 Hello, all right Okay so Welcome to Drupal Drupal 8 deployment that continues integration Today I'll talk about continuous integration and Why do you why should you care about it at all? and I'll try to take a look about What do we have in Drupal 8 and what tools will help help us there So my name is Amin and I'm I'm the CTO what? Okay, I think I should stand here. I'm the CTO at Analym We're a Swiss based company and we do business-to-business mobile web solutions and An enterprise Drupal We also recently far launched something new which is Which is called safe Swiss cloud and it provides developers with virtual data center capabilities But I will talk a little bit about that later So there's three things that will cover in this talk First is why should you care about continuous integration? the second thing is I'm going to share some of the tools and practices that we've used to achieve some of Some of the goals that we had and I will also review How how Drupal 8 is going to help us and be a better tool for continuous integration? All right, so why why do we bother with continuous integration at all? So how many of you are familiar with the situations here? anybody just you okay so you you we We had this issue also in our company and we had to spend like Hours to prepare before we go live and bringing configurations from all different We have distributed team by the way, so We had to make sure that everybody I'm audible. No, so we had to make sure that everybody Puts the configuration into the right place then we gather all that stuff then we go and apply try to apply in these configurations manually of course it breaks all the time and After we're done Most of the times we spend another week of fixing bugs that are related to the deployment itself It's not like not functional bugs. It's things that arise because we deployed this thing Not a very good way So and we have we had to deal with So if anybody remembers that our website that was so fragile and broken that you know at every time you look at it Basically breaks This wasn't deployed properly Yeah, yeah, all right, I think this is better Anyway, so some of the sites are so complex and you have too much You you've already spent a lot of Time working in it and the configuration change from time to time or change a lot And then you have this fragile website that you can never get to work properly and Then there's this when you try to integrate Basically the whole integration hell doors open Anybody have this situation nice scene, right okay, so So we decided that we don't want to change that and we decided that we want to deliver faster and actually Increase our quality. So not only Not not one of these but do both at the same time We wanted to spend more time developing functionality for our clients not working on bug fixing and Especially bucks that that shows up only because you integrate the sites into a live environment or because you you're doing some deployment And of course we wanted to get a happy client and one of the team to stay relaxed and be happy and not always trying to Do bug fixes and stress stress out So How do we do it? So the first the first important thing that you absolutely need to do is and this is basically trivial you need to maintain a code repository and There's no other way you can go around this if your code is is Somewhere that where you you don't trust the place where the code is then you're basically not set up and you need one one also one important thing is that you need to have a branching model and your developers need to actually understand the branching model and You need to make sure that everybody knows what to do and which which piece of code gets gets committed to which branch We use get of course and we also found that using visual tool Which is called github? Sorry get lab, which is very similar to github, but you get to install it on your own premises and just use it It's pretty pretty useful and pretty Provide some some at least you can share links of commits and so on and By having this branching model also you can They cut the collaboration of the team will improve because they all want to they all know what's what they are doing There's no code. That's that is go that goes to life or Or is submitted? You know we without any good reasons All right, so the second important thing and this is also very important is Automate automating the build and What do we mean by build? For for us when we say that we automate the build basically it means that The build should be able to get the code from somewhere from a trusted place Then you can do some checks on the code Run some tests like unit tests or whatever Then you use a database with proper data And then you maybe do a couple of things like revert the features of Drupal or do things and at the end you need to have Working Drupal site. So that's that's what we call a build Another important thing with the build is that you need also to know understand Where are you putting the build? What's what's the environment that the build is going to run on? so So basically you need to care about How PHP is configured how a patch is set up and all this stuff needs to be also part of the build Because if you build something on one environment, it can work on another environment It can completely crash and break then we have What we've done is that we we've done our in the daily development and integration into a lifelike environment and This is also very important. You shouldn't be Deploying or integrating into an environment that is completely different than the than the live because after that you'll get Obviously you'll get into trouble Yeah, so once we automated the build we were able to build the more often and Basically what we what we try to do is to make the build also run the tests and Sometimes tests or actually simple tests are very slow So we're not like building them always so our build is parameterized We can you can say to run the test or just to skip the test if we're if we're running the build on a Test server we just run all the test if not if you're not trying to build it locally, then you just skip this yeah, and After so it's also important to run your test on a clone of production and It's important also to test your deployment procedure itself. So Before you deploy on on production on the live site you need to You need you basically you can check out the whole code and bring the live database to a test server And then you try to deploy there over and over again until you're sure that the deployment is okay And then you can move all that and do the final step on the live site Yeah, so that's automate deployment and basically you need to it's always good to be sure before you go to the live environment you need to Practice it over and over on a staging server, which is basically a clone of the live environment Another thing here is that you We we automated all our migrations or any it depends on the project, but if you have any migrations or any Content that needs to move from one structure to another structure. You should also do this all automatically and Never let anybody go to the live environment and actually do some clicks in order for some things to work because This is basically a potential cause of issues. So, yeah our results what we've achieved is Now we are doing one click deployment You only need to run one script and the site is Can be both built and deployed to the live environment We were spending like three hours before that gathering the configurations Gathering preparing making lists doing all sorts of manual stuff and Now it all can be done in 15 minutes, which is basically the time that it's taken most of the time is taken to To clone the database from if we're deploying it is on staging So we just clone the database from the live website We sanitize it and then we run all the deployment on a real production database Yeah, so after that after we we've done all that and because we we were able to always test The deployments and the builds over and over again once we once we say that The thing is ready or the website is ready to go live We barely have any maintenance and or any any kind of bugs that that the client discovers and Which which also means I've it means a lot for for the team and for the levels of stress and the team again a team focus shifted from bug fixing and all doing all this annoying annoying stuff to creating really useful features and It just feels much much more better than to do new features And not you know get stuck in bug fixing So so that was continuous integration and why you should use it And some of the results that we achieved now I want to talk about Some of the tools and the practices that we use and how we actually Achieve that Yeah, so I'll reuse some of the tools for Drupal 7 Because now we're building in Drupal 7. I'll mention some of the limitations and problems and after that we'll go To Drupal 8 so here's a bunch of tools that That basically I think most of the people familiar with I mean the first line Update hook it's a powerful thing, but it can be bit complex features anybody use these features All right, anybody completely happy with features Okay All right features is a very good is a very great module and it's I think it goes like to 90% of the configuration management Think maybe 95 but it still have its limitations and some of the limitations are for example You need to have exportables your module needs to have exportables in order to To make it work with features And basically features was built to to distribute Features not to manage configuration or not to deploy feature have five of our ten things and now the clients The client want to edit or make changes on the live site with a feature and all these five things become Always bound in the same feature So what we do is we unlink things from from feature like we remove stuff from it and then we can Export it back or we can just continue working with only the things that that we we do need to To change and this unlink functionality is available in a module called feature tools right so we also use drush and Drush basically is the Swiss army knife of Drupal The the important thing with drush is that you can really do a lot of Automation just by putting a couple of drush commands into a shell script So here here are a couple of commands that we use and Yeah, so drush can enable disable modules and features Which is probably everybody familiar with Put the site on maintenance or set variables, which is also very useful if you want to Set a variable then and set it during the deployment or do some more complex logic with it Very neat feature is the synchronizing The data and sanitizing it so you can really you can get the data the database sync the database from production server sorry and if you use this sanitize switch it will go and basically anonymize all users data on your database and This is basically done to prevent to prevent spamming your Your clients or whatever, you know emails or that in the database you have It just anonymize everything You can always sync files from from the production side, which we also found very important You do need the staging site or the even the development site to always look the same as as the live site Then you can evaluate code with with drush So basically this is also very useful. You can just write some Drupal code within After drush EV and then you it will get executed and then the same way you can You can execute SQL queries And of course you can extend drush and write just about any complex functionality or any complex PHP logic that you need a good example here is for example if you need to Do some migrations in your website? You can really control that very well with a custom Drush command where you where you go and change some structures or move some content It will be part of the deploy Yeah, so I talked about the importance of environments and For for environment scripting or for making sure that the environment is always up to date with the code We use a tool called the salt stack Anybody familiar with salt stock? No Okay, one two So you should you should really check this out. It's it's very easy to install and The basic idea is that it's it's a configuration management system that uses Data files also like it stores the configurations in YAML and it's a remote execution System so you can it's pretty much like I think like puppet or puppet or chef you can set it up with master or masterless way and What it does is that you can so what we do is that we when the environment when an environment is being set up or booted it asks for For its configuration from a salt master server that we have The salt master server basically Basically is a centralized place to keep configurations So we have basic stuff like make sure that Apache is on make sure that PHP is configured make sure that there's enough run for PHP and so on and Also, the salt master is smart enough to go to the git repository and I'll actually check code Check the code of the project that is being deployed now and add things on top of that so a good example of this is a Some of some of the developers use some functionality that needs a library like the CURL CURL library, maybe and When the developer commits the code that uses the CURL library can also in the same in the same project Just go ahead and modify the salt Salt configuration files and Any build that will happen after that will automatically install the library for you on on the environment so it's kind of it has a genetic Configuration and it also can go to git and bring some project specific configurations Yeah, and salt master can also push configurations To already running environments so you can basically go ahead and push some some new things to your production or staging environments So in terms of infrastructure one more thing that I want to share we use the vagrant and cloud stock and Basically what we do is that we use vagrant to make sure that all development happens on similar boxes Anybody familiar with vagrant all right cool, so We basically have images that we pre-built and then the same image is used to fire up Vagrant boxes and we use this image also on a cloud infrastructure that we built based on cloud stock and This infrastructure basically is where we have a development integration server. We have the staging and production So the same basic image is used After that I as I just mentioned the salt master goes and make sure that the environment is is Completely ready by checking git if it had if it need to install any extra stuff on it on the environment and Yeah, and after I know only after that with the Drupal star we start building Drupal on top of that we also have Jenkins which also checks git and does some automated the stuff on our productions there cloud on our cloud infrastructure like Into checking out the code to dev always running tests Running tests on staging and so on and it gets some report and report back Yeah, and so the infrastructure that we've built actually is It's quite good and we are trying now we are selling this or we're providing this infrastructure as a service So if you're interested go and check it out Yeah, more tools So we use Jenkins that I just mentioned. We are exploring with thing thing can be used also to build right now we're building with shell scripts and drush scripts, but thing is probably a more powerful tool to do that and Of course for testing we use simple tests And we use selenium to do kind of a smoke tests on top of that and The problem with with simple tests is that it's pretty slow So you won't be able to most likely won't be able to run it on on your development environment Or your local development. Yeah so Basically the main limitation that we have now is Is configuration management with features we get it we get into troubles with that and the lack of integration into external tools which may be the WF to Zia module can can resolve that so what about Drupal 8? What should be what should we be watching for in Drupal 8 and of course Drupal 8 is still a work in progress but the unsurprisingly the most Anticipated thing in Drupal ends in terms of configuration in terms of deployment is the configuration management initiative Which is basically a core API that That It's a core API to deal with all configurations so you don't need to to rely on any exportables It's file based it uses YAML files and so therefore can be versioned so all your configuration can be Can be put into a version control system Yeah, the configurations are cashed into database in the database and Yeah, by default The configuration manager module. It's pretty simple and have a very at least what you get now when you install Drupal It's a very basic workflow So this is this is an example of a workflow again. We have kind of the same as the features we have The database we have an active directory and a staging directory and both these directories actually reside in in a subdirectory of files and The subdirectory is called the config and it's followed by a hash So what you what you get immediately when you install Drupal 8 is that you can basically export from the development environment if you export your Configurations it will provide your kind of a config dot tar file you export that and You can you can go ahead and immediately put it into the staging Just unpack it and copy it into the staging subdirectory in production and then when you're ready you can go to the interface and just sync it and Then basically your production system will switch between the active and we'll think Will accept all the new configurations The thing that we we I didn't find any any solution for is that Once or maybe not that convenient now even though the documentation mentions it is that having this hashed or Hashes in the config directory will kind of make it a little bit hard to To put in git so we have directories that that are under the files, which is already kind of a not very Convenient ways. I mean we don't put in our projects. We don't want put the files directly in git We don't track the files that thing in git, but even if you do that you will have A directory with hashes which may change from environment to environment and therefore Yes Yeah, yeah by default. Yeah, but but you will still have the hash probably will still have the hash because it's built into there in this For security reason It's it's not like every time, but it's not You cannot guarantee that it will stay the same through through all your environments So one way to deal with this maybe writing and drush command that will bring the the hash or be able to figure out the the hash from From Drupal and then deploy things but so far this is kind of a It's not it's not It's not used yet and therefore that is why you know, this is a good place to write contribution modules so But that's that's not all with Drupal 8 We have We have several new APIs in Drupal 8 that will definitely help deployment and several new modules features 3 is going to be written on Based on the new configuration management API and Maybe it will have some configuration tracking, but at least it will it will use the configuration management API to Package things as usual and just distribute them There's another API which is called the state API in Drupal and it's basically It's basically a place to store configurations or information That is not That is related to the system state like for example, you can use this for build numbers of of the of the Of the site you don't want to migrate or push build numbers from one Or maybe not build numbers. Maybe the last time the Cron run there's something that some variables that you don't necessarily need to push from development to staging to production All these things should stay on their Environments another thing is composer. So you can this will definitely help managing dependencies and differently help automating the build So we didn't use I didn't see any examples yet for Red to the state API And we're not using composer yet But this is something that we definitely will look at and this is this should help us automate the builds Yeah, one one more major thing in Drupal 8 is the shift from simple tests to PHP unit and Basically, this means that we can write more faster unit tests and probably benefit for test driven development in test driven development practices because If you if you're building tests and they are very slow, it's it's not going to work for you If you're trying to do test driven development, you need to write your test Write some code and then always try to make your code pass this test and we get We get coverage reports also with PHP Unit so to do is for Drupal 8 is basically the next steps are we we need to finalize the APIs as mentioned as was mentioned today by Dries and We need to start writing contributed modules on top of that in order to basically to make sure that the APIs are suits what we what we need and We also need to take a look into integration with different external continuous integration tools so conclusion I Hope everybody will go and do some continuous integration It's really important it helped us a lot And it's it's pretty easy to start it's you don't need to To do complex things if you just try to automate the build as a first step with this will give you a lot of opportunities and possibilities Then Drupal 8 is it's pretty good I think it's very promising and a lot of issues will be It's it's still in progress, but a lot of issues looks like that it will be resolved with the Drupal 8 with contributed modules Using of course the APIs And And we all need to basically to go and experiment with all these APIs and invent new workflows for For building this so thank you very much