 I think we will start slowly. Welcome to our session, thank you very much for coming. I hope you have enjoyed lunch and I will not make you too bored so you will not fall asleep. So today we are going to talk about continuous integration practices and some things that we have developed in our company and that we actually open sourced and how they changed our development life and how you can actually do the same for your organization as well. So let's start with introduction. My name is Yuri Gerasimov. I'm working in the company FFW. I'm working as the developer, technical lead, and architect, so I'm mainly involved in development. And this is my colleague. My name is Andrey Podonenko. I'm a DevOps architect and mainly I'm a developer as well. And let me do some introduction to our company. So we are pretty big agency. We build a lot of websites, mainly in Drupal. And lately the problems that we were facing because we have so many offices and so many development teams that every development team does their own thing. And this is kind of join of all best practices across our company that we're able to gather. So first of all, I would like to do some perspective of my career and I have worked in multiple companies with different teams and each team basically works in their own way. And I would like to talk about how that was and how that involved to our continuous integration workflow. So usually development in Drupal has two pieces. So the first piece is the database and we know and we love Drupal because it's so configurable and flexible and at the same time we hate it because it's so hard to move the configuration around. So one of the things that we need to take care in our development environment is moving database. Another thing is the code, of course. Usually development infrastructure for small teams and some agencies is that we have one development environment. And what means that there is some server, it can be within the company office or it can be cloud solution. It's somewhere where all developers are working with. So they do all their configuration there. When they work with the code somehow it's got updated and this environment it's pretty easy to set up and there are multiple ways you can work with it. So sometimes you have one big server with multiple projects on it. They can live on the same domain but in subfolders. Sometimes it's the different domains but the same server but anyhow physically it's the same equipment. Also some companies using the other solutions that are on the market. For example, Acquia, Pantheon, there are multiple providers that have some support for development in Drupal and this support is that you can have production environment, development environment staging. So you have some tools to work with. When you work with the code, well when we used to work with the code, I'm coming from time when we were working with CVS a little bit. I mainly was working with SVN and because it's the tool that it's much better than of course pushing your files with FTP but still it didn't have that flexibility of working with branches. So basically everyone was just pushing their code to master and it was like pretty fine, pretty convenient. If especially if you have some CIS admin in your team they were able to have this post commit hooks in your repository. So every time you push them code to the repo it got reflected on the Dev side. And that is working great when before you actually start facing some problems. So some of the problems that you can have. When you push the code it has some bugs. And in this case you really need to fix those bugs before doing the release. And we sometimes were able to manage to get the code to the state when it's stable and we were not pushing the broken code to the master branch. But still if the sprint is pretty long or functionality slips into multiple sprints it's too easy to make some bug in the beginning and then in the end when you need to deploy the staging environment for the client to test you find out oh we have some bug and it's like not very good for client to see. So you're really failing the deadlines. Or of course you can work over nights and it's pretty stressful fixing bugs. And you can plan your sprints to have like okay we develop five days and then three days fixing bugs or maybe two days depending on how good your team is. But still it's you put yourself on the risk. And this is something that is not very good thing to do and especially nowadays we have some tools that help prevent this. Another thing that is important in development process is actually code reviews. So it's like the very first and probably the very best thing that you can do to improve your code is to make the code reviews within your team. And this is pretty hard to do if you just let everyone to commit to master. Because you can do the code reviews but you do it after things got committed to your stable branch. So in this way you probably can catch bugs but you cannot prevent them to slip into your code. And also it's kind of like you cannot block the bad code getting into your master. So you don't have any mechanisms to actually prevent that to happen. So still you can organize your sprints to have like one day of factoring two days back fixing and then you are like semi-ready to get for the release but still it would be no much nicer to have something like you block your bad code to get in the database. So database it's something that we all deal with like in Drupal 6, Drupal 7 we of course can use like features but not everything can be exported there. So we used to work with like let everyone to configure just like everything on the dev side. Again, that works fine until somebody actually breaks another person's work. Like and it's very easy. Like you can have a view, you can have multiple view modes, oh not view modes, displays, right? And then somebody is working on one view, somebody's responsibility, another display and then it's very easy not to check the option to override the settings on this particular display and here we go, you have damaged the work of other person and it's very bad to notice that actually when you are in the end of the sprint and you are doing deployment for the staging. So that's one problem. Another problem is that when you would really like to experiment on your local environment and then push everything to staging dev and production, you can do this but you need to repeat all the actions there or you need to play with exporting to features. So in some cases you, well the recipe if you do any configuration on dev environment you basically do it wrong. You will not let other people to review your work because they will review only your code changes and you really cannot increase the code quality. Some things that we improved in this process, we were able to deliver some scripts to actually pull the database from dev and staging environments to the local environment but that was actually it. Usually we just installed Backup and Migrate and it was working fine. Just remember to disable it on production. Some problems when you have single site I mean single server that you're working with on your project so because you can have multiple projects they are actually sharing the same resources. If you have like one team that's fine. If you have two teams probably it will work. If you have like five, 10 teams working on the same server they will start fighting for the resources. That's for sure. A lot of projects they need migration for example and you need to rerun migration from time to time on your staging environment. This is pretty intense process. It can put your resources down. That's like one thing. Another thing caches. So sometimes you work with memcache right and if you have multiple sites working with same memcache instance of course you can have like keys and stuff to separate caches and that will work. But you will definitely find the person who find more convenient just to restart memcache on the other server to clean the caches. And that's like not that bad but sometimes you will just put other teams little bit down. They will have some lags on their work. The worst thing is the Apache Solar. That's the thing that actually eats memory and if you will let other people to like do some manual configuration they will need to restart solar and if you are indexing all your content during the migration and they restart solar it will just fail. So you will have some cases when you will really start fighting with other teams like you don't have granular control over your development process. It's getting very bad when we are talking about varnish. So if you have varnish configuration it's getting because varnish is configured like globally per server it's very easy to mess it with other projects. So especially when you have multiple size that need different configuration of varnish this is just like it's very hard. So if you need to configure varnish it's much safer to just go to different server. That's for sure. Another problem that is not very technical but it's more organizational wise because usually when you have one server and you can really mess with other teams work you usually have like one person who is a system administrator of that server who don't give any pseudo rights to any other people maybe team leads but he is the man who actually manages everything he knows the configuration he knows the implications of changing them and he is doing the releases to other environments and it's all good if he works fine and he's capable to work over times and things like that but it's very bad if he got sick. So if he got sick like all your teams even they just cannot do any releases and he is like single point of failure like your actually development server. So this is the practice that you get into in this environment and like other developers they will not like to get into this and then you are in trouble because of you have only one person. Well, if he will got sick it's like he can work from home and stuff but if he will resign or leave the company it will take some time for the new person to actually understand the things because there was like 99% no documentation so that's like normal and I would like to talk what we have changed in our development workflow. So things that we have changed the local environments because every developer has his own preferences in like what operating system to run and things like that it's sometimes very hard to have some consensus of okay we have our development environment PHP 53 or 5.4 and then local environment of that person installs with PHP 5.5 and he doesn't really want to spend time back porting the ports and alternative sources and play with that and another person can have one version of solar and libraries oh it's even worse when it gets to the SAS compilers I'm personally not a big fan of finding well with fighting with all those dependencies it just like fails for me but it's lack of my knowledge so but I know that team members can have similar problems like me so like when you start having these issues with multiple environments you start spending time like running around and making sure that everyone is working at least similar environment and if you find the bug on staging environment it can be replicated on other environments as well so that's like problem number one problem number two is actually introducing new people to the project this is something like we have big teams right we have like 400 developers like huge and when our team is like I can see that oh we are failing reaching the deadline for the project I start shouting and running around like we need developers and probably in few days I get someone and the deal is usually okay this person is coming in like how much time you need for him to help you and you say like yes like probably one week and then he starts working on a project and it takes like two days for him to actually set up the environment and this is something that is very bad so we have switched to using virtualized environments so we are using vagrant boxes and we develop the set of scripts that let you set up the vagrant box so we don't send the box itself because it can be like five gigs easily and we have remote teams so passing five gigs especially updates to those five gigs that will take same time then just configure the stuff from scratch so we just have configuration in we used to use puppet but now we use ansible but it doesn't matter I mean you can delete your box you can restart it and it will configure it for you so that was like huge improvement especially joining people who are using Macs, Linuxes, Windows, different versions of Linux like everything that was great currently we are using Ubuntu 14 we used to use Ubuntu 12 but it really depends on what your requirements are in our open source project this is just by default but in vagrant it's pretty easy to change the version so it's mainly so this decision was made because we are using Debian on production environments but it was just working fine on Ubuntu so we switched to it another thing we have changed now a development environment that we don't do any database changes in configuration like none and it's very important to have that because you can actually do the review the code you can see the conflicts of the developers if they are touching the same pieces of the site so we are using of course features and we use hook updates it's all this code-driven development model that is around since I don't know a few years already and this is very important another thing that we do we have two kinds of development workflow so when we start project from scratch and we don't have any content we start with installation profile so we develop everything from scratch we just reinstall the site all the time so every time I do commit and push the code and if it's got merged it will be deployed to the staging environment it will be reinstalled from scratch this is very convenient much better than just pulling the database around with the content and so in this case we take care about the demo content we can control it usually we don't use the same demo content across the project so we try to get images from the old website of the customer so they can feel that it's something that they are going to get in the end and we sometimes pull the content from their existing website or if their business is I don't know about stocks or some nature we just pull articles around and they can feel that demo content is relative so and regarding the content we deploy it from the code I know that there are multiple solutions there like features UID probably you heard of and there is others like demo content something but if the website is simple like just notes not that many references around exporting the content into features is fine when you're getting a lot of references especially entity references and like taxonomies and taxonomies with other references like you can have like multiple relationships between entities this is where you really need to control much better how things are working and it's you will get the bugs in features UID so currently we just use our own code so it's just the module and using hook install we just deploy all the content in the way we like and another thing if you're using penalizer that's probably only the way to go code this is probably the first thing that we have started to actually changing so we really like to work with the GitHub and the best thing that we have there is actually pull requests so with GitHub pull requests it's very easy to the code review and that's like the main idea of this workflow so we're using pull requests and for every pull request before it gets merged to the master we do the code review when projects started usually like team lead do the code reviews in the beginning like to ensure that architecture is like followed but currently we have all our team members pretty mature and everyone understands how things should work we have some kind of standard so everyone is doing code reviews of his colleagues also we do the code style checks and tests runs so on every pull request we actually check if the code style is met we use the Drupal coding standards and if something is wrong we just post the message about the problem and person who is creating pull request he needs to fix them before actually asking anyone to do the review we do the security test so it's another scanner what is important for the code review code review can be pretty painful in the beginning because it's like we are all friends we are all colleagues but when we start pointing fingers to each other like this code is bad and this one is bad like people can get it a little bit personally especially like if we are colleagues and we were working for a long time that's like perfectly fine we will have a beer on Friday and like all problems with code reviews are resolved but if it's like new person coming in from the other team who doesn't know these guys and then first his pull request get like 10 code reviews reports and he will like oh probably it's a wrong place for me so it will take some time but that's like really a good way to improve the code and especially when you have experience from the other teams like very easy example like there are ways of like if you're building multilingual website you build it a little bit different than you build like only one language right you take care about the references you make the decision about whether which model of the translation you are going to make and you need to remember wrap everything in T function and things like that when you don't do this and you don't have much experience with that probably you will miss a couple of points if you have another person who is coming in and doing code reviews and he has his experience and yours experience you really will exchange your experiences pretty fast on the code level so this is very good thing not only for the increase in the code quality but also for sharing the knowledge so we had some cases when the people from other teams were getting in and they were telling me like yes that's good but it will not work for multilingual site and like we have this problem we build the website we were working on it like for a year and a half and now they asked like oh guys can you translate it in Spanish oh yes no problems and like that's the moment my problem started right so because we were not following some practices and we didn't build this multilingual websites we had the issues so it's very good to share the knowledge in the code reviews and you need to be open for the code reviews so it's not about just like saying to the person oh this comment has typo you need to be constructive so remember that if you are pressing too hard on that person he will do exactly the same to you so like this is kind of balance we already have on our team but if you don't do code reviews this is something that you will need to work on as well another big thing that in our process when we do a pull request we actually beside of doing all these checks and code reviews we actually build the website in specific folder on the build site where people can actually go and test how things are working and that's great for multiple reasons so first of all the person who is doing code review can ensure that things are working and then we also can let our QA people to check the site and this is when also we are removing a lot of things out of the release cycle because like one developer sees the ticket usually like if ticket has like five points he got over excited about first three he implements them and two others he just skip I mean he's so happy he managed to solve this hard problem he pushes to the pull request and he has another colleague who is also developer right so he will read it in exactly the same way so he will be very excited about first three points and the problem that is being solved and nobody will take a look at the fourth and fifth and QA person on the other hand he is not excited I mean it's just like piece of functionality he needs to fix and to check so he will be this boring checking every single step of test process and he will catch bugs before you actually merge them to the master so allowing QA on builds that boosted our quality level pretty high some monitoring that we are using on our build servers that when we work with them so we do the monitoring so we use Moonin uh... we also deploy we're working with dark a lot and uh... with other hosting providers as well so we have some boiler code uh... baller plate to deploy things to occur uh... what we also do on our servers and in our workflow we do the visual regression testing and I will talk about this bit later and uh... also we check when after the big migrations we also check if the site is actually working and all your roles are working on it because sometimes when you migrate like fifty thousand pieces like of the content you will not notice of this infinitive red directs this is kind of nasty problem and some other things as well visual regression this is the tool that we are offering as the platform it's available on the internet you can log in there give it a try basically what it does it creates screenshots of your web pages the ones that you have identified so we can scan your website on four URLs and then you can do comparison of the screenshot sets so we can do the set of screenshots you can do the release you can do another one and you can compare them and then you will see what pages have been changed uh... also nice thing you can actually compare the environment so we can do the set of screenshots for your production for your staging you can compare them and then you can find what changed uh... this is something that we uh... using currently on our project specially during the releases to staging and then to production and you're welcome to register on the site you have like hundred URLs free and give it a try and let us know what you think maybe we will definitely improve on it uh... also you can do authenticated users some like HTTP access pass through so a lot of use cases covered this is how diffs look like for example uh... some URLs health checks so this came from the problem when we had to migrate the old website that had a lot of images this website was about nature and birds and people were so excited about like birds and like uploading huge photos to the website without thinking that actually twenty megabytes picture embedded to the pages not the best idea for your mobile phone when we migrated the content of course we pulled like everything and uh... we needed to check this twenty thousand pages like if actually we imported something bad with these huge images so we wrote a scanner that was checking the responses of all the pages we were we were analyzing the site map and uh... we were also evaluating the size of each page with the all in line images, JavaScript, CSS all together so we were able to actually find those twenty thirty articles that were like too heavy and uh... it's written in goal language and it's like hundred to two hundred lines script so if you have some large migrations give it a try it can be useful and uh... now we are going to talk finally technical and uh... we will talk about the actual technical details how you can install and use the CI box uh... this is the solution that we have outsourced open sourced and this is what something that Andrei is going to talk about thank you hi guys so welcome to CI box uh... you can follow the presentation by just going to this link it's uh... link to github repository where you can check the files that I'm going to talk about so what is CI box? it's not an application it's a really weird product because currently it's a bunch of scripts that uh... was hardly tested by our team for a lot of projects and it consists mainly from two uh... parts these parts are Ansible playbooks if you are not uh... familiar with Ansible it's a provisional tool uh... that uses YAML for scripting very handy uh... style like uh... name of the uh... name of the step and command that should be executed and conditions like when it should be executed so uh... uh... CI box consists from two two parts like uh... the first part is the script for spinning up your continuous integration server uh... along our development process we decided to have uh... one server for a project so when we have a file server five projects we have a server for every project itself and with Jenkins box you can just run it with small configuration changes and in approximately fifteen minutes you will get totally working in Jenkins server with pre-configured configs and another part is uh... actually uh... Project 3 Initiation playbook that will prepare for you right now by default is Drupal 7 with all the files in it uh... like uh... the scripts that will be used for uh... provisioning for uh... running tests for instance in your site uh... so how to deploy this uh... product well this product is like an infrastructure installation product so first of all you should prepare your server it can be server like digital ocean uh... instance for example or you can use your internal server hardware server or even virtual machine based on Ubuntu long term server long term support release you should change two lines like uh... IP address of the server and run Ansible YAML that will prepare all the software uh... into that uh... box next uh... second step you should uh... initiate your project well if you start your project you will start from the uh... Drupal like Drupal core then you will put there some models but from start it's like an empty Drupal with barctic theme so this github.yaml will just uh... grab the latest Drupal from Drupal org put there all needed scripts that shipped with our product and uh... you will just need to get in it and get push it to the github it can be private repository it works fine with them so and another and the last part is you need to make change to Jenkins like put the credentials to github or api key to hipchat if you use notifications and vice versa and after one hour you will get totally working uh... infrastructure that will do all the stuff you talked previously so what we currently ship with Jenkins box it's like a lamp stack uh... with optimized configs and SSL support we have there a bunch of sniffers with standards from you might know the model coder uh... we check in compass files by using SSS lint it's like a big pain for our frontend developers because they don't like this tool but helps us to improve a lot uh... of quality for the frontend also we are using security linters then you will get the stable version of Apache Solar with already preconfigured Drupal configs from Apache Solar model from Drupal org uh... Selenium and Behead testing and uh... also after more than a year of development we shipped the preconfigured MySQL config because you know that uh... you need to obtain results from scanning and build site very quickly because you shouldn't wait for a build site for one hour right now our uh... Drupal sites spin it up with from one to ten minutes and of course the Jenkins with needed plugins and preconfigured configs uh... as well so when you first open the Jenkins UI after provisioning you will get only a few jobs like pull request builder actually this job will do all the stuff uh... around handling the new pull requests within GitHub uh... also there is a skeleton for deployment plans like out of the box you will get ability to test current master branch by putting the IP address of your server slash demo and there will be always latest version automatically rebuild it by if there are any new changes to master branch and also you know that servers uh... that you're using for say infrastructure is not so large and if there is no more space there is a server cleaner which will uh... trigger the space script and uh... clean up all the tasks uh... one time per day and uh... about uh... uh... project structure we stick it to some uh... structure that is uh... very similar like to the structure that is for aquia product product uh... aquia hosting like you need to put all your drupal files into doc root and uh... uh... we have all our provisioning tools for the grand machine uh... in inside provisioning uh... folder and there are some tests you can put their behead testing that will be executed on if you will select optionally for every builder or even only for demo site and there is a vagrant file with config so uh... when you start to work with uh... new project you will just need to clone this repository and press vagrant up and uh... in the middle there is a list of scripts that we injecting into the drupal uh... code base like reinstall script uh... sniffers tests so this also are uh... ansible playbooks uh... that leaves with the drupal code base in your project and uh... they will be used for uh... running all that stuff with spinning up build site running tests running sniffers for github and inside the box folder uh... we have refactored our code base for ability to approach not only drupal like drupal seven in drupal eight is a totally different systems and if you want to work with drupal eight you just need to provide uh... some environment variable in uh... new ansible script like uh... uh... in drupal seven you need to clear cache with one comment in drupal eight you need to use another comment and all other stuff is just going to be used for both system as we equal and a few words about vagrant box right now we decided to use same uh... scripts for spinning up c i server and for local environment like uh... we have to be confident that the sole server in your local environment is totally the same that will be used for testing uh... builds on c i server and it should be the same as for production server so uh... we use uh... same scripts for for provisioning our local environment in remote environment as well and you might know that some systems like windows didn't like uh... ansible and we are using a trick we put in all the ansible uh... scripts inside virtual machine and run them there so now you can use our vagrant box like in open base d even because if you can start vagrant and virtual box you will definitely get working version of virtual machine so just vagrant up and you are ready to go coding now let's talk a few words about developer point of view when you start work first time with this system the first touch to continuous integration will be like this comment in your preparing pull request and in two or five minutes you will get comment like this with links to sniffers uh... that uh... uh... where run on uh... over your code with jayce hint and the last link is the link to the build site itself so uh... you can open this link and test uh... how your code work on the similar environment as you bought uh... if there are any issues you will definitely catch them there and uh... how it looks like a big picture so you'll get a new task from your team lead and you start coding adding files to repo right now we decided to work uh... every developer has its own fork or main repository it's mostly cure uh... because sometimes developers just jit push and overwrite master uh... branch so now we decided to use new fork for every developer and uh... developer can do everything he wants in his own for his own fork and uh... can break master branch after that developer when the task is finished uh... he prepares the pull request and he is a part where the c i server start to work c i server uh... grab the code and prepare everything you've seen on the previous slide then developers should wait for the comment check if there are any issues with sniffers linters if if there are issues he should fix them and after all the issues fix it he should send his work to uh... another developer of the team and took ua for testing if the task was done as expected and if everything is fixed after that only the task can be measured into master and uh... there's a very uh... nice steps that we uh... added to our process is a steps for review the guy who knows how the uh... feature some particular feature works is a developer that uh... current that definitely did that it's called so uh... we decided to create some administrative step like creating steps for review for example if you works on some uh... slideshow you know how to check it for example you just write these receipts like go to these enter these save the dot and issue and she you should see this and with the steps for you all of most of the box uh... catch it by developer itself and uh... after and with the steps project managers uh... are very happy because uh... for during preparing to demo to client they just copy paste these steps and uh... uh... there are no bugs because everything already tested by the guy who did the code review the qa and the guy who did the uh... task as well next house c i works uh... it's a bit of technical information let's imagine that you are jenkins server so when there is a new pull request uh... on github jenkins uh... has a custom plugin we have made a few changes to the source code of that plug in for ability to post all that links to uh... github comment and this uh... plug in triggers he found that new pull request is coming uh... then uh... here and this plugin uh... runs uh... the script the script reinstall dot yaml this script is in the code repository so uh... uh... after that after uh... he runs reinstall or sniffers depending on your configurations you can uh... at first runs nifers but after that if there are any errors you can skip uh... running build because if you want to get results uh... fast for for example if your project is huge and rebuild takes uh... uh... a lot of time you just run sniffers and if there are any errors you stop there but uh... usually uh... after sniffers we run in reinstall and right now i'm talking about profile basic flow when you reinstall site from the profile in drupal so the reinstall part is like drupal uh... site install and profile name then if the project uh... builds successfully then there are test scripts that uh... runs and if there are any issues uh... you will see the link to test results and uh... also the Jenkins can uh... handle comments so you can talk to Jenkins like if you want to get two or three builds two three different sites for this pull request you can just write retest this please retest this please retest this please and you will get three links to unique sites for testing and also uh... we have a SQL-based flow this flow is more uh... universal because you know that only drupal has profile installation and if you need uh... there is a part in uh... every project there is a point in every project when uh... content managers become to put new content into the uh... site and you can't just reinstall your site because content will be lost so uh... just by changing one variable uh... inside reinstall dot yaml you will get an ability to uh... change the reinstall part with different uh... kind of uh... installing like grep the database from production uh... imported into the unique database run registry rebuild clear cache change variables uh... clean sensitive data and vice versa and all another part is totally the same so this workflow can be used even for non drupal uh... cms is like we were used it for uh... symphony for big tree cms for wordpress anything that will feed requirements like php project and database for it and uh... during using this workflow we got some rules uh... like uh... uh... one day we uh... face it with the issues that github block it our uh... both user both users that it is a user that using for posting that all that messages to your project from continuous integration server so we decided to have uh... both user per protein and also uh... we are using one c i server per project because uh... when the project is uh... uh... in the final phase you can know that uh... you know that uh... there can be fifty or sixty bills uh... per day so it's really busy and not convenient to test uh... sites on the server when you are trying to use multiple projects for one server also uh... there is a rule for development like you never should match your own pull request because you will break the rule of manual code review uh... also you need to add steps for you because it's help helps a lot for qa and minimize the time for it uh... and uh... uh... for some teams uh... we're using uh... uh... selection of who will do review for your code like for our robin principle like for one task i'm selecting first guy for another time i'm selecting second guy because uh... we we want to improve knowledge sharing in the team and it helps improve a lot and also there is a rule never push directly to mine rep a master right now get up uh... started to have an option for protecting uh... branch so this rule like can be uh... enable that checkbox and everything will work okay also there is a tough rule like you need to keep two siblings of every role in your project because when you have one senior and one junior is not an easy for junior to review the code of the senior his hands will be like this and uh... for responsibility it is better than when you will uh... assign a task uh... over back to the guy who did review for the code this improves responsibility a lot and few words about responsibility shift you might see you might see that uh... all the scripts that are using by jenkins i'm sorry the project doc root so every developer can create pull request with like adding new step to reinstall script for example for cleaning some table or enable some model and uh... after merging every everyone in the team after pulling the code will be used will be used this script so you don't need to have ops in your team ops guy is the guy that knows uh... really well how the servers work but he doesn't know how the development is going so he's not aware of what the face of the development that's how we uh... decreasing the gap between dev and ops we uh... growing a lot of devops in our team by using this workflow and of course like every product uh... this product has bottlenecks like uh... we have dependency from github this see i can be easily uh... change it to use for github beat bucket we already created the plugin for beat bucket for ability to post there that links for comments so when github is down you can do only local development without code review and second back is the fc i server down uh... teams get stopped on the pool pool request uh... pool uh... on code review step so you can continue to work but uh... you uh... have no that convenient feature like build new developers should follow the rules you reset a little bit about that and it's can be really tough for new developers because there are a lot of rules code quality should be really high and the developer is not going to be is not familiar with this he will be really confused and uh... the books must be a team member definitely because our developers should be aware of how jenkins working how solar working and uh... this can be not so easy to approach for small teams manual review manual code review is like a battles uh... sometimes we have a battles between our developers because one like this and other like another code style and there are a lot of talks talks between developers also there is this there is a place where we need operations like when we work on a really huge projects when the database is two gigabytes importing the database every time when the build is coming is it tough task so uh... here where we need an ops for in for improving the process maybe preparing database removing some of the data and etcetera uh... by using virtual machine uh... there is a requirement that your local develop development machine should be really decent because uh... virtual box use a lot of hdb and it should be really perform so ssd is a must and for project managers with this flow uh... there is no way to assign a task for five minutes because when developers start to work on the project he he should pull the code from the vagrant up and if he has no latest updates he he should run vagrant up provision so all the uh... software should be reinstalled into the virtual machine and only after half an hour he will be ready to work on the task itself so after that it's a code review and so so minimal task is really one hour overall system is pretty complex because uh... well uh... right now see a box contains approximately fifty technologies and it's not easy to understand how it works when you see a lot of scripts thanks to our awesome team we right now we have a nice weekie with quick start and you can follow the steps but anyway it's really complex and it's not so easy to start for new small teams like if you have a small workshop it can be really hard to change your uh... workflow to use see a box there is a way to start to use it like uh... not use all the project but only some parts of it like you can use reinstall script for running it uh... for running it to reinstall your site locally every time it's it's really handy or if you want to put into your process tests you can use test scripts and or even a background box if you don't want to use all that's the infrastructure you just need a good work in virtual machine here it is uh... and of course uh... the best way is to contribute to see a box because when you are contributing you learn a lot and for uh... for the Drupalcon Barcelona we put an effort and made stable release so we stick it to stable versions of plugins, Jenkins, polish it a lot of stuff uh... remove it uh... abandoned code and right now you can just try it and it's really stable it has we believe it has no so much bugs and here is a list of what is included in release and upcoming features so right now aquia deployment plans are ready to go but they are still not merged into master because i'm not happy with them they are not so universal like we are expecting and some of our projects are using gulp workflow is for frontend like you don't need to recompile every time your source files you can just uh... install that workflow uh... it's not complex but it helps you a lot to speed up all the process of frontend development and uh... we want to create something like legacy cms integration for example uh... i've been involved into integrating big 3cms nobody knows about the cms it has maybe ten installs but uh... it takes for me uh... twenty hours to for adding it to cibox like changing the environment variables and to put in database backup and everything is going just smoothly and of course if we have some projects on drupal 8 and uh... cibox works well with drupal 8 but right now drupal 8 is not stable so we are not included all that scripts into the stable release here you can uh... see a lot of presentations we've made in different events blog posts and of course linked to wiki that is well-organized and i guess that's it we are ready to answer some questions and uh... invite you in uh... fifty minutes uh... will be uh... both about best practices of continuous integration in room one three two i believe so welcome to our book right now we have uh... it's about good flow yes a question so right now we're using a different branches for different environments like master branches for development environment a staging branch is for staging environment production for production so when we are want to deploy the code from master to staging we just creating pull request wait for all that sniffers and if everything is okay manage it to stage in and after that the copy of demo job that you might see on my uh... on my presentation it just looks for change in uh... staging branch and run deployed to environment we have a microphone there so this session is being recorded if you have questions maybe it will be good idea to talk to my phone or we can repeat the question as well this one working for me uh... yeah i was gonna say thanks for sharing this information uh... the first question i had was about the security tools that you run uh... some sort of a security audit that you mentioned at the beginning what are those tools exactly so right now we're using Drupal security rules for sniffers uh... it is used for uh... approving applications from sandboxes on Drupal org and another tool is Drupal model security checking and i believe that's that's it for today i'm assuming they have some sort of like uh... maybe a drush command that you run on the command line and it says yay or nay right yes security checking model is has its own drush command and its command written into the sniffers uh... script and it just shows you the link to the log of output of this comment thanks and the second question is i saw there is this output from uh... something called a ppbot on github right so reacts on a pull request and it basically outputs you know the result of all the checks so is this included in the repo or is this something homebrewed that you guys are using no uh... this this is not included in the repo this comment is uh... just a file that is forming every time you run the uh... build so when reinstalled dot yaml uh... runs uh... all the output of it puts into the uh... comment info dot md file this is the part that we uh... made by our self to improve the Jenkins itself and after all of the steps are done in within Jenkins this comment just uh... this content of this file just copies to the github comment that's all it's not a bot it just basically sends through like github api it just sends this comment yes it's just sends through github api let me add on this a little bit so that's a standard Jenkins plugin that is github integration and the way it works you need to provide the user that will gen that Jenkins will use to actually check if there are new pull requests coming in and that's the name of the user so it's not something excerpt that we have developed we just call that user ppbot and what it does after the jobs it just can output the content of the file to the comment stream and that's what andrew has explained we just build that file and it's got uh... display there any questions if you have any specific questions so you would like to see a demo uh... and have really like hands-on session uh... the both will be the best idea for you to come thank you for coming