 So, today we are going to talk about multi-dimensional testing workflow and actually we are going to talk about a lot of things regarding continuous integration in our development workflow. So, let me introduce ourselves. My name is Yuri. I am working for company FFW. I am coming from pro people part of this company and this is my colleague. I am Andri. I am a DevOps architect. I am coming from pro people too. Currently we are FFW. Yes, and one of our responsibilities are managing development teams and like we have a couple of teams that I am leading and some of the challenges that we are working in during execution of our projects we are going to cover and I would like to show you how we have improved all our development workflow and how we have managed to do a lot of testing during our development before actually things get committed and pushed to repository and appearing on our staging environments and so forth. So, first I would like to talk about some historical things like how usually development workflow happens. I have been working with Drupal for like six, seven years. I worked with a lot of different teams and now I am kind of person who is managing very, very different teams and I am getting involved in all the workflows and I know a lot of how people used to do things. So first we are not talking about really, really old workflow when we had FTP server and just pushing the code through FTP. We are already on the stage when we are using some sort of version system and usually how it happens. So, if you are not a single developer, right, if you have several developers they usually have some sort of central development server where they can configure the site, where they push the code and sometimes it gets automatically deployed to the server and like some companies they have their own server somewhere in the back of the room and some companies are using the some SaaS solutions like hosting and like Acquire, Pantheon and there are like a lot of them and this workflow has a lot of different elements. And for example, when we are working with the code, like one of scenarios is that we have some SVN or Git, we all commit to that master branch and we push the changes and we get them immediately deployed to our development environment. And this is where we have all our changes. We see the merges, if we have conflicts we resolve them and something like code review getting very tricky because like when we have a code it's already in master, it's already deployed. So, if we see something like that we need to roll back and sometimes by the time we do a code review there are not only one commit happened, there are like multiple commits and they're like stacking one another and maybe we already got some dependencies and sometimes this rollback process it's fairly complicated. So this is how it usually happens. There is another aspect of development of Drupal website is configuration and because we need to deal with the database it's also very interesting because multiple people have single database that is on like development server and they do some configuration changes and usually that happens on the development server. And again, if something went wrong we understand that something went wrong like last week and we need to roll back and do the changes and database is already messed up. And if we need to roll back we of course we have some backups but if we roll back to some backup we already lose some work that has been done afterwards. And another bad thing about doing so is that we have different configuration that we don't track. So sometimes we don't know when that change has happened and it's hard to review things. So for example if my colleague has changed the view that I'm depending and I realize that something happened to the view and my display is broken and I'm changing this view and my colleague starts screaming like oh you have broken that view on my display. And this kind of conflicts is pretty hard to manage if everything is happening just like click work on development server. Some other problems. When we have a single server usually how it happens like with the small teams when we have multiple projects all the projects are running on the same environment. For example if you will have several teams who are working on different projects once you start eating all your resources of the server for example on migration or I don't know some other very intensive jobs like all office will start screaming at you basically. And this is the problem of sharing resources on the single development server. So when you do development your developers are using their local laptops desktops and sometimes it's pretty hard to reproduce the environment that you're going to work on the hosting. Like some projects they have different versions of PHP configurations Apache solar when you start running into configuration of varnish you cannot really find the worst task for junior developer than actually asking to configure your production environment. That will take him like a long time. So all this onboarding process is getting pretty hard with this kind of workflow. And there is another thing that is pretty usual with teams that there is only one guy who does deployments to production. And if deployment needs to be needs to happen all stress goes to him. So he is the person who doesn't sleep at nights he's fixing all the production servers. So it's getting hard to do deployments as well because we have all knowledge for only one person. Sometimes it's like multiple people but still it's pretty big risk because we don't have any automation we really depend on people and that usually well sometimes it fails. So it's about big distance for the development and operations and other things like one projects require different configuration and it starts to interfere with other projects. So these kind of problems usually teams face how we started solving things. So first we started with the problem of local environments. We have started experimenting with vagrant and currently we are working with vagrant based on Ubuntu and the way we started we found very nice website that called pubfit.com. If you haven't used it probably you should take a look at it. The main idea is that you can configure like with your click works selecting some check boxes what configuration of vagrant box you like. So we can choose the version of Apache or Nginx or MySQL or PHP and extensions and all that kind of things that will take you some time to configure manually but it does it for you. So we started with that configuration but we ended up rewriting it basically because pubfit is doing that with puppet. If you're familiar with this tool it's provisioning tool that does configuration of your servers and we were just breaking our vagrant box too often. So we decided to switch to Ansible and we are pretty happy with the solution database. This is something that a lot of different teams are facing and we are now following code driven development. So basically this is technique where you don't do any manual changes in your database at all. So you use things like features, you use hook updates and if it doesn't help you just put like different MySQL queries in your hook updates and in this way you configure your website. This helps us to see the changes, the actual changes that are happening and then we can at least track who has broken things and when we see some conflicts happening like in feature we can resolve them in way more efficient manner. So in our workflow that we have actually two workflows. So in the beginning of the project if it's like new project, no heritage, nothing we start building the website as installation profile. So every time we do deployment we actually rebuild the website from that installation profile and we see how it works. When we start having real content so we cannot actually reinstall because we will remove it. We call that SQL workflow. So we just have central database but we use that database only for the content. So we don't actually log in there as super admin and start changing configuration. We don't do that. Everything happens on code level but we just pull the database to our environments and then we apply all of our hook updates and we also run all the changes in features with hook updates. So sometimes they like to have features in default states but our teams, we just found it not very good time spending so we just revert only features we need and sometimes even we revert the components of features we need. So we're getting pretty granular and features allow that to do. How our coding workflow changed? So we have switched to GitHub completely on our teams and that helped us a lot with great feature of pull requests. Pull requests is basically when you do changes you create branch and then you do your coding in the branch and then your quest changes to be merged back to master and this moment called pull request. And nice thing about GitHub is that it's very, very handy for doing code reviews. So we shifted the code review from do it afterwards, it has been merged to do it before changes being merged to master. So we have, we always have not only one person working on the project so at least two and they do code reviews of each other and of course if there are three, four then it's much easier. And we also solved another issue that is very important that we solved the kind of solving, the issue with knowledge sharing with the project. So there is no such situation when there is only one person know what that code does. And it's very important because like people rotate, they leave for the vacation, they come back and especially in our environment when we have a lot of teams in FFW sometimes like we have downtime in one team and people like guys starting to help on other projects and then, oh, they got a new project, they just run away from the project so it's a very good idea to have a code review so really a lot of people understand how your code works in general. Another thing that we have started using is the continuous integration. By that I mean we have Jenkins, we have GitHub plugging for Jenkins, what it does, it allows you to do a job, it has a trigger for the job that is pull request. So when we have a pull request, we can trigger some jobs and what we do for that, we run all the code style checks. We have a lot of JavaScript, Lint, CSS, ASLint, like all of these kind of checks and if any of them fail, so depending on our configuration of course, if something fails like build mark failed, so we don't allow actually bad code to be merged, we have pretty strong gates for that. And another thing we have some scanners for security and that is another static analysis of the code. And another nice feature that we do is we spin up environment based on the pull request. So actually our QA people, they are able to see, okay, this is the pull request, this is your issue, this is how it should work and this is the URL that we can actually open that has all the changes from the pull request. So we can log in there, test the functionality from user experience because like developers they are very optimistic, they are very fast, they really like the job and for example, if Dickett has three paragraphs, you usually end up developer reading first two and he starts coding and in this way, like if he is friend developer, because he is exactly the same guy, right, so he is checking first two paragraphs and of course nobody reads the third. And what happens then QA person starts checking and QA people, they are really like boring these guys who are actually reading all the words and then they start like, oh, this is not done, this is not done, you should finish the job and then we can merge it. So that is a pretty common problem and pull requests are really helping that because it is not very good experience for your clients when you deliver not finished tasks because like if you have a sprint, it is better to say like we haven't completed that functionality yet than to actually deliver just half of it and half of it will be broken then to switch to SQL-based flow to keep all those content within your database. It can be done by passing variable to reinstall.yaml playbook called name and pp environment station and everything becomes rebuilt from database from the continuous integration server and the third thing is when you are trying to connect continuous integration environment to already the real project being developed not by maybe a team, this phase should have some rules like you should make all those security checks, code style checks optional because at the start you will get something like two or three thousand errors that is impossible to fix in one pull request. We call it lazy fixing and a few words about responsibility. Responsibility being shifted from operational guys to development team itself because all those scripts, sniffers reinstalled.yaml and tests being stored with the project itself. Any guy from development team can change the process at any stage, at any step just by creating pull request and changing the script in the middle of the project when the project has such need and there is a magic when you don't need really operational guys because those guys are not familiar with the project. You should spend a lot of time to pull them, push them a task to with this approach team members can do pretty much the same as operations and more effective. And of course we have a bottlenecks because we have strong dependency from GitHub. When GitHub is down you can do anything because you can do pull requests, you can clone repository. When CI server is down you can't obtain builds so you can create pull request, you can do manual code review but there will be no builds. Also there are issues when you are trying to inject new developers into the team because they are not familiar with this workflow and you should spend some time to study them. Also DevOps must be a team member. You know DevOps is more than seniors because they not only good coders but they are good administrators of the service. Sometimes there are a lot of battles between developers at the manual code review step because you know code style differs from guy to guy. And when your project is large your builds become slow because when your database is huge you need to pay attention to optimize that step to remove some non-needed data because every time when you try to import 10 gigabytes of MySQL you will take forever, you will wait forever before you will obtain your build. Also you need to empower your local environment with good decent desktops because you know Vagrant consumes a lot of disk speed and with SSD it's good enough to use. And the last thing is more about project management, the minimal task is more than one hour. Well you can do hot fixes, you can create pull request to production branch but after that after change deployed to production you need to create the same pull request to staging and to development for keeping changes for future. So here is a list of links, the first link is to the project itself, we made it open source for DrupalCon, two links to documentation, first link is about the CABox itself, second one is about reinstall script because it's complicated script and a few presentations that shows our continuous integration from different sites from previous camps and cons and a few blog posts. Jeff are you here? No, when you meet this guy at DrupalCon you may tell him a lot of thanks because we've been using maybe 50% of his roles with our workflow for installing MySQL, LAMP stack. Thanks Jeff and please evaluate us and if you have any questions please. You can use microphone, yes. So the question was about presentation being online, yes we're going to post it and this session is being recorded so you will be able to see all the stuff multiple times. And also I wanted to add that we are very interested of other people giving this thing a try, we currently, we have multiple offices and currently three of our offices start using this workflow, we already said that we have already in production like 10 to 15 projects and now all our teams are starting our projects with this tool but different companies, teams they have little tweaks that we are happy to incorporate so it will be great to have some feedback from you guys as well. Okay, question? We will repeat the question. So let me repeat the question. So the question is about how that works with the clients when they want to do some changes and how I think they are using this system. So first of all we're really not encouraging them to go to coding level and mostly we are dealing with clients that don't do much coding themselves but anyhow when we show them the system it's if the developers are pretty good and we usually deal with these guys they can pick up things fairly easy after we did a couple of tries but we really encourage them not to do any commits directly to master branch and we still work with pull requests and we also have cases when we have multiple teams working on the same project and we still keep the pull request workflow. Usually they don't touch the YAML files that can cause all the system to break and when we do releases to like production environment, let's say to aquia environment we remove all the stuff before we actually push it to the environment. So we keep things clean and transparent and we didn't have any problems with that workflow. I'm not sure I understood the question. So the question is can we use Gitflow workflow with Git? So Gitflow, yes, you definitely can use it. I'm not sure how it plays with GitHub because Gitflow already does all the very nice stuff with starting release, applying hot fixes and then completing release and it does a lot of good things. Maybe you can answer that. You can just extend Jenkins with Git plug-in and maybe tie it to some text or I don't know because it's a little bit different from the GitHub itself because GitHub uses pull requests but Gitflow is another thing. So we've been trying to use it with different kind of systems and it's ready to be adopted to them. So just never used Gitflow. So for the Gitflow, if we have a branch that can be pushed to GitHub, we are happy to do everything and it's actually the business of our developers how they manage their branches and what they push to GitHub and what they don't. So you can probably run all the create feature stuff with Gitflow, commit all the things and then you can say, okay, this feature is ready and then you push it to the GitHub and then we take it from there. All right. Then I hope you all get excited about the tool and we will be very happy to discuss things further or even give you a demo if you like and thank you very much for the attention.