 Welcome everybody. Thank you for joining us. Today we'll be talking about a recent collaboration between Forum One and Mercy Corps on our project to improve the performance of mercycore.org. We hope you'll leave today with some insights on site performance concepts, how performance can be measured, and how it may influence user conversion. Just a quick introduction. My co-presenter is the lead user experience designer at Mercy Corps, Drew Betts. And my name is John Brandenburg. I am a developer at Forum One and a member of our dedicated support team. This is the business showcase session for Forum One. And we're talking about the Site Speed project because it came out of our support work for Mercy Corps. Just a little bit about Forum One. We focus solely on working with mission-driven organizations that are working to make great changes in this world, like Mercy Corps. Our full service agency provides digital strategy, design, development, and ongoing support services. Which provide long-term strategic support, development, fully managed hosting. And now to tell you a little bit about Mercy Corps, who they are and what they do. I'm going to turn this over to Drew. Thanks, John. For those of you who don't know who Mercy Corps is, we're a global humanitarian aid organization. We were founded 36 years ago in response to the refugees that were fleeing the war and famine in Cambodia. And since then, we've helped 170 million people in 115 different countries survive emergencies and build back better. So when natural disasters strike, like the 7.9 magnitude earthquake that hit Nepal on April 25th, Mercy Corps responds to meet urgent needs for water, food, and shelter. Some of you may also know that two days ago, a second earthquake hit Nepal, this time 7.3 magnitude, causing even more destruction. So after the crisis, Mercy Corps stays and partners with the local communities to help them rebuild. And this is one of my favorite pictures, and this is a young man in a Mercy Corps carpentry program in Helmand Province, Afghanistan. So you might be asking yourself, well, what does site speed have to do with Mercy Corps? Well, our website is critical for us in engaging visitors as well as raising funds for our efforts. And so last year, I was just kind of doing some testing on the site, and I noticed that it loaded really slowly on my phone. So I got to researching site speed, and the first place that I landed was a free site. This is webpagetest.org. It runs a URL, even specify a browser, and connection speed, and it runs a report, or it runs a test, and then comes back with a report. So you can see, or maybe you can't because it's really small, we had a load time of 4.8 seconds. This is a desktop connection using IE9 on a cable modem. I thought, well, you know, that's tolerable, it's not great. Webpagetest.org also gives you a letter grade for various other site metrics and even links underneath those to help you improve, and you can see that we had a lot of room for improvement. So if desktop was bad, mobile was worse, it was much worse. We had a load time of 19 seconds for an iPhone, and I could just picture people giving up when they were trying to come to our site before they even got there. Webpagetest doesn't give a letter grade for the mobile tests. Oh, sorry, thank you. So it turns out that slow site speed can have a negative effect across a number of different aspects of your site, and the first, if you're at e-commerce site or your site like ours, is your conversion rate. And this is a slide from a study that Walmart did in 2012. Actually today it's an initiative that they did in 2012. That red line is the conversion rate and those blue bars are the load times. And for every one second of improvement in load time, they experienced up to a 2% increase in conversions, and for every 100 milliseconds of improvement, incremental revenues go by up to 1%. So they also found that increased load times also had increased bounce rates and decreased revenues. And first, somebody as you can imagine like Walmart at their size and scale improving a little bit had a tremendous impact. Now, you might be wondering how Walmart and Mercy Corps are related. Well, they're not. But there really isn't any research out there that I could find that looked at the conversion rate for nonprofits like Mercy Corps and Site Speed. So we're kind of off on our own here. So as I mentioned, there are other negative effects to slow site speed. It lowers negatively impact your SEO. If you have a slow site, Googlebot can't crawl more of it. We know that at least since 2010, early 2011, that Google has used site speed as part of their page ranking algorithm. It lowers perceived quality and credibility. So when users come to your site, it's going to detract from that. Frustrates users. I think we can all relate to going to slow sites and, of course, increase this bailout rate. So slow sites is a problem and we needed to improve. So I'm going to turn it over to John to talk about what to do about it. Thanks, Drew. So you know who Mercy Corps and Forum 1 are and why we're so interested in site speed. Let's talk about what site speed is anyway, how we measure it. We have a few different metrics that we use here. When we talk about site speed, first is page load time. This takes, as you might expect, measures how long it takes to load a page with all referenced assets. This also includes any third-party scripts that aren't perceptible to the user, so a page that might take 15 seconds might appear to load in under two seconds. It actually might take 15 seconds, but this can be okay because it's all about the user experience. Next to what we call is time to first byte. This measures the time passes from when a request is sent to a site to when you begin receiving data for that page. This is a pretty critical value since it blocks downloading all of their assets on page, so it can really make a difference. The third measure to talk about is start render. This refers to how long it takes requests to process before the user actually begins seeing the page rendered on their browser. This is actually a pretty complex subject. There are a lot of different drivers of these values. Some we have more control of than others, hence a couple of items you see across that on this list. You could say a CDN or something could counteract that, but I'm just going to say we don't have much control over this. I'm not going to go through each of these and this isn't a comprehensive list, but I will call out caching and downloading assets as some items that we have control over on a Drupal installation. There are a few important things you need out of the tools you use to measure performance, something to analyze the page structure and behavior of your page, as well as how optimized your assets are and how your images are compressed, your CSS or JS are well optimized. Also, something that can form load testing, so simulating large amounts of traffic that are visiting your site. Here are the tools that we used. You already saw the slide Drew showed you earlier from webpagetest.org. They had a REST API that we utilized and it allowed us to script these tests. It was a good tool for that. They also provide videos of your page actually rendering. Google page speed is great. It gives you a rating and separates it out from desktop and mobile. It seems to place an emphasis on payload size overall. Apache Benchmark is great for simulating heavy loads on a server and already comes packaged with any Apache web server installation. We also utilize the paid for service New Relic. New Relic provides a wealth of data on a live running production instance. And we also Google analytics to assess user engagement which we'll define for you and explain how we measured that in a bit. For the future I consider using a tool called Jmeter. A lot of my colleagues recommended. This organizes a lot of similar tools in a fashion that allows you to organize your actual test runs. Because we found that sifting through the mountain of data that we collected was quite a challenging of itself. But regardless never to make these tests more consistent. I wrote a script which I posted on GitHub. Also the user engagement script was done through Google Tag Manager and the source was through this article you see. And we'll be making these slides available online later. So we have our instruments and what we had to do was come up with a consistent way to apply them beyond just scripting it. So Drew will walk us through that. So when we started this project it was really important that we have a consistent process to follow because we wanted to show the impact of our efforts. So first we decided on the metrics and the tests. Then we decided on the changes we were going to make and we prioritized them. And the changes were for prioritizing were based on the perceived amount of impact that they would have as well as the level of effort required to make them. Before we did any of the changes we did baseline testing on our staging and production servers. Then we developed and applied each of the changes measured them in staging and production and repeated the process. So kind of some general notes. Where we could we tried to simulate as closely as possible the experience of our users coming to the site. So use the same mobile browser use the same desktop browser we didn't use any virtual machines these yield inconsistent results when you're doing testing and so we kept the staging and production environments as identical as possible. The big exception to this is load balancing. We at Mercy Corps load balance our production servers and load balancing staging servers would be it was going to be just too time and cost prohibitive. So metrics so John's kind of covered this so quickly go through page load speed you talked about that. Conversion rate word defining conversion rate is the number of visitors who came to the site divided by the number of the visitors who made a donation. And if your site is encouraging users to take some other action that will be your conversion rate. Engagement rate is a new metric. We wanted to see if by increasing the speed of the site that people who were coming to our article section would read more articles would they make it farther down the article that they were reading would we see a lower bounce rate. And that's where that Google tag manager script that John talked about came in. You can see it over there on the right so it basically fired off all of these events and you could see these in Google analytics. And the last was response time. Synthetic measurement and Apache of how many requests that the web server could handle and how fast that it could respond to those requests. Alright so we get the measurements, we have the process what to improve. Thanks Kendra. We identified a number of items on the site but before we go through that though I just when we're talking about site speed in Drupal I feel obligated to call these items out to any person who manages the Drupal site and if that's you take note go check your site after the session or after the next session. The page cache is really simple normally. It saves your page and reuses it for all users. It can get tricky as we'll see in a moment because we ran into a bug with this on MercyCorp. The next item is compressing your CSS and JS. Since your browser can only download between 5 and 8 things at any one time you want to minimize the number of things you have so that leads to better performance overall. On the same admin panel you see what's called the block cache. If you are unfamiliar with this the caching in the block cache is a bit dumb sometimes so do be careful when using this it can lead to issues where you see dynamic blocks repeating in areas where you don't expect. If you're unsure on how to do this or want to see some actual quantified data on what these can do blog post on form1.com Getting back on track to MercyCorp.org we identified a first set of changes which ended up going around. We chose these because we believed they would have a low level of effort for the impact they would provide. First we enabled views caching on a number of views across the site. This was going to be an easy win and only required a simple configuration change on each view. The page cache is normally a simple option but we're seeing some unusual behavior around when users are being served cached pages so we wanted to dig into this a little further. This is an alternative back end cache so we wanted to set this up because it's rather simple to set up. The first change we made was enabling the views caches which allows us to cache many of the individual elements that make up a page. This is pretty straightforward to change for example but it can lead to issues with cross cache contamination. So for example a lot of the country pages on MercyCorp.org is the open layers based display mode which displays maps for the country. But we ran into an issue where if you visited say the Afghanistan page you'd see the map of Afghanistan. If you went to the Nigeria page afterwards you'd still see the map of Afghanistan so we couldn't use views caching for open layers based views. We also added the views content cache module. This handles cache exploration for views similar content types are updated. This is helpful for keeping our lists of content up to date. The results of the views caching showed a pretty modest improvement which is what we expected. It's also good to note though that the impact of your views cache will correlate with the complexity and amount of data you're displaying in your view. Just a quick explanation of the charts you're seeing. Some from the Apache benchmark tool provides an option to output a CSV file. They show the number of requests served under a specific time for that percentage. So 20% of requests were served in 3 and a half seconds or less. 80% of requests were served in 3.8 seconds or less. This is across a thousand requests each on the left you see the level for 10 users concurrently and the level for 50 users concurrently. Admittedly to properly test this we need to have the page cache disabled so under normal conditions you wouldn't see such a large jump up to the 50 users. If you can't read that the scale is quite different about five times that of the chart on the left. I probably should make this proportional. Next we're seeing inconsistencies with the page cache and we just had to address that next. I previously mentioned that this is typically a simple thing to do on a Drupal site. However in the case of MercyCore cache pages were being sort of erratically and we just needed to dig into the code and it turned out that the session global variable was being made and what this does is it will create a session for a user regardless of whether they enter an anonymous. This is actually the all logging users have sessions but sessions for anonymous users cause the Drupal page cache to not serve those cache pages. We went through and provided some alternative means to serve this store these session variables. We used cookies instead that MercyCore changed our cache lifetime from five minutes to an hour. Now this resulted in more stable caching behavior we're trapped to see. I'll argue you get 200 from cache. I'm also going to jump into the next measure we implemented which was setting up the memcache module. This is an alternative caching mechanism and it is much faster than the default database cache. One module that complements memcache well this module as you could have guessed caches entities like nodes which helps improve the performance across many areas of a Drupal site. In simulated testing mean time first byte went from four and a half seconds to 1.5 seconds on the MercyCore donation page and as a follow up to the test reports Drew mentioned low time dropped by about 50% to 2.3 seconds. The time first byte dropped was about 65% to 0.35 seconds. All these measures were ready just in time for a serious event that's still unfolding right now which was the aftermath of the 7.8 magnitude earthquake that struck Nepal. MercyCore has one of the largest humanitarian teams in the country and is in a unique position to bring aid to the people affected in the region. So these measures were ready or nearly ready just in time when this tragedy happened. Over the course of that weekend the 4-1 and MercyCore teams worked to adequately test these changes and push them directly to production. At the same time the MercyCore team also set up mercycore.org to use a content delivery network or a CDN. When disasters like these gained international media attention people desired to help and pour into organizations like MercyCore.org through their public websites. So these fundraising platforms need to be ready to handle this type of traffic. Anecdotally the MercyCore team noted that they used to sweat when they had 400 users on the site but now they welcome them with open arms. Here you see that Daily Gupal Analytics report starting on the April 24th the day before the earthquake on May 9th when traffic piqued 116,000 visitors. This is the new relic report after the last deployment. The markers indicate when the deployment happened and which led to a much more stable system as you can see from the flatter lines. And this slide shows the impact of the content delivery network. This is the number amount of data that's being fed out of the CDN. I know. Yeah, so actually at that high point there on April 30th Amazon served 23.7 gigabytes of data for us and that took that load off of our servers. And that was thanks in part to the CDN module. I'm going to probably not say his name right but Vim Lears. Thank you. That's awesome. So did our site speed everts have an impact for our desktop and mobile users? So here's the desktop before up above and here it is down before. This is same IE9 using the cable modem connection. We shaved 1.3 seconds from load time 0.648 seconds from first byte 0.744 seconds from start render and visually complete we shaved an entire 6.4 seconds off of load time which is great. Our letter grades improved so we still have some work to do on compressing images. That's going to be some of the next work that we do and we got an F4 cache static content and that I think is a little unfair with the scripts that load after the main content loads and we can't do anything about that for mobile. So there's the before up there and then afterwards impact we shaved 2.89 seconds off of load time and 2.51 seconds off of start render and web page test this is I think a little odd it says visually complete we didn't save any time but I can tell you by looking at the videos we definitely did so we would have liked to present to you today a longitudinal study where we were able to say well we made these changes we would wait long enough until we had enough evidence to make another set of changes and then test against the previous changes but the earthquake kind of changed all of that so what I did is I went back and I looked at the last major and I responded to which was Typhoon Hyann in November of 2013 and I took a similar 15 day block around that event so up above there is the top line in the graph is Hyann and then below is the earthquake and we were improved across the board compared to Hyann and I admit it's not a fair evaluation but the best that we could do as far as when we rolled out the content engagement script on April 1st we had enough time to at least form a little bit of a baseline so this time since we had the script in place I did a comparison between during the emergency versus the baseline for it and actually surprise we were down a little flat actually in terms of the engagement not just a 5% difference for our article section but that was kind of surprise and then lastly conversion rate so there is the conversion rate that we saw in the Nepal earthquake and in the middle there is the baseline so compared to Hyann we were up by 3% for desktop 1% for tablet and 2.6% for mobile and then against the baseline and that baseline I didn't talk about this before that was all of last year because we had a lot of seasonal traffic I just took an entire year of website traffic and created the baseline so we were up compared to that 5.5 2.83 for desktop and 3.1 for mobile so we're not done yet yeah we'd like to continue our work with Mercy Corps the graph here you see is sort of the ideal framework you'd like to stack you want set up with a Drupal site some sort of reverse proxy cache sitting in front like garnish another CDN and then going down to your Drupal site so a few more items we want to implement this is an ongoing project and but so far we've managed to make some very significant improvements to the site speed of Mercy Corps so thank you we have another session starting in a few minutes here I guess we have about 5 minutes but if we have any quick questions we can take those but we'll probably take the bulk of questions out in the hall after this next session so thank you very much it is available oh no I meant because I don't know the link oh yeah we'll post these slides online it's under my github account name John Beberg alright so I think we're the second half of this so thanks for coming my name is Jason Ford CTO Blackmash Solomon Gifford he's one of our DevOps engineers this is the first time we've actually did this demo in front of someone it's going to be a live demo so hopefully it works exactly, cross the fingers, the toes sacrifice a few chickens we're going to see if it actually works out but what we're going to try to show is a tool that we've been working probably over the last three years or so we've been doing a lot of DevOps work on workflow so what we basically came up with was a more standardized way of doing this instead of just doing it one off per customer so we created this thing called Cascade and what it basically becomes a workflow deployment tool that is flexible for what you want to do instead of having to just do only Drupal or only WordPress or only Node, they can basically do any of these things that is a lot to do get and actually have an SSH connection to go into that box so it's not really just pointed at one simple thing because most sites just don't only have Drupal or only one one thing, they're using other things as well, AngularJS or something like that that may or may not be impactful to what you're trying to push into so the first thing is we always get asked why is it so hard to do dev workflow there's a thousand different ways to do it none of them are really wrong as long as they work for what you're trying to do but the problem is you have to have a certain skill set so you have to be a developer and you have to also be operational at the same time because certain things that you make decisions on may impact the site uptime and those things then impact SEO scores and impact other things as well but you have to have it so if you don't have the whether it's you manually going into the box and actually typing commands or whether something else is actually doing those things for you you still have to have something that says I need to deploy code from here to here to here or a database as you go from different stages whether it's development whether it's staging or whether it's production so and that flexibility is what I was kind of talking about before is not at any time do you have one technology you're dealing with most shops will pick the right tool for the right thing so maybe Drupal isn't the right thing or maybe it is but you want to have that flexibility and then hopefully that same workflow logic happen around that Dev and Sysimines always fight so we're always developers always have this thing saying that Sysimines are always the ones causing the problems Sysimines always say the network guys are causing the problems network guys always say the communication guys are always causing the problems it's this line of contention it's always there so trying to add these things together that's the problem that's the problem space that we're trying to at least help with a little bit so Cascade stands for this basically customer administrative administrative system configuration and automated deployment engine very long but basically what we came up with is we have this interface that is the front end to Jenkins, GitLab and this application it's a node front end that basically gets your requests and then pushes things around depending upon what you tell it to do so this demo setup we're going to do today we have basically four vagrant servers that we're running on this laptop so one is the Cascade server itself that basically we have all of our tools on that we've built and that's our workflow engine and then we use Red Hat OpenShift for development so anybody that doesn't know what OpenShift is it's basically an automation platform as a service, the Red Hat's created you can get both the free version which is Origin or you can get the enterprise version which is more geared towards J Boss and other things other than just PHP containers the second one for staging we use Docker we basically spin up a Docker container with Drupal inside that and then we have two KVM virtual instances that are running to show that we can deploy to multiple frontends instead of just having one place to dump this code in a real world setup we might have physical servers, in our cases we see a lot of people with physical, virtual maybe it's not even in our network it could be outside at the end of the day what we're after is you just have to have SSH so as long as you have SSH we can attach to those things it could be a container it could be really anything that you want to have to so with that I'm basically going to turn it over to Solomon he's going to kind of walk through some of the deployment techniques and stuff that you need to talk about when we're talking about workflow and then discuss some of those pieces of this puzzle and then actually do the demo itself alright so in a normal development workflow you have code you work on your local laptop you push that into your Git repo you want to put that onto your dev environment you push that to staging and then eventually once the customer has accepted that you push it to your production environment the database on the other hand except for when you're first deploying a site brand new the database goes the other direction you want to take a live database pull it down to your staging and if at all possible bring that down to your development as well and a lot of times on Monday morning or every couple of weeks on a Monday morning it might be your job to get someone within your office's job to do all those database dumps and get staging back up or get dev ready for development and we saw a need for this to be automated it's something that's repeatable something that is done all the time so we wanted to solve that problem obviously there's exceptions to this workflow so sometimes you might have multiple dev machines so you might have different branches that you're working on you might have two staging machines because you're also doing load balancing in your staging environment and of course multiple productions that is in our demo today we have two vagrant boxes representing two staging I mean two productions but of course this could be 11 web heads or whatever your your scenario may be so we have this normal workflow for code going upstream databases coming downstream but Cascade doesn't just only support those directions but yes you can restrict within the interface through permissions so that you don't want to accidentally push your staging code to production once you go online so I know this graph might seem a little confusing at first but here you are you and other developers you commit changes into some repos so we provide an installed GitLab as part of our solution and if you're not familiar with GitLab I think GitHub only it's open source and you can install it on a box and so as part of the Cascade solution we give you a VM that has Cascade and GitLab and some of the other tools all installed on the box already we have to set this up we set it up for you so the first thing is like I said GitLab so I think GitHub only it's installed locally within your environment and all the Git commands work it's Git clone to get your code down to push it back up so we have our source control and what we want to happen when we push our code into the repo is we want it to automatically pull to dev changes immediately or within a few seconds live on the server so you can test it essentially in a real environment and in this scenario or in this picture you see the guide Jenkins that's the official Jenkins logo so Jenkins queues up the task to actually deploy the code from Git into the dev environment the feedback loop that you see here going around the edge we don't set this up for you but we give you access to set it up but you can set up feedback loops to send emails to hook into your hipchat or IRC so that you can know when that has happened so that your whole team can know every time dev has been updated with the latest code and then finally we want to deploy that code in this model so Jenkins is pointing over to this big A A stands for Ansible Ansible is an automation framework you create roles, you create tasks and I'm not going to go deep into Ansible at this point but to suffice it to say we inherited the runner class of the Ansible Ansible is an open source project we inherited the runner class and created our own wrapper and this wrapper then essentially can log into your dev and your staging and production and deploy the code into the right places and so that's generally speaking all the pieces of the puzzle there is one more piece that's not on this we also install LDAP so that you can have permissions management create users remove users etc reset your password etc all as part of this it's very configurable so we can do more environments than just dev staging and production multiple web heads already talked about that if you're familiar with Github you know that you can put things into groups you can create a well your user is your default group but you can also create essentially a company and that becomes like a group so you can create groups and GitLab supports the same and so in our Cascade interface we have this concept of a group as well this helps you especially with permission management so that you can have multiple sites in the same group and give different developers access to different sites and of course it supports not just Drupal it also supports non-standard Drupal sites so as long as your code can be deployed then Cascade can work with it Cascade knows how to deploy Git and it knows how to move databases around alright so at this point I'll show you a little bit of a demo sacrifice your chickens please so hopefully this does not need any internet connection because again it's running as on vagrant on this box that said I did have vagrant crash for me the other day in 20 minutes to get it all back up so hopefully that doesn't happen today alright so when you first come into to this you see actually the dashboard I'm going to actually go to the dashboard here and obviously we're having a little bit of display issues with this resolution this actually is mobile mobile compliance so if I had if I use a different resolution here the stuff on the right hand side will drop down below but anyway you can see that we have this concept of a group and so our sites are going to be in the groups we also see on the right hand side different machines that are in in your environments so in this case we have a development box two production boxes and a staging box and there's information about those so for example we have the domain name and the IP address and a description that we give you the option to restart services from on each of those machines this doesn't just have to be Apache in my secret we also support Redis, Memcache, whatever services you want we can restart SSHD if you wanted to stick to it you also see a workflow log so these are the things that other people have been doing so you can see the blame here in these examples it's our demo account but it could be your developers alright so we'll select a group so we created a group here at Drupalcon in it we have two sites and I'll show the Drupal 8 demo and so this is your this is your main landing page for each of your sites and you'll see that it shows your different environments so in this scenario again we have a development a staging and a production the staging here is a little lower on the page and again if you need two developments if you need multiple sets of staging we can support that but obviously the normal scenario is development, staging and production and we'll make one other mention that is that we have some customers who want to have their development staging on the same box and we can do that so this doesn't necessarily mean that this is multiple machine per environment we have one customer who has dev staging and production and their database all on a single machine not my recommendation but that's what that customer wanted alright so you can see also that there's a link to preview the site and I'll go ahead and click for example on the development one and you can see that I have a Drupal 8 install here and I can also show you that this is the production etc so I can show you I can essentially click on these links and it goes directly to the site to that version of the site so for this demo I'm going to go ahead and go for example to the production and I'm going to make a change I went ahead in this demo to set the permissions so that anonymous users can add content just so they don't have to worry about logging in and logging out so I'm going to go ahead and create an article and of course we can do the hello world because we always do and we're going to save and publish this and so here in our production environment we can see that we have data hello world I'll go back to the home page so you can see that article and if we go back to our staging environment you'll notice it just says welcome to black mesh it doesn't have that hello world content so what I'm going to do now is I'm going to do a deploy of our database from the production environment down to our staging environment so that you can see the changes so here at the bottom of each of the environments you see the workflows again through permissions we can remove some of these workflows you probably like I said earlier after you go live you don't want to have a workflow of sending your staging database to your to production but for this example I'll go ahead and take our database from production and I'll deploy it down to staging and I'm going to click the button it's going to say are you sure hopefully you do a sanity check and select ok and down here in the workflow log you'll see that this is currently in progress and I can click on this and see what it's actually doing and follow the steps and if you know anything about Jenkins Jenkins essentially runs scripts scripts can have output we're essentially taking that output and pulling it all the way up into the interface a couple things to note about this first of all it's making a backup of both the destination and the source databases this means that if you need to roll back you can we don't have a rollback mechanism in the interface but that's why we have a support team the support team can put a support request and we'll get that restored as quickly as possible obviously this is a base Drupal 8 install it only took 37 seconds for this to happen obviously databases get much larger than that this is also on a vagrant box so there's no there's no real network issues to be worried about but I'll go ahead and show you this was our staging and if I hit shift refresh and everything worked I should see so our database has successfully moved from production down to staging and I won't go through that whole process and make you wait for but I'm going to go ahead and get started on a deploy in the staging environment down to development and I'll come back in a few minutes and show you that that has also happened okay so obviously the other piece of the puzzle is code right that's the thing that's more interesting so instead of editing Drupals settings.ph index.php and doing something weird I decided to do something else weird I created a simple ASCII file that has the Drupal logo in it actually I didn't create the logo so I will say that I just copied that off the internet I should have put the attribution there but anyway you can see it's in the root folder and I have two versions of this so I can push this through the paces I've created little links here just so I can save you time me typing these but essentially you can see that the dev and the staging and the production all have the ASCII version of that so on another screen here hopefully you can see this yeah hopefully you can see this I have my my code checked out now to tell you where you could have gotten this sorry about that there we go we have a little button here that shows you how you can do a git clone I mean obviously we all know how to do a clone just showing you this this gives you the URL so that you can easily do it and again it looks a little bit strange in this compressed interface so I've already done the git clone here's my code if I do a cat on ASCII you can see well it doesn't look very good but that was the code for the simple version I have this magnified so the people in the back can see it so that's why we're not seeing it as well but I have another version here I have an ASCII 2 version so I'm going to cat the ASCII 2 version into the ASCII version and I should do that correctly there we go and I just made this change so I can do a git status and this is just obviously just git commands I can see that my ASCII.html has changed I git add git commit git push origin now to talk about branching we associate a branch with each environment you have dev staging and production we create three branches you can do the staging and be mesh production what this means is that if you have a hot fix that needs to go live you can check out the staging branch then do a push into the staging branch and go directly to staging and skip your development stage obviously that's not recommended for most situations but we said hot fix right so in this scenario you've already made a bunch of changes it's already on dev instead of taking that everything live that was already on dev you can go directly to staging let's take an even more extreme scenario so you've already you have stuff on dev you already have stuff on staging you're about to go live then you need to do the hot fix yes you can also pull out just the production branch and go directly to production if you if you so desire again not recommended but again it's a hot fix so in this scenario be mesh dev is our dev branch and so we're going to push to that and if everything works correctly and it did earlier today within a few seconds our dev box is going to update so again there's the old here is your new so this code has already checked out and it's just a little bit different of a logo to show you that it did not deploy that change to everywhere I can refresh my staging and my production and show you that staging and production is still the old code back in the interface I can now deploy this code to from be mesh dev to be mesh staging so we're essentially doing a merge between the branches before we know what a merge is we do a merge and then we do a checkout this does imply that it's important when you do a hot fix that you make sure you merge those changes also down into the lower branches so the deploy has been done again we can watch what is happening and while this happens I'll note again that this is live in the background I talked about this concept of Jenkins or this application Jenkins that's in the background I won't go into this in detail except to say again you can plug into this so let's say you wanted to run a testing suite every time every time you push to dev go right ahead and do that let's say you wanted to also push the code out to Bitbucket or some other repo maybe GitHub so you wanted to run Travis or something like that you can hook into that create notifications etc let's see where are we in the process it found the branches it created the merger class it accepted the merger class and what it does is it goes to dev staging and production at this time and says are you the latest version of the code and if not it checks it out so at this point I should be able to go to staging hit a shift refresh and we'll see that staging has now been updated development should not have changed production is still there and I will go ahead at this point and get a deploy your production started won't make you wait for it in this scenario we again are taking our code now from staging to production I mentioned LDAP is sitting behind this so again permissions are granular so for both GitLab for Jenkins and for Cascade all three pieces you can you are giving essentially a master account and you can use that account to divvy out permissions as necessary again you don't want after you go on live someone pushing a staging to production and that essentially concludes the demo this will be finished in just a couple seconds but while that continues are there any questions so Jenkins is calling an Ansible wrapper that we built and that is what's doing the deployments so that is essentially logging into dev doing the git clone or the git checkout or the git pull and then it logs back out of the machine it's also the thing that's doing the database dumps for example so it logs in to Ansible knows how to log into machines how to do commands so it logs for example into your production database has a database dump executes an rsync over to the new machine logs out logs into the new machine doesn't import all we could have but in this scenario we had other things that we wanted to do with with there's some other pieces to it that we're doing to ensure machine state so where Jenkins doesn't do that we could write stuff to do that but at the end of the day it's not what it's meant for and also that would require us to create Jenkins templates and then people customers could jump in and modify the templates and muck themselves up and now we're forced to go and fix that so it meant from our perspective that just making a wrapper do that is the best route any other questions yes sir so this is a service that black mesh offers so we give this basically we sell this as part of our stuff much like you get anywhere else the main difference between this and others is that this is dedicated to you this is not multi-tenant this is single-tenant it is single-tenant to you from the perspective of that's why we can allow you to plug into Jenkins you have access to this stuff you have access to GitLab you have access to all these hooks and all these different changes that is completely for you and everybody has to fit in this deployment in this window and that said there was a lot of stuff that we did with Ansible that we have made open source if you check us out on github port slash black mesh we did put as much as we could out there so there's a lot of stuff there was another question I'm going to show an example of we'll run an error handle in this error if somebody does that hot fix in production and doesn't do that does that get caught? the foundation when you do a hot fix is to go ahead and do a merge back from production down to staging and staging down to dev you can do that through the interface or you can do that through get checkout I will say that the interface will give you errors so I showed essentially in this scenario everything was green so if this had been an error and you clicked it you would essentially see exactly what happened and we've run into the very scenario that you've mentioned and it will essentially say in this scenario that the merge failed because there was conflicts and it gives the link actually into gitlab where you can see all your pushes just like github has a list you can go into here and now see all your pushes but also you can go in and look at your merge request and you can look at your merge request that failed and get your diff and see exactly why you can't resolve it through the interface that still is going to require you to do code changes and do pushes but yes you have the tools to figure out why and to be clear how we delineate what we do and what you do as a developer is that you know your code and we're expecting you to know Jenkins jobs and stuff like that but what we're doing is we're basically putting that wrapper around it to make it look nice so that you don't have to know you have to manually go and type all those commands all the time, every time and that's at the end of the day we're system administrators that know how code works we're not developers that's the delineation on the line it's a very fine line and it's hard to go on one side or the other but we try to stay on that line as much as we can so if you don't have Jenkins you don't actually have to know Jenkins unless you want to he's right, you have to know Jenkins if you want to do more with Jenkins but just out of the gate you don't have to know anything about Jenkins in fact if you don't want the login you won't give it to you Is there anything that prevents this tool from being used to manage servers through the third party? No because it's SSH it's similar to a beanstalk service you can use beanstalk to do anything anywhere so what we've made as a policy is that if you have services with us and we give you this platform to use that with our services we'll work with you if you have services elsewhere to basically integrate with those as well Any other questions? If you want another demo or a more closer look at this come down to our booth we're pretty much at the front come find us, one of us will be able to talk to you at least a little more in depth in particular, maybe how we've been how some of these things are happening or things of that nature we're happy to talk about it Thank you