 So welcome to GitLab CI, a template for reusability. I'm Francis and I'm a project specialist at Cold Front Labs. I work on automating tests and workflows for our projects. And I'm Nick. I do a lot of like front-end JavaScript stuff at Cold Front Labs as well as Drupal theming and a little bit of module development. So today we're going to talk about GitLab CI. And before we get into GitLab and what you can do, all the amazing things you can do with it, we're going to quickly go and talk about continuous integration and what it is. Just a note on terminology. I'm going to use the words merge and merge request quite a bit today. These are similar to the terms pull and pull request in other systems. But in GitLab you say merge, so just so I don't lose you on the terminology, it has to do with the practice of integrating code into a repository. Or yeah, the act of integrating code into a repository. So at its most basic, continuous integration is the practice of merging changes into a central repository on a regular basis and running automated processes, builds, tests, etc. to verify that the build is functional. Instead of keeping code on individual computers, developers merge their code into a shared repository frequently, generally on a daily basis or more often. So that everyone has access to the latest version of the code. Depending on who you ask, there are anywhere between like 3 and 30 principles of CI. Today we're going to keep things simple and highlight just a few that everyone pretty much agrees on. The main tenet, I guess, would be that code is stored, maintained, and updated in a single central repository. This doesn't mean on your production website. This means by a version control system such as Git. Changes to the code should be small and modular and merged in on a regular basis. And every commit should be tested. Automated self-testing builds should be run to ensure that the build is working. So some of the benefits of CI have to do with reducing time spent, tracking down bugs, speeding up code integration, and increasing visibility and transparency. Because you're constantly running tests on your code, bugs are found quickly. It's also generally easy to isolate a bug, or the cause of a bug, because code changes are kept small and in small manageable chunks. Another benefit of making incremental changes is that there's less likelihood of integration conflicts. When coding conflicts do arise, they're generally pretty easy to deal with because you're only dealing with a small number of changes. That decreases the chance of making mistakes when dealing with conflicts and also avoids compounding issues from arising. Since the code is stored in a single place, everyone always has access to the latest version and can tell the status of the build at any time without having to track down individuals and ask how a feature is doing or where a bug is at, or all of that stuff. So the major takeaway about CI is that it saves you time. Having everything in well-tested, manageable chunks speeds up the development process and your automated CI processes give you confidence in your code because it is being built and tested with each commit. The workflow for the CI process is simple. You have your central repository where your code is saved. This might be your master branch if you're using Git. And changes to this code are made by making copies of the code and merging in small changes back to the main repository on a regular basis. In our project, we follow a feature branch workflow for CI. We create branches of code for building and testing, and then these branches are merged back into our main master branch. Finally, processes such as code checks, tests and builds are automated so that they are run on every commit. Every change you make to your code is tested and verified. Some examples of automated tasks include running a linter to check your coding syntax and standards, running compilers to ensure your code is building, and running validation tests to ensure that functionality is working and to check for regressions too. Whether using feature branches or committing directly to the master branch, it's important to remember the continuous and continuous integration. To realize the benefits of CI, you have to commit often and keep changes to a reasonable size. For example, it's much easier to read something like this in which a dependency, a single dependency, has been added to the project than to read something like this where there are many, many changes, like images being added, dependencies added, components created, and configurations changed all in a single change. So sure you can read through large volumes of changes, but it's much quicker to deal with things in those small chunks that we talked about previously. Okay, so Nick told me I shouldn't spring pop quizzes and presentations, but does anyone know what CD stands for? Just yell it out. Okay, continuous delivery. So actually it was a bit of a trick question, but I guess it actually gave more people the opportunity to get it right. Continuous deployment or continuous delivery basically is following the CI process, but then tacking on automated deployments to your automated processes. These deployments might be triggered automatically, which would be continuous deployment, or continuous delivery in which case you are manually triggering your deployments, but they're still part of your automated processes. In our examples today, we're going to include automated deployments as part of our CI. Okay, so now that you all are masters of the CI CD lifestyle, we're going to get into our main topic here, GitLab CI. Before we go into the nitty-gritty GitLab CI stuff, we're going to tell you why and how we use it. So the first and most obvious reason is that GitLab, we host a lot of our projects on GitLab. We have for internal use, serve personal projects or clients, and GitLab CI is built into GitLab and it's free to use. The next one is it's really easy to configure. We'll get into this a little bit later, but essentially all you have to do is add your GitLab CI file into the root of your repository and you're done. It'll run every time you push. Next, it offers really good process control. You could run individual pipelines without necessarily having to make code changes. So if you just needed to run a test again, you could go through the GitLab UI, go into pipelines, and run your stuff manually without having it to pick up on changes and go. And you can also save artifacts to your build. And what this is basically, when we run a production deployment, GitLab CI saves a compiled code version of our build, which is the artifact, so that we can go and download it later if we need to. We need to figure out what went wrong or was it broken when we built it and all that kind of stuff. And for those of you who know our boss, he also told us to use it and so we had to. So at Co-Front, we use GitLab CI for a lot of different things. We run a lot of code checks and linting on our PHP and our SAS and our JavaScript code. We use it to build and deploy our Drupal 7 and 8 sites, UJS projects, and even Docker images. And we run validation like backstop, codeception, AX, and a couple of other things on our sites. And we run security audits using Drush, Composer, and MPM. So now that we've told you why we use it and what we use it for, let's start things off with some basic terminology. The whole GitLab CI process is kind of broken down into four main things. Runners, pipelines, stages, and jobs. So a runner is basically a machine, typically a virtual machine, that takes all of your code from your repository, all destructions in your CI file, and basically executes any tasks. It runs things. GitLab provides a set of free runners for you to use if you're on GitLab.com. There are some usage limits. They're pretty high though. I think you are allowed to run 1,000 pipelines per month. Is it? Okay. Typically, you would create your own runner so you don't run into any limitations, and you can optimize it for your own processes. Runners are what makes your CI process go. They run your pipelines. Pipeline is a pretty generic thing. It's the wrapper term for your jobs that you're running. It basically just contains all your series of automated tasks that you run on your code, and they contain multiple stages. A stage is a category within your pipeline. You may have a lit stage, or a build stage, or a deploy stage, probably all of those things. The order in which you declare the stages are the order in which they are run. They run consecutively, which means basically one stage must complete before the stages after it can begin to run. The lit stage must finish before the build stage begins, and the deploy stage has to wait for the builds to finish before it can also run. An important thing to know about your stages, though, is that while they define the general order of your pipeline, they don't actually execute anything. That's what jobs are for. A job is an individual task contained within the stage. It's where the functionality of all your CI processes are written. You can have multiple jobs contained within the same stage. For example, when you're linting something, you probably maybe have JavaScript and SAS. You're going to want to run tests on your ES lint for JavaScript, style lint for your SAS code. It doesn't make a whole lot of sense to run ES lint and style lint in the exact same job. Yes, they're both linting processes, but they're for two different languages. They should be separated into two different jobs. Within your lint stage, you could have an ES lint job and a style lint job. Another important thing to remember about jobs is that unlike the stages that they're contained in, they run in parallel. When your lint stage begins, both your ES lint and your style lint jobs will start running at the exact same time. As long as there are runners. As long as there are runners, yes. Once all your jobs in a stage have completed, that stage is then completed itself, and then the next stage with all of the jobs inside of it can start running. So let's visualize. So here you can see we have some groupings of tasks. All of these tasks are what makes up your pipeline. In our pipeline, we have three categories, lint, build, and deploy. These are our stages. Within each stage, we can see some individual tasks. These are jobs. We have ES lint and style lint in the lint stage. We have a job called build in the build stage and a job called deploy in the deploy stage. So when a pipeline begins, all the code and instructions are sent to your runner and all the jobs within the first stage will begin to run. Now, just because all the jobs started at the same time does not mean that they will finish at the same time. Fortunately, before the build stage can begin, all of the jobs within the lint stage must succeed. Once all the lint jobs have succeeded, our build can go, and this process basically just repeats throughout the whole pipeline. Each stage must finish before the next stage can begin. So let's take a look at configuring a basic pipeline. So Nick mentioned the CI file, which is used to control your pipelines. Right, I forgot a slide, which I always do. Here we go, to create a pipeline. So GitLab CI is controlled by file called the GitLab CI YAML file. This, as I mentioned, is used to tell GitLab what to do. It defines the pipeline. And GitLab will automatically detect this file when you place it at the root of your repository. So let's go have a look at that file. So you can see at the top of this file, I have defined the stages. In this case, we have a super simple example. We have three stages, lint, build, and deploy. Down here you can see the jobs. I have an ES lint job in the lint stage, a build stage, and deploying the deploy stage. Now, to see how well you've been listening, can anyone tell me why we would have three stages when we only have three jobs and we could just have them in a single stage? Why would we have these jobs in individual stages? Exactly. So you need them to run one after another. So in this case, you can see that we have very simple scripts. The scripts are what tells the runner exactly what to do during the job. We just run a linter to check syntax, run a build, and then deploy that build out to a server. You'll notice that in the build stage, we create an artifact, and that's what is used, grabbed by our deploy stage, to copy over to our web server. Let's go take a look at this in the GitLab UI. That's a little tiny. There we go. Sorry, I didn't check that one. That's okay. Oh, and something I didn't mention. I'll just click back here. You can control when artifacts expire, so you probably don't want these, if you're billing as much as we are. You probably don't want those artifacts sticking around too long, so you can control when, how long you want GitLab to hold onto this for you. All right. So you can see that the latest on the Pipelines page of the GitLab UI, you can see that I have my demo branch here, and I've actually had all the stages succeed. You can click on a stage to see the builds that were run within that stage, and you can see here that my linting task actually failed earlier. So let's go have a look at the terminal output for that job. Here you can see that it ran the NPM install, which is fine, but it did fail on the NPM run lint because of a syntax error in my main.js file. If we go back to the Pipeline after fixing the linting error, let's go have a look at that build that we ran. You can see here that those job artifacts can be accessed, downloaded, or you can browse to the files that were saved in that artifact we specified in the file. You can also, kind of a handy feature that I like is if you create merge requests, you can go and see the most recent status of the Pipeline that was run against that branch for the merger request. Another handy little tip, you can also run jobs again, which is something Nick mentioned, and you can also run whole Pipelines if you have a reason to want to run a specific branch at a later point. Going back to our GitLab CI file, I'm going to go to version two. There we go. Something you might have picked up on in the last version of the file, I'm not sure if anyone was paying that much attention, but we had some repetition going on in our job scripts because I had NPM install written in every script parameter. In this version, I've moved that up to the before script, which is a special keyword in GitLab CI, which will just run all of those commands you list in the before script array prior to every job in the file. There are also a series of other keywords such as after script so that you can kind of reduce redundancy in this CI file. As you can imagine, as we add jobs, things easily get, you'd have a lot of repetition going on. In this version of the file, I've also added two new deploy jobs. I've added a deploy dev and deploy QA because I didn't always want to be deploying to my production server. Once we introduce jobs like this, I don't actually want to run all three of those jobs every time I run the Pipeline. We've added these only and accept parameters to make sure that we deploy to dev whenever a commit is made to a branch, except in cases where we commit to the master branch. Then the deploy to QA will happen whenever we make a change to the master branch. I've decided to trigger my deploying to prod manually because I don't want to accidentally have that happen at any point. Taking a look at these scripts, you can see that actually there's a lot of duplicated code here because pretty much everything is the same between my deploy jobs except for the location where I'm actually copying that build to. This gets especially troublesome when you imagine adding multiple validation jobs to test code on all of your servers and adding we get to the point where we have nine stages and probably 20 to 30 jobs. It adds a lot of bulk to your CI file if you're repeating code. What's the answer to this problem of bloat templates? Okay, so templates. The reason you probably have come to this presentation. Okay, so the LBSIA templates are pretty much exactly what they sound like. They're pre-made jobs that you have created that you can extend to fit the needs of all your stages and reduce the code. You can have scripts, stages, onlys, accepts, dependencies, anything that another job could have. That's what a template can also have. All of these parameters are inherited by the job that's extended to template and you can customize and override each parameter whenever you need just in case you need something a little different. In our previous example, we had three different jobs for deploying to three different servers. We could easily create a generic deploy template to handle all of these. So let's take a look at how we would do that. So here we have our production deployment job. I've taken the liberty of removing some of the functional, the screen went away. You mentioned backstabs. Yes. What exactly do you get just from a result of a build? Essentially. I'll let you. Copying the files back to get that so that they're attached to your job and so that they're accessible by the runners to use in addition to calling them. Because if you compile your code and then the next job needs to deploy that code, it needs to be attached to that. So it just makes the next job be able to... GitLab has like a storage for all of this stuff. It has some temporary storage that you can use. Typically you would make your own caching server. So we're going to see you deploy... No, not ours. For the most part, we also have our own GitLab instance so we control the whole server. Yep. I noticed you used NGAM install. Yes. Have you ever used NGAM CI? We sure do. Yeah. I actually just had a build issue when I was trying to make the demo and I wasn't going to troubleshoot it so I just went back to install. We're recording. It's faster in there. It's faster in the lock files, I suspect. Because fun fact, your NGAM locks like the package-lock.json file is read first and then the package.json file is read to install any updates. So your lock file does squat all and then you use... Well, that'd be... Okay. Okay. Caring off. So we have our deployment job here for prod. So I've replaced some of the functional code with comments because you guys get this. We don't need to see all of the code. So we know by looking at our previous examples that this job looks pretty much identical to our deployment jobs for dev and QA except for the server. So they're all part of the deploy stage. They all owe the SSH key. They all are sync. They're all basically the same. So the first step in turning this into a template is to rename our job. So we've named it deploy template because we're just telling everyone what it is. So you'll note the period before the name. This is a handy little tool that tells GitLab that this is not just an ordinary job. It won't run it because it's hidden from the CI process. So next we'll get rid of our only and accept parameters. You could have default values for them if you wanted which is totally cool to do but for our purposes right now we won't worry about that. So now looking at the script parameter the only thing we have left is the big problem. The server is hard coded right into the script. To solve this we will take advantage of variables. So variables allow us to really harness the power of GitLab CI templates. Just like any other language, variables are data stored, key value pairs, you're good. You can declare variables in a few different methods and levels which we won't go too far into but essentially you can declare them per job, per pipeline, per project and even per group inside of GitLab itself. So you can use variables pretty much anywhere you want within your builds except for a few small exceptions and those exceptions can be found in the GitLab CI documentation about variables. Is your scope on there? Yes. So depending on where you would declare them that's where the scope is. When the CI process begins on your runner all of the variables in your various scopes are loaded as environment variables just into your terminal so you would access them like you would any other command line environment variable. So let's see how we can use variables to finish off this template. So to declare a variable in a template we use the variables parameter which is a basic Amel Array. So now that we have the target server variable available to us we can simply remove the hard-coded server and plug in our variable. So now as you can clearly see our slides are using bash so we access our variable through the dollar sign and then the variable name. Obviously if you're using Windows Batch or PowerShell or something there will be a slightly different syntax but you get the gist. The tip is to actually use our template to allow jobs to pull from this newly created template and extend its functionality we use the extends parameter. This basically tells GitLab that when it gets to our job it needs to go look at our template first and use that as the base job then look at our new job and make any overrides or customizations based on that, you know, like a template. So let's look how we can use this template in our deployment job. The first step is to tell GitLab that we are extending our new template using the extends parameter. Next we need to make use of our target server variable and give it a value for this job so in this case just drop for it.com Finally we'll add our only accept parameters back because they're specific to our job we need to make sure that we don't accidentally have a dev build deploy to production that would be unfortunate. So following these steps we could upgrade our dev and qa stages as well jobs rather as well So here you can see we've extended our template declared our variables and made our only accept rules pretend they're real only accept rules. So while these templates help reduce duplicate code it may not seem so important in this situation we only have a few jobs to execute the same code but when you have multiple types of builds multiple types of deploys a whole bunch of tests that run before and after those deployments the repetition can add up really quick. Extends and variables is great they're all very nice to use but what if you have multiple projects on GitLab that need these templates You start off with one project with pipeline and template then you add another and another one more templates, more projects more pipelines, more builds it's just getting ridiculous So you may be like our good friend Leo here and think that that's just way too much work to support all those projects with their CI builds and all their templates and you're right it is way too much work we would know we used to do it we would have all of our clients projects and all of our projects with their own CI build files all using these sort of templates we've made in our files all extending them and then if we updated one of the projects we would then have to make sure to go through all of our other projects and update those as well to make sure that all of our builds are on the same stage there and this is where the final key feature of templating comes into play the include parameter the include parameter is a global parameter just like when you're declaring your stages it goes right at the top of your GitLab CI file it functions as a basic list of URLs all pointing to different gamma files when you add a gamma file within the includes parameter GitLab CI will basically look at the URL go to that URL and embed all of that AML as if it were written in your CI file itself this allows you to have a single source for template files all of which can be included extended and customized through all of your builds individually without having to update constantly so as an example these two slides are exactly the same as far as GitLab CI is concerned so not only does this allow you to include templates across multiple projects but it also drastically reduces the amount of code you have in your CI files so we'll end our official slide presentation with some important notes to keep in mind when you're working with templates all the parameters like script and stage they're unique this means that if your template has a stage and your job declares a different stage when the GitLab CI runs your job stage will be the only one that's respected it will not look at both of them a really important example for this is your scripts you may have a lot of default scripts written in a template if your job has a script parameter all of your default code in your template crushed it's gone and you have two scripts so if you need to customize the script for this one job even by like just ever so slightly you would have to rewrite all of the scripts for that job unless you can somehow work it into variables like we did the exception to this rule is when a parameter is basically just a list of key value pairs like variables in that case that will merge those two parameters so let's say you had your target server variable in your template and then you added a site alias variable in your job itself your target server will be respected they'll be merged into a list another important note is that jobs are limited to one template combine multiple templates into one which is only a temporary limitation coming in GitLab 12 they're changing the extends parameter to take a list of templates so you could have multiple templates combined into one job where that could be useful is like for us we have our basic set of like only in accept rules for deploying to different environments that we just copy throughout everything all of our projects follow the same tagging and branch naming standards it would be nice for us to have a template that is just the dev deployment categories and extend that with another template so that when we make a dev deploy we can say take the deploy take the dev deploy and then we're good so we do have I believe we have some time left so we can do questions although we had a couple however we can also show you how we deploy Drupal with GitLab CI and it's going to be very technical because we're basically going to show you how we template it all out so I actually have it here okay is this big enough? Do you guys want a little bit bigger? good so essentially when we're making a build we have our template for building Drupal which essentially is just making sure composer is run and we add some default artifacts just naming our artifact and how the artifact will be saved and then we kind of customize it what we do with our templates is we kind of make templates from our templates to import into our jobs because basically our dev build is going to be the same pretty much every project we do our Drupal dev wants to build the exact same way so we have our build template then we have just a job that is the dev build and then we include that throughout all of our projects so that if we change the dev build once anywhere all of them will update and make sure they all deploy the same way so here you can kind of see some examples of the artifacts here is a good example of where parameters that our key value pairs are merged crushed so in our template we declare the name of our artifact and what paths it needs to save as the artifact and then in our dev build we have artifacts again and we're adding a new key to tell it that our dev build should expire in one day scroll that up a little bit so in this case artifacts in both the template are kept, they're just merged into one so then we have a slightly different process for our release builds where we get into a lot of crazy templating is our deployment stage so we have two different kinds of deployments I won't go into initial deployment just because it's a weird use case and we have to deal a lot with Drush which I don't want to talk about dealing with Drush so essentially how we deploy to Drupal is we build it all through Composer and then just use Drush site aliases to arsenic everything where it needs to go so the whole site, aside from the database is built on our runner and then we just copy it all over so you can see a lot of this is where variables come into play because you see we have Drush site aliases options SSH options our alias group our alias alias type SSH everything and then we run a bunch of other commands to get Drush up to date we even wrote our own backup script because Drush9 took out the ability to run backups on your site because why not it doesn't work so we made our own little PHP script that is essentially Drush archive so we do a whole lot of things we run the Drupal fix permissions script that you can get on Drupal.org with a little customization by us and essentially it's all automatic we can go from no dev site on the server at all to a completely functioning dev site with just clicking the CI build run so it depends on which one we're running we have our initial Drupal deploy this actually creates a database and gives it all that with the MySQL URL our normal deploy we just update the data and move on and then after we've deployed everything we actually run a bunch of updates so we automatically run like Drush up DB we import all of our config and we do some more permission fixing just because we've noticed that GitLab CI sometimes screw up some of the permissions I did consider showing you a full demo of it like running but sometimes it can take upwards of our entire presentation time to fully deploy like a functional Drupal site so that didn't really make a whole lot of sense unless I hit play at the beginning and then showed you the end result and so then we have it split out so we have our update template which has our basic stuff and then we extend this template further so that we update our feature environment specifically we update our dev environments and update our QA environments our prod environments and most of these are just taking from the template with a small exception of like we changed the Drush alias type variable just so when we run Drush it knows what server to go to what to do another example are we on oh yeah we are we are done aren't we ok question where do you decide to use SSH keys? so our SSH key is loaded I can actually show you we've gone like all out in this template essentially for all of our projects we just have a list of imported files and that's all our CI consists of for the most part so when we load SSH keys they are variables stored on a project level so in GitLab you can go into the UI and actually make variables and those are inserted into every pipeline in that project which is where we keep the actual keys yeah they're not committed to the repo so they're stored as like secure variables that only maintainers and owners can actually access and then basically we have this before script run and all it does is it looks for our variable and just loads it in and that way all of our jobs can access our dev and prod servers as needed and another cool thing although if you want to talk to me like after this you can come find me because we also have stuff that like when our lint fails it actually runs a fix and gives you that artifact so that you can download that apply those changes and commit them back up it is technically public I'm just going over with Matt, our boss just if we are actually going to like give it out fully when the slides become available you'll get the URL yes okay well thank you