 Hello and welcome, everyone. Just want to send a warm welcome to you. Thanks for joining us today. We're really excited to have you here. In this webcast, we're going to be covering GitLab for complex CI CD, robust, visible pipelines. My name is Darwin, and I'm a Senior Solution Architect here at GitLab. And I'm joining you today from Pennsylvania. We'd love to hear where you're joining us from, so feel free to drop your location into the chat so we can get an idea of where everyone's coming from. Before we get started in earnest, I'm going to cover a couple housekeeping items. Essentially, we're very open to you asking questions as we go along through the session. There's a special place on the Q&A icon in order to submit questions. And please go ahead and enter them as you think of them, but we'll be deferring them till the end of the presentation and handling them at that time. Also, if you're experiencing any technical difficulties, feel free to reach out to me on chat, and I'll try to help you as much as I can. Our presenter today is Joel Krusewick. He's a Solutions Architect Manager from the Chicago area. And we're going to be launching a couple of polls as we go through the webcast to learn a little bit more about you. And then that way, Joel can also tailor his presentation to some of your concerns or what license level you're at or that kind of thing. So I'm actually going to launch the very first poll right away. And if you can just take a minute to respond to those, we'll leave it up for a couple of minutes and then close that out. So we'll take a few minutes there and then... Okay, people are responding so we know everyone can click on their polls. That's great. So without any further ado, I'd like to turn it over to Joel for our main content. All right, thank you, Darwin, and thank you, everybody, for joining us. Today's topic you'll note is on GitLab Premium. However, we're going to add value to everybody, no matter what level or what GitLab package you're on. So my goal here today was to give you some tips, to give you some insights into some of the things that are provided to you by GitLab that you may or may not have considered so far. You don't have to be someone who's done a lot with GitLab, however, if you have, I hope you'll still find a lot of value in the things we're going to talk about today. Now, you note the title here for complex CICD and robust pipelines, what we're going to talk a little bit about is performance and then we'll talk a little bit more about microservices and we'll talk a little bit about configuring multi-project pipelines and some things like that today and some of the performance things to keep in mind as you do that. So without further ado, let's get started. So when I take a look at what GitLab puts out on our website, it often looks something like this. So if you've been around for a while, if you visited our website, you've probably seen some diagrams like this. I pulled this particular one out because it focuses on the primary use case, right? This is the thing that most people have bought GitLab to support, the idea of source code management plus CI and increasingly CD. We get a lot of questions on CD. Now, within this diagram, on the left-hand side there, when we're talking about coding and committing, there's a lot of times that we need to understand the auditability, the traceability, the mergeability, what limitations or regulations do we have on who can do what, that change control component. And Darwin actually did a great presentation on this. It was the first one in this series that you're currently in. So if you haven't seen it, look for the one on rich change controls that was recently sent to YouTube as the predecessor to this that talks more in depth about that left side of the diagram. For today though, we're gonna spend our time talking about the CI CD side. And there's a series of questions that we frequently hear. Darwin and I spend our days interacting with customers on a regular basis. And so what we wanted to tackle is some of the more common questions that we hear essentially. Number one is performance. How do I make my pipeline faster? We hear this every day. Someone's always looking to speed up the things that they're doing today. And so we're gonna talk through some very practical things that you can do with GitLab CI in order to accelerate your pipelines. What registries should I use? When we're talking about this from the perspective of Docker, Maven, NPM or even any kind of dependency proxy, there are a lot of different registries available. We'll actually go there first today to talk about those items that are available and what the advantages of using them inside of GitLab. What about microservices? What do I do with these things? How do we orchestrate these? How do we make sure that we can see what's going on with all the dependencies or across maybe 50 different repositories? How does that work? We'll touch on that today as well. What deployed where? Can I see the code that went out to my target environments? What does it look like within those environments? What is the health of the Kubernetes clusters and pods that may be hosting those environments? Again, we'll be looking at some of those things as well. Two last questions. Who approved the deployment? So how did it get there? What's the visibility to what that is that was shoved into production or into the staging environment? Who put it there? What are the changes? Who's approved to do those things? Okay, there's all those controls that are part of this whole function. And last but not least, if I'm using GitLab CI, can I use this with my other source code managers? So not everybody can jump right out of their BitBucket or jump out of their GitHub environments right into GitLab for SCM. If that's you, then today's webinar applies to you in its entirety. Yes, you can do that. Now it does require GitLab Premium in order for that to work seamlessly. And that's of course part of this content today. But know that no matter what you see today, the things I'm gonna discuss and describe can in fact work with other source code management systems if those are things that you have to use today. All right, with that, let's jump into package management and registries for just a minute. Now you see the little S and B there that's starter and bronze. This also could apply to core packages, but we're just talking about the enterprise version of GitLab today. So the idea here being container registry is readily available to everyone. Now, if you don't see container registry in the GitLab version you're using today, chances are your admin hasn't turned it on. It's really that simple. However, there's also other usability limitations on this. You're not gonna be able to access that registry unless you've got personal access tokens, OAuth tokens or the ever convenient CI token that allows you to access and pull and push Docker components and Docker containers up into this registry. Okay, so those are some regulations, some gates before you can actually leverage the container registry. Most people are familiar with this. They've used this instead of using Docker Hub or using ECR and AWS, they're using our local container registry. And so that's one of the ways that when we start talking about performance we're gonna come back to these pieces and the why that it's so important to consider using these local registries. Now as part of GitLab premium or the silver package on gitlab.com, you're now able to leverage things like Maven, Conan and NPM registries and very soon Nougat. Okay, these are all different package management solutions that we have in place inside of GitLab today. There's one third one that a lot of people aren't aware of and it's this idea of the dependency proxy. Okay, this is yet another one that is in development right now. It's active but only for public groups today. So there's a lot more development and filtering around this happening. But to leverage this appropriately into a pipeline might look something like this where I'm pulling a container from my registry that I'm going to leverage. And maybe I have upstream dependencies defined and those will be stored in my dependency proxy. That registry there along with this other container registry can provide the components needed for me to process my builds and the rest of my CI pipelines. When I'm done with that CI pipeline I may upload a container to the container registry which could be leveraged for say a Kubernetes deployment or I could store an artifact into the NPM registry for instance, which that package then may be pulled into a separate configuration pipeline of some kind from the CD side and distributed into one or more environments. Okay, so you can see that these work together in a pretty interesting way. Why is it important from a performance perspective? Well, a couple of reasons. One being of course, the closer everything lives together the less network latency we have to deal with, right? This is pretty simple idea that the more co-located things are the less time it takes for things to travel between the destinations. So if your container registry can live closer to your CI and your package management can also live closer to your CI engine ultimately at the end of the day you're saving time and energy. Now, one thing I want to mention here is this is a pretty common solution, right? It's a pretty common approach that we're seeing used by a lot of GitLab customers today. There are some exceptions to this where if you're doing all of this say within AWS environment or something like that, you may want to consider using ECR at times or something like that to support whatever is closest from a registry component to where your runners reside and to where your code deployment is will touch a little bit on runners again in a moment. And just as a quick screenshot example of what you might get with the NPM registry, all of the packages that are uploaded must be scoped and must be versioned. So and as such, you see here the versioned and the scoped labels and then of course the uploaded package just leverages both of those, how convenient. So the next step here, I wanted to get into some really practical tips. If you've used GitLab CI at all in the past then you may be familiar with the fact that we use YAML files, YAML content in order to manage the pipeline. There's some tips and tricks to this and some common usage components that are really important. And so I wanted to get into these. Now complexity unto itself is something to be aware of. Remember first that complexity kills DevOps. There's just something too complexity that we want to keep in mind every time we start a project like this. So we're gonna begin with just some basic steps for a single project pipeline. Some of this is gonna be actual YAML content and then we'll show you some visualizations. So for instance, in this case, the first things first, if you've used any GitLab CI in the past, you know we separate into stages and jobs. Stages being the sequential flow or the serial flow, one thing happens then the next thing happens. Each stage must follow on from the one before it. Inside of those stages though, we have parallel jobs. So these jobs can run in parallel within a stage. So obviously the more we can maximize the number of jobs within a stage, the more performance we can bring out of our pipelines. And so you might see something like this where you've got a couple of jobs that run under the prepare stage and then a bunch more jobs that run under the build stage. Now obviously the jobs in the build stage might be dependent on the prepare stage. The prepare stage might be pushing something into those registries and repositories we just talked about. So we may have to leverage those during our build stage. So this is a pretty simple concept and it's one that most people already understand. But is there a way to speed up parallel jobs? Well, it turns out there is. We've released directed acyclic graphs. The idea here being, I have the ability to move between stages without necessarily finishing the jobs within a given stage. So in the previous example, you see I had to complete crop pictures and enforce relative links before I could move on to the build stage. On the right hand side, you see we've got Linux and we've got Mac components, two separate environments and there's actually three different stages present. What that would look like is three different stages. However, in this example, you see the bottom is all green already. Because of that code word needs that you see in blue on the screen, the bottom line there was able to complete and do a deployment before the top line got past the build stage. So think of it this way, if you've got a complex pipeline, you've got especially something with a complex monorepo or one giant repo to manage a lot of services, you can actually expedite certain facilities for certain jobs within your pipelines and run them to completion while a large systematic job may take a long time to process. It's not gonna hold you up. So this is something I love to point out to people the idea of using DAG as part of your pipelines. And you'll notice this slide as well as the last one, both are speaking to the generic starter and bronze and core, the baseline user set. These are not something that are in GitLab Premium itself, but I feel like they're really important parts of the usability of GitLab that we have to consider. So a couple more things you might wanna look at, run rules. So we're working actually on a run rules engine specifically, but in the meantime, my favorites are only and accept. These rules allow you to say, hey, these code files did not change, so I'm not gonna run these jobs or I only wanna run on certain branches or only when we're merging to master. We've got the ability to break it down and say only certain jobs should run at certain times and that's of course gonna save us time. If we can save a full stage at a time, even better. So we wanna be careful with what jobs are running when, especially as complexity increases, the larger some of these repos get and the more dependent services we have, some of these run rules can save us enormous amounts of time. Think a GitLab CI YAML file, being something over a thousand lines long, we certainly don't wanna run all of those jobs every time. Another example, and this has come up a lot recently in conversation that I've had with customers, the idea of runner caching. So the runner is the executor of the jobs. If you have a CI job, the runner is polling for it, picks up the job, executes the job and there's the option to store the artifact there on the runner. So we talked about the registries, depending on where that runner is actually executing its code, we don't necessarily want that runner to return artifacts back to the registries, pull them out for the next job. We'd love to have that work stored local. So these artifacts can be stored right there in the defined paths on the runner itself. So maybe that's in a cloud, maybe it's in a target destination, somewhere close to your deploy environments. This is one way that you can really, really speed up your pipelines is by leveraging caching to the fullest extent possible. One other quick tip on this, if you're leveraging caching today, tags are a great way for you to make sure that you're always using the runner that's optimized for your caching exercises, okay? So when you're doing the caching, when you've got tags that you're leveraging, you can ensure that you're gonna go right back to the runner that you need when you most need it. Another quick one to point out here, if you're doing testing, especially, we have a parallels keywords. So you can parallelize a whole bunch of tests and get through them quicker. This is pretty self-explanatory, but I certainly wanted to call it out as an option that's available to you. And then last but not least, we have the options of after scripts and before scripts. And before scripts is where we can see a lot more difference. And so the idea here is there's an option to either use the before script or use the first job in order to configure things that you're gonna use in the rest of your pipeline. Okay, that first step is important. So if you have a lot of container preparation, like you're gonna build up in a before script, it might be a sign that you should instead actually maybe use a Docker file or something like that and use that to build your container instead. Okay, so you might wanna turn that into a separate job, not necessarily just a before script. The more preparation and work you're doing there, it can dramatically accelerate or decelerate build times when there's a lot of build dependencies. And so the idea here is you're getting ready for the build. You want that to be as fast as possible. I encourage you to just take a look when you're doing something to set up a container or to prepare for a build, look at how much work it is, look at how much complexity is, look how much rework you might need in your initial jobs and compare and contrast. In fact, you could even run it both ways and see which way is faster. We're finding some interesting variances in this conversation recently. So I wanted to make sure I brought it up here today. However, this is all just simple project pipelines, right? This has been kind of a single project, a single pipeline kind of approach. Let's start layering some complexity on this. So when we start triggering downstream pipelines, the idea that I've run a pipeline and now in order to take next steps, maybe I've got more dependencies to manage or I've got microservices based, build that it's gonna be across multiple repositories, that's where something like this comes into play. This is optimized by GitLab Premium. You've got a script here that you see on your screen before you, that's just an API based trigger. So if you like working with APIs, this is certainly one way you can do it where you're actually triggering a downstream pipeline with GitLab, leveraging the GitLab API. It's not my preferred method, my preferred method is actually to use a path-based trigger that allows you to trigger a downstream job or downstream pipeline, using the path to the project, just using GitLab Pathing. So you've got the full path to another project that you see listed there. This example is a little more complex. I also added in the idea of passing variables in, specifying a branch in that target project and the strategy piece there, which when it says depend, ensures that the upstream pipeline has had to pass before the downstream pipeline will execute. So just a few things there that you can do with this that can add some layers of complexity and some gating mechanisms on what's going to actually run. So what's the visualization of some of the things that we've been talking about? Well, first and foremost, this is that single project view. And this particular view is actually of our auto DevOps pipeline. So when you see this, this is GitLab auto DevOps, an example of what we believe to be a really good, sufficient pipeline to get you going. And so you can see the build, the test, the security scans in there, the code quality job, the review app that gets spun up, dynamic security scanning that happens because the review app is now live and even a browser performance scan. This is all part of auto DevOps. However, the one thing that I wanted to point out in here and the reason I pulled this screenshot up specifically, we talked a little bit early on about the registries. The code quality job in auto DevOps will pull natively from the web, okay? It's gonna natively go out and say, I need the latest code climate image before I execute the job. That is the largest variable in my pipeline in my personal project right now. That's the one that can take up to seven minutes to download the current image depending on a number of different variables. Seven minutes is a huge variance when you're talking about a 20 minute max pipeline. So this is just one of those examples where if you've optimized the idea of your code quality scans by hosting a local image, you might expedite your pipelines dramatically. So let's increase the complexity. Here's another pipeline that actually deploys into multiple environments. Now, this deployment itself, that job where it spins something up is great, but when we get to that trigger cross projects, that's where the magic happens. And that's where it's really important for GitLab premium usage because it's required for the visualization of this downstream trigger of the new pipelines. So what we're gonna do is continue forward. So those last four items that you saw with Android and such, I clicked the first one and got this whole other downstream pipeline that showed up. And now I can click into potentially the website where you see the yellow and get yet another pipeline rendering out to the right. So the layers of pipeline complexity that you can get into when you start talking about downstreams and configuration custom deploys and things like that, this can get really, really complex. The visualization is what helps you ensure that it's right and also helps you be able to see what's going on across a whole lot of jobs running all at once. There's one other view I wanted to call out real quick and that is our operations dashboard. Again, as part of premium, this particular view gives you things like what pipelines are executing. You can see here, I've got a pass and a failed pipeline for different projects as well as passing with warnings. I have the ability to quickly click into any of these and see what's going on. I like to see this as my at a glance pipeline activity or my at a glance pipeline health view. So the more pipelines you have running, the more projects or repositories that you have active, obviously the more value you take from a screen like this. All right, now you're gonna notice, we're gonna move into some deploy scenarios here and you can see that this is a different format to the slides and I tip to the hat back to Darwin there. Thank you for the last webinar in this series. This is some of his content that we're re-leveraging. So last time we focused a lot on the left hand side of this, the idea of all the change controls to get us to the point where the artifact has been created. Who has approved that code? Who has done the actual merge of any code that created this artifact? Now the artifact in this example is that puzzle piece in the middle. So think of this as something that's ready to go out into multiple environments. And this particular example takes that application. This is kind of that SAS type approach here of multiple artifact deployments for your customers. Okay, so if you think of it as you've got customer A and customer B there, perhaps each of these has a custom configuration and a deploy repository or something that's showing a different CI or CD functionality there at the end before it goes out into these different environments for your customer. The idea here being I've got one artifact that I've stored in our local registry and I'm gonna reuse that along with custom configuration on the CD side to push this stuff out to customers. And each one of these could be a different type of deployment, right? These could be config only builds for per environment, customize configuration. So that's one example. Here's a different one where it does kind of the same thing but in this case I've only got a couple of different configurations and I've got one most common configuration that we're gonna use to push out that information and that production deploy into all these different identical customer environments. And that's what you see there on the bottom. So these are different CI YAML configurations that you would set up. Some would be localized within the colored stripes there to say this are custom configurations for the deployment. However, that one on the bottom would actually be a lot of different deployments with a single configuration. And it would just be written in the YAML files accordingly. One last example. So when we talk about monorepos we talk about the idea of these giant repositories that are trying to do a lot of things. One of the top questions that I receive is, well, so we wanna do microservices when I do a multi-service approach, do we use lots of repositories or do we use one? I did the research, came back with a very unclear answer for you of about 50-50 is what we see in our customer base right now. So there's no right or wrong answer. There's no clear answer on exactly how to approach this issue. But there are some tips that I can give you from a pipeline perspective when it comes to doing something like this with a monorepo, okay? And that is you don't necessarily wanna manage all your CI within a single file in a single repo. So what you see here is the idea of an include. This again, back to the idea of GitLab CI, YAML coding itself, the include function allows you to pull in other existing YAML configurations and leverage them in your pipeline. So maybe that code is external, maybe it's part of a regulated security component to your pipeline that has to run every time. This ensures that that piece comes over cleanly. There's also the idea of an extends in the pipeline. And this is saying that you start with a plugin template and then you're adding value from there. You're adding extra functionality from there. So the difference here between the two is on the left-hand side, this is your primary pipeline and you're pulling building blocks together, kind of like the Legos of GitLab CI, if you will. On the right-hand side though, the most of the pipeline is already built and you're just gonna add value to it. Both of these are convenient for a couple reasons though. If you're in a regulated environment specifically, this now has components that are being pulled in that might be locked down. These are things that can't change. We have to run them with every pipeline. It's really important to include them. The extends is more of a standard approach. So this would be one where we have a lot of different code that's going to be built and tested the same way. And we would begin with a regulated pipeline of some kind that we can extend on from there as needed. So let's shift gears entirely for a minute. We talked a lot about pipelines, but we also mentioned visibility. And so I wanted to get into some of the Kubernetes insights you can get from GitLab. GitLab is trying to make it easier for you to get into the world of Kubernetes and to leverage existing clusters. And so in this case, I actually took a snapshot for my existing project. Now in my existing project, you can see it's all on the right side there just project level clusters. And I took this screenshot for two reasons. One was that this is project level clusters. We also have group level clusters. So all the projects that are part of a given group can leverage the clusters that are defined. However, this image is also just one that's kind of good for a laugh. I have a multi-cloud setup here, but you can see I've actually killed three of the clusters. If you look closely, you'll see I've got a dead one overloaded one and something that's disabled, okay? I'm still active in multi-cloud. The first line is a GKE cluster. The third one you can see clearly stated AWS. I have the ability to deploy from a single project into multiple different clouds, okay? And anything that's denominated with the dev environment scope for deployment into the dev environment, then that will go to the AWS environment. Everything else will go to GKE. When I do those deploys and I go to the environment tab within GitLab, what I will see is something like this. This example shows you about 20 to 25 pods that are active for my production deploy inside of GKE. And you also see that I have five of those that are defined as canary deployments. So those five are called out specifically as something different. And I can click into any one of those boxes to actually be taken to the terminal view and actually interact with those pods real-time. On the far right-hand side though, you see a little box with an arrow sticking out of it up into the right. That will open up the environment itself and show you the contents of one of the pods. So the website that you've deployed into those environments. If I reload that several times, it's gonna hop through the pods and it's gonna show me a canary on one out of every four or five loads and show me what that looks like. Now the canary deployment for any of those who haven't necessarily leveraged that is like an incremental rollout of sorts that basically allows you to test. So that might be my new website UI with a core set of people who log in or a randomized set of people who visit my site. This can get enormously complex and this particular screenshot I called out because of two things. One being that you see there's a lot more pods that are available in a more complex setup. We call these our deploy boards where you can see all the deployments across Kubernetes like that. But I wanted to show you this one specifically too because the top one shows an incremental rollout is actually occurring. The pods are being created. Those are the green boxes that don't have the fill yet, the fill color. The first ones are actually fully deployed and it's rolling out across them. So this kind of visibility you can kind of see your environments being created, rolling out, especially as you do them incrementally. And you can see that on these deploy boards. There's also the items at the bottom of the screen. And I apologize for the clarity of this particular image. It looks a little fuzzy to me. It's interesting. I don't recall taking a picture of this. I didn't know you could blur a screenshot. But it turns out those bottom half dozen or so are actually just bare metal servers and virtual machines. I can see into those environments. I can monitor those environments from a performance perspective, but they're not gonna have a deploy board because there's no breakout into pods and the like. So these are all my defined environments of various types on a single view. And we do have the ability to see deeper into Kubernetes because of its complexity. One thing I did not call out I didn't have in this particular screenshot is the idea of auto monitoring for these pods. So these pods that you see that are active, I can also track the performance of my Kubernetes cluster and the pods themselves. So we can see CPU and RAM utilization. We can see when things are headed in the wrong direction after a commit, all that kind of thing that we'd like to see. And that's automated by a simple one click install of Prometheus into your Kubernetes cluster early on and it will dynamically monitor all those pods for you. Again, I didn't have that available as part of this screenshot today, but it is a really nice component of what you're looking at here today. So just to wrap up on this real quick, these are some of the things that we talked about today as part of GitLab Premium. These are all different items that we walked through specifically. We didn't spend a lot of time on CI CD for external repositories, but again, everything you saw here would work the same way within GitLab, even if you use an external source code management solution. And the thing we didn't spend a lot of time on is the visibility piece. So we talked a little bit about Kubernetes visibility. There's of course a lot more visibility features to GitLab, including things like work management, workflow management, merge trains, analytics, feature flags and a whole lot more. So I just wanted to make sure I mentioned some of those items here. And at this point, I'd like to pause and see what kind of questions you may have. Thanks a lot, Joel. Great. Awesome overview. We have a question here. How do I ensure all of my pipelines include a core set of standard jobs? Yeah, that's actually a really good question. And we touched on at least three ways to do it today. And there's actually a fourth one coming. So the best methods to doing that include the triggered pipelines. So the idea that every job and pipeline that runs will now trigger a downstream pipeline that is the core included set of jobs that we're gonna want. The better way to do that would be the include or the extends that we mentioned where you now have a core set of jobs that are gonna be leveraged repeatedly. And those jobs would most likely be regulated and only available to admins and maintainers inside of a different repository. Sometimes those repositories are strictly created to control the content of these core job definitions. So that's something that you might wanna look into. It's oftentimes too, we get questions about locking that, right? So how do I make sure that this content can't be altered or tampered or that we ran the security jobs that we needed to run? And there's a way to do that with file locking and code owner files. So if you've seen code owners as part of GitLab, it's where we can define the owners of various files within the repository. So locking a file and defining the code owners will only allow those code owners access to the files. And that way, we create some controls around what's able to happen or not able to happen inside of the GitLab CI YAML files. However, all that is leading us to a new feature that's coming soon and that is an actual ability to specifically lock down who has access to the GitLab CI YAML file itself, thereby enforcing the include functionality that you saw today. Great, thanks, Joel. That looks like that's our last question. I just wanted to let everyone know that you'll be receiving an email, a follow-up email that will include a link to today's videos. I'm also just gonna quickly post in chat the link to the previous one that was mentioned. And we'd also like to encourage you if you've been joining the series to go ahead and sign up at the same landing page you did for this session. For our next session, we have another one coming up. And feel free to share all of this with any colleagues or anyone in your organization or people you know in other organizations who'd be helped by it. So we really thank you for joining us. We thank Joel for a great session and we hope to see you again on the next webinar. Thank you.