 I'm Mohammad Mehdi, I'm a Principal Dev Ops at Verizon Connect and this is our Papa Bear, Keith Braddock, he's a project manager with our Verizon Connect team. How many of you know what Verizon Connect does? Except the Verizon Connect team. Really? But wait, there was someone who forget how many of you know what Verizon Connect does. All right, so Verizon Connect is a part of Verizon, a few years ago Verizon bought a bunch of companies and clubbed them together. We basically are in the telematic space, fleet, including fleet and ELD and compliance, ELD is electronic long devices, which track how long a driver, a commercial vehicle driver is driving how much the vehicle is idling, all kinds of it's needed by the government. And asset tracking, fleet tracking, all kinds of yummy stuff with respect to the fleets. Our biggest thing is that we developed a way for parents to track their children when driving. That's it, that's the biggest thing. That's one of our products, yes. Next slide. All right, why are we here? So we're going to talk about our journey with GitLab, how we started, what kind of challenges we faced and how GitLab helped us resolve those challenges and what our goals were. And at the end of the presentation we'll have some Q&A if you guys have any questions. So initially what were the challenges we faced? I mean, initially when we started this project almost two years ago I would say we had legacy systems, Java based monolithic applications and those things. We had Git, Bitbucket, Jenkins, all kinds of stuff. We had Jira for issue tracking and separate our ticketing system and everything. We wanted to convert everything to microservices. And we had a lot of issues in terms of how we were building our data centers, it took more than 30 days. We had a lot of disjointed processes. We had manual deploys and it was not fun. I mean, we were spending too much of time on doing stuff manually and fixing errors manually. And that was one of our biggest challenges. So we decided to move to microservices and we decided to invent or write everything from scratch. Next slide, please. So what were our goals when we started looking, the infrastructure goals when we started for this? Architecture. So we wanted to, at the end of the day, rebuild our data centers in less than 12 hours instead of 30 days. Increase the development velocity, cost consciousness. We did not want to get locked down into a particular vendor or hardware vendor or a software vendor. It can be anything. So we went with open source solutions. We went with open hardware, open compute. And also we wanted to have, go with a solution which also provided, was open source by the same time we had the capability of providing vendor support if needed. Automation. We wanted to be 100% CI CD. Nothing should be done manually. Nothing should be deployed manually. Especially server and network provisioning. That's where we started. Then eventually we tried to deploy the applications and those things. Integrated validations, infrastructure as code and everything should be repeatable and predictable so that we are not again spending time doing manual stuff. Extensibility platform with swappable components. For example, if we want to, let's, we did deploy everything in a private cloud which we built using Docker and we wanted, now we are moving towards AWS. The builder, our goal was to build the solution in such a way that we don't have to rewrite the whole thing when we do that. And that's what's happened right now. We didn't have to do much work except for the AWS part where we built the newer infrastructure. Apart from that, everything else is just repeatable. Just go ahead and deploy it. Word lock in everywhere, hardware, software and cloud. Future proofing. Flexible and extensible, easily scalable, cloud adaptable and platform agnostic. As I said, we can run it on a data center. We can run it on a private cloud. We can also run it on a public cloud like AWS, Google Cloud, Azure, you name it. Next slide, please. So this is our very high level diagram of how we implemented our sites. We have two different sites, site A and site B. And these are mirror sites. We have production over there. With mirrors, the production over here, we have a non-part over there and non-part over here. And these are the different environments we had, like dev, test, set, preprod. Some of the environments were needed because of the kind of customers we had, especially for them. But we had more than 360 physical servers and those things. But it was big. I mean, right now, we are also adding AWS to this picture right now. So when we are globally expanding, we have stuff in Europe. We have stuff in China. We got stuff in New Zealand, Australia, everywhere. This is a very high level logical architecture. We wanted to make sure we had service by service failover without affecting other services. Service by service isolation. We used VXLan. We want to use McVillan. We are not there yet. But that's the goal we wanted to go with. We are using any cash flows, load balancing. So this is our site A. And this is site B. If you look at it, there are just mirror images. There's nothing different. We have different services running over here, different clusters. And this is our infrastructure stack. We have a hardware, which is open computer, Dell-based, open hardware, Ubuntu, Docker, SEF, Anycast, console, Kafka. It's a spring-based Java applications and GitLab. And everything and every code we are writing goes in ransomware. We try to avoid using scripts because with scripts, what happens is over a period of time, when you think about operations, somebody has their own favorite language. The next thing you know, that guy left, and nobody knows that language, now you have to maintain that script. So we decide we are not going to use scripts. That will be the last resort if we can't do anything with Ansible. I mean, Ansible is so flexible if you really love it. And it's been really great. We didn't have to actually write any scripts until now. So what did we do? So we decided to go with some of the automation goals. Simplification, fully automated, easily repeatable and predictable, zero touch and zero downtime deployments. Automatic tracking, standardization Ansible first, a bash Python second, as I told you, I mean, we never had to use it. But just in case, those are the two languages we want to go with, then same CI pipeline every site. Let me talk about the same CI pipeline every site. Basically, we created only one single pipeline, and you can actually take that pipeline and implement it in your sites which are coming in. You don't have to rewrite anything. That's how we modularized and customized the whole pipeline over here. And the last and not least, end-to-end visibility, server network configuration and provisioning, public and private cloud adaptable, automated maintenance and healing. Why did we select GitLab? As I told you, I mean, when we started, we were looking at certain things like open source solutions and vendor support if needed. We already had Jenkins and we used Jenkins in our legacy environments and those things. There were lots of issues we had, and but when we decided to look for it, we went with GitLab. One of the reasons is because it was free to use with an option for premium support. It has a very simple UI and easy to learn. It's not difficult as we have seen in the previous demo. It has some really great features and it's very easy to pick up if you have some experience doing CI CD. Some of the other features which we really loved, actually, we didn't find with any other CI CD tool, is project management features which this guy provides. It's basically, GitLab replaced a bunch of disparate systems for us, like Jira, Bitbucket, Jenkins and anything, there's a lot of stuff which we did not have to have or use them again. So GitLab provided us with a one-stop solution for us. Core review, CI CD, issue tracking, source code management, audit management, deployment flexibility and platform agnostic and chat ops. Chat ops was also one of the important features we were looking for in this case. So how did we implement our automation plans? Everything is configured in Ansible. That's a single source of truth. GitLab issues to create and track new features, manage projects using the project management features, implement repeatable and reusable modules, separation of changes based off environments and one pipeline for all the environments. So this is our very high-level CI pipeline and deployment management, GitFlow stuff. If you look at it, it's very complex, but at a high level, basically what it does is if you follow the documentation of how GitLab Flow works, we try to implement that over here. We try to go with separate branches for separate environments and deploy that based off testing and everything as if one, let's say we are introducing a new feature or fixing an issue or something like that. It goes into the first, you create an issue, you create a branch based off that issue and go ahead and commit your changes and everything, push it out, it gets tested and once it gets tested and everything, everything looks good, you merge it into a, let's say, itest branch. Then everything gets tested, gets merged into the third branch which is your pre-prod branch and eventually it goes into the master branch which is our production branch that gets deployed into N2M, excellent. So this is the high-level flow which I just talked about. Create an issue, create a branch, commit changes and create merge request, approve it. We actually have a sandbox server where we actually don't test the whole N2M pipeline but we just the specific components which we want to test which where we made the changes for. So we test it over there, triggers a pipeline, deploys component, it's if it fails, we go back and puts it back in the queue of the developer engineer and says, these are the issues, go ahead and fix it please. If it passes, it goes to the next one, dev branch kicks off the same thing, same thing follows over here into the high-test branch and eventually it goes into the production branch. All right, so this is a sample CI file which you are using. As you have seen in the previous demo, we saw a sample CI file, this is one we are using. So we first create our variables, then we define our host based on different environments. The good thing about this one is that if you look at as we go down I'll show you one more thing is this is based on the environment. This is a global one which can be overwritten at the component level, that if you want to deploy a specific component at specific environment or a site or something like that. Then these are before script, then these are different stages, these are just a sample stages and this is one of the job template which we are using for Ansible. And if you look at it, right, if you keep going down, so in this one, this is the job, right? So this is where you can override the sites if you want to specifically deploy any particular components, one thing like that on a specific site, that's it. Then you can actually override it over here. So this actually, I mean, if you look at the whole CI file, it's actually modularized in such a way that you can take one component and deploy it anywhere else without changing anything. So that's how we decided to create this CI file. So we're gonna talk about what I actually was a hard convert to GitLab. Part of it is I'm advanced in age and so I was a clear-case, clear-quest person in my earlier life, which I thought that was nice to hear. Then I became a Jiro person because I believe that management needed to see a level of extraction but not all the details. And so it took some convincing from the young pups but once I got in to GitLab, I actually had to change my thinking and that was a welcome exercise to actually have to go through that process and re-examine how we look at how we track issues, how we track progress and how we reported that was extremely welcome exercise for me to go through. So one of the things that we did and kind of at the time we started using this, there wasn't the, was it platinum? Is that the highest version? Ultimate. I don't know why I wanna say platinum. I don't know, is it platinum, gold, I don't know. So the ultimate, it was not yet in production. So a lot of the project management features enterprise management features are actually in that version and they actually incorporate a lot of the features that many of us that are in this world of managing programs and things of that nature we're asking for. So I do highly recommend that anyone look at that product but I can show you how we were able to use the premium edition, right? Kate and we use the premium, okay. The premium edition to actually do a lot of what we were trying to do. So here's an example of a Wiki and one of the things I love about this tool is developers love it and that's actually good. It's easier for them especially those that are command line developers that wanna do everything from the command line. The ease of commits and comments and one of the things that wasn't talked about earlier was the ability to have open up a discussion at the review point of before it commits made. Those are really nice ideas and we actually did a lot of doing that. But one of the things I love is that I can report out in an enterprise level to senior management where things are using their Wiki page and that was an actual welcome feature because I can include the actual item by tagging it inside of mix with text so that then that particular, you can see this one's closed but each item it would actually give the stage and where it's at within the reference next to the verbiage. So what we did was we used an interesting combination of milestones to track releases. So just to give you an idea when we built, building our infrastructure using infrastructure code, all of the code is actually in GitLab. All the pipelines are run out of GitLab. So we had a, we were releasing basically updated scripts every, I think it was every month, I think we were looking at. And so we ran it from a, as you can imagine someone has to communicate to executives who don't wanna know about agile, who don't wanna hear about the edicts of agile. They just wanna know when they're gonna get their cookies. Given their cookies, their dessert, they just wanna know and bank on it. So we use the milestones to track that, to give us an overlay or a view of that. And then that allowed us to actually look at the boards and leverage the boards to actually give us the information that we need. One of the great things I did like about the labeling structure within GitLab, it allows you to do multiple labels to give you the opportunity to do, to look at a project 360 degrees, to look at where we are at all aspects of it. And you'll see here when we look at the board. So right now the board is showing the development view, but let's say I wanna know, what are all the items that are currently in code review? I would go to that particular board and would show me where they are and what status they're in. If I wanted to look at items that say, hey, I'm curious, what's the June release look like right now? I can go to that because we use effective labeling. I was able, we were able to tell where things are for the June release, what's in the backlog, where things are, have there anything's been closed for that release? One of the things I love, there's the go live board, there's the planning board. So for me who has to do all the planning and has to say what can we do, what is possible, this allowed me to look at what we're planning to do overall from a program perspective for the life of the actual program. Here you can see how we able to use labels and basically what we did was we said, okay, there are times when we're gonna need to know what was the effective system, right? Was it SEF, was it DNS, was it Docker, Docker Swarm? What was the actual, was it a hardware issue? So you can tell, we not only use it to track how our projects are code going throughout the project but also we use it for support. So again, when we say this took over, we use it for everything, we literally use it from everything from soup to nuts. We manage our deliverables, we also manage our support issues, we were able to actually report on and what I used it for from a reporting standpoint or kind of an overall enterprise monitoring is I was able to effectively use the labels using the labeling method that we use to say, hey, how, what are the things that are causing us the greatest issues? So when we look at budgeting planning for the following year, are there resources or are things that we need to re-examine based on what areas gave us the largest amount of tickets to solve for? And I was able to use this system to do that. The other good thing that a lot of people don't talk about is that at the time, Verizon now is moving completely over to GitLab, but at the time we had Jira and we had for enterprise reporting outside of our group, we were able to link this into Jira and actually report out, never having to leave the GitLab environment. So that was a great desire, need for us to be able to fit the corporate needed while still staying within our existing system, allowing our development teams to stay in the system they love. And that was a good feature. And again, you can see we went from area to, it was interesting how we kind of mapped this out. And a lot of this, you talked about the status, we talked about even we used, sometimes we use vendors to help us and we wanted to assign projects to know where they were. We actually use that as a label that allowed us to say, okay, how many items that are on our core deliverable are basically out with our vendors and where are they in their face? So again, great flexibility. It's the flexibility that frightened me in the beginning, but once I welcome the change in my perspective, it's the flexibility that allowed me to do everything I needed to do and able to structure it in a way that I can report on it. And I think that's the biggest key. Being able to effectively use the boards to show me whatever view I need to see in order to maintain an eye on where things were throughout the particular project. Good question, please. How does your tagging structure is really involved? Got multiple levels of tags and then there's colors that go with them, right? You had this list view where all the status slash whatever, they're all green, et cetera, right? How does that happen, is that? That's a great question. So in the labeling function, I can go here, I can edit, and I can choose a color. I just choose a color, mm-hmm. Is there anything in the product that allows you to say, get everything that's status related, or is that this multi-level tagging mechanism pretty much manual? Can you rephrase that question? What I'm saying is that if you wanna report upwards on status, and I'm not being critical, I'm not really impressed, I'm just saying, if you're trying to report upwards on status in general. So you're talking about overall status or if something was outside of the board? So there's a board, the development view that comes with the product that has out-of-the-box status. I use that for normal status. What I did was also create, and what you can do is create multiple boards to give you different views of that normal status. So I can segregate the data based on if I wanted to see where it was, what particular vendor that we were using, or if I wanted to again see highly affected areas, or just wanted to see where we were as it relates to a release. But if you wanna see status on everything that's in there, the out-of-the-box development module just basically gives you status of things going through their particular workflow. But this allowed me to use theirs, but also create more complicated workflows based on what my needs were. Yeah, I have to say, I think this is a rare, just in my experience with the product, I think this is an advanced use case. Yeah, I was a nut. I was a nut. It was me refusing to give up everything that I needed for my safety net. But actually it turned out that if you think it through, and I worked a lot with GitLab, and they worked with us to really kinda walk, talk through the issues that we were experiencing, what we were trying to do, and they helped with my education as far as changing the way I looked at things. We really wanted to absorb the way GitLab, the way they do things, the way they do conversation development. We were trying to implement a lot of that. So it was an educational transformation for myself, but it allowed me to figure out how to accomplish what I knew executives wanted to see while not disturbing the developer's workflow. And out of all the products I've ever used, and I think I've used almost all of them that make sense to me, even Microsoft products in some point in my life, unfortunately, this is the most flexible I've ever seen any product. Thank you, Kate. And again, I mean, that was one of our main very critical factors, I would say, going for GitLab. It really helped us out, replacing multiple systems and not paying license fees and all kinds of stuff and maintenance issues and now it removed a lot of headache for us. Now coming back, impact of automation, it reduced our complete data center deploy process to under almost eight hours. Yes, that's true. I mean, we almost deploy our whole data center, rebuild our data centers every other month or something like that, just to test our pipelines and those things and we do that within eight hours. And it's really amazing. Streamline deploy and build processes, enabled issue tracking and resolution as Keith talked about, enabled auditing and tracking of changes. When, who did the change? When did it happen? What was the end result of the change? Did it get deployed properly? Was it, what happened when it got deployed and all kinds of stuff? You can easily track using GitLab over here. Most, now we started spending more time doing proactive work than reactive work in the sense like now we can think about how can we improve the availability of our application? How can we improve? How can we redesign some of this stuff which we want to redesign? So basically, GitLab gave us a lot of time back in terms of what we're doing and where we are spending time on, I would say. Obviously eliminated human errors, specific role-based access controls. Role-based access controls, that's the other really good feature about GitLab in the sense that you can give very specific roles to specific people to deploy on specific environments and those things. What can they do? What they can do? GitLab has some really robust feature built around it. Last but not least, increased frequency of releases and deploys. Yeah, I want to talk about that a little bit about that last one because one of the things that I loved about it from a program perspective, so you don't slow things down, we could have multiple people reviewing and approving during co-review, but only a select number of people that actually have the authority to merge the master. So you won't slow yourselves down by giving a lot of people to do code review, but if you want to protect that master branch, you can actually within the tool restrict the ability of those people to do that. And that was sweet, especially when some people who are up late at night don't know what they're doing and they don't do CI skip and they actually commit a text file that actually kicks off a pipeline in the middle of somebody trying to rebuild an environment. Yep. I won't tell you who that was. We've been there. We have done that. Everybody's done that. Yep. Points of success. So we are almost at the end. Speed, we reduced as I talked about the speed at which we deployed our applications, our operating systems, our devices, hardware, network gear, you name it, everything. Agility. Agility, we can use the same pipeline to deploy multiple components at different platforms or different OS or you name it. I mean, this GitLab gives you the flexibility to do whatever you want to do without redoing anything. Go back, go back. Yes, sir. Time savings. Obviously, end of the day, it saves you a lot of time and because you're not doing work which you normally would be doing like troubleshooting deploys, wide fail, what happened and all kinds of stuff. Release cycles and lastly, auditing and reporting. Lessons learned. Standardization. What did we do? Programming language. We made sure we used single programming language. In this case, I would say Ansible which is a single source of truth. We asked everybody not to use anything else. Adoption of open source rules and software and hardware and everything so that we can swap components, we can deploy or build our pipelines in such a way that we can deploy them to any open source solution which we want. Today, we like Ubuntu, we are going to use Ubuntu as our OS. Tomorrow, we might go with Red Hat, who knows? I mean, we created our pipelines in such a way that we don't have to do much work to redeploy Red Hat or CentOS or whatever we want to do. Operationalize, ensure our ad-importancy. We created roles for tools, alerts, and dashboards and everything. We modularized the whole thing and created repositories, separate repositories for separate environments and separate, we also wanted to do with separate based off the kind of application we are doing, but I don't think we decided to stop it because we were getting too much fragmented in those things, so we decided to stop it. Next documentation, we documented our change processes, code is your document, we don't have a separate website or a page or anything like that which talks about how things work. You just need to go into an Ansible repository and take a look at it. You will understand everything, how everything works. That's how we wrote the whole code and everything. Last, and again, project management. Modularize, separate repositories by function, consolidated roles, logically segregated CI pipelines.