 As you might have noticed, I am a solution architect. And I'm going to talk to you about a journey. And it's a journey about solution, solution that I designed the last few years to improve our DevOps chain. But as you know, all solutions start from a problem. And this is where I introduce my colleague Simone. Hi, everyone. I am Simone, and I am a problem architect. So I like to raise problems. Well, making this journey with Matteo in the last few years has been quite an adventure, I would say. And yeah, we are here today because we would like to share this adventure with you. But let's start from the very beginning. So let's roll back in time. Let's go to just a few years ago. We were not DevOps butterflies. We were a really small team. Our workflow was really bare bones. So we had like an issue board. We mostly organized our work there. We split it in tasks. And then, unrelated, we had a subversion repository. We will check out the code from the repository. All the development and the testing was done locally on our own machines. We will just take care of regularly deploying the artifacts that we were producing to our QA environment, which consisted of multiple virtual machines. And that was it. And on the other side, completely separate from us, there was this operations team that would pull the packages that we published and manually take care of deploying and making releases to the production environment of our customer. And this workflow was really problematic for us because there were multiple teams involved and where there are multiple teams, there are communication issues, everything was manual, mistakes were bound to happen. And in fact, they happened. They happened a lot. There were a lot of integration mistakes, last minute bug fixes, and all of these impacted our workflow because we had subversion, okay, and we had these release cycles. What we did was, okay, we know that we will have trouble during the release. We have to organize ourselves. We split the team in two parts. One part will work on a next development branch on new features, because for the customer, new features can never really stop. The other part of the team will stay on the main development branch, which in subversion is called the trunk. And just keep committing these bug fixes. And at some point, hopefully, we will be able to merge the two. The problem is that this feedback loop lasted like several months at some point. And when we finally had to merge, we had a lot of problems. Merges in subversion are never easy by themselves, but in those months, the code base changed so much that sometimes it was really hard to make these merges. It will take us multiple days, and we introduced even more regression, making the merges. Sometimes we even lost features by mistake. So we knew that this could not really continue, and we would not be able to succeed and scale with this approach. And that's where I knocked on Martel's door. I didn't really knock because, you know, it was in the desk next to me, so I didn't have to knock. But I told him, okay, we need to find something better. We need to evolve. What can we do about this? Well, we know that we have to evolve, but how? What were our necessities? We have a source control tool that does not work very well, and we have a bad workflow. For this reason, we choose to migrate to Git and adopt the branching strategies to reduce the mistakes and improve the release and feedback loop. I have a great different solution, and in the end, for several reasons I choose GitLab because, first of all, it's free, and we can install on our server, continuous integration and continuous delivery are built in to GitLab, and it provides collaborative tools for collaborative coding. And now that we have GitLab in our tool chain, we start to evolve the status quo. Phase one, we want to choose our branching strategies, and we choose GitFlow because it's the most famous, and it has a common line that could help the developer transition to the new tool. And now that we had the new tools to organize our daily job, and we could discuss the issue on the project board, it is a reference to the issue in commit message, we could not notify via Slack when someone pushes comments or something important happened on Git repository. We also automated the published flow and with GitHub Runner and CI Script, the tar version with the project code base, and we stole on the QI server a Chrome script to pull the new version of Docker image from the Docker registry. And with this I think I solve all your problems, Simone, right? Well, I have to admit it's a nice improvement on our initial situation, very well done, but we almost immediately face some limitations. In fact, okay, you have a full Chrome job, and it regularly updates our environment, but it's still very far to a real, true continuous delivery. So it's not always aligned with our developments. And also having these virtual machines costs us a lot of time to maintain them, to duplicate environments, to create new ones. All the times we spend there, it's time that we cannot spend improving our process, our methodology. So yeah, we have feature branches, we are more organized than before, but we are not making any code reviews, we are not making any... I got it, I got it. We have to evolve again. And I think we can solve most of the technical limitation by evolving our infrastructure. We can install Kubernetes cluster managed by Rancher and move all our tools application on this cluster. And with this infrastructure solution, we can implement a full continuous delivery. And now that we have this I Kubernetes cluster, we also can install Sonocube for software quality and Minos to speed up as a cache layer to speed up our pipeline. And the immediate advantage of this solution are that we can create a new environment very quickly and with low effort. And during the second phase, we also version our environment setup in a Git repository. And now that we have resolved all our previous problem, we can focus on merge request and code review. I think that's all. Well, okay, let's say that we improved also the methodology. We automate more, we are able to work better, we are able to iterate faster, we get more customers and that is really nice. But we start facing new problems now that we have new customers. We start facing some challenges with scaling and growing as a company because now there are some customers that start using our software and maybe they don't always use the same version of the software. So some customer for some reason stays behind. And this is where the branching strategy which was felt falls apart because you know, Gitflow is really nice, it's really structured, but it does not cover all use cases. Specifically in this case, it does not envision a way to keep supporting multiple versions of the same software. So we have our software that is at the 2.0 version, a customer is stayed on the 1.0 version, asks us for a bug fix because we are supporting that version. We can open a not fix branch and make the bug fix, but then it's unclear how to merge the bug fix to the main development branch. Maybe because the code does not exist anymore in the latest version. And even if we can merge it, it's unclear how to increase the version number. So there is something wrong there. We need to keep the not fix branch open forever, but that's definitely not ideal. And also wherever we are making a release, there are a lot of manual steps that we have to make. We have to open branches, make manual changes to change all files, version numbers, remember to merge the branches, push the text which we regularly forget and we mess up the releases. So we need to find a way to make the release process less involved and scale for different versions. How do we do that? Yeah, you are telling me that the branching strategies that we chose does not fit very well with our needs. But it's not a real problem. We can change the branching strategy and we adopt the GitHub flow that has two variants. The first one is called environment branches and in this variant we have the master branch and one branch for each environment. And for example, when we merge from master to staging branch, the CI pipelines update the QI environment. The advantages of this approach are that we release our software with a single command and we do not take our release commits because the branches reflects the history and the status of the environment. Okay, that's a lazy movement but what about the releases? Yeah, for a project that need to support multiple releases we can adopt the release branches with GitHub flow and in this strategy we have one branch for each supported version. When we fix a bug, we open a branch from master, fix the bug, merge the fix on master and then cherry pick the commit from master to the release branch. And in the both variants, the feature branches still exist. And with this branching strategy I'll solve all our problems. Well, maybe you solved my problems but you didn't solve the problems of the other 20 other guys. Who? Well, Matteo, while we were busy solving all these problems and finding these solutions, the company has grown. We are not three guys in a room anymore but there are a lot of people that works for our company and they are split across multiple teams and they're facing problems too because we are consultants, we work with a lot of text-texts and we are split across a lot of projects, okay? We care about having a CI pipeline for all of them but how do we maintain these pipelines? So yes, we succeeded in establishing a common set of stages and jobs that all our pipelines have to define. So we know that all our software has to, all our software pipelines have to install the dependencies, link the code, run the unit tests, the end-to-end tests, some quality checks, then build, publish, deploy, usual stuff. In practice then the pipelines for different text-texts are implemented in different ways. And this is an issue because it's a lot of burden on the developers and also even when two projects even use the same text-texts there are inconsistencies across the two pipelines sometimes because someone maybe implements a job in a different way from another. And when I find a bug on a CI pipeline I have to go back on all the others and fix the bug. So all the time that we were spending earlier on manually deploying and publishing our packages I now spend maintaining these pipelines. So I think we need to solve that and go back to actually developing the features. Yeah, you're right but this is a common problem for the developer. We have duplicate code in different repository and to solve this problem we can move all this CI script in a single centralized repository. And with all script in this repo we can define a reusable and templates organized by technology and stages. And in this way we can import the right template in our CI script and customize them if needed. The project CI script are now simple and easy to maintain since most of the time we upgrade the base CI script and all projects exploit them for free. For example, all projects that are deploying Kubernetes could import and extend the template job that you can see on the left. And in the extended job on the right is enough to set a few variables to get a working deployed job without any knowledge of Kubernetes. And in some cases is enough to get, is in some cases important template is enough to get a working job. We try to keep this script as readable as possible. And to do that we define some high level function. So I, as you can see, I can go through the comments and easily understand what is happening. So in this case I, this job template, upgrade the project version, install the dependency, persistent environment URL, and finally deploy the project on Kubernetes cluster. But to do that we define mesh function that hide the complexity of the comment. This is not an idea, but we, I came up with this idea by reading the GitLab how to develop source code. Now that we have these CIScript, these CIScript are very useful for us. And another example is when in the last year we developed a reactantive application and reactantive application had three technology stack, the node stack and the two native stacks. And we could reuse all the node CIScript and implemented only the two native CIScripts for the native parts. But another problem is that all our GitLab runners are deployed on Kubernetes and it is impossible build an iOS application without a Mac operating system. To solve this issue again is enough to install a GitLab runner on a Mac server. And with that I think that our evolution is completed. Well, I admit that, okay, we solved all our problems very good, we improved a lot in the last few years, but I think that if there is something that we really learned is that we are never really done with this journey, okay? So the mindset that we acquired is that there is always some part of our workflow and now we make things that can be improved or made easier or just automated so that we can focus our effort on more important stuff. And in fact, this is what we are doing still today. So just for once, I'm the problem architect, but just for once I present a solution. So this is something that we did in the last few weeks. So automated releases, GitLab gives you the possibility to publish releases for your software so you get this nice changelog with the links to the artifacts and it's very cool. We wanted to automate making the releases when we publish to the environment branches for example. So what we did was just creating a changelog bot which pulls the relevant part of the changelog depending on the version we are releasing and invokes the GitLab API to create the new release. And this is all very cool because it's another part that we do not have to think about. So yeah, I think this is the really important thing. So just to wrap up some takeaways. What we learned. We learned that how the company grows impacts not only how we are organized and our methodology, but also which technological solutions work well or don't work well for us. The important thing is that whenever we find that something is not working anymore, we are not afraid to try alternatives, find something better as for the branching strategy. What was really nice for us looking back is that the GitLab supported us from the very first phase of our evolution and worked very well for every problem we threw at it so there was nothing we were not able to do with GitLab and we are thankful for this. And also we drew a lot of inspiration both from the GitLab blog and the GitLab source code itself so yeah, I'd say that we made the right choice. So yeah, just to finish a few words about ourselves. We already presented ourselves, I'm Simone, here's Matteo, maybe we may be the solution and problem architect at WLD, which is the company we work for. It's located in Lugano in the south of Switzerland. It's a beautiful city by the way. You know, there are not only architects in a company, there are also a lot of developers that put a lot of effort in actually adopting all the solutions that we think about. So we just wanted to have this slide to share some moments that we have with them which we think are really nice and funny and thank also them. It's also thanks to them if we are here today. If you want to know more about us just find us later in the crowd. There is also the CEO somewhere there so you can ask whatever you want. So yeah, now we're really done. So long and thank you so much for the fish. So I want to thank both Simone and Matteo, like this is the journey that we're all on. The way you guys represented this is perfect. I'd encourage you if you guys are not doing this in your company, think of it as a problem solution, right? Challenge one another, go through that evolution. This is a culture question. So, you know, they've done an amazing job implementing and using a tool that happens to be GitLab. That process is the one that we want you to walk away with. That culture is a part that we want to be a part of if at all possible. We're honored that we were able to help you guys through this. Well, it's just honored you guys here to share that. So with that, we have a couple of minutes. We want to open up if anyone has any questions. I've got the mic, but if you can learn from anyone, I think they've been on a journey that would be well worth it if you have questions. If you have, ah, perfect. Hi. So obviously me being here, I totally am with you. Like all of this sort of stuff, changing the mindset and making sure that you're constantly asking questions. Totally with you. It's very hard to convince people who aren't in that mindset to get in that mindset though. What sort of, how do you, aside from bluntly telling them to get their ass in gear, what sort of the way that you've found to be able to engage with people and actually get them to think about, yeah, we should probably be improving. You made the question. No, well, I think that, I will answer. I think that on one side, we are very lucky, okay, because all our company employees are very young, okay, and enthusiast about software. So whenever a new solution is proposed that promises to improve their workflow, improve, you know, the results than they can obtain from their effort, they are always quite on board with it. So I think that's the key thing. And then there is the other side, okay, so which is the customer, because of course we are consultants, so sometimes we always have to, we also have to convince customers, that's harder because maybe they are less prone to, you know, change their tools or change the way they organize their workflow and you know, sometimes it has taken also several years to convince them to make the switch, but if you show that there are objective improvements, then usually people get convinced then and try new things, that's I think the key. It's just showing them not asking you to trust you, not asking, yes, not asking them to trust you because you know, that's a leap of faith, but if you just, if you can show the numbers, then it becomes obvious what to do. So in a way, it's like we're gonna go ahead without you anyway, as soon as you realize it's the best way forward, you're gonna be on board. Yes, we are lucky because we have a lot of customers, so some jump on board earlier, some later, but you can show across one another, okay, look what we did for these other customers, it worked very well, this is the tools we are using, this is what we are achieving, this is the speed at which we are releasing, it's working, so maybe try this as well and yeah. So, thanks. You're welcome. Firstly, thank you, I thought that was very well presented, especially as English isn't your native language. You're welcome. What has been the biggest blocker through this whole process? What's been the biggest challenge? Well, probably putting up with all the requirements required work to switch the tooling, so the work we did on the infrastructure part, the Kubernetes part was really demanding, you know, we did it voluntarily of course, but sometimes we even spent, you know, holidays tinkering around and ducking and trying stuff on with Kubernetes and deployments and so that was the really demanding part. Once it's set up then it's more graceful for developers to adopt the tools and just use them as tools, but setting that all up is very demanding if you do not go to a cloud provider, we don't yet, we have our own machines on premise, so you have to manage the whole stack and for a small company it's quite an investment, so that I would say was the most important. And that's why we work with a ton of different people because that is the biggest, the Kubernetes piece, it's amazing, but it is a non-trivial adoption piece to a cloud native journey. There's other ways to do it, the fact that these guys did that, so their journey, most people skip over that and like I'll go to a managed service. I will say bravo in learning that because you're gonna know how that operates and knowing the guts of how that works, really, really critical. So if you're looking to do it, be willing, if you're a leader, be willing to set aside that time for your team to learn it, it's certainly well worth it and the trade-offs, totally wonderful on this journey but that may be the biggest piece and that's pretty consistent when we talk to customers. It's not the GitLab and the CI CD piece, it's definitely how do you run your operations the Kubernetes piece. Other questions? Thank you, so maybe just a quick question on this part on Kubernetes, so you almost started from scratch in knowledge on Kubernetes, maybe, years ago. I imagine my, I have a team, I'm sharing a team, some are, they have knowledge on Kubernetes, some don't, how do you, how do you think that these people with no knowledge of Kubernetes should start or what's the hint for really getting to it quite quickly or efficiently? Well, disclaimer, not everybody, not all the developers that work with us, no Kubernetes, okay. For us, the important thing is that a lot of them don't have to, okay, so it's not necessary that they understand everything that is going on on Kubernetes, so we do a lot of knowledge sharing to make sure that they understand the principles behind the cloud native, what is Kubernetes, how it works on an high level of abstraction, but then there are some responsible people that actually get events dirty and work on the deployment scripts and they find the services, the invasions and so on, so it's like a pyramid of knowledge, but I think it's fine, then people can decide where they want to stay, if you want to stay at the top and just know everything and put the ends in the infrastructure code or they just want to be users of the infrastructure and just reuse the templates, we define the setup a few variables and that's it, that's why it's working for us, it's scaling because we don't have to teach everybody all the concepts behind Kubernetes which are quite a lot and takes a lot of time to get started and productive.