 Jamie is one of my favorite people in the universe. Because we've managed to get him to talk here about GitOps, but I also get him to co-chair the OKD Working Group with me, which is the open source side of OpenShift. And he is just a wonderful speaker. And today he's gonna tell us a little bit about GitOps at the University of Michigan. So if we can let you introduce yourself and I'm gonna mute myself and let you go. We're a little behind time, but take what you need and we will do Q&A in the AMA. Awesome. Let's see if we can get... Nope, there is a little bit of echo in there. I'll try to get done like right around 3D5-ish. So thank you very much, folks. And a little bit about myself and about this presentation. We'll start with that. Actually, let me share my screen and actually show you the presentation. And okay, the ubiquitous question. Can everyone see my presentation? Looks good. All right. Excellent. So Cloud Native Development and Deployment with Kubernetes and GitOps, what we're calling ICPSR Ops. And I am Jamie McGuera. And so the agenda for this quick talk is an introduction and some foundations and where we are at ICPSR. And then some suggestions for DevOps engineers and developers. And then as Diane mentioned, question and answers will be moved to the section at the end. So an introduction about ICPSR, the Inter-University Consortium for Political and Social Research provides leadership and training in data access, curation and methods of analysis for social science research community, the Social Science Research Community. We provide research data to organizations across the globe via a variety of web services and portals. Recently, we began to transition our application development and deployment to the Cloud Native Paradigm in particular, Kubernetes and GitOps. We've learned a lot during this process and would like to share that knowledge and experience with others to shed some light on the challenges and advantages of moving to Kubernetes in the Cloud. And about me, I've worked in education research technology for 25 years with a focus on higher education. I've worked for the University of Michigan for 18 years and started working with Kubernetes in 2017. As Diane mentioned, I'm co-chair of the OKD Working Group and I'm a member of the GitOps and Fedora Core OS Working Groups contributing as time permits. So a little bit about foundations of this and folks have seen this. Just a reminder, this is probably the third or fourth time you'll see this today. So GitOps is, this is the definition from open GitOps, declarative, versioned and immutable. It's pulled automatically and it's continuously reconciled. And this is important because all of those components together I think are necessary to really get the full advantages of GitOps. So all four of those. To start at the most basic level, I always like to show folks thermostats, right? So thermostat, you declare what you want the temperature to be in the room or in the house as the case may be and then there's a process of monitoring through a sensor, the temperature, the current temperature and it tries to reconcile what you've declared and what the current state is. And of course this harkens back to cybernetics which is the science of communication control theory that's concerned especially with the comparative study of automatic control systems such as the nervous system and the brain and mechanical electrical communication systems. That's from Merriman Webster, Merriman Webster. And it was defined in the early 1900s by Norbert Weiner in the book of the same name Cybernetics and it can be applied to systems great and small and the control mechanisms and feedback loops defined in Cybernetics aligned with GitOps or vice versa. And so that's the appeal to me of GitOps. And so cloud native computing and ISR, how we got where we are and the need for GitOps. So the problems. Although development was done within Git, the changes were not accurately synchronized with the cluster continuously. There was drift between what people were submitting and what actually was in the cluster. Microservices were not deployed in unison for applications. The promotion of applications through the stages, it was not well organized. And so there was a lot of problems with that. Database schema, solar schema, patchy solar, AWS resources and other external resources were not synchronously deployed with the code. So you'd get people promoting code changes, developers promoting code changes. You'd want to push those up through the stages, but then you'd have to do manual changes or issue imperative commands with the respective tools, AWS CLI, et cetera, et cetera. Or maybe you'd use like Terraform, the binary, but all of those things were still being done in a way that wasn't completely synchronized with the code deployment. And the clusters themselves were not easily provisioned and restored, right? So disaster recovery, very important, the ability, provisioning and disaster recovery, the ability to provision clusters in the state that you want very quickly and efficiently and to restore them to the state, keep them in that state over time. So the solutions in a general sense, we separated out our application code from the deployment code. We create code to define external dependency configuration, and I'll be talking more about that. Now we stored the microservice code and cluster configuration in the U of M's GitLab repository, and we created Helm charts for the services and for the infrastructure configuration. And we automated the process of loading computer code and cluster configuration continuously with Argo CD. So a little bit about what we have for infrastructure. We have Kubernetes clusters and development is OKD, which is the upstream of the commercial OpenShift product. We use that for development and the quality assurance stages. And for production, we use OCP for our user acceptance testing and production stages. And in terms of deployment resources, what we've done is we've separated, we've created separate repositories for each microservice deployment. Each repository for the deployment has a Helm chart for the respective image and environmental variables and whatnot, and the Helm charts are packaged and published to a registry. This is a step that not everyone does, but we like the ability to move things around and not be reliant directly on the repository. So we actually do the Helm chart packages and pull the packages down. We create application level repos and application Helm charts to allow for quickly collecting the microservices into an application and deploying them as a whole. And I created a Helm chart library in ICPSR that allows us to very quickly assemble Helm charts from reusable template components. And the deployment repositories also have pipelines and pipeline runs for external dependencies such as databases. So spinning up databases in AWS, for example, and things of that nature so that as we deploy an application freshly into a next stage or into a new cluster, that the dependent resources are spun up as well. And then this plays into the reconciliation process with that. So here's the deployment repositories here. We have my ubiquitous testing deployment Helm chart, which is called taco. Ty always use that for testing. I love tacos. And then we have applications and these are all Helm charts, repositories with Helm charts and the associated resources. And you'll notice that one of them is the app deployment. And then there's other ones that are the API, the UI, et cetera. So these are all in individual Helm charts, but then the application one assembles them. And here's an example of the application Helm chart. So we have the, on the left side here, you can see a typical Helm chart folder with the respective values and testing scripts and whatnot. And then on the right side, you can see is a zoom in of a reveal of the chart.yaml file. This is the file in a Helm chart that defines your dependencies. And as you can see here, we're pulling in the Helm packages of the microservices. And Christian actually has a name for these. They're basically empty Helm charts. So there's no, there are no templates or very few templates in this top level Helm chart. They're basically used as a reference to all of the microservices. And so this is the library that I wrote, which basically has YAML files that allows to very quickly assemble some common manifests in our Helm charts. Elasticsearch credentials, database credentials, data site, OAuth credentials for node applications, and then OAuth for Java applications, which has slightly different needs. And you can see also in there an S2I build config utilizing OpenShift's S2I build technology. So we have actually build configs that are templates as well. And the cluster configuration Helm charts contain much of our day two configuration, or as I call it hour two, because you spin up the cluster and you have these Helm charts and you have your Argo CD and you just get things running very quickly. Manifests within these Helm charts are active directory binding, backup configuration project templates, links that go in the console, the GUI console of OpenShift, setting things like session timeouts to meet our security requirements, configuring the cluster monitoring operator and more. And these cluster configuration Helm charts allow us to have consistent configuration across our clusters and to organize those resources in a way that they can be very easily continuously monitored. And then there's the pipelines that make all of this happen, right? The getting the stuff from the repo and created out in the world for the external resources or initial deployments before Argo gets to it. So basically we have new pipelines that we created with Tecton, which allows for easy pipeline creation. There's the GUI tool that allows our developers and all of the DevOps to be able to very easily create their own pipelines and insert new tests and tools. These are used for modifying the deployment resources, such as the Helm charts, like the image tag in a Helm chart. Also for modifying Terraform and other pipeline components, like other pipelines components and more. And we automated the publishing of Helm chart packages so that those packages are up in a Helm registry. And also deploying container images to different projects and registries. ICPSR has a lot of security requirements that require a lot of redundancy and a lot of sort of siloed configurations between different stages and between projects. So we often do a lot of copy of containers to different projects and registries as opposed to just allowing cross communication. And of course, Argo CD to continuously monitor the Helm charts and other resources and reconcile them as needed. Here's an example of a Helm chart for managing pipeline resources. And it groups pipeline resources based on function and allows for clear version management and Argo CD can reconcile against the chart for full GitOps. This is an example of a Tecton pipeline with a webhook. So you have the binding and the listener, which is basically what listens for the incoming connections and then maps that and then directs that to a route, which then directs that to a pipeline. And so we actually have this in a Helm chart and then a values file allows us to tweak and have particular values when this chart gets deployed. And then configuring and managing external resources. We have repositories that contain external configuration components, which are applied via the pipeline for Terraform, for AWS, email templates for Amazon's SES service. Schema for databases, schema for Apache Solar and a lot more. And so we're getting to the point where the promotion of a Git commit hash means not just the code, but the developers are working and the DevOps engineers are working to make it so it's all of the resources. That Git commit hash represents everything is ready for all of the resources necessary to go to the next step. And we do use customize. We use customize via pipeline. I found it's best suited for singular resources, things that aren't part of a collection, where there aren't multiple things that need to be collected for this to work. I modify pipeline runs existing in the deployment configuration repositories and I modify files that require only very small changes at one value as it were. And here's an example of a pipeline that we have in our GitOps process. So we have what's called a comment gate. And so I'm starting to build sort of a bot, basically infrastructure, where using sort of a chat ops model, commands can be issued from the Git repository to trigger the pipelines. And so we have a comment gate. I wrote this to filter comments and you can set what comment you want to look for and who it should come from. And that allows the rest of the pipeline to run. And so we've got a Git clone. We've got to generate a new Helm chart and then package package and then publish that package up to the registry. And so here's our process basically. We've got developers and DevOps working together with Helm charts, GitLab, OpenShift OCP, Argo CD, all of this running on top of AWS. And you can sort of see a little bit of the flow there, obviously there's a lot more nuance going on, but that's the basic process. And the results of GitOps, of implementing GitOps, the microservices can be reused quickly and efficiently speeding up the creation of new web applications, configuration for external dependencies is deployed consistently. Developers can spin up new web applications without server admin intervention, including building one-off instances to test functionality. And the process of promoting applications through development, QA, UAT and production stages is efficient and less prone to errors. And resource usage can be tracked at a higher rate of resolution as you're watching things get synchronized and what's needed in this feedback loop to get pulled and what gets modified in the Git repository in terms of deployment, your pods, CPU changes that have to be made, it's all part of this bigger picture of feedback right in a system. And clusters themselves can be scaled up quickly as well with this GitOps model. So suggested next steps, if you're interested in getting into this, separate out your deployment from your service code, strongly recommended, it makes things a lot easier in terms of management and allows you to, if necessary, sort of control access to these things and separate out the need of developers necessarily to have to worry about learning Helm pipelines, et cetera, if they don't want to. Or worry about making changes to that. Create top-level application repos in Git. I found this is very helpful. Create a group, a Git group, within that an application level repo and then your microservice repos and then references to other resources, if necessary. Create top-level application Helm charts, as I showed you, those sort of empty Helm charts that just reference the respective microservices. Clarify delineation of software for an application and software to multiple applications. So start thinking about what's gonna be used once, an API or a microservice or a GUI microservice or whatever. What's gonna be used once in this application or what will be used in multiple applications? That will affect some of your design decisions. Health checks for all microservices are important to contribute towards sort of this intelligent life cycle of pulling information and then modifying Git resources and then in turn, RgoCD pulling those back up to spin up changes as needed. And also a mental shift to providing software for consumption in other applications. And think and communicate in terms of Git commit hashes. It's counterintuitive to some traditional developers and software engineers and sys admins to think in terms of Git hashes, but it's helpful because once you start going into this model of Git having everything, that hash is the point in time. That's all of the resources where you want them to be for deployment. And avoid using floating tags if you can. You want a single source of truth in that Git commit hash. Improve your documentation such as read me and changes files to better understand what the automation has done and is happening. And that will help folks embrace the GitOps automation and reconciliation process when it's clearly explained to them and it's explained how that actually works in terms of their particular software projects. I found in a lot of my work in leading this transition to ICPSR is that you really have to frame it in reference to their projects in particular. Here's a list of resources. This will be available in the PDF of the talk. Thank you very much. And of course, none of this could be done without the hard work of developers and DevOps engineers in ICPSR and the support of the GitOps community Christian and a lot of other folks that are doing the great work to flesh out these ideas. And so that's my presentation. Thank you very much.