 We're going to start, I think we're one minute early. So one more extra talking time at the end for questions. Welcome to our presentation and demo about building a self-service database as a service for your internal developer platform, which actually follows on from the talk Michael just gave, which is really cool. My name's Dan McKean, and I'm a, sorry? It is on. Testing. Can you hear me at the back? No. This should be, that's a lot better. I can hear myself. Cool. Yeah, welcome to our presentation and demo about building a self-service database as a service for your internal developer platform. My name's Dan McKean, and I'm a product manager at MongoDB. And I'm George Hanzaras. I'm the engineering director at MongoDB, working with Dan leading our Kubernetes efforts. We're responsible for enabling our customers and users to run MongoDB in two ways. Firstly, self-hosted in Kubernetes. So our enterprise and community operators or using the Atlas Kubernetes operator, which is designed to manage Atlas resources or cloud resources through Kubernetes, using Kubernetes primitives. So before we even jump into the agenda, we're going to start with a little story that's going to kind of flow throughout our presentation. A small team of developers was working hard to complete their work by the end of their sprint. As normal for them, they were using manual processes to manage their infrastructure, including configuring their MongoDB database. This had worked well for a while, but as the team grew, they had more services to manage, and they were onboarding more people to the team. And one day, two developers, Jack and Jill, were working on two different projects but using the same database. Jack implemented a new authentication mechanism, and Jill implemented TLS. And unsurprisingly, they cut themselves off, both of them off, from the development database. They were eventually able to fix the problem, but it took them a long time. Fortunately, this wasn't a production system, but they instantly realized how bad it would have been if it had been serving customers. Hands up if anyone here can relate to tripping over somebody else's work. Yeah, most people. So they realized that they definitely needed a better way to manage the configuration for their systems. And we're going to talk today about how they actually did that. So throughout this presentation, we're going to build upon the story. We're going to talk about how this team changed their approach to managing configuration changes, addressing their intermediate needs and their problems. Then we're going to talk about how they began to standardize their toolings and processes across different services and systems and different teams and delivering a better developer experience for themselves and for the whole organization. Then we're going to talk about the step that the whole company took in order to standardize those systems across different teams. And finally, we're going to show you how you can experience the same benefits through a short demo. Before we jump in, we're going to hold our hands up about the fact that while 90% of this presentation isn't specific to MongoDB, it's largely focused on enabling a DBAS to be used in a development environment in an IDP. We have obviously chosen to use MongoDB for the database here, which is probably quite unsurprising. You could swap out the database with any other, as long as there's a Kubernetes operator. As Michael talked about just now, we're effectively offering a pass service through our Atlas platform. And our Atlas operator enables that. So while you could use another database, we're not exactly going to recommend it, because we're MongoDB. So with that said, let's get back to the story. Jack and Jill's team unsurprisingly had a retro after this happened. Again, it was just development, but they still knew to do something about it. Finally, well, they identified that they needed to make a couple of changes, identify the problems that they were facing in their approach, and find some solutions that stopped this happening again, let alone in a production environment. And it came down to two core changes, or a couple of core changes. So they tripped up on a few overlapping problems. Team visibility and awareness of each other's changes. And neither of them made a bad change, but they didn't. They weren't aware of each other's changes. And if they had been, they'd have avoided the problem. But better than just relying on human communication, they could have done peer reviews for the planned infrastructure changes. They already did code reviews of each other's work, but not for the changes to infrastructure. And finally, there was no easy way to revert the changes. Perhaps some of you have guessed the solution that they landed on. So they decided to start implementing infrastructure as code. And they also decided to start storing their infrastructure configuration in their existing code management system. And for the sake of this example, they were using GitHub and for the projects that were using the databases. So that seemed like the easy choice. So they started storing configuration file for the databases alongside their code. And this ended up with a number of benefits. So first off, they started doing pull requests similarly in the same way like they did for code changes. They did that for infrastructure changes. Through peer reviews, each team member knew what changes everyone else would be applying. And if they'd still managed to make conflicting changes, they could easily revert those. So this all worked great, but there was a catch. They'd implemented this for one of their projects, but they hadn't gone round to doing it for the rest. And as a result, their services were managed differently using different tooling and even running on different types of infrastructure. And this was costing them. So with a lot of different tools and processes, it was hard for existing team members and even tougher for new starters. Even hiring people with the right skill set and experience was tough. In their systems where they'd not yet adopted infrastructure as code, over time they were seeing the configuration drift as people made small changes that made it hard for anybody else to know the current state and to use the system, let alone troubleshoot anything that went wrong. They were experiencing infrastructure drift. And lastly, the security team was also breathing down their necks. The number of systems made it hard for the team to keep track and keep the business safe and the team system secure. It also made it a nightmare web of roles and missions that each new team member needed or collaborators even. And this was painful for everybody in order to get access that they needed to do their jobs. So all of these complexities of managing infrastructure made them realize that they needed to standardize their team's tools and processes a bit more. So this time there were several changes they realized they needed to make. While none of them came for free, they knew that what worked best from existing tools and processes they could replicate across different systems. So they decided to use a standard platform and Kubernetes was their first choice for a few reasons because of the self-healing capabilities, declarative configuration, the active ecosystem of the complementary tools and the rapid and expansive scaling which matched their company scaling. And also it offered a high level of customization to match their different needs. So one of their systems was already running on Kubernetes so they decided that they had enough knowledge to start running more of their systems. They also settled in a standard way of implementing infrastructure as code through GitOps with our Go CD. As we discussed, they already had success with infrastructure as code but this time around they wanted to standardize and automate the process of deploying changes even further. So this gave them a consistent and reliable method of applying the changes from their repository in their Kubernetes infrastructure. Finally, they had much far fewer tools and much simpler processes to worry about and that gave them more time to work on their applications and as a result this increased their velocity and of course their happiness. It also limited the rights and the permission needed in the Kubernetes clusters so the security team was really happy. No one needed to make edits in the Kubernetes cluster anymore. Everyone just pushed in their GitHub repository and Argo did all the rest. So with these improvements in place, our team was running like a well-oiled machine but as they'd been making this journey the problems were getting with service across all of the other engineering teams at the company. Alignment across teams was becoming a hot and often heated topic of conversation. No two teams were working in the same way. Engineers moving between teams had to effectively re-on board in order to do so and this also limited collaboration. Any new team had to decide and implement on there tools and processes and that meant reinventing the wheel costing them valuable time. Governance and policy enforcement were also struggling. Writing a policy that could be easily applied across teams was a nightmare and enforcing it was even tougher. Finance weren't that happy either. It was hard to attribute cost to teams or projects. And finally, self-service was suffering. What little shared infrastructure they have wasn't set up for self-service. There wasn't much shared infrastructure and trying to do it on each team's own systems was painful. So after they identified all those challenges they had a lot of discussions and they realized that they kind of needed to standardize across all the teams and as a lot of discussion is already going around about this topic, they kind of decided that they wanted to adopt platform engineering and build an internal developer platform because they wanted this to be centrally owned, centrally managed and highly self-service. So additionally, every team needed to start their application data somewhere. So they decided that one of the first components of their platform should be a database as a service and they also started discussing about the technologies that they would use for the platform. And they made a few choices. First off, we mentioned Kubernetes. That was the easy winner for the underlying platform for the IDP. The CNCF estimates that about 70% of platforms are already built on Kubernetes. So there was a lot of prior art available for them to choose from. Okay, MongoDB Atlas was their choice for the database as a service. They considered different vendors, ended up with Atlas because it's already an as a service offering. So they didn't want to manage their databases on Kubernetes, although they wanted to manage it through Kubernetes. GitHub, as we said before, was the choice for storing infrastructure as code. Argo CD was their selected choice to automate the deployment of infrastructure configuration files from their Git repository to Kubernetes. And finally, as I just mentioned, they did want, although the databases weren't running on Kubernetes, they did want to use Kubernetes to manage everything. So they started using the Kubernetes Atlas operator which helped them manage their Atlas resources in a Kubernetes native way. Now, a very quick introduction to what the Kubernetes operator is. It's a service designed to extend the default control plane of Kubernetes to support custom logic and custom operation of a specific application. In this case, it's an external application, namely Atlas. So using the operator allowed Jack and Jill's company to use the same tooling and processes to manage their databases as they already used for applications running in Kubernetes. So Atlas databases are running one of the three major cloud providers, but with the operator managing configuration running in Kubernetes clusters, it allows users to configure databases with configuration files that can be managed in a repository agnostic of where the databases is running. This saved them a lot of time to implement new tools, new processes, understanding new architectures, and so on. Better yet, using an operator to configure the databases allowed templating to be shared out to development teams. This created standards of databases and made it really easy for developers to start onboarding and start using, just copying the templates from a repo in their repo and starting deploying their databases, customizing it really easy. And through Argo CD automating the deployment process. And finally an operator helps prevent configuration drift and that's probably the biggest difference of other traditional infrastructure code management tools. So the source of truth is a repository and it's not great to make changes directly in the infrastructure. So the operator does periodic reconciliations and ensures that what is in the repository is the truth that we see in our infrastructure. And if that's not the case, manages to make the changes when needed. So Jack and Jill's company had reached a great place with great consistency across the teams. But they made their journey to an IDP and their D-bass over time and quite a few iterations. And if they'd known where they were gonna end up, they'd have likely taken a more direct route. So that's what we're actually gonna show you in our demo. Our shorter journey is split into two parts. First, deploying and managing the database through the Atlas Kubernetes operator. Ensuring that developers didn't have to learn something extra. They could just use the same tools and processes to deploy infrastructure as code that they were already using to manage their stacks actually running in Kubernetes. And secondly, automating the process through a GitOps approach using GitHub and Argo CD, which made it super self-service and really easy to use with templates that could be provided centrally. So when Jack and Jill's company realized that there was an operator available for their databases, they chose it using it to become the management layer for their databases. In particularly, the Atlas operator can be installed in a few different ways. Through Cube CTL, through Helm charts, or even the Mongo CLI. And for our demo here, we've decided to show this easier way of setting it up because they were already using the Mongo CLI to, the Atlas CLI to manage their database resources. So they simply run one command and in the background that installs all the necessary resources in the Kubernetes cluster. We can see all the operator pods running in our Kubernetes at the moment. And besides that, we also have created the necessary API keys because we are gonna be managing an external resource using the admin API. And in the secret, we can see that we have the API key which the operator is gonna be using to manage the external service. So now that the operator is installed, we're gonna create a new project. This is not good to use. We've given access through everything. So a simple Cube CTL apply and we see that the new project is created in our Kubernetes cluster. So we see the Atlas project resource. And then we see that the operator in the background through the Atlas admin API has picked up the resource and has created the Atlas project in our Atlas account. So then we go and create a new deployment. Similarly, a simple YAML file, the Atlas deployment resource is the one we're gonna be using. We can define the size of the cluster, the region, anything we want. These are Atlas specific configurations. We apply the custom resource and we can see that the custom, the Atlas deployment custom resource has been created and the Atlas deployment, the MongoDB cluster is being now created within our project. So using the operator enables you to manage your database in the same way that you manage your applications in Kubernetes. But that didn't necessarily solve all the problems for those, for that team that we were talking about with Jack-in-Jail. And it wouldn't necessarily sort those problems out for other companies either. Complexity at scale. In a company with many services, many teams, many engineers, you need tooling that scales and standardizes across the teams. Rights and permissions are a challenge. You need to give everybody the rights or rather without it, you need to give everyone the rights to directly manage services on production Kubernetes clusters. And without self-service and automation, a lot would be being done manually. Every new engineer has to learn how to do it and you don't get the benefits of the consistency. So engineers don't really need or they don't even want to manage their Kubernetes clusters and their Kubernetes resources directly. So they usually prefer to focus on their code. So infrastructure as code stored in Kubernetes repositories was part of the solution but Jack-in-Jail realized that centralized, centrally managed Kubernetes clusters with Argo CD installed provide a level of abstraction that saved a lot of time and effort for the engineers. So as a first step, we need to install the Argo CD operator. We just need an Argo CD namespace to install everything there. And then installation is automated through a script which automates the installation of everything and we can see that the operator pods are running in our Argo CD namespace. Now the next step is creating our D-Bus application in Argo CD and it looks like something like this. We create an application custom resource. We define the repository we're gonna be watching, which revision and which Kubernetes cluster we're gonna deploying our changes up. We simply create the application and then we are ready to, we see our D-Bus application. And now we are ready to start creating databases with this and through this automated process. So to deploy the Atlas project, we use exactly the same CRD as we did before, the Atlas project resource, but instead of running kubectl apply, we just do a git push in our infrastructure repository. And after the push, we can see that Argo CD has picked up the change in our repository and then goes, creates the Atlas project custom resource in the Kubernetes cluster. And then similarly under the hood, the operator picks this up and creates the Atlas project in Atlas through the Atlas admin API. Similarly, we use the same cluster yaml as we did before, the Atlas deployment custom resource. Again, git push instead of kubectl apply. We don't need access to the Kubernetes cluster at this point and Argo CD picks up the change, creates the Atlas deployment custom resource in Kubernetes and that is created in our Kubernetes cluster. So this was the end of the demo and probably you were expecting fireworks, but when we manage infrastructure in production environments, not having fireworks is a good thing. So with all this done, we've reached the point where Jack and Jill's company had reached after all those iterations, managing databases in production, deploying databases in an automated fashion with very little risk. And if you also want to kind of experience this transformation, see this demo, find the code for all this demo, this is a good time to prepare your phone. In two slides time, Dan is gonna show you a QR code which is gonna enable you to reach the GitHub and the resources to kind of go through this transformation journey. Cool. So as with the many things, the value of the whole is greater than the value of the sum of its parts. All the components we just showed you just now from Kubernetes even to Argo CD to Atlas and the Atlas Kubernetes operator, each provides value in isolation, but it's in the overall databases as service plugged into their IDP that the value really shines. And as I mentioned at the start, you could theoretically swap this out with any other databases as long as they have an operator available, it could be running within Kubernetes or as a cloud-based service like Atlas is. So self-service empowers users and is almost always faster than developers having to figure things out for themselves or having to ask someone else to do them. Central teams are able to focus on supporting the teams using the databases as a service and the IDP and work on improvements that benefit all. Developers are free to focus on the value add work which often leads to faster innovation, shorter time to market and greater productivity and usually greater satisfaction as well. Centralized onboarding in common processes and tooling between the teams allows for much more rapid onboarding, enables developers to move between teams and enables collaboration across teams which can help drive business agility and retention of developers. Cross-team collaboration as I just said is now possible with different teams able to work on the same projects and contribute to each other's. There's significant cost savings that come from the use of fewer tools, gives the company better ROI and allows for the effort to be spent on maintaining those central systems. And finally, as I kind of already alluded to, developers are happy and more engaged and everybody loves it when their developers are happy. So overall a well-implemented databases service and a great IDP can contribute massively to the success of an overall business from reducing cost to improving retention to shortening time to market. All of these things help any modern company succeed where great ideas alone are often not enough. And that's pretty much it. So through the link in the QR code, as George just mentioned, you can find a landing page that gives you a load of information and it would replicate this demo, see the tooling we used and find out a little bit more about the services that we talked about. Before we take any questions, we also wanted to take this cheeky opportunity to plug a talk we have on Wednesday. That one's called patterns of multi-cluster and it covers a lot of concepts and challenges around running multi-cubanettis cluster workloads. And I'm surprised that we've been talking about how that we do that by using our self-hosted MongoDB to provide a database as a service just spanning multiple clusters. We're really excited about doing that talk. It's a new one for us and it's really, really cool. I'm biased, of course. And now I think we have a few minutes left for some questions. If you do have questions, please make sure you grab a microphone or go to that central microphone to ask your questions so that anyone remote can hear. Yeah, if you have a question, just head to that microphone or it's probably the best way to do it. Hello. Can you guys hear me? Okay. Yes. So I have a question regarding the access management. Like, so we have been using MongoDB Atlas in production. So one of the issues that we face is like how to provide access to the MongoDB Atlas console for different teams, different users. And like for, we have the way of like, we're using Kubernetes operator, but it's mostly for creating the database clusters and there is projects, right? So how we can kind of manage access for the developers? Do you mean in terms of rights and permissions or in terms of network access to the database? Both. Okay. I'll start with the rights and permissions because it's slightly easier and our operator actually enables it. The solution there is that we allow you to create users. We didn't kind of go that far because we wanted to focus on the kind of programmatic management of a database as a service, but you can create users using the MongoDB Atlas operator. You can even create custom roles or utilize inbuilt roles and that allows you to give really, really fine grained access to the databases in Atlas. Not only the databases, but the kind of the deployments that hold multiple databases and the collections that exist effectively as tables within each of those databases. The networking access is slightly harder and it is a challenge we as a company are trying to solve because we're aware that when you're using a platform as a service like Atlas, providing developers a service way to achieve connectivity to it can be a challenge. I'm not going to go into it too much at the moment because we haven't got a great silver bullet, but there are a number of different ways you can do it, but it also depends on which of the three cloud providers you're using to actually run your MongoDB Atlas databases. That can change a little bit because obviously the networking solutions across the three clouds are not completely common. So we generally tend to use like VPC, like we were using AWS, right? So we tend to use like VPCs to connect to those MongoDB clusters. So you guys have been working on that like through the operator, if you can create those VPCs. Yeah, so we create a VPC for any project in Atlas which can hold multiple deployments and you can set up private link to connect to those. You can do that through the operator and actually we're making a change soon to be able to allow you to set up the cloud provider end of that private link as well. There's still some considerations that we can't automate because you kind of need to think about them a little bit more on the cloud provider end of things to make sure that all the access is set up as needed, but we do and we are working to continually improve how we provide access to the database in the networking sense. Guys, thank you. Thank you. Any other questions? Cool, thank you. Thank you. We'll be hanging around for a couple of minutes. So if anyone wants to ask us any questions, privately feel free to wander over and say hello. Thank you. I have a question. We'll be at the back. Oh, sorry. One more question. Go. I was just checking out, right? You've made mention of three cloud providers. So I wanted to ask, can one use Atlas on-prem? Sorry, you're asking about Atlas on-prem. Okay, so at the moment we don't offer Atlas on-prem. Atlas is not only a heavy degree of automation around the databases themselves, but it's also a number of extra features of functionality that makes it a developer data platform, as we've termed it. That includes more things than we've got time to talk about today. We do offer MongoDB community and we have an operator for that, so you can run that in Kubernetes. You can also run MongoDB Enterprise within Kubernetes because we have an operator for that. It's not exactly Atlas on-prem, and although the operator makes it much, much simpler than running and managing all the database stateful sets and all of that sort of stuff, it's unavoidably still more complicated than using a platform as a service like Atlas. We do have a lot of companies using it across a variety of industries, and it does work really well if you do need to run your enterprise databases on-prem, but we'd still recommend Atlas. It's just that much easier if you can. Yep, thank you very much.