 I'm Lisa, as was mentioned. I'm also a CNCF ambassador, which is why I'm wearing this scarf and also shout out to our brother, Eor. And so happy to be here moderating this amazing session with some of my favorite people talking about one of my favorite topics. So you've all heard the joke, I'm sure by now. I've said it almost every time I've done this talk that there really isn't any such thing as a stateless application, like maybe the only stateless application that exists is Hello World, possibly. So everything has some form of state. So we're going to dive into that a little bit deeper. So if you're wondering why you're here, wondering if you can run stateful applications and communities, I know you all know you can. But we have some experts up here who have taken it to another level. So why don't we have everybody introduce themselves starting with Gunna. Thanks for joining. I'm Gunna Maripuri, senior director of product management for NetApp. I actually work with a lot of customers, including Scotty and Scott. So thanks for your hope to have a good panel. Hi, I'm Scotts Rovich. I work for HSBC Bank. And I'm the global container engineering lead. And I also own Kubernetes globally for on-prem and hybrid solutions. My name's Scotty Miller. I'm a technology fellow at DreamWorks Animation. And I oversee strategy and architecture for infrastructure, including Kubernetes, virtualization, and legacy. So not legacy. Heritage. The new term is heritage. And hi, everybody. I'm Diane Patton. I'm a technical marketing engineer with NetApp and focusing on Kubernetes solutions. I love it. You haven't stopped saying heritage for weeks. I just learned it today. I thought of it. Everything old is old and classy again. Very funny. OK, I also just have to do one more thing to embarrass. I love to embarrass the panelists. But in case you haven't downloaded this, bought this, read this, Scott has written a book. You can't see what I'm showing them. He might be able to if you can see my screen. Yeah, I had to. Get this book. It's available on Amazon. And he's an expert. We had to validate. We have at least one expert on the panel. I'm sorry. They're all experts. OK, but back to us. So we're going to talk about the different phases of the cloud journey a little bit. And the journey to containerization. So Scott, why don't we start with you? Sure. So our container journey started in about 2015. We had a relatively unique opportunity to re-implement the production pipeline for feature film. So a pipeline is a collection of digital content creation applications, data sets, workflows, how you hand that stuff all around. And we decided the new hotness was cloud native. So we wanted to re-implement using microservices, including the databases that were being used to support these applications. So we took a microservices approach to databases and wanted to deploy those in containers, in Kubernetes clusters, as well as the application front end and services. And the journey's been great ever since. We've gone from pre-docker from LXC containers and our own orchestrator that we hand wrote. I don't recommend that. Moved into OpenShift. And then upstream Kubernetes is our current deployment. And then a little bit of Azure Kubernetes service when we need cloud. And Scott, as he mentioned, Scott owns Kubernetes. So in case you were wondering who owns Kubernetes, because we were all wondering who owns Kubernetes, Scott owns Kubernetes. So tell us about your. Well, first of all, I'm a bank. Doesn't sound as cool as when he said animated film, all the pipelines. But without us, he couldn't do films. So you need money, right? You couldn't pay for them. That's true. So we started our journey about the same time, around 2015, 2016, actually a little bit before with Mesosphere. So if anybody in here has been around that. And that kind of went for a couple years and Beverly took off for us, other than Jenkins. So we decided to get over into, of course, Kubernetes. And in about three short years, we've grown my infrastructure to about 3,000 nodes, 65 clusters, and 70,000 containers. And next year, we expect that to grow by five. So a lot of stuff going on. So before we get into the questions, I want to find out a little bit about you. So a couple questions for you. Who here, we want to know where on your Kubernetes journey you are in. So who here is running Kubernetes? Show of hands. Keep them up if you're running Kubernetes in production. More hands went up than were before. That was interesting. OK. Keep them up if you're running stateful applications on Kubernetes in production. OK. All right. Well, that's actually more than I would have thought. I just saw Eric cruise through here. Eric and I did this talk about two years ago in Barcelona, a multi-cloud version of this talk, and asked those similar questions. And they're a fraction of the hands that just went up just two and a half years ago, CNCF, KubeCon, Barcelona. So we also want to know who you are, strategists versus implementers. And actually what we really mean, like Kubernetes admin versus an infra admin. And by infra admin, we mean like heritage, infrastructure, VMs. OK, so show of hands, infra admins. OK. And then Kubernetes admins. OK. And then the rest are strategists. OK. And just for my own personal, architects and operators and software developers. OK. Thank you for that. Super interesting. And I know some of those roles overlap, especially in some of the smaller companies. I know that. Some of you wear all the hats. We all wear all the hats. So I want to talk a little bit about what applications you're running. So I introduced myself as a CNCF ambassador. But by day, I'm also head of DevRel at Cockroach Labs. So we just published a report, not Cockroach, but the DOK community, one of the ones I'm involved in, a Kubernetes report. You can download it. It's available as of last week. Really interesting information about who's doing what on Kubernetes, who's doing it, how who's making money with it, how long they've been in it. But one of the questions was, what applications are you running on Kubernetes? And the number one answer was databases. So by day, I work for Cockroach Labs. Yes, Cockroaches exist during the day. And so that is my day job. And so I was really happy to see that databases were the number one app that the enterprises, the 500 enterprises that the DOK survey queried. That was number one. Number two was analytics, up from number six last year. So that was interesting. But let's hear about the applications you're running in Kubernetes. Scott. OK, so as Lisa was saying, what do you run? What kind of apps? In our case, we're a bank. So payments, it's pretty important to a bank. Payments aren't flowing. HSBC, there's trillions of dollars a day that flow through this. So our payment system is on Kubernetes. It takes you to be migrated. And it requires storage because it's running Kafka. So if anybody in here has run that, you need to put those log files somewhere. If you put them on the overlay file system, you're going to get 80% and evictions. So that's just one. The other app we have that's pretty popular to do this with is IBM MQ. And this one gets more interesting. You have one container in, we'll say, one cluster, another cluster. We replicate that data from the prod to the DR. So replication of data is not just a heritage thing. It's actually something you do today. By doing that today application, we have saved 72 cores of cost just for one instance of that business. That's my trading platform. So I like to tell people, those are both 2-0 apps. If we're not trading and we're not moving money, DreamWorks gets no movies or something similar. We have to have that. Yeah, it's amazing how that all works. So the application stack that we run, it's a collection of microservices that implement the asset management, workflow management, and data management features of our film pipeline. And each app, there's about 50 microservices that make up a collection of apps. And each app's persisted state in a database. So we made a cloud-native microservices like Decision seven years ago where databases should be deployable just like any other Kubernetes app. So we deploy a minimum 3-0 database cluster for every app. So we've got about 1,500 stateful sets running a little over 550 database clusters deployable by the developers. So if a developer needs a database, they say, I'd like CouchBase, or I'd like Kafka, or I'd like Elasticsearch. Whatever it is they need, they can deploy it using the same CI CD tool chain, the same mechanisms and APIs they're used to for deploying their application itself. So the interesting thing there is we not only use databases to persist state for the applications, we treat databases like Kubernetes-hosted applications from the beginning. There's some tricks there, but we'll get to that. So if we can add one more to some of the apps that Scott and Scott are talking about. On privilege of me working at NetApp, talking to a lot of end users, people are actually using not only databases, they're also using Jenkins comes across multiple times. People are actually running thousands of instances. One of some of the customers we work with, they're running like what, like 90-node clusters, gigantic clusters like 150,000 builds a day, gigantic deployment. So I think stateful applications are here across the board DB2s and CI CD pipelines. Yeah, and so we're not trying to tell you what the definition of a stateful application on Kubernetes is. We really want you to understand the implication and what's the implication of not doing this? I mean, and first of all, you're not gonna put everything in Kubernetes. So let's start there. What aren't you gonna put? What don't you wanna run in containers? I wanna run everything in containers. And the reason for that, containers are not only a way to run things, they're a way to package things, a way to distribute things, to a way to isolate an application from the heritage environment. I'm getting pretty good. We're using containers outside of Kubernetes as a way to package up interactive artist applications so that I can give them a self-consistent version of a DCC app with the right plugins, runtime libraries, that environment. So containers as a packaging distribution and self-consistent way to run it, to deploy an app is brilliant. Kubernetes is a way to orchestrate a lot of containers. So I can't think of anything unless there's something active in the implementation that doesn't wanna be on a container. I mean, the intent- Everything, legacy apps, everything? Heritage apps. The intent is, that's kind of what I was talking about this earlier, we've got a deployment strategy. If somebody needs to run a thing, it's container first, VM second, bare metal, hardly ever. And the only exception to the bare metal rule is the HBC environment and everything else is a container first and then a VM if it has to be. You had some things you weren't gonna... And say, of course, when the question comes up, containerization, I'll use the line. If I could, I would containerize my dog. So it's like anything, just put it in there. Especially Linux, let's be honest. So you go to Windows, it's a different story, right? You can do a lot with windows.net, IAS. But in my opinion, you can move that monolith over. Microservice is great, of course, right? But that monolith, MQ, good example, that's not exactly a microservice, but it's saving us a ton of money in CPUs. Now those CPUs, because it's not on a VM anymore, means no more patching. I don't have all these agents running. I'm not paying for operating system licensing. And of course, overall containers will use less resources. I don't think I know of an instance where a container takes more than that app on an actual VM. If there is one, I'd like to see it just so I can know, but I've never seen that. Just to complete, given that these are like no windows, they're not using windows. If you have windows containers, you can run them too. Yeah, and I've never worked with a customer that basically said, we won't want to deploy this new application and we don't want to use containers. So they're always going to containers as the first option. Okay, so we, for the most part, have known each other for a long time. And in my opinion, you've been early adopters, Scotty. I mean, like really early adopter, probably Scott as well, but I haven't known you as long. But the, so, in general, you're ahead of most of the folks that I talk to, most of the folks in the user group that I run in the Bay Area, San Francisco Bay Area. So where are you in your journey? Like how far along are you, I would say, in this forever journey? You're never gonna get to the end, but what stage are you at now? It's not a journey, really. It's a lifestyle. You have to decide this is how you want to deploy applications. We're all the way through kind of the first generation of deployment and we're doing some refactoring. One of the things that you get when anyone can deploy a database with Kubernetes is you get a lot of databases. So one of the things we're doing now is a little bit of refactoring. We're doing some multi-tenancy work. We're trying to reduce the total number of instances, but make them bigger. So we're gonna leverage the horizontal scalability of a database in Kubernetes as a way to do more with fewer instances because then you have fewer that you have to upgrade, fewer you have to patch, fewer you have to track. We're at the point now where we're still trying to hire people who know how to work Kubernetes and understand the data model and understand how to write operators and understand how to manage an ecosystem like this. That's probably one of our biggest challenges. So even after seven plus years of doing this, finding enough people who want to support, develop, and enhance an environment that's running databases as Kubernetes items is pretty challenging still. For us, we've, like I said, it's been about five, six years. We're pretty far along that journey. We're multi-tenant already. I always have been. Four years ago at Coopam, people said I was crazy. Mark heard that. Multi-tenant, why would you do that? Exactly why you said. From day one, cluster control. You don't want control planes going all over the place. Just better resource utilization. Now our problem is finding people and then you're working in an environment that's highly regulated. That is not fun. If anybody in here works for government and or a financial institution, it can be like molasses. But luckily for Kubernetes, we've been going really fast. So it's a very rare occasion. And I do have some problems. We still need education with developers. So we still have a lot of developers that need help. And that's understandable, right? They can code, code the heck out of Java, but they don't know how to make a deployment or they don't know how to do a PVC or they create an ingress rule from Mongo, which of course won't work. So we're along, people are getting there. As I said, payments, tier zero. You can't get much more important or scary. That goes down, I go down. That means I don't have a job. I can hire you. It's good talking. I think a lot of people here would hire you. Another thing that came out of the Kubernetes report, they were asked what problems people are having. Number one, security. Number two was talent. Hiring the talent, the right talent, who can write the operators that are necessary, not enough talent out there. And interesting stat, 60% of the enterprises, these 500 enterprises that were queried, were saying that we're hiring this talent in the next three to six months. So even though everybody has hiring freezes and we're in some recession, not recession, everyone is hiring for this. Who's hiring here? Raise your hand if you're hiring. Okay, if you're looking for a job, look at who's got their hands up right now. It is a coveted skill and it is desperately needed. So how did you go past this? I mean, where do you send people to start, to learn, to get on this journey? We try to do a lot of training internally. Since one of our big stateful application pieces is the databases, we started with our DBAs and tried to teach them, you need to administer a database, which you also need to understand its runtime. You need to be enough of an SRE to understand the Kubernetes environment. You need to be enough of a observability analytic type person to understand the behaviors of your application in this different environment. More importantly, you need to understand failure domains because the way Kubernetes nodes and Kubernetes pods can fail is different than like the way a VM or a physical machine might fail. So we tried to train from within where we could. I wish there were university curriculum people here who could start to teach something other than just how to code in Java because that's not all there is. You actually have to run the code you write. I think the universities need to kind of step up as well. So we do some outreach to try to get people trained up, but that's a five to eight year pipeline to get people through school. Yeah, for us, I actually went internal for a lot because being blunt, it's easier to learn Kubernetes and HSBC policies because again, highly regulated. We're in 63 countries and my team is global. So they have to deal with all of those regulations and what we'll do is the same kind of training and there's just pretty good book that somebody told me about, but aside from that, we do working groups inside the bank. So we do them at all different hours. So Hong Kong can get on, UK, us of course, and East Coast and we just keep training. So you get questions, I ask people what they wanna know and we do, let's say hackathons, but not quite that complex. That's just Scott. I think these days, there are so many resources including your book, there are good training programs and more importantly, the skill set of organizations is actually growing up, the Kubernetes skill set. So we have a couple of SRE teams that actually run our fully managed services. We're all self-hosted. And we have internal training as well for Kubernetes and of course there's always external training that you can take online too. Yeah, it's not like five years ago when we were starting. I don't know who, I'll start at that long ago, but you couldn't find somebody to talk to. Or it was very rare, right? So now there's so many more people and the community is really good. Yeah, I'm gonna put it in a plug for the community. There are so many resources and self-guided coursework and certifications, I encourage anybody, even if you're not gonna be an operator to learn how to be one so when you have issues or your team complains, you'll understand why. Okay, let's talk a little bit about data lifecycle, like apps and data protection needs upgrades, green. I know you wanted to talk about this a little. You had some examples, so do you wanna go first, Kevin? Sure, yeah. I think one of the things that we have been talking about is once you put the stateful applications in production, given that most of you guys actually raised your hand when you did Kubernetes into production, when you bring stateful apps, it introduces some challenges. Like one of the things that I see talking to customers is Kubernetes is on like a nice kid and say every four months, you have a new version coming. So because of the changes and sometimes not pretty useful functionality, unlike the heritage apps that Scotty was talking about, the upgrade cycles are not like six months, nine months a year, like people really want to upgrade a nice kid and so on, keeping up pace. So the big thing that comes is like now, if you're running stateful apps, like some of the DB2, I mean, some of the databases applications are Kafka, Jenkins, et cetera, you really need to think about like, yeah, I'm upgrading it, but am I causing any disruption? So for example, when we talk to some of the customers, the disruption is massive. There may be like 18,000 developers depending on ACACD pipeline. So you just can't afford to bring it down, right? When you're thinking about like a database instance, like Scotty was talking about is billing, like I actually come across some billing users, right? I mean, he's talking about payment system, I'm talking about some billing customers, that app, that's your revenue source, you can't bring it down. So when you think about, one of the, if there is one thing is like, when you think about blue-green upgrades, just think about the stateful application more holistically, how do you take care of the state in the application and what are the policies that you'd like to look into? Yeah, one thing I'll add to that is, it can't be your source of truth. So what I mean is you've got your clusters, you better have backup, right? A lot of people say, Kubernetes doesn't need the backup. Well, that data still sits somewhere, right? I don't care if it's NVMEs, it happens to be in a filer, solid-fire EMC, it doesn't matter, right? You still need something to back that up. So also know your storage, know your storage classes, what you can do with these storage classes. You have to understand ReadWriteOnce, ReadWriteMany. In my opinion, ReadWriteOnce is completely useless or close to it, right? So why, and I'm gonna steal your sign, sorry. If you try to do a rolling upgrade with the ReadWriteOnce, guess what happens? It sits there, because it's trying to link the new pod into the PVC. And you've got a multi-attacher, basically, if the pod deploys on a different node, the new pod. Yep, yep, yep. So we actually ran into that, so. And then people call me and go, why doesn't this work? I went, well, can't do that. So it's all part of that life cycle. And that pretty much takes away the whole point of Kubernetes also, because the way to get around that, or one way to get around that would be to use to recreate, right? And then you don't have your rolling upgrade, unless you want to start working with affinities and things like that, which make everything a lot more complex. Yeah, the other thing to keep in mind is where the data might live. We made a conscious choice to put most of our database-durable data in the cluster distributed amongst the worker nodes using various virtualization techniques to present back a PV. That makes upgrades of the storage environment and the Kubernetes environment and your applications all dependent on each other. So it's doable, it's a good choice for databases because you get really good read and write latencies and really consistent IO without jitter. But the downside is you've created another dependency in the Kubernetes cluster, which you can't just reboot a node because it's misbehaving. You have to think about what other data services might be affected. If you put your data outside, say some sort of a storage array, we're for the durable storage, then you have to protect it using its native mechanisms. So you have what you do in Kubernetes to deploy and operate, then you have what you do in the heritage environment to deploy and operate, and you've doubled the amount of operations and work that you have to do. So there's always gonna be a trade-off, but think about the recreatability of your data, the location of it based on performance and capacity requirements, as well as the overarching life cycle of, if it's gonna last for five or 15 or 100 years, you're gonna upgrade the databases, the Kubernetes cluster, the underlying hardware, the storage hardware, at least once, if not every three or four years. Okay, I've got maybe two more questions for you, then we'll open it up to the room. Kubernetes first comes out, people think it's gonna be everybody's savior, let's just do everything with it, it can do all the things we found out, it can do all the things. What were some of the, you talked about where it went right, what are some of the places where it went wrong, cause that's always more interesting to hear, right? Golly, where did it go wrong? I think that to me, one of the interesting early on things is the fundamental behavior of Kubernetes, if a pod doesn't run, it just starts it again, and again, and again, and again until it finds some, either you get tired of it, or it finally ends up running. You can't do that with a database node and you can't do that with a storage node, you have to think consciously about where it resides. So we spent a lot of time working on operators to make sure that we could carefully tell Kubernetes to stop being itself and do something more natural to the database. Where else did it go wrong? I don't know, there are probably, I can't think of any, everything's working right now and I wanna jinx it. I can't think of any, I can't think of any, I can't think of any, I can't think of any, I can't think of any, I can't think of any, I can't think of any, I can't think of any, I can't think of any, and you start twitching, you know, you get PTSD. Yeah, on our side, it was a, I thought anyways, but. I'll speak some more and you can do stuff. And so, some of the challenges, these guys have been doing it for a long time. I work with some of the customers who are kind of maybe here into it and maybe two years, like relatively speaking, early in their life and the maturity. I'll talk from a data perspective, there may be other issues that you guys can chip in. You know, how you provision storage, like Scotty touched like some very key concepts, you know, where data is residing, where your storage is coming from. But the other thing is also how you're provisioning that storage volume to your part, the persistent volume, how your storage volume gets mapped to a persistent volume. Some of the, I don't want to call it a mistake, if you guys are using don't get offended, it is, now people actually use like the traditional way, you know, you call your storage, it may ask him or her to create a storage volume, map it to a part that's called static provisioning, right? Like right now, like, you know, every storage revender, like, you know, if you guys are running in the cloud, pretty much every cloud provider is providing a CSI, CSI driver, that gives you dynamic provisioning and some other thing that Scott was talking about, it lets you provision different kinds of volumes to match your quality of service requirements or availability requirements. So there is a lot of power, you know, that you don't need to make the same mistake, you know, people who have done, because they're constrained, there is no CSI five years ago, but now I think you don't want to make that mistake. Yeah, and we are still using some entry drivers for some of the storage deployment and migrating off of that is in planning. It's gonna be tricky. And by using some of the CSI drivers also, you can also hand some of this off to the developers themselves. So this way you can basically maybe create some application patterns where the developers can then say, hey, you know, I would like a certain type of storage for my application, a certain amount of storage, then they just need to click that through automation and then they can get what they need in order to deploy their application as well. Yeah, actually that's one thing I do, like as I'm going back to my IBM MQ, the developer can actually cut that disk over themselves. Unlike the old days of like EMC, you might call your storage team and say, split this and then they go and do it. The developer can actually just run a YAML and it cuts it over. And when they're done, it can sync back, both async or sync, depending what you really need. I'm actually working with a different bank that's working on putting all the things together. There are no more banks. We were talking earlier, actually Scott, you've been going to a lot of sessions here and shout out to the community. There's actually a lot of work that has been done and you were talking about in the last session, some of the things that are coming out with Kubernetes that are solving a lot of the problems that we used to have to try to solve manually. Yeah, and the session after this one's about SQL in Kubernetes, so if you're interested, stay. But some of the other things I found out about is like the ability to treat a database as a service connected back into Kubernetes, some of the things coming up make some of the issues that we both ran into over the years kind of non-starters. The switch to CSI as the way to both dynamically and statically provision your storage is interesting. There's work on dealing with affinity to make sure that your stateful stats are running disjoint to make sure they're adjacent to the storage that are ever improving, that's important to make sure your application performs and is restartable. That was kind of leading to my last question about the future, what you see coming in the ecosystem and what are you doing in the future? Yeah, so again, I'll go back to the data perspective, so please add the rest of it. From data perspective, one of the things we see is, Scott kind of mentioned in his thing is there are those so-called heritage apps also have best practices. There are certain things that actually keep this mission critical applications running, right? He talked about don't think back up as a second thought, but not only just back up business continuity. Some other customers, especially when you try to do, if you're already in the cloud, you really need to think about, oh, what about cross region failures? If you're running on-prem, you really need to think about can I replicate that kind of infrastructure in my clouds? So really the replication, Scott was telling is very critical, like synchronous, asynchronous, so that's keep in mind, sometimes some of these capabilities are outside of the CSI as it stands today, but I think those are all some of the things that'll actually give your user's life much more easy. One thing I'd like to see get added is more mount options for a PVC, so we do a lot of things externally as well. So you might go to an NFS mount that Windows uses or a mainframe uses and then Kubernetes don't use. For Kubernetes to get there, you put every node in the export list. That means any pod can try to get that. There's nothing built in the Kubernetes that can stop this. So we do it with OPA policies, which is just another manual step, but it works. I'd like to see some other security that could be involved in there or more NFS4 options. Do Kerberos authentication with it rather than just IP. We take requests here. You don't have to just go directly to the maintainers. We'll just shout it out and we want these things to happen. But how are you simplifying your state flow application? Simplifying. Well, that was kind of the gist of like, what is the future of you that you're... One of the things that we struggled with since we did choose to offer 10 different database types to developers deployed as in Kubernetes, how you get credentials passed back to the application? How do you get a standard way to do a quiesce backup? We need kind of all the database vendors maybe to get together and come up with a community standard for what the operator looks like that manages a database. What your health endpoint looks like, what your status endpoint looks like so that everyone reports the same stuff. I shouldn't have to have 10 different sets of making sure a database is healthy for 10 different database types because they're all fundamentally databases. And there might be some specialization. I'd just like to see some agreement about everyone doing a database deployment in Kubernetes the same way. Okay, sorry. We did promise we will take questions. So does anyone have any? There's one there. Please wait for the microphone. Oh, first, okay. Real quick, I know we're time limited. As folks look from moving to Stafel applications to Geodistributed, do you have some top considerations or recommendations for Geodistributed applications Stafel like databases? Our model for geographic distribution is there'll be two instances of the database. The databases themselves will replicate with whatever their native engine is. We're looking for ways to maybe replicate using the storage underneath, especially when you scale, let the storage replicate and then let the database replicate and then it can reconcile itself still in early discussions. But for us, it's all about getting a deployment in two different clusters with a configuring database replication. So that's a multi-cluster behavior. I don't know if your question had to do with partly, you gotta worry about your round time network traffic. It should be like 10 or under roughly. Anything, at least for synchronous, right? Async's gonna be a different story, but you gotta make sure you have that. And bandwidth, there's a lot of stuff going. Yeah, money buys bandwidth, but latency is forever. So pay more attention to that. Yeah, we'll take last question. And after that, we can talk after. It's amazing some of the statistics that you pointed out, Scotty, about how you're offering so many different database technologies and you have a platform team that has built all the tools needed so the developers can straightaway deploy these. But there's also like a cost associated from a support angle, so to speak, to because these are not just application parts, these are actually databases. So you can't just have them go down all the time and if it does, you need a whole support model. So wanted to understand what kind of challenges you have seen over the years in helping build that SRE team or kind of solutions that you had in that aspect. There was one question I had and just one more. What kind of tooling did you choose in your entire solution in terms of CI-CD, package management, open source tooling and what kind of decisions that you made to choose those options, so to speak? I'll start at the end. So we choose open source where possible and a commercial supported solution were necessary. And necessary usually is not having an expertise. So the pattern is use a commercial solution, gain the knowledge and then switch to the open solution. Tooling and CI-CD pipelines is a whole other couple of sessions. But one of the things we did is we put our databases in separate physical clusters from the stateless part of the application stack. So that we can treat those physical clusters differently. They're provisioned with different hardware. There's disks in the database clusters. We have different operating procedures around them. Probably the biggest operational challenge is when debugging something, the DBAs have to put on their DBA hat and their SRE hat to understand if it's the database, is it Kubernetes, is it the network, is it the storage? And it makes them a broader discipline than just a DBA used to be in the heritage days. So it's both a training and an ambition challenge. Good question. We actually had that question on here, but I didn't have time to ask it, so I'm glad you did. So that was the last question. He's shutting us down. All the other sessions have gone way over. It's okay, we'll be here if you have any more questions. But I really want to, first of all, thank Guna for putting a lot of this content together and for getting a lot of this panel organized together. It was a pleasure hanging out with you. And I want to thank everybody here because these are not just users and users, but really contributors to the community.