 Cool. Hello everybody. Yes, welcome to a panel about Kubernetes. The idea was we actually got a lot of Kubernetes submissions of all kind of different people. So as I said, instead of using a lot of space in the sessions, because we want to have a lot of different diverse topics and tractors together with the advisory committee, we said, okay, let's do a panel instead. So we brought all the people that has submitted sessions or are using Kubernetes together and they are all sitting up front here. So I'm very happy to have all the people here. We're going to let them introduce themselves. But maybe just to shortly gauge the room so we understand a bit about the knowledge. Who of you has never touched Kubernetes but has heard about it? Okay. Like touched as in not running a cluster. Cool. Okay. Who runs a development cluster? Yeah, that's good. And who runs it fully in production? Cool. Okay. Okay. So we have a good mix. So for the panelists to know. Yes. So today we want to hear of different companies how they're using Kubernetes, why they're using it, how exactly they're using it, what is good, what was bad, and then we also want to hear a war story at the end. So we want to hear your biggest pain points or your biggest explosion or your biggest whatever you had, because I think we don't share these stories enough. So it is very new technology and so we want to hear also how it fits. But first let's do a short round of introduction and please also state how you're using Kubernetes. So how is it managed, is it not managed, all the different types of ways that you can run it. Yes, my own name is Michael. I'm the CTO of Amazio. I'm not going to talk too much today. I'm just going to do the moderation. So we're going to start with Kevin. Hi, I'm Kevin Bridges. I'm the founder and CTO of DDEV by DRUT. We use Kubernetes in production on Google Cloud. So it's not a managed service. We tend to offload as much of that as possible. But we do have several clusters running and a fair amount of production work happening there. I'm Tom Tugut. I'm a Systems Engineer at Amazio. I've been working with Michael. So based in New Zealand and we've been using Kubernetes for about two, almost three years. And we currently manage the GapCMS documentation in Australia where they run a few hundred websites all on a dedicated Kubernetes cluster. So yeah, we've had quite a bit of experience running with Kubernetes. My name is Boreen Lorton. I'm the CTO of Splunder, the company with the carrot. So we are using Kubernetes as pretty much everything from creating projects for developing environments all the way to production. So it's a very heavy focus on having a fully automated system that really enables a really good flow of work for developers. So very, very heavy focus on the developer experience. Hi, I'm Brad Jones. I'm a freelancer in Denver. I know if you're looking for somebody to consult with you, I'm available. But I'm mostly captive the last year to an agency in Denver called fruition, which is doing probably a transition that's been heard in many of you from sort of a weird mix of bare metal and hypervisors, and God knows what, to Kubernetes environment. So sort of a little bit maybe different from, so the other folks up here, we're not running a platform per se, like a commodity thing for sale. We're doing more of an internal product. But we've used this project to move everybody to a sort of unified Docker local development environment. We're running the same images locally into production. And we are using Google Cloud. So we're taking advantage of a managed public cloud product. But we're also using Rancher as well, quite extensively with GitLab and doing some interesting things with authentication. And proxy, so we can talk a little bit about that too. Cool, thank you. And yeah, so we want to jump right in. And the first question we want to know is like, how did you actually got started? Did you just woke up one day and said, I want to use everything Kubernetes now? Or like, what was the process? How did it came from A to OK? So I guess I'll start. Get the fortunate benefit of sitting next to my desk. I started basically using Kubernetes as a result of the Drupal work. So it started very early in Drupal and kind of rose with Drupal as Drupal gained more traction in the enterprise. So as the sites we worked on started getting larger and more complex, the needs to facilitate the transition to the operations team became more complicated and complex. As the sites grew in scope and scale and size, the server's farms that we were using started expanding and we needed a way to very quickly and easily manage those. Eventually started working at Acquia as a cloud systems engineer and saw a lot of things there that were very interesting to me, but decided to move on to more of an open source way of handling infrastructure because I think there's a lot of value in open source and communicating with how we're solving these problems. Kubernetes is the largest open source project in the world and it was an obvious choice as we started evaluating what to do. I think that the real challenge has always been timing it. We've known that we've wanted to use Kubernetes for a long period of time in production. We've known that we've wanted to use Docker for a long period of time in production but timing it so that these technologies are mature enough and stable enough to be able to use has always been an interesting part of it. For me, it's the bleeding edge. It's fun. There's a huge community behind it and I get a lot of value out of that. For me, it was actually three or four years ago and primarily we're using more traditional listening companies. I was working with them at the time and found Docker and Kubernetes or container orchestrators to work with continuous integration and things. So we'd use it for mainly splitting up ephemeral environments and testing on before we deliver that out to the production website. But back then, I mean, people would say, you're crazy with Docker and production. That's changed completely in the last couple of years and even Kubernetes has changed completely in the last few years. So it's become a lot more stable, a lot more solid and a lot more users. So the other thing that really interested me as well is the fact that it's an open source project. It's got a lot of backing from a lot of companies and moving really quickly and so it's really good minds working on this great product. It's just really cool to work on it every day. Yeah, so actually we are an agency so our main work is actually building stuff, not hosting stuff and for years we've been trying to find the right hosting partner that would just have the right process for us and we didn't actually find that. We tried with many and there was always something like it just didn't work exactly. So we have our own hosting infrastructure, like the traditional with virtual machines and so on, it actually works well. It's cost effective, it's stable, but what we saw is that pretty much the size of our operations team needed to scale linearly with the number of sites that we were hosting and this was not very effective and also it led to interruptions or delays. For example, you start a project, first thing you need to do is you need to ask someone to get a server and this is something that we wanted to go by that and really have a much more modern flow and we started actually using container so we wanted to use something that was simple. We thought that we had a very brief look at Kubernetes and we thought hey, this slows to complex so we'll go with the simpler version and after a few months with the simpler version we realized hey, there's actually a good reason why all these complicated things are there in Kubernetes and so we switched to using Kubernetes about a year and a half ago, and there's a certain learning curve but it's really a lot of fun then the more you learn about those dedicated things like new concepts and so on the more you realize why those are needed and that's actually been a very exciting learning process for our operations team and it's really exciting to also involve our lead developers, our architects into understanding what kind of possibilities we have so it's not only just Kubernetes as a replacement for another type of hosting but really as a new way of thinking about applications especially as we're moving away from just like hosting simple Drupal to hosting Drupal with different microservices with different technologies and seeing how they work together, how they're deployed sometimes independently, sometimes increasingly and so on so it's really about enabling new types of development processes and we still don't want to be a hosting company we provide hosting just because that's what was needed to get the processes that we want Yeah, to the question of sort of like how do you get into Kubernetes or work out into Kubernetes like Docker is the gateway drug to Kubernetes because Docker or virtualization with containers is really what we're talking about and Kubernetes has won the orchestration debate when I first got into doing containers and especially like deploying containers it wasn't entirely clear that Kubernetes was going to sort of win that like in retrospect now it looks like it was a done deal but I started using a product called Rancher and Rancher 1.0 was its own thing and it orchestrated containers on VMs that you provided it but it wasn't entirely clear that Kubernetes was going to be it but now we're at a place where it's very obvious that that's available and there's great managed products for all of the public clouds but I'd like to underline what Florian was saying it's not just about hosting and it's not just about running workloads for Docker or for Drupal I mean I think it's a way for Drupal to continue to get off of the island just like Drupal 8.0 got us off of not invented here to proudly found elsewhere a good example of this is that sometimes Drupal is not the right answer for the thing that you're doing or you need a complementary tool to play in with Drupal and a lot of times I think we get very narrow minded if we're like a Drupal shop and we do Drupal and we're going to solve the problem with that a great example of this is something that I'm sure a lot of clients have asked you for before we want to upload files and have like a directory of files for people to download there's no need to do that inside of Drupal but Kubernetes and Docker allows developers you don't even have to ask an ops person necessarily we solve that problem by deploying own cloud into a project and we proxy through a directory on that and it gives we let own cloud do we can run an image that we get from Docker Hub and it works great and clients love it so it gives us the ability to do that or run Redis or Memcache or whatever you don't need to call an ops person to install this thing for you anymore so I think this is a conversation more broadly about not just Drupal but like empowering developers to build cool stuff because the virtualization allows it to cool thank you so we heard about a bit of running your own cluster running using managed clusters maybe questions specifically to Florian you said you're using Google Cloud and have you ever considered running it your own and what was the decision point behind that like managing your own cluster well actually we started with Google Cloud because we wanted to just get up and running quickly so we said let's go with Google Cloud as best Kubernetes like managed Kubernetes supports also the most recent version and so on and then eventually we'll move to on-premise if needed and we realized actually we don't want to move on-premise because there's just so many things that are taken care of for you and yeah it's just a very good ecosystem and it used to be in the past that many clients wanted but like they didn't want to touch the cloud at all it was evil it was like everything and then we see that the cloud ecosystem has changed quite a bit so it's not like if you have if you want a European data center it's going to be in Dublin and that's the only option you have and for example so like most of our clients are in Finland and just having a data center in Finland which is cool by seawater which is a pretty cool thing unique data center design so it makes a fun story and certainly it makes it actually a selling argument rather than us having to convince look the cloud is okay and so yeah we've seen a major shift there and really the reason for going on-premise I see is like a political reason just policies things like that but I don't really see a technical reason not to the cloud cool and Kevin how is it for you I think you went from self-managed to now managed well we started off with managed Kubernetes essentially you need to make some really sound business decisions to understand that choice I think typically if you're a larger enterprise you have a lot of virtualization already in place you have a lot of compute resources available to you then potential and a large support staff to be able to or operation staff to be able to manage Kubernetes then you might want to look at managing it yourself outside of that I can't think of very many use cases where a managed Kubernetes instance makes a lot of sense it's very complicated you tend to be on the bleeding edge you have better access to more complicated authentication algorithms you get features maybe a little bit quicker but by and by the real benefit of not managing Kubernetes is that you get a lot of security updates you know there's a consistent stable base for you to build on and it allows you to focus on what your applications are doing on top of Kubernetes instead of being caught up in the tasks of managing Kubernetes Kubernetes is a complex system there is no denying that and taking it from you know A to Z essentially is very complicated for most organizations it's not something that we wanted to take on lightly we started off that way and learned very quickly that it's just much easier to do it managed cool thank you so Brad you mentioned before the gateway drug Docker if you could go back at the first time that you had like that you started using Docker and thinking about Kubernetes now with all the knowledge you have of running stuff what would be the single thing that you would have loved to know before you started that's a really good question but it's also the goalposts move so fast right so when I started using Docker probably this is very similar to a lot of you like there wasn't even a Docker compose locally couldn't provide you a volume right you had to use a container that you sort of shimmed to be a persistent volume right so if you're going to get into Kubernetes journey I think the big thing is understanding that you're going to have to if you've got a team that is not used to changing things right if your local development process hasn't changed in the last five years your local developers are going to have a really tough time adjusting to that change it's got to happen though so it will accelerate the pace so I think understanding the biggest thing for me with Kubernetes is understanding the model and what's in scope for something that you should still manage yourself right even on GKE right if you're managed cloud right you still have questions about should I use cloud SQL right every cloud provider has a managed SQL option we we elect to use it because running our own database right running you could cluster a database in Kubernetes it's something that we don't want to do right so you still have to make decisions about what do you want to get from your cloud provider what do you want to self manage and I think the big thing that I'm seeing a lot of teams now struggle with is and maybe Kevin has got some thoughts on this like the idea of your local development environment being like a Docker plus kind of solution like DDEV right how do you we've elected to run the same images locally in development that we then deploy into production some shops like to use sort of these composed images and then during your CI process build something that they deploy there's not a right or wrong answer there but it's worth thinking about because your developer teams have to then understand how they have to understand more about the production deployment workload than they might if it if you're just used to sort of turning your code over to Aquia and having it run on a black box I know you would so you know to address the question of how do you get container parity between your local development environments and what's happening on production there's a project out there that we're very very much focused on right now to accomplish that for us and it's called Cloud Native Build Packs Build Packs are something that you might recognize from Heroku back in the day it's a Cloud Native project that's basically being run by Pivotal and Heroku right now and Build Packs allow you to certain vent or solve a lot of the the shortcomings that you're going to find in things like Docker files so a very quick example of that is let's say in your local environment you're running a Debian stack you might want to put Git in there you might want to put GCC in there because you need to compile something you might want to put Node in there and you put all that in one container because it's easy however you don't want that to go up into production because that increases your it increases everything it increases your security vector it makes you more vulnerable in a lot of ways it increases the weight of your images Build Packs provide a mechanism to be able to take each of those layers that are put out by a Docker file make them more programatic and aware of each other so for example you can do a conditional step inside of a Build Pack layer that detects if there's a composer.json and if there's a composer.json in there it will add composer do a couple of different steps and allow you to delineate between what your build time environment is and what your run time environment is so essentially you can create different containers using the open container initiative not Docker specifically but the output is containers to run local development environments and run production level containers with a fair amount of parity they're not exactly the same so it's a good purpose and they give you that assurance that what you're working with locally is going to work in productions and follow up question to that how much is it usable like on which state of maturity would you say it is we've reached full production parity with what we've typically done with Docker using Build Packs we have some example open repositories it did github.com I believe it's Build Packs is the repo name but we're using that in production now I'm just going to continue and say that not only just from level development to production but also through UCI environments so you get that full congruency to be very confident when you're actually going to do that deployment that way your deployment is fully regression tested and ready to go it's also really helpful for security the classic example that I use is if you build a Docker file you might go through and include your base image that might be Debian you might add PHP on top of that and then you do everything that you need for your application which might be Drupal if that Debian image ends up having a security release that happens you have to rely on that maintainer to go through and do that security update for you and then you have to rebuild all of your Docker files if that doesn't happen and you want to deploy that security update then you have to essentially fork the entire stack pull that security update in and rebuild all of your images you're taking on a lot more technical data, a lot more work with something like Build Packs you have the ability to go through and rebuild the entire stack by just focusing on one layer of it so worst case scenario you go through and need to pull that security update to Debian yourself then none of your other code needs to change in a very manageable way and it's just a fascinating technology Cool, I think that also brings us to another interesting point in Kubernetes is the speed of development Tom, how is it for you to like, how do you keep up with the speed of new things to come up and to use them and these things? Yeah, I mean it has been a crazy ride for sure I mean, Kubernetes has been moving at an incredible pace I think there are about three monthly release cycles I need the contents of Kubernetes upgrade Who wants to take a picture? Well, it's recorded anyway, so if you have it forever So, yeah, everything can be definitely challenging but it's also exciting as well There's a big, big community and that's one of the most important things so there's a lot of people working on these projects The Cloud Native Foundation is very impressive in terms of its scope and size and members and support and so, yeah, it's a pretty crazy ride but I mean, it's exciting as well to see that over the last couple of years what we've gone from to where we are now and things like, you know, build packs, operating there's a whole lot of new things coming out all the time it's, yeah, it's been a great ride and I think it is just getting easier and easier and especially with people like a lot of you guys actively working within the Drupal community that's also really good for the Drupal developers as well Who has one resource or say that you would say follow that or read that every day to know to keep up to date with communities or what are your personal habits of staying up to date? Actually, I think on one hand there's a lot of things happening and at the same time I see that there's a lot of effort that is being put into making sure that nothing breaks so you can always update at least for Kubernetes itself there are certain things like Helm charts, for example that's a whole different story there's the possibility to just stay at least I feel like you don't need to read everything that comes up on this day or this week you can leave your front-job beta one API definition and it's going to work fine so I don't feel like it's I'm not feeling stressed about having to just always keep up the way that you would, for example, get in the JavaScript ecosystem so this is something that I see that yes there's a lot of things happening I'm pretty excited to see the new stuff that comes in but I feel pretty safe about the stuff that's already in actually staying there and being quite stable On that point I think there's the point of the question I think points to the pace of development with Kubernetes is so rapid that if you want to do the latest hotness all the time and you want to like, I can't imagine right now telling my entire team we're going to convert the build packs but it's like, I just learned about it and Kevin's really into it it might be a really good fit but especially if you're taking, I've been doing a lot of change management with helping to explain these new processes to developers and I think Drupal, we've gotten through this fog where there was a lot of Anjana about this is not a product for the copyist anymore this is a very enterprise-y product but we still have a lot of people like, I'm self-taught a lot of us are self-taught you don't need a CS degree to do this so I think flattening out some of that change and understanding that if you have a Dockerfile that builds today yeah, you might need to make sure that you keep up with updates and that your upstream works but you don't have to be constantly filling every bit but one thing that I liked about what Kubernetes brings to the table and maybe Docker word broadly in containers is that I write more shell scripts today and everything old is new again you have to understand how the internet works you have to understand how routing works you will work with way more reverse proxies I write more nginxconf files now than I ever did when I was using more managed products from upstream and that's a good thing that's a really good thing that your developers can understand how all of that works so I think if you take it as an opportunity we don't have to be learning about the new hotness let's do a lunch and learn about writing really great bash scripts because every Docker container I run starts up by doing a bunch of bootstrapping and that's a good thing we don't have to be constantly chasing our tail with the newest hotness I mean there's been a recent sort of mantra in the community as well which is just make Kubernetes boring but I think that's been awesome because it actually is just far more stable in the early days this thing was flying and there was huge changes even in point release but now it's about making it boring making it easy, making it stable so making sure that what you run now there's going to be running in the previous time there's going to be new updates and for me I think a lot of that answer relates in similar ways to today as close to core as possible integrate with the community Kubernetes has a very strong community go to KubeCon it's a fascinating experience it's a very tight community you have the ability to network and talk to people to understand what's going on pay attention to the projects that the CloudNative Foundation is shepherding the ones that they're paying attention to the ones that are incubating those are the ones that you should be focusing on if you find yourself doing something like fighting an API or you have some very specific thing that you're trying to accomplish because you're used to doing it a specific way you might want to pause on that development effort for a little while and rethink the way that you're approaching it by exploring what the community is doing in the Drupal Slack we have a Kubernetes chat room encourage you to join that and start talking with each other there we have a regular meeting that we try to do where we try to have this conversation with the Drupal community so get involved, speak up, learn from each other that's the best way to do it and try to stay as close as possible to core because varying away from that is going to lead you in a scenario where you might be drowned in technical data essentially so don't thank Kower also and to answer your question one place to stay up to date one thing that would really recommend if you're using Kubernetes and a few things on top for example we use a traffic for Ingress just subscribe to release notes for these projects because each of these projects has a really good process like new version comes out there's a release candidate it tells you what changed some bug fixes that affect you but also very often it's new functionality that you think oh this is cool, I see where this could be used in our stack and it doesn't come like an endless flow you don't need to be watching each commit individually but those release notes are really really helpful and this is something that I get a lot of my information that I actually need I get it from there cool, thank you so let's go into the success stories world so we want to hear what was like in the last couple years something that you looked at and was really like a point there's like wow I didn't imagine that this is possible or something that worked really really well but yeah what is something that you can share that's really well for you I'm obviously pretty excited about build packs those have been working very well for us I'm also very excited about the operator controller patterns that have come out in Kubernetes and the native way of communicating with the Kube API being able to extract basically give interfaces on top of the Kube API that we can expose to customers that are really fascinating so understanding our back understanding the security controls essentially has given us the ability to rapidly iterate so if we want to bring Ruby onto the platform Java onto the platform or any other CMS onto the platform we have the ability and a pattern to do that very safely and very effectively and that pattern all of these patterns combined the better we are, the better we are for everybody so I think that's going to be fantastic Yeah, so from my side I think I see Kubernetes as a really great automation platform it's the ultimate automation platform to declare it into what you want and one of the best success stories for us is actually a big migration project for Kube CMS that we did last year and just through over a weekend we basically were able to migrate there was 102 scientists that were run by that just fully automated just basically kicked it off ran took a whole weekend mainly using the transfer time but to be able to do that and replicate that and during that process we're actually constantly testing and deploying and testing that migration process and kick it off and have that run through smoothly was pretty cool really excites me as well is the operators framework I think that's just a really good model to be able to deploy objects and extend Kubernetes some things I've been looking into is the ability to have a Drupal custom resource definition so you actually can define your Drupal site as a single Kubernetes object deploy that to a cluster and the cluster sort of works so something that I'm keen to talk about today about Kubernetes I guess my success story is a bit on a different level I guess in general like having a setup where I see that our developers created projects and it's running and I had like I had absolutely no involvement our operations team also had absolutely no involvement and just happened and I think the best one we had a bit of an emergency situation last week and we needed a website very, very quickly and so one of our developers actually created a new project from scratch and so from creating the first code to being in production with the domain and SSL certificates and being production ready 13 minutes that wasn't expected like that I hope these kind of situations don't come up every week but actually being able to do something and that I think it's pretty awesome and if you look at any kind of like the research that's done on the DevOps movement and the kind of key metrics the time it takes you from writing code to having that code deployed in production that's one of the leading indicators of productivity and I think this is really pretty cool to reach these kind of numbers I would agree there's a lot of really interesting cool stuff coming down the pipe but the biggest successes are actually rather work a day for us very similar kind of experience where teams when I don't hear from teams that's really a good thing in the sense that we similarly use traffic which is maybe that and an advantage but the ability for people to deploy a workload especially for staging it's super important to be able to like have TLS certificates for staging and test in an environment that is on parity we don't have the same SLA for our staging environment but we have the same infrastructure and so just being able to go from a world where we didn't offer TLS environments because we had to buy certificates like people are still buying certificates let's incorrect we'll give them to you for free but you have to automate that so automating out a lot of the pain for people and even being able to do things like when a new database goes into our Cloud SQL we have been able to use Kubernetes, it's not a Kubernetes feature but we've been able to use a lot of that ability to deploy services, we have a little service that runs on a cron job and uses Kubernetes primitives to back up databases so we don't forget to put databases into our backup regime now because Kubernetes something we haven't talked too much about is a great gateway drug for cloud native too Kubernetes is just a one way to get you into a cloud native environment and so we now use object storage a lot more than we used to we now use whole disk snapshotting, right, we use Valero for backing up disks and we get disk snapshots right out of the box every persistent volume we have in Kubernetes just gets backed up so that's been a huge, automation has been a huge factor for us, I don't have to set any of that up so I don't have to forget to set any of it up so now when we need it back up we know it's there cool okay so last questions for each of you now we want to go into the dark realm and so we want to hear about what was the worst that happened so far, deleted clusters lost backups production sites now for multiple days who wants to start so we've actually been in production for a couple of weeks now so no major per stories in all the sites that we have so far are relatively small but one thing that's actually a simple way to break your cluster just especially in DevLockman sets a bit too much resource allocation and you think why would you get things to be slow by allocating more resource and the problem is that we had those resource allocations for the PHP containers but not for the underlying services that guarantee that the cluster is running so we ended up with those PHP containers reserving everything there was absolutely no load on the server but it was reserving everything and then the things that actually needed to run for the cluster to be running they weren't running and so we pretty much like essentially locked down our entire cluster just by allocating too much so this kind of thing it's very easy to do I've got two horror stories I'll share very briefly one is even sort of pre Kubernetes sort of a trope that the difficult problems in cloud native and especially Kubernetes is persistence it's really every hello world tutorial in Kubernetes land is like run a container scale it up to 50 and it's load balanced look it hits all these different nodes and it's like well yeah but I need to persist my files have it and it's like congratulations you're a Kubernetes actually so that's very difficult or very early on even pre Kubernetes I had just launched a beautiful new multi site Drupal site it was running in Docker running on the client's infrastructure because they insisted that it be on-prem and I persisted all of their persistent file storage in a directory that would not survive restart so the first day that that node restarted guess where all of their files went right into the round file didn't make that mistake again but we've all made those mistakes for sure the other piece that I will just mention you will have to if you're gonna go on this journey and even if you do manage Kubernetes a big challenge is that you have to manage more of your infrastructure yourself right so a great example of this is on Ingress Google's recommended way to do Ingress is to buy their HTTPS load balancer and it's 30 bucks a month per host right you can put multiple certificates on it but you can't really you know scale it out to multiple clients so you have to buy a load layer 4 load balancer and then do a lot of your Ingress yourself which is not a problem but like I'm not a TLS configuration expert and so I have gotten a lot of heat from a CEO who's like my wife runs Windows 10 and IE 11 or whatever it is I don't run IE right and she gets a certificate error right because you have to go in and manage a lot of that configuration yourself you have to really be on top of you know what ciphers did I select and that kind of thing and we run into problems like that because clients will you know client that sells supplements to old people and they use really old Android phones and we were using optional client TLS mutual authentication for some features and it broke all of that on Android it just doesn't work on Android and I didn't know for like three months and our clients telling us you know we lost hundreds of thousands of dollars who knows if that's true but you know you will run into that because you don't get to point the finger at anybody else anymore it's all you know yeah for us I mean we've been lucky but it's definitely some days but the great thing about it is the self healing nature of Kubernetes really so you can actually lose that whole host and recover and sell but yeah one of the one of the worst case well the interesting ones that we ran into was just having Redis deployed in like a high availability setup and we thought you know we really just tested fine we were testing this on some availability zone so latency wasn't so bad but as soon as we can switch that on to be a highly available so spread across three availability zones in a region that latency just caused massive issues you know sites were just running like just so slow and it was very difficult to work it out because you couldn't it was hard to replicate because if the internet or the web pod would set it on the same availability zone as Redis there was no problem but then if it wasn't there was a problem so it was one of those things there's lessons that you learn that sometimes you know Redis or Memcache is always a good idea with Drupal but sometimes in a highly available setup they can be damaging for us it's you know the very first large scale deployment we did involve right around 200 or 300 sites so we had gone through and it was our initial implementation which we tested load testing on everything was fine and we were doing very static load testing across a wide range of things but you know to Florida we didn't quite have the resource quota set properly, didn't have CPU limits set properly so once we got involved with the noisy neighbor problem where one site was doing much more than everything else around it suddenly the entire system just ground to a halt so you know I can't stress enough take a look at your resource limits it's resource quotas take a look at your CPU limits make sure that you're allocating properly for your machine you can't over allocate it's a difficult thing you need to make sure that you understand what's happening there cool alright that's it thanks for the stories thanks for being on the panel that was it I think if you have more questions please reach out as we heard there is the Kubernetes Slack channel and talk to the blog and there's a blog what's happening we have a Kubernetes buff from 5.30 to 6.00 this afternoon so if you'd like to come by and talk about Kubernetes we'd love to have you