 Can you guys hear us in the back? Yeah? OK, cool. Awesome. Ladies and gentlemen, welcome to the serverless Drupal room. If you brought any servers with you, please place them outside the door. This is the serverless room. This presentation is going to be very conversational. Myself and Salim, and we'll do our introductions here shortly, we talked about this. We did this for real. And we went back and forth, and I'm like, I don't like that answer. Did you mean this? You didn't mean that? What did you mean? So I interrogated him. I beat him up pretty hard. And we got a lot of stuff, a lot of information going on in here. Yeah, we're good there. What does any of that have to do with why you're here and what's going on? Quite a bit, and we're going to find out why. Let's find out together. Starting with me. That's right. I'm the first one. So I've been in the Drupal community in my 15th year. Levenic came in at Drupal 5 and thought that was so powerful. It was at the time, but we're doing much better these days. From the Midwest, Indiana, happily married four kids, 11 at all. Synaptic Blue is the company I started. It's just a solopreneur, me. So you probably haven't heard of it. And all these pictures real quick. I did have a meet and greet with Red Al Yankovic. I do play guitar. That's me running a spotlight. I'm also a journeyman, stagehand, if anybody cares. So that's kind of fun. And I used to be younger. Something else you can learn from these pictures. I used to have hair. But that is me. OK, if it was just Doug, it'd be a comedy hour. Yay. Yeah, so I'm going off a scrape. Are you saying it's too soft? OK. Put it in the mic. So I have to tell you, it's ironic, but I don't care that much about serverless. What I care about are the benefits of serverless. And I have to tell you, one of the main benefits of serverless is that we were able to get up to 90% savings on hosting from using serverless for some of the clients. So this session is more than anything else. It's a case study on how we were able to take a $1 million hosting bill down to 127K. And it was not by rewriting Drupal or re-engineering anything. It was by taking a reference architecture, AWS reference architecture, and implementing that for the client. And this is something that anybody here can deploy and use. So we're going to talk about that in this session. So I'm Salim Lakhani. And for the last six years, I've been automating AWS with different clients. And DevPanel, the company is the point and click control panel that automates Drupal on AWS and automates Drupal deployments on AWS. And just to give you an example, one of the things we did was recently was we did, in 17 minutes, we did a deployment, which was taking one of our clients. I think it took him four months and three people to set up. So they signed the same day we did that. But today, we're not going to talk about DevPanel. We're going to talk about how you can get started with serverless. And we'll walk you through the pre-built reference architecture. And hopefully, you'll be able to save a ton of money yourself. And I realize some people here might be allergic to AWS, but the patterns and the concepts we're going to talk about over here will apply to all platforms. So we'll keep it at that level. All right, so the next slide is all about DevPanel. So what the heck is serverless? I hope we all have the same answer, but maybe not. If you don't see a server, then it must be serverless. There is no server here. This is a robot. And think about it. It's efficient. It scales. The cost is reasonable. And customer satisfaction is much higher that way. You too can experience serverless today downtown at Top Burmese in Portland. They have robots bringing your serverless bringing your food. So that's pretty sweet. Take a screenshot. I ate there in February. It's very tasty. OK, so if you go and mention my name, they'll say who? So when you talk about serverless, what is the first thing you think of? Lambda, right? What is the first thing you guys think of, right? Lambda? Yeah, OK. So that's mainly what most of the people think of. But to build, to actually run Drupal on Lambda would take massive re-engineering. And the benefits of running it on Lambda would be the same as, I think we can get the same benefits of running it on Lambda from using other technologies that are already available on Drupal. And if we run it using some of the other serverless technologies, you'd be able to run Drupal without re-engineering it. You'd be able to use all the modules that you like and so on. But for that, there's already a pattern that's already built by AWS. And that pattern looks like this. So I'm immediately not happy. Subnet, Sorority, Fargate, I'm lost here. I don't have a CompSide degree. And I'm not AWS certified. So I'm kind of done at this point. I'm ready to go do something else. Yeah, so it might, I know it looks complicated, but it's not. So I was going to say, don't leave. We'll walk you through this. It's not that complicated. And it's a pre-built infrastructure that you can actually deploy using scripts. The repo is out there. And you can actually download the repo and run the scripts and it'll deploy it for you. This is what they call a reference architecture. And it was built by AWS DevOps experts. And you don't need to know Kubernetes. You don't need to know DevOps. You don't need to have a degree in AWS. This is something you can actually deploy yourself, download it and deploy yourself. So it follows all the best practices. So it's built on technologies like this, these kind of technologies. This is AWS Fargate, which is their compute, serverless compute engine, ECS, which is elastic container service, EKS, their elastic Kubernetes service, Aurora serverless database and the EFS, which is their elastic file system. There's shared file system. So you can use these technologies to actually build serverless Drupal infrastructure. So the serverless infrastructure diagram that you saw uses some of these, most of these technologies, I should say. And with these, when you deploy Drupal, with these you're essentially deploying containers and the platform takes care of all the scaling and the platform takes care of all the infrastructure for you. So all you're doing is deploying containers. And the benefits of this, as we've conversed back and forth for a long time, was no machines to maintain, which I was excited about. Platform provides computing resources as needed, which is related to, you can pay only for resources you're using, also exciting. Scalable on demand, gotta be. Flexibility and speed of deployments. I'm a developer, I like that. Extremely reliable and fully managed services. So, winning all the way around. Do you have any examples or case studies of this? Yes. Imagine that. So this is a project, this is a proof of concept project we had done for Voice of America last year. And I use this because it's a, it was the largest savings. I mean, we'll go into clients, we'll save them 10,000, 20,000, 50,000, 60,000. But this was the largest savings that we were able to demonstrate. So, we'll use this and I'll show you some of the numbers here. And this was really big, so. This is, how big is the project? Two billion hits and 50, what's that? Did you develop everything yourself or how much was out of the box or how did that go? So, we had to do some development with them. We worked with AWS on this. So, this is something that was bigger than what we had done before. So, we worked with AWS on the architecture, on everything, but we brought to the table, we brought the whole DevTest Live infrastructure from our control panel. But we had to work with them to do tag-based deployments, to do, what else were we doing? BlueBrain, the ability to roll back deployments and things like that. And when we did that, it was available to everyone. But this is something that was really huge. It was two billion hits per month. And that was massive. That was just massive. They had 50 total websites in different languages. And they had four main sites on Drupal. Their main site was getting about a billion hits. Oh, yeah, I think close to a billion hits. Or I think the, yeah. So, we were actually planning on scaling this out to, we tested the infrastructure out to handle two billion hits. And they had 46 other legacy sites that they were trying to move, they were planning on moving to Drupal. For the four sites that were on Drupal, they were paying a million dollars in hosting. And they had a single code base and they wanted to deploy all the sites from that single code base. That was kind of the background on the project. So, these are some big numbers. So, we had, for this project, and like I said, we worked with AWS. We had, we were able to get them down to about 127, 128 annual hosting bill. And it was with equal traffic and load. And we used, for this, we had used on-demand servers or on-demand infrastructure and on reserve, or reserved infrastructure for pricing. So, that was kind of the basic background on the project. That was actually over 87%, it said 80%, but that's, either one is great. Does that include the dev panel portion of fees or your stuff? Right, so that didn't include the dev panel fees, that was just the AWS fees. Our platform fees are based on the savings we create for our clients. So, the more savings we create for them, the more, you know, so we're incentivized to create more savings. So, we take a percentage of the savings. So, that's a good question, so. So, some people in this room are probably thinking they're gonna go back to work next week, talk to the IT director or CTO of somebody and say, hey, listen to me. I can, I can, it's really impacting and make some crazy savings here. You know, they're gonna be excited, perhaps, hopefully, by this, but they're gonna get a question back to them like, but does it perform? Right, so that was one of the things, the questions they had as well. So, we had to do load testing and this was, you know, it passed all the load testing. We did both cold testing and under load, we did testing for spikes and so we stress tested it and we targeted the origin, which is passing the caching and going back to the origin. And for performance testing, we used web page test and lighthouse for doing performance as tools and we got comparable performance with the production. We got a little better than their production performance but, you know, that could have been tweaked, I guess, but, so. And then, the whole thing was done using, the entire thing was done using auto scaling. So, we monitored the clusters, the clusters went up and down, all the infrastructure auto-scaled correctly and, you know, it gracefully went handled all this. Auto-scaling is nothing new. People might be falling asleep. Is this special auto-scaling or just auto-scaling, auto-scaling? Is this special? No, right. So, if you've worked with the cloud infrastructure and if you've done auto-scaling, then this is just normal auto-scaling, so. Yeah, but not easy on this game. No fluff, no fluff. So, we can go into some questions that we get. This is, these are some of the questions we get. No, most of the time when we talk to clients, like what is auto-scaling, how it works? So, in auto-scaling, when you have like fixed capacity and your traffic goes beyond that capacity, your site will crash. And if you're buying capacity at peak page views or you're paying for your peak traffic for holiday season or whatever it is, right? The rest of that capacity is essentially wasted and you're paying for it all the time. So, auto-scaling essentially adjusts for that and you're having the systems essentially adjust the traffic for you and follows your traffic up and down and the infrastructure will adjust according to your traffic. So yeah, even getting started in this, you know, the diagram I'm not certified, et cetera. I wasn't sure where the front door was to this building. So, to get started, this is where you would, this is something that you can go to. This is the infrastructure. Here's the repo you can download and essentially follow the readme. So the first thing I have to tell you is this is infrastructure as code and it's built around AWS's cloud formation template. And this is very similar to Terraform, Terraform if you've heard of Terraform. So if you want to run cloud formation, you can essentially Google how to run cloud formation. It gives you the instructions on this repo. But essentially just follow the readme and you will be able to deploy this infrastructure. This URL is different and this URL actually walks you through step-by-step on what this diagram does and how it's set up. Are the slides on our session node yet? Right. So I've put a link to all the slides on the session node so hopefully you'll be able to get to that. On the DrupalCon website. Right. We shouldn't assume people know what a node is. So in this architecture, you can see that we're using Fargate to run these containers. You've got the three different regions. So you've got Drupal containers running in three different regions and that gives you the load balancing and that gives you the redundancy for running Drupal. It gives you the multiple zone so if any of the containers goes down, you've still got the website up and running and you've got the serverless Aurora database cluster running in the background. That automatically scales. You don't have to manage the cluster. You don't have to set it up or anything. You've got the EFS which is serving your code and it's just the elastic file system that's serving your code and your files and you've got the load balancer upfront which is distributing the load to all the three different regions. So all the three different availability zones. So this is all running in one region and that's what this diagram illustrates. And the nice thing about this reference architecture is that it can be extended. So if you're looking to run things like Memcache or Redis or if you wanna add S3 buckets to it, it's very easy to extend this reference architecture. And this is one of the things that we had done on the project as well. So we had taken this reference architecture. We had done something very similar to this and we had extended this. This is when we were working with AWS. So we took something like this and then we extended it with Solar and with CloudFront and the WAF. The WAF is the firewall. So that protects you from hacking attempts and things like that. The CloudFront also comes with DDoS protection. So it'll protect the website from DDoS. Basic DDoS protection and then they also sell advanced DDoS protection. Question there. So standing up the initial reference architecture is a bit turnkey in that regard. But adding a Memcache or adding a Redis or Apache Solar or whatever, that's gotta be a little more complicated and just turnkey. Right. So, sorry. I stepped away from it. So yeah, so the question was yeah, if you wanna add the additional components, you have to work with either AWS or somebody that knows AWS. For example, but they're not that hard to deploy. For example, the WAF, there's a pre-built recipe to deploy WAF. You can essentially walk through it on AWS. So if you Google for AWS WAF implementation, it'll give you, you know, it takes half an hour or a couple hours maybe to deploy it and it's a pre-built recipe again. That you can use to deploy that. How else can we go through this? Can Drupal scale to zero? And what about small sites? If your site's small, are we in the conversation? Right, okay. So again, those are all good questions and we get those a lot. So there's different patterns of going serverless. The pattern we saw is well suited for, this is the multi AZ pattern. The multi AZ pattern is you're running Drupal and multi and different like data centers within a region. And it's a reference architecture. It's good for, you know, it runs in one region and it's good for most Drupal sites. But if you're running large mission critical sites, there's also multi-region deployments which is beyond the scope of this presentation but it uses some of the newer technologies that are out with AWS and we're building those out for some clients now. So you can actually run, I don't know if anybody knows but AWS, some of the AWS regions went down last year. Has anybody heard of that? Yeah. So that's when we got, that's when we started getting requests for doing multi-regional deployments. So now you can run Drupal and other applications in multi regions that way. And then for static sites, there's another pattern for running static sites. And this is something you can actually, you can actually put out some links here if you wanna check this out. It's a project for AWS, I mean for Drupal called Tome, which I believe will work with, it was set up for Drupal 8, I believe will work with Drupal 9 as well. And if you're hosting static sites on AWS, you can, it'll cost you $1 to $3 a month or I would say $1 to $5. There's also another company out of, another company that actually does converts your Drupal sites into static sites. It's called Quant, they're called Quant CDN. So check out Quant CDN and we're partnering with them so that we can do DevTest live and then convert the live sites into static sites at DevPanel. But Quant CDN can take your site wherever it's hosted and turn it into a static site and then just host that. And that will allow you to turn off your production site and just host and serve out the static site. And when you hear the first static site, you can even think of, and many people do think of corporate blogs and things where it's not a lot of interaction and what not going on. So if you're gonna create a blog, even once a day, you can just shove it out to a static environment and not have any server, that'd be really serverless. With static sites, your sites are amazingly fast and they become hack-proof. So that's really good if you don't, if you don't wanna be on the edge of your seed when there's a patch that comes out and you wanna run and patch it. So if you convert it to static site, then you have some time to go patch your sites. Online IDEs, there didn't exist another do and it's kind of all the rage, what's up with that? Yeah, so that's another good question is that if you're using serverless, can you use online IDEs with that, right? So AWS has a built-in IDE called Cloud9 and it gives you all the things like code autocomplete, it'll give you multiple panels and code hinting and things like that. There's a link here, you can go check it out. It supports over 40 languages, it's got integrated debugger, your terminal and editing history and chat so you can actually do collaboration with different developers. File revision history, keyboard shortcuts and all the bells and whistles, so. I'm curious, has anybody tried AWS Cloud9? Anybody? No, yeah. Hands up how you like it, middle you don't, it's okay, wish you watch it. Yeah. Oh, good, okay. The endorsement, that's great. One endorsement, there you go. So CICD, you gotta have it, are we boxing ourselves into some limitations or anything here? Yeah, that's another good question. Can we do CICD? So CICD is, you can. What is CICD? Yeah, so CICD is continuous integration, continuous deployment, so you can actually do your, we're building containers from your, if you're running on local and you want to, you build your containers and deploy those, you can actually just integrate with GitHub, Bitbucket, GitLab, you can bring in Jenkins or CircleCI and just build out containers and deploy those, so those work just fine. So if you have a flow you like, you can reproduce it rather seamlessly, it sounds like. Yeah, so you can set up your own pipelines and build your containers and deploy them, so. Take it. Yeah, all these questions are mine, I really interrogated the snout out of this guy and got all these answers. My devs use local dev environments, some Ddev, Lando, Dockful, et cetera. I have MaxPCs and Linux. Can we get started on day one? Yeah, so again, this is, I don't know if you've ever, like what we do is when we have, when we set up these environments, everyone essentially syncs up to a repo and then we deploy that repo. So again, it's the same CICD deployment pipelines, but yeah, everyone has their own local development environments and then they, we just deploy using CICD from that. So no limitations there. Multiple remotes, dev test and prod or dev test and live, you got to have it, but sometimes you just need a couple extras to do some avant-garde crazy stuff. Is that moving heaven and earth or what? Yeah, and it's, I want to say yes on that as well. It requires some setup, but yeah, you can, if you're deploying, if you're setting up CICD, you can deploy containers, you can deploy different branches, you can deploy feature branches, you can deploy any branch you want and you're essentially setting up different container, you're deploying different containers to your environments and that's, you know, you've got, you've got in the cloud, you've got, instead of having a server, you're just deploying any number of containers, so. No limitations there. And I got to this point in the show, our discussion over for many months. So it's just long-term, I'm gonna do this on day one and am I good for six months, 12 months, 18 months? Is something gonna change? I'm gonna outgrow this, I gotta change again. What is the story on that? Yeah, so I guess you can outgrow it if, you know, if you're gonna outgrow Amazon or, but yeah, you know, it scales, it automatically scales, so I don't think there's a limit to outgrowing it. I think the better question is, is it what you're doing today, are you gonna outgrow that or is that sustainable? So those are the questions you, I think that's the better question, so you outgrow what you're doing today. I remember that answer, I felt put in my place. What are you doing now? Okay, so we gotta keep up with the times, it's not just monolithic Drupal sites, it's decoupled and progressively decoupled sites, it's also not just Drupal, decoupled WordPress sites are being deployed on these things, any limitations there? So this is good, this requires some work, but if you deploy decoupled or progressively decoupled sites, then it requires some work, but you've got two classes of containers and you've got Drupal that you can scale separately and you've got the front end that you can scale separately, so yeah, you can do that and you can do separate scaling for that as well, so. Sweet. Server logs, pipeline logs, et cetera, that's been a boner contention in some of my dev operations. Do we have access to everything we want? Yeah, anybody use Splunk? Yeah, so it's essentially we take the logs and we put it out to another log aggregator, we'll use CloudFormation, I mean, you will use CloudWatch for that or we use Splunk or we'll use some other log aggregation. Sweet. There's OpenSearch as well, so that's another new thing that we use. So there are some people doing this already, is that considered serverless? Pantheon, not platform of this age, amazing. Is that serverless? Yeah. Is that serverless? Yeah, yeah, yeah, so it's all serverless, it's, most of them are using serverless. Acre is using, yeah, Acre is using a mix of it, so I think they're moving more and more to serverless, but most of the other platforms are using serverless. What you're paying for, I think is, most of them charge you at the peak, plus you pay a margin. I mean, I'm not knocking it down, they have big infrastructure and stuff. So they're charging you on the traditional model, which is you're paying at one capacity level, but on the back end, it's all serverless, their capacity, their infrastructure moves up and down, depending on the load. We did good, so we do have time for questions. Yeah. 20 minutes, right? Yeah. So that was our conversation over many months and I hope it looked a little natural, but that's what it was. Questions about serverless? You can reach out to us and then there's questions here and there's questions on the Slack channel as well, so what questions do you guys have? If you're just in a boff, just tweet out, mention my name, Matt Dugvan, and we'll join you. We should probably put the wireless over here and invite individuals to approach the stage. Do not bite. Or pass it around. You know what? Sure. Pretty basic question, but being in healthcare, right? We have difficulties getting legal contracts with AWS, so you talked a lot about AWS, so have you tried to do serverless on Azure? Yeah. And what's your experiences there as far as implementation and timeline? Yeah, so we're working on porting over our infrastructure and our implementations to Azure. We get that a lot, especially with the government clients and stuff, so we haven't done that yet. We're working toward it. If you're interested, you can contact us and we'll work with you, so. Sure. We use a lot of pentane. Yeah. Basically it does all of these things really fast and throughout need to maintain a container and so on. It's so much more beneficial. Is this beneficial over Pantheon? I mean, if you want to explore saving cost over Pantheon, if you're, then this is something to explore, but if you're happy with it, then I would just stay with Pantheon. Like there's nothing wrong with Pantheon at all. And in the case study, they do a very good job. Yeah, they do fantastic. In the case study, we had an organization that was doing their own server in-house. It was very costly. So a lot of organizations just won't even touch something on a platform, you know. So they're gonna roll their own anyhow, so this becomes. And a lot of times we'll go into organizations where they want to actually run their own, they have a mandate for running stuff on AWS. So we will help them. Like I was saying, there was one organization, government organization we were working with recently that had spent four months and three people trying to do a Drupal deployment. And they actually timed us when we went in there. And it was 17 minutes. And we weren't timing ourselves, they timed us. And I'm like, oh, you know what I'm saying? It sounds like really the question here, correct me if I'm wrong, what you're trying to get at is, if you have the stomach for slash need to cut down on paying somebody else's margin slash willingness to roll your own, do this. Right. If you need to offload your complexity to somebody who promises to give you a button to push and you can deploy, you're gonna pay a premium for it. Is that what you're saying? Pretty much, yeah. It's the state of Drupal hosting today, right? So, okay, I just wanted to clarify that. More technical fun question. Your architecture.gram is nice and complicated with many overlapping boxes. And it's not my diagram. I just wanna, yeah, I just wanna, like if I were to draw it, it'd be more complicated. So, yeah, I just wanna point out. I've drawn many complicated. Yeah, I get really excited and get carried away. So this is really good. To grind the favorite acts of mine, some of that is based on the fact that you still need bind-mounted storage, right? If you really wanted to go multi-region, for instance, I don't think you can do that on elastic box storage multi-region, right? So at that point, you have to get more complicated. But the storage becomes the limiting factor? No, no, there's a multi-region EFS now. So that's a new thing. And you can use that as well, so, yeah. But yeah, it's brand new. So it's something that was brought up with the region failure and that's something that AWS has been working on. So, yeah, but you're right. The storage was the thing that was limiting factor in going multi-region. And with multi-region EFS, that becomes possible because there's already multi-region aurora and in a global aurora database and there's the global accelerator. So using those, we're able to set up the multi-region infrastructure with it. Any questions over here? Yeah. Hey, Slim. Hey. My understanding about Fargate using that over ECS is that it's cheaper, but they might, the Amazon kind of reserves the right to shut down your instances. Is that a risk in this architecture? So what we do is we run multiple containers and when they shut down, it's designed for failure. Right. So, and underlying, we actually use, in most instances, unless the clients object to it, when we do our implementation, we actually use spot instances, too, because in a spot instance, you know, you're paying $70 for on-demand. In spot instance, you're paying $7 a month. Right. So we use a bunch of spot instances and instead of running one container on an on-demand, we run two containers on two spot instances, or we'll run multiple containers on multiple spot instances. And if they go away, it's designed for it to go away and you end up saving a ton of money that way and it's built for redundancy and high availability comes with it. And you get, essentially, from that design, you're able to actually serve more traffic, too. So. Great, thank you. In the back. Hello. I'm curious about dev instances and how much it takes to spin one up, what it costs when you're not using it and like, is it very convenient to spin things up on-demand as needed and, you know, much cost savings along those lines for sites that may not be on for weeks at a time, but others would be? It's pennies, right? So if you're, I mean, depending on how much you're running it, so, and I have to be like, I have to tell you how we run it, right? Like, we run our, we spin up dev instances on, I'll use some terms here, but we use dev, we spin up dev instances on dev sites on spot instances and we use EBS for storage. EBS is their elastic block storage and with the spot instances, when even with the dev instance goes down, even if the dev site goes down, the data is not lost and we automatically spin up a new, it automatically spin up a new spot instance and with that, you can just pick it up, but so it takes about five minutes. So if the spot instance goes down, you know, you just go get a coffee or something. And then you come back and then it's back up again. So it happens once in a while, but spot instances can save you a ton of money, but some clients want to run it on either on reserved instances or, you know, they, depending on their infrastructure, they'll do, they'll say I want reserved infrastructure or they'll want spot infrastructure, so. I have a question over here. Yeah. Hey, a couple of questions around performance. You said that you did this initial load testing, which was artificial and I'm just curious, did you find that reality pretty much matched your initial tests with artificial load? And do you find in reality that there's, that while performance on average is fairly good, you occasionally get like, you know, the 99th percentile is quite long, you know, quite long tail? Yeah, we actually took reality and we tripled reality. Okay. It was, we blew reality away. So it was all like, it was all tests. So we just took reality and just like. So you just had a much higher load than anticipated together. Yeah, just to see, just to see if the infrastructure would handle it. Okay. And that was one, you know, it was something that we had to satisfy, we had to satisfy them. Yes, yeah. Another question if I may, just like, everything sounds so wonderful here. Was there anything that didn't go amazingly? Was there something that you're like, ah, this will be easy. And actually it was surprisingly tough, like any gotchas or foot guns that you encountered. We had some hiccups with like blue-green deployments and we had some hiccups with the rollbacks and stuff like that, especially when it came to databases, right? But those were things that we had to work with them and figure out because those were hiccups we had during testing. So. It was sorted out by the time you actually, it was sorted out in the long run. But when we were doing testing, it was a difficult process just to do the blue-green and rollbacks. That was the most time-consuming thing just to figure out how to do that. Would it be rude to have one more question? Rude on. Like long-term maintenance I'm thinking here. So you mentioned we just have this new service from AWS. Over time you anticipate that you need a certain level of knowledge of AWS services so that as you main, I appreciate that once you've deployed all things are good right now, but if you don't have the Pantheon or the platform who's kind of looking after things for you and making sure that maybe five years later you're using a different service from the one you were using before because it's more appropriate. Do you feel that if you go this route you ought to have someone on board who's kind of aware of and keeping an eye on AWS as part of kind of the maintenance even though you're not necessarily doing much but to have that understanding rather than black-boxing it completely if that made sense. Yeah, we do it for our clients, right? But if you're rolling your own then yeah, you need to stay, I would say you need to stay up to date. What I showed here is something you can create and get started with. If you want to go in that direction then I would encourage you to learn more about AWS and be involved more with it. So, but you can't just stand it up and walk away from it, so yeah. Can I put it back here? Yeah. Hey, I wanted to ask about multi-site or not necessarily specifically Drupal multi-site as it's supported by Drupal core but like hosting multiple sites in this kind of configuration that's something you've tried or you have any experience with? Yeah, so these guys had 50 sites with the same code base, is that what you're talking about? Yeah, so we've done that with this, on this project there's another university we were working with that had 700 sites so with the same code base so we did a set up of that. So yeah, it's possible, it's not difficult to do, it's just a different set up than what you would find on Acquia or PlatformSH or something like that. Were you able to leverage a lot of the same infrastructure and just add additional database instances for that kind of growth? Yeah, each site had its own database. And since Salim doesn't wanna do shameless self-promotion while I walk across the room, a lot of the problems they've solved for clients they've rolled back into service offerings so the blue green solution was made and now it's an offering. Hey, maybe first the question that you asked, yes, I can say I'm CTO of AMAZE, we use pretty much exactly that. Yeah, yep. One question though, you talked about fully managed, there's still stuff that you have to manage, like for example, you're gonna use a PHP Docker image that you need to make sure, like let's say there's a PHP security issue coming out, AWS will not update your PHP Docker image for you, that's something whoever manages it. So how many people are actively monitoring and managing, maintaining this infrastructure even though a lot is managed by AWS? Yeah, that's a very good question, so thank you for that. Some clients have the people, they have the teams to manage that and some clients will actually have their own images that they want to manage. So we are able to launch their infrastructure and their images on their accounts, we manage everything in our clients' accounts, so we don't manage anything in our accounts, we manage everything in their account and a lot of times they'll come to us and say, we've got this infrastructure and we've got these images, we want you to manage that for us. So we'll do that for them as well. But a lot of times they'll have their own teams, if we're managing it for them, then it's, we'll take care of it, but it's generally, it's not that difficult to update the, like if, we just need to rebuild the containers and redeploy a new version of the container and the new version has the fixes in them, so. Right, right, so essentially that's managed by the SLA or the expectation that the client has if they want to do it, if they want us to do it, if they want somebody else to do it. We work with a lot of different government contractors and sometimes the contractors want to do it, so. As far as like local development, do you run a very similar like shape architecture locally? You just run a pen, like a PHP image or what exactly do you guys run locally? Yeah, we don't use, and on, if you're asking on DevPanel, what I talked about was cloud, cloud nine for the folks here who just want to do, you know, AWS directly themselves. If you're talking about DevPanel, we actually do serverless for local Dev environments as well and we use, we give you VS Code in the browser and we give you PHP MyAdmin in the browser, but that's all running on serverless environment and it's all through that point click control panel. So, but that's if you have DevPanel. If you don't have DevPanel, there's the cloud nine is available through AWS and it's a very good product, so. Is there a short question about chance, it's a tail end? If you're interested in doing a bop on this and getting together to talk about things, bring it up in the serverless channel on Slack as well. If you are not going to do AWS as you can't stand it and you find some resources about Google Cloud or IBM or what have you or Azure, you want to share those in the serverless channel on Drupal Slack, that would be fantastic as well. Yeah, so thank you for being here. Thank you. I want to thank Doug Bannis.