 All right. Good afternoon everybody. Welcome back to the afternoon session of Dripple South Shorts If you could find yourselves a seat Little in-person joke So this afternoon, we've got the DevOps stream for the next 45 minutes. My name is Toby Bellwood. I'm the product lead at amazing IO We've got in this session. We've got Margie Sonec from morphed He's gonna be talking to you a little about Docker letting you know what doc is all about and how it works and some of the Experiences with it and then we're gonna pivot seamlessly into a conversation about DevOps and we've got some star guests That'll be part of our round table for DevOps So as we go to the presentation think of you a really good DevOps questions line them up hit them in the Q&A live Q&A section And we'll try and get to them as we go to the panel after the session, but I'd like to welcome Margie up on a stage Hello guys Yeah, okay good So I'm Margie. I'm from morphed. I'm one of the two count funders co-founders And I'm a DevOps engineer. That's what I do most of the time helping our developers Gets their coach into production This is a DevOps round table as to be said, thank you Toby for the introduction But I thought I would have a little lightning section as lightning talk about Docker we were hoping to attract some people to this So this is just a little bit of lightning talk about Docker and the hardcore Alma doesn't have to be Discussion will be after so Let's try again There are no doubts that you have Docker already somewhere Because Docker is everywhere hosting providers have been using Docker for years Gavcms the current version of Gavcms runs Docker on Kubernetes or OpenShift Cis these days use Docker containers. GitHub action. Let's you choose at the beginning what? That you want to run it on Ubuntu. You have to choose what container to run your Pipeline on and local development use this Docker heavily We can see here like if you use Lando use Docker If you use Gavcms, Ahoy and Pygmy Pygmy you use Docker Doxel you use Docker. I think people basically use Docker every day when they develop this slide is just to Pause a little bit on what Docker is, you know, because I still remember when Docker came I don't know 12 15 years ago. Well 10 years ago. We were calling it virtualization, but these days It's just It's it's basically like a platform for developing shipping and running applications You put them in a Docker container and then run them Somewhere and you run them using the OS level virtualization capability. So Docker itself is not virtualization. It's it's a way of shipping Developing and running containers It's just funny that I borrowed this definition from the Docker com website. You can see they didn't say open source platform They just said open platform. We can see that it's an enterprise now Docker terminology just the basic one There is a Docker image a Docker container and Docker hub have seen people having a little bit of a difficulties to Defy what is what I mean developers who just run Ahoy up and Not sure about the terminology. So I thought I would make out just a little bit of a little bit of talk So you have so you need to build your Docker Docker image somehow you either Build it yourself by running a Docker build or you may pull already existing image And if you pull existing image, you will do it from the registry, which is on the right Very very often Docker hub frequently, but this day you have also get up container registry and Then you start it and when you start it you take the image and you make a running container from that image And you can have more than one So that's basically all these three picks And I just came up with a little metaphor how to think of Docker image Docker image is like I think of it as a binary as a package you download say you download Firefox on your Mac You downloaded that package and then the container is that package running the program running And it can run multiple times when you run the Firefox you install when you run it You can run it multiple time if you have multiple profiles So that's basically how image and Docker image and a good container work and out of funny metaphor I came up with is that if you are a PHP developer you can think of the Docker Image as a PHP class with all its definition and then when you instantiate the class you create the object Many of them more than one you can make create more than one object from that class You know, that's actually the running container here, I'm just Showing you how you can run a program without installing anything. So here I decided to execute the chaos a program I didn't have anything installed. I just have Docker installed and you can see that it's actually Pulling the image on the second line because I don't have it locally. So it pulls it from the registry in this case, it will be Docker hub and Then it it starts it and because I gave it the parameter. Hello Drupal South you can see that It popped up in there. Well, is it a whale because somebody took the original and this this is coming back. How is a Docker? image built and There is a file called the docker file and this is Definition of this previous program. I just showed you here. So this is how how the How how the image could build so you see that if they are using Debian stretch as a base image So there is a docker image, which which has Debian stretch in it. That's that's the start then you just install the cow say package APT package inside that and then you copy your I think it's the ASCII car art the docker cow that that basically that that that's this the whale and That's it and then you say what what should be executed if you don't provide any parameters That's how easy it is. So in docker file you define how to build your container. I also just want to I can see people mixing ahoy and docker together, especially people who have been doing just gap CMS and nothing else and And they run ahoy up and so from for them ahoy and docker are very things which belong together I'm saying they have nothing in common. They are two different things and it's because ahoy is just a tool of Defining commands in a YAML syntax and saying if I say ahoy Margie it executes this if I say ahoy Peter executes that so When you look at the definition of ahoy PS and this is the output output how our PS looks like you can see containers there I've got CMS containers The definition is the first four lines. It actually runs docker compose PS. So when I run ahoy PS It executes docker compose PS. So the ahoy is just a wrapper as I said docker compose because I just talked about docker compose so it's a tool for running Multi-container applications together. So here is a just a quick example how to run a stack for Drupal So I have three services. I have my SQL service. I have a web server running and from nginx and I have Drupal container, which is PHP FPM and you can see that the nginx the last one in the middle the Drupal they shared Slash web directory. That's where the code of your website is and And There are dependencies. You can see that the Drupal service depends on my SQL and the web server depends on Drupal That's that's that's all it is and it's very simple and the last thing I want to mention in my 10-minute talk is just about developers running their containers without really looking at the containers which might be the case of CappCMS development You might end up with with very old images or too many images on your System, so it's always good to look at Whether you have images you don't need obsolete it. So, you know by running commands like docker image Or docker ps and I'm looking at sorry looking at images which might be stopped but not being used So there are commands which you can use to clean up. For example, this one I provided here docker system prune and you execute that that offers you to delete every single image which is not currently running and Also the networks related to that that's that's a big cleanup I don't want to talk about the details now, but you need to know that you have images and there might be many versions of them and You need to clean up now and then otherwise you will run out of the space one One detail I want to Mention here because I have seen people hitting it See here I have when you basically when when you refer to an image with when you start a image and say latest It looks at your local Better you have image already and it's called latest and if it's not it downloads the latest image So say you're running gopcms and you have php 7 latest and You did that the half a year ago and then half a year or later you again you want to start php latest and Doctor demon looks whether you have the latest image already and if you do it stars it But you might be actually running a half a year old image because you you said latest and you did not explicitly pull the current latest from the repository, so Just be aware of the latest doesn't mean latest until you verify that you don't know you actually docker pull that latest one and I also find it really helpful to actually put the The specific version of the image because the latest always maps to a specific version of that Container image to be sure what you are actually running because latest on my laptop might be php Latest on my laptop might be completely different docker image than php latest on my colleagues laptops Just be aware of that. I have seen developers not realizing that and struggling with old code without actually knowing why That's I think it's most of that I wanted to say there is just one a practical example I thought I would offer as just a little bit of the thing about for example, if I run backstop Js, which is regression tool for regression testing I use this alias on my machine So when I run backstop it basically starts a backstop container if it's not present on my system and downloads it and it mounts my asset that is my My current directory. I'm executing this backstop command from and and runs that so I can jump to any directory when I have backstop files back backstop Config files and run backstop without any installation without having any Independencies and I can do it on my local and then I can do the same within a CI This is just an example and if this looks really horrible when you look at that what it just executes a docker Called backstop and I try to mask a Google analytics doing like DNS poisoning. So I'm actually not hating statistics for the side when I'm analyzing it But it's very simple few did once and you have a tool. So What I want to show by this example that you can have multiple tools like that Which are actually wrappers over docker Comments and you don't need to know about them, but they are easy to run That's basically all I wanted to say. I know it was a little bit like But I didn't want to go into any details. I just want to Tell you a lightning top Thank you. Perfect. Thanks very much Margie Quick question for me. How many is too many docker images to how on your local machine? It really depends on your memory I 32 gigabytes now and When I have more than 20 I can I can feel that it really depends on how they are running how many CPU Yeah, I know that was a joke. I just checked. I've got 590 docker images and I system ruined about a week ago So maybe they are idling running running close to limit. Okay, so thank you very much Margie for the talk What we'll do now is we'll pivot seamlessly into a sort of round table about DevOps I'm going to bring a couple of extra people up to this stage But if you've got questions put them in the live Q&A tool thing and we'll see those and we'll see if we can get an answer to them and We'll take it from them just waiting for There we go Greetings everybody. So If we want to do a quick round of Intros let us know who you are and why you DevOps Okay first my name's Nick shoe I'm ops leader previous next and I DevOps because I like to help people. I think that's I Yeah, I know I generally I think that's kind of the line Like I worked in help like a help desk thing for five years and then built tools there And then kind of went in I went said I wanted to be a developer and then And then kind of went into DevOps because I was trying to help developers. So, yeah, that's that's my reason I'll go next thanks Nick. I'm also Nick Nick Santa Maria senior DevOps engineer at the Victorian Department of Premier and Cabinet and I'd DevOps because I Needed an escape plan from Drupal, but that hasn't worked too well because I just keep working for Drupal companies Nice And I'm Tom to good. I'm head of sales at amazing. I owe Based in Auckland, New Zealand and I DevOps. Yeah, I guess I like Nick Stanza around, you know Helping make make developers lives easier, but at the same time. I'm also Automation nut just love anything about automating things and tickering So that's another reason I got into DevOps Awesome, well, I might head straight into a Shameless linker question and Off the back of Margie's talk. How do you think Docker has influenced the DevOps landscape? I mean, I'll jump in. I think it's it basically invented the DevOps landscape In a way it really revolutionised every aspect from local development continuous integration development environments production now with Kubernetes Every single touch point is now containerised and and that was not possible. Well, not widely available until Docker came along and Built their great tool kind of reinvigorated it a little bit to like the DevOps movement because before that it was kind of you know standard, you know host infrastructure like very, you know, like robust standard infrastructure But then like all these talks around a K. How do we bridge the gap between Dev and ops and you know, certain people like big Organisations taking that leap but then but the smaller ones not as much I guess because it was really a and is really like a movement around people and and if anything like the Docker movement just Through yeah, just accelerated the heck out of it with tooling that then invigorated everybody and then got them to You know get on the bandwagon as well and kind of rethink their processes in the meantime, so Yeah, I agree. It also sort of lowered the barrier to entry. I think it sort of made it a lot easier I mean traditionally you were dealing with you know, you know data centers servers this sort of thing And although the DevOps principles were around it was just a lot harder to deal with right So this really brought it down, you know to your local machine where you can build everything Replicate everything, you know, and then take that all the way through the stack through your UCI and production things I still remember when you know, I was starting with infrastructure as code 15 years ago, there was no way how we'd run it on your local, right? You were running in on servers, but that completely changed you can still have infrastructure as code I'm actually running on your little little laptop. That's a beautiful thing. I Don't know how many hours I wasted packaging virtual machine images for vagrants To save other people from having to you know run like puppet or Scripts or things like that to actually fire up an environment like just for the docker piece like yeah, that was pretty So But but think about the company too like like the docker company like started with the tool and then the company kind of grew Around it and became this massive beacon and then kind of led the way and now it's sort of shrinking back down if anything too More of that initial tool. So it's very It's very interesting to see that journey as well Quick quick throwaway question from Sean. What's your favorite docker image? What do you go to Margie's is obviously Cal say That was fun I'm not sure it's my favorite, but I was looking for something people would know and it's reveal. Yeah, sorry My my favorite images pick me pick me up Looking to be alive. Yeah What are you the most? Yeah, that's that's probably gonna turn into hosting shameless plugs. Yeah, it's our skipper CLI. Yeah I'm just scrolling through right now Most used one is the AWS CLI because I can just never get Python installed natively and not have issues with you know, Python 3 and Python 2 And that goes back to Margie's point about being able to run your tools inside images. I stumbled across whale brew Last year or the year before and that's home brew but in docker images. So Most of your favorite tools are ported into docker images and run a ball by a single command If you like that kind of thing So, I mean when we talk about DevOps, you can't ignore the The more recent trend that is talking like it's DevOps is becoming dev sec ops Be really interesting to know how particularly as people who run sort of large infrastructures In the DevOps space, how do you go about performance and security monitoring? It's tricky, right? Because it is that it because it is that DevOps factor like it is the people like Like one question, you know, like that gets thrown to round like that security focused Like folk folks kind of instinctually go. Oh, like ops should answer that but it's like no, no No, it's also the apps like it's it's the whole picture, you know, everybody kind of has to work together Otherwise, you know your infrastructure could be as secure as as anything It doesn't matter if you know if people aren't if it's not happening on the other side and vice versa. So it's like security is definitely a you know, one of those things like a That really teases out, you know comms between teams and people working together more than just, you know Take more than maybe like the wall that we set up With all these toolings where we went just package an image and then we'll ship it for you, you know that was kind of like the You know where the DevOps movement used to be about like let's talk and collaborate. Maybe the Maybe the tools went a little too far, you know and automated away all those comms and you can't really automate around security So and having those discussions. So yeah Anybody else want to take it Yeah, I mean, I have a yeah, I mean, I think DevSecOps is you know a big move and it's sort of a subset of the DevOps movement It's really really grown and it's just in the last couple of years and trying to really bring that shift left model Into security whereas the old classic model was you build your application You deliver it you hand it over to some security team to pen test it or do a review and then knock it back and patch it the idea is you're bringing in this automation tooling into your DevOps workflow and You know code scanning static analysis Whole bunch of tools that you can do and actually catch things as early as possible And you know, we're seeing a you know a lot of encouraging use out of this and it's becoming really really important to You know if you can get this that you know You're getting a deployment processes up and faster if you have to wait for security to come back It really slows things down. So Yeah, lots of scope and lots of exciting New enhancements on that area for sure I'd hate to focus on the tools, but are there any tools that? Like are enabling you to do that shift shift left? I mean, there's a whole bunch of tools you can plug in You know, I mean, you know some of the things that even get like github brings you like the get Guardian You know coming through and actually scanning your repository for put, you know sensitive information Yeah, I mean, there's other other tools that I probably can't think of right now, but you know, there's a yeah Totally depends on the use case Yeah, I Think back to your point Nick that they're probably the number one thing that anyone can do is good code reviews and just making sure that the stuff that's getting in there is Sounded eject We played around with the idea of adding some, you know, like a you know, trippy sort of vulnerability scanning to our to our build pipeline and It just raises some potential risks around You know, what do you do if there's a vulnerability, but it, you know You feel that the the risk is low enough to be happy to deploy it So, you know, haven't those as kind of build gates is Not a solved problem for us, but definitely something we're considering Yeah, and there can be a lot of noise, right? I mean we use trippy as well You know, especially when it's coming from upstream dependencies, right? So it's which completely out of your control. So it's definitely a balance that you have to sort of achieve. I Think I've read a study which says that like one third of all the official images have some kind of like vulnerability built in because They thought I cannot find the document, but that was like a half year ago So like even the trust that image, you know, because it's like official php image Well, hopefully that one is simply enough and you can look at the source But they are complex images you trust and you have no idea what you are running and you you know That's one thing and also if you guys have seen Get up actions fused get up get up actions and you you see these what is it called actions to actually running inside your The the repositories the the plugins whatever I figured the terminology So people just use many of them But none of them are indoors by github like they are like a few of them and you are running somebody else's code from their master You know, and you are not realizing that they put it in the customer pipeline and you are not realizing that so I Started like I have a few clients We have a few clients who use these and and say okay You either have to fork it so you can audit it or at least you need to lock it to a comet So you actually know that you know that the comet you nobody can tamper with that comet because you have looked at it You audited it and he say I know that this comet has nothing evil in it But it's just like blindly running the latest in your pipeline. You never know when they inject something, you know Like this is such a I can see this coming. This is but that's been a problem a long time You know, I remember people used to give, you know, a lot of heat around curling to bash, right? You know just having a one-line kill bet some, you know products recommend that's the installation process But effectively pulling down a docker container is Pretty similar right again contain Seriously that that image scanning can be pretty important It's it's kind of going to the attacker like Instead of previously where it was like, you know curl bash like what would you like to run? It's like what would you like to run plus what tools would you like to like, you know to come with it, too? Do you know like it's yeah, yeah fully tested down, you know As someone who spends a lot of their time in upstream docker images Yeah, it's it's a minefield out there like I can trust the official ones the official PHP image official Python image, but Every so often you come across a fantastic tool that is too good to be true And you're like well, how long can I pull it from this person's repository for before? They push a weird image, and I know nothing about it. Yeah commit signing is one way of getting around Pulling those sort of the commit hash of the of the docker image But you just got to be so vigilant and coming to Tom's point of shift left It's no longer a security guy who sort of pops out is headed head at the office once every couple of weeks and goes Hey, I've done a security order and I found this We've all got the tools now. We've all now got the responsibility and Yeah, the tools are being produced like trivia. There's a couple of other really good software bill of materials type tools The in the onus is on the developer to know what they're using We're familiar with it from a like a composer packages NPM paradigm, but Yeah, I don't think we think fully enough about it from a docker point of view particularly And the same goes for the other side like as a lot of us are platform or Developers like we are developers just you know on top of platforms or development pipelines, and it's the same deal Whether it's you know a pH, you know whether we're writing PHP for the platform or writing go or writing, you know it's all the same same issues at that level as well, so Yeah, there's a lot of parallels So so one question this has been asked by Andrew is has anybody got any tips to speed things up with a docker development on max? Yeah, yeah, by the next laptop Yeah, yeah number one answer to that question is that But yeah, probably learn learn a lot of other location in depth and how that works and operates and I think is the is the best Advice I could give Yeah, it's probably a new way that I can tell Okay, I have seen people struggling Mac users running Lando having massive performance issues, right? I'm going to excuse myself, but recently looking at the cues they I think Lando mounts your home directory you know like in and even if you are running a Little project on Lando and there is an other process copying anything Modifying anything in your home directory, which is not related to your project It's still being replicated inside, you know, like and you are wasting wasting this IO of virtualization for that so There are ways how you mount only the necessary stuff like SSH key some things and like so that somebody pointed it out recently in the Lando Issue Q that this might be a massive reason because it mounts all your home inside If you if you jump inside the docker container running Lando, you realize that you have your home documents movies in there Just good to realize and whatever is touching them. It's going inside Yeah, it's funny the problem never really went away when it went from vagrant to docker compose because it was always like a file system mounting performance style issue So we kind of just ended up in the same realm where it was like you either use the baked in You know mount solution where you have the right permissions So you don't end up like churning your local files to like 777 or something really silly to get around that or you end up Yeah, or you end up doing like NFS like an NFS mount, which is a bit more performant, but more to maintain so Yeah, and just add to that. Sorry. I thought it was building images But yeah for when you're syncing files if you can do things like not have synced dependencies So if you build your dependencies in your image and you don't have to sync them that that's often a way you can get away with faster There's also some really good writing out there I think like shade this plug for our own blog But our number one most popular blog article was written four years ago about improving docker performance on Mac Since then I know Jeff Gillings done a couple of updates There's been a couple of changes But I think the problem the problem still exists and it is So watch what you're mounting and make sure that you're not just blindly mounting the whole thing because that's what the default is So really if you're just working on a theme Mount your theme if you're just working at like build as much as you can in the image Don't rush out and buy a brand new and one Mac thinking it's all gonna be sunshine happiness and Rainbow's yeah, that there was a substantial amount of rework that had to go into that from what I've seen Check my commit history sometimes Just to mention that you know your gop say gop CMS images will not work on your new Mac Right because they are not built for this infrastructure just to realize like But buying a new Mac book for a developer you have to go for Intel because otherwise they will not be able to run CMS images You're running different images locally right at that point and you're deploying a different image as well, so I don't know Still unresolved Yeah, my what are people? like my experience of the last two weeks building like a full suite of multi architecture arm 64 AMD 64 images is that Yeah, there's there's a lot of caveats in there What Tom says it is right essentially you're running a different image locally than your own production At this stage the chances are that it'll be more broken locally that it would be in production Most production workloads are still running on Intel boxes but there will be a point in the future when the running a Kubernetes service on the Boxes in AWS or wherever is Economically more feasible and suddenly the whole problem pivots and stuff that works in your local suddenly doesn't work in production because you've got that multi image split I'm trying to build the same image at the same time for both I Could do a 20 minute talk on Docker build X and manifests and local registries and I still wouldn't get anywhere near the actual crux of how confusing this thing is But that's an interesting problem because yeah folks like AWS and like all the cloud hosting providers are getting into the silicon game and starting with their own ARM chips, which is yeah varying like What we thought was stable or what we thought was stable ground not so much anymore What does everybody run for their local like or what does that org like what's the standard tool for like local local Drupal development? Ours is just kind of standard Docker compose at this point We maybe use pygmy sorry. Yeah, and Lando I guess that split Yeah, we used London. Yeah, and for a gap CMS. We have to use ahoy pygmy. Yeah, hopefully it's coming. Also, I Mean like into London Yeah, and we're all seasoned doc compose with a hoy over the top Interestingly, we've got some of our guys running Windows machines and using WSL and it seems to go reasonably well So there's another another curve ball in the mix at some point in the future. We're gonna have to Follow our DevOps tooling to Windows I Sorry, I just do this one. I had that developer who has Windows which is a unique unique case But they had less trouble running Docker The BN Docker packages inside WSL So they had like a local performance file system wise because the kernel Linux kernel inside Windows is very well done And there is no trouble with file system sharing as opposed to Mac, which is ten times slower So it's like actually user on Windows WSL to can run a native Lando on native Docker Without any performance issues. That's that's like a game changer Isn't it a bun to inside it's like you're already in the VM, right? So you're loading up Docker so you're already sort of virtualized in there and then you're all the time is just Yeah, so there is no performance on That's fascinating for me to it wasn't I open it right like it keeps going into Suddenly the Mac is the Olympic platform, right? Everybody but every developer running Docker has trouble with performance because it's on Mac and you know Fascinating like I've also been working quite a bit with get pod and get hub code spaces over the last couple of weeks and Those I think are gonna be pretty game changing Being able to run essentially your local workload in a pre-provisioned Container that's built from an image and in the case of both get pod and And get hub code spaces you can define what the image has you can set it up for local development You can add the tools and people just spin it up in a browser Like this whole Mac versus PC versus Linux like you can do this on your iPad now And that's there's gonna go I think that's gonna be a bit of a shift back to how DevOps was in the yes We can provide people with Docker images that run exactly the same thing But if we can sort of provision Re-provision the environments that it's all running on we know that people at least Should be set up for success Yeah, as an example that you've been building those in one images in a cloud service of a that a Mac many running in the cloud So you can actually just connect into others machines to actually do most of your development, which means you don't need all that power Sitting in front of you It means you can do it from your iPad on the beach or skating down the center of Auckland What about my ID man, that means I got to use VS code I Love the S code I'm sure they'll have a beam in the cloud soon for you Quick question from Dennis Can you recommend any training pathways courses for new joiners to enter Drupal DevOps? Like not particularly his eyes particularly for onboarding for govc-m-s-c-i doctor Drupal But what do you think are the best resources out there for who are the people to follow? Where do we learn from? Jeff Keeling That's the Probably unanimous he puts out great like he's written books on Ansible and Kubernetes he puts out great YouTube content Yeah He's the guy go and follow him Plus one to that for sure We don't specifically to Drupal. I don't know if there's anyone else I can think of that would have so much content So much knowledge Foundation run some courses and Kubernetes and they have like multiple and they have some really good content like I think I have a friend working as a developer and DevOps engineer for Linux Foundation And he works on that testing environment What I have looked before they have really interesting content and that's how Linux Foundation makes money I basically providing training and certification Kubernetes and you know, so I'm pretty sure there will be some Talk about But that's interesting though because like there was like especially on my journey like there was like very clear tentpole folks to follow like as all this stuff was happening like But now it's kind of yeah, it's a little bit more disturbed like dispersed or maybe more specialized Yeah, like the Jeff Keelings and those kinds of folks like I mean you had used to have like your Solomon hikes and Kilty-high towers and those kinds of people who you know You'd follow and kind of go on that journey with them as everybody else was but um, but now yeah, you Yeah Quite interesting. I'd really encourage people as they're on the journey to document it like write about what you've done Use like the services like dev dot-two medium cough are Are really good at putting experience out there and finding other people and doing the same sort of stuff that you are I find it really cathartic writing about what I've done because it helps really cement it in my mind I read a blog post last year sometime about using Using Docker eyes in a Docker image with circle CI to CF services already and it's still like by far away The most popular article I've ever written and I'm like I I remember it doing it and thinking this is so cool I wish someone else had written this before me Hopefully I can help someone else but yeah, you write something from your point of view. It doesn't have to be perfect It doesn't have to be polished. It doesn't have to be Like cross-checked and edited and reviewed by 15 people. It's more if it's your personal experience It's it's good to share with people and maybe you too could become the next Jeff Keeling And to close off the last question I have is yes, Carl. This is an RGB microphone Red light means hot Okay, a few minutes left. What's gonna be the next big disruptor in the DevOps or DevSecOps space performance monitoring Hands down. Oh, that's that's kind of my my take on it like we've We've put all the tools in place that give us much deeper hooks into where our applications are running to the point where you know like like take New Relic for example like New Relic Was Stable for a really long time like that was the product like the APM was the product But now with things like pixie like it is very very compelling Where it's kind of like this lower level tool where you don't need to in small that the guarantees or the thing that they're putting forward is The fact that you throw these agents on your entire cluster and then it'll just start to pick up Performance bottlenecks and the like I think we'll see way more of these kinds of tools and way more things pointing out Application bottlenecks and performance issues that we didn't have before Yeah, I can go I mean I would say it's more your abstraction basically like a lot more abstraction where still I think we're still in the early days of that That moment where it's all very very configurable And you sort of had to be in that way where you know, you set up clusters and you put nodes and you've got to sort of tune everything I think I think there's there's a lot of scope for that to be just abstracted away And you can already see some companies and products that are doing that already I don't think we're done with serverless and and yeah that and the distributed tracing and and kind of the observability caped like Where would a really vital system? So it's kind of like a blend of common mixed answers I think like certainly Back in the day sort of New Relic and Blackfyre and whatever used to be really good at the application I think there's a realization out from a DevOps point of view There's a whole stack thing in here and it's it's not just your application There's the infrastructure it runs on is the infrastructure tuned is your database tuned is there's all these other things And there's all these layers to peel back and it's it's not as simple as optimizing a php call or Sorting out a node library There's all these other elements that are going to impact how your application performs how it scales how it Performs when it isn't scaling So yeah, I think there's there's a lot of additional work to be done in Understanding how your application runs when you when you launch it isn't just launched It's the watching. It's the tweaking. It's the configuring all right, um, I'd like to thank you all for your For questions. I'd like to thank the panelists for coming and giving up some of their afternoon to talk about DevOps I know it's a struggle to get you for talking about DevOps. Um, so Thank you very much. Um, and thank you all for attending. Um, this Drupal south session It's been a fun day and I look forward to another one in November See you guys. Thank you Thanks everyone. Thank you