 Hello everyone, thanks for coming in at the I think this might be the last time slot the last day before the Designs of it tomorrow who out of grassy who all is staying for tomorrow tomorrow's Friday sessions for the Right on yeah, who's local and who's local from Austin? I'm really proud of the rest of you for sticking it out this long this summit. Thanks for coming So brief intro, I'm Luke Heidecker engineering director at Selenia. This is John Stanford vice president for our product delivery and Sorry product and engineering at Selenia He's been around since like I don't know the PDP 11s and Both of us were working on distributed systems things like that for quite some time At Selenia, I'm part of the delivery side if you were in an earlier session And consulting side John is on our product side. So we got two of the three arms We also do a lot of training for clients both in the space of OpenStack, but also Kubernetes containers and things like that all really focused and driven by open source infrastructure We have offices On the West Coast San Francisco New York City and also in Tokyo, which is our newest office and in Seoul, South Korea We work a lot with clients both on the product side and on the consulting side for global 500 finance Media companies and and of that other customers that nature I'm gonna go to hand it over to John to talk about some of the background and then What comes next? So so our product Goldstone Started out as a I'd say a fairly monolithic Django based Application and as it grew and we started to incorporate more features We ended up with a pretty large sprawl of back-end technologies and front-end technologies and Ingestion technologies and all that good stuff. So fundamentally as just a quick overview of what the product does We bring in logs and events and metrics and structure data from OpenStack and start to you know present that binding with some of the knowledge that the consulting side of the house brings to us and You know, you you'd recognize a bunch of the stack components And I'll show you some of those things as we go through one of the of course big challenges is that Getting a developer to deal with this stuff in a monolithic sense on their laptop is challenging So developer spin-up was really tough for me. So there's lots of components. We have three language frameworks today We use Python Django We use JavaScript and backbone and we use Golang for some of the some of the back-end and smaller components that gather data We also have two variants of the product an enterprise and an open source Variant and and so there's public and private repos and all kinds of stuff like that and different packaging strategies for each one And we have a small but growing team of people. So when we bring in new developers I you know as the person who you know wears about ten hats that I rotate around every day You'd have to deal with developer spin-up and and want to make that as efficient as possible So so we can get people working on the fun stuff the product So Developers, you know a lot of them we Choose to develop on Mac. It's very convenient and nice and that's where we started from But as I started adding some people and consultants and things like this We start getting Linux variants come in the in the mix I've got a guy doing stuff on Mintos and and you know all of a sudden I've got some some sprawl not to mention the targets for production And we just have to try to rationalize that and it started to get painful For developers we tend to stand up at an open stack node In a virtual machine next door to the you know next door to the to the code base That's running obviously in production. That's going to be a much larger cloud and in our lab There are larger clouds to play with so we had to come to terms with that sort of thing as well So the initial state, you know as was fundamentally the bowl of spaghetti Not necessarily a code although there was a fair amount of that in the early days But but even from an intro infrastructure perspective the number of components talking to each other and and you know The ports and configuration and logs and all that stuff So so the stack, you know as it stands today consists of those things in production and potentially multiple instances of those things And the developer needs a reasonable subset of that to be able to just get the data from infrastructure To you know do the work they need to do to interpret that kind of thing So, you know some of the big challenges of the project was growing in in scope and and you know code base And all that good stuff the team was growing The the knowledge that people you know people were coming in with about Virtualization and operating systems and all that good stuff Varies, you know quite a bit. I mean, you know fundamentally. I'm looking for good developers. It's helpful if they have Infrastructure knowledge, but not always a requirement if they're you know if they're really good front-end developers I don't expect them to know a lot about you know setting up networking and Virtualized storage all kinds of fun stuff like that. So we need to you know figure out ways to to make this possible for them Rapidly iterating is is one thing that I'm really you know big on so I come from a background where development You know my last big project that I worked on we had an 11 second compile time and you could really you could really go Put a line of code in and quickly see The impact of that on the on the real code base You know in a very quick time and so when I come to this world and and we're you know trying to get Things up and running you know some of these changes take a long time and and and we'll talk a little bit as we go through about You know just sort of finding that balance Beyond that, you know, we want to kind of ship a product every couple weeks and or ship a version and and you know Make that releasable to the world so Those are some of the forces that we're really challenging us at the beginning and You know our goal here is to talk to you about how we've tried to overcome some of those things We haven't baked at all. I'll just get you know, there's the sneak preview. It's not all fixed but the reality is that you know, we've we've come a long way and and dealt with that we also had challenges around Different distributions and we'll face this more as we sort of widen the scope of our product to hit You know hit different distributions that we work with and different releases of open stack come out And then again our open source versus commercial variants of the product So Repeatability is a big thing. We have dependencies on environmental stuff We have dependencies on on management or sorry dependency management the versions of the libraries that we use in our code If those things change, you know, all of a sudden every developer has got to pull that and and update in the monolithic world They're virtual e and v that they were running in and it just you know, if they forget to do that All of a sudden the code doesn't work for some reason or another and then they need to you know And we need to communicate about why things are not working and do some triage and all that stuff and and it As the team grows I expect that would become a very difficult challenge to keep everybody synchronized in that type of world And then finally, you know the integrity of testing is important We need to be able to verify that Developers have a very similar testing environment It's certainly for unit and more and more towards integration testing that as we do in our CI pipeline and and our continuous test and delivery kind of stuff, so That was fun We have some scalability issues to deal with single nodes for ease of development So developers are working on a single node thing multiple nodes or multiple versions of components for production How can we you know quickly go from one to the other and and verify that things are working nicely there? and so containers, you know kind of came up as a as a way to Potentially solve some of these repeatability and composability problems and we also have some you know longer-term product aspirations Selenia, you know, I think we've been around the open-set community for quite some time I've been here since Hong Kong Hong Kong was my First day at Selenia. I arrived in Hong Kong. I started working at the open-sex summit day one a couple years ago and Prior to that, you know our folks have been around the community, but but fundamentally we identify ourselves as an open infrastructure company Luke was here an hour or so ago giving us each a presentation on kubernetes and we do a lot of stuff with Docker and and Devops and things like that. So from a product perspective I look at at those guys to lead me and and if they're headed that direction I want to make sure that my teams have some understanding of this technology so that when you know when when the leadership says Hey, we should have a docker, you know a docker mark management platform I can go ahead and do that without having to go retool everybody Also the trend in the industry has changed a bit and Towards you know leaning towards containers as being a more safe safe route. They're starting to see it in consulting quite a bit You know, so so we want to be up on that to just stay current and keep our people Engaged in the cutting-edge technologies that are coming. So I'm going to hand it back to Luke for the next Next section So this piece is how do we kind of Transition some of these monolithic deployments and and into a containerized solution that's Composable but also reusable by the developers in their day-to-day activities so that you don't fall into that sort of trap of deploying in one way for sort of production versus Test and development we can continue the artifacts through the entire process So the first things we looked at was standardizing the base image sort of the first iterations of of using Containers we were using the official images And Found it sort of difficult because some of the maintainers decided it best to use an Ubuntu base image or a Debian base image Or then some centos images and there was very little standardization and we found it used very useful to Start out with a standardized base image that we could reuse that sort of helped us both in the standardization of libraries and methods but also in the build-time iterations that we we could share the layers of those images between multiple multiple images within the rest of the components of The Goldstone server architecture the next piece was again I kind of hinted to it was figure out ways of decomposing The image requirements so that we could layer the dependencies and we were reusing as much of our Docker images as possible so it starts out with that base image of in this case Debian and as you know, we're looking at ways of sort of talking Later about sort of the future improvements but Debian sort of allowed us to still use package managers and have a fairly Minimized image in comparison to something like even Ubuntu and then We included one of the other lessons or was sort of moving to a where we were Including the gold the the Goldstone code as the last layer within that so that developers could iterate quickly And not have to wait on all the prerequisites and things like that being installed So we'll show you right now sort of how we kind of went about some of that with the initial base image here We start out. We're looking at the image Installing the prerequisites on top of a a base Unicorn image that we have created we have versioned one of the issues we saw Was that the way that Docker images even the fish the official images? That the way that they tagged and versioned those images changed Significantly between projects and it caused us some headaches. We'll go into later That we really it's very important us to for us to keep a consistent Base image and a consistent experience instead of dependencies and things like that for developers So that was the first step then we go down to the the environment statements We wanted to have an easy way to override statements of need be as customers go into production But keep things consistent between environments. So we expose certain environment variables Have a consistent work directory And then go into this is broken across several slides because it's Nice and robustly long we go into breaking out the build requirements and the prerequisites the build requirements are only needed for the initial pip install or Other other packages and the pre-rex are something that should be included as these As the image goes and runs whatever applications on top of it Again in this case G and a corn You're starting to see here if it would be from from my sort of standpoint It would be really nice if this was declarative a little more declarative so that I could sort of say I need these things installed figure out whatever the base images get it installed rather than some of this Mashed nation between every image, you know, we end up using a lot of these same sort of Characteristics throughout all the base a bit throughout all the Docker images And it gets a bit redundant especially if you change your methodology so that's sort of one other lesson learned we found is we're adopting these Containers is that you know keeping things consistent, but and making sure you're looking across your entire catalog I think we have Yeah, a dozen or so images and That continues to grow as you know, I just decided to create other images and I want to keep things consistent It's it's sure would be nice if that could be a little bit declarative So we go into making some of the basic directories and in part three One of the things we found and we'll go into so again lessons learned here is that Initially, we had a lot of these layers As optimized as possible so that the build wrecks and the pre-rex were installed We installed our software and the and the pip install for requirements with for Django and then the Goldstone server But we then wanted to remove the build wrecks and clean up the image some And we could we're able to save a lot of space the downside is that that caused a much longer build time so if the it's that weighing of of build time versus the iteration frequency and if if a developer is trying to build that image and See an instant feedback Maybe change some requirements change versions on their local machine Waiting 20 minutes can be frustrating and you know You know go get some more coffee isn't always the answer Sooner or later you get all copied out and it's nice to so in this case We actually separated out these layers so that developers can easily change some of the Python environment requirements without having to Completely reinstall the pre-rex every time they rebuild the image So we take this image. We have a Docker entry point script that really just is able to read into in the environment Files and start it in a repeatable way We then layer that on top with the app image and that last We have several images that use this base image this base genocorn or genocorn whatever it is a Image and this one is the application image. We also have like celery for example would reuse that same Those those base dependencies as a base image this one takes the goldstone server code and overlays into the after And then everything else the entry point everything else works Without need to change it We then take that and we're using Docker compose for development certainly what we're also Able to port this in the other orchestration frameworks including Kubernetes But wanted to walk through real quick that it allows us to easily do that linkage easily be understandable developers fairly quickly I think with the onboarding for this was fairly straightforward And it allowed us to also do some of the logging and ingest that back into the development framework So we could see actually what was going on within the Docker images As an example in the developer environment We changed things slightly. We have a different environment file We add some volume so that developers can as the Docker images running They can make changes within the the code for Django and Instantly see feedback without having to rebuild the entire image as long as those base dependencies don't change They could also easily boot it the image do virtually MV and and test some of those things out a lot easier than having to rebuild The image every time once they're able to test that then they can put that back in and throw a CI System then rebuild those images and have a go through the full set of unit tests and integration tests and things like that So I'll head it back to John to talk about the next piece great. Thank you Look, I want to make a couple points before I move on to the next slide This really you know highlights a couple of the I think the key advantages to Docker What Luke said about you know developers kind of instantly getting this stuff I think stems from the fact that you've got very few places to look to see a fairly Wide view of what's going on here. You you know start with the compose file. You go. Oh, I see it's got an app server It's got an elastic search server. It's got a log stash server. It's got a you know, various other types of containers And then if I look at the the difference of this and the production I see that I'm overlaying, you know the code for for the developers so that they can iterate quickly And that was you know finding that balance has been has been an interesting experience I think we're still searching for for the right balance. I'd love to get you know iteration time, you know Cycle time faster, but but at the same time I don't want to compromise the the end-to-end, you know repeatability of these environments And so we'll continue to work on that But I think it's fair to say that our current state is less like a bowl of spaghetti and more like a table full of Bonchon or something else that comes in small containers We've got isolated functionality that we can triage and do some, you know Assessment of and work with in in that fashion when we need to Versus trying to sort out what's going on across all these different Processes on the same virtual machine or the same hardware Where you have, you know config scattered all over in different directories And you've got log files going everywhere and not consolidated and certainly not time sequenced in one place So so we get you know some of that kind of stuff So I wanted to talk a little bit about, you know, what our topology looks like now We went from that sort of, you know monolithic view over the world to something that people can grok pretty well the you know the configs and the Environment files and all that stuff is organized into a into a tree structure Within the repo For the top layer of images the the actual ones that support the Goldstone product Developers typically don't need to dig much deeper than that to see what's going on Rarely do do they get down into the the base images to do something unless we have an explicit Goal of versioning those images or you know bumping a version in those things and everybody's got a cascade up and And we'll address that as it as it happens, but but fundamentally this is the layer they work with so a Developer sees two different experiences whether they're an open source developer or an enterprise developer an enterprise It's the same base image for say Django and celery, but But we bring in a different set of code So we you know we're bringing in the open source code plus, you know plus the enterprise code into those images They don't need to think about that all they need to do is use the compose file for enterprise developers of the compose file for for open source developers and and the enterprise the enterprise images come from an enterprise repo that's authenticated and the open source Just come from the Docker hub which is is open. So that's worked out pretty well. There's another variant of developer I Don't think that I have that slide in here The the other variant of developer is that is the split between front end and back end developers And that's purely our removal of some back-end, you know debugging type of containers like if I've got people working on search You know, I might want to have them have a cabana instance to back up their search exploration stuff And and it's not something we ship with our product But it's something that they can use during development to quickly get to get to a query that against elastic search That gets them the data they need to fill out the you know fill out the You know the the feature of the product that they're trying to address Same thing with flower flowers a celery monitoring tasks So a front-end guy doesn't really need this or gal doesn't really need these things In their day-to-day and and so we we just have another compose file that doesn't have those in place Similarly in production You know, we we take out those debugging, you know kind of containers We have the open source and enterprise variants that are fundamentally the same except for that last layer Of copying code in but we also remove some of the some of the developer image-building kind of tools like the Like the entry point scripts and the Docker files and all that stuff doesn't ship and doesn't need to be in production so we kind of ended up right now with sort of for developer compose files and and to Production compose files, and then there's a CI one that's similar to developer But uses images instead of building containers in place So we we're trying to I'd love to get that down. I think Luke more importantly would love to get that down to a much smaller number For sustainability and all that stuff and just just repeatability across the life cycle But so far, you know so far it seems to be a fairly good balance so You know what? What have we done to sort of get the development teams up to speed? You know, how how did we how do we go from monolithic to you know to a container-based world? The first big thing is that it became a lot easier to script the setup of a developer environment It's literally, you know clone the repo Docker pull the the images from the compose file or you know use the Docker compose Pull it pulling up kind of commands to get images on board a little bit of setup still For back-end developers, they typically want to have a virtual ENV because they may want to run tests outside the container They may not want to get in the container for all this activity We'll talk a little bit about that in the lessons learn section and some of the you know overcorrection kind of mistakes we made as we went through this process, but Fundamentally, it's it's a lot easier than trying to run the whole application on your desktop You know on your on your Mac laptop With all the various, you know brew tools and virtually and v-tools and library differences and all that stuff So that smoothed out quite a bit Getting the developer environments Started, you know, we've got scripts for that that just basically set up and start all these processes We'll talk a little bit about how far to go with that stuff in in our opinion I'd say there was a bit of overcorrection there as well I'm sort of giving a sneak preview, but then the other piece is the The version management of these images and how it associates with the versions of the code base itself So we've got scripts to do bumping and building and pushing images around up to repos and things like that And and that's an area. That's probably still a work in progress So, you know, what is the learning curve for a developer you come onto my team, you know, oh boy You've got this application. It's all docker-ified. You know, what what do I need to do and how do I get it? Fortunately, we're in a position where our training staff offers a docker 101 kind of class and and we can lean on them to get that Information, but you know, but fundamentally, it's not there's not that much You need to know about docker to be a developer on our team You know, you need to start containers and stop containers and List what's running and maybe delete delete containers and images on occasions in the basics If you're you know, if you're managing the the container Strategy Development, then you probably need to know stuff about compose and a little more deeper detail on docker networking and things like that But you know for the for the front-end developer or the front of the back-end developer It's it's you know, it's pretty straightforward one of the things that became really I'd say a lot easier was using Dockers, you know consolidated logging from all the containers spitting them out as JSON and Developers can see all that stuff and so they can see the sweet sequence of events that maybe led to a problem They could see it across the database and the elastic search and the application server Celery which might be having a recurring task that's causing trouble. You can kind of see it all streamed by captured if you need to We haven't taken it any further like shoving that into an elastic search instance, but You know, I think it's in a form that we could pretty easily do that if we wanted to and replicate that Strategy for both the production and the developer viewpoints Like I said the back-end developers might need a little more networking storage networking and storage experience, but typically the back-end developers That that that we find that are a good fit for the team have a little bit of that anyway They kind of come from this community or Docker community or other other sort of virtualization heavy kind of communities We're still learning this last one the the the association of versions of docker images to the associations of commits in You know in a git repo is a is a challenging thing And if you're working on anything, that's even a moderately complex code base And by moderately complex I usually mean more than one or two developers Possibly using sub modules multiple repos, you know all that kind of stuff It becomes a really interesting challenge to keep everything in sync and you kind of end up with some chicken egg you know kind of things and and You know generally that the pushing of images up to places where you can share them with your distributed team Is a is a slow process and you kind of want to minimize that to the times when you've actually changed images So that that's something that we're still working on But I think we're getting you know closer to something that that makes us all you know relatively happy So so on that theme, you know, I'll just do one small rant semver is really neat if you if you use it not just say you use it so the major version, you know for big API changes minor version for a little functionality and backward compatibility stuff and patches for for really bug fixes and things like that is a great mental model for versioning software it would be a great mental model for versioning images, but but It it gets conflated with Docker when you have Really two things that you're trying to version the the thing that you're putting in the container and The contours of the thing that you're putting in the image and the image itself. So There's you know, we this is one of the reasons why we end up building our own base images because we can intercept this upstream image and make sure that they haven't had a Released an image with the same version but different, you know different Docker behavior So that that's my small rant and I think that's the last one I have in here So we'll just move on and I'm gonna hand it back over to Luke to talk about some of our, you know more questionable Moves as we've gone through this process Thanks Yeah, so one of the first pieces we saw is just in sort of the exuberance to adopt containers and get things as optimized as possible As as clean as possible We've found the need to kind of take a step back and look at the optimization between the build time and as I mentioned the the developer iteration time and and periods of those iterations so that so that you're not punishing the developers for the optimization of the build time or the run time just and and vice versa and The other thing piece there is making sure that you're not you're still providing ways that developers for both Development issues, but also in production issues that you still give a method towards debugging the application within the container The next piece we tried and found some issues But a lot a lot of times just the developers day to day was in the use of data volume containers It was an early sort of recommendation with with docker is a good way to sort of keep your data separate from from your application containers You ended up with sort of a double penalty of Volumes sticking around especially when you were in CI in the CI environments or in developer environments Consistently trying to iterate on that and then you find yourself with a bunch of data volume containers And that there's a lot of better methods for abstracting that storage and keeping around the volumes just by just for example declaring a volume in your Within compose at start time rather than having that as an entirely separate volume that runs And they've improved that some and we that we had some interesting fun times with those Some of the time also the the notion that we were like trying to abstract too much from developers and too many scripts and You know things to make their lives easier and instead of Allowing them to kind of better learn docker and compose and how things were We're combined together into the application as a whole really a basic proficiency in the Both the commands and internals of docker and also the orchestration platform is We just found very important and very helpful for developers both in just day-to-day usage of it not feeling frustrated When something goes wrong, but also in just better architecture for the applications themselves, too The last sort of thing we kind of look back is that trusting of the upstream With their way of doing tagging their way of doing versioning One of these updates may not be in the same cadence so that for example on the base OS image You know, it's very easy to miss a very critical update if if your cadence is different from the release cycle of Those those base OS images even One of the other pieces is the mainstream or official images Don't use the same versioning and tagging standards So that you know while one might have some way of of sort of tracking the difference between Postgres nine dot four release You know container release one versus container was two they don't and It has broken code upstream or downstream for us in the past. So this allows us to Speaking of that, I have a nice little docker version reminder kept just pop up my screen The yeah, very helpful of them The so having some of those same standards being able to control some of that stuff and how things are released and have that Standard base images have been very beneficial for us That does remind me that the one of the nice things with onboarding Has been the differences in the new way of doing some of the docker for Mac for example That it the docker machine was a large painful set between, you know, do you like fusion? Do you like virtual? Virtual box virtual box or hate it with a passion Or can you run it natively with Linux and things like that? Even those differences with causing like for example with volumes causing a bit of an issue for a while there for us This is certainly a good move in the future if they don't introduce more bugs Instead of helping us fix them So some of the course correction was with some of the issues I just mentioned we're using our own base images We probably still don't have it right, but we've moved to using a commit hash for our image versions So we can be very specific in the versioning I'd prefer something more like a channel tagging where you have an alpha beta and stable release And that you just trust in your ci and your tests both your unit tests and your integration tests to fall back on if If those versions don't work out But that's something we you know just between the developers and and Stuff helping them with some of the ci stuff Hopefully we can figure that out at some point and it's certainly if you guys have a more creative way or or more Sound way of doing it. We're always kind of check checking out what might be the best strategy for that The other piece is the composition variants of Wanting to that trade-off between wanting to have a single way of composing the application as it went through the entire Life cycle from development and tests and unit tests and integration tests and just sort of a production version Ci right now that that only really it mirrors production And only has some slight changes just given the nature of the platform that we're using I still think that could be improved some so that it's being deployed the same way Every single time no matter what But yeah, yeah, we'll continue to have corrections on them sure every build we're getting better Which is great, you know trying to keep keep Keep track of the build times and all that stuff and trying to reduce them as we go Factoring in anything new code that we've created new new images that are in the in the composition But but I think all these things are headed in the right direction The the version stuff is is probably the biggest challenge I think that that we're facing and it's it's it's really a balance between I don't want to be pushing every, you know Every time I do a commit or even every time I do a pull request I really don't want to be pushing image versions up necessarily certainly if they haven't changed Even if very few since we're since we're adding some code in there that has active steps in the in the In the docker file even even if those layers Haven't changed Significantly the the code base has and that's being copied in and it causes You know causes a new layer to be generated in the image virtually every time and and that has an impact on the time of the push so So we're we're somewhere in this balance of you know tagging with the you know with the commit hash and a branch name and and You know manually pushing Images when a developer has remembered that they made a change that affects the the docker layer of the application And and that's you know that remembering piece is the piece that I'd like to you know Automate somehow and do some detection based on get comparison against the against the branch or pring into or or whatever So we'll you know like I said continue to work on that Luke's been very instrumental in making this a successful effort So some of the things that I just noticed and figured Cheryl So it are some of the helpful commands that as developers are coming on board that They had to run quite often and that sort of ease their pain there That's the type over there, but you get the idea That being able to easily remove Images and start with a clean environment a cleaning consistent environment. What was one? Then being able to selectively target it was very helpful also in being able to Remove all goldstone related images for example the The next one we found out was was being able to you know again developer of being able to easily get into the container And exact some commands deep especially debugging commands Check out internals run things manually And the other thing was the difference because we use a lot of entry point scripts being able to Push to the developers that they need the differences And that was one of the sort of fundamentals learning experience was making sure that they know That there is such thing as entry points what they do why why we're adding this seemingly unnecessary complication And then giving them a method to also do the same thing for images So that one's really that one's really interesting actually this command is the one that Is probably helps you with your most frustrating situation is your container didn't start and you don't know why? This is the one that will get you into the container Around the entry point and to a shell where you can start poking around and and possibly editing files You know config files to quickly cycle and and figure out what went wrong in your container and get it up and running so that you Start to inspect the code base, especially if the issue is with the startup itself, right? Yeah, that's yes exactly Okay, so you know so where we headed Some of the things that that I think You know are on our radar for future obviously it's you know, I think the message is pretty clear It's all about finding that balance between developer cycle time and and consistency of your environment through its life cycle so Today we have a the base layer of of Docker images out in their own repos and we're starting to add Add testing to those things as an independent unit to make sure they're functioning the way we expect them The Goldstone, you know that the product layer Images are are still in our get repo so they haven't been isolated I'm not I'm not a hundred percent sure that that's the right thing But I am pretty sure that we should have you know isolated testing for each one of those things and to be able to test the Perimeter of the images and make sure that they're doing what we expect As we take in more features from our product management layer You know the the notion of I'll say microservices very loosely At least composition or decomposition of the application and putting those features off in their own containers Seems to be catching some you know catching some interest and seems to be making Development easier in the sense that you can focus on on the feature with the boundaries of the feature and Develop it off there and you just need to sort out how to get data back to the to the main You know data layers, which is also why I don't really call it microservices. We're not doing much service discovery It's it's you know the compose file satisfies a bunch of that stuff for us but I'll give you an example we just added a canary feature. So we've got a little container that runs its 16 megabytes I love that part it runs every five minutes a job to start a virtual machine in a cloud Attach a volume, you know check a floating or add a floating IP do some other stuff and then tear it back down And then report the timing data for that stuff back to Back to the main goldstone application so we can you know help customers evaluate over time Whether they're API performance and the underlying services subsequently are degrading or or staying consistent or you know Hopefully improving because they've you know better capacity and better performance but Those are the kinds of you know microservices kind of things that will start looking at every feature and going can this stand alone by itself You know or is it somehow too tied to the data? And if it is then why is that and have we thought about something? Architecturally wrong to you know that makes it not decomposable into something that we can use Finding smaller footprints for images. I think our biggest image right now is like 600 megabytes. It's not huge It's it's huge by by some standards, but It's reasonable although seeing you know things like alpine And and you know this this canary container running at 16 megabytes. I go, okay Well, you know at any component that doesn't have a reason to be running on Debian or any other base You know we should look at this and see if we can make installation and spin up and push and build the whole life cycle faster That would be awesome And then of course the the versioning strategy, which I think I've I've hit on enough today So I'll leave that one alone, but we'll keep working on it though. So I want to give a Luca a chance to Plug an event that's coming up That that he's really near and dear to so we'll wrap it up But just want to give all I give you all a heads up my separate life Focusing a lot on on the Kubernetes SIG. We have a Southern California meetup in LA. She's El Segundo If you can make it it'd be great hosted by AT&T There's gonna be talks by some guys from Google actually about continuous delivery pipeline using Jenkins and Kubernetes Selenia, we're gonna have a very large scale deployments of Kubernetes clusters on OpenStack and then also there's a good talk from Amgen about implementing paths on top of the enterprise and Any time it's up on Twitter. I'm on the Kubernetes slack a lot Jx Stanford on Twitter. Just you ping me I will note that we're hiring both consulting side and development side and probably training side So if anybody's really interested in what we're doing, you know, maybe I've seen our booth and Got a got a pitch from what we're about what we're up to please reach out to us You can you know Reach me on Slack or you can email me at John at Selenia or hit our careers page and look at what's going on If something out there doesn't match what you do and you think you're you know, you're a good fit for our company Please just you know Give me a shout. I'll put you in touch with whoever the right person is and it's not me And on that I'll say I know we went a little over time, but Questions we'll take a couple if the guys in the back or the room will allow us to or afterwards to some. Yeah Thanks a lot guys. Okay. Thank you