 My name is Seth Goings. I'm the VP of Engineering at Deus. Love go-karts. Have never driven a Formula One car. Sounds ridiculous, sounds way scarier. This is my Twitter handle, Seth Goings, at the bottom. If you have any other questions, talk to me at an open space, or if you see me wandering around the halls. I work for a company called Deus. This is notoriously hard for people to pronounce. So we're going to participate in a little group pronunciation session. So it's Deus. Can we all say it? Deus. OK, all right, that's good. You'll see a demo of the actual product later today. So I'm not going to talk much about what the thing does, except for the word Deus comes from an old French word, which is a raised platform in a room for dignified occupancy. So a fancy word for saying a platform. And obviously, we're a platform as a service, open source operation. So I want to talk about what it actually looks like to go from go-kart speed to Formula One speed. But I first want to talk about who we are in this room. I would say we have a lot of similarities to race car drivers. We probably go faster than we probably should. We crash a lot. We see a lot of failures. Have to rebuild the car a lot. We are in control of numerous variables, accelerator pedal, the brake pedal, the steering wheel, and how this complex system operates. We're also driving these systems to usher in the future of technology into the world. The other thing that I like about this analogy is that we as engineers, operators, developers, enjoy living in the zone with a laser focus on what we're trying to accomplish. It takes a lot of focus to drive a car that's going really, really fast. I have no experience doing that, of course. So why are we here? Don't we want to figure out how to go faster? Competitive markets are challenging to be a part of, and we want to go quicker than our competitors. We're also kind of impatient as developers. We want faster feedback all the time. So we can stay in that narrow focus that like in the zone feeling all the time and feel productive. So how can we measure this speed that we desire? Why not make it analogous to a track? So we have a track with various turns, various things to navigate. We have this build aspect we have to navigate with maybe some unit testing, which Leon addressed in the previous presentation, which is awesome. You should be doing it and monitoring as well. Then we also have this deploy aspect of software development. We also have this test aspect, which is like an end-to-end test or a larger set of tests. And then we have to deliver this to our customers, because if we built and deployed and tested all this stuff, we actually want to make money off of it, hopefully, or deliver value to our customers in another fashion. So in Deus V1, we had something that looked like this. So Deus is dependent on all these CoreOS components, SED for a key value store distributed across the cluster. Is everyone familiar, or is this all new to people? Who's familiar with that middle bar? So like 40%, 50%. So SED, key value store, distributed fleet for scheduling containers, ConfD for managing configuration across all the nodes, Docker for actually running your container. And then CoreOS is the operating system. So there's a lot of layers that we are managing with Deus V1. Deus, the actual thing that we produce is just a set of Docker containers that are deployed to a registry that then we pull down into a cluster for you to start deploying Docker containers on top of. So our lap time for any change in the Deus V1 system looks something like this, 20 minutes to build, 25 minutes to deploy, and 45 minutes to test. Delivering is essentially free as I see it, because if we pass all the tests, we merge that into master, which is available for anyone. We host all our code on GitHub, so master gets updated. Anyone could grab that change, so it better be good. So our lap time in total was 90 minutes. In summary, we had kind of like a go-kart. It's fast, but not quite fast enough for us. Some of the reasons why it was that slow is because we had some container scheduler woes. Or actually, these are in general my complaints of the platform. We have fleet, which is resource unaware. So it's just going to schedule a container to your cluster, maybe not in the best place, depending on what resources are available on certain nodes. We also had tightly coupled components. If you remember that diagram I showed, we got a lot of components to try to wire together and get a working system. We also had a tight dependency on whatever nodes we ended up deploying on top of. So we used something called Cloud Config and to bring up an AWS cluster or a Google cluster. And inside of that Cloud Config file, you were already kind of locking in the implementation of what exactly your Deus cluster had in it in the future. And then we also had this problem with just general distributed systems problems. So distributed key stores and distributed file systems are a really hard problem. And this, I like to call, created some Seth instability. Who is a Breaking Bad fan? Do you remember this scene, where he's like driving around and he's like depressed? That's kind of what I was not that low, but that's kind of what it felt like. You're driving a go-kart. It should be fun, but for many reasons it's not. And you're a sad, sad man. So with Deus workflow, which is V2, we wanted to fix all of those things that made us sad. This is what our lap time looks like right now. So four minutes to build, one minute to deploy, and 10 minutes to test. How did we get to this 15-minute lap time? How did we go six times faster? We started this effort in October too, so it's not like all of a sudden we switched cars and now we're going faster. It's taken six or so months to go six times faster. But I wanna talk about something before I head off in this direction of how we got six times faster. This is my personal soapbox topic, and it's when engineers start to engineer. Does anyone kind of know where I'm going? You do. So I'm passionate about what happens when engineers go full-on engineer. It may look like the world or your company or your coworkers are waiting for you to swoop in and save the day with some programming heroics to bring all these parts, these awesome things like Terraform and these fancy-do things that maybe you were seeing today in presentations into a perfect working harmony, but I would say resist that immediate urge. If you're still not convinced, remember this picture. This is what we did with Deus V1. Deus V1 depends on a ton of components that theoretically work together nicely, but in reality, we had to build a ton of glue. With every bit of glue that we started to add to our system, we started to be the ones in charge of maintaining this whole set of components and intertwined things and all this glue and we're jamming glue onto the top to make sure it's waterproof all of a sudden. So now we're responsible for the maintenance of all these things because we have created this system and dug our own graves, so to speak. Einstein has this quote, which I think is very appropriate for our world. It's totally possible to solve all these problems in distributed systems, managing containers by figuring out how to do it yourself. That's fun, maybe do that on a weekend, but a wise person avoids it. Wouldn't it be awesome if you never had to worry about that system that you built on a weekend ever again, like it'd be cool to create something and then just go on and see, oh, someone else built this better, why don't I just adopt that. So with deus workflow v2, this is kind of what we did. We said we only care about making it easier to deploy containers into a cluster, making it easier to run the application that you actually care about. And Kubernetes started to bring in all of those components that you saw before, etcd, fleet, all those things into one package that they said, here, we're gonna maintain this, we're gonna provide this API for you to use, we're gonna create this stability for you. So once again, what value do you want to provide with your software? Do you want to write that application that gets you really excited about that idea, or do you want to get mired in the details of how distributed systems work? I really liked Dave Hahn's point yesterday, in the morning, the guy from Netflix, that said make sure you're not doing undifferentiated heavy lifting. Do you actually want to lift this huge amount of weight of distributed systems yourself? And my mother said this, just because you can doesn't mean you should, just because it looks like a cool problem doesn't mean you should solve it. Let's go back into Formula One racing. In a pit crew, we have a ton of people involved, but they all have one really specific job. They're really passionate about getting that wheel on. There's one person that he dreams about putting the wheel on just perfectly, and then the socket goes in and puts it on, gets it all secure. So maybe consider what you do in life a little bit like this. Get really good at figuring out what you should be doing and what other people should be doing. This is true in the workplace. There are people that spend their days figuring out how to attach the wheel in the best and quickest way possible, and then there are also people that love figuring out why SED is failing in this large deployment. So I'm done with my soapbox now. I didn't bring one. It's kind of hard to find a soapbox these days, but I can still have one. So with DSV-1, this is where we get into more practical things. It took us 20 minutes to build, and in DSV-2, we have four minute build time. The left side is DSV-1, how we built things. We had one get repository, one Jenkins job, and then a ton of Docker containers coming out. In DSV-2, we decided, well, let's only respond to things that change, only components that are changing, because some components move faster than others. So we have more get repositories, and we have a Jenkins job or a CI system pulling changes from that get repository and deploying just that Docker container into a registry. This gets us into a world of the fight between micro-repositories and mono-repositories, which I don't really want to discuss, but it kind of explains how we're responding to only things that change, not rebuilding the whole thing every time. The second aspect that we attacked is the idea of deploying our platform and not provisioning and deploying. So in DSV-1, we were provisioning and deploying. If you remember the cloud config kind of narrative I gave earlier, we had to build all these machines specific to the DS platform. We were gonna deploy onto it later. In DSV-2, we said, let's just deploy it to a Kubernetes cluster. So that's where we got that major shift. This is what it looks like to create a Kubernetes cluster on Google Cloud Platform. It's basically like saying, I want a Kubernetes cluster, give it to me, five minutes later, you have one, and then we deploy our dais platform on top of Kubernetes using this kubectl utility. But how do Formula One teams go even faster? There's two aspects of a Formula One car I find absolutely fascinating about going from the pit or going from the shop to actually the racetrack. And the first one is this tire warmer here actually heats up the tires to more operating conditions, what the tires are used to being run at temperature-wise. Then the other aspect is this machine right here is essentially a portable boiler. And if you notice those lines, there's like tubes going into the engine. The pit crew actually heats up the engine before it even runs because the engines are built with such tight tolerances that the engines are essentially seized at room temperature and they have to be brought up to like 100 plus degrees to run at the way they were designed. So with DSP2, right now, we're actually working on a way to kind of take that idea of warming things up before we start deploying on top of Kubernetes. And I like to describe this as grabbing warm Kubernetes clusters, right, like getting a warm Kubernetes, warm cuddly Kubernetes cluster from somewhere. So imagine this is you, you want a Kubernetes cluster as fast as possible and you see this pool of Kubernetes clusters out in the wild. Now enter the black box to solve all problems, grabs a Kubernetes cluster for you from this pool and gives it back to you. This makes you a happy cat because then all you need to do is deploy deus right on top of that warm Kubernetes cluster. That's how we get that one minute deploy time instead of waiting five minutes for a cluster to be created. This black box is what we're calling, at least for now, the Kate's Claimer. And it's brand new, just was something that some engineers got together in a room and said, we need this to speed up what it's like to improve the software that we're building. Now, if you remember my first soapbox, I said, don't build it, don't let your engineers go full engineer. We kind of did that, but if you remember, this is the value we're providing, right? We want to get our platforms stood up as fast as possible by doing the least amount of work. And this was a really good solution for us. Now, software, if you remember, from the first talk, always needs tests. But how do Formula One teams test their components? Formula One cars have 80,000 components. How do those teams test that stuff? They have things like break dinos, running a break and just breaking it to see when it fails, the qualities of when it gets hot, what it does. Same thing with engines. Imagine if you had to drive around a car at like max engine speed all the time around track and that was the way that you tested things. Good thing there's stuff like this so you didn't have to drive for eight hours straight. And then there's something called a vehicle characteristics simulator. So this is when they create simulations of actual drivers and simulations of most of the components in the car. And they put those two together, run it and see what happens. Then they can tweak things, either how the behavior of the driver will change over time or maybe how components change over time. So with DSV1 you remember our testing cycle was about 45 minutes, which is a really long time for tests. These are end to end tests. So this is like if you use our command line client to deploy a container in the cluster, the container starts, all that stuff. So in DSV2 we had a 10 minute test cycle. How do we get there? We run the tests in the Kubernetes cluster itself. We don't have to worry about any dependencies. We just say here Kubernetes cluster that's running that dais platform also run this thing that exercises our platform. So you see we run all these tests in parallel, which was quite the feat of several engineers time. 15 workers, 15 threads of this test system. We only have 134, 137 tests right now, but those are testing that full, like as a user, this should work, right? This takes 10 minutes versus 45 minutes. This was not an easy task, by the way. It looks cool, everything's green, but we're still struggling with as distributed systems are hard and things working in operation and in parallel are hard. Parallel testing is also hard. We use this parallel test, this 10 minute test cycle, to vet out changes from the community, from our own changes into our repository, and it makes it a lot easier to say, okay, when the tests are green, let's merge that change because it's a really cool idea or a feature. This is what it looks like on our Jenkins instance, which is public, it's ci.dais.io. You can see there's actually, this is from yesterday, there's a pretty good trend of not a lot of tests failing and a lot of green on the left there. So in DeusV1, we had something that was like a 90 minute development cycle. This is a go-kart pretty fast, faster than a lot of other places I've worked, but now we have something that is a lot faster, 15 minutes around a track. That sounds more like formula one speed, right? But I also wanna talk about what this feels like, going through this six months of change from going really slow, fairly slow 90 minute cycles, 90 minute lap time, to a 15 minute lap time. This quote from Mario Andretti is slightly over quoted, but if you haven't seen it, commit this to memory. If everything seems to be under control, you're just not going fast enough. I think he's 80% right, but it's not just about going fast. It's about how you accelerate, it's about how you slow down, it's about how you take a corner, how you line yourself up around that change. And it's also about the environment that you put yourself in to gain acceleration, gain speed just by being around other people, like in this room, right? Like you probably have learned more in this past two days than you have learned working at your desk for the last six months, say. And this is where I get to, this is something that I repeat to all of the people and one-on-ones that they're really frustrated by something about technology or people, is that growth is never comfortable. If you go to the Unser Go Kart track, down I-25 and 36, every time I've been there, they say take some ibuprofen when you get home because tomorrow you're gonna hurt. Because you don't experience G-forces, you don't experience all that bumping on a daily basis. But if you do that enough, your body starts to get used to that kind of stimulation and you're just used to, oh, I got the muscles to deal with that stuff. On a Formula One track, your neck will give out if you're new to that environment within an hour. If you watch the Top Gear Formula One special, Richard Hammond also says, I can't think fast enough, I see the corner, I tell my foot to release the gas and the other foot to press the brake, but at that point I'm already through the turn and my neck is like this. So eventually after you practice over and over, striving about what you want to see changed in the world, you're all of a sudden more comfortable going faster. Then what do you do? You go faster, you press that pedal down more. So I'm Seth from Deus, I'd love to talk more about this talk in an open space, I'd love to hear what you all have done to change what it's like to develop, to improve the world in which you live. That's it. How much time? Okay, I got five minutes for questions. Bring them. Go for it. So your question was what advice do I have for people that are trying to get to continuous delivery and kind of incrementally going there, bringing in tools to serve individual purposes instead of just like jumping whole hog into like here's a solution. I think this is where prototype projects come into play. So maybe you spend some of your time trying to build the thing yourself and seeing how if you can think about how what you've built after a couple of years, what directions it's headed in and then you see, oh, that actually looks more like cloud foundry or open shift or whatever. Because I actually deal with this all the time. Like we can only see like a little bit in front of us and then you talk to someone that's like adopted everything continuous delivery like Netflix and they have all these things and you're like, that's way too complicated. It's because they incrementally improve to that place. So anything you could do to learn from others and hear about the pain that they felt when they went incrementally and jump to where they're at now whenever that's possible by taking like what they have built or what they're starting to adopt. I think hearing from Netflix and the unicorns is super helpful in helping the horses of the world get to that state. What are their questions? Go for it. I have thoughts about containers. Yes. What are you trying to get at with service virtualization? I'm not sure I know what your question is asking. Like I love containers and I think services should be running in a way that could be reproducible. I think virtualization gets us that. We should talk after. What else? Go. Oh no. It's J. Paul Reed. So the question was what about the people that they are going slow and it's still painful? My answer to that is this one. Speed isn't all about, like velocity isn't all about speed. Everyone kind of goes through the cycle of, hey now we need to work on accelerating, we need to work on cornering, we need to work on slowing down. There is a case of like maybe you're trying to go too fast and you need to learn how to manage to get a go cart around the track and make small improvements somehow. So it's that whole go slow to go fast. It's like the orbital physics idea. So I mean, it's all about being uncomfortable and trying to improve from where you're at. What else? That's about five minutes. Go for it. All right. Cool, everybody put your hands together for Seth.