 OK. Thanks for coming. Hope everyone's had a good KubeCon. When I first proposed this talk, this was actually kind of a little bit cynical. And based on the description in the schedule, I've changed it a little bit. But I've kept the parts that I think are most applicable to people who are coming to KubeCon. So what we're going to cover, our story. I work for a company called Lytics. And we transitioned to Kubernetes from GCE and jumped onto the GKE system. So this is kind of the story of how we transitioned from GCE to GKE. And the stumbling blocks that we kind of ran into along the way, as you can guess from the theme of Oregon Trail. We ran into some issues, overcame some stuff, and kind of ended up an interesting place that not a lot of companies seemed, and I don't think a lot of companies ended up in our position. But I think we actually uncovered some cool stuff along the way. So who am I? My name is Josh Ropo. That's actually not on the slide, which is great. I've been doing data engineering and platform infrastructure engineering for most of my short career, about six years. I'm a gopher, and I really like Kubernetes. I'm definitely a user, though. I haven't contributed to core. And I haven't really had the time to dive super deep into stuff. But trying to stay up on the trends and where things are going, I like asking why and doing thoughtful design of systems and trying to do things as good as well as possible. And I don't like being woke up at 3 in the morning when Elasticsearch goes down. And I like stable infrastructure platforms. There's primarily why I was so interested in Kubernetes when it started coming on the scene a couple years ago. And so stability is a good thing. So the Elasticsearch story. We're a customer data platform for marketers and web developers. We enable personalized web interactions. So the classical example is the recommendation engine from Amazon. So you look at a page, and then you see the other items that you might be interested in. We enable you to, based on content recommendation and machine learning, we enable you to create a list of front. We can tailor personalized experiences to your users based on our generic engine and entity resolution. We were standardized on Go since 0.9. So we've been a Go shop for a long time. We have a little bit of R and Python to support. And originally, one of the first things I did at the company was we were on AWS. We moved to Google Cloud Platform because of stability and performance. We started adopting a lot of their technologies to replace things like Kafka. So Kafka was a pub sub, Cassandra was a big table. It's been a really nice platform to build on. And as far as the spectrum of monolith versus microservices, we're kind of in the middle right now. We haven't gone full microservices adoption, but we're not tied down to a single monolith. So we have an API tier. We have a workflow management tier, which we call a metaphora. And then we have an event stream. We have a stream processing system that's distributed across a number of group of deployments. And that does entity resolution, machine learning classification in real time. So where we came from, we were originally on Saltstack. And that is both powerful and terrifying. You can take YAML, run it through a Ginty template, and then generate more YAML based on that. So it's a very complex system, but it is pretty powerful. So obviously, I'm not going to get into why Kubernetes we're at KubeCon. We all know all the great points of what Kubernetes is good at, but for us, it was primarily the idea of going into VM management is not very appealing because you have to build all these heavyweight tools around it. And we're deploying a single binary. So using all these heavy tools to manage things like operating system kernels, managing all those upgrades, it just is overburdened for just trying to deploy something very simple. So when Kubernetes started coming on the scene, we're like, this is good because we had played around with Docker, but any of the one who's tried to deploy just Docker to production knows that that doesn't go very well. So the goals of moving to Kubernetes, we eventually want to get everything on Kubernetes. Right now, we only have our own core application services in Kubernetes, but we're working on trying to get our other dependencies in there as well. We run a few SED clusters, and SED is kind of a core heartbeat for our distributed system mechanisms. So one complication that you do kind of see, and it's kind of visible on the screen, is that we have a deployment system called linkgrid, which can be scaled from one. We might need to have from anywhere between one to n number of these deployments. And upfront, that creates a little bit of an issue because that means you're going to have to define YAML or template YAML declarations for these deployments from one to n. And so I wanted to have a good solution to manage that going forward. So before we could even get to there, we had to do some application preparation. A lot of our services were kind of built in the older model where local state was available, and it would be there with your service crash and then came back up. So we had a few things like BoltDB, files that were on a server, and if something crashed, it was checkpointing its last location. But in the Kubernetes world, if you kill a pod, it needs to be able to reschedule somewhere else, and it won't have that state anymore. So you have to rebuild your application to accommodate that fact that persistence storage shouldn't be a thing, and you should always use distributed stores like GCS, maybe SAF, just something that's over the network. We also really utilize a lot the HTTP pre-stop hook, which enables us, in a few cases, we have long running workflows with integrations, with external services, that in the case of shutdown, we have rate limits on some of the integrations we work with. And if we're just only able to trickle data and there's no checkpointing that they allow just because their API doesn't support it, we need to be able to let that finish. So the HTTP stop hook on a pod is actually super useful in that case, and it's been pretty rock solid. So when you request a pod to terminate, it actually does wait that full time, because effectively, the way we use it opens a connection, and then you just hold that connection open, and until you're ready to shut down, you just hold it, and then when you release it, the pod terminates, and it gets cleaned up. So Kubernetes doesn't hard kill it right before it, right when you send the stop command. So this took a little while before we finally got all of our services aligned up to work well in the Kubernetes deployment model. And once we got there, though, or another thing we ran into was we have heterogeneous workloads. So across some of our workflow machines, and we have some processes that will consume way more CPU than others, and depending on where they get distributed to in a set of pods, there might be one workflow that uses 100% of the CPU, and for a few minutes, and then shuts it down, and others, there might be five others that don't. So you can have one CPU, one, there's a few different vectors of resource allocation on this. So we have some clients which can use this. We have some clients which, using the same workflow like an export to Facebook or something, or BigQuery is a good one, will consume a lot of resources on that one pod. If it's a smaller account, it might be burst for only a little bit, but it's a flat distribution across a set of pods. So in the VM system where it's hard to create and allocate VMs just because it's a little bit ops-intensive and something you don't want to do very often, you end up having to buy more compute than you actually need. So you end up in the situation where you're just wasting money. One of the really cool things about Kubernetes is that there's a difference between what you schedule pods for and what they actually are limited to. So if a pod needs to burst for a single workflow, it can. Just for simplicity's sake, on the VM utilization, let's say that there is one pod that actually is running hot most of the time. And let's just say hypothetically that these other ones are kind of hanging out, not doing a whole lot. Well, with Kubernetes Scheduler, if you combine all those and to say like a two core machine, you could schedule all of these VM, all of these pods onto the same VM. And the Kubernetes Scheduler will assign all of those to one pod. One will be able to schedule all of those onto one VM, but they'll actually be limited to only running on using one core at a time. But if they all buffer up to the point of, or if they're all limited by CPU, they won't be killed. So they can all fight over CPU, but chances are if they're all running along at this average percentage usage, they will actually be able to fit nicely into that VM. So on the trail of the Kubernetes, obviously YAML, lots and lots of YAML. I'm not the only person who thinks this. And some people love YAML. I'm on the opinion that coming from salt, white space is difficult to think about sometimes. And the fact that YAML doesn't have any termination characters means that you can't format it. So there's a lot of times when YAML just becomes more confusing than you would like it to be. And managing tons of YAML files, it's like, unless you have scripts to getting back. I think for static resources and rules, like your RBAC settings and services, I think it works fine because those things don't change very often. But for things that you might want to manage and update regularly, it seems like they're or create that one to end number of, I think it's a little bit problematic because you're going to have to manage it somehow. So obviously, Helm is one of the popular YAML templating engines. And that's not necessarily it. Helm is a pretty cool system, but it's still that templating engine. I think that the Tiller design is actually pretty good, but I think we can do better than YAML for our own application stuff. And now it's KubeCon, so there's a bunch of cool announcements like Kson, it looks pretty interesting, and there's other things like compose. But again, you're taking a Docker file and you're converting it into another type of YAML. So just as an example, we can actually use Go to actually, that's out of order. So another option. Client.go, it's a client library. It's extracted from the actual Kubernetes repo, so it's not quite as much code that you need to import. If you've used it before, you probably experience some pain. It's a little bit tricky to get organized correctly the first time you set it up because the dependencies are complex, but it does afford some pretty cool stuff. So initially, I jumped into it because I'm a bit of, I like Go more than I like YAML, so I was curious to see, all right, is this gonna work out well? And you're compiling against the Kubernetes source code, which is really nice because I would often find myself in the early days of Kubernetes looking at YAML files, trying to figure out what went where, and there would be mismatched versions and examples, so sometimes you'd find something that was using a certain structure a certain way and then you have to match that up and you pretty much end up digging into the code anyway. So at this point, it's like, well, if we're gonna have to dig into the code, why don't we just use the code itself? And one of the cool things about actually writing the library is that, or using client.go, is that when you run compile, everything is type tracked. That's really nice, you know at least when you go build that everything lines up correctly. So this is kind of like an equivalency. On the top, there is a YAML definition and on the bottom there is the same thing written in, specified in the in.go. And as you can see, YAML is a little bit more concise and a little bit more readable, but the verbosity of the go example is very, it's nice and clear. We have hard typed fields like name and volume source, which define how you need to build this structure. And at any point, you can actually dig into these types and actually see what the documentation is and how it's supposed to be used. So I'm gonna show you a little bit about that in VS Code. So right here, this is an actual example that I've been working on for the past couple days, trying to make it an example for people to start using the client.go. And it's just like, you can just pick this up, pick up this library and start playing around with it, because everything's already rendered, everything is already defined. So like you can just pick this up and start playing around in this get repo. But in this example down here, we're creating an API v1, beta two deployment. Obviously, we're still running into the numerous number of versions in here of Kubernetes deployments. But if we highlight this deployment, we can see a nice little structure definition pop up and then we can dive into what the type actually is. So now we're actually diving into the client.go library. So we've jumped into the vendor folder and now we're in the type of deployment. So now we can see that there's four fields on this type and these are the specs. So this is the deployment spec, hit control, dive into it. Now we can see all of the fields on this and we get nice, I mean, it's code. So it's not like a nice HTML page, but this is the documentation for what Kubernetes is actually using under the hood. Okay, so the first thing I kind of did after digging into the client.go library was I built a CLI tool just to kind of see like does this work? And initially it was just like list the pods from some cube context. So the cool thing about building a CLI tool is that you use the same cube config file and you just specify with flags and parameters to the client creation function which cube context you wanna use. So if you were running your mini-cube, you would pass mini-cube and I have an example of that later. So at Lytics we actually started using this to define some of our deployment objects and I actually created a superset object on top of that called a deployment set which is effectively a specified set of deployments that all share roughly the same type but are just like named, have separate names and separate a few different fields like metadata tags to define their role. So in prod we have about 10 and in staging we have three. But those are very clean to create because effectively you just create a nice static function which defines all of them. So we still use cube CTL in our deployments which is unfortunate. I would like to live in the world where Kelsey Hightower's demos are because those are great but we're not quite there yet. Rigging is a pretty limited scope, it just does creation. There are flags to mutate the types at runtime so, or not at runtime, but at creation so we can update the number of CPUs that we want all of the pods to have allocated to them. Just a few things about if you are building a CLI make sure you specify and make sure that the user is defining which cube context they're actually trying to use. Cube CTL is actually kind of loose on that which I find surprising. Like you don't actually have to specify the context it'll just use whatever you have configured in your cube config which is a little risky because Kubernetes has never made it easier to accidentally the whole cluster. But, and then also sem for all of your deployments. Obviously, Kubernetes is moving really fast and that's great but it also means from user's perspective stuff is moving really fast and things are changing out from underneath you regularly. On the right there is a list of SIGs of special interest groups and this only goes to C. There's a lot of SIGs out there and it's hard to pay attention to all of them. Things like replication controllers, third party resources, those are already dead at this point. You shouldn't be using them and I know that talking to people here at KubeCon everyone is kind of, all the users of Kubernetes are kind of dealing with this. But everyone on the, everyone who's actually building Kubernetes wants to go faster. Like they want releases every month, not every few months. So we're in this like weird kind of, I heard Tim Hawken describe it as a U shape or like we're in the bottom and everybody like users want it to go slower and the developers want it to go faster. And back when Kubernetes was unversions it was kind of the Wild West. You were always in the code trying to figure out what stuff was actually doing. So surviving all of these changes, the client-go library has had a rough history. I started using it at the client-go version 1.4 and it was like, oh, this is kind of cool. And then 1.5 came out and they created another path for the 1.5 client and they just shoved everything in there. And then they kind of realized, oh, this is a bad idea cause we're just appending tons of code to the same Git repo. So they ripped that out in version two and then from then on things got a little weird. You can see this matrix on the bottom right and the check marks are where things all just line up well. So we finally have reached a solid point with the client-go version five and Kubernetes 1.8. So things are actually lined up at this point which is a really nice thing. And if you need to update from 1.5 to five I actually have a document which I need to write a blog post about which is about how things change and where your things shifted around to cause it definitely got weird. But things are improving a lot. So from going forward I would, I hope that things are stable now and it'll be a lot simpler to upgrade. So again, when you're, if you decide to use client-go think a bit about your design, look at how the interfaces use your, or look at the interfaces of the types that you're gonna be interacting with and think about it a little bit. There's some chaining that goes on that makes some of the client-go functions a little bit hard to use. So think about it before you jump into it. Fender, everything. If like make sure there's no code that is coming out of your go path. And there are some issues with depth right now on at Lily's talk yesterday she mentioned that RBAC has a little few issues. So if you try to use RBAC in client, in the latest version of client-go there you might run into some issues with some API versioning. And that's a bug in depth actually right now they're trying to sort that out but be wary. But also I don't, again if you're, yamble seems like the appropriate place to define RBAC rules. Don't ever mix any other projects with the project that you're trying to build your client-go code with because Kubernetes moves so much faster. So like we have like our, I tried moving client-go into, or the dependencies into our main repo and we use a lot of Google libraries and we use a lot of GRPC libraries and we also have SED and all those three together just create this massive storm. You cannot find the resolution of libraries to make all those things work. And unless you can match the velocity of Kubernetes project you're probably gonna end up caught in a difficult place. And it just, you won't be able to resolve the dependency trees. So build your client-go library in its own project and don't try and merge it in with any other projects. A few things that we used to just kind of get us over the fence onto Kubernetes. We have, we use Kibana, or we use, it used to be our only log storage, but there are things like Google logger to our Stackdriver logging technically and that is one option but the search functionality is a little bit underwhelming and there are some nice features that people really like about Kibana and we wanted to keep those. So we actually used something that Glider Labs created and I forked it and made a few minor modifications. If you are interested I would be happy to show you how to make those changes as well but effectively it created Daemon set, runs on every single node in your cluster, reads the logs from Docker and then ships them straight to your Elk cluster. But you don't need log stash in any of this. It just writes the log files directly to Elasticsearch and it just requires a simple mapping to make all that work, or Elasticsearch mapping. And I've deployed that six months ago and I haven't had to touch it ever since. So as far as software goes, I'm pretty happy with that one. Event logger, originally logs weren't bubbled up from like the useful Kubernetes events. So if you do kubectl get events, that's actually a useful stream of information to know about what your cluster is actually doing. Created a little tool just to read that out and rate it to logs. It's actually super useful. You can create metrics off of that. So when pods are dying and you don't know why, or if pods are dying, it's a good way to create a metric on it and then notify. Yeah, so you can alert from failures that show up in the event stream. And then some simple use, I would use loggerous JSON logging and then modify some simple things so that it worked well with GKE logging, which practically means you add a field and for severity. This is kind of a unique one, but if you use something like statsD for metrics and graphite or carbon, Prometheus is well designed in that you can give it a list of pod names as part of the path for metrics and it handles that fine. It writes it correctly, but if you're using an older system like carbon, it writes every single metrics path as a file system directory. So if you use pods, that will just, if so, if it's using pod IDs, that's just gonna explode. So created a distributed, effectively a naming flattener. So you can effectively share different names. So something that has a very, something that has a normal pod name that has a replica set and then a unique identifier, you can squash that down to just like one and then one, two, three, but then when pods exit, then they will release that and then someone else can claim it when another pod starts up. So you effectively recycle the name set. So the, and then the advancement of our CLI tool was something we called Ceph. It's not the end goal, but it's a really useful tool for our staging environment because we do a lot of integration tests that run over like terabytes of data. And then we look at the results and make sure that things are looking correct. So to facilitate that, Ceph actually runs that deployment set. And it's a simple integration with Slack. You send out an update command and then you give it the tag and it will update in this case, three different deployments with that tag. And those deployments roll out all the pods and you're up and running. So this was pretty useful. It provides nice visibility to our staging environment so people don't like stomp on each other. And it's a nice tool, but it is again, your issuing commands. It's not as clean as like something that is just reactive based on your GitHub. It's not something, it is still a manual tool effectively at this point. So we want to get to the point where we're using like an operator pattern, but we're not quite there yet. But all the types are defined. So we are in a good position to actually move that way. And this is a nice step in that direction. We used Slack, that's a thing. Slack is a difficult API to work with. I would cautiously recommend putting very strong interfaces around every interaction with Slack because who knows where that API is gonna go. But in its design is very similar to Helm's Teller. So it sits in the cluster and communicates with the master and then it, we issue commands to it. And one nice thing about operators like that is you can apply guardrails. So you can do checks on like the current state of your application or you can do checks to the command that you just issued. So validate that the image is in, has been built correctly by CI and is in the repository before you try and apply it and then potentially take down a pod for no good reason. So with that, I'm gonna try and give you a short demo of actually using the client-go library to do end-to-end testing without having to pass YAML files around and cat scripts. So like typically whenever I see like some project there's like all right to do end-to-end testing you run this bash script which evals this other bash script and then the output of that and that runs a bunch of COOP CDL commands and applies them. So what I've been working on and committed to I think less than 40 minutes ago was this project that I'm trying to build as a showcase for using client-go. So, all right, so let's go to the actual test. So what this test is gonna do, it's going to communicate with my local mini-cube. It's going to use that deployment that I showed you earlier. It's gonna call this function create the deployment structure. We're going to create a connection to mini-cube and then we're going, like actually this is creating the connection, we're going to create that deployment. We're gonna validate that it exists which is not very sophisticated right now but and then we're going to delete it afterwards. So, and this is all in go, there's no shelling out to do this and then we're going to watch. So it created the deployment, there is the pod and it should go to terminating pretty soon here. Or not, the demo gods are not being nice to me today. Oh wait, no. Okay, so it did delete it, the replica set is still going. Yes, all right, so the replica set is still available, so which is why the pod is still there, so. And I also alias kubectl to ktl, so if that's what you're wondering what's going on. So that was all in go, no need to shell out and use a bunch of YAML files and kubectl to throw stuff around and check zero codes. So using client-go is a bit of an interesting thing because it is a lot stickier and heavier to pick up than YAML because you can just copy a whole directory of YAML and run it against your cluster and see what happens. But I think it provides a lot of insight into what is actually going on and it provides sanity for me. Like when you actually can look at the fields that you're trying to use and dig into them and see what everything is, see all the specifications for that type. I think it's really useful. I think other people should explore it and especially now that it's not, there's not so much wonkiness with versioning going on. I'm hoping from here on out we will have complete feature sets for every release so there'll be a new client-go version which doesn't line up numerically assemble-wise but hopefully that'll tie very nicely to the upcoming Kubernetes releases going forward. I've been talking to a few, I talked to a few of the people working on client-go and they sounded like hopefully from here on out it'll be solid. GKA as far as our actual migration has been rock solid for us. Upgrading has been super easy. We haven't really run into any issues. Like occasionally a kubelet will die and in the early days I didn't really have to go kill them every once in a while, but it's been rock solid for a long time. So if you are considering GK or Kubernetes managed systems it's been pretty great for us. So we eventually did make it to Kubernetes. Not everything went to plan. We didn't get all of our stack into Kubernetes yet but Elastic Search is a fickle beast and we haven't quite, I don't think anyone has quite figured out how to get Elastic Search in Kubernetes yet. I haven't seen any significant demos. Hopefully table sets will get us there at some point but I know that there are some memory allocation issues that come up. But for us the ability to use deployments and our stack actually fitting in deployments has been super useful. It alleviates a ton of burdensome operation time to actually get our app out there and have it being resilient. And being able to over bin pack and let Kubernetes scheduler kind of handle our own scheduler scheduling issues is really nice. So we can oversubscribe to our compute resources and the Kubernetes scheduler allows us to still have high availability without having to shove everything into vertically scaled pods. It's really nice when you're using client-go because you only have to interact with one API now. You don't have to interact with the compute API and then the image API for typical cloud providers and client-go is really nice. So the theme of this conference has been Kubernetes is now some percentage more boring as infrastructure and hopefully won't be breaking so much going forward. I'm really looking forward to service meshes and what those will provide us because being able to the Istio's TLS automatic automatic encryption of traffic is gonna be a big game changer and really nice. And auto scaling and client or in cluster resizing is gonna be really nice going forward. So that's all I got. Thank you.