 Good morning. Good afternoon, everyone. My name is Aleksandar. I work at Pivotal. I work on Cloud Foundry Container Runtime and Crenate. Hi, my name is Akshay Mankar. I used to work on the Cloud Foundry Container Runtime as an engineer. Now I've moved to the closed-sourced part of the PKS program. And this is our friend, Octo. He's the octopus at Pivotal. He'll be helping us present this talk. Yeah, so we will talk about off-top-reading Bosch packages and avoiding boring work. So important part that this talk is will be most helpful for the Boschers' authors. We expect you to know what Bosch list is, how you can develop it, and yeah, usually, if you develop it, you'll see benefits. So what we will talk about. I'll talk about why do we need to do this. Probably you know, but we'll just take up a little bit, then how you can do it and how we're already doing it. And we'll raise some concerns during our talk, what we can do better. Okay, let's start. So I joined Pivotal almost three years ago, and after deploying with Chef, Bosch was incredible. Bosch deployments are so nice. You can reproducibly deploy all the time. You can deploy it in different infrastructure, manifest the versionable, so new manifest, new version of your deployment. It's so nice. And part of it, it's Bosch list. Bosch list contains all the packages. It's reproducible and it's self-containable, so you don't have to access Internet to deploy something. And the self-containability is done by Blobs. So Blobs help us to use some versions. It's Bosch packages that usually some other authors provide. You upload them to the cloud, and you have to commit on your source code. And Blobs are awesome. There's actually one problem. Problem with Blobs is that, yeah, problem we want to solve is this Bosch dance. I think all of you saw it. It's when something got upgraded, you need to add new version of Blob, then remove old version of Blob, then upload. There's a new Blob to the cloud, then commit, push it. Seems very easy. And, yeah, looks very easy. Unfortunately, not as easy. So here's just two months ago, I forgot to upload Blob. That happens. It was, I added new Blob, forgot to upload it. Before that, I forgot to remove it. And I do this mistake even with Octahelp. What about you, Akshay? Yeah, I did something, but I would very much prefer Octo to not hear it. It's really, really bad practice. I once uploaded Kubernetes version 1.6 as 1.7 while upgrading 1.6 to 1.7. So we all thought it got upgraded, but Kubernetes was still old. And guess what I did? Why don't we have a commit message here? I was so embarrassed, I forced push on top of it. Please don't tell Octo, he'll probably kill me. Yeah, so that's even part of bigger upgrade thing, isn't it, Akshay? Yeah, so that dance was small, but look at this bigger dance. How do you even know when upgrades are going to, when you have to upgrade a package? You have to wait for some team member to notice the package has updated, like they have released an upgrade. Then they would create the same old boring story in Tracker or whatever your project management thing is. You have to find the old story, copy paste things, and be like, yeah, update things. And then next thing, you have to convince your project manager, your team that we need this upgrade. Then again, that condition goes on, people go like, okay, we'll put it somewhere in the priority. It makes it to the top of the backlog, and then developer starts doing all of these things. Again, you hope they don't make mistakes. And then pipeline runs the test, and then you ship it. So all of this, how do you feel about it, Alex? I personally like it with Octopus. We like it, but it feels repetitive and little bit boring. Maybe a little bit, just a little bit boring. Some boring things. So first of all, this find new version. It's relatively easy. So we work on CFSR and Kubernetes, package and Kubernetes, and this Kubernetes is relatively easy. Okay, new version of Kubernetes, there will be some post on hack and use or somewhere, like okay, let's do it. But if we do some dependent package that no one cares that much, even Flannel or some container images that we use, you need to go somewhere, probably to GitHub and check the new GitHub release. And I need to do it every day, once a week. It has to be procedure. It's quite boring. And that fun part of the boring thing that I can see is, yeah, development work. And since it's boring, we all do mistakes, even Octo does mistakes here. It can be just one script that can do this for you. And if it's boring, I'll ask you to do it, actually, obviously. As the doer of boring things, I would probably make more mistakes or I'd just hire a Terminator to do it. And if it's boring, it'll terminate it. Like, what's the point of doing this boring thing, right? So again, what would Terminator do? Terminator would be my bot. I call my bot Terminator. It would be like it notices the upgrade. It'll create a story for me. And then maybe wait for us to prioritize it. And then we'll talk to PM, have that conversation. And then when it reaches the top of the backlog, Terminator will pick it up and do it, right? It's simple. Does this sound good? Like, if I don't do it and terminate it? Yeah. So, okay, you saved your time. That's fine. But I know, I think that probably Sean, who is here, he won't enjoy if I ask him every time to prioritize a bump of new package, especially it will be some very, very boring package. He'll ask me, why didn't you prioritize it? Can we, like, sneak it? Yeah, I think you're right. And waiting is human-like. We, as humans, like to wait for things alike. Make sure we're doing the most important work that the customers would benefit from. And because our time is valuable, we prioritize things. But bots' time is not that valuable. No offense to any bots that are here. But their time is not that valuable. They are sitting, they're just... And also, it's cheap because what is bots' time? It's some CPU time on some VM on cloud, right? It's cheap. So, how about this new world where the bot will notice things and the bot will ship things and we can go to Alps and hike? Yes, that's good. I mean, that usually works good, but unfortunately, unfortunately, we work on Kubernetes and it has some, sometimes there are some problems happen that sometimes they introduce some breaking changes, even in patch versions. I won't even talk about minor versions where contract may change or something will happen. And if some new features appear, bots usually don't expose it because bots can't read release notes. So, maybe let's optimize this bot and make him read the release notes. I don't know. Maybe automating bots to these release notes. Sometimes I can't understand them. I don't know how to write code for bots to understand them. I would much rather take some compromise here and bake in some flexibility in the bot. If I notice that bot has upgraded something wrong, maybe I would say, hey, how about you stop for a bit, don't upgrade things, and I will revert the bot's things, right? Revert whatever changes bots made. I have to make sure the bot is not super reckless and just doesn't... Like, I revert bots in and bot goes, oh, you're using wrong package. Let me upgrade again. So, you don't want that. So, let's... We should have a pause button in our bot. Like, it should be able to pause. There are other problems that we could... Other ways to solve this is like Semware. It is the next best thing after reading release notes. Yes, release notes give you much more depth in depth detail, but if the packages you depend on religiously follow Semware, they would not break contracts in miners or they would not make major changes in patches. So, we could make bots listen to only minor changes or patch changes depending on what package we depend on, right? As a next step, we could also say that, hey, how about you don't push to master? Our master is supposed to be much more tested, much more stable thing, send us a port request. If bot sends us a port request, we run a PR pipeline. We make sure everything works. Maybe have a person go through it and see. Maybe have a person go through release notes, check out deprecations and see if there can be any minor enhancements they can make. So, we could do all of those things. And not only we could do, we actually do. You can do this, all things. So, we do all the things. So, we have several pipelines that you can access by this URL or they will be linked to our Git repo. So, for example, in Docker release, we just say Docker will only use supported version of in Kubernetes. So, we won't bump it much. We won't bump it higher. So, we won't go on the latest. And in Kubernetes, it will use both. It will bump only for patches and send PRs because sometimes break scenes. And, yeah, for HCDN funnel, just go ahead. If it's a break, we will think what to do. And for GoLand, we'll talk about it a bit later because it's a bit more complicated. But how does it work? And it's like, can you tell us? Yeah. How would the bot notice things? Like, you might think it might be super complicated by its six lines of JSON in concourse. Thanks to concourse. Sorry. So, you just tell where the repository is. You just say, hey, use this access token. And there's this GitHub release resource which comes with concourse which can monitor releases on GitHub. So, whenever somebody makes a release, it will trigger a pipeline. And when the job gets triggered, what does the job do? It does exactly as I would do or any of you would do, but much more precisely. That is the blob dance. At least half of it, right? And then it will have to do some changes to the packages because ideally what you want is you want your blobs to have versions in them. As in, if you have Flannel and your Flannel file in the blob is just called Flannel, you might get confused like, did I really upload it or not? Did it really work or not? So, for visibility at least, you would ideally want Flannel 0.10 and next 0.11 comes, you would remove 0.10 and add 0.11. So, because this name changes happen, you might have to change something in your packaging, script and specs of the packages. And usually those are very, very tiny things. You just basically change them with said or if it's more complicated, you could do things. Our things are very, very simple, so we just use said. Yeah. Now you would be like, how does it push code? Just like we do. Just check out a branch and push, add everything and commit it, right? You could also make it push to master, but again, as we discussed, it's up to the package and how comfortable you feel upgrading that package. Now let's talk about the Golan thing that we used. Yes. Bosch vendor package. So, when I joined updating Bosch, languages was quite annoying and until someone pushed me, I was like, no, maybe later. Unless there's something important because I have to go in each release, update this blob and even with automation, it's doable. But luckily, Bosch team heard us, or maybe they had this problem as well, so they introduced this vendor package. What it will do? They actually maintain the release with package, with version package for languages and CFCI. And when new version release, they automatically add it to their release and we can use it in our package with simple command, which will be on the next slide. So, for example, for Golan, they support, you can either follow the major version or follow the minor version and you just run the same command again and again, which is this command. Very simple. If you want to know more, go to tomorrow's talk by Maya and Maria. It should be very interesting. And yeah, this is what we do in our pipeline and I want to explore a little bit more here, what's happening. So, we get the version of Docker release and why we do this because we want to bump it in Docker that we use for our unit tests. And that's what probably you should do as well, because sometimes on bumping of major packages, go becomes more strict and you don't want it to fail on your integration time, on your deployment time, but fail in your unit test time. So, that's why you should bump your versions in your Docker image that you use in CI. That's like a minor caveat. And one more interesting thing about vendor packages, basically all your releases, all your packages in all your releases can be vented. So, as for us, for example, we can use kubectl and we thought about extracting more small releases that will use kubectl so they can get this kubectl from our main release. And one of the benefits is that you won't have to bump packages in all Bosch releases and others that it will be single blobs, so less uploading, less compilation if you don't use precompiled releases. And it just works. And now when we bump everything in our Bosch release, we finalize it. Next step is the manifest. So, you want to bump it in the manifest. It's very simple. Again, concourse pipeline. Code is not readable, but you can read from our repo. You just get the release URL and run Bosch interpolate command for the manifest and push it. As simple as that. So, links to our CI and to our CI repo. Check our bump pipelines. They're very simple. If you still have questions about it, we can answer them now or we can answer them after the talk or ask us in our Slack channel. And thanks for all these sources for our images. And have a nice CF summit. And happy October.