 You you ready perfect right on time so run off last for Lucy and she's going to be talking about DevOps and Distributing DevOps for fun and profit. Yes Hello, thanks for coming So my name is Lucy and one of the well my team I worked as a bunch of things We maintain a bunch of Services and platforms and tools that the rest of the developers use and one of those is a thing called PSC Li or Piscali Which is a CLI we've written in go that Basically, I like to think of it as a user experience layer for launching a bunch of darker containers containing Tools that the developers want to want to use So and Before I talk to you about this tool I need to give you a little bit of brief background about where it came from Didn't just bring it out of nowhere Gonna talk to you about How it started becoming really really useful when we made this CLI in go and Then talk a bit about how we've started open sourcing it because this is fast and after all So in the beginning the problem we had that we need to solve was new starters at the company we needed a way of Giving them a single thing that they could install and run all the tools they needed We didn't like the idea of giving them a document saying install this configure this here's some settings No, we wanted them to have a single thing that would for the most part just work and Our initial solution to that was a vagrant image that contains a bunch of stuff And that worked for a while, but as the company grew bigger that wasn't really a sustainable solution For start teams across the company are sort of semi-autonomous and what that means for us Is that while they might use the same tools? They didn't necessarily use the same versions of those tools Which for us meant potentially different versions of VMs for different teams, which is not a sustainable solution These things are fairly big which in noise developers when they have to download a new one And they take a long time to build which is not a problem in and of itself because you automate that but it means that the testing cycles when we create new versions this are really awkward But the biggest problem with this was that it took ages to start up and for developers to update to the newer versions of these Which basically meant that they didn't and A lot of the issues that they saw is to do to having Old versions of stuff and not having the same version on their laptops as on CI So about this point in the timeline when we were getting bored of this Docker for Mac came out of beta and One of our team came up with a brilliant idea of why don't we just replace this VM with Docker and That looked like this which Yeah, it works But that's not a thing that we want to give to our developers and we're getting developers to update a VM every now and then is Awkward enough making sure people are running exactly this and Making sure hundreds of people across the company are running exactly this is Not going to happen So while this worked in our team and it did everything that the VM did We needed a better way of distributing it Which is where go comes in So we created a tool called PC li which is written in go It interacts with Docker SDK Same code that the Docker client uses It uses Cobra and Viper to wrap stuff up into a nice CLI developers can just run and There's a lot of nice nice library code that we've got in there as well That we can run prior to launching a container So some of the nice features we get out of this It's able to update itself so developers Previously updating that the end was a pain in this case. It's just one command and updates Tools that are run with this Run against whatever is in your current directory, so you don't have to clone your code twice Once for outside your VM once for inside it But another feature we've been able to add to this is that it's able to run without Needing your code to be in your current directory. It's able to run against a git repo which stores in a sidekick data container Which in CI is helpful Another thing that's useful for that is that you can there's a parameter that we've added to it that lets you use a remote Docker server But all of that's not particularly interesting on its own because there's lots of things to do that and The reason I think that this became incredibly popular and my company is that We are able to add arbitrary code prior to launching a container Some of the things that we do fairly frequently in this tool include authenticating the user with vault Which lets us get access to secrets that the container might need An example of that is anything that needs access to AWS we Use vault to generate short-lived credentials which our security team loves us for because now there's no excuse for the most part for long-lived iron users But it also lets us do nice little things little User experience things stuff that in workshops. I've noticed people make mistakes or find frustrating Like if they've not configured get or if they've not got an SSH key So they can't clone anything from bit bucket yet. This will detect whether they've got that and prompt them Tell them what they need to do So they don't have to Ask why is this broken because it'll tell you and Then once we have this tool and basically all the developers had it We realized that by accident we managed to distribute a thing that lets us run Any code that doesn't require containers at all. So there's a bunch of stuff in here That doesn't even use containers at all But one of the most interesting things that we've found by using this is We've got anonymous usage statistics in this so we know which versions of this tool people are using we know What people are running with this? so we found out recently that 25% of people in the past month are using the latest version of this and a further 35% are within a few patch versions of it, which was unheard of back when we had a VM that took 45 minutes to update So I'm going to show off a couple of things that we have in this can I get our friendly neighborhood mic stand I Feel like I should make people pay for this I'll pay you in stickers So some of the stuff in here So most of the stuff in here requires access to the work network. So disclaimer. I'm faking some of that So it's built in Cobra. So some of the stuff we get for free includes tab completion So some of the commands that are included in this there's a lot of them most of them running containers Bunch of global flags that are common to everything I Mentioned this thing has the ability to update itself So there's two commands that are part of this first one PSC live version tells you which version you're running And checks artifactory where we store all the go binaries for this Check if there's a new version and PSC I update updates you in place of the new version the incontruable go updates library is pretty good for that So, oh no, I'm on 7.0. That's not the latest version. Let's fix that and now I'm in the latest version. That's like Significantly quicker than updating a VM So what else I've gone with a very complicated example of One of the containers that we run in here, but it's a fairly common thing we do in this tool In this case, this is the AWS shell, which is something that Amazon provides as a wrapper around their CLI but The reason I'm using this is because it is an example of us Authenticating the user with vault getting secrets getting a list of all the accounts. They have access to generating credentials and Then at that point it knows what it needs to do to launch a container So it launches the container Binds your AWS directory, which contains the credentials and Also contains the history that this tool needs So I'll show you what that looks like So if I say basically AWS shell First it prompts me for my help that credentials and that's definitely my real username password hunter to How are you typing? Magic. I'm psychic. So there's all the list of accounts that I have access to I picked dev account I say I want admin access to the account So it's going to download Docker image This was recorded at a slightly higher resolution. So that normally looks better than that but then at this point, I'm in the container and I just press up and That's the last command I ran the last time I had this container on my laptop In this case, it's just checking the content of a particular S3 bucket But yeah, so that's an example of what launching a container using this thing looks like sure a question Yes, those come from vault as well So the question was when it showed the list of roles, where do those come from? And yes, those are One of the features of Hashi called vault as well. So That's where that comes from as well as the list of accounts So another example said terraform. This is something we use to manage our infrastructures code and This is one of the things that teams across the company like using but they are not necessarily on the latest version so Hashi corp has a fairly frequent release cycle, which means that When they release a new version, it's not necessarily backwards compatible So we so if I do If I'm gonna need my backstand because this one is one that requires me to actually type So if I ask it which version I'm currently running I'm on one oh 11 one Two hands typing is easier. I can also ask it what versions are available. So those are the versions we've added over time And if I can if I specify that I want a specific version It'll use that So, yeah, and that applies to basically anything in there But this particular command this is one I use a lot in the workshops and I've seen the frustrations people have So I've added some nice usability stuff to this when you saw the list of accounts for example Anything in this tool that uses AWS you can specify the account as a flag But you can also use a config file and by default it will check for a config file in your current directory Or it'll check for one new home directory if you can't find it And also because we manage all the AWS accounts, we know which roles exist. So I'll show you what this looks like Here we go So have a look in the config file It says that the code that exists in this directory corresponds to that particular account So if I run this in debug mode to get some more logging. So what we see We see that it uses this config file from a current directory. It's chosen to use that account Because we know how this tool works and we know which roles are available in each account We know that plan only requires read-only access So it drops me down from admin which I chose earlier down to read only and the rest is basically the same sort of thing Runs this in a container in this case. It's just some test DNS records. So that's not particularly interesting So that's that part I said this is open source and Basically this tool has sort become the victim of his own success at work People have been asking us. Can you add this other tool to it? And we've been thinking. Yeah, that's a useful thing to do in something like this But that's more stuff that we have to maintain and we don't want to maintain lots of things We want to maintain a small number of things So there's some refactoring work we needed to do with this anyway So we instead of just refactoring that in place we Split out some of it. So while there's a lot of cool cool stuff in PCI at its heart It's a thing which runs Docker containers and functions in a nice CLI So what we've done we've abstracted away the Cobra and Viper and the Docker SDK stuff that is part of basically and Put that into the open source Kelly and This lets us do stuff like setting sensible defaults like we basically know that anything that will be written with this Would want the opportunity to run from your current directory or from a git repo possibly in a remote Docker Stuff like that, but their philosophy behind this is that while we are doing stuff by default We want to let people access Native Docker stuff and use that Preference if they have special stuff that they want to do So demo time again So I've got a version that uses this called loosely named after myself because I'm vain and So I'll show you what one of these things looks like So main is not that interesting. So I'll show you the root package first thing it does I'm defining CLI variable which is available to all the other commands in this new CLI I'm calling it loosely Setting an init function which just sets short and long descriptions. That's does the help text stuff I could do all sorts of stuff here. I could define a bunch of global flags, but for this demo. That's all I need So if I run this It'll show me all the tools available into it And a bunch of global flags so Docker host again a bunch of stuff for interacting with git repos if I want to and Yeah, so I've got so show me before I show you how this thing launches containers I'm going to show you how it does simple functions and so My version function I've got so from my CLI. I'm defining a new sub command version Which just prints out which version we're running I Define a function that just prints out a bunch of variables from my From my version library, which I set at build time when I compile the binaries And then the task Which is the run function of this command just runs this function So if I do tells me it's running o3 1 that global debug flag and Where's it if I specify that we see a bit more information when it was built which get commit it was built from Terraform again I'm going to use this as an example mostly because I really like Terraform and I use this myself To manage my live DNS records So what does this one look like? This one is slightly more complicated. This one actually does use a doc container So again command is a new command from this CLI. I'm saying Okay, yeah, can you hear me? Okay? Fantastic You may sit down So Yes, in this case defining a new command off my CLI short description shows up in the main Help her text long descriptions shows up when I ask for help text for this tool specifically And I'm defining a flag and in Cali, which is the open source version of this a Flag corresponds to a Cobra flag and it also corresponds to a Viper config So I can specify this profile field Both when I'm running my tool and in my config file and you saw what that looked like when I did that with Terraform earlier So I'm defining my Task for this which in this case is a string which Cali interprets to be a docker image So by default it will download this image and just run it I'm also defining an init function which happens first in this case It just takes the value of that profile flag and passes that in as an environment variable So I'll show you a bit about what that looks like So I'm going to use a git repo for this This is a private repo that only I have access to so it's going to use my SSH key to clone this and I'll use the same commands that I did before plan in this case I've previously cloned this so this Sidekick container still exists. It's just checked whether it's up to date, which it is and It's going to check if my DNS records are up to date Which I think last time I did this they weren't but I'm not changing a live DNS records in the middle of a demo so let's leave that and Another one this is the simplest possible example I could come up with for running a container in this I'm using Vim just because I like Vim So the command for this looks like this So All I'm doing to find a new command off my CLI set the task to this image and I could have to stop there I could have used an alpine image with Vim installed But of course this is not normal Vim. This is Vim plus all my configs or my plugins It's even got go format in it Which means when I save it the Vim go plugin will automatically format that for me because there's no way I will write go code that doesn't automatically do that for me. So, yeah There's a lot of stuff that I could do with this I could rewrite the entirety of what we've got in PSILI using this if I wanted to and why will at some point I've got a dedicated 10% of my time at work that will be doing that But for now, that's probably all I have time for So I've explained where we come from with these massive VMs that weren't particularly scalable I've explained where we are now with PSILI being this CLI that runs Docker images and I've shown you a bit about what the future holds for us with Cali as this Abstraction that this library that we can just use anywhere we want to make a tool like this Links to source code on screen now the Cali library itself my loosely example There's another one. I don't have time to demo today that someone in our company has made in the past two weeks Thank You Alice This is called statically and it's what a couple of us are using to manage our Technology blog at the moment and of course Ashley McNamara is fantastic go for images without whom this would be a significantly more boring set of slides So yeah, that's all from me unless anyone has any questions. So I say that again See that right. So the question was how do I make sure that the user inside the container is the same user? user ID as outside the container and this is something that I deliberately simplified for this talk on Mac OS, it's simple. It doesn't matter because Docker for Mac makes that simple on Linux We do actually have some extra stuff that we need to do so I'm figuring out a way of standardizing this in Cali We it's a solved problem that we already have solved at work Which is you just pipe in the user ID and and you use a group ID as environment variables And then you have something in your entry point which makes Make sure that the user IDs match. It's not particularly nice, but it works Is it so the The statement was that that's not particularly secure because you can just pass in any environment variable you want and yeah, that's true It's not great. It's not it's the solution. We have at the moment We need a better solution in future, but that's what we're working with for now anyone else Sure So the question was how do we Decide where to mount the volumes from right? So stuff like the dot alibis directory with the credentials and your other stuff in there We work on the assumption that those are stored relative to the user's home directory at the moment We haven't found anyone at work that stores them anywhere else if we do then we'll add something to make that configurable And Everything else stuff that runs from your current directory. We just say use the current directory And if it uses home directory use that Yeah, I hope that Answers your question So the question was when we mount the current working directory How do we know where to mount that in the image and the answer to that is usually that We mount it into a directory usually referred to is usually Temp slash temp slash workspace and then that is the workspace that we Use when we're launching the container. That's usually how we do that If we need to do anything special for a particular image, we will configure it per image But that's the default that we usually go with Thank you. Oh and I have stickers for the logo if you want some because stickers are important And the stickers will not be on this desk They will be outside just so you know If you want to give a bunch of stickers and we can pass them around we can throw it with it