 This talk is something that I put together, and fell out of the class that I talked once, called The Screaming Hars of JIT, JIT for Systems Administrators. No, I'm not gonna pronounce it JIT the entire time. I'm just kidding. Right now, I head up the Dev Ops Group over Future Advisor up in San Francisco, recently acquired by BlackRock. It's fun, it's exciting. Please apply, we'd love to have you. Okay, obligatory hiring pitch aside, we have a lot to talk about. Feel free to toot at me. Do you have any questions? Throw a hand up. We'll go from there. All right, let's get started. Okay, I'm going to start with a promise here, that everyone is going to learn something here today. I don't care who you are. If you're one of the GIT authors and you're sitting in the audience, you're about to learn exactly how I mistreat your baby. And one thing that I wanna point out here, I have a bit of a reputation for building talks around me making fun of things, and this talk is no exception. The difference is, is that every stupid thing that I'm about to list in this entire presentation is something I've done inadvertently. So, if you can't laugh at yourself, who can you laugh at? So, there's no shame in coming and admitting that you can't necessarily do something. Okay, so let's start at the very beginning. Who here uses GIT? That's an impressive showing. Who here wishes they used GIT? Who here is just raising their hand because they're staring at their phone and wanna pretend they're paying attention? Good, good. Okay, at its heart, GIT is extremely simple. It tracks changes to files and directories over time. It lets us collaborate with each other and share those changes back and forth. Let's work on teams that are distributed, both in distance and in time. But it's extremely simple, which is why in these talks, the next slide invariably looks something like this. Because, let's face it, there's an elephant in the room when it comes to GIT, so let's introduce it. It doesn't matter who you are. Eventually, GIT leaves you feeling like a moron. It gets very deep very quickly, but the only question is how long you can hold on for the ride before you've gone off the deep end. And it seems like there should be some reason behind it. Why does GIT get this complex? Well, Acropoe of Nothing. GIT was written by Linus Torvalds, who is famous for three things primarily. Yes, he did indeed. He names all his projects after himself. We'll see what the third one turns into. But in any case, at this point in time, Linus Torvalds is famous for three things. Right in GIT, which is why you all are presumably sitting in this room. Creating Linux, which is why everyone attends this conference and why everyone in this room, more or less, has a job. And for tearing people to pieces, a mailing list. Yeah. So GIT, despite its complexity, GIT does let you do some incredibly powerful things. Powerful, of course, being a euphemism for stupid. You can commit large binaries to GIT. You can commit atrocities in GIT. Tends to operate in a relatively, sorry, from a build repository first, shouldn't we? What an amazing concept, the demo guns. GIT in it. Now, local operations on GIT are really fast. So when it hangs like that for a little bit, you've probably, that's your first indication that something might not be quite right. I'm not as much of a genius as I thought I would. Okay. So now we see that in addition to committing an ISO, we have built out the GIT sub directory that tracks all of the changes that go on in here. And if we take a quick look at that directory, it's now 69 megabytes, which is a totally reasonable thing to stuff into a Docker container. But that's neither here nor there. So, okay, I've done something stupid. I've committed a large binary to GIT. That's okay. The GIT status, it tells me the file is on track. It's no longer being watched by the GIT repository. Huzzah! But notice we still have a 69 megabyte GIT repository. Changes in the past, things that have been committed to the repository, stay there forever. It's awesome. Yeah. So the way that we're not often committing large binaries to GIT, but what we do and we want to fix it, in order to want to wind this, you've really got two options. You've got a relatively complex command called GIT filter branch, which you can wind up going through and removing all references to it. And then you wind up having everyone else realize that, hey, there's a mismatch between what you think you have and what the server used to have, the rewriting history when you do that, which is bad. Or you can use a Java tool called BFG, which will go through and tear through things. It takes a little bit of time and it's annoying. Now, here in the real world, if you want to go ahead and store large binaries in version control, look into two different projects. They're designed to solve this problem and they do it a lot more effectively. GIT annex was written by Joey Hess, a former Debian developer, and Github wrote GIT LFS, which they are pushing harder and harder all the time. Now, most of us aren't going to be spending time committing binaries into GIT, at least after the first time we do it and get yelled at by management, but we're often committing things that we shouldn't be, like secrets. Replace large binary with private key for an API key for Amazon or an SSL certificate or a password and you begin to see the problem. Now, a future advisor, we are committing secrets into GIT. We're just encrypting them with GPG first and then using GPG renderer for Solstack to end up turning those back into a plain text. That's a sensible and reasonable way to get to work with it. But remember, you can go back in time just because it's not in the current revision doesn't mean it's not there somewhere in the past. There are stories about people exploiting systems by doing deep dive searches on Github and back on Google code when that was a thing. Let's move on to a terrible idea. You can, you know, I don't even know how to describe this. Let's just go ahead and show you. That's how we learned to type. So, remember I said GIT can make anyone's head hurt? So, get in it. It creates a .git repository. If we take a look inside, we have different things that point to the latest commit. It keeps track of different branches. There's a local repository specific GIT config. The usual things here, but these are all files that wind up controlling what the GIT repository does. Well, these are files. I care what happens to them. I should probably track them in something. The sad thing is that this works and you can go deeper. If you're ever doing a fine and things like this hierarchy turn up, something very wrong has happened. On a topic here, let's take a look now. If you've ever done this, you start typing a command like GIT status, for example. And then you have to turn away and look somewhere else to actually copy the cop repository, whatever it is you're looking. Then you come back, GIT status. And there it is, GIT's telling you you're a moron. There are ways to fix this. Now, what's in that file? Very simple, just GIT dollar ampersand. Sorry, dollar ads lab, good Lord. You can tell I'm losing it. And it winds up working really well. You can now get GIT status. But wait, there's more. Something very similar to this will make Docker, Docker, Docker a reality. It is turtles all the way down. Okay, so GIT does have its own alias support. You can build GIT aliases, but that doesn't always tend to go quite far enough because for starters, you have to preface all of them with GIT. So if we don't want to do that, we can back out into the shell and wrap shell aliases around lengthy and expensive GIT commands. For example, now, why in the world would I want to do this? If I do this in the right place, there we are. Who accepted this talk? Is that it opens new frontiers in being a jerk to the people you work with. I really feel like this should be on GIT Hub's marketing page. We can have the slogan, I'll license it via Creative Commons. So we've all heard about rebasing master. What that effectively does is it goes back and changes history. So everyone working on a repository suddenly doesn't have the history and the thing they're working on that they thought they did. It changes history so everyone else gets screwed. If you see a group of Ravens, they're called a murder. A group of whales is called a pod, not a docker. Thank you. A group of developers is called a merge conflict. Not a developer, big surprise there. I work in operations and I don't have time for this. So I use this. It's a lot simpler. Oh, we're just the department of no, it's easier. Okay, so we've shown that GIT can make us feel stupid. Why don't we let GIT help us feel a little bit smarter? Really wish there was a better way for Mac to transition more elegantly than that. Okay, type out something like that and then GIT tells you, hey, Jack, didn't you mean this other thing? Well, there's a way to fix that. Now, what that command does, and I'll break that down for you in a second, is actually kind of helpful, except for the part where it can screw you if you're not careful, but that's computers, right? This is more or less a type what I mean, not what I say thing. Now, what this is, we're setting a GIT config option. Note the dash, dash global. A small detour, there are three places that GIT config will alter things. Globally, which winds up doing this in your users.gitconfig in your home directory. The system level, which is an etsy GIT config, which is a really great way to confuse anyone else on that system. Or by default in the repository itself. And so that's kind of awesome. What the help.autocorrect does is it says, go ahead and if there's only one right answer that could pot that I, did you think I'm trying to type, go ahead and use it. Now what the eight does is that is the number of tenths of a second it will wait for you to hit control C before saying, screw it, I'm going anyway. We sometimes call that, yeah, we sometimes call that the penalty of having to wait because you type out something. Set that high enough and you can make it really painful. It's kind of awesome. But it doesn't work on everything because this winds up only working when there's one right answer and it's not as smart as you would like it to be. For example, how many times have you tried this? So you add a remote repo, things are great. You want to push your review, you want to push to where it's going. And it tells you that no, you if you want to push it to the same name branch upstream, you have to explicitly tell me. And it's frustrating. Fuck. And then it works. And it works again. This is a program that's been making the rounds. It's on GitHub. Most notably more than any other technical thing that they have done, they have managed to get the first Google result when you Google the phrase, the fuck. That's amazing. OK. So, humor aside, I do have three legitimately useful tools that I want to talk to you about today and they do tend to play rather nicely with each other. So, who here views management as perpetually looking for something shiny like a moderately intelligent squirrel? I see a few hands, including some people who work for me. That's good, good, good. As a manager myself, I have to say you're not far from wrong. So, when I first started at my current employer about six months ago, they didn't really have anything resembling a monitoring system, and we fixed that. But while it was coming up, we had a big monitoring board next to my cubicle that was dark. So, what we wound up, what I wound up putting up there instead was something very similar to this. It's a program called Gorse. And what that does is it lives in a Git repository. You're running a Git repository. It's an open-source software program. It lives in just about every different type of, in any type of package management system you've got for every operating system. And what it does is it provides a visualization around every bit of history. So, that happened in that repository. So, we would see fun things, like our CEO way back in the day going ahead and making some changes. And then our founding engineer would go in and immediately revert all of them. And it became a great, it was a great way of looking at where we had come from, where we had, where we've been, where we're going. And once we replaced it with a monitoring board, people didn't come by and look anymore, because they would just like to come and tell stories and marvel at the history of what we have done. So, that was kind of nice. Don't ever doubt the power of being able to show something shiny, even if it's not as informative as you would hope. G-O-U-R-C-E, it'll be on the board at the end. I'll link to all the tools I'm talking about. So, does anyone have more than one Git repository that they care about? Unless you work at Facebook two years ago, your bread's probably out of stock. They were famous for having a single monolithic repo. And it sucks, because when you have 15 or 20 different repositories that you care about and you get on a plane and you realize you're missing the latest changes to that one repository that you really need in order to get something done, like put your slides together for a conference talk. So, Mark Atwood posted a decent fix here. Specifically, you just punch a find command into a for loop to find Git repositories, iterate over it, you fetch the latest changes in every repository's parent, and I would think half of you just fell asleep. So, this is really awesome, except for the part where it's shitty. Because I'm going to do this once, but I'm not gonna go ahead and run this every time. So, I tweeted back at him. So, I took his stupid loop, but instead of just running a Git fetch, I ran a different command. In this case, MR register. And then I don't have to remember it anymore. And he tweeted back, this is awesome, thanks. And the point here is that we're always learning. Mark is the director of open source at HP. Maybe HP Enterprise now, I haven't been keeping up with the various changes. And this is a guy who's forgotten more about software than I'll ever know. We're still always learning about this one weird trick that no one has pointed out to us before. We're also usually not the first person to encounter a particular problem. So, when you start realizing this is annoying, there's probably a better way, ask people. So, let's talk about what MR is and does. It used to be called MR in the tradition of open source projects, of having a name that's virtually impossible at Google. And then they said, ah, this is actually a really handy tool. Maybe we should change the name, which makes it easier to find. But now I never know which is the proper name and I invariably use the wrong thing. They've managed to take one of the hard problems in computer science of naming things and made it worse. So, step one is you can run, you run MR register inside of each repository you care about. You can do it by hand. You can do it using a for loop, like the one that was on the board a few minutes ago. And it doesn't really matter because you only have to do this part of it once. It goes ahead and it builds out a file called .mrconfig, the typical Unix fashion, one of your .files. Then you go ahead and run MR update and watch as it updates your repositories. Specifically, I give it MR list. It spits out a lot of things that didn't show up because of the weird resolution on this. All right then. Then I tell it to MR up, which, if the conference Wi-Fi behaves, no guess, there we go. It starts in series updating each and every one of the repositories that I care about. If I take a look at my MR config itself, note that there's an include there at the very top. Note the include at the very top where I'm including my entire .mrconfig directory, which means I can drop individual files into there. So, if I have, let's say something for the front end team and there's a list of repositories they need, I can build an MR file and just pass it out to them. They can wind up using this. I can also, if you take a look at specific environments, for example, you take a look at puppet or salt, you see a bunch of different repositories that in turn wind up being given specific instructions of where the parent lives. Yeah, MR is not, the list is not working well with Teamux at the moment, but it spits out 30 different repositories, which is really the relevant part of it. You can also break this down into groupings by going into subdirectories. If I go into my workspace directory and do an MR list again, and type it properly, in turn it breaks it down to 12. I've never seen it do this. There's something strange with the projector, anyway. So, at this point, that winds up letting you sub-target among different groups. There is a slight movement toward getting actual better grouping logic handled in this, but it hasn't gotten anywhere yet. Patch is welcome. And it's not just working with Git. It works with virtually every version control system that you can think of. The notable exception that's not on this list is Perforce, but Patches are welcome, which in the open source community is how we say, go and get, you bother me. I mentioned how we can limit operations by using the same directory structure. I tend to like hierarchies for things like that. We're on the front end directory where I just have those work repositories and a back end directory, and they both live in a work directory, so I can just go drill down a lot more effectively. Now, I wouldn't know that this tool existed if someone hadn't told me about it, and that's really what part of the community aspect of open source comes down to. A Stodgy German friend told me that it existed, and I thought this is ridiculous. Why would I ever use, oh my God, that's kind of amazing. Now, I bag on them a lot for this, but Docker absolutely nails this. You can't get people to stop talking about Docker, and that's kind of fantastic. Anyway, my crazy German friend told me about this in conjunction with a tool he himself had written called VCSH. This is effectively a version control system that built on top of Git for your home directories. Who here has a global .files repository where they keep their things like .zsh or .bash hersey, their .vmrc? Right, it's a common thing, and it usually looks a lot like this. There's countless numbers of them on GitHub, and it's fine, but there are aspects of this model that are kind of crappy. I don't want the Git repo that has my personal AWS credentials in it. Yeah, I know, I keep those in Git, I still do terrible things on my work machine. And vice versa, I don't want my work credentials from a Git repository living on my personal machine because that's a great way to get fired and sued. So what I do instead is use something called VCSH. Okay, yeah, the listing function in this works at least. Okay, what it does is it lists out a different series of repositories that are controlled in this. Let's, for example, take my Git config as a great example. I run this, it then winds up instantiating a shell session underneath where I am now, and if I take a look at where I am, I'm still in my home directory. I haven't changed anything about that, but it does take advantage of how Git works internally to wind up with a Git root that lives somewhere else. And if I do a Git status, my .git config has been modified, but there have been all kinds of file changes in my personal home directory that are not being picked up. The reason that happens is if I can't .git ignore, I have a wild card in it. What this means that in each repository, I do need to explicitly force at any file that I wanted to track over time. That's a little bit of a hurdle, but it winds up paying for itself later in time. The idea is then I have my Git config living in a completely separate repository from my SSH config, and neither one of them know anything about each other. I can make my changes, but I'm ready to go. Then I hit Control-D to log back out, and I'm right back where I started in my home directory. I run Git status now, and it doesn't see it as being in a Git repository, which is handy. It will also avoid the submodule problem among other things. So what this does under the hood is you have your home directory, you have your home directory, and right under that, you have a .config directory. This is stolen from some of the Debian people. They tend to like this particular approach to things. I'm lukewarmhearted myself, but it works. The idea being that we then wind up spreading things out, so you have a list of things that are available that VCSH can manage, and you can pull these with MR, which is typically how you would handle this. And the ones you want on this machine, you quite simply have a sim link in your config.d directory that wind up pointing back to the original source of the repository. So you can effectively turn repositories on and off with this. Therefore, because I've never pulled things except for what's linked in config.d, while I know where those other repositories live, I never have their data locally. Then in turn, when I wind up checking out the repository, it does under the VCSH sub directory in repo.d, which then is where that repository actually lives, but checks out its files into my home directory. I have one more trick that I do wanna show you that has been extremely handy in that as I've gone through GIF, a lot of people have had, have something that does this already, but a surprising number don't, so I'm going to point this out. Now you've been seeing it throughout the entire presentation, but I haven't called attention to it. It tells me what, you'll notice it next to my prompt and the ridiculous Dockerized hostname that's a hex character that winds up being populated. It tells me that I'm on my Git master branch and there's a red one with a dot next to it. And what that tells me is that there's a file that's been modified and it's been added to Git staging area, but it's not yet been committed. So I go ahead and I make that commit. Now my prompt has changed and you'll see that it still has a green check mark which tells me that things are decent, but I'm one, but I still have a commit that needs to be pushed up. Notice the one with the up arrow next to it. It tells me that, and Git status tells me that I'm a head of origin master by one commit, use Git push to publish those commits. What this, what having that in my prompt does is twofold. One, I have to type Git status a whole lot less and two, it gives me a constant sum of reminder maybe I'm not on the branch that I think I'm on. If I'm about to make a commit to a feature branch and I see the word master there that should set off a quiet alarm bell in my mind. Now increasingly, since I first built the original version of this talk, GitHub and others have made it harder to forcibly push changes to master if you don't, if you disallow it on a particular repository, which is great, but not everyone enables that. Not everyone can enable that due to various workflows. And ultimately it tends to not be particularly a panacea which is why I do like having that subtle visual clue. So here's a link at this point to everything that I mentioned today. And I'm gonna leave that up there for people to take pictures of, jot down while I answer any questions you folks may have. I keep thinking their hands are raising, nope, just pictures. There we go, hey, a hand, unless there's a camera in it. What's up? Okay, to repeat the question, I was asked what graphical interface am I used to manage Git repositories? I come from a sysadmin background, I use the Git command line client and that's it. Other people like things like Tower, some people like I think Git K is in somewhat frequent use, but personally I've never made the leap into using a graphical client for it. That's mostly preference at this point. From my point of view, I can always drop into the command line tool if I can solve that, then what the graphical elements do don't necessarily matter. I do understand that's a highly opinionated perspective for me to have, but I'm not, unfortunately, I'm the wrong person to ask. Anyone have something they use for a graphical client they love? Yeah, I remember Lassian did have something I know. What is it, source tree? Yes, yes, yes, I think someone at the office might use that, but I wouldn't swear to him. Sorry? Magic? Fugitive, oh well, VIM fugitive, yes, yes. Yes, yeah, you can do that too. Ultimately, this is one of those things we can wind up, you can take a survey of a room like this and wind up, and spend the next 20 minutes listing things on a board. There's a lot of answers in that space. Find something that works, if that's the direction you want to go on, I can suggest finding something that works for you, try them out, most of these things are free. Of course, other questions, comments, concerns, criticisms, snark, hit me with it. Well, you know one of my favorite things to do in the open source community is disappoint, really, it feels like I'm standing through a room full of my dad, it's great. Hobson's choice I was given was sub modules or sub trees. I really don't like either of those. Sub modules are annoying specifically because it's not always obvious what they're doing and how they work. Sub trees are supposedly the better way of doing things. Personally, I avoid it with things like VCSH, which if you leverage it correctly and move it away from just a home directory, you can use something like that that works a bit more effectively. But it's one of those awkward questions that everyone happens and haws when you ask them because there really isn't a good answer to that. For those who aren't familiar what sub modules are is you put a git repository inside of another git repository, not in the crazy way that I did inside of .git, but if I go down to the level and then create more repositories, how do you wind up keeping the parent repository informed of the changes in the child? Yes, what I use for that is something called ZSH, it's a ZSH specific plugin, which I'm noticing not as many people are using as down here as they are up north. So I tended to, I tended to not necessarily call that one out specifically. Some people love PowerLine for that, other people use AirLine and there are a few more that go along with that similar naming convention. I'm sorry? Bullet train? Yeah, that's one of those situations again where there's 50 different answers to that question. But again, the way I picked the one I found was I just started playing around with it one day and moved through a series of different ones that I found when I liked. If you're using ZSH, feel free to tweet at me, my Twitter handle's on the next slide and I will go ahead and dig out the one that I'm using and get an answer for you. Other questions concerned? You again. Good question, a little off topic, but are there valid reasons not to use ZSH? Well, again, a lot of what I, the same reason I don't use a GUI, forget, is because I spend a lot of time logging into systems that are not directly in front of me and in order for me to use things that I'm comfortable with, I should at least understand BASH. So if most it comes down to portability, if nothing else, BASH is everywhere, ZSH is not. Other questions, concerns, snark? Thank you all for coming.