 Renounced. Okay, what you're all you? Yeah. So hello everyone. My name is Arami Alhami. I'm a principal software engineer at Cementic Corporation. I do DevOps there to take care of the public infrastructure and do DevOps operations day in and day out. Helping me present today is? Most of all got me his son and his protege. So today we're gonna talk about people always say, don't bring your work home. Okay, well how many actually adheres to that here in the audience? Oh, we have a truthful audience today. Awesome. Okay, so this is a version of bringing work back home. But before we dive into that I want to talk about some some some other when you go back in computer science go to the early days. Some of the people who put a lot of the hard work in and have a lot of insights into how the development process works. And one of my favorite codes from Larry Wall is mentions the three great virtues of a programmer. Okay, and I believe it encompasses everything about a DevOps person. First of all, I am lazy. Right, and I'm saying that and my VP is staying in the standing in the back there. So I am lazy and the reason I'm lazy is I want to write code to solve all my problems. I want to write it once. I don't want to deal with it later. Someone wants to deploy something. I don't want to wake up in the melanin and do it for you. I'm gonna write code to do it for you. If something breaks, I write code to prove that it's your fault, not mine. And that's how life goes on. Also I'm impatient. If the computer or the system or the automation doesn't do something the way I want it to do, then I need to make sure that it does that. Okay, otherwise I'm gonna spend more time on it. The idea is laziness need to conserve energy. Impatience, I just want to move to the next cool thing. Right. And hubris. Nobody can ever tell me what I did is wrong. Right. And some of the people who work with me know I have the three basic rules. But in order to prove myself right, I always need to write code that always executes properly. So with that, we're gonna start going in my day to day life. So I go to work. I go and I start my day. First thing I do, I do an Ansible run on the stage environment, just to see who was playing in Pune and who in India and who was playing with the environment. And I go in and I start seeing what all these changes that happen overnight. Because the last thing I do in the day is actually do another Ansible run. And for those who are using other automation, I'm sure this applies to you. But you want at any given point in time, especially if you're not updating packages, that this always okay is the sum, change should be zero. And what really bugs me is not all these changes. What really bugs me is this is 32. And this is 17 from the same class of servers. So someone really went nuts on that box. Okay. So kind of my morning starts like that. I start getting angry. I start opening outlook. Unfortunately, enterprise. So I start opening outlook. I start composing my angry email. And I start saying, guys, who logged into the box and change something? And I'm trying to compose myself while I do that. So to take my head out of things and cool off a little bit, I go to Terraform. And those Terraform is infrastructure as code. It tells me what change and I have five things that need to be added. So people actually deleted stuff. And nine things that need change and one that needs to be destroyed. It was so off base that Terraform gave up on it. Okay. So now I am really at a space where I'm going to say, I don't know who you are. Yes, I'm going to find you. But I'm going to go and I'm going to rotate your SSH keys. And you're going to keep working two days to figure out how you're going to log into boxes in our system. Okay. So that happens. And, okay, that's my midday. I composed my email, rotated the keys by the afternoon. Now it's time to go home. Right. So there's no place like especially if it supports IPv6. But I go home and I start looking at all these devices that I have at home. And these are just samples of the devices, not including device counts. Right. So you have your set of mobile devices, Android iOS, your voice assistance, different IoT devices, your laptops, your media streamers, your raspberry pies, the different things that you find hidden somewhere in your network. Right. And you start looking at that and you realize that actually a slash 24 is not enough. Your home network now is a slash 16. And that's something you're considering moving to the hundred dot IP space because one night, one night two is not, it's not cool enough anymore. Right. So the device count in your home network, it's becoming an infrastructure. The number is approaching and factorial. Okay, you have network bandwidth requirements right now. And also, everything keeps changing, kids come in, guests come in, and everything's changing from under and on top of you and you're under pressure to keep things working. And you need to maintain security, all the IoT stuff, you have your personal information at home, you have your photos, you have everything there that you need to keep safe. Also, you need to make sure that you have your configuration in the right place. You have, for example, you have you may use your same laptop at work, for example, at home, but at home, you open up your browser using a different persona. Okay, you may have to use a different set of dot files because now you're committing to GitHub as opposed to your internal game. So you need to maintain all these things. And you're not root on all these boxes. Imagine all that happening at home. And suddenly after coming back home, I'm back into my OCD. Right. I'm back. Okay, dad. It's okay. That it's not work. It's scale. Okay. Thank you. Thank you. So all that stuff triggers OCD for you. So you actually need to go back and try to streamline. That's what I mean by bring your work home. Actually, go home, bring DevOps back to home right now. Okay. And what we're going to present right now is basically an opinionated approach of running DevOps at home. Okay. Anything that I'm going to discuss right now, I am sure there are at least three people in the room who are twice as good at me at anything I'm going to discuss right now. Okay. However, this is how we framed together and how we made it work and how it works for us. Just like people, for example, some companies package Postgres, some companies package Kubernetes as their opinionated approach of running it. This is my opinionated approach of running home. Now, there are certain trade offs you need to keep in mind. The first trade off is security. And when it trades off with convenience, not everyone at home is a software engineer. Not everyone at home can make that balance work for them. So you need to work between security, convenience, usability. Yeah, I'm going to put all my IOT in a network. But if I want to turn on a light bulb, I need to switch Wi-Fi and then turn on the light bulb. Okay. So it defeats the purpose. Also hackability. So do I buy the thing that I can flash and change the ROM and update it but it's going to break every two weeks? Or do I take something that I can really work with? But it's not as hackable as others. Also software licenses. We're in an open source conference. People tend to go with open source tools as much as they can. But sometimes you may go lenient just because it works better for you. So the first place we're going to visit is the home network. Okay. Now there are two main components in our home network. The first one is the Ubiquiti Edge Router X, a wonderful device. Okay. And a set of routers. In this case you can substitute either two Wi-Fi endpoints by the setup that we have right now or one access point that can actually have two networks, two SSIDs that are actually separated. Now why do I need to have a home network that is working? If something goes wrong in production. Okay, in a company. If something goes wrong in production. What will happen? I'll get an alert. I'll have 15 minutes to respond to pager duty. Then I'll look into instigated. If the internet goes down at home. My biggest fear is it's 15 seconds before someone starts screaming, the internet is down. The world is ending. Right? So it's a production network. Okay. And most of it's going to walk us through the network topology. So basically it starts at the ISP modem which transfers the internet to the Edge X Router. And the Edge X Router has four networks. Two wired and two home and two wireless. One of each for the IoT and the home network. This brings the question, what are the features of the setup? The Edge Router X is a good choice because it's less than $50 and can have up to four isolated networks. The networks are semi-isolated which means that the traffic from home can go through to IoT but the traffic from the IoT cannot go through the home. This is a good thing because what happens if your light bulb gets hacked, you don't want the rest of your home to get hacked. And this is a reference that we got to help with getting this information. And next we're on to configuration management. OS configuration and DNS. Who here does not know what configuration management is? Raise your hand. My dad's going to get mad at you. So yeah, I'm going to get mad at them, especially that one of my interns actually raised their hand. So the second topic that we're going to discuss, thank you, Moosop. The second thing that we're going to discuss is your personal infrastructure as a service. So again, this is my experience based on the tools that I use and I have knowledge with. So for provisioning my infrastructure, I use Terraform. So in order to make things work for me, you know, you have the set of raspberry pies that you have set up at home. But every now and then you need a public server sitting on the internet. So my vendor of choice in this case is DigitalOcean. Provision the instances using Terraform. You get amazing DNS capabilities from Cloudflare. And I use Ansible for orchestration. And the link to my orchestration code is available in the bottom here. And so all this is great for the infrastructure that I have. The next step after that is, what about home? My IP address changes. I don't yet have business internet at home. So which means that my IP address is not very static. So there is this amazing service that's called DuckDNS that's deployed on AWS. It is wonderful dynamic DNS. It works with every platform known to man. If you use any kind of IoT device or if you're using any kind of ROM for your routers, or if you have Linux windows, et cetera, there is a command, there is an executable they can use that to run on a cadence that will always update your DNS. You can, it's so good you I use it on my laptop for a while to figure out where my laptop was traveling over time. So it's a really good service. Alerting. How many people set up New Relic or Dynetrace in their home network? Okay, because I want to stay away from you as much as I can because I don't know who will deploy Prometheus on their home network. Okay. I'm sorry. Stranger danger. Okay. So for alerting, I just use a basic, I use Slack. It's good enough, it does my basics. And I'm going to take you through an example using a tool called Slack T that's developed by course hero. It makes life easy. Okay. And then the last thing is a chatbot. What's DevOps with a chatbot, right? So I use her bot and I thank Justine here who actually introduced me to that piece of software a couple of years ago. And it's amazing. It grates with Slack. Anything grates with other backends like IRC. I know people don't know what IRC is here, especially that Slack is actually turning off IRC. But I am on IRC. I admit that. So it has different plugins into different networks. And this is one of the cool things I like about it, written in Python. So it's a wonderful tool. So I'm going to take you through some of these things one by one. And hopefully we will learn some stuff today. So the first thing is the Slack T alerts. So if you look here at this command, basically, I'm calling the Slack T shell script. So it's actually a batch script. I'm calling it to send an alert to Slack with an attachment that says danger that reboot required. And the condition being that I use I'm a Debian user. So if the reboot require file exists, run it at a cron job every two hours. If it exists, I get an alert. And if you go to Slack, and you go to that channel where that is posted, and you change your notification setting saying, whenever a message is posted alert me, that way you will know whenever a box needs to be rebooted. Okay, because unlike in production environments, I do I keep automatic updates going on. Okay, the follow up example is, if you set up an email server, sorry, an SMTP server on your box, you always get an email that looks like this from your box whenever you do an app update. Okay, it's called at list changes, right? It sends an email whenever that happens. What I do is actually do a loop where you loop the email into Slack T itself. So the email gets sent to Slack T Slack T pipes it to Slack. I don't want to read any more emails. I get enough of them already. Right? So just use Slack instead. Okay, and the code that I posted before for the Ansible for the Ansible orchestration cover this component to for Slack T. And cool things you can do you can choose one color per host, you can give it different bot names. So it keeps you sane. That way, your boxes are updated. You don't need to worry about it. You just need to check Slack every couple of days. It's already open. And then Erbot. Okay, so Erbot, you can write multiple plugins in it. One of the things that buddies at work asked for is like, we want to know what the cryptocurrency price right now is. Okay, well, not anymore, right? But but I just wrote a small plugin that's called that says you get a ticker, I call it BB eight. And it comes back with the current prices. And this is on the 18th. So don't sue me for seeing these prices. Then we're going to talk about dot files. dot files is the bane of everyone's existence. You have them right. You have them right. Once you don't want to touch them again. Because if you change one flag or Tmux just deprecates one variable, you're lost. You don't know where to start. You don't know where to end. And that blog that you checked for that specific purpose doesn't exist anymore, right? So what we're going to use here is a wonderful tool called yet another dot file manager. If you don't get anything from this discussion, okay, except this, I feel I had a win. Okay, this is an amazing tool. Okay, that takes care of your dot files. And what it does basically, it wraps around get it doesn't try to reinvent version control. It doesn't try to force you to version control the way you want. It doesn't ask you to put your files on a specific directory and sync and sim link them. No, every file stays where it is, where it belongs at all times. It wraps around get. And then you just use the get syntax. So if you want to clone your your your your dot files, you just say yet him clone. If you want to push, you do yet him commit yet and push. You don't change anything. It just pipes the it connects with get it wraps around it and you're done. And it's a single repository. So you don't have your vim repository. You don't have your team x repository. You don't it's one place to keep them all. They're all host aware, which is my favorite feature. So what happens is your dot files on your Mac are different than your dot files on your Debian box are different than your dot files on your CentOS box. And your SSH config file in your personal box is different than your SSH config file in your work box, right? Now yet him uses annotations where you can say, if this is a Mac, use this version of the file. If it's a Linux, use this version of a file. If this box is labeled as work, then use this version of the file. If this box is labeled as server, so it's a server that's sitting on the internet, you don't want to push your keys there, you can set it that way. It also supports GPG. So if you have any secrets, it'll encrypt those secrets in a blob and push that blob up and only decrypts it on the proper box that has the proper GPG key. Okay, it's a really amazing tool. This is the link to the to the tool's repository. Okay, so in a nutshell, what you do, you do an init, you add, for example, a file, you commit it, you add your dot files repository, and you push it, you go to the next box, you add a clone, you give it the URL, you're done. Okay, not only that, one of my and if you're interested, I have my files after sanitization posted up here. One of the things that I love is a feature that's called bootstrap. So you download your, you have your dot files, but you didn't download your Vim plugins yet. You didn't download your Tmux plugins yet. If you're using ZHA, ZSH, you have your plugins that you still need to download or you need to update. If you're on a Mac, you actually want to run an update of brew. Okay, so it has actually a cool bootstrapping feature, and I'm going to try to make this work. Let's see if it works this way. Okay, so I'm doing it on my Mac box. So I'm going to bootstrap the box. So I cloned, yeah, them right now, my Adam repository. And the first thing that it does, it does a brew update. Okay, and it goes and updates all my brew file, all my, all my packages, including the UI ones, and then it went to ZSH, and it's upgrading all the repositories. Now it jumped onto Vim, and it's upgrading everything for me in place without me doing anything. And in any other bootstrapping work that I need to do, it's done. Then what I can do is I check the Adam state in case I change the file, the Adam status, there's nothing there to change. Okay. And also you can check the file structure for Adam, how it stores it. So it has a .yadam file in the root. It has a bootstrap file, which we just executed, and they get repo. But it's a bare repo. And it manages its way through the files. It actually, this one protects you from tweaking stuff the wrong way. And that's one of the things I like about it. Okay. And then if you want to list all the files that are tracked, you can list it this way. Okay. And these are the files that I'm tracking using the Adam at the moment. Any questions so far, by the way? So I wanted to take you through the Adam bootstrap, I believe we have some time. So we'll do a quick run of these. So this is the bootstrap file that I have. And some of the basics that you're used to, check the file system. If you're running a Mac box, you just make sure that brew installed. If you have brew, then update your brew bundle. And a brew bundle is basically a file that has all the files that install, that all the packages you install using brew. And you just update the list of files and you call the brew bundle update for you and it takes care of everything. I'm using antigen for ZSH plugin management. So it checks that also. Then it goes and checks for Vim and make sure that it updates the plugins for me. Does the same for Tmux. And then it makes sure that Go is taken care of, the director structure for Go. Also, if you upgrade it to Terraform version 0.10 and above, there's a directory structure that you can keep in place if you want to share some of the plugging caches and some of the information. So it maintains that for me. And the good part is, I write it once. It runs on all my environments that support Bash, for example, in this case. And since I don't use Windows, I really don't care about Windows. So Microsoft is sponsored. Sorry. Sorry. So this is one way to manage it. Now that we're down with this, another thing is some of the entertainment you do. Now I commute about 50 miles each direction when I go to work. And there is a lot of time that I can spend reflecting, thinking, and sometimes wondering why I'm doing this. Right? And one of the ways I used to pass time is listening to podcasts or audiobooks. Now if you want to get audiobooks to cover 50 miles, 100 miles every day, you're going to lose a lot of money. Okay? Let's put it that way. So I found out, since Microsoft likes to hang out in the library a lot, that they have almost all the libraries that I've been to in Southern California have a subscription to a service that's called Overdrive. Okay? Overdrives provides e-books and audiobooks. Okay? And the audiobooks were previously provided in a proprietary format. Now the audiobooks are actually provided in mp3 format. So you put the book, if it's, if you can, if you can put the book on hold, if it's not available right now, when it's available for you, they'll send you an email, auto-check out. You can go download the overdrive file and install it on your box and the overdrive client is compatible with wine, by the way. So that's actually a cool thing. They didn't break it yet and I'm hoping they don't break it. Okay? Because I don't have one of those boxes to run it on. So now that I have the mp3 files, I really don't want the overdrive app to run things because I'm not comfortable with it and I already streamlined all my podcasts listening. I know what to do. I have my layers, my levels, my pod files. So why not add audiobooks? Right? So I wrote a small tool that's called podcast feed generator, which basically you tell the podcast feed generator. These are the mp3 files for my audiobook. This is the name of the audiobook and here is album art if you're interested. Okay? And it'll generate a podcast feed for you. Right? You just need to post it on one of your boxes on your home or if you've got one of those digital ocean servers online, just scp the files up, point your mobile device to it, it'll download the audiobook and you use the same app you're used to all the time to maintain everything you have. Okay? So it keeps track of it, you can snooze it, you can rewind and it makes life a lot easier when dealing with these things. Some advice on other things that we that I'm not covering here just because in the interest of time. The first thing is try to move away from SSH agent in favor of GPG agent. After GPG 2, a lot of changes remain including compatibility with SSH agent. So now you can have one agent that handles your GPG keys and your SSH key and you can actually make an SSH key out of one of your subkeys. Right? Now why bother with that? One of the things that a lot of people use right now is that they secure their private SSH key using a passphrase. Right? Well if you use GPG, you cannot you don't only use a passphrase, you can use a hardware token with it. Right? So you can you can have your you can safely move your key between your laptop which may get stolen and your desktop or the machine that's stationary at home or at work without worrying of your keys being lost. Right? Or someone getting into it because you need a hardware token to access those keys. So this is an extra layer of security that you should consider. By the way there's a key signing party later today if anyone is interested in signing their GPG key. Also try to enable to factor authentication and FIDO. I work for a security company so it makes sense to give some security advice every now and then. This is like bringing your work to scale. Okay? I saw Ubiqui in the Expo hall. I am an avid Ubiqui user until I got a USB-C laptop. Now I need to rethink my usage. But they're a wonderful company. I highly advise you to get one of their products. Use it to secure your Google, your GitHub, your social media accounts. It's a wonderful tool. And also consider using anti-malware and storage solutions. So if you so happen go to the semantic booth, yeah product placement. If you ever so happen to go to the semantic booth in the Expo hall we can talk to you about some of the security that we have around anti-malware and solutions for your work. So if you have Prometheus cluster that is $20 large at home you may want to have some security around that. Now I like it. We're moving to something that actually I think we coined, we're not sure, we looked on the internet. We're not sure if we have it, if somebody used it before. But we're introducing a new concept that's called T-Nops. And I'll let Musab take it from here. So T-Nops is an agile memetology for completing responsibilities such as homework chores and rewards and it gives my dad visibility over them. For the record I like con bond benefits from. Homework is the next topic we're going over. So this thing is Canvas. The teacher, so basically we use it at school and teachers put homework and stuff like that on it. So when it's put, when the homework is posted on Canvas it gets sent to Google Calendar which then goes through if this and that to the twoist so my dad can see what homework I have. And this is an example of the of the final product of the homework intake flow. And next round to the homework and chore completion. So basically to do, when I mark my homework or anything like that I finish it goes to Erbot and then it says the message on Slack to my dad. So he knows that I completed it. Some future things that you like to do is the is use parent portal but does not have API support so we can't use it. Parent portal is basically where all the grades are put in. And then use Erbot for calculating rewards like things that I get for doing stuff like doing my homework or chores. Also we like to switch to Discord because I'm more familiar with Discord and I'm a gamer so it works. Also Slack is for people like dad. Okay. And thank you. Thank you very much everyone. That's a good presentation. Thank you. Any questions? One at least one? Yes. Oh of course. Put the pipeline back. Which one? The intake. Oh okay. Oh one down. Oh so to-do list is one of the to-do list apps and it has a nice integration it has a lot of cool integration with F if this and that and it has a nice API interface that I can work with. So if you go to the next slide Musa please. So what is that for the link that we posted up there? The applet? So if you actually- I'm going to post a slide on the scale website. If you go to that applet that applet will do the job of grabbing a calendar invite whenever it's posted and pushes it into to-do list for you. No it's a separate different place. Yes. I second that for the benefit of people who are not on listening we're opening a new campaign switching parent portal to support APIs. I'm not sure how TNOPs will feel about that that we have visibility to the grades real time but yes. So I try to avoid that as much as possible. So you saw that I- the main things that I usually sync are either storage or shared storage or my- what do you call it? Or my dot files. That's what I generally use. So everything is either in a Git repo or in a NAS that is shared between everyone and I just back up that component on its own. I try to avoid the use case that I need for syncing stuff into multiple devices. Like I said just central location make sure that's a hard like if you have a NAS make sure it's backed up make sure it's hardened make sure it can live for a long time. I would rather do that than dealing with conflicts it's working on this box or not. We have enough trouble with Dropbox or Box as it is at some point. I don't want to deal with conflicts like that at home. So it's a way of just changing your workflow to adopt to what you can give time for basically. Yes. You're more popular than I am. The next one. This one or the next one? Oh okay. Yeah. Are you done with the picture sir? Yeah. The network to publish. Rami are your slides going to be available online? Yes we'll put them up. And which one? I'll take it. Network to apology. Yeah. Any other questions? Don't shy away. We coined teen ops today. You were in the room. I tried that. I tried. I tried JIRA. Trust me. I tried. Didn't work out. Yeah. We needed to invent a new agile methodology. Nobody can be like my teenager. All right. Well thank you very much. That was an awesome presentation and that brings DevOps Day LA to a close. As a reminder at the Portal Restaurant, El Portal Restaurant, six to eight. The wonderful folks at Sunnetype are putting on a party tonight. So go join them. The scale is going on all weekend long. Looking forward to catching you all in the hall. And with that we are done. Have a great evening.