 It's almost looking like a Windows computer for a while there. Can you see it? Do we need to bring the lights? It's really clear. Yeah. Ooh. I have, see, I should disable that. Flux. That looks a lot better. A lot more white. So, as I said, I'm Bjorn, and I'm going to time y'all and see how long it takes before you fall asleep, because this is probably going to be a bit boring if you are web developers, but sometimes we have to deal with things that aren't on the web, and sadly not in Ruby. So Bjorn, web developer, DevOps, and Pythonista. So come on to the Python meetup. We're very friendly, nice bunch of people. And the big question I would like to ask is, in general, they're like, what do you guys think about your code bases? You feel good? Yes. You feel it? Awesome. That's a real self-assured people. I'm really happy to hear. Like, do you test them out? Yes. Of course. Oh. That's good. What kind of tests? Unit tests? Honestly, tests. Oh, yeah, I've had those problems. I'm just going to append this temporarily. That's not going to be a problem, right? Yeah. Thank you. Annual testing, all that good stuff. Integration testing. Want to make sure your web browser keeps going, rolling where it goes. JavaScript error doesn't stop. It doesn't hit the client, hopefully. Infrastructure testing. You guys do that? How do you check your infrastructure? That stuff is actually going. KSMonkey, we also have the end-users testing it, which works if you're a startup. And for some organizations, it's not entirely good. One of the other ways you usually do is, you have something like Ansible or Chef, for-puppet running and use-side. This is what an environment is supposed to look like. Running some integration on top of that, and you can test out what you're doing. So this is kind of where we come into what we are doing. So I am currently working on a fairly big project with Neon ThoughtWorks, who probably would love it at a point like that. They are also hiring. And because of the nature of the project, we do a lot of file transfers. Well, not a lot, but we do a bunch. It's from legacy systems, systems connected to some mean frame somewhere that we can't touch. It has to be file transfers. They're SFDP, they're encrypted, packaged, making sure the network connection stays up and all that. And all of this integration is handled by a bunch of different teams, and we're just trying to, you know, set spider in the middle, do our thing. And we don't run the servers where our scripts need to run. And because of this, we ended up in a position where we have to run bash. And the only thing we had a lot to run is find, grep, set, and all of these fun things. Which, you know, I've used those before. I've written my 10 line bash script to filter something out. It's great. When I had to do production stuff, though, I was looking at it like, oh, this is Red Hat servers. This is good. I know Red Hat. Red Hat got Python, which I also like. It has Pearl. Please let me use anything. And they go, nope. I was like, please, Pearl, anything! You got the nose tricking me, though. So, yeah. So what to do at that point. Looking at bash, it has aged. It has been around for a while, but, yeah, I've seen prettier things. I spend most of my days coding in Ruby, so I definitely did. But we wanted to do some kind of TDD. And what are the options we have available if we want to do TDD with bash? And so we found this thing called Sharnes, which is a portable shell library, to run an Allah's automated test for Unix programs. It was originally built to help test out Git. So, obviously, it has a good program. It's testing out and making sure it actually works. But what does it look like? Doesn't it look nice? Oh, yeah. If you look at that, that's basically saying, this string eval that and tell me if it went green or not. I've seen that before. That's Angular. I don't want to do Angular in my code. I don't want to do it in the browser either, but... No. Okay, so, we figured out we're not touching that. We're staying away. There are plenty of other frameworks, but they end up in the end coming out of the same thing. So, what do we do? Well, we know... One thing we do know about our program is that we know that there's going to be an initial state. We know what that is. Then, we know that we're going to fetch that and we're going to get it to a specific state. All of these things. And this reminds me of something. Integration testing. So, what do we know? What do we use for integration testing if we're Ruby developers? Also can. Yes, thank you, Winston. Now we went white. So, Cucumber is great. How many have actually used Cucumber in here? That's a good number. Could be more. So, the good thing with Cucumber is that you can write your stuff in so-called business language. You can read out a story. You get a good idea of what's going on and you can just fork out and do your little thing and then it comes back and tells you what's going on. It's just black box. You don't have to care wherever it's being run. Even if it's a shell script running on a remote Linux server somewhere. It also helped us to actually be able to do a red-green refactor on our bash scripts. So, it's pretty neat. To be able to do all of this, though, we had to use a really weird tool chain. So, I hope people know of the GNU. This is the GNU Organizations logo. Anyone know what the GNU stands for? Yes, GNU is not UNIX. They're recursive acronym. UNIX hackers. So, they like the recursive stuff. So, another thing they like is something called M4. M4 is a recursive macro language. And for people that have been doing SysOps stuff, they like the recursive stuff. So, they like the recursive stuff. For people that have been doing SysOps stuff, they've probably heard about Sendmail and that you configure Sendmail once and then you run screaming away and never touch it again. M4 is the reason for that. That's why people say that. Just warning. Use postfix. Make files. Because who doesn't need make files? That's good stuff. GNU Core Utils, which is things like find and date and all of these things. Bash, of course. And then, cucumber and some SSH. Because we'll do that later. Some small stats about what we did. So, because of this project because it's so big and has so many files, we needed to figure out a way of actually combining our files so we could write a smaller script and then just compile the big one like, you know, requires. So, we use M4 for dealing with our requires. That's why that is in there and it's placingly fast. So, we have something like 616 lines of shared libraries yeah it's it's grown today 2000 ish plus lines of scripts played over 40 of them all tested luckily the biggest pipeline we got has 15 scripts on them running across three data centers and nine servers lots of fun total of 15 and a half thousand 16 and a half thousand lines when we compile them all together I don't quite understand how those numbers make up but I'm trusting DC so and the largest largest script when it compiled is 700 lines where 142 when it's not compiled so that's like the meat of the script just taking all the library parts and getting it together so I mean 142 lines of batch script this it is a couple of screens but it's not too bad you can actually look at it and you can make sense of it but still and our cucumber scenarios there's something like two to have 2,400 lines 300 scenarios and two and a half 2,000 steps takes a little bit less than 10 minutes to run which isn't too bad and if you want to write bash like this how do you make sure that he actually is something that will survive you because right now I've done a lot of this I know pretty well how it works and I feel a little bit sorry for the guy that's gonna take it over after me someday I've been doing what I can to make sure they don't have to hate me but you know you want to make sure you should stay away from that and the reason for this is because there's so few people it's actually done bash on the size where it you start to have to think about making sure it actually looks good and you can compile together and deal with it it because 10 lines that's what you do so try to keep it make it lean clean all of that stuff use functions name your things try to avoid the bash shortcuts they're like you can there's a lot of places inside a bash where you can do some really nice short version of things and then there's a long version an actual spelled out name it's like pearl magic variables stay away from that especially we want to make sure people can use it afterwards so simply try to keep it beautiful and because beautiful is better than ugly explicit is better than implicit simple is better than complex complex is better than complicated the center python so if you have python a computer just open it up and type import space this it is wonderful and so I shall script like this what can it possibly look like so this is one of our basic work courses it's just downloads the files from one server it checks if there was anything downloaded add some checksums because it's the first time it's coming in I want to make sure the file that I receive now is the same file I receive one at the end of the pipeline and pushes it on it doesn't clean up doesn't look too bad right come on it's not too bad I mean the syntax isn't overly clear at all the time like the if error the dollar percentage and what the dollar question mark and so on like it's not too but same time you could follow this along if you if you were looking at it so just give an example of what all of it is so this if error it is a function it's just a normal function call function we made ourselves so the affairs the function this is an exit status of the previous command that you ran this is the error code that I want to return in case there was an error and this is the message I like about it and all of this just tells me whenever I get the log mailed to me from whoever was running it that this is where it went wrong and I can actually figure out and go and see what happened and read the log and see okay we changed the permissions in that folder again please stop doing that and when it comes to these error things this complete side note do you guys know what the success status code is on a unit system why is that yes nothing went wrong no errors which it is one of these things I've noticed that a lot of people haven't actually gotten around just keep around keep it's good thing I know cucumber so this is an example of a cucumber file or gherkin it's called and it's business e-language and given given when then and so on these write down all the steps that you would like your thing to take and well this one is very wordy because I have a lot of paths in it because it's the only way I can actually test my thing it's also fairly readable it's not our speaking but give it a try and when you're doing this thing just test it where you deploy not in production I know that's the cool thing to do unless your customers test it out not always an option but if it can't go for it if you're deploying on Linux God say test it out on Linux before you do anything because you know that your max if you're like me and I use one of those I actually use BST tool chains for a lot of options like your grep on your Mac is different from your grep on your Linux box your set is different and they're not to speak about what happens when you have to run the Solaris God vagrant if you have heard of that which is a virtual it's a it's a way of bringing up a virtual machine from scratch on your computer and getting the same environment up usually together with something like Ansible or Chef or Puppet and being able to get the same thing over and over again obviously is a great for these scenarios that I got I have something like 16 user accounts 250 folders special setup for SSH and a bunch of other stuff because I need to make sure the environment looks like it does for my clients and that's horrible and it runs in about 10 minutes so from doing all of this nasty work what did I learn well I can now feel fairly confident that I can change even the Nautilus little bash script I got and then I can see that it works will run through my pipelines and I won't have to feel too worried and that's a good thing I like sleeping at your bash code will be more readable if it's less idiomatic because all of these double ampersands you see everywhere by a very succinct unless you actually know what's going on the people just freeze up the same goes if you're doing the double ampersand and double pipes which is the and or or rather if and else which no just don't stay away set minus x is an option you can have in your bash script and they will print out every single subsequent command that bash runs for you so if you do echo it but it will print out plus echo and then you will see the response of your echo they will do that for everything that you do it makes it super easy to actually debug your thing when you send it over to be run on the servers and it will save your ass if you enable it I'm pretty sure you also have to log to another file that you can get a succinct message in case there's some other error but be able to get it full is great set minus you you get whenever you try to sign to a variable and just spell out args instead of args it will just blow up instead of noticing that way too late always good but I'm also Python guys so I like it when things explode set minus e x is an error so Python thing is that if you have a dictionary in Python you try to call something that doesn't exit we just blow up because the assumption is that you will catch exceptions you get nils both are good ex good way of dealing with things but different set minus e we'll make it so the script will blow up if anything goes wrong anything including an if statement doesn't help a lot it looks like a good idea but trust me you're staying away default past default variables are great they're super confusing to use because what the hell just look at that syntax but really useful especially if you need to be running through some weird environment you must not override one thing on that particular server having different configuration files to your scripts are great because then you can just tack it along at the end of the script so you can test it and rerun and you can test multiples and areas you don't have to modify your script so really useful plus you can similar and go out your scripts together when they're the same across multiple things a lot of us don't know that much about the unix that lies at the bottom of pretty much all that we do all day or where we deploy so I really suggest looking to new core utils look what your shell can do because you can probably learn a lot and really worth it and surprisingly I had fun at this like I've had my fair share with my fist meeting my table but I also had fun any questions or comments how much does set that's not a concern of mine the my things the things we're doing is slow anyway because we're doing encryption and decryption and packaging and networking and so I wouldn't know no we already had a vagrant set up at work and I know how to use it so I didn't want to tackle that issue exactly that's wrong so when somebody's using it I always like to ask them so what is your experience of it not just in that what you're doing there but also specific also with rails specifically how do you use it where do you find those benefits so the looking at the project at work we use it a lot for the browser automation set part of it up that's that's one big part of it I also like it for the integration stuff because I just need something where I can explain I want this to happen and it's something happens which is a black box which we don't care about what it is because we defined it somewhere and then I can look at the results and it also has the advantage to me that even the people that are like the ace or probably managed to look at the thing and again idea what's going on because they get bonus they get brownie points with the client when they can show that and say this is this is the good thing this is the thing you're paying us for that's my point of view I would like to point out so I think that has values when I started out with cucumber I I replicated my exact unit test in our spec in cucumber and it's like what am I doing this this is a waste of time yeah like you live and you learn plus working with the team with people has been doing this for years really really lets you see how effective it can be used correctly yeah and it's also the fact like the work that I'm working on is split into multiple parts and we receive messages from like we don't only have file integrations and having something that can actually ping one side of the app and see that it travels through to the other side of the app and all the transformations to happen on the way has gone through I mean you can do that to an extent with unit testing but that I mean if you start having that through the entire chain something's wrong with the unit test because they're going to be dog slow so you test test the individual part but something I see that comes together you can do that with other tools as well obviously but it's a good one I find yep so we're not done yet the first scripts took about a month and I spent I think we spent about six man weeks on it like on the pair three weeks roughly for the first set then it's been continuous continuous additions to that from that onwards but I think as somewhere around 8 to 10 man weeks in total doing all of this we have a lot of logic in there like that's all I can say there's there's a lot of stuff that I would rather not do there it would have taken me a lot less time I don't think the lines of code would have changed but it would faster because arcane syntax and invocations is fun I have the beard and everything but no thank you that's it yeah if I was good at it yeah I would definitely do it like the only question would be if I would be allowed to run it but if I if I was a possibility yeah it would be really neat I would also be fast but I must admit one of the things I really love with with the bash grips as far as just the fact that I can see every single step that it takes easily and if I wanted to add something like that to my Ruby scripts or my Python scripts or anything I would have to let the bug statements everywhere or run it through a wrapper or something yes that's what he asked if we were thinking of running a compiled program and sending it over no I didn't because I'm honestly not very good at those things so thank you all right can we give a round of applause I couldn't resist I love this thing so up next we have Justin