 So let me paint your picture you're at the end of a death march and you spent the last three months getting all the features ready for your prospective client But there's a problem How is the client actually going to install the software you figure what the heck you cobble a table? Cobbled together a table attach it to an email hit send and clock off. It worked in dev. It's off problem now You rock up at 945 a.m. The next Monday and your inbox is exploded and Unfortunately the boss wants your head The customer couldn't install the software and when they did eventually get it running They got owned 15 minutes later by the loaded by the latest open SSL exploit So how can you get better assurance that the software you're building and shipping does what it says on the box for this? We're using service back So what is service back? Service backs a framework for testing infrastructure and applications It's built on top of aspect a popular Ruby testing framework It uses standard aspect syntax so you can use all of the existing aspect functions There's lots of resources on the web to help you get started and it's generally pretty simple You can run it via SSH or using vagrant or locally Just going to look at a quick example of output from a service back run So here we're looking at a log file the flapjack the flapjack log We're asserting that it should be a file that's green. So that's past It should have permissions six four four which it does so that's past but It should be owned by flapjack and it's not So that's failed. It's red. So it's all pretty pretty obvious to see and if you look further down you can see The assertion owned by flapjack should return true, but it got false and at the last line You can see this the stat command that it's actually using under the hood. That's part of service back You can run that command in the target environment to replicate the replicate the output So it's pretty accessible and helpful and tells you tells you where to go So why should you use service back? One big reason is to automate your QA Humans miss things in QA and they don't necessarily do things the same way every time QA can humans can only do QA during certain parts of the day. It's considered bad to lock them in the basement quite sad Humans are hard to scale. It's difficult to just get more humans during the busy periods of testing But service back will keep testing and testing and testing The same way every time at any way of the day or night at any time of the day or not rather You can validate your expectations So you can run service back against your development code to check that files are being created as you expect It's a more concise form of cucumber tests You can fail early you can have service back run as part of your build procedure and when the test fail Just have the entire build fail. You'll never publish a build that doesn't work because we've all done that You can validate your your output For CLI tools you can run commands and check that the standard out and standard error actually give what you expect Because service back integrates with vagrant you can easily test on multiple versions of multiple operating systems This of course can be automated so you can test all your operating systems using different vagrant boxes during the one build. Which is nice With all of this you have a much better chance of your release working as you expect as a bonus You'll free up your QAP for to write more service back tests and capybara tests for the GUI elements, of course Cool. So how might you use service back? I'm just going to look at some quick examples of the resource types that service back makes available to you First as a developer and then as a sys admin so as a developer Package installation So it's one thing we use it for is So if you install the package in the target environment, typically a VM or your local Environment you can run these run these service back tests in a check that your is installed And this is also testing that it's that it is enabled Enabled on boot. Sorry. That's enabled on boot. Yeah Command execution so service back can log into your test environment run a command Verify the exit status in this case should equal zero. So it didn't fail That the standard output includes the word port and the standard error includes the word interval or whatever you like You can check for running processes So this is when we install a flapjack package on Ubuntu and Debian We have a flapjack automatically start as is the convention on Debian based Platform so after the packages installed we can assert that flapjack is running and then the processes arguments should include server start file presence and permissions So we're looking at the log file. The log file should be created There should be a file. So not a sim link or a directory or something else. It's just a plain file that it should have defined permissions But there's a 2 or 4 Yeah, that's a good question. I don't know. I don't think anyone's actually thought of that yet We certainly haven't needed to use it in the flapjack project We can look up the docs after the after the main presentation and check it out So yeah file ownership So who the files are owned by and which group or groups. Sorry group the file has You can test the hashes of files This is doing a sharp 2 5 6 some of its et cetera services So that's verifying that the file hasn't been corrupted or that it's the right version of the file Maybe you update that file as part of your part of your package installation And finally you can test that a port is open So when flapjack starts it listens on port 3080 by default And this is just yeah verifying that uses net state under the hood to check that port is in fact now open So there are various ways that you can use service back as a sysadmin and one of the big ones is after running puppet on a server Service back doesn't just test a file existence. It also checks the type I whether it's file or a sim link or a directory And you can also check the contents of files by checking for strings and regular expressions This can be restricted to finding strings before or after another string in the file or between two strings as you like So the spec provides full user and group support so You know including home directories and login shells And which users should belong to which group as you can see up there You can also check the file systems about it as you expect including the type and options here the roof file System is of type ext4 and it's mounted green right and if you're unfortunate enough to have to support PHP You can check the settings of that are correct to Another thing service back is useful for is making network changes. So you can Verify that your IP tables rules are as you expect and you can actually limit this to a particular checking rule is in a particular chain and Service back doesn't only check the host that you're on It can also look at other hosts from the host that you're on and try and connect the Try and check that they're reachable on particular ports in particular, you know TCP or UDP Service back doesn't forget about Windows There's a bunch of specific Windows resource types that you can use in service back. So just look at a few of those File versioning as here. We're asserting that the WAPI dot DLL should be version 7.6.760.256 IIS application pool That there's a default that pool it should exist and should have dot net version 2.0 installed IIS website default website That it's enabled and running and it's in the app pool default app pool and has physical path see I net pub dub dub dub Can examine the Windows registry for you check that you got certain keys and values in your Windows registry What I really like about Yeah, I believe so. Yeah, yeah, because it's a standard aspect thing Pretty much anything in aspect you can test you can say should not equal should not be here Yeah, anything like that Yeah, that's part of the standard sort of aspect things negation Yeah, so what I really like about service back on Windows is it Saves me from having to remember and or look up how to do these things myself because I don't normally Touch Windows, so that's great and we can you know, obviously test all of these things on multiple versions of Windows or automatically So just a sort of quick recap where we are you can see our service back is great for validating your assumptions about how servers are built and How your application behaves after installation for application developers? It's not a replacement for the tests you might be doing already such as unit tests or integration tests, but rather lets you Test the overall behavior of your applications install and that are run successfully on all of your supported platforms So we use flapjack we use service back in flapjack got the right way around this time So flapjack is monitoring tool. It does notification Routing and event processing. So in layman's terms, it takes check-out port from Nagyos sensor and friends applies filters and works out who to notify and how For this we're building packages for Debian Ubuntu and CentOS using omnibus We're using service back to test our packages before they are made available to our testers That would be the human kind of testers So yeah, we build packages we test them with service back and assuming that all passes we publish them But what about the different operating systems and releases that we support? We use Docker and yes, this is the manager Docker reference in this talk as all must have We take a pristine image of each distribution add the newly created package and run service back on each Docker instance itself But not all tests are relevant to all operating systems service back allows you to add an if-block to your test which we use to You know which we use to run some of our tests only on Debian based operating systems because that's all they're valid for Here's a demonstration of our current service back test suite including a few bonus fails So yeah, that looks pretty similar to aspect Yeah, look at all those pretty fairies and you can run section by section like aspect and everything else and This actual script just brings up the vagrant box and automatically applies puppet to install all the relevant packages and install I guess and all that kind of crap and Yeah, just does it and runs the test and then you come back how long later and it's all done, which is awesome Just do and how many questions Are there any plans to Have a semantic understanding of the config files for like the SSH config So rather than doing regex you can actually say does it have the key port and value because if port 22 was in a comment It would still match that regex Yeah, that's an interesting one. Certainly you can restrict, you know, you can say it should appear in this particular point of the file and you can obviously You can obviously use carrot and dollar to to kind of force it to to match a full line of saying port 22 Which means I can't have a comment in front of it But yeah, I don't think it does support key value pairs at this point You might be able to tie it in with August perhaps Which is something you can use it in puppet to to model to very you know understands file formats different config files for different different systems maybe So you mentioned that it lets you do OS version-based checks To decide whether to run particular tests. Can you do? System introspection based tests are like looking for particular config files to figure out what kind of system you're on Because we had a similar capability in beaker and we ended up rewriting it to do feature-based testing Because the OS based testing turned out to be a major maintainability hassle Where a new distro would come along and your assumptions would all be wrong so and you couldn't Ascertain what what version of the distro it became you ended up having to track all the distros to figure out Which checks you should be running And so as people like did things like adopt system D or switch from ntpd to crony and that sort of stuff It became easier to just look for the thing you were interested in rather than worrying about what distro you're on Yeah, right. That's interesting Yeah, I Believe that you can limit aspect tests like that So you would be at a limit you would be at use exactly the same to to use subspec You know you can wrap them all in describe blocks and Yeah, and use if blocks that way. So I think you can wrap the entire thing and if if you want Yeah, sorry if this has been answered already and I wasn't listening Can it run as a service so it's constantly monitoring for your server? If the conflict goes out of our spec, you know some monkey logs in and Bypasses the the accepted configuration method. I don't see why not while true run, you know bundle exact aspect Yeah specs of spec, whatever. Yeah, it's a I know that some people actually hook it up And they're nagging or so whatever monitoring system and execute the checks Continuously as part of the monitoring system. So things fail Yeah, get notified It's just a it's just a command line you run. So you hook it up to anything really here Bonus points if you hook it up to a big flashing red light saying you broken shit again Because that's always the best or a giant hammer that hits people your faults. Yeah That sounds good Do you have any more questions? Excellent. Okay. Thank you very much