 Can everyone hear me at the back? Good. Welcome everyone, thank you for coming. It's nice to see a full house. I'm Richard Wall, a senior developer with ClusterHQ. We're hiring, so come and see us at the booth. I'm going to talk to you today about a technique we use at work for testing a piece of software that I work on called Flocker. In particular, a technique that we use for ensuring that the APIs that we write and the implementations of those APIs are easily testable, and a technique we use to ensure that all of the implementations of the API, the real implementations and the fake implementation are in sync and that they behave the same. So, I'm going to start with an introduction. The talk is going to comprise a quick discussion of the problems that we're trying to solve using these verified fakes. Hopefully, an in-depth discussion of the solution that we've come up with, and then I hope that I'll have some time at the end for questions. But if anyone has a burning question, then just put your hand up while I'm talking and it'll be nice for me because it'll interrupt. I won't have to be speaking all the time. First of all, let's talk about one of the... There are a number of problems with testing APIs using unverified fakes or ad hoc mocks or stubs in your tests. These aren't ideas of mine. These have been discussed previous icons, for example, by a couple of guys called Orgy, I think he pronounced it, Orgy Fackler and Nathaniel Minister, and they gave a great talk entitled, Stop Mocking and Start Testing. And so, they say... Well, I'm reading here from some sort of a transcript written by Ned Batchelder. Everyone at Google where they worked made their own mock objects. We had n different implementations to the mock, and when the real code changed, you have to find all of those mocks and update them all. And this is a problem that I've seen in Twisted, for example. Oops, I'm not used to it. So I've got quite an interest in the Twisted project. I did some work on the Twisted Names module last year. And so I wondered if this proliferation of unverified mock objects affects Twisted. And it does, because Twisted is about 12 or more years old. And so I grept for fake classes across the Twisted code base. And there are at least seven implementations of the Twisted base protocol. And six fake reactors. And the one I'm probably most guilty or responsible for is the five copies of the fake resolver in the Twisted Names package. I'm not sure... They may not all be... There may be good reasons for some of this duplication, but I'm sure that some of these could be... Are examples of fakes that could be replaced with a single verified fake which could be used throughout the code base. So that's Twisted. And it's guilty of having these unverified fakes. The next problem with unverified fakes and mocks is that they don't... It's easy to write some ad hoc class which behaves just about the same as the API that you're trying to test. But the trouble is that they're often inaccurate from the very start. And then they grow more inaccurate as the actual real API develops. And so I was looking while writing this talk frantically over the last couple of days for example of this. And someone mentioned to me at the stand a library which I'm quite interested in called Nova Docker which is a driver for OpenStack Nova which allows you to create Nova instances as Docker containers. And so I wondered how do they test that the Docker driver that they've written behaves properly with a real Docker daemon which would be slow and expensive to run versus an in-memory version of the Docker client. And I found that they have actually implemented a fake Docker client and it suffers from these problems that I've just described in that the fake has got out of sync with the real Docker client. It may always have been out of sync. It may never have... Because they don't run the tests against both the fake and the real client they may never have been in sync. So here's an example of what can go wrong. So we've imported here both the mock client from Nova Docker and we've imported the real client from Docker Pi and we instantiate them both and we see how the real client behaves when we create a new Docker container. So all you have to supply when you create a new Docker container using Docker Pi is the image name upon which the container is based and it returns a dictionary containing the ID a 64-digit ID of that container and a list or a summary of any warnings that have occurred while that container has been created in creating that container. And that's not the greatest API. It's a pretty ugly... It should really be returning something like a class or a record of some sort containing that information rather than just a dictionary but it is what it is. And then we look at the Nova Docker the fake implementation that I found and we can see that the first thing I tried to do was to supply it with the same keyword argument and it fails immediately because they haven't used the same argument names and then this is just one example I went on and I tried and tried to create a fake container using their API and I found it doesn't return the same result it doesn't raise the same exceptions it doesn't have the same optional arguments as the real API and some of this may be because they've based their fake on an earlier version of Docker Pi but I think actually that's not... there are some real problems here which mean that I can't use this fake in my code which I quite like to do. Okay. So. Can we do any better? Well. Let's think about the ideal for a verified fake. Ideally we want something that provides the same interface as the real API we want to be able to run the same tests against the fake as we run against the real client and ideally ideally we'd like the fake to be maintained by the same author that wrote the real implementation that's quite rare but I hope that that will become more common and I'll show you an example in a minute of someone requesting just that. So. I haven't explained... I haven't really explained what Docker Pi is but I think probably most of you are familiar with it. It's a library that wraps the Docker Demons REST API in Python and it is something that we use at work quite a lot so it just seemed... It's something which I'm familiar with it seemed an easy target to pick on but there's lots of examples like this so don't think I'm picking on this in particular. Okay. So I wondered when I started writing the talk is anyone else thinking the same thing that I am and funnily enough there is an issue on Docker Pi's GitHub someone has come and asked and made the statement that any usage of Docker Pi requires unit testing the latter requires a fake client that does not require a Docker demon to be running providing a mock implementation will avoid every single user having to re-implement its own mock and he says this is a nice to have well it is a nice to have but it's probably... Ideally every library would come with such a fake so that we all don't have to re-implement our own dodgy, unverified fakes. Okay. So what I'm going to show you now is my crude attempt to write a fake, a verified fake for Docker Pi. Any questions so far? If not, I'll carry on. The first thing I wrote was a... It's something I'm familiar with from alternatives but I haven't used base classes that are available. An implement a Zope interface instance that you pass to it provides all of the methods. Doesn't go so far as checking the arguments if that's something which I've put that test in place. Notice that the test is defined in a mix-in class. We're not actually defining a test case here. I'll show you how this works in a moment. But with that test written we can run the tests against the real and the fake with the classes, the Docker Pi client class. I'll show it again. I just want to show you some results. In order to get this result a twisted unit test is tests.integration test real Docker. I'm also running the tests, the unit tests tests.fake.client tests. So we have those two with the integration tests the slower tests which exercise the real Docker demon in a separate module from the fake. That's a nice way to that's the way we organise our tests existing test modules but it's nice to keep the two sets of tests separate. In such a way that the tests the test mix-in which we saw earlier and the fake implementation are put in a public module. They are which means that they're not hidden, they can be imported by any libraries that consume this package that is discovered by the the way the test runner discovers the tests or the way the actual test cases are produced are by way of a test case what this does find a setup method. A similar case pass in this time the fake Docker class which has been produced by this factory. This dynamically this test class gives us a place to document fleshing out the interface. So far all we've got is a test for the interface a zip interface test which we'll describe in a minute the passes and we read it in the same way with the same image and with the same parameters and you're wanting to re-exercise public so you're trying to in creating a container you need to create the container and then ensure that that container is listed or you want to clean up all the containers that are available are listed and you need to create some containers so you can't quite implement it method by doing this sort of stuff is that you want to start from the very start if you're running the test against a real Docker client you need a way of cleaning up the containers that you've created after and so to do that if your aim is to only use the public APIs you have to implement a remove container method test the remove container method until you have a way of listing and creating containers so again you have to be practical about this and tempting when you're trying to overcome the problems I've just described temptation is to test the sort of implementation details you might have private helper methods inside your implementation you might start trying to write tests for those but usually it's not worth it and it's usually just sticking to the testing only the public parts of the interface the parts of the class which have been defined in our ZOP interface that's as far as I'm going to go with showing you the implementation I've done of this fake Docker client but I wanted to briefly describe some other examples of this maybe not all of them some of them are not quite the same the first one is the one that I've most recently worked on at work we were tasked with creating a system for our software which allows us to attach to create and attach and destroy our tests against AWS although we do and we don't want to have to run our tests against OpenStat but we do have to do they're very slow tests to run what we really want to do is run our tests for this iBlockDevice API against a fake a sort of simulation of those those two cloud block devices you can have a look at the code in GitHub here I'll post all of these slides up to the top you can have a look at the code and you'll see that we've implemented a loopback simulation of these we've got an implementation of iBlockDevice which creates a loopback file system attaches them to one or more to one process and detaches them and attaches them to another process and by doing that we have a way of testing running our tests for this API much faster than we would if we had to run them against the real and rack space for example another example I'm interested in but I haven't yet tried developed by a guy developed by amongst others some of the twisted developers Glyff from Twisted Matrix this is a system which allows you to create a fake version of the open stack REST APIs and this is something we definitely like to use at work the trouble is it doesn't yet implement the cinder APIs but that's something again which I'd like to try and do and what this is is a web service which can be primed with both successful and error cases and it's a web service which you can make REST API requests to access the state as the requests come in so you can say create a NOVA instance and then you can ask that fake API for a list of the NOVA instances and it'll return you the fake that you've just created the details of the fake that you've just created and it even I think simulates the keystone authentication for open stack as well which will be really useful another one from work a colleague Gitamar develops called Elliott a structured logging library and from the start he's built into it various features which make it easy for us to test the code that uses Elliott so he's implemented a memory logger and a he has done it the way I've described he's got an interface called iLogger and a memory logger which implements that interface he's gone further than that he's actually in the public um Elliott package he provides tools for testing that your code logs the correct messages and logs the correct errors so that's a really good example of this sort of technique and finally I thought worth mentioning is a recent blog post by Gliff about going even further writing your fakes in such a way that you don't pollute the fake with attributes that aren't actually present in the real implementation and it's got a really neat way of leaving that separation which again I haven't yet I haven't done that with my code for this talk but that's the next thing I'm intending to do so I think we're about out of time quickly to summarise we've seen some of the unverified problems with unverified fakes and mocks we've looked at some of the some examples of those problems um we've quickly briefly shown how we might go about writing a verified fake from scratch and we've seen some examples of other um other python examples of this technique and that's really the it's only a half an hour talk so that's all I've got to say I think I'll leave it there Thank you for your attention Officially we don't have any time for Q&A so you're free to go to lunch but if you'd like to say and ask questions then you're free to do that and does anybody have questions?