 Welcome everyone. My name is Lekha Jeevan. I'm a software developer in test with Rackspace and I'm Glyph also a software developer at Rackspace Today Glyph and I are here to talk to you all about Mimic an open-source framework that allows for testing of OpenStack as well as Rackspace APIs Mimic has been making testing across products in Rackspace a cakewalk And we think it could potentially do the same for OpenStack-based applications as well someday today We're not there yet, and your contributions will help us get there soon. I Originally created Mimic to help with testing Rackspace autoscale because I hate waiting for tests to run Also, I was creating it as I was creating it I realized that it could be something that could be reused for multiple projects across Rackspace and OpenStack and Glyph volunteered to help achieve that. I saw Mimic had a lot of promise But needed some help with its implementation. So I came on to the project and I've been helping improve its architecture Making it more generic and modular So before we even begin to dive into the details, let's take a quick sneak peek at What Mimic does and how easy it is to get started with it. So in a virtual amp we pip install Mimic run Mimic and Hit the endpoint and Mimic returns the authentication endpoint to be able to authenticate and get the service catalog of All the services that Mimic implements While today Mimic is for a general purpose testing it was originally created for Rackspace autoscale So let's talk about that and see why we needed Mimic So Rackspace autoscale is an open-source project a solution that Rackspace utilizes To automate the process of getting the right amount of compute capacity for an application By creating a deleting service that is scaling up or scaling down servers and associating them with load balancers In order to perform these tasks autoscale depended upon three back-end Rackspace APIs one was the Rackspace identity for authentication and impersonation cloud servers for provisioning and deleting servers and Load balancers for adding and removing servers from the load balancers as they were created and or deleted Rackspace identity is API compatible with OpenStack identity v2 and Rackspace cloud servers is powered by OpenStack compute Rackspace cloud load balancers is a custom API as Autoscale was interacting with so many other systems testing autoscale did not mean just testing features of autoscale But also that all the systems that autoscale depended on if any of them did not behave as expected Autoscale did not crumble and crash but was consistent and able to handle these errors gracefully So there were two kinds of tests for autoscale One was the functional test that validated the API contracts these tests verified that the autoscale API responded accurately for given valid or malformed requests and the other was the system integration tests These were more complex tests these verify the integration between autoscale and Identity compute and load balancers for example when a scaling group is created one one such test would verify that The server on that scaling group is provisioned successfully That is that the server not only went into an active state, but also was added as a node to the load balancer or If one of those servers went into an error state Then that autoscale was able to reprovision that server successfully and then added to the load balancer in the scaling group So all these tests were set up to run against our production environment and here are some observations I had whilst writing these tests Servers could take a while to build sometimes five minutes sometimes ten minutes or longer and the test would run that much longer and Sometimes tests would just fail randomly because a test was expecting a server to go from build state to an active state but the server would go into an error state and Test for such negative scenarios like actually testing how autoscale would behave if a test Server did go into an error state could not be tested because it was not reproducible consistently However, the overall test coverage was improving and I continue to add tests oblivious of the time it would take to run the entire test suite and later we even added these tests as a gate for autoscale's merge pipeline and Things were not going so well because tests were slow and flaky and nobody wanted to run these tests locally because not even me when I was adding more tests and Our peers from computer and load balancers were not happy because we were using all of their resources for our autoscale testing and They were so not happy that we were pretty glad we were in a remote office But we had had enough this had to change we needed something to save us from these low flaky tests So now that we've had enough how are we going to solve this problem? Since we've been proceeding from the specific case of autoscale to the general utility of mimic Let's go back to the general problem of testing for failure and proceed to the specific benefits that mimic provides When you have code that handles failures, you need to have tests and ensure that that code works properly And if you have code that talks to external services those services are going to fail and you're going to need to handle When that failure happens you need to have code for that But if your only integration tests are against real versions of these external services Then only your unit tests are going to give you any idea whether you've handled those failure cases correctly your positive path code the code that Submits a request and gets the response that it expects is going to get lots of testing in the real world Services usually work and when they don't the whole business of service providers is to fix it so that they do work So most likely your positive path code is going to get exercised all the time You'll have plenty of opportunities to flush out those bugs If you test against real services though your negative path code will only get invoked in production when there's a real error If everything is going as planned this should be infrequent, which is great for your real service But terrible for your test coverage It's really important to get negative path code right if because if all the external services you rely on are working fine Then it's probably okay if your code has a couple of bugs you can work around them manually But if things are starting to fail with some regularity in your cloud and that is to say if you are using a cloud You that that is exactly when you need to make sure that your system is behaving correctly Accurately reporting those errors measuring those statistics on those errors and allowing you to stay on top of incident management for your service And your cloud and take advantage of those nice SLA is that your provider gives you Even worse when you're testing against a real service You're probably testing against a staging instance because you don't want to disturb production of all your testing And if your staging instance is typical then it doesn't have as much hardware doesn't have as many users It doesn't have as much networking gear as your production environment and every additional piece of hardware or concurrent user or network Partition is an additional opportunity for failure. So that means your staging environment is even less likely to fail. I Remember the battle days of the 1990s when most projects didn't even have unit tests Things are better than that now OpenStack itself has great coverage. We have unit tests for Individual units and integration tests for testing components together. It's a marvelous future. We're all in today and We all know That when you have code like this that we need to write tests for this part And one popular way to get test coverage for those error lines is by writing a custom mock for it in your unit tests So if we can't trust real systems for error conditions Why isn't it sufficient to simply trust that your unit tests cover your error conditions and your integration tests make sure things work in a more realistic scenario For those of you don't recognize it This is the mock turtle from Alice in Wonderland as you can see he's not quite the same as a real turtle Just like your test mocks aren't quite the same as a real system There's a reason that the mock turtle is crying He knows he can't quite do the things a real turtle can do just like your test mocks can't quite replace those real systems Let's take a look at a simple example from OpenStack compute In June of this year OpenStack compute introduced a bug making it impossible to revoke a certificate These lines of code at fault here are these two additions. This is not a criticism of Nova The bug has already been fixed. My point is that they fell into a very common trap The bug here is that Chider does not actually return a value Because the unit tests introduced the change also introduced their own mocks for Chider Nova's unit tests properly cover all of the code But the code is not integrated with a real system in a way that verified in any way What the real system did in this case the real system being pythons Chider In this specific case a nova might have simply tested against a real directory structure in the file system because relative to the value of Testing against a real implementation new folder is not a super expensive operation However, standing up an OpenStack cloud is a little more work than running Makedur If you're developing an application against OpenStack deploying a real cloud to test against can be expensive error prone and slow as Autoscales experience has shown Creating a one-off mock for every test is quick But error prone good mocks rapidly become a significant maintenance burden in their own right as they're making sure that they validate all of their behavior against a real implementation Autoscale needed something that could produce all possible behaviors like a test mock but ensure those behaviors accurately reflected a production environment It should be maintained as something maintained as a separate project not part of the test suite They can have its own tests and its own code review to ensure that its behavior is accurate Since we've been proceeding from the general to the specific right here Or we needed a realistic mock of a back-end OpenStack service is where the specific value of mimic comes in So the first version of mimic was built as a stand-in for services such as identity Compute and rack space cloud load balancers the services that autoscale depends on The essence of mimic is pretending So the first thing you must do to interact with it is to pretend to authenticate mimic does not validate credentials All authentications will succeed and as with a real identity endpoint mimics identity endpoint returns a Service catalog that contains all the endpoints for the services implemented within mimic So a well-behaved OpenStack client will use the service catalog to look up the URLs for the services that it wants to use And such a client only needs two configuration values to be able to communicate with cloud One is the credentials and the second one is the identity endpoint So one such client would only have to change the identity endpoint to be that of mimic to start using it So also when you ask mimic to create a server it pretends to create one This is not like stubbing it does not return static responses when mimic pretends to build a server It remembers the information about that server and will tell you about it in the subsequent requests Mimic was originally created to speed things up So it was very important that it be fast both to respond to requests as well as to help developers that get set up fast So it uses in-memory data structures with minimal software dependencies almost entirely pure Python No service dependencies no configuration and it's entirely self-contained So let's look at how we could use mimic with the Nova command line client So here's our config file that has all the environment variables Required to run the Nova command line client or any other OpenStack command line client As you can see we've set the username password and tenant to just be random values and The auth URL is said to be that of the instance of mimic that was running in from our previous demo Now let's continue where we left off with our previous demo. So we already have an instance of mimic running We're going to pip install the Python Nova client We're going to make sure that the config file has the URL set to that of mimic Source the config file and verify if there are any servers Well, we didn't create any so let's create one We're just going to create one with a random flavor in image So the server went into an active state immediately on creation. Let's see if this behavior is consistent. Let's create another server So we created server 2 and even this server went into an active state and Mimic knows about both of these servers that were created and both of them are in an active state Now we're going to delete the second server that we created and Now mimic has only one of the servers We did the same thing with autoscale We pointed all of our tests to run tests as well as the autoscale API to run against an instance of mimic This reduced our test time exponentially before mimic the functional tests the API the ones that validated the API contracts would take 15 minutes to run and After mimic it would take only 30 seconds to finish and In the integration test if one of the servers remained in building state for 15 minutes longer than usual Then the test would take 15 were longer to run and complete these earlier would take three hours or more and With mimic it takes less than three minutes and is consistent All of our dev environments now are configured to run against mimic One of our devs from the RaxFace cloud intelligence team calls this developing on airplane mode as we can work offline without having to worry about the Uptimes of the upstream systems and get immediate feedback on the code being written But Leica, what about all the negative path testing stuff? I was talking about before does mimic simulate errors How did this dev VM test autoscales error conditions? Well, Glyff, I am glad that you asked me that question mimic dust simulators Earlier when I said mimic pretends to create a server that wasn't entirely true mimic can also pretend to not create a server It uses the metadata provided to it inspects that metadata and sets the state of the server respectively So let's go back to that demo where we left off and see how this can be done So we have the one server that we created Now let's create a server with the metadata server building 30 This puts the server state and build state for 30 seconds and Glyff will tell you how you could prolong it or not and So we have two servers one in the active state and build state We can also create a server and have it go into an error state using the metadata server error true So now you can see that we're able to create servers with two different states For the purposes of order scale it was important that we have the right number of servers on a scaling group Even if the number of attempts to create one failed We chose to use the metadata error Injection so that we could use the same test to run against real services as well as against mimic for order scale The expected result is the same number of servers created irrespective of the number of failures But this behavior may also apply to many other applications because retrying is a common logic for handling errors However the first implementation of mimic had some flaws It was fairly racked space specific It only implemented services that auto scale depends on and they were implemented as part of mimic score And if you needed nn points You needed to run it on n ports and not just n ports, but n consecutive ports It allowed for testing error scenarios, but only used the metadata method And this was not supported by some clients that would probably not let their users enter metadata while creating a server Also mimic was not implemented to be multi region It used global variables to store state which meant that it was hard to add additional endpoints with different state in the same running mimic instance mimic had an ambitious vision to be a one-stop mock for all open stack and rack space services that needed fast integration testing However, it's architecture at the time severely limited the ability of other teams to use it or to contribute to it As Lika mentioned it was specific not only to rack space, but to auto scale on Balance mimic was also extremely simple It followed the you aren't gonna need it principle of extreme programming very well and implemented just the bare minimum to satisfy its Requirements there wasn't a whole lot of terrible code to throw out or much unnecessary complexity to eliminate There is however a corollary to Yagney, which is eventually it turns out you are going to need it So as mimic grew other services within rack space wanted to use its functionality and a couple of JSON responses and a couple of global dictionaries We're not going to cut it anymore So as one does we created a plug-in architecture Mimics identity endpoint is the top-level entry point to mimic as a service Every other URL to a mock is available from within the service catalog as we were designing the plug-in API It was clear that this top-level identity endpoint Needed to be the core part of mimic and plugins we each add an entry for themselves to the service catalog URLs within mimic service catalog all look similar in order to prevent conflicts between plugins mimics core Encodes the name of your plug-in and the region names specified by your plug-ins endpoint here We can see what a URL for the compute mock looks like This portion of the URL which identifies the mock being referenced is handled by mimic itself So it's always addressing the right plug-in then there's the part of the URL that your plug-in itself in it Handles which identifies the tenant and the endpoint within your API each plug-in is Called an API mock and that's an object that has only two methods catalog entries and resource for region and that's it catalog entries takes a tenant ID and returns the entries in mimic service catalog for that particular API mock APIs have catalog entries for each API type Which in turn have endpoints for each virtual region that they represent This takes the form of an iterable of a class called simply enough entry each of which is a tenant ID a type a name and a collection of endpoint objects Each containing the name of a pretend region that says ord because that's a rack space region, but it could say the moon a URI version prefix that should appear in the service catalog after the generated service URL But before the tenant ID and of course the tenant ID itself for each endpoint Resource for region takes the name of a region a URI prefix produced by mimic core to make URIs for your services unique so you can generate URLs Into your own service if you need to generate them as part of your responses And a session store where the API mock they look up the state of the resources. It's pretended to provision for this tenant before It returns an HTTP resource associated with the top level of any given region this resource then routes the request To any tenant specific resources associated with the full URL path Once you've created a resource for your region It has a route for the parts of the URI that starts at the end of the mimic Plug-in URI path and here you can see what the nova list servers endpoint would look like using mimics API As you can see it's not a lot of work at all to can turn a return a canned response It would be a little bit beyond the scope of this brief talk to go through a full tutorial So if you haven't been following all of the code That's fine. You can always come by and get hub later But hopefully this slide which is a fully working response shows that it's pretty easy to get started And now that we have most of a plug-in written Hand wave hand wave. Let's get mimic to load it up To register your plug-in with mimic you just need to drop an instance of it into any module of the mimic dot plugins package This is really a complete plug-in registration. That's all there is to it This of course just shows you how to create ephemeral static responses But as like I said previously mimic doesn't just create fake responses. It remembers in memory what you've asked it to do That session store object pass to resource for region is the place you can keep any relevant state It gives you a per tenant session object Which you can then ask that session for any mock specific data you want to store for that tenant So all session data is created on demand So you pass in a callable to go with your API mock object which will create your data if no data exists yet for that mock for that tenant Note that you can pass other API mocks as well So if you want to inspect a tenant session state for other services in factor that into your responses For example, you have a service where you want to Pretend that you're gonna get a certain type of response if the tenant is over quota in their nova mock It's easy to do this the pattern for inspecting and manipulating a different mocks data It can also be used to create control planes for your plugins so that one plug-in can tell another How and when to fail by storing information about that future expected failure on its session We're still working on our first error injection endpoint that works this way It's still in a branch and a pull request by having a second plug-in tell the first what its failures are But this is an aspect of mimics development that we're really excited about Because that control plane API also doubles as a memory for all the unexpected and even potentially undocumented ways in which The mock service can fail So anyone testing a product will run into unexpected errors That's why we test but we don't know what we don't know and can be prepared of prepared for it ahead of time, right? When we were running the test for autoscale against compute We began to see some one-off errors like when provisioning a server the test expected the server to go into a building state and Remain there for some time and then go into active But it would remain in building for long or indefinitely or would go into an error state and Autoscale had to handle such errors and we changed our code to be able to do that and mimic provided a way to test these Consistently however, like I said, we don't know what we don't know. We were not anticipating to find any more such errors But there were more and this was a slow process for us to uncover such errors as we tested against the real services And we continue to add these errors to mimic Now wouldn't it be great if not every client that depended on a service that had to go had to go through the same cycle Not everyone had to find all the possible error conditions in a service by experience and have to deal with them at the pace that they occur What if we had a repository for all such known errors and everyone contributes to it? So the next person using the plug-in can use the existing ones and ensure that their application behaves Consistently respective of the errors and be able to add any new ones to it mimic is just that it is a repository of all known responses including error responses and it has an endpoint called presets that Add today lists all the metadata related error conditions that can be simulated using mimic In addition to storing a repository of errors mimic allows for finer control of behavior beyond success and error You can determine the behavior of a mimic service in some detail We're not just here today to talk about exactly what mimic offers right now But also where we'd like it to go and in that spirit I'll discuss one feature that mimic has for controlling behavior today and a couple which we'd like to have in the future Appropriately enough since I'm talking about things now and things in the future the behavior control I'd like to talk about that mimic has right now is the ability to control time That is to say when you build you do something against mimic that will take some time Such as building a server time does actually does not actually pass for the purposes of that operation Instead of simply waiting 10 seconds. You can hit the out of band tick endpoint Like so it will tell you that time has passed like so Now you may notice there's something a little funny about that timestamp. It's suspiciously close to January 1st midnight 1970 Mimic begins each subsequent restart thinking it is the beginning of time that is to say in 1970 January 1st At the Unix epoch and if you want to advance the clock you just plug in the number of seconds since the epoch is the amount in your First tick request and mimic will appear to catch up to real-time instantly If you previously created a server with the server building metadata that tells it to build for some number of seconds And you hit the tick endpoint with that number of seconds That server and any others that are in that state will now show up as active as they should this means You can set up very long timeouts have servers behave realistically have things happen at the same time Which is often hard and integration testing But in a way where you can test several hours of timeouts in a second if you need to If you have legacy integration tests that need to run in real time and have time dot sleep in your test suite You should fix that and ignore this slide But if you can't do that right now you can ask me make to pay attention to the real clock with the dash dash real time command line option, which disables the time advancing endpoint Another feature that isn't that is not implemented yet But we hope to design later is as we've spoken about the ability to inject errors ahead of time using a separate control plane Which is not part of a mox endpoint. We've begun working on branch doing this for compute But we feel like every service should have the ability to inject arbitrary errors As like I explained mimic and already inject some errors by supplying metadata within a request itself But this means that in order to cause an error to happen You need to modify the request that you're making to mimic and it's not exactly the same as the request that you would want To make against a production system, so your application isn't entirely unmodified What we'd like to do in the future is to put the error injection control plane into the service catalog with a special entry Type that you're testing infrastructure can talk to and your application can ignore That way your testing tool would authenticate to mimic tell mimic to cause certain upcoming requests to succeed or fail before the system That you're testing has even talked to mimic in the first place your system would not need to relay any expected failure data itself It could use naming conventions. It could use other sorts of out of band information To match that request to an expected failure and so no metadata would need to be passed through What we'd really like to build with these out of band failures though is not just a single feature But an API that allows people developing applications against open stack to make those applications as robust as possible By easily determining how they will react at scale under load and under stress Even if they've never experienced those error conditions in a real test environment So we need you to contribute the errors and behaviors you've experienced mimic is also based on a networking framework that some of you might know about which has features such as a built-in DNS and SSH server It'd be really cool if when a virtual server was booted it the advertised SSH port which currently doesn't respond Did give you access to an SSH server albeit one that cheaply created local shells or just a container or something instead of a Provisioned virtual machine Similarly if we had a designate mock it could be really cool to have real DNS entries so that you could point your your DNS at local host So mimic can be the tool where you do not have to stand up the entire open stack To understand how open stack APIs behave or to just play with it Mimic can be the tool which enables an open stack developer to get quick feedback on the code He or she is writing and not have to go through the gate multiple times to understand that upstream failures can happen once in a way One of the things I like to point out to potential new contributors is that mimic isn't really real software It's tiny. It's self-contained. It doesn't need to interact with the database or any external services And since it mimics exclusively existing APIs, there are very few high-level design decisions As a result mimic is a really easy project to contribute to if you're looking to cut your teeth on something open stack related mimic is a great place to get started So please come join us build mimic together We can make it an error repository or a repository for all known responses As we mentioned earlier mimic is open source. There's the github url to it We're maintaining all of our issues that are in progress or are in plans for works in the issues tab You can also chat with us on pound pound mimic You can start by using mimic giving us your feedback or better yet start contributing to it by adding plugins to services that do Not exist today. Thank you So yeah, so if anybody has any questions, I think there's mics in the middle of the room The question was how do you track consistency between real apis and mimic apis? There's a quote of one of my authors of my favorite authors has this quote strength of will and faith in our holy cause We uh We We've tested mimic against so mimic is currently still part of the gate for the rack space autoscale product And that means that we validate this on a pretty regular basis Just through that usage and really that's because we want to be able to mock Unusual errors that come back that are unexpected that might only exist in certain versions That sort of validation is probably going to be a fairly informal process going forward But the goal is to get mimic into more and more kind of Gating project processes where it is compared against real open stack on a regular basis So it's manual, but it's distributed Well, we hope across multiple teams one team is doing it all right now Also, we have 100 test coverage on our unit tests But the idea is that we probably should have tests that run like whatever responses that we get from mimic and validate that against Real services too. So we know that both of them are in line Let's say I'm writing my production code. So I need to change my production code to use mimic for testing Is that how it will work or like how do I inject failures without changing my production code because Well, so it doesn't really serve the purpose, right? Yes, so the the question is how do I inject failures without changing my production code? So with the autoscale product, we didn't actually change autoscale itself Autoscale would just take a some parameters for how to create a server and we would relay those along Similarly load balancers have a similar facility So if you have a product where you can pass metadata through then you don't need to modify your code Uh, and if you have a product where you can't specify metadata We're totally working on that. We think that's a really important use case The question is isn't it more appropriate to call tests run against mimic unit tests than integration tests because mimic is not a real system That is uh, I I think that there's probably a right answer to that question But I think exploring that is a deep philosophical discussion about test terminology Mimic is in a middle place between unit and integration tests because you would want to like not sign off By testing it only against mimic But you would want to test against all your dependent system so that you know Your application is working fine against them before you put it out in the real world and test it against real services itself Yeah, so so mimic definitely is uh Not supposed to replace your integration tests. You still need to run your integration tests But the goal is that what mimic should be able to Help you with is making it so that by the time you get to your integration gate It's always passing already because you got really rapid feedback earlier in the cycle Um, so it's sort of unit testy. It's sort of regression testy. It's sort of acceptance testy It's sort of functional testy. There's there's a whole bunch of different tests. Yeah See, I'm sorry in the back there. Um, is there ongoing work with the neutral api With which api the neutral Neutron api Not currently, but we could definitely start working on it We're looking for what plugins that could be of most use for people so that we can start using How many apis do we mock? So we mock very few right now the the main goal of this talk is really to get folks to start contributing to it We've got um compute. We've got identity These are partial mocks. Um, yeah, do you have any idea how much percentage of compute we mock? At this point About like 30% maybe So yeah, so we're we're trying to make it more complete. We mocked exactly as much as we've needed so far Yeah, so yeah, I think swagger is a tool if any tool like that gets like officially adopted and Stamped by the open stack api committees that yes, we would definitely like to have that. I would love to do that too yeah, um So have we ever thought about generating mimics responses from uh api discovery features rather than manually doing everything um Yes, we've definitely looked at that a little bit. The problem is some of the things we want to be able to mock are Outside the formally defined api like for example network connections that hang is one of the things that we have On our list of features to to add and there's nothing in the api specifications or the discovery features that says Sometimes your sockets don't work and but that's an important failure mode that a lot of clients don't handle very well Any other questions? Thanks. Thank you very much for coming