 I'm Alex Bretts. I work in the ZenServa QA team at Citrix. I'm going to talk on the test of the service and the introduction to ZenRT. I'm going to talk about a few things that define the test of the service. The term has been used a few times already today, for example, without really any explanation as to what it's about. Give a brief introduction to ZenRT. A quick little demo, nothing particularly special, because it would take a long time to do anything really involved. And then talk about how test of the service could be used with the Zen project. And then, obviously, Q&A at the end. So, there's lots of definitions for test of the service, as we can mean some things such as outsource manual testing, things like that. But the definition I'm going to use for the purpose of this talk is an automated test platform that is a self-service platform, so developers can submit their own code to be tested. They don't need to go through somebody. They don't need to get authorization or anything like that. They should have a good-sized library of tests. There's no good submitting code, and it just says, oh, yes, it works without actually doing much to it. You don't want to have to be writing tests for everything you submit. It should do on-demand provisioning, so it should start off with, essentially, clean machines, not have to prepare anything in advance, just have the test system take care of that for you. And obviously, as we're talking about Zen here, we'd like a good variety of hardware. This note, obviously, with Hypervisor, we kind of want different CPU types, different chip sets, all that sort of thing. Brief note there, this is not the Citrix Auto Support Tool, it also advertises itself as TAS. That's apparently tools as a service. I'll talk a bit more about test and service in a bit, but just to jump into ZenRT now. ZenRT was originally Zen regression test. There's a lot more than that now. It's an automated test platform that we use at Citrix for pretty much all of the Zen server testing. There's a few bits of manual testing on the GUI, but all the core stuff is done in ZenRT. It's a Python-based framework. It was originally created in 2005 by James Bulpin for what was then ZenSource. It started off as a bash environment with make files and bash scripts, and then turned into the Python core we've got now. As of August 2013, it celebrated executing its 500,000th production test job, and in September, we open sourced it. Now, I'll say up front at this point that the open sourcing was, to some extent, a lobbed over the wall, which obviously isn't ideal, and we're trying to sort of improve that now. So, what are the capabilities? The key thing we've got is job scheduling, so you can submit jobs to it, and it will schedule them and run them as and when. When you submit a job, you specify what its requirements are, so how many machines it needs, any particular CPU-types, particular memory requirements, storage requirements, that sort of thing. And then ZenRT knows what all the test machines it has are, and will essentially work out what machines it can run it on and set it going. This allows quite high utilization of hardware, so with the production set up at Citrix, we get quite a lot of use out of our machines every night. They're basically running, every single machine is basically running solidly for about 12 hours at least overnight. So we have full host provisioning support. So I mentioned the clean machine principle. We don't want to be affected by whatever last happened to run on the machine, so the first thing we do is we wipe it by basically power cycling it, pixie booting it, installing whatever we're working on, be it a Zen server or probably a version of Linux in the case of Zen. We have integration with storage and network devices, so we can talk to storage arrays like NetApps and things and provisioned LUNs and nice SCSI targets and that sort of thing. And integration with network devices, so we can create VLANs on switches, configure LAPP, turn the ports on and off for testing link bonding and things like that. As I said originally, it was called Zen regression test, but now it does a lot more. So as well as basic functional testing, you can do scalability tests, stress tests, soap tests and performance testing. On that last point, the Zen server performance team actually makes quite extensive use of ZenRT because it lets them run lots of tests against different builds, get the results all automatically and they've written their own little tool that takes the results and creates graphs and does analysis of them. We have a large variety of guest operating systems. Currently, 31 different Linux distros, 26 Windows guests. I think those numbers might have gone up a couple since I wrote this. And there's a lot of workloads available. So we've got commercial things like burning tests and then open source tools like NetPerf, IPerf, LNBench. These can either be used to actually get results out of, so for benchmarking or just to generate load for stress testing and that sort of thing. So mention earlier the scheduler. So what we've got here is a little layout. We divide up our test machines into sites, each site having a controller. The reason this is done is that the controller is where the actual test code runs and while Python's not the worst language for memory usage, it's certainly not the best. So if we tried to run everything from one controller, we'd basically cause it to melt in a big pile. So we separate test sites into sites, each site having a controller. It's nice and simple. The controller then has its own storage allocations, own networking devices and knows all about them. So when you actually want to submit a job, the user submits a job to the scheduler. The scheduler then has a look at its sort of database of machines and their currently running jobs, identifies a suitable machine and schedules it on that machine. The controller will then periodically check in. Once he finds a job, it executes it. Nice and simple. All the communication with the scheduler is a very simple HTTP API. There's various tools. There's a command line tool in the demo. There's also within Citrix, for example, at the moment we have a Windows GUI tool that just makes it quite easy to monitor the status of jobs and submit them, cancel them and that sort of thing. Within Xenote, we have an object model. So all the real world objects are hosts, guests, all that sort of thing are modeled as Python classes. The reason we do this is it gives, well there's two reasons. One, it gives the standard interface a method. So the test case doesn't have to worry about starting a VM. It can just call the start method on the guest object and the library code will take care of it, which makes it very quick to write test cases. The other reason is by maintaining this model we can keep track of the state that we expect the system to be in and then check it later. So rather than every time going, I've started my VM or I better check if it actually started, just call start, the object model knows that VM should have started and will go in automatically in the library code, check the VM is running, if it's not, it needs some diagnosis to try and figure out why not. So to say here, objects provide check methods in order to ensure the actual state is what we expect. And we also, obviously, being an object-oriented language we can use inheritance to avoid having to implement check methods and library code multiple times. As a brief example of this, the different tool stack thing I mentioned, this is a comparison of a, it's a very simple method, it was just one that was short enough to fit on a slide, comparing the create network call, which is just to create a virtual network, from the libvert implementation we have, with the zen server or zappi implementation. So, sent you from a test case, you don't care about all this, but in the libvert case, it's using the virtual command and in the zappi case, it's using the XECLI to run a command. But from a test case point of view, you don't care, you don't see the difference. A few definitions are useful at this point. So with Insanity, a test case is a python class that derives from a base class, that at minimum implements a run method. There's other methods that can implement, see one of them in a minute, but has to implement a run method, that's where it actually does the work and passes or fails. We then group those test cases into a sequence file. This is just an XML file that can group them into serial or parallel blocks, so we can say run these tests in sequence and it also has a prepare section which can define the environment that's needed, so how many hosts, in the zenn server case where there should be a pool, any guest that need installing, any particular storage that's required, all that sort of thing. A job is then an execution of a sequence file against a specified input, so zenn server that would be a build, case of open source then that would likely be either a built version of zenn or possibly some RPMs or whatever. The next test case, hopefully it's readable, the aspect ratio is going to be wrong on the projector, I'm not quite sure why. So what this test is, the actual issue of this test is just a regression test for an issue we found in the field that in a certain situation large packets could cause a problem in I think a net back and the guest would lose connectivity. So the details of the problem aren't too important, but this test just reproduces the scenario and makes sure that it doesn't happen. So the first thing it does here has this prepare method. This runs before the main body of the test and it basically sets up the state. The reason we separate it like that is that if the prepare method fails, obviously there's been a problem, but it's not the problem the test was trying to detect. So it just means something went wrong, could be another bug, could be an environmental issue, but we know it's not actually the thing we were trying to test that's failed. So the first thing we do here is call get default host. That's just a way of saying I just need a host, I don't mind, I don't need any specific one if we had multiple hosts going into the job we could browse specific ones. So we just call get default host and that gives us the host object. We then for this test need a guest so rather than having to worry about any sort of complicated configuration files or anything like that we just call create basic guest and passing the type of guest we want in this case a sentos 5.7 vm. That's literally all we need to do is then take care of that, install the guest for us, make sure it's working, find its IP address, talk to it, copy a load of useful tools and things to it and hand it back to us. We then in this particular test need to install the IPerf tool and this is a good example of the inheritance of the library methods. We just call the same installIPerf method on both exactly the same code, nothing complicated there, it's just simple inheritance and it'll go away and installIPerf for us. Final bit there is just called check reachable, not absolutely required but it's just a belt embraces, make sure the guest hasn't died after we installed IPerf on it. In the run method we then actually have the actual core body of the test. These step entries are basically, they're essentially comments but by doing them like this they actually appear nicely in the zennoty log so if you're trying to triage a failure and figure out what happened you can see the step comments and see what we were trying to do as well as what we actually ran and what happened. In this case we set up an MTU on the guest and the exact guest line here and the exact DOM zero. These both just are calls that run via SSH so they just SSH's into the guest and runs whatever we need. If we had a Windows guest we actually have a little XMLRPC demon that sits in the guest that can run commands on it as well so we can do a similar thing with Windows. So yeah, we start up IPerf on the host the IP table's bit there is just for the default zennotyver file, we'll block this. Run IPerf on the client sending some packets and after we've done that we need to make sure the guest is still reachable. If the guest wasn't reachable that call to check reachable there would throw what's called an XRT failure exception which would cause a test to fail and be reported as a failure and would get suitable data. Zennotyver then once this test is finished either pass or fail go ahead and collect all the logs from the host that we might need so virolog messages and those sort of things and if it can from the guest it'll make some attempts to recover the guest in this situation so if we had lost connectivity re-starting it, see if it comes up try and get as much data as possible. So when we get on to the brief demo this is just going to be this test running this is the XML sequence file for this test details aren't too important but basically it's telling zennoty that this is a Clearwater machine Clearwater is zennoserver 6.2 just saying we want a single host nothing special, that's our prepare section and then just what test case we want to run the TC identifier at the end there is just an internal reference in the Citrix bug tracking system so I've set this going earlier today because the actual host install and guest install would take a while but I got told it to pause at the end of the prepare stage so hopefully if this works so hopefully this is readable you can see the pausing for user assistance pausing for demo purposes so that's where I told it to stop so I'm now just going to interact with this the zennoty CLI so this command basically will connect to the running zennoty process and just tell that test to continue and if we watch this screen hopefully shortly, I'll just get rid of this when it next pause to check there we go, it's now starting the test so you can see it running the commands that we had in there so it's set the MTU it's stopped IP tables it's now started IPERF on the client and shortly it'll hopefully come back so it tells us the guest did not lose connectivity so that'll work fine it would then go on and do log collection but again I told it to pause to avoid flooding the screen with unhelpful messages so that's a very brief introduction to zennoty so then the question is how can test the service and that's not necessarily zennoty but it's an option for it be used with a zennoty project so why are we trying to do this at all the goal here is that the advisory boards willing to fund an infrastructure to, well, these are from the advisory boards August meeting minutes to increase upstream zenn hypervisor quality including quality of latest CPU and platform features I won't read the rest, you can read that yourself which is obviously a good thing now one question that's come up about this in the past is why would we need to do this for zenn if Citrix is already doing it with zenn server and the reason is two fold one is that zenn server isn't necessarily using all the features of zenn anyway the other is that if we can find issues before we actually make a zenn release that's a lot, lot better zenn server only consumes zenn once it's actually been released it does back ports and patches and things like that but it only takes the releases so if we can get these issues found and fixed before it goes out that's going to be a lot better there's quite a number of challenges however we're doing this so compared for example to the implementation at Citrix there's certainly a load of security challenges at Citrix zennotys on an entirely private network only Citrix employees have access to it so there's not really any security issues there abuse of resources or anything like that because there's contractual remedies if anything happens and the other then big problem is obviously we're installing various guests including potentially old versions of windows that might have all kinds of vulnerabilities that are unpatched we don't want to let those loose on the web we'll have ourselves a botnet very quickly so those are some issues that need sorting from a licensing point of view there's potentially some questions around windows licensing it's not quite clear how MSDM would apply here and that would need to be resolved and there's similar issues with various commercial benchmarks and test tools that zennotys uses it's not really clear what category a license would fit into because it's obviously not a single user if it's open to the development community the remaining two sets of challenges are mostly just financial issues so resource wise we'd obviously need some hardware to run this on some hosting for it some members of the zennotys advisory board have sort of suggested they may be willing to help here so hopefully that one can be solved quite easily and then the next one is the ongoing management and maintenance of this sort of environment so hardware fails needs fixing there's potentially new hardware once going regularly and the software environment would need maintaining and fixing up for any particular issues so talked about zennotys and talked about test as a service so the question is zennotys can make a good test as a service environment we have seen that within Citrix the question is, is it the right choice for a test as a service environment for the ZEM project so what we're trying to do here is a proof of concept we're actually hoping to have this ready for this event but unfortunately a few things slipped and it's not quite there yet we're working with Oregon State University's open-source lab try saying that a few times quickly to set up a three-host environment where we'll do access to via an SSH gateway that just gives us solves most of the security issues we'll obviously have to restrict what guests can do and things like that and initially a limited set of tests because we don't want to put a huge amount of time where a lot of the zenn server ones will just work but inevitably some of them are going to be making assumptions about the way the system behaves that they aren't quite valid with ZEN the big to do here as well as actually getting the hardware set up is we need to put an implementation in to actually use the Excel tool stack as opposed to the ZAPI one however that's not a massive amount of work because of this object model we use essentially we just need to implement the minimum set of methods to boot VMs up and that sort of thing and then that should be done okay so what are the next steps here we need to obviously complete the proof of concept implementation that hopefully will be able to get done very soon there was the first meeting of the ZEN project test framework working group yesterday I'm not quite sure how that goes unfortunately I couldn't make that because I was travelling up here at the time we need to once we've actually got this proof of concept running we need to try and encourage people to actually give it a go try running some tests see if it's useful assuming that ZENRT is decided as the right answer and that the whole principle is actually worth doing then we'll just obviously to iterate address any of the remaining challenges that are there particularly the licensing ones and that sort of thing and then publish the code and the live environment so as I mentioned earlier essentially the open source release we did of ZENRT we basically just chucked the code over the wall partly because our internal version control system had various bits of history and things like that that we couldn't release for commercial reasons and that sort of thing so what we need to do if we actually take this forward is set up a public repository, probably a get repository create the live environment and work out how that's going to work in terms of submitting tests and all that sort of thing okay I think it's a bit early there but are there any questions? I don't know if we have the mic so if you go back a slide I've got a a slight question about the order events here which seems to assume that the community will decide whether or not we like it before we've got to grips with the code and given that essentially we want a testing system that's ultimately owned by and can be updated by the community surely these things should happen in another order that's a good point, I mean I guess the difficulty here is that because we are still using ZNRT obviously day to day for ZN7 testing we want to try and avoid creating a public repository that then diverges and has to be brought back together balancing that versus the benefit of it being in a repository as opposed to a table to have a look at and play with The code is out there, yes it's out there in a form of a table rather than a version of controlled repository This isn't a question about whether it's a table or not, I can turn a table into a git repository just to find what I can't do is take your target or your git repository, whatever it is and run it and try and change it in order to understand whether this is the thing that we want to have I think what I would like to see is I'd like to get to criticism of it I'd like to try running it in a little toy instance and see how easy it is to add a new test case and so on A couple of things to say to that The code is out there, there are some instructions on how to set it up I have heard today that someone has tried it and struggled so I'll need to try and figure out what's going on there We did test it ourselves following the instructions several times, seemed to work but I've obviously missed something I've also been working on and there's a link to putting some documentation up on the Wickey, that's a work in progress at the moment there's still a lot more to put up there a lot more to sort out so in terms of setting up a test instance that's doable I'm happy to help That's no problem there's no objection to that obviously The other thing to mention is something I didn't cover in here because I thought it was going to not have enough time although I should probably have done One of the things you can do as NIT is submit test case code with a job so you don't have to have it you can just submit a job with a specific test case and it will run that for you so if you actually want to try writing a test case you can do that without having to actually put it into the actual deployment That's interesting Does that run on your control? Yes Any environment that we set up for the Zemproject would be a separate environment that would have to have some level of access control to avoid any problems there It's not the exact info there Any other questions? Thank you So you talked about the provisioning and setting up hosts and guests and so on Obviously back in 2005 there weren't very many ways to do that these days there are things that have similar functionality perhaps some go further, some don't go as far Cloud orchestration stacks for example do some of that We've got lots of provisioning tools part of the DevOps community that we didn't have before Do you see that one future could be to use NIT's library and test cases but move to maybe an off the shelf provisioning tool? That certainly would be a good thing to try and do at the moment it doesn't need much maintenance but obviously anything that's implemented separately will need maintenance work and if there's off the shelf tools that do it and probably can do it better than the current code can then there's no reason not to move to them Obviously it's balancing the benefits of that with the time taken to do it and the risk of issues but yeah Any other? I missed the beginning of this so you may have covered it so I apologise but could you give some thoughts on a better worse equivalent of OSS test? Bit tricky is to be honest with you I haven't done very much at all with OSS test myself so I can't really give a comparison that would be any way fair because I don't really know enough about OSS test to comment on that I'm not sure if anybody who has used both anybody at Citrix for example who might have used both but was that ten or ten a while ago though rather recently? It has improved quite a bit since then I will hasten to add Do you want to say thank you? I don't want to get into a future comparison if nothing else it's slightly a bright angle anyway What I will say is that a lot of the work has been done by the Centre of Performance team to add support for Libvert for testing KVM and VMware for getting comparative performance results actually means that there is now the abstractions are a lot better and more thorough so actually implementing other support for other toolstacks like Excel shouldn't be a massive job How many OSS tests do we have right now? The main weakness as I see it in OSS test is as a body of software is that it's got not very many tests in it obviously our installation isn't on the scale of MRTs but that's just a funding problem not a code problem but it's improved It does now have a slanderlone mode that you can that multiple people have successfully used and contributed test code Do you reckon? Any other questions? A point on the OSS test and MRT relationship a a kind of predecessor to OSS test was XM test back when we had XM Zendy a long time ago and that and ZNRT actually proved to be quite good to work quite well together so ZNRT was very good at providing the provisioning the report in the integration with the rest of the world but it could then wrap and run so basically getting the unit test value from XM test with all of these high level system and lab management facilities that Alex has been talking about today Is he saying you could merge the two somehow together? I'm saying we did with XM test I don't know enough about OSS test to know what's possible there but I think we're going to be finding out in the next few minutes how OSS test has its own obviously it has its own host allocation system a job schedule and things like that I don't think OSS's main value is in its set of fairly minimal test cases although it does provide a useful set of automatic scripts for installing various kinds of guests and things that maybe ZNRT doesn't have so at that level merging things might be plausible So Do you have another one? I have a question Can you talk a bit about because typically you don't want to run if you have a large body of tests do you have like profiles where you just say well I actually want to run this thing or how does that kind of old side of things work? So within Citrix we have various things that run regularly or automatically so for example there's every time a build's made it gets put to a BVT that just makes sure that we can install a host, we can boot VMs do some very simple life cycle ups so just essentially a smoke test just to make sure it's not disastrous and then there's other things we can do so for example there's a regular nightly test that gets run on the current sort of typically the current trunk branch ZN server that runs usually 200 machine hours in a night if not wait on that But nearly 2000 machine hours of time just running lots and lots of different tests which we expect the vast majority of them to pass but they're mainly there to pick up when someone's put something in that causes some regression somewhere and then in addition to that we have sort of bigger tests so we have a stress test sequence that runs over a weekend and some sort of long haul test that runs towards the end of a release process so that's sort of the automatic stuff we have but from a manual point of view people can then submit jobs choosing any subset of tests they want to run so you could say ooh I know I've just changed some code that's affecting networking I'll run this set of networking tests and it'll run them and give you the results back So you've talked about test as a service quite a lot but is this also being proposed to do a regular automated test to be part of the push gates be used not because some developer asked it to be but it's part of the release process and it's part of normal ongoing testing I don't know why it couldn't be Is that really a question for you for the testing working group committee or whatever they're called Yeah I mean essentially there's no reason it couldn't be whether it would need to be matured enough to have enough of a set of tests to be useful to be a push gate or anything like that but yeah there's no reason why it couldn't be Yeah I guess ultimately you know this is how we want to use this thing If we do that's kind of up we have to have a discussion around in the community how we want to use this and so on and so forth and I guess the first step really is to just get something out there to start playing with it because it's a lot better to play with something concrete than something on paper that was my thinking about it and then we can just see how it goes and how it might be useful Does that make sense? Yeah that's right Run out of time already so thank you