 If you're wondering how to basically make your tests more resilient, right? When we are talking in the context of making your systems more resilient, the tests play a very important role in that. And if you're wondering how to make your tests more resilient, I think Paul is one of the most qualified person to speak on this topic as one of the original co-creators of the Selenium project and many other very interesting projects that Paul has helped the community with. So I think it's a real honor to have Paul here to talk about how to make your tests resilient and it's using one of the frameworks that he's built. So Paul, if you want to take it away and show the magic. Okay, thanks, Naresh. Yeah, I'm the Selenium guy. Oops, I've got a lot of my slides. So as Naresh said, service virtualized HTTP, make your tests resilient with subvertium, which is the new thing I'm working on. So this presentation is for people who code or understand code or those that manage the people that do the coding and I'm going to feature code and demos. As Naresh mentioned, I helped start Selenium 16 years ago. My contribution was writing or organizing the writing of four languages, Java, Ruby, C-Sharp and Python for that. The technology Selenium at that stage was HTML only and we moved it to four languages and there's many more now but we knew at the outset we had to target four which were the hot languages for enterprise use in 2004. I'm also author of a book, trunk based development and branch abstraction. Yesterday I spoke on branch abstraction. I love fast builds. I hate using shared environments and that's going to be a theme throughout this presentation. So service virtualization is the field of science that all of this applies to and we're going to refer to it as SV or service virtualization but when you see SV or hear me say SV, that's what I mean. Right, so a recap. Now, Naresh went into some of this yesterday. If we have, let's say, an application that we have in production and we know technically we broke it up into a number of pieces, let's say there was a front-end, there was a back-end and there were some other back-ends that our back-end works against database services and that front-end thing today, we might call that a back-end for front-ends. It could be a node tool chain, webpack involved, zipping all of that up and getting it out to a static system in production and then that's going to call why services, maybe rest, maybe not and then inside there we have some logic and this one in lime green here might be JavaScript or it could be Java or C sharp or a dozen languages these days and then service one and service two, we're keeping there as gray because we're implying they're outside our team, could be vendor applications or they actually could be in our own company, sibling teams, same release schedule, different release schedule, it doesn't matter but really the piece that's ours here for this hypothetical team is that front-end and that back-end anyway. So let's focus in on the back-end and then imagine all the testing pieces that we're going to have to do for that. So I've got it done in the bottom right corner there so you can just keep looking at it and then go back to this larger picture. We have the rest layer within this application and we could say we have some logic in there that we could test with pure unit tests. So we might say pure unit tests are any one of a dozen test frameworks but they're directly invoking logic using the programming language in question and they're not spawning threads or using TCPIP in any way. So this has been the classic bread and butter for developer teams, agile developer teams for years and JUnit wasn't even the first but really for most it was the first unit test framework for enterprise lands that allowed us to go and test the code we were making as we were making it. We might also say that that same rest layer has some business logic that it could invoke and in the modern age and at least for the last 10 years we've been using mocking frameworks to mock out that biz logic as we invoke the rest layer again for programmatic purposes and we might say those are still unit tests. Moving one to the right, can you see my mouse? You can see my mouse. Integration tests, right? So this is controversial but let's say integration tests are not pure unit tests. There are a number of things and in this particular instance we're going to say they involve HTTP. So we have an integration test that would poke our rest layer using HTTP and I've put a question mark below that because with a number of strategies including mocking still that we could use to neutralize the invocation of the back end logic. So those three classes of test against that red rest layer we might say gives us 100% test coverage but if we did we'd be lying because that's pretty much impossible but let's say 95, 96% test coverage and we'd be super happy with that as three sets of tests that would execute two together one following the first two anyway. Next layer down, biz logic. So if we're doing our job right we're exercising the biz logic with again with direct invoke. No HTTP involved. We may or may not be using a mock technology to mock out the layer below which was we were euphemistically calling mock service, ABI and persistence. So let's go down one more. It's probably bad that I've used two shades of blue there because that's not really coming out on this presentation but we again can use unit test to mock those things and now we have more questions. Sorry, unit test to test those things we've got more questions around how do we mock services, databases and things like that and there's a job here for a sharp tech leads to choose the right way of testing this in order to make it fast and resilient but that's us drilling into our testing obligations our test automation obligations for this back end. Now mocking, not the first but possibly the most widely known mocking framework for enterprise devs would be Mockito. Lots of mind share has gone into copying the Mockito features into other test mocking frameworks for other languages, JavaScripts and others. In Python I think there's a port of Mockito from Java also called Mockito for Python but I don't think it's complete and these are for in process mocking so this when you're going to use a programmatic function and invoke to directly invoke a method call in something else that's not really the real thing, it's the mock thing but we also have a class more to do with service virtualization where we're going to go over the wire in some regard so that would be for wire mock at least it would be HTTP for Mountibank, another well known piece it would be any wire protocol binary or text web or not for postman which is a little bit shaded there it would be HTTP again but really just the list 3 would be too short a job of communicating what these are so we need to say there are dozens and dozens of commercial and open source choices for wire mocking and this field isn't new it's 10 years old maybe 8 years old okay so let's say we should drill into how we share services or should we share services whilst we're developing so top left we see development workstation so this would all be in this case 4 devs within our team and should they share a mock service that's somewhere else like in a place you could have stood up something and we'll call it the mock inventory if we gave it a business name if it's business was inventory so that could be invoking something within the company and as we're all sitting here remote now they'll be around the world or across the VPN leveraging or on the right hand side should we make this dev workstation situation only rely on the dev workstation and the answer is possibly yes or probably yes when my book and I think Nourish would agree a definite yes so what we really want on a dev workstation is to only invoke mock services that are also co located on the machine in question like I have a Mac here right so I want everything running on my Mac somebody else could be on a PC and they want the same situation running on their PC just fine so I think I'm going to say and I hope people will agree that it's better if we have a reliance only on the machine in question and we don't have a reliance on a shared central thing so if we decide that we want that locally what are we going to do for getting the data that is mock you know this mock services representing inventory or some other business named thing do we want to have our unit tests and our service itself coming from version control or do and then as part of this test situation do we want to have a database feeding the mock data that we're going to use in a test situation or what I'm going to claim we should do should we also reach into source control to get our mock data if I have tests and business logic in source control it seems natural to me and others to have the mock data in source control too in fact one of the motivations for making Selenium in 2004 was because the incumbents which was WinRunner at the time stored its functional test plans in a database and that database was proprietary it wasn't even your choice in database it was built into the Mercury tool WinRunner so as Agilists we knew that wasn't we knew we wanted to be able to go and see source for the functional tests in source control co-located with everything else about the application so we made Selenium work with source control from the beginning in fact your choice of source control it was probably CVS version back then it was just being started in 2004 so then we say if we've made that decision to store stuff in source control the mock data what do we want that mock data to look like what is the variation of that mock data so I'm going to stylize in the top right corner here three unit tests these are probably bad test names given all of our understanding of given when then in the Cucumber era but let's say they're okay for us to understand so we want to test the purchase works with some credit we want to test the purchase fails with an empty cart we want to test the out of stock Haltz purchasing and there's possibly if we were thinking about purchasing there's another 600 tests we could run that our purchase scenario is good and bad and if we could code them all and they're perfectly resilient we would code them all so we think of those as three methods three functions in our test technology JUnit Mocha dozens of others and unit for .NET people but we want to imagine what there's this script would be or what the source is for the mocks and right now I'm going to say text files okay so if the text file for each of those scenarios is pretty close to the function name the method name then I've got a one-to-one mapping between the two some some of the service virtualization technologies choose a programmatic way of setting themselves up so you could be coding them in YAML or you could be coding them in Java and saying when you get this then give this answer but I'm not going to go into those those pre-exists subartium subartium is choosing to use a text format so how do we get those mock interactions in a test invocation so we're going to have our tests we know what tests look like they're those three functions and they have some body you know and every test has an obligation to do some form of setup an interaction or invocation of something the thing you're trying to test and then a bunch of applications assertions that would follow to say whether the test was good or bad in this execution so we might say that as well as mock service which we spent a lot of time talking about we also have real service so if we had our tests those three tests that could run in record modes we could in record modes and with the real service up and available we could record those three interactions and store them in source control text files and clicking another switch and this could be a command line switch we could then try playing those back and our tests should pass in record mode and should pass in playback mode playback mode would read the same text files from source control and as a set we can commit these together we might if we're doing DDD we might work on the little library we were making the unit test that would call the library and at the same time because we've been working on tests and we were focusing on service tests we might also commit changes to the recorded conversations in one a single atomic commit so we could roll things forward I could come up with some contrived things but we'll be going down a rabbit hole if we talked about some particular scenarios where we can describe coding activity as we regularly understand it so anyway we're going to go into demo now at least we're going to talk some real code one of the things we've done for subvertium is we've focused on a particular world bank climate API so this is a HGP service on worldbank.com and it gives us back climate data so particularly rainfall for different countries for different periods of time so you can find out what the rail rainfall was in France for a particular decades and it gives it that back in XML but also you could ask it for Jason or others but it doesn't matter we're going to focus on HGP XML conversation as provided for free by the world bank but we're going to imagine that our team that would use this didn't want to cut and paste that HGP using code all over the place and made themselves a tiny library and in this case the language of choosing is TypeScript and whilst we're going to go into TypeScript now we could just as equally imagine Java Kotlin, C-Sharp, Perl, PHP Rust, Ruby, Python and go as the technology that the dev team in question was using anyway subvertium JavaScript is the library we're going to talk about and it comes with a series of six tests and as we go through we're going to talk about the record mode the playback mode and the direct mode where there's no subvertium involved so really that's not six tests, that's 18 tests anyway so I'm going to try clicking on that link and it did come up for you which is great so what we have here is a project on GitHub it's buildable but it is an exact description of what is it, implementation of the thing I've just talked about so what we have is a function so if you're not from the JavaScript world call out a method and the method takes a from year and month a two year and month a country code so that would be GBR FRA and it gives you back a number which is the number of millimetres of rainfall that have fallen so this particular source is pretty small 75 lines of source representing the API in Python it's a bit shorter in Java it's a bit longer I think you get the picture I'm going to hit the back button and I'm going to talk about tests so we have we have a mode of operation called direct so there's no subversive involved for this and we could say that we have a test and this is a spec runner what's the spec runner Mocha probably anyway test average rainfall for France from 1980 to 1999 so we can see that method that function that we had just talked about with 1980 and 1999 and FRA being passed in and then we're going to get back a number millimetres or meters I don't know so 913 of something is the expected rainfall now of course I didn't know that I just called the API and then wrote down the results and now I'm doing an assertion that we expect the result that came back from that wire call to equal the output so this is remember we do invocation for tests in the modern age and then assertions so we've made this just a few lines in Python these ones are a little bit shorter in Java they're a little bit longer but they all roughly look the same because we're doing test average rainfall for France is one of the known things that we test anyway so we said this was direct there's no subversion involved but let's have a look at record and this is of interest to the developers and test automators who would be doing this it's roughly the same that's Great Britain here's France so the same number the same invocation the same assertion but this one came with additional setup so we did some stuff to tell subversion to do some smart things as it's doing its stuff and as we're working on this after each test is run subversion is being asked to write the record back to the source back to the directory structure which will be committed with the rest of our changes so this one was the same this call for France we differed in that we had some before or setup and we had some after the test is done we need to keep the recording so for completeness sake let's look at the playback there we go there's the France one again identical and then we had some before going on this is a a clue that we can pass the test name to subversion so it knows which text file to load was doing the same in the record and then there's no save operation when we're finished because this is playback so we just tell subversion it was done and it can stop being the middleman right so this sweet this demo exists for Java Kotlin C sharp rust go and others it's roughly the same but it's obviously in a language and it's coupled to a version of subversion that is written in this language in question so we're not writing we're not interoperating from JavaScript from TypeScript we're not interoperating with a piece written in Java for subvertium which is a different choice we made for Selenium for subvertium we choose him to write the library subvertium library in the language in question or at least the language family so the Kotlin one can be used from Java or Clojure or Scala anyway there's a GitHub organization and multiple demos and multiple libraries so back to the presentation next slide a live piece this is a bigger demo this one features Selenium but don't get hung up on that we're using Selenium because we're loading a web suite of tests and the suite was written by Pete Hodgson six or seven years ago and people used it to demonstrate a compatibility with an idea there's a to-do application that can do stuff anyway the mocha suite is the thing we found interesting because it does a series of 50 odds IOs to a back end and we're sitting in the middle of that recording that conversation so I'm going to play this this is a YouTube video and you can see yourselves online and it does have a voice track so you can hear me saying something different to what I'm saying now but here I'm going and this is running a compatibility suite for TypeScript again but it's loading Selenium Selenium is going to a target URL the URL is running a series of tests slowly as you can see them the green check marks this is invoking something in Heroku very slowly the piece in Heroku is a Ruby piece that's not important just that it forms a wire API using RESTful services and that finished and that confirmed that all the tests in that mocha suite passed using Selenium assertions now it's me re-running so I pulled the command back and this time I'm saying playback and this one's going to do the same thing load up Selenium in the middle in Selenium go to the URL that was quick so all the tests that ran the first time in what amounted to like 45 seconds ran in 3 seconds after Selenium had loaded Selenium loading is never that fast and here we can see the final result page 1, 2, 3, 4 this is about what is it 16 tests there they ran so fast that it's possible you didn't see that progressing frame by frame over this distance if you go to the same YouTube video yourselves you'll see it you might be able to pause it the reason you use service virtualization is because you want that speed so that's the biggest suite that's not the way people would normally use subvertium they're not going to use it in conjunction with Selenium they're going to use it on its own for invisible tests but because this is a demo to you guys I wanted to use something that was visual right so we had talked of record and playback and this is true for all the eternity of development recording trying to invoke a test that works over the wire with somebody who uses pieces of stuff is slow and flaky flaky means that you could be observing in a continuous integration system you could be observing failures and then somebody says retry it and then it passes again you know slow means that what could have been a 1 minute build is a 10 minute build because of all the integration tests if we talk about record modes we're still slow and flaky and we could choose retries as a strategy for getting recordings to work if something runs 5 times and it passes once in record mode that's good enough for me but I don't want everyone else in the dev team to be parroting what I've just done in that achievement of running something 5 times to get it passing I want the rest of my developers to use playback and to observe fast and resilient so subvertium is this man in the middle technology in record mode or it just wholly replaces the real for all of the mock data that had stored in text and there's a number of ways in which you could use subvertium within your dev team so in-house you could use subvertium to buff yourselves from another team that would be upstream of you supplying you with web services they're on a different release cadence a different schedule using different technologies they have different standards they might not even be an agile team they could be a waterfall team and most of the time they're not shipping something you could use so you just record the last one that they used it was good and keep running against that until they've got a new version ready for you to use and you record that one and redo your tests you could do that with a vendor if the vendor is unreliable or slow or incommunicative or if the vendor is super powerful and they don't care for you and your dev your CI or your CD or if the vendor calls you and says could you hit our sandbox a bit less that happens you know because you're good at CD and your vendor's not they might say could you just not be such a pain and you might be pleased that you recorded their APIs and you're running against playbacks and then it could be that it's a vendor that isn't so powerful and ignoring you or flicking you off and it's a vendor that you can actually talk to about co-development and go in the same direction together and use and converse around the recordings which I didn't show you I should really quickly show you that so we're back to that demo project source mocks where was the one for France here it is this is the way GitHub presents it because we're actually recording in a markdown not in text but it's nearly the same thing if I show you the raw format you're looking here at raw markdown so that's a head in that's a subhead in this is a code block so we have headers for the request request body nobody forgets we have response headers that came back from World Bank and we have the response body which is a long piece of XML that's been pretty printed here so all recordings look like this and we see interaction number zero and in one of the case of one of the tests in one of the case of one of the tests we see Great Britain and France as data meaning we're going to average the rainfall for two countries so there's interaction zero interaction zero and then followed by interaction one so as long as they keep recording and playing in the same order we can programmatically use these in a build back to presentation present so different versions of Severtium we have done or have in progress Ruby, Python, Kotlin .NET Rust, Go, Elixir, Node.js which is written in TypeScript available to the JavaScript folks too and Dart Dart is the language that Google have made to go with Flutter which is the new hotness for Fat User Interface Design for presently mobile phones Android and IOS but soon to be desktop too we have not started Swift, PHP Pearl or R and obviously this is insane it was bad enough in 2004 with four languages so we multiplied that up key concept so we talked about this business of recording and then asking devs to just use playbacks so we imagine here you can see my mouse currently right so this one at the bottom right this is developers on their workstations they run stuff before they commit and push but they do commit and push and then if we're doing the GitHub pull request model something wakes up Travis, CircleCI, Jenkins it works up and it runs the build for that pull request branch just to make sure that it's good so that makes a good or bad determination which is pass or fail and it would run compile, unit test and then the integration test including these subvertium using tests and then we also have after we've merged into a master or trunk Jenkins, Travis or CircleCI wake up again and they rerun exactly everything they did previously but this time they're making a determination as to whether something in the most valuable place trunk is good or bad, pass or fail and in this particular case if we're a high throughput super team we probably get Jenkins, Travis or CircleCI to revert the commit if it fails in trunk so it gets rolled out and then the build goes back to green again and the dev who broke it inadvertently accidentally or whatever fixes it in their own time and then goes back through the pull request cycle but this one here is the continuous use all of these three but on this side of this diagram which is an attempt to describe this concept we have an imagination that there's another job that attempts to re-record the same web API so it's the same tests but it's flipped that switch that says use record mode so it's going to overwrite those markdown files and this one is not something we run every commit it's not hourly it could be daily or it could be weekly depending on how often the vendor or the sibling team updates their web API and this one the job of this is to do the recording pass the tests as they would have done but also to fail the build if there's any differences in the recording versus the last time we did it that's easier said than done but we've got some little pieces built into subvertium to allow you to neutralize things like date differences but in the end you're going to do this job's going to do a get diff and if the diff says there's any differences on those markdown recordings it fails the job which would signal through the alerting system that somebody should look at why the recordings are different today than they were yesterday and then you make a decision to the vendor secretly change something in their API and they do do that even when they say they're not going to or is something broken on their side that retry didn't fix still there's things you'll do there to make this more resilient over time additional commits or conversations with your internal supplier or the vendor around you know what's going on here what is that you know test data could disappear and you would bake that into your tests is one of the common scenarios for agile teams anyway some pioneered this concept years and years and years and years ago and they called it technology compatibility kits others call this contract testing in the modern era and there are many other technologies that speak to this including the stuff that Noresh was talking about yesterday now I think this is my last slide help me out here okay so this is me and a bunch of people and every now and again somebody rolls up and says can I help you do this thing so that Elixir was most recent but I need Swift and PHP the two biggies that haven't been started yet we've got a way of working here that allows us to do one baby step at a time you know a little cone like way of developing something that's eminently stop startable and I find that fun even when I don't understand the language in question but I'll pay with you on your language but I'll help you make the right decisions about how to build this thing out and then as well as building these towards completion there's plenty of room for more speakers on the tour there's many more conferences than I can cover there's meetups and you know if we manage to cross over and to successful then we get our own conference series in two three years from now you know if Selenium is more than one conference on the world stage then Sovertium could be too smaller maybe than web UI sorry or web page clicking technology different challenges but it's still ultimately valuable to to the community right so I have a few minutes to go and I think that was my last slide yes it was so I'm ready for questions all right awesome thanks Paul are there any questions from the audience if you have please put them in the discuss under the Q&A section so the first question I had is you know what happens when a service does not exist if a service does not exist already could you create the test data by hand yes so you could proactively create the test data while the service is still not available keep running your service actualization against that and then whenever the service is available you can validate your data with that well let's say it's you and your dev team right and you want an inventory service from a team inside the same organization but they're waterfall and you're agile and you can't wait for them right so what you do is you work on your little library you hand write all that mark down you know I want Jason back it should look like this this is test scenario one his test scenario two and you write all of this stuff and as the other waterfall team brings themselves up to speed you use this as your contract with them like hey buddy could you make this thing yeah yeah no no you choose your language you know Ruby's great Ruby's great but we're a we're a AWS team but what you do is you negotiate inside your own company using the mark down for name scenarios like you know we talk about shopping carts and inventory here's me trying to purchase something from the shopping cart and you're the inventory service or whatever um yes in one word okay cool perfect awesome uh just moving to the next question here uh you could use an older version of the response you could be using an older version of the response data and then the provider could have moved to a new version which means a contract incompatibility yeah is there a way to keep the test data in sync yes but it's going to require human interaction so that nightly job which is going to attempt to rerecord right it discovers that the vendor moved there at service forwards so that's in-house meaning it's a sibling team or it's a public service including Amazon um you know that Amazon doesn't have a division between your sandbox version or the next version of what you would use and the production one and Amazon just has production so using Amazon for one or Google you could find that they just surprise you with incompatibilities in things so your nightly job or if you wanted to run it twice a day or whatever is going to discover incompatibilities as it attempts to rerecord and it's going to flag that through the regular alertings system to say hey these 18 tests failed after 412 a human should probably investigate this so it's definitely not continuous integration it's discontinuous but it's a safety net your team works against playbacks as fast as they can develop but you have a safety net that's double checking against incompatibility without that having to be somebody reading release notes or going on a comfort call with the vendor you're just going to catch it when they change the wire API and fast-fail tests in record modes meanwhile even though those failed the devs are not impeded because they're still running against previous playbacks cool sorry the last question from me so from what I understood is Serpentium users in the playback mode it uses the over the wire mocking stubbing approach is that correct yeah both record both record and playback are emulating in the middle right so either one of them could affect the outcome of the test but there it's a man in the middle you know two modes of operation as a pure man in the middle or as a proxy over HDP but yes it is over the wire you know it's not in language correct so my question was if that's true then why would you need language specific implementations if I'm so what I was a user of Mountiebank previously and I was trying to make a Java back end but Mountiebank is written in Node.js if I'm in a test suite test suite or testNG I need to on a just-in-time basis bring up Mountiebank on my local host and I've got choices I can spawn a process per test or I can spawn a process at the beginning of all the tests that I would run I need to change the actual mocks per test so that's process choreography a process on a machined Mac Windows Linux and another process which is Mountiebank Mountiebank has its own dependencies its own install tool chain blah blah blah which process coordination isn't easy in a fast test loop so I'd rather actually just have this as a library in the language in question meaning if this was Java I would just use new and start and that'd be a thread not a process so I take the hit I'm porting this to every one of 12 languages but the benefit is it's blisteringly fast super reliable itself and it's possibly easy to set up just like dragging in Mochito another dependency in my list of dependencies for my build tagged as the test technology I bring in subvertium I'm using it like an API it always works unless somebody's locking you can see the port here port 61417 if somebody's already listening on that port then probably my test suite is going to fail but it's always me that's listening on that support so I didn't kill a previous test suite properly cool makes sense thanks again Paul and this is a really interesting work that you're doing here again to push the envelope on making our test more resilient and making the developer experience much more better for people so thanks again for that and greatly appreciate you coming to the continuation