 Thank you very much for stopping by. So I'm going to be talking about generative testing. It's a little bit different from a lot of the testing that we do today. It has some interesting properties, as well as some interesting caveats and challenges. So first off, just real quick. Basically, I'm just some guy, engineer at PayGarden, giving me normal places. I work a lot on functional programming stuff. So we're actually a closure shop. I do a lot of work in OCaml as well. So I'm approaching this from a very functional programming mindset. And this talk has some pretty strong caveats. So first of all, this is mainly meant to be a thought experiment. So we use a lot of these techniques ourselves, but they're not necessarily things that you're going to be able to graft onto your existing code base right away. They take a fundamental rethink of the entire architecture in order to be able to pull off a lot of this stuff. Therefore, it's very well suited for greenfield projects, stuff where you can kind of come in and lay a very strong foundation in the beginning. I assume things like a functional stack, and I mean functional in a functional programming kind of way. So Scala maybe, Clojure, OCaml Haskell, stuff that lends itself to very strong, immutable, persistent data structures, even immutable databases, in this case. That'll come back to help us a little bit later. And then this is not a basic talk. I assume that if we're doing this kind of thing, we all understand how we're going to have a fully parallelized setup. So we're not running a million tests sequentially in an hour. There's a lot of parallel that needs to happen here. And that needs to happen at the browser level. That needs to be able to happen at the server level, even for your testing infrastructure. And then your runners and your reporters need to be able to run in parallel as well. With the Greenfield project and thought experiments, a lot of times I think there's value in saying, rather than figuring out what can we incrementally do better about our existing code base here? What if we were starting from scratch and we wanted to kind of create a perfect setup? And we'll never get a perfect setup, but we want to try something totally new. And you kind of see what you can achieve given no restraints. And then based off of what you can do there, you figure out how you can slowly start to graph that back and backport that onto existing code bases. So with that, kind of start off with what is testing for? I think you guys will all agree that testing is for exploring state transition space for invalid states. Is that right? I mean, it's a terrible definition, but actually it will help us kind of think about this a little bit later. So to give an example of what I mean, let's just imagine we have a symbol cart on a PsyTorn app. We just have a couple of steps, right? Step one is we're gonna add an item and that's gonna update our states. So now our item IDs, we have one item in there. And we can take that state and we say, all right, is that state valid? So yeah, sure. We remove an item, it's still valid. And if we add a non-existent item, that is not valid. So this is what I mean by an example of a state transition, which is invalid. We should not be able to represent this kind of thing in our code, usually. Oftentimes this takes a form of maybe we do several steps and then we run a in order to build up the states. So for example, I go to the site, I log in, I add several things to my cart and then I click on this thing. And really what we're doing is we're building up states until that final step and then we're running a test over that final transition where we say all right, whenever I click checkout, I should have this many things in my cart or whatever. So this is kind of useful because whenever we're traversing the state graph, there are times where we wanna actually narrow in on one specific transition. So testing today is largely kind of example-based where we hand write unit tests. And this includes things like page objects and whatnot. It also includes things where we use an IDE maybe to record our tests. And it's kind of an interesting idea because we think we can kind of manually numerator all of the important examples. Really is what we're saying whenever we write one of these tests, right? Whenever we write a cucumber test and we say, as a user, I do this and then I do this and I do this. And we kind of feel like we've covered a very important thing there. But actually there are millions and millions of variations on the thing that we're not really capturing. We're kind of hoping that we've got the general structure there. In fact, we're very optimistic about it because there's a combinatorial explosion between features. So if you have one feature, you can write a couple of tests and that's pretty easy. If you have two features, the interaction between them, you now have to write a lot more tests and three. The number of tests just kind of starts to explode. So for this talk, I'll just go over a little simple app. It's called Osmium. It's a book summarizing site. You guys recognize most of the ideas. So basically we can submit a book to this and we can give a summary of it and we can rate the book. That's it. We can log in, view our account, update our account. This is about as simple as you can get for a kind of real-worldish demo app. I think we can all agree this is a very, very simple app. This app does very, very little. And just to show you what it looks like, there is, it's meant to kind of emphasize how simple this is. I've kind of circled all of the different actions we can do from here. We can log in, we can sign up. We can view a book and we can list the books up at the top. On the login page, we can log in. We can log in, we can sign up. We list books. And then from the viewing books page, we have a few more actions. So like we can see already that there are a few actions, but overall still very, very simple app. A site flow from this might be, we can for example go from the listing books page to viewing a book. So if we're looking at the list of books, we can click on one and view it. If we are not logged in from any page, we should be able to sign up or log in. We saw that on the footer. If we are logged in, we should always be able to log out. We should be able to view our account. And then from the viewing our account, we should be able to update our account. Now if we are logged in, we should be able to submit a book. We should be able to rate a book. We should be able to update a book. So pretty simple still. We have just kind of a very simple state flow or site flow diagram here. But actually it kind of helps if we don't think of it as a flow diagram, but rather as a tree. For example, if we start off at the list books page, what options are available from us from there? So just like in the previous example, we can view a book, we can sign up, we can log in. But if we now traverse to the next step, we go to the login page, we can sign up. And then from the sign up page, we can log in or we can list books. We back up, we go to the login page and we successfully log in. We can now log out. We can also submit a book. We can list books. But now because we traversed this, that actually the state is a little bit different. Because now from this listing books page, we can log out. We can view our accounts, we can view a book. And if we're viewing a book, we can still log out. We can list books, we can view an art account. We can update the description, we can rate a book. And then backing up, if we go down this branch, we can view books, log in, list books. You guys kind of get the idea. So the idea is actually it's, when we think of it as like a site flow, we tend to think of it as like this kind of closed loop. And actually it ends up being this ever-expanding tree of state possibilities. And this is incredibly difficult to enumerate in tests. It actually, for each one of these branches, there's an infinite number of sub-branches gently. Like very few of these end up terminating. And so whenever we write or record tests, we're actually tracing a specific path through this tree. And we're hoping that kind of we've captured the canonical example. And that's gonna be enough to protect us later on whenever we do some refactoring, adding a feature of fixing bugs. And this actually gets worse whenever we consider concurrent access to a shared resource. So simple example, let's say that we have two tabs open in a browser. Tab A logs in. The tab B goes to the add books page. And it fills out the form, but does not yet submit it. Tab A logs out, and then tab B submits the book. Normally whenever we're looking through that tree, we would expect that through the linear flow, you wouldn't be able to do this, right? Because we've logged out, so that's no longer a thing that's accessible. But actually because of the concurrent nature of this, like maybe a user will do this, right? Users are encourageable, and like the way that they will find bugs in your site. And so now we have this question, like what's the resultant state? Did we create a book? Like if we have it submitted by ID, like the user that created this book, is it null? Did we write a test for it? Did anyone think to write a test for it? It's kind of the challenge there. So an alternative is just not to write or report tests. In fact, you should generate them. This is by John Hughes. He's a creator of QuickCheck, which is a very famous generative testing and property-based testing library for both Haskell and Erlang. So, or alternatively, as we just heard in the last talk, we should automate all the things, even writing the tests. So the core proposition that we're gonna say here is that it is a much better use, or it's a much better idea to use infrastructure, computers, rather than Cuban times, Cuban time. Computers are really good actually at enumerating all of the branches of possibilities, much better than human beings. We tend to lose interest if we say, all right, what if I click login seven times and then click log out and then click login again? Very few people write that test. But the computer doesn't mind, the computer will do it. And in fact, if we can teach the computer to do this, then if we have, if we write properties about our tests, for example, that we say, whenever the user is logged in, they should always see a log out link, right? That now generates, or we can now generalize that over, sorry, if we take that property and the ability to walk through the site and to generate all these different examples, that one property now is able to expand and scale over millions and millions of tests. So we get kind of constant effort, no matter how many tests we run, versus the linear effort that it takes, if we want to run a million tests of handwritten ones, we have to write a million tests. So this scales much, much better for covering all the state space. So there are kind of two properties of properties, two classes of properties that we can assert. One is about, no matter where we are in the graph here, we have certain properties that we want to assert about the state. So for example, as we're traversing it, we should never see a 500 error page, right? The user should never be able to do anything that causes a 500. Similarly, they should never be able to do anything that causes a JavaScript error. If the user is logged in, they should always see a log out link and vice versa. There are all these kind of very fundamental basic properties that we can assert, no matter where we are on that graph. And then, but more interestingly, we can actually target specific transitions. And we can say, for example, that if we're at the rate book page and we've just kind of come out of it, well, one thing we know is that the user must have been logged in in order to do that. That if we're keeping track of it, then the number of ratings for a book has to increase by one. The updated ads timestamp should have changed. So we can do a test on the properties, on the state, but we can also test on the page, on the rendering. So for example, the log out link must be visible. We can do things like, if we've updated a book, then we must be at the view book URL. After we've edited the description, it should send us back to viewing the book. Same thing with log out, count, sorry, submit book. Each of them has a specific set of properties that we want to ensure that no matter where we came from, whenever we are going into it and wherever we leave it, that there's the state conforms to certain properties and the page conforms to certain properties. So there is a challenge in teaching a computer to navigate our app. You can do things like, for example, so as we saw, every page has an inbound kind of edge and outbound edge. So all the other pages that link to it and all the pages that it links to. And the page model, as we saw and we've seen before, can explicitly list these. So we can actually encode in the page model and we can say that this page is a larger transition to these other pages. And then you can use this kind of a predicate. You can say that, given this state, here are the pages you can go to and here's how to generate the link to them. But this is pretty clunky and it's actually really prone to rot. I'm not a huge fan of this idea. If I have to go back and maintain something and if I forget to update it, and the tests just kind of silently don't cover something, then I am likely to get in trouble much, much later. So I want something where the computer just kind of does it on its own. So in our case, what we try to do is actually capture at render time. We say, given this input, which is some immutable piece of data, even the database is immutable, we say, all right, here's the input. What are the outputs that are available here? So we actually rendered the page to an intermediate form in closure, it's known as hiccup. And each of these, so rather than actually just hard coding a link inside of the page or whatnot, we use functions to say, all right, generate a button with this action name. And so we can then take that data and we actually extract that. We walk over the data structure and we say, all right, here are all the possible next transitions that we can go to based off of this input at this node. We do a couple of metadata tagging. And then what happens is, as we generate these nodes, we then walk, we keep them over to the side and we can walk down them later on and we just basically just recur. And then that's how we actually are kind of teach the computer to navigate through our site. And then each step along it away, each time we go to a new node, we can run the appropriate assertions. So now the computer knows how to render a starting point. So now we just have to list our starting points. It knows how to figure out where it can go from there. And it knows which test it should run at each node. So just giving you an example of what this might look like. This is an array that was generated by one of the tests. And it's gonna just say, at the start, go to the homepage, click on this action, fill in this password with this. You can see that in this test case, it really likes clicking on this element for whatever reason. And then it fills in the email. This is the kind of test that a human probably wouldn't write, right? It seems not really relevant to click on something five times. But oftentimes you'll find that there is some states that ends up changing because of submitting a blank form over and over or whatever it might be. So to give you an example of what a test might look like, we wanna know that no matter where we are on the graph, we should never show an error page. There should never be a case where the user sees some error. And so there's a bit of setup here. Can kind of ignore this. We're just getting a new browser and then we tear it down down here. What we're saying is I want you to walk the graph in steps. So starting from the homepage, I want you to render the page, get all of the steps that are possible and I want you to walk down each of them and then walk down each of all of those subsequent ones, five layers down. So there's this huge explosion of number of possibilities. Now each time you do that, I want you to run this function where you give me the current action and also the path that we're on. And I'm gonna run some assertions based off of where we are. So in this case, it's easy. I don't really care about the specific node and I just say, I wanna make sure that I don't find this string in any of these pages. And if I do, I wanna record the path. How did we get here? What was it that we did in order to trigger this error page? So quick demo. This has some assumptions. So we have a few fully immutable database. We have a reference to the transparent application where we have states passed in. We have the database is actually passed in. We can muck about with it as much as we want. And then if we pass in the same database later on, we'll always get the same output. And this is really important because as the computer is doing crazy things and traversing randomly through our application, we wanna make sure that whenever we replay that same path, we get exactly the same output. So in this case, I'm just gonna look at here all the failing paths that we have so far. This could be tough. I might have to talk quickly. All right, so we have no failing paths. So we're gonna start our server and then we're going to run our tests and we wanna walk 11 steps down the graph. So we're gonna go to the first page, get all the children, walk all those, get all their children, walk all of those, run all of our tests, and we wanna make sure that there are no errors at all. And so you can see actually we've hit some errors in this page. We expected not to see any errors and actually we did. So now we can take a look and say, what were the paths? And the paths actually corresponds to repro cases. It's literally saying, if you go through and you do this, you will see an error page. So here are all the different ones and we just wanna look at the last one, which will be the longest one probably. And you can see that if we just do this and you can see that it involves clicking several times here, we actually see an error page. And then you can take that and you can give that to a developer or you can put that in, you can save that for later, for a unit test or whatever. But now the computer has actually generated a valid test for us to make sure that we would not have written almost certainly. And then you can do kind of cool things where we have two different classes of tests here. I have some user tests where I wanna make sure that every time a user is created, that there is valid, that it always has the properties that I expect. And I wanna run, what I'm gonna do is create 250 threads up here. I'm gonna run that test where we walk 11 steps down the graph and then create 250 more threads at the same time. Walk 11 down those. So 250 times, 500 times we're generating the graph and walking 500 or 11 steps down and running assertions across that the entire time. So we start with no assertions yet. We've made no assertions and we have no failing tests. And this is sped up because we're spinning up a lot of stuff. This is Sauce Labs. There are some significant infrastructure challenges to running all of these tests. So you can see that it's actually hitting our site. And there are a lot of failures in this test. And I really like up here in just a moment. So you can see we start to have 286 assertions have been made so far. We have lots of failing test cases that we can actually use for repro later. And at some point here, we go from 133 to 559. Is this like we literally like break the scale? It's supposed to be halfway. Yeah, I just love this. Yeah, so I mean this is really cool. So now we actually see how many assertions did we make? We made actually close to 13,000 assertions on that when we have 116 cases that failed. So we can actually take this and those are repro cases from now on. All right, so there are a lot of challenges in generative testing though. So in particular, figuring out the right properties to test. So this requires a very different mindset because we're very good at finding examples of what we want to test. This is what user stories are all about, right? Where we have a specific story in our mind. And what you actually need to do is kind of lift that up a level and you need to abstract that to what are the general properties that we're actually looking for at any given time. And this is actually, it takes a lot of practice to get into that mindset and to internalize it. Another thing here is that reproducibility is critical. So it doesn't matter if you have that repro step. If every time you go through the site, a bunch of state changes in your database and the repro case is different every single time. You need to make sure that the app is very core reproducible. And flakiness is a really big deal here as we'll see a little bit later. Every time the test fails once it needs to fail again in the future until you fix the bug. Within a single run. Another thing is you get really long tests. So you saw that run pretty quickly. It was sped up I think three times but it's still pretty quick. But some of the tests can be thousands and thousands of steps long, right? We only went to 11 steps in there but we're blowing up a lot of browsers in order to do that. And a lot of times the thousand long step repro case, a lot of the steps are not relevant. A lot of times you click on the same element 10 times doesn't really change anything. And so if you give that to a developer that's oftentimes very frustrating. Because as a human being what do you do? You start to kind of do this binary space partition where you take out at one and you see does the test still fail and you take out another one and another one and you kind of try to shrink it. But it turns out that the generative testing people have thought of this there's this thing called shrinking. Which is pretty cool. I'll come to you in a second. The other big challenge here is the extreme scale. You are probably not going to be doing this on your laptop. In order to run, that one that I just showed will run all rights but in order to hit a million tests per hour you're gonna have to run probably 10 dozens or hundreds of servers in your testing infrastructure. And that's not just because of the test but actually because of shrinking. So shrinking, super cool idea. So we have this very, very long failing test. 2,000 steps. Which is great, right? We found a bug. We didn't have to do anything, we found a bug. But the bad news, super long. Diagnosing this bug may be longer than trying to go and figure out how to like repro it yourself. And I love this idea of like a user reporting this bug at some point. Just for having like a 2,000 step thing. And then trying to debug it. But as a human being what you would do is you take out some of the steps and you would rerun it and then you'd see if it still fails. And you would see like what are the relevant pieces here? And it turns out that's actually exactly what shrinking does and it just repeats it until it has a minimum repro case. So you can take this 2,000 step case and it will turn into a seven step case where it says here is the minimum like set of steps that I found from this long list that still produces the same error. And so this is why reproducibility is incredibly important, right? Because we need to know like if you have flakiness and the test fails for a different reason then shrinking is not going to be able to do any work at all. It needs to be able to actually find the same bug by running it again. And then kind of stepping back again to the infrastructure if you imagine you have that 2,000 step test that's a lot of subsequent tests that you then have to run to figure out what are the minimum number of relevant steps. So I think for tests, recording has its place but it's not in the browser. It's kind of in the database. So one of the nice things about this functional setup is because it's an immutable database and it has structural sharing all the way through we can take a snapshot at every step along the way. So as we're running this kind of crazy number of tests over the database, we keep a copy of every database that was generated and a share structure so it's very, very efficient. And then later on we can actually go and run tests over the state of the database. So rather than running super slow tests in the browser we can run very efficient tests in the database. So when we need to run browser based tests we wanna do page rendering and that kind of thing but we can do that and then whenever we wanna run specifically state-based tests, we can do that here. Yeah, so kind of the question is like why do we care about like a million assertions per hour and the title, it's pretty cool I think first of all. But really it's about if you think about how deep you wanna go in these tests, right? You want the computer to be able to test very, very deep in each one of those paths. The deeper you go, the longer the session you're representing. You also want them to be able to traverse over the breadth. So you wanna make sure that you're not just checking what happens if I log in, log out, log in, log out a million times. You also wanna try some things like what if I log in and then I submit a book and then I do this. You actually wanna get a breadth of coverage over as well. And then once you find these bugs, like the computer then needs to be able to shrink it. So there's a lot of assertions that happen in this. So infrastructure is a challenge here, but tractable in our case, yeah, we just, yeah. So generative testing has been used in lots of areas but mainly in memory kind of context. Nothing that's super slow, nothing that's blocking because of the huge demands it takes. So CircleCI is what we can use to scale out our servers in the test infrastructure. We use self-slabs to scale out on the browser side. And there's some pretty cool future areas here as well. So quick show of hands, anyone notice a problem like with the generative testing? Anyone think of one case where it might not be great, where it might miss some bugs? Anyone? Back there? Yeah, so there are cases where, so Erlang has a system called SimpleCheck, which is actually really good about this kind of thing. It's called linearizing the test cases. So if you see these kind of successive states that depend on one another, then you can then assert that there must have been some linear way of doing it or else it's a failure. So there are some kind of cases where this has been thought of but another kind of interesting one or technique that's based off of this is concoct testing. So it's a combination of concrete test and symbolic execution. And it's a terrible, terrible name. Like I always feel childish saying it but it is a really, really cool idea. So give me one second, let me change the color on this. One of the challenges or like one of the big cases that generative testing might miss would be something like this, where let's say that we're going, we have a function that takes a user and we wanna decide if they're like a super user based off of how many internet points they have. And we have three cases that if they have fewer than 42, they are not. And if they have more than 42, they are. And if they have exactly 42, then the whole app blows up. So we have to generate a very, very specific value in order to trigger that third case. Like we can run infinite number of tests, more than 42 and fewer than 42 and never hits that 42. So we can actually use something where we bring in code analysis. So code analysis is something, it's a bit like static analysis where we will take a function and we'll examine all of the code paths that are in there. And so in this case, we see that there are three code paths and that they're guarded by three conditions. And so what we do is we then back propagate that. What are those conditions and what are they reliant on? In this case, they're reliant on solely internet points. So on the input there. So then we will take the cases and we'll constrain what we can generate for the inputs to make sure that we actually exercise that branch. So repeat for fewer than 42, for more than 42 and then finally repeat it for exactly 42. And so we can constrain it and we can make sure that we now have three classes of values that we can generate and we can make sure that we'll actually execute every bit of our code without having to teach it how to execute our code at all. Unfortunately, so this is really cool. This is like cutting edge. There's just a recent, this is an academia still basically, there was a recent implementation of an Erling. However, this is really hard to translate into for example, a Selenium test. It's still really unclear how to do that because what happens is whenever you hit a page and you load up a page or a part of your app it's unclear what code you're executing. So there's a lot of research left to be done on like how do we know whenever we hit a page what codes we're hitting and then how do we change the input for a page to make sure that we hit the right code paths. Another really cool thing is predictive testing. So the idea is just like we did the, we had that kind of declarative that simple data structure that was a repro case for us. We actually record the actual user's site or path through our site and then if an error is triggered we automatically email the developers without repro case. And then another thing we can do with it is we take the previous 10,000 sessions and this is something I did beforehand but we take 10,000 sessions or a rolling window of the previous 10,000 sessions and before we would push any new code we would actually take those 10,000 and replay them against the new code. And so then we would have this idea before we push this code is this new code going to break anything that our users are actually doing? Yeah, so just basically in summary humans are really terrible at manually enumerating all possible stage space and computers are really terrible at being creative about it. But they are very good if we teach them to find edge cases. So we just have to give them a little bit of help. And most important thing is don't write tests, generate them. So I'll just show you a quick example of predictive testing. So this is a little demo chat app where it uses the same structure. We're going in, we're changing state, we're updating things. And then we're going to pull up a debugger and you can see we have the same number of states here and we can just kind of scroll back and forth. You guys have seen this probably with things like, I don't know if the ID does this, but it's basically a time traveler, right? Where we can go back and forth. But if we have those state-based repro cases, we can actually take the user's session, we can load it up in our browser and we can actually scrub through it and see exactly what happened whenever an error is triggered. So there's a lot of possibility if we start to go into this generative testing model which requires everything to be kind of data-driven. So just real quick, almost all the ideas here, the work and whatnot is done by Sebastian. He's a coworker of mine, he's a cool guy. You should follow him on Twitter. My company is cool with me coming out here which it was along flights and they were super nice to let me do that. So I want to thank them and yeah, happy to take any questions. So irrespective of generating tests, but let me just decide, don't forget, I'll answer the first question first. So generative testing, so it's not about parallelizing tests, right? Generative testing is not about bringing test times down. In fact, it uses a massive amount of infrastructure. Like it is required that you already have this stuff there. What it is about is saying, I have described the general way that my application states works and I have some properties that I believe are true here and I want you to try to disprove me. So I'm gonna have the computer actually search through that and try to prove me wrong and then whenever it does find something, give me the minimum case that shows that. So this in no way is going, in fact, you need to have all that infrastructure because in order to run a million tests per hour, you need to have a lot of setup, yeah. So second question. Well, humans are definitely not doing a million tests per hour, right? Yeah, also, I think nobody's, so people do this but just not in Selenium as of yet. So because there are big infrastructure challenges around, it's a lot easier to run a million, so for example, a really kind of classic use case of generative testing is let's say you have an algorithm that you're implementing and you're gonna implement a very slow version of it but it's very correct, it's really easy to read, right? And you can kind of look at it and make sure that it's right. And then you're gonna implement simultaneously a very optimized version of it. And so you're going to say that for every input into the first function, the output should match the output of the second function. And so now I can actually use the first one as a test case for my second one. And this is very easy to do for in-memory stuff, right? Like, there's almost no latency. You could also use this for refactoring of a webpage, for example. You could say that I have an old version of the webpage, we're implementing it in a new language or whatever, it's a terrible idea, but let's say we were. Anywhere I go on the old page, it matched the new page. And you could say, try to prove me wrong. It just hasn't really been done in the Selenium world because there are difficulties in how do you actually get the browser to traverse your site? How do you make that efficient enough so that you can run a million tests per hour kind of regularly and it's not really a big deal? Like, I think a lot of us need to move in that direction. But right now everything is, and then the last bit is like everyone, like the status quo is to write your tests manually or to write specific example case or to record them. There's very little thought over the general properties of our application. So I think very few to no people are doing this right now. No, so it doesn't, in some ways it replaces it. So the way we always say it is, so we have, for example, the predictive testing. So the first thing is, we have everything our users have done beforehand. We run that through our new code to make sure that we're not breaking anything that they would want. We then have generative testing that explores a bunch of the state space. And then after that, if something actually makes it past that and we get an error from a user, so we'll get a nice email with the repro steps, exactly right, and say, here's how you get this user's case. In fact, you can load it up in your browser and you can scrub through it. Often times we'll take that and generate an automated unit test based off of that. So it kind of augments it. So most of the leverage is gonna be in writing the generative tests, but you can take the specific examples that have gone, like, have made it through those gauntlets and then just use those to make sure that they don't happen again. Say this again. Oh, yes, definitely. So there are often cases where, depending on what level you're working at, right, where maybe someone logged out, but actually there was a bit of state that wasn't cleaned up, and that was only apparent if you did a couple of other things to the site and then you submitted something over here. And like, you know, as humans, we always imagine when we write that logout code, it's so simple. There's no way there's a bug in this code, right? And then there's no way that this bug here would affect that thing over there. And so that's where a case where having the computer actually go through and say, by the way, I found this bug, which this is basically the mirror of having a lot of users, because if you have a lot of users, this will happen. They will do generative testing on your site, basically. Users do crazy things, where they'll log out and they'll go and do a bug and you'll have a hard time debugging it. Oh, sorry. Oh, so we use Clojure, and then we use Selenium. So the Clojure is a functional Lisp on the JVM, and then we use the Clojure WebDriver. Yeah, I don't. Yeah, so the code is up on GitHub as well. I'm happy to talk about the specific implementation. Like, it's not as relevant as the ideas, but yeah, we use Clojure on the JVM. We use Selenium WebDriver stuff. Someone stopped me if we're running out of time. I don't know. Yeah, so it took quite a long time to kind of think about what are the properties that I actually want to assert about my code? But I think just after you do it for a while, you get a good sense of it. It does mean that you need to be a decent developer. This is not a manual QA kind of thing. You can take the manual QA test and try to figure out what's the actual property that they're testing for? Because underneath each example, there's actually a generalized property that you're trying to prove. And so you kind of learn to suss out like, what is this thing is actually doing here? But really, yeah, so like I said, one of the common ways is to compare two alternative implementations where you're trying to do a new thing, and it has to match up with the old one. That's really critical, but you kind of, for whatever reason, are kind of scared to touch the old code. And so you just want to make sure that the new code matches the old code. So that's a really nice way to start with it. And you get a sense of how all the pieces work, and then you start to figure out, all right, what are my generalized properties that I now want to, that's entry level, and then you can go to the graduate level, which is I'm going to now write my own properties. So the unit level is a lot easier, I think, because it's generally gonna be in memory, you're not gonna be bringing up a browser and tearing it down and whatnot. So you have a, like one of the killers for development productivity is the turnaround time, right? So if it takes me five minutes to test something, I only get to test that a couple of times per hour, and that's really, really, like that kills my flow. If it's a unit test or integration test, you can probably keep that turn cycle down into the milliseconds, right? If we hit it into it, it runs a million tests, or a thousand tests, it doesn't matter, and gives you an answer. And I would actually encourage you to start there, but I think from maybe you're thinking about the philosophical direction, like how do we encourage developers to do this? I think it just goes to, like, drawing out that state's diagram was a big kind of mind-blowing thing for me when I started to realize, actually, I'm just scratching the surface here, and I don't want to spend all of my time, like with these examples, and then once you actually get a couple of test cases in, it's like going from no testing to testing, right? Where you say, actually, I did have a few bugs, and I'm glad I have testing. And when you go to generative testing, and it starts to find bugs that you didn't realize were even possible, and you think, oh, I'm really glad that someone did that, then you start to feel like this warm, blanky of security around you. So it's just kind of an experience where you have to gradually get someone in, get them to have a few success points, where they feel like, all right, I feel good with this, and then once they feel that sense, I think it becomes this obvious thing where you want to invest more time in it. Data-based API, oh, how, so, how is this, when you look at my review? Yeah, so like I said, this is a thought experiment, right? So, I mean, it's not that we actually do this, but this isn't a promise of you can use this today in your existing code base. So there are ways to do it, where let's say you were to, for example, I don't know if Oracle has this, but if you had a mock in-memory database, where it provided the same API, this is kind of crazy. I don't think it would be a good idea at the Oracle level, but then you could keep a copy of every database. So we use Datomic, and Datomic actually provides this out of the box, where every transaction is actually kept in history, and we can always record, like, we can zoom back to another point, think of it like git, we can go back to a previous state, and we can fork the database and use a clean state from there. So we can always go back to any previous states, and then that helps a lot with the reproducibility, because we're pointing at that point in. You can see what the rate of, there's one, there's one there that, how many, like, two, six. So it depends, so it's a really good question, actually. So we actually give every test its own database, but its own database, in our case, is just an in-memory data structure, sure. So everyone gets their own environment to call it, their own kind of clean, pure environment, unless we are testing for concurrency, and we want to know what happens when we have multiple users testing on it. But that's a very explicit thing versus this, like, we'll just randomly throw everyone altogether on the same database. One more? Oh, so we're already doing Selenium for a bunch of other stuff, right? We're already driving Selenium through the page, and we're making assertions about the rendering of it, and we're trying to mimic real-world behavior via the browser, or via the mobile app, and kind of while we're there, we might as well make these assertions. And oftentimes, we do, we won't use necessarily Selenium to verify the status code. We'll just use that, like, we'll look up the request ID, and we'll say what was the status of that from the server point of view. So, you know, points the effort at the right level, I guess, right? Use Selenium for what it's good for, and then use the server and the back-end and memory data structures for what it's good for. All right, thank you very much.