 Cool, okay, so I'm Andrew Noyes, I'm from Snowflake Computing, and I am here to talk about simulation testing. So for some background, within the context of FoundationDB, a simulation test, we're basically simulating a distributed system in a single process, and the outcome of this test is going to be determined by the random number generator. So and in particular, like this sequence of numbers returned by the random number generator, and this is really nice because if we, like, say have a pseudo random number generator with a particular seed, and we want to reproduce our test failure, we just run it again with that seed. But having it, so, but having a pseudo random number generator kind of constrains the possible outputs of your, of this interface here, in particular if you want to get very unlucky for your, for your test, and you want to return zero, like six times in a row or something like that, that will never happen with the fair pseudo random number generator. So I'm going to talk about kind of an alternative idea where you make the input to the program just some sequence of numbers returned by this. So you, you have more flexibility there, and let's kind of see how that, how that goes. Okay, so we're going to be using this example bug throughout. Basically we've got some integers here, and we want to maintain this invariant that each one appears exactly once. And so what we're going to allow clients to do is we're going to allow them to swap elements. And this preserves the invariant, so there's, there's a swap. And our implementation of the swap is going to have a bug. So there's, there's two tasks, right, so this guy can get executed separately from after the wait. And this looks really contrived, but we do really insert wait statements here and there occasionally for things like trying to fix a slow task, or, you know, lifetime issues, various things. And here's the, here's the bug. Okay. And this is what our clients will be doing. We'll have a couple of concurrent clients running. They'll kind of wait around the amount of time. And then they'll, some of the time they'll check our invariant. And the rest of the time they will swap two random elements. And this is what we're looking for here. This is our minimal counter example. So we've got two concurrent swaps, and they're going to interleave. And they're also going to swap the same element. Here they're swapping, they're both swapping nine. And then immediately after, we'll check the invariant. So this is what happens. So this kind of represents the first operation here. We're reading zero and nine. Then we're reading eight and nine. Then we're writing zero and nine. And then writing eight and nine. And what you see is that nine now appears twice. So that's what we're looking for. That's our minimal counter example. Okay, so if you were using, so I'll kind of describe the strategy foundation DB currently uses for scheduling tasks. So we've got a fair random number generator. And some of the time we are going to delay for longer than requested, basically. And that's what's called a bug-ified delay. And this is kind of like kind of the probability distribution foundation DB is currently using. And it's kind of skewed toward zero. So most of the time it will either delay for exactly what you requested or slightly more. And so let's see what needs to happen here. So to find the minimal counter example, we need at least two clients. One of them needs to get their delay bug-ified. The clients swaps need to intersect and check invariant needs to be called immediately after. And the probability of all of these happening is pretty low. So here's kind of how it did. So I tried this, one million seeds, and this is the shortest counter example that came up with. Which is a little bit expensive to fix in terms of following what's going on because most of this isn't interesting. There's kind of where it happens there. And then you finally check the invariant, eventually. So here's kind of the idea here, is instead of a fair random number generator, we'll just have the input to the program, be the sequence of random numbers that we're going to use here. And so this kind of shows how unlucky we can get with this random number generator and this replay random bytes class implements the random interface I showed earlier. And basically what can happen is you can request two random numbers and then you can crash if they're the same. And with libfuzzer, it will actually find that input. But the random number generator won't. And then so here we compare kind of the quality of the simulation. So this is the shortest counter example we know about versus time. So the fair random number generator finds a counter example and then slowly makes it slightly better, but it never really gets to the minimal one. Whereas libfuzzer finds the minimal one very quickly. And then another kind of idea, another way we can get unlucky is if we stop looking at, if we stop trying so hard to execute things in order, we could just ignore time. We notice a couple of things. So this is kind of the current thing. We've got a priority queue that executes tasks ordered by the time they're scheduled. If we just pick a random task, that also finds the minimal counter example really quickly for a couple of reasons. One is that it's a lot faster so we can run more simulations. And the other one is that we kind of took away the delay is unlikely to be bugified part of it. And so here's some takeaways. So it's possible to write bugs that are kind of expensive to catch in simulation. We can also apply existing tools for finding bugs to simulation by making this more general input instead of a seed. And making your simulations not realistic isn't necessarily helpful. We already do this with bugify a lot. So don't worry about that too much. In FDB server, but yeah, that's it. Thanks.