 This is a tool for randomized testing. The paper's actually about Haskell, but the language doesn't really matter. And since it's such a practical paper for doing real things, I'm more going to just show how the techniques are used and give you a concrete idea about how to actually put them into play. Essentially, it's a technique for randomized testing of your programs and testing properties about your programs. So, say if you have a concatenate function and you're trying to test that it does the right thing, you have, sorry, I'll use a reverse instead because that's what's up there. So you want to write a function that reverses values in a list and you have some properties about reverse that you can test on it. With reverse, if you have two lists and you reverse each of the lists and then you add them together, it should be the same as reversing the lists added together before the reverse. So that's this property here. Now, when I demo this, this is all enclosure, which is a lisp. So it'll look kind of funny with all the brackets and everything. But to read it, it's really quite simple. You just move the bracket one element over and that's essentially the same thing, really. So let's come up with a reverse property. Okay. So let's write a reverse function. Let's write the property first. So my reverse of v. So this is just saying for all v's which are lists of ints, generated lists of generated ints, we want to prove the property that my reverse v, I'll prove this second property first because it's easier, my reverse v equals reverse v, my reverse v. So if we reverse it twice, it should equal the original list, right? I haven't implemented reverse yet. So we'll implement a reverse function. We want to say, let's say we take the last element, last element and we add it onto my reverse. So that's not going to work either because that needs to be a list as well. This is a very naive implementation. I just don't want to use it there. So let's just try it out. My reverse, now I've got a property, reverse twice. You want to try out this property? Okay. I know really the language, but I'm not confused because I'd be expecting that when we are doing the reverse, we will somehow pop the element. So here it seems that we are putting the element at the beginning, but we are keeping the list the way it was. I would expect the same as we need to last call. In my reverse call, we will have to do something to remove that last element. So how is that working? Yeah, well, we'll see as the properties find the bugs. Obviously, it's not working right yet. Which is hopefully going to help with the live coding because it's kind of fun. All right. So stack overflow exception, right? Let's go if empty call return that. Otherwise, concat, right? Okay. So stack overflow exception still. Well, stack overflow exception because we're never actually taking the element off, right? So, but last, right? Okay. It passed the 100 tests that were thrown at it, right? The seed that's there. Sorry. I keep forgetting them. So when things get evaluated, they get evaluated at the bottom. So we'll look at that. Maybe I'll make it a little bigger. All right. So, okay. And the seed that's there just gives you the seed for the random number generator that was used to generate the examples. So if you want to run the tests again and get the same result, you can, right? So I've implemented a very bad implementation of reverse and tested one of the properties on it. So now let's try out a different property. So let's try reverse concat. So we want to generate a vector of ints and another a vector of ints. And we want to test that reverse v concatenated with reverse w. Ah, my reverse. Yep. Thank you. No point testing the standard library. Now, let me just think about it for a second. So, reverse v, it's w. Concat v, w reversed. Yeah. So we can test this property now. So the failing test was 0, 1. So let's just see what, maybe I'll mix this up. My reverse v, w. So one thing that's kind of cool while I think about this for a second is that the failing test is a vector of 0 and a vector of minus 2, 0. But one thing that's done is it's shrunken down the example cases that's used to something that's a bit minimal that we can test on to make it a bit simpler. Yeah, should they be reversed? These two? Yeah. Yeah. Okay. Yeah. I had a feeling. Cool. All right. Now, let's say we want to test property. So this will give a good example of the shrinking that's used. So we'll generate a list of integers and let's say we want to make sure that there's no value 42 in that list. So it's generated an initial failing case of this vector, which is about 10 elements long. The shrunken version is actually just one element long, which is 42. And it uses a kind of shrinking that can shrink on integers themselves. So I think this should shrink to 0 instead of 42 every time. That's because 42 will actually shrink towards 0. So it's kind of nice when you get, in a list of really big integers, you might just get a whole bunch of zeros. It allows you to kind of eliminate that value as the cause of it sometimes. So the hard thing I find about testing in this way is you've got to think about the properties of your functions. So it requires a little bit more thinking than just unit testing because unit testing, okay, this value should give this value. You actually have to think about what your function is doing, how it works, sometimes as mathematical properties that you need to think of. The simplest one usually is if you can just apply a function and reply another function to it to actually get back the original value. So that was the reverse property from before. Another way that's quite useful is if you have just a much slower version or you can think of a much slower version that's fairly naive, but very easy to write, that can be pretty useful. So imagine, let's say, you're writing the function min, which just goes through a vector, finds the smallest element of it. You could trivially implement it by just getting the first value in a sorted list and you can just use the standard library to do that. Often, if you've got some code that you're trying to make faster and you have the slow version of it that was written and you know it's correct, then you can just take that, write your new version, write it faster and then use the old version as what you test against because it's known to be correct for what you think. So sometimes you'll find bugs in the original code. Yeah, or how inputs relate to the outputs. So for the example before with the reverse, we've got two reverse-concatted lists equals the reverse-total-concatted list. So we've talked about that. Okay, so with this test check, well, with test check, which is a quick check implementation, you can use sample on a generator. It just samples the generator. So it basically just calls into it and grabs a value. It's useful if you want to see what your generator is doing. Maybe you have one generator and you want to actually ensure that only some of the values are used. GenInt. So here I've got an int generator and I want to make sure that only positive values are actually returned by it. So that returns a generator itself. But then if I sample it, now I've only got positive inputs. All right. This sample by default does. But when you actually test the properties of your system using quick check or test check, it will generate vectors or arbitrary numbers of your values, depending on your generator that you're using. You can make them of any sort of size. There's ways to actually say, okay, only one vector is from this size to this size. So sample isn't actually returning a generator. It's just a way of applying a generator and getting some actual values out of it just to have a look at them rather than to use it in your testing. Does that make sense? Let's say we have... we want to select from some names. Bob's is not a name. So you can use GenElements. So just randomly selects from the list that you've got. Choose. Most of these are very similar to what's described in the paper. And you can find them in most quick check libraries. So choose allows you to only generate within a certain range. Return is kind of... I'll avoid using the n-word. Just returns a generator which will just generate the value that you've put in it. So if I GenSample, GenReturn3, I'll always return 3. Sometimes you just need it when you're writing recursive generators. I'm really powering through it. Does everything make sense? Yeah. So... Just leave yourself to it on-site if that's easy. That's not going to be a problem for me. Do you see it? If it moves it, it's going to be a problem for me also. How about that? Is that better? Sorry. Okay. GenBind just says if I have one generator, if I have one generator, you can essentially... I'm thinking of fmap for that one. What does it mean? It just means add one. So yeah. I gotta forget. Okay. fmap says if you have a function and you have another generator, apply that function to the generator, essentially in return another generator. It's kind of like peeking into it and just applying the contents of it. It's often useful. So let's say I want to generate ints which are powers of 2. Whoops. GenFmap says powers of 2, v2 to the v. All right. GenInt returns a generator. Then I go to the sample. Let's have a look at some of the values. So it's returning powers of 2. Let's just try implementing a function with a bug as well. So myink says we're silly and if v is equal to 0, we won't do anything. Otherwise, we'll add one to v, 0.4, 0.45. Now we want to test a property of it. So let's say probably myink. We can say since we're always adding something to it that the new value should be greater than the old value. So we'll go to greater v, myink, v. So for all v's, v should be greater. Myink v should be greater than v. Cool. Finds the incorrect statement. And so one thing that's kind of useful is to use Gen1 of which is a way of mixing generators together. So let's say we have a very simple one. So we've got my power of 2 generator now and we want to mix it in with regular ints but only occasionally use the power of 2 generator. That needs to be in a list. Now we have a new generator. Sometimes it'll return the int, sometimes it'll return the values of the other one. So now what I was thinking was, well, is everything making sense still? Yep. As I rush through it, I can try some sort of bit more TDD type testing. So let's say we have some sensors and you've got data that's coming in from sensors and every time you get a sample from a sensor you want to increment account, right? So what I'll do is we'll build a generator. The sample is kind of an event generator. So we've got events coming in. We'll just say sensor1, let's say sensor1 pings. So the event type is ping. The sensor is sensor1, right? Now we are going to pingPaste property. So positive, so positive or 0, right? So the number of events that come in should be either 0 or they should be positive. Can't be negative, right? So we'll generate some events and then we'll call our function event. We haven't written our function yet. All this is essentially saying is call the function handle event initially with an empty map, dictionary essentially, on all the values, all the events each time in turn calling it with the updated sort of count plus the latest event, right? Now we need to run and write handle event. First v ping, then we want to counts v plus 1, right? Otherwise we'll just return counts or just ignore all other types of events. Does this kind of make sense? So we've got handle event of ping sensor 1, sensor 1, 0. It should... It's not right. So what this is doing is if you get a ping for this sensor, just strip this up, sorry, ant type, sensor ant type is ping. So what we're doing here is we're adding to counts with adding to the dictionary under sensor and we're adding one to the count. I still have a problem. This is why life coding is a bad idea. There we go. No pointer exception. So we get counts sensor. I'm not doing it right. It does the job, right? I've got a property, plus or zero, and we'll have a test and we'll see what happens. Sorry, wasting time. Okay, so no pointer exception. It's the same thing that happened before, right? So that's because we're just trying to add one to the count. Yep. So we fix our bug because basically what we're doing, what we need to do is if count is nil, so current count... So we've got the current count, yep. If current count is nil, then we do something... Sorry, new count. We want one. If not, we want plus one to current count, right? Can you say nil for me? Yeah, I'm just trying to avoid getting too lispy, because otherwise, yeah. So now we want to suck counts. Sensor new count, right? Okay, it should have worked. It should have worked. Why didn't that work? Okay, so it did work, right? So it's found one issue. Now, let's say we want to make it so that we have... Not only do we have pings, but we want it so that sensors have to be registered before their pings are recorded. You know, everything before the registration should be ignored, right? So let's make this ping generator and we'll make register sensor generator register sensor one. Now, we want to event generator be one of ingenerator or register sensor generator, right? So one of those. Now, it should be able to just put it in proposal zero still. Gen one of is wrong, because it's not in... Okay, cool. Now, when we register the sensor we'll say I'm making this so I can handle more conditions. That's what cond is. So let's say event type is register sensor. Is it register or register, yeah. That type is register. Then we want to say counts. You want to say sensors. And we want to add the sensor to it, right? So all this is just saying is it says it's saying add it to the list of sensors, right? And otherwise just return nothing. When I change the event thing I'll say sensors is a set with nothing in it to start, right? So everything runs still. Okay. So now we need to test the property that all of the values before the register are ignored, right? So for all events do that. And so this... I'll just explain what I'm thinking first. So if we reverse the list of events we could just take all of the events after the registration and ignore them. And the sum of all of the events before the registration should still be should be greater than or equal to the actual count. We should have ignored everything before it, right? Makes sense. So we'll say I'll just make a handle events function which does this for us. So that handle events V should be equal to reverse V. So we'll say drop until we hit... Sorry, did I say unregister? I meant sorry, drop handle events. Someone can... So essentially we're saying from the list of events we want to drop every event until the initial registration, right? And that should... the result of applying all of the events where we've dropped it until the initial registration should be equal to applying all of the events because it should have ignored the initial events, right? So let's see what it says. I haven't added a test for that. But prop before registering, ignored. Is there only one test there? Okay, cool. So that doesn't make any sense to me. Nothing. Sorry, I flipped back because it didn't make any sense. It's saying that it passes this test because there should be cases where it hasn't ignored it. Right? That's because it... Yeah. It should be until I've hit a register. You're right. And it should be first. Yes. Okay. Cool. Okay. So this... test it found was that, you know, a ping nothing else. So what we should do is we'll say if the event type is ping and sensors counts has V in it already. Sorry, not V, sensor. So sensors are already in senses and it's a ping increment account. Otherwise, we don't do anything. And it's passed. Now we could... How much time have I got? I think you'll get the gist of it. So I was going to have an unregistered sensor, but I think much point in going too much further. Any questions? Sorry for the rambliness.