 Hello everyone, and thanks to TDR for coming. I don't know if you're using it all at the first time, but thanks for calling us to have an interview. So, my name is Charles Barrowas, and someone talked about computer-based testing and finding more bugs with less effort, which sounds pretty good. I work with a company called Amia, that's how I say it, and I've been doing computer-based testing for the last three years, pretty much, in some sense, statistically, so I've really been thinking about since that time, and I kind of want to pass on why I think it's a really interesting way to test it, how I think it's a better way to test it, and why the other one should kind of be looking at it as what they should be using at work. So I want to start by, you know, asking the question, or, you know, posing the question, what's the point of testing? Why are we here? Why do we continue to use all? And in some of them, I think we're going to be able to know when to write the right software, correct and reliable software, so that we write the code, and we're going to be able to check it in with the production in some sense, and have it not break us, have it not blow up in the user face, and actually have it produce the correct outcome. So testing is often at first line of the test. You've written a code, or maybe you haven't written a code, and you're writing that test to make sure that you get the answer that you think. And also then, over time, I make sure that that test is up in your impressions, or that it's a break. So I kind of want to, you know, really kind of ask the question, is testing giving us what we think is good? Like, are we getting up out of this? Are we, you know, are we really finding what we can find a part of our production? I think the answer is no. Well, not always. And I want to talk in some ways about when I talk about testing, and when I'm talking about you can have ways of testing. So the idea here is that, you know, we kind of capture this one thing, these particular arguments of dysfunction, or terms of particular output, and that's it, and that'll capture that one thing. So how do you work out what the example should be? Should know about the function, know what edge case is to look for? Well, I kind of want to use this as our special code. So I'm using Scalar in the code here, but the point is it's definitely not about Scalar at all. So this is actually the substring function from Java, or the JVM, but it could be in any language. Might as well have some way to take a string and reduce the length. So let's say we've written a substring function and we're not interested. So we write an example. You take ABC, you pass in one, you get ABC. So then we add another test to make sure that the length check is correct. Maybe we actually check that the unitary works the way that we expect. Maybe there would be numbers, maybe there's more numbers. And I guess the question is, how many are enough? When do you stop and say, that function now works, so we never have to add another example over again? I'm not very good at this. I think I've tested this code enough. It should be fine, and of course it doesn't work. It doesn't work. You don't have to come in and break it down with your program. So, I guess the thing is, to write a really good example test, you really have to know, I think, both the HKS. You have to know what you're going to get and make sure you test both sides of the HKS. Instead of us coming up with these examples, why don't we get the computer to do it? Why don't we get it to effectively generate a whole bunch of these examples and then see what we can find rather than us having to do it manually? So, let's rewind way, way back to 1999, 2000. So, this paper came out, and this library came out and we checked from Colin Carson and John Hughes. So, I've been around a long time. The idea here was that you would generate these examples to see where you could prove or you could check about your programs. We'll see that in a second, John Hughes is one of the two authors of the Pre-Check library. And since that time, since 1999, he's gone on to the founder company I'm not going to try to pronounce this. And they basically test software for a living. They go into organizations, and I don't know if they do, but hopefully they can make lots and lots of money to find positive software that the people who test it can't quite break in the ways that they can. So, I want to talk about the best example from their website, which is the one where they went into Volvo and made the money to go in and actually test how the software could see what they could do. And so, in a particular case, they were testing this sort of past system and passing messages around the car, talking from the stereo and the brakes and all sorts of things like that to see what could happen. And they have the different parts of the car being outsourced to different suppliers. There's a whole bunch of code here that they've written from these different suppliers. And so, when John Hughes company went in, we checked there are a whole bunch of properties, and they found a whole bunch of parts. They're both learning the actual supplier code, but also, like, in the specification itself. So, the only thing was the code bad or organized in some way, but also that it was never going to work. Like, the specification was just wrong. So, the other one, the one I'm probably testing by reading Matt earlier, is the one called Yes. So, it's written in a sort of maintained by a guy called James Curry, who writes an amazing blog that's called Call Me Maybe. So, if you're using any of these databases or distributed systems in production, I highly recommend you go and read one of them, because it's fascinating what kind of files it has to be. So, it only uses property-based testing in a sense to kind of throw, you know, sticking up one of these systems and test them in crazy ways, and basically it was all of them arriving in some ways that people didn't expect. And so, I like this one in particular, you know, Kyle has basically been kind of standing for, if you have one of these systems, it's just a badge of honor to have Kyle around looking at your, you know, your product, your software to actually make sure that it works the way you think it does. So, people are taking this very seriously now. Okay, so, we'll look at our motivation. Let's talk about Pitchak. Since 1999, Pitchak has now spawned as kind of usable in pretty much every language I can think of. So, 31 languages on Wikipedia page. Let's have a look at sort of how it works and what you can do with it. So, I'm probably using this term incorrectly, but the idea here is refreshing the idea that you kind of randomly generate data and throw in your function and we assist them with looking for performance or security costs. We're not really looking for any of those students, in particular, but we assist them with different ways to operate that option for how to throw exceptions and how to just not work. So, let's go back to our substring, you know, example before. Let's just look at one example. And let's ignore the output. We don't care what it actually works on. We just make sure that it doesn't break. Okay, so this is what we start asking. So, we need some way to generate a string. I mean, maybe just call some sort of random string function. Let's do it in a way that we can kind of call it over and over again. And so, in ScalaCheck in particular, which is the ScalaGridder or one of them of Pitchak, it looks a little bit more like this. So, the idea here is that you can pass it away to generate values. So, like, generate some of your A and here's a lot of the random values so you can make sure that it's correct on them. And so, now that we have this for all I'm sure we can then, you know, there's a lot of different ways of providing a way to generate random strings but also ways to generate new generators. And so, now what we can find is we can see that we're, you know, generating a random string and we're generating a random string and we're just going to see what the substring does. And if you ever use a substring function on Scala or Java before, you'll know what's going to happen. It's going to block pretty quickly. It's going to block because, you know, the integer, the offset is going to be greater than the actual length of the string. So, now, I just want to ignore that and say, okay, we'll ignore that, that's not going to be a problem. Let's have a pre-condition to say, well, just ignore anything where the integer is less than the string length. And then, this is a way to sort of say in Scala check or pre-check. This is a pre-condition for the function. So, let's try to break it in. Basically, in a very simple way now the integer is less than zero. So, not only is it, you know, if the integer is greater than the length, it can also be less than... So, let's sort of update our pre-condition to say, okay, if the string is... make sure that the integer is less than zero and less than the length. And so, finally, we actually get a, you know, from Scala check that, in fact, your substring function is not going to block assuming that pre-condition. So, we'll see all of these examples over, you know, in the next couple of slides, the most beautiful about this is we found two, two, you know, hj is effective without knowing anything about substring. And we never used java before. We saw that it could work two different ways just by running a very, very simple property, very simple fuzz. And also, what's really wonderful about this is now that we've written this, we're going to have more confidence in this function. We don't have to change the test. We can just crank up, crank the link before. We'll run this a thousand times. We'll run this, you know, 10,000 times, and you can use the same property, the same fuzz to really crank up the times that it runs. And maybe just, maybe just find a bug, or maybe when it runs in the world, we'll look three weeks and we'll finally generate an example of the fuzz up. But this is where things get a bit tricky. So that's all we'll do with the fuzz. We can just throw some arguments in the function and see where the fuzz up. But this is where, you know, I remember when I said now we're probably testing, I got to this point, things got really, really annoying because you end up in a situation where you generate the arguments you put. It's not very inception, but how do you actually serve that it's doing the right thing? And I always guarantee if you try to do this or people do this, you'll make sure the light's up streaming in. It'll just keep going if the light goes in and you do this and you call this other function. And you actually end up having two affirmations of the same function. And this is not really the point where this is always the point. And there are better ways where there are kind of tricks that you can use to kind of tease out the good properties. So I want to call that some sort of, this is another function, this is another thing I hear about, this is how I should write a property test for it. So the first one would be easy is the idea of a round trip. So let's say we have a function that goes from some dial type to a different dial type. So in this case, we're going to take a string and we're going to put it at the last, maybe put it at the UTFA. And then take it at the last, it should be where you go back to the string again. It's just sort of a way you go to and from the dial type to some action. So if we look at any code, this is the exact thing there's a way to go two bytes from the string and then there's a way to go from a dial type into a new string. So we're going to write a tripping test. It really couldn't be any easier. We're going to say generally around a string, any round string, called two bytes, go from a dial type again to the string and they should be the same. If it's too much of a different property, they should be the same. And that's actually not true. Of course, depending on the operating system, depending on the locale we set, in terms of UTF-16 characters are going to not be reversible. I just want to tweak the test slide just to make it more obvious, which is in fact what we could do is write a property to make sure we're testing what these characters say. Well, that's not only around the generator string itself, but it's also around the generator character set to see what happens. And then that may not actually tell us more about the sort of what's going on. So we can see that some characters here to the sum of strings are going to break. At least for how these functions are supposed to behave. But again, knowing nothing about this function, knowing nothing about the internal level, with the basic interface, where we now know something about this, which is we should be very careful about what we're coding to use if we're ever calling this function. So if we were calling this within a function, within a function, we would have something kind of like where depending on the characters that we're using, you'd be looking at something where a user might not see, you might take a better amount of code than something useful. This is just something we're talking about So let's now test you at a time which is a Java library for you know, data manipulation. So we're going to randomly generate a date time and we're going to try to test the date format so we're going to basically print this date to a string and then we're going to pass it back into the same date. This is a couple of years back now but it used to be a lot with this basically knowing, well, I can't handle passing with a sort of time zone art. So again, this is what I tested about the idea here is again, nothing about your time writing it at all but just writing a tripping test so they can work out their work in the system and the examples that we're using in the test weren't good enough. So I do a lot of JSON coding work I do a serialization of a HTTP request so often we'll have functions that go from user type into the system to the JSON representation and then come back again to the actual original data type and again, the tripping test is basically writing itself which is from JSON should be the same. Now, I just want to certainly tell you about your code so let's say you're already in some way in terms of the strings in some ways that don't line up but I think what's more important about this is when testing will be the actual JSON library so the JSON library that we're using didn't handle the microsecond precision of dates we didn't know that but when we start testing the property as you can certainly find it out. So this is all just about testing your code we've also been testing code that's actually doing like database stuff so let's actually call it Dialoghack so let's insert a user into Dialoghack and then let's go back up again which should be our entry test so let's insert a user, get it out and we think it's the same so apart from getting jobs in your code many years ago it was written correctly but once you get all those things you also start to learn things about Dialoghack so if you're going to test the process you're going to find your characters so when you try to pass the screen your characters basically get stripped down so again, not necessarily Dialoghack is just an invitation we learn that just from writing a trimming test so if I'm going to kind of hang on about it just a little bit I'll look at some of the Dialoghack testings and you're writing really two line of code kind of go to a common particular value you start to discover not just the box of your code but the box of some of the edge cases and the systems you're holding to start what happens when you want to test Dialoghack which is kind of what I was talking about when you were writing things but this is now quite intentional so let's say we have another way to go from the screen to the different path the different code maybe unoptimized and now we can assert the answer to these two functions will be the same so in some ways a contrived example will be if you ever have to write a story you should write an error based test so let's keep it on the user's law function we don't know how much it works but it's an invitation for the Java sorting and then let's just write a really simple error based test that's saying for any array of these the sorting of Team Sort should always be the same it's something really slow and horrible and it's something that we can look at the code and go you know that's correct and this is a bit of a long stretch but it was actually like in the Team Sort algorithm in Java for very, very large arrays there's probably if you can imagine a world where thankfully we are up high enough with this with this particular test you would never be able to write an example that would find that so again let's go back to the database style stuff we have a query that's going to look up users from the table the error based test in this case is a very, very simple operation on the list in memory so we're going to insert a whole bunch of users in the database we're going to pull out a list by user function to retrieve the user from the database and now the error based test is a very, very simple use of manipulation so we're basically modeling or kind of recreating what the database is doing just by this sort of hide so this is certainly and you may find you may find tasks in a manipulation software correctly or in other ways so this is for example really but the idea here is again if anything is not working well you can take this test and it will tell you so an example for my tasks is where we had our own data representation for performance reasons and past health reasons and to do so we actually had our own data representation like this so we wrote to and from our time from our data time to our data so this is just for testing and we had a tripping test for that so we said we'll just make sure these functions are broken so we can pull out and then in other cases so where things get interesting is that I mentioned that we've got to write a day plus function so taking a given a day and some other increasing or minus 5 and some other set so I only had to write one test I just had to write this one so I generate a random date I generate a random integer I'm going into my function and then it's also going to your data you just get your orders plus a function between world tests and then it works in even more cases and then you go back from your data back into your data format and then even if I didn't know anything about data type 1 I don't even know how to code but this particular test will tell me everything I need to know so I love ORB based testing in particular for finding these you know finding you know you basically come up with these cases and then just keep writing it over and over again until I know that my function passes I don't know the point here so the ORB based testing is particularly good in cases where if you've ever taken like an optimization path this is going to be the fastest or something and you take this thing here you can now have a free ORB based property if you've ever talked into a database or a file system you can also then write ORB based tests because you can recreate those things with a simple list to format manipulation so another kind of thing to look for when you're writing properties is if the function you're writing is meant to be either vocal or it is if you run it twice you should get the same result if you run it more than once and so I do apologize this is not a very good example I should give you the same, just go into team 1 so you shouldn't change the order of the same time you shouldn't drop it twice that seems to be the element I went hunting for bugs so I looked for the word under my own code you could help me take back with 4000 issues maybe they're not the last ones but the point that I like is this one, so it's not a serious bug but there's a tool called Rust Format so it takes the source code in Rust and you run it and it will format it so it's always in the system so it is, let's say we run it once and it produces exactly the code so ideally if you run it again it gives you the exact same layout but of course the bug was that it would start to repeat the entries into certain cases so what I like about this bug is that it's someone that writes a literature going well, you know, they actually have a test of this that would seem important but they came back and said we did test it we tested against all these hard coding samples so maybe there was a way to generate more examples and then I've also been interested in the style format as the exact same and other problems, so obviously formating code is obviously something that's hard to get rid of with other codes so it probably makes sense to mainly not find this kind of old story but I think we will so we still haven't found a way to describe this file of test, which is sort of a thing that's kind of a general approach when you kind of test certain kinds of functions but hopefully the example will help you out so let's come back to the substring example and I said this is really hard to test and it is if you just generate a string and you just generate an integer you're going to have a hard time testing as much without recreating the substring the trick here is not to just use those arguments the trick is to generate all the need to then sort of manipulate arguments so that you know exactly what the answer is going to be so what you can do is you can generate two strings you can generate two strings so that you know the second string you generate will be the answer to that support of the first string and then you can add that together to get the input so just to make sure that people know what I'm talking about if you generate an A and a B C the length of that A is one, substring it off and then you would get to a B C so the trick here is to kind of if you ever find yourself writing a property and you're just generating exact inputs to the function unless you're doing a parameter test or a workplace test you probably need to do something like this you need to generate more information than you need to know the answer to the function and you'll often need to know the answer if you find yourself recreating what's on the right-hand side of the equal sign you're probably not generating enough stuff so the last of the kind of the practice here is sort of the where you're really trying to cert properties out of the code so if you're running over a function maybe you'll order the arguments changes but there's a certain property you're going to come into play so this one's a bit harder to describe but what we can say is we think that if you call to an A is one string the length of that A string should be the same as the original string you would think but this is actually not true I would have said that of course that's true, but in fact there are certain strings where you do A the number of bars in the string will then get decreased because of the you need to go to the store or whatever it is so again, you know this is such a lot I would have to do with A so by writing it there and saying this is really more of an A this is really more of an A you might find a button in your code or a bit of a function as you call it so this leads on to the second aspect of property-based testing which is the idea of shrinking so the idea is that once you once the property testing is generating these values finally becomes a case action or an explosion section or the assertion is not true it does more work for you to work out what those actual values should be I'm not sure what I mean so if we go back to this two-log case example you can imagine if we might be in an error in the property testing library it would seem like an error or assertion or you might just stop and say well, here's an error so you can imagine it actually going to be so of course it's going to be the error in the bigger strings eventually one of those characters will be that character that doesn't quite have to go looking forward if you run it again you're going to get a whole different error and somewhere in there is only one character or maybe it's the problem so the view of shrinking will be that most property testing libraries will do this so if it's an error case it will then start to work out start to shrink these binary values into something useful so you originally found this error like after a much effort to generate this big string but it then starts to kind of break it in half and then when it finds the most part breaking it in half maybe it's a combination of parameters but it will kind of do the extra work for you to tell you this is a bad example it's going to be hard to explain how this works without trying to go code but it's going to give you a feeling for it which is that choose property testing find the smallest odd number so we're just going to sort of all introduce our algorithm which is not true but it's really the odd number and I eventually get that way back down to one again so to give you just a quick idea it's just going to be a quick visualization of how this might work so now we can start to see what's happening which is generating all sorts of crazy values eventually it gets a lot more than a test that effectively fails but it doesn't stop there and just keeps coming back to zero getting a video, getting it smaller and eventually not quite finding a search way back to one one is the smallest value of the test and it doesn't work so Trinket will become useful I'll mention Trinket and AI on the talk when we're talking about state-based testing so I want to talk about the other side of property-based testing now which is not the properties of cells like insertions but the other really important part of that is generating these values which is a really critical part of the property so if you start back down it's going to look at some sort of insertion and we're going to test it in the workspace at Bob so that's your place just with the property-based testing cell let's try fuzzing it let's fuzz it with a whole set of random strings and see what works and of course we're going to actually have a validation or a user name saying a lot of our user names can't be blank so we have a pre-medition a property saying okay ignore all those user strings that are for the blank and then they also will say well hang on you can also handle a path through the user names and their spaces and you have another pre-medition you have another pre-medition it's just good to go off the bar and maybe if the strings are going to have these names it's really good to see them so this eventually will actually do an error from your property testing when I bring it on well I gave up because what's happening here is I'm generating so many random values and nothing of it is actually being through the test and you're wasting your time it's just going to see you because it's actually really handy features so you don't have tests that are literally sitting there forever but it's a whole bunch of depending on the property testing I'll be using there's not the annotations in Java or Python, there's a particular function as a scalar but it doesn't really matter there's ways to kind of build up these variables so in this particular case I'm going to combine the use of the general and just one function which generates at least one value and then you choose what I'm going to say even these two values or integers generate any value between A and Z like low case string so we're saying you look basically at the low case string between A and Z there's one or one length but let's not stop there because this might actually be true for all the user names in the system which is exactly what we want so let's extract an addressable function from this I'm not just going to have this in one particular property we're going to extract out a general user function that we can then use across to the code base then as we test the user the user, the migrate user every possible function they're going to use within every user's one area so we're kind of reaping the benefits of having one function that tells you what this is going to mean this is the edge case that we can think of and over time as users grow extra things like let's say we have an age so let's say we have user settings you can then just change your user generate function and it will be in one place and then all your other properties are going to be used in every benefit of that so I want to tell a funny little story about another sort of a story from my history where we had a previous company I was working at we had a bar where for a particular part of the database we did a user name eventually after a couple of days of tracking we worked out the problem was that one of the users had a last name of a model so that's a strange name thank you so we do what we do we do what we expect we can fix that we can make sure we migrate user analysis it never happens to you but I want to come up with an approach to this from a property testing or at least from a generated perspective so of course in a property testing world we already have a test we already have a test that's generating around the users but unfortunately we have a family in the case of any other level but rather than changing the test and writing another test let's leave the test on an actually generated generator let's do something a little bit different and say every so often generating another string one time out of 20 so even though we're already using genders and delete user update user migrate user all the tests that we use earlier are all now using these genders so we can now make sure that we don't just have this one bug into place over here which you have guaranteed that the bug doesn't come into the system any before the test is in there okay so that's the generator it's worth it's worth it's best to generate a score pooling up the kind of the worst edge cases and that's what I'll show you the tests so I'm going to try to talk and finish up about the thing I want to say base testing which is really, really exciting when it's coming in part of the property based testing even though I was going around in some sense along the property based testing so I want to go back to this example we had earlier this is actually a test from before so we're testing the database where we're inserting the user into the database and then getting that out again but of course this is really on the what you can imagine doing with testing like our user table for example so you can imagine maybe we should actually test the fact that you generate twosers when one of them comes back so the ESP is actually correct maybe we might as well delete these options so now we're going to test that in some way so tick one in there and make sure that you don't this is a way some of how many examples how many of these property examples are enough so this pulls very familiar this kind of feels like the same problem which is rather than we're trying to kind of capture examples in the data but now we're also talking about capturing examples of the actual series of steps that we want to call so this is the idea behind the same as testing let's get the computer to generate the same steps as we want so the code here is a bit more funnier than my other stuff but the general idea is this is a model the state that we're trying to test so in the case of a user table there needs to be users this is just something as simple as that and then let's model the different operations that you're trying to run so we're going to model the insert we're doing and then depending on your state based testing library you're going to have a whole bunch of functions that you can benefit from there like actually doing the insert so actually running the database insert it also has then a separate function which is a way of doing the need memory if you are state based or version of that function you run the version do they run up? it's a system in a consistent state so we do that in the insert we do that for delete or any of the operations that we're running we're testing it earlier you kind of wrap this command operation style you know a lot of other things and then we do this how we're going to use our sand tools probably testing in the same way so that's just what I'm going to say using the gen1 of function operation and then just generating a whole bunch of them running the sequence and then the test may be made in a way like this I generated these three steps so I generated insert and insert and then again and it didn't value the way that I expected it apologies again this is not a real example this is not particularly useful but you can imagine you know it's actually finding real things so in fact I want to show you a real use case of where this actually happened in the wild so level B is a key value store from Google and so I don't think it was someone from Google it was someone who went up and wrote a property test for this I've got to say a test for and then we came out eventually with this bug so it found a series of 17 steps to reuse everything else that you buy in level B so this is for people at this point this is not a 17 grand this is a 17 step after shrinking so you can imagine a whole series of steps and usually we've done crappy reviews as well the state-based property test found these 17 very simple steps to actually say this is what happened so then the guy could then raise an issue and it's Google saying you've got a bug in level B but my favorite part is when I was looking at it I looked at it and it said I will fix it for you and then I came back with a new version and then that same property that I was changing in lines of code then found another bug it took 33 steps to reproduce so you didn't have to do extra work but the same property and then you got another three steps eventually Google fixed that one and then it probably ran a hundred times in a time so I think what's really exciting about state-based stuff apart from this is obviously pretty cool but where do I see my key users is that what is it we try to test to compare it for the system so writing in power code is really hard getting it correct is almost impossible and the testing is I don't even know how to test in power code it requires a lot of work and you never really know how to get it right if you use a state-based testing approach some of the state-based testing libraries will then just use this kind of function so run that state-based test I showed you earlier we'll use it again and just crack up the number of threads to see what happens and then maybe it comes back and says we'll actually so what they often do is they'll generate a series of sequential steps and then run a whole bunch of parallel to make sure that the post-production is all kind of passed so this is kind of I don't think this is the one that we've been really excited about this stuff so this came out in 2009 so it's been around a very long time and unfortunately the parallel testing stuff is not in most of the public testing libraries so I've been hoping that with the great interest in probably testing the rows we can really start to immerse in making this work so it's in the Scala 1, it's in the early version now it's in the Hasselborn testing in different public testing libraries so just to kind of finish up just want to kind of walk you through what we've talked about so it started with the question why are we testing what's the point of testing and I kind of say that hopefully we just roll on the same page we're trying to arrive a reliable correct code and that's the aim but again the progression of the process does exemplify testing writing down high-quality examples and we make sure that we think about the ways it takes us up front when we're really doing the testing we can and it's an apple-based testing really working for us why not get the community to basically generate these examples and see what we can do with that paper we saw the idea of buzzing so without knowing anything about the session of the test without knowing what it's correct on let's just throw some examples of the codes to see where they're priced and maybe that's what you can find that just needs to be doing this like, you know, null characters in your part of the database in the maze of the whole point of the session in one of the libraries we're using we talked about that when you actually did a prior research and needed to get a tricky particularly when you passed exactly what was to be a function and how you tested that and just really reduced some of the function itself we saw the power of the trimming test, so taking a function that just may be and then by the writing or finding another function that goes back from your writing and then making sure we deal with what values are the same so we saw that null character coming out and we also saw the we also saw the error-based testing sort of approach where if you just find another way to get that same answer then you've got to do a better property to that so again, this is really good when you actually try to do an optimization in code if you write something that needs to be slightly faster if you need to take something going by that other function that you're testing as well as you do because it will kind of make sure the test does what it does I've called Explained Rebuild an example where we sort of generate more inputs as much as we needed to then make sure that we knew what the answer was going to be we saw that probably testing might be shrinking so that even after a double level of time that A is going to defend itself they're actually going to try to work out what the real, really kind of small possible cases so that you can think about what this is like, what for, what for, what for this is something that we do to the person who writes this line rather than having to go on hunt for the actual reason why we break we saw that investing in generators was really important so having generated and making sure that the generator was the most possible user was really important and then you start to find impressions of things like nulls, you can then increase the coverage of your generators so that the color will be a test I've got very silent about play-based testing where we're all dealing with very complicated state-based systems or on your dial bases so there's all sorts of distributed systems that we deal with and they all come down the path of complicated tests so you don't need to necessarily be very important because there's a really good way to use it to generate the different systems of tests so that's what I mean by a poor line of students that say well, since 1999 Kyrgychek has really taken a long line and some ways it hasn't changed a lot like there's definitely some twists on your ideas but certainly the poor idea of generating examples in a variety of sessions hasn't really changed since 1999 and so if you're at work tomorrow and you're writing some jar work there's no reason why you can't just drop in and write a little closet for that to be a test and see what it's like and not want you to get a longer code if you're running a JavaScript question and you want to make sure that it can again not get under defined or all sorts of different cases it's just dropping in one property not having to have a rewrite or change the language just to get the benefits of what we're doing today so that's me Python has a few good law documents like a proper hypothesis which I kind of figured out recently is definitely one of the better property test libraries so I guess hopefully you will kind of get a sense that I really think that by doing a property testing you can start to kind of see in those new cases not your code, your code obviously works but the library that you're using the system that you're calling because you haven't thought about your QA engineers the one you'll be telling you about but instead you can see those errors and you write that code straight away I guess I want to end on the thing that you need more liable or correct by finding what you guys are talking about you can do it without having your liable or code you can write these really simple properties and then get a testing where so on the other side I just want to lead you up here so if you're interested particularly around the patent stuff if you're interested in what you're trying to do if you're trying to write a property and you get stuck this is the best reference guide you get shopped for a kind of profit or what versus just really in terms of the best product again if you're interested in property based testing you have to watch what I'm talking about because it's really really passionate about property based testing and all kind of the same idea you're just talking about safe based testing but in particular what you said I've always said when it goes into more detail it's really mind blowing just how simple these tests are and just how complicated the box will be and lastly all the kind of slides that I've talked about and you can actually check out around the properties that way so that's all I've got, thanks for listening we've got time for questions who has questions yes I will run around look at it from the test pyramid right, UI test pyramid UI unit that you can have where will this be used most of the time where would it come on for you that's a good question I'll do it too UI testing UI testing UI testing UI testing UI testing UI testing UI testing UI testing UI testing UI testing I think from what I've seen, safe based testing is a really good approach so you can model your user types in this particular area and have your safe based test then randomly generate the things so user events you can really test your UI that way that you wouldn't have to do manually that's the best example I've seen but I'm sure you'll also test different parts of your code to the smaller properties as well don't know how to sum it up anyone else can you elaborate a bit more on how to modify the property, so I'll be surprised to find a upstream property how we do this little test about here this one's a bit tricky I'm looking more at samples so the idea is whenever you find yourself doing any kind of work on the right-hand side how to get easier to write how to execute this kind of test if you are so the other example I could be using is I'm going to say you're returning users to the database and you want to find the users by postcode for example another example would be generate two of these users make sure that they're generating around the postcode and then sign the postcode to the first user insert both of the users into the database so they know the answer and the answer and then you model it which is four users out of that exact postcode and those inputs are something you generated and you know the idea is you want to work back from a crazy situation where I know what the answer is going to be you work back from that and actually there's a little bit of a test going to be more of a statement but I'm going to be able to do the logic again if that makes sense it's a bit tricky I guess it's very hard to wait in here for what you might do but again the important part is you find yourself writing the code here on the right-hand side like the actual logic of the function then it makes a step back and I know more about the system and I really hope and I really recommend that you just have to particularly go and look at that sharp block to work with any of those really good examples about you know what you're doing with the situation what you're doing with the situation so I'm going to answer that it's just something you're going to have to get used to there's something over here over this side there actually the actually the I was going to use this I'm going to use this because in our language you can pass negative numbers and it feels like this kind of constructing the cases is more for the case where you know you have control over both sides but the minus one or more of those edge cases they're partly produced in this kind of um I think they can be a bit more properties so this is where you definitely want more than one property so you might write the exact property for positive numbers and you might do one for exactly positive just a bit of numbers where you might take the length of the string you're slicing and then it makes it as maybe maybe you can share it three strings and make sure you know the length of the last string so I think you could do this way but just writing one is not necessarily what you need to do like some of you might have three or four these are couple of cases you know of that so I have to say it will still be under the term of things that you know yes, yes, yes so there are some cases where if you can't write the over based test if you can't do any of the little tests some of them yes you definitely have to come up with some more edge cases but now you're not coming up with three examples you're coming up with the three things that will cover your problem completely so it's definitely like a functional test that you can write a problem that you can use because you really describe exactly what the function does rather than just ABC one or ABC minus one you can now describe the behavior of what the minus one is actually going to do you can definitely try some more if you can Charles, thank you very much you got me very excited for this state based testing it's given me a way of testing by United State Automata so that's really exciting 39 steps or whatever so thank you the second thing is I'm not very trouble I now know how to test by United State Machines I'm trying to get you to give me a free literature search have you read anything in the literature about testing machine learning all this that's a good question that's a good one I couldn't help you because that doesn't need to do that but that's set of answers no, I think when we do machine learning stuff we're too we definitely struggle with that solid testing there's still plenty of room for different patterns and different approaches for those things that's a good question that's great any other questions maybe one more that's on me so I was just a little curious I was just wondering could you elaborate a bit more on your threshold for say deciding that something is satisfactory satisfactory tested at what point do you say okay I can understand this enough this enough is relatively not free is there a set of general criteria that you follow or is it more flexible is it more tailored to specifically doing that testing that's a really good question it's a way to say what the problem is so the problem for leading on time whether you're doing a hardware testing or a virtual testing you do the remote one and maybe you just do a test where you're going to have a new relation where you can introduce a state based on so I want to talk about that and we have to use a state based testing so that's where you go whether you're serving users so I'm kind of excited as well to try a state based testing see how many of those properties don't have to write anymore so it's a way to stage you and so forth I certainly feel a lot better about properties that I do with these tools I know you're going to stop but now there are properties particularly again that are all you can kind of do that if anyone else has questions please join me in thank you