 And so I'm gonna do just a little bit of stretching no one here is obligated to do that with me but if you want to take a part in this we can all maybe like stand up, get our hands up, feel us on the side, get those arms out in front, oh yeah here we go, feel the burn. I mean Nasty's just coming, did you? Alright awesome, thank you for indulging me in that. Don't do all of this, it feels so much better. Cool, so today we're going to be talking about property based testing and generative testing. And this is actually not the talk that I submitted, nor was it the talk that was accepted. But after Day's keynote yesterday I was really inspired by the idea of thinking differently. We are in an incredibly early and fertile time in a lizard. And I think it's really important to question the status quo, question the things that we've been doing. And for me property testing is that thing, that's the thing that's hit me the most in recent time. As I would think differently about a problem. I was very happy that after some drinks I went up to John and I was like please can I give this other talk? And he said yes, so here we are. Side note, I also have these awesome elixir sugars with elixir in a bottle which is just really cool. So if you want one of those you can find me at Elixir. So what are we going to talk about today? We're going to discuss testing kind of as it is today. We're going to discuss property tests. We'll look at some basic examples of how to use those and then we'll look at trying to model a real world application using properties. So let's kick it off. So testing. Who here believes testing is a really good idea? Who here tests? I saw someone's hands start to go down. I get it. Testing is interesting, it's also very challenging and it takes time. I think if you were to sort of map my career happiness on an XY graph like this it might look something like this. Down here in the early days I started out writing C for embedded systems and I did not even know that testing was a thing. It took me getting into Java and learning JNU to even discover that something like software testing was a real thing. And then I had the great fortune of joining sort of the Ruby community which is not a very popular thing to say anymore at elixir conferences. I actually got a lot out of being in the Ruby community right on because we had these great testing tools built right into the language. That was really important for me. And so my happiness started to go up when I discovered TDD. I discovered there was a way to write software tests. It was really cool. And that took me a long ways into my career until at some point I began to plateau. I sort of refer to this as like the quiet contemplation phase where I began to sort of question what it was that I was actually doing. And I wasn't getting the benefits from testing anymore that I used to get. And it led to this sort of period which I can now call the trough of disillusionment. Where I sort of fell into the camp with like I don't really know why I'm doing this. I don't want to do this anymore. It doesn't benefit me in the same way. And of course now I'm all the way off to the side because I have discovered and enjoyed the property test. As you will soon see. So it's really hard for me to talk about testing in my life without talking about TDD. The two things are really intertwined together. And in case you don't know what TDD is, I'm going to stop using this acronym and just find what it is and say that TDD is test driven developer. And the process is pretty straightforward in theory. You write a failing test. You write just enough code to make that test pass. And then you refact. And you continue in this cycle over and over again. You add new features by writing a failing test for that feature. Describe how you want that feature to... What you want that feature to do, the behavior of that feature. And then you write it up just enough code to make that test pass in your factory. And also in case anyone is feeling pedantic out there, which we're all programmers. So unfortunately those two things go hand in hand sometimes. I'm using BED here as well as an example. I use the terms BED and TDD interchangeably. It's not really important. Ideally if you do this, you will gain some base benefits. For instance, you'll get validation. You'll validate that your algorithm works in the way that you believe it should. You will get protection from aggression. You won't have as many bugs anymore. You won't ship broken code. And thirdly, and probably the most important, you will gain design. By allowing your laziness to sort of take over while you write a test, you'll make testing easy. And through that, all these good things will sort of shake up. You'll do things like dependency injection. You'll separate your concerns correctly. And a lot of people talk about this like test provider mental gargles. They provide this sort of easy row that you can take and keeps all the bad stuff on your periphery. Very important to note though that tests directly couple to your implementation every single one of them. Now that's not necessarily a bad thing. We often want that in many ways actually. We want to create a contract with our external APIs. The downside of this though is that as soon as we change this API, even if we're only testing it at the boundary, and even if we only change things internally, we'll still cause tests to fail. Because every one of these things is a compliment. There just isn't a way to write an example-based test where this doesn't happen at some point. And so that kind of leads to a meme, which is like write as few tests as possible. And you might have heard some people talk about this. So I want to try to illustrate some of these things. And I want to do that using what I believe to be an incredibly contrived and strong-man argument that will help me make my point. And I'm telling you all this upfront because I know it's a contrived and strong-man argument, but I hope that what we can do is see past that a little bit and see into the truth of what we're trying to do here. And so that all being said, let's TDD the edition. Everybody ready? I know, it's 2.30 on Friday. So how do we do this? We start with a test, obviously. So we'll test that we can add two numbers together. We'll say that we assert that adding 1 and 1 is 2, adding 0 and 2 is 2. That's pretty good. And we'll go ahead and stuff up that function call. Now if we run this, it's weak. We've got a failing test. So what are we going to do? How are we going to solve this? Well, again, let's remember, we should always write as little code as possible to make the test pass. So let's just return 2. Now we don't even need these pesky arguments. Don't just mark those as unused and we'll move on for our lives. And look, we have a green dot. We all love green dots. We shove this up onto Heroku or wherever we open up our new edition as a service. It's awesome. And everything's great. But wait. 3 in the morning, the pager comes in. If you get pinged, hey, there's an issue with that add function. It doesn't seem to work with add 3 and 4. Well, we wrote tests. How do we have a problem here? Well, what do you do when you have a problem in production? You write a test for it. Obviously. Because you don't want to have that bug anymore. So we do write a test. And they will assert that adding 3 and 4 is 7. And now how are we going to solve this? Luckily, in Elixir, we have all the power in OTP. We have nets. And furthermore, we have pattern matching. So we are very equipped to go solve this problem. So we'll just quickly inline this because I just want to save space on the slide. And we'll go ahead and add a new clause. If we see the 3, we'll return 7. And now we have green dots again. We'd rest easy. But wait. I found an edge case with add. It looks like it doesn't work if you try to add negative values. This is really wrecking my weekend. So again, what do we do? We add a test. We'll go ahead and just pick something arbitrary. Like add negative 1 and 4 is equal to 3. Run that and get something that fails. That's good. And now we're ready to go ahead and solve this. And again, luckily, Elixir, it's one of the best languages I've ever used. We had this great thing called guard clauses. And we can use those guard clauses to go ahead and protect ourselves from these pesky negative values and return the right value. And now we run that. We'll be green dots again. Bump the green merge button. Deploy it to production. That looks great. I think we finally found all the bugs. So, I told you this was a contrived strawman argument to make my point. But I do think it's important to analyze what's going on here. Because at the end of the day, this is not actually that different from how we write a book. And so in order to kind of tease this apart, I think it's important to step back and say, what have we gained from this? Well, how have we gained validation? How have we gained professional progression with design? Let's start with the top. Have we gained validation? Absolutely not. We didn't even use the plus operator. There's no way that we've actually validated this algorithm works. And so that's just right up the window. How have we gained protection from regression? Well, let me give this like a half a slash. Because we have protected ourselves from regressions that we've thought of. Or regressions that we've seen in production. But we haven't protected ourselves from things that we can't see. From the sort of unknown unknowns problem. And how we gave design. I actually think this is up for debate. I personally still do a lot of TDV. The difference is now I throw the tests that I would have done. I do think I gained some benefit from doing that. Others have different opinions. For every research paper that says that TDV is valuable, there's other papers that say that TDV is the only way to find bugs or whatever it is going to be. My opinions were probably just really bad at writing papers about software practices. But, you know, your mileage may vary. But I do think it's important going back to this metaphor, sort of these tests providing Gargoyle's idea. I think that's really cool. It's just that I don't know how efficient it is to drive down the highway smashing into the Gargoyle all the time. Which is kind of unfortunately what tends to happen. One thing to remember here is that there are bugs in your car. All of you. There are bugs out there in your car and they're waiting. They're just waiting for three in the morning to pop up. And unfortunately, there's no, it is an untenable problem to sit there and try to write enough tests to find out. You could be there until the heat death of the universe. And if you want to spend your life doing that, that's fine. It's not what I want to do necessarily. And beyond that, what about bugs that you can't easily write tests for like a race addition? Like especially in Alexa where we're going to have multiple processes all sort of working at the same time. We haven't fully eliminated race conditions. How are we going to write unit tests for that? How do you do that? So these are sort of the problems that I see with testing in sort of example-based testing in TDD as I've been doing it at least for the past several years. And if that's the problem, obviously I'm hoping to present a solution and if you're out here at this talk, that solution is property tests or one of the solutions. So for those of you who aren't familiar with property tests they're sort of, most everybody cites this paper for the birth of this idea by John Hughes who's going to class on talking about QuickCheck. And this was a tool written for Haskell in order to apply property testing, apply generative testing to algorithms and data structures. And since then it's been ported to a ton of other languages. Basically every language that matters at this point. In fact John Hughes went on to write one for Erlang. It's an hour-licensed product and it's probably the best QuickCheck implementation now. There's even one Crocker row one for JavaScript I haven't used it, but if Crocker rowed it I assume it is called the good parts. So, but the idea is that we're going to, there's kind of a thing that they fixated on in this is that typically with example-based tests we have something like this. We have an expected value and we have an actual value and we compare those two things. And you could basically vote on every test that you write to something like this. And then the problem with this though is that it's incredibly overly specific. If you know the output that you want to have then you can't possibly interact with all the other possible input data that you could have. So for instance if we had like a distribution like this you would pick a couple points on it. This doesn't actually completely satisfy all of the different pieces of data that you could input into your algorithm. And so instead of this what we really want to do is we're going to draw a box around all of this and say well it kind of fits into here. So what we'll do is we'll start by just generating data. In this case we're going to generate integers this could be really whatever you want. We're going to pass that into some function, some property and then we're going to look at the output. We're going to make some decisions based on the output. And this thing in the middle you'll also hear me refer to this as an invariant. So it's important to define what an invariant is. An invariant is a function quantity or property that remains unchanged when a specified transformation is applied. In this case you can kind of think about an invariant or a property as a law. A law that your algorithm has to conform to. So with all of that sort of head knowledge let's look at how we apply this. The first thing we want to do if we want to go back to our original example is we have to ask ourselves this question what is true about addition? What is true about addition? This is really hard. And this is actually where property testing becomes incredibly difficult because you have to stop and reason very deeply about the algorithm that you're testing. And so if you're like me and it's addition you go look it up in Wikipedia for this talk. Or you can, you know, if it's data structure you might be able to look at the rules of that data structure but we need to figure out what those properties are. This can be very challenging. But if you think about it for a while you might remember that there's this interesting property called addition which is if you add zero to a value you get the same value back out. This is called the identity property. So let's write a test for all. So we'll start the same way we would normally while we're testing our stubbed out function column here and then we're going to use QuickServe, thanks Dave for our tests. And we're going to start, do that by saying P test. And then we need a value. So we're going to generate an x value. And that x value would be an x value. And then finally we can actually go ahead and write our assertion. We'll say if you add x and zero you should get x back out. And if we run this we should get a failure because we haven't implemented that yet. And we can now go back and implement it. For the sake of learning we'll go ahead and take our same tactics we did when we TBD this and we'll go and write the least amount of code to get this to pass. And so in this case we're just going to return x. Alright so that will cause that to pass. So now we need to think about more properties. We need to think about other things that help us to find addition. And again if you think about it for a while you may remember there's a property called commutivity which means that I can add x plus y. And that's the same thing as saying y plus x. So let's write a property for that as well. In this case we'll generate two different values x and y. And we'll go ahead and add them in different orders and we should get the same value. And now we should get a failure. And you can see what's happening here. We're actually seeing, it's actually showing you the values x is 0 and y is 1 and that will cause, that will exercise a failure for this algorithm. Which makes sense if you look at our documentation. So we need to figure out a piece of code that will satisfy this test. And the thing is, I kind of remember that multiplication is commutative so let's try multiplying these numbers. So we'll multiply x and y and we'll see what happens. It looks like we broke the first test, the identity test, because if you multiply it by 0 you're going to get 0 back out. Again elixir is very cool and we can pattern match on that case and we'll have a guard there to look for 0 and then we'll just return x if that's the case. And then we'll run that and we'll get that to pass. It's important to note at this point that neither one of those properties even taken together is enough to define what addition is. So we need more. And there's one other property it's called associativity. It allows us to, it means that we can add like say 1 plus x and then add y and that's the same thing as padding. So let's write a property for that as well. Again, we'll start off and we'll generate some values x, y, and z and then we'll add them in different orders and now we should see a failure. Which we do. Awesome. And you can see the values that it's happening that it's trying and that will cause us to actually get a failure. And now finally taking together all three of these properties we can do nothing but go back to our original implementation no more shenanigans and actually implement it using plus. And that's because at this point we have defined the essence of what it means to add such those three properties together. Any one of them alone isn't enough and this is sort of a really powerful idea that you can describe sort of the base principle of something with these different properties. It's very, very interesting. So now that we've gone through some of the basics let's look at how we take this knowledge and apply it to a real application. So I wrote a little Phoenix app it's a luncheon voter so you can go and say this is what I want to go get for lunch today it's all using channels and whatnot and I want to write property tests for this. So we start that by modeling the application. Again going back to this diagram we need to generate some values we need to pass them to a property and then we need to make a decision and the first question we have to ask ourselves is what is the input domain here? Like what are the possible values that we can generate? We can think about we can derive that list of we can derive that input domain from our users. So if you think about a user the way that they show up to your site they'll arrive and then they can choose to vote and they can vote for the middle one they can vote for the one on the left and then they can vote for the one on the right. So what we can do is in terms of a finite state machine this might be an example finite state machine of a user they start the logged out state they transition to the logged in state they go back to the logged out state and then once they're logged in they can also vote in which case they'll stay in the logged in state. So what we can do is starting from a given state we can generate transitions so for instance and then we'll be in the logged in state and then at that point we can say we can make an assertion for instance something like did we actually get logged in? Once they're there we might generate another command like a vote command and at that point we can also make another assertion something like did the vote increase and what we'll do is we'll start generating just lists and lists of commands so start with one then another and another then fails and then what will happen using a technique called shrinking the library will actually pick out or help us pick out the list of commands that will actually exercise a failure and then return those to us so what we'll be left with is a set of commands that you can run that will actually cause a failure to happen on your site. Now list of commands might look something like this we might have two different users and then for instance Chris might vote for one and one and three and Chris might vote for two and this might cause a failure and at the end of it what we'll get back out is a list like this which is much more useful. So how do we write this in code? Well Chris uses a pollution to generate streams of data and so we can kind of take advantage of that so we'll just define a function called generate commands so it'll take a list of names and then we'll return we'll return a list we'll return a list of things to do that list is going to be a list of votes and we're going to generate 20 of them because it's about all it's going to take to actually exercise a failure in this application. You can see we're actually calling into a second function there gen vote and passing in the list of names and we can define that as well so gen vote is going to be a tuple and it's going to look like a valuable vote and it's going to choose from one of the names and we're going to also choose one of our lines one two or three and these are the different things you can vote for and then once we've done that we'll return to the property so let's start simple and say something like well a user's vote should always increase so let's write a property for that we will start off with our standard test we'll generate a list of commands in this case we'll pass in just a single and we'll go ahead and do a little bit of setting up the world since the votes are a global thing we'll just need to reset that before the test starts and then finally we'll tell it to run commands at the end we'll get a result back out of it and we'll assert that the result is a true or false the implementation for run commands is incredibly straightforward it takes the list of commands and it increases over time starting from some initial state so for instance in this case we'll start from zero and at true and then we'll run each command and then the run command down here is not very interesting actually it just calls into the module based on the state of itself so as I said we can model everything as a state machine so we need to define that state machine so we'll do that and we'll start by having a vote action so this is something that a user could do and in this case we're just posting to the API but this could be anything you wanted it could be you could actually use a browser and test with the actual browser and test the API and kind of test whatever order you want this is just implementation detail it's easier for it all but you see what's going to happen here we're going to do a close we'll make some new votes and then we'll turn it okay once that's done we need a way to transition to the next state so we'll have a vote next that we'll define that will allow us to sort of say this is the next state we should be in and this is the value of that state after we're done transitioning this is what it should be and in this case all we're going to do is go and update the count for our name and the thing that we go in for and then finally we have a post a thing and a way to sort of make an assertion and this happens after we've transitioned in this case we can actually get the real result the actual result that came back and compare it to our expected result and it's nice because at this point we actually kind of get to get back to our expected versus actual which tends to be easier to reason about and so now we're up to go ahead and run this test and if we run that you'll see it passes that's kind of boring so we should come up with some other properties for this and I think another good one might be that users shouldn't affect each other's votes like you don't want to vote when you can't see someone else's vote out so we'll like that test as well it's going to look very similar to the first in this case we're going to generate two different users Chris and Jane we'll reset our vote counter again and then we'll go ahead and run our commands and then we'll start the result so it looks very similar the big difference though is that instead of running just commands this time we're going to instead run parallel commands and what that's going to do is it's going to take the list of commands it's going to split them and then it's going to actually run both lists basically and this should be interesting if we run that you can see we'll actually get a failure so I added some debugging in here this is the first set of commands that it actually tried to run and then this is what it shrank down to and this is really interesting because it says Jane votes for one and Chris votes for one we don't get our expected results and I just think this is the coolest thing ever because I got this working at one point and I just giggled for like 15 minutes because it was so exciting because this means like we found a race condition but we found a bug without having to do without having to look for that bug but that's awesome, that's so exciting and of course if we go and look at the code you might be able to see from this example where the actual bug is spoilers that might have written this ahead of time so that it would definitely have a race condition in it but what's happening here is that we're getting the votes out we're updating and we're putting the votes back in and that means it looks something like this like two different clients are talking to the vote counter they both ask it for the count of votes they both see a three then they both update it and then they both put that value back in plus one is four so we've lost somebody's vote in that and this is easy to go through at this point, we can just go and linearize both of those rights so with that I think it's important again to step back and look at what we've gained from this so this is our list, this is what we wanted to get out of TDD well do we get validation? well yes because we've actually been able to describe the entire input domain now each time you run the test it's not going to run it with every possible value that would be untenable but over time you will run it with more and more random velocity and you will begin to find edge cases and in fact if you run this in something like a continuous integration environment it will continue to find bugs free we've gotten, because of that we also gain protection from regression we can protect ourselves from the regression and we can protect ourselves from the regression that we haven't even discovered yet that we haven't even thought about yet and finally, do we get design? I would say that we have because by stopping and thinking about your users and by thinking about the ways that they interact in your site, you are going through the same process that you would have with TDD you're thinking about the problem which is the whole point so in this talk we have seen how to think in properties we've been able to see how to model applications very easy ways to generate data, how to generate commands and then how to model users as finite state machines if you want to learn more about this stuff there's a ton of resources on it I totally recommend the original paper it's a little dense but it's worth reading most of the state machine stuff all comes out of a paper called finding race conditions in furline with quick check and pulse where they find a race condition and deaths I believe and a lot of that is taken from there other than the deterministic schedule but if you want to know more about that talk to James in the back and there's a great, a bunch of great talks out there as well so go check all those things out and finally I can only leave you with a question which is if you could write less code and find more bugs and if those tests that you write continue to find more bugs and you can continue to iterate quickly and provide value for your customers would you do that? Thank you