 Welcome everyone to the session on property-based testing with Magda. We're so glad you could join us today, Magda. Thank you so much for being here and yeah, without further ado, over to you, Magda, all the best. Hello, everyone. My name is Magda and I'm very happy to be able to be part of the functional conference here and to be able to talk to you about one of my favorite topics about property-based testing. But to be honest, when I first met property-based testing, I wasn't too impressed. It was during my first months in Scala about four years ago, when everything was still new and strange and difficult to me. And at that time I noticed that each Scala project that I joined has something called property-based testing among the tests in it. And at first I thought, oh, that's just another Scala thing. So I will get to do the same thing that I already know in Java, but I will have to learn to do them the Scala way. But when I really got to understand what they're about, I thought, wow, now this is something different. This is something amazing and revolutionary. And the reason why I thought that is that because in property-based testing is the library who does the heavy lifting for you. So your role is to specify some rules about your code and the role of the library is to find the bugs for you, is to find the holes in your understanding of your business requirements. So that's why I thought that it's something really awesome. And my next thought was, so why has no one told me about this before? It's something that you can easily use outside of Scala, even outside of the world of functional languages. So why am I learning about it just now? So today, I would like to do just that. I would like to show you that property-based thing exists, that it's awesome. I'd like to show it to you in the code. And if you haven't yet heard about it, then I would like to encourage you to try it. So the plan for the next 45 minutes is I will show you how property-based testing is different from any other kind of testing. I'd like to show you how it looks like in the code and how you generate the data. And then in the second part of the presentation, even though right now I may sound like a salesperson saying that this is just a great thing, but in the second part of the presentation, I will tell you what is some of the fine print? What are some of the challenges that go with property-based testing? So let's start. And the best way to start would be with an example. Let's imagine a sample domain that we would be working on and then we would like to test. Let's say we have a calculate discount method. This is the method that we would like to test. It takes an instance of customer and a local date of today. And the customer class you have below, it has the customer name, the date when the customer has joined, and their date of birth. And the algorithm, the business logic of the calculate discount method is that if the client, if the customer is with us for less than a year, then they get no discount. If they are with us between one and two years, they get 10% discount. And if they are a client for more than two years, they get 20% discount. And to keep things a little bit more complicated, let's add a special birthday discount so that on the day when the customer has their birthday, they have a special 30% discount just on that day. We will be working on big decimals, so 30% will be 0.3. So now, how would we go about testing that in a traditional style of testing? As opposed to property-based testing, we can call traditional test example-based tests. So if I were to test such a method in traditional style example-based tests, I would write a couple of test cases. So for example, the first test case should calculate 10% discount. I would prepare an instance of the customer class who has joined one year ago. They have some date of birth, but it's not important because it's not their birthday here. And then I evoke our business method, and I have an assertion at the end to verify that the discount is 0.1, so our 10%. And I would have a similar test case for 20%. So again, I'm creating a customer who has joined two years ago, doesn't have a birthday. I call the business method, and then I call the assertion to verify that the discount is 20%. So this is cool. And now let's test the birthday discount. So again, I'm creating an instance of customer who has joined just now, but it is their birthday today. They are 32 years old today. And I verify that the discount for them is 30%. So now with those three test cases, I have the main logic of the method cover. But of course, I know that this is not enough because I have to also test the edge cases. So what test cases could I think of here? Let's say that for a customer who has just joined, we want to verify that they get no discount. So I prepare an instance of the customer who has joined just now, and I verify that the discount for them is 0%. And what other edge case could we think of? Well, let's make sure that the discount doesn't grow indefinitely, so that for example, if a customer is 10 years with us, they don't suddenly get a 100% discount, and then we would get no money. So let's create an instance of the customer who has joined three years ago, but doesn't have a birthday today. We calculate the discount for them and verify that it's still only 20%. So right now we have three test cases for the normal happy buff and two edge cases. And with such set of test cases, I would say, okay, that's enough. I did a pretty good job at testing. I've got the edge cases covered. Everything is correct. Everything is green. So let's move on with life. Let's start implementing the next feature. And now how would that be different in the world of properties with property-based testing? So if this is property-based testing, then what is actually this property? This is some kind of a rule about your code. That you define that you want to say that it has to be true no matter what the input data you have for your method. So it has to be always true. And your role is to think of such properties, such rules, and the role of the library is to find where this rule is maybe not true here because you have a bug. So what kind of properties we could think of here, for our example domain? One property that we could come up with would be that no matter what the customer and the date of today we are getting, the end result, the end discount is never less than zero. So it has to be a positive number. And on the other hand, another property could be that no matter what the input data, the discount is never bigger than 30%. It has to be always true. So as you see, these properties don't have to fully define and fully test your algorithm, your business logic. It can be, it can test just some part of your logic. It can be some kind of an invariant or some kind of safe boundaries. So that no matter what crazy data you get to your method, the end result is within those safe boundaries. And this is one of the ways where property-based testing is very good to make sure that your code behaves. And what other properties could we think of here? We could say that if we have two customers of which one of them joined earlier in the past and the other joined later in the past, then the discount for the older one has to be greater or equal than the discount for the younger one. This is something that has to always be true for the two customers within that relationship. And similarly for the birthday discount, a property that we could think of would be that if we have a customer who has a birthday today and another one, then the discount for the birthday customer is always greater or equal than the discount for any other customer. So this is four properties that we could add here for our domain. So now let's see one of it in the code. But before we move to the code, let's say a bit about libraries. So property-based testing has been with us for over 20 years now. It has started with implementations in Haskell and Erlang. And since then, a lot of the libraries have followed. So right now, no matter really what language you are using, there's probably one or more libraries for property-based testing created for this language. However, not all of the libraries are created equal because as I said, the library here plays a huge role. It does the heavy lifting for you. So we can have the support of the library which is very sophisticated, very helpful, or you could have a very basic library which helps only a bit. So it's a good idea to investigate a bit about which library is the most mature, the best for your language. And where does Scala, sorry, where does Java lay here? Where is it? In Java, you have, I think, more than 10 libraries that I could found for property-based testing, but there are three that I recommend to you that you can have a look at. Which one to choose is a matter of style, really, and preference. Some of them are related to the first one, related to the framework. So whether it's Java 4 or 5, some of them are framework-independent. And this quick theory has this specific DSL which can be to your liking or maybe you don't like it. So you have to take a look. However, all three of them are great. And during this presentation, the examples will be in Java. This is one of my favorite libraries for property-based testing. It's very mature. It has a lot of features and it helps a lot. But everything that I talk about can be easily done with any other property-based testing library in Java or in any other language. The concepts are the same. So let's go to our example that we have been waiting for. We are writing the property that the discount is never larger than 30%. So the body of the test is pretty straightforward. We calculate the discount and we have an assertion that it's not bigger than, it's actually less than 30%. Then at the top we have the property annotation. But all the magic is here in this for all annotations. Because what happens here is we say to Jakewick, dear Jakewick, this is my property. Please generate a thousand valid pairs of customers and current dates. And please verify that this property is true for every set of data that you have randomly prepared for me. Well, I said randomly. But in fact, this is something more than randomness. This is something that I like to call nasty randomness or vicious randomness, malicious randomness. Because what the library will try to do is it will try to find tricky data for you so that it breaks your code. For example, if you had numbers, then you can make sure that you will for sure have a zero, the minimum number from the range, the maximum number from the range, and so on. If you had strings, you could be sure that among the data generated you will have an empty string. You would have very long strings. You would have strings with various car sets, maybe Japanese kanji, maybe Polish characters. You would have white spaces, non-printable characters, and so on and so on. In terms of lists, for example, you would have empty lists, very long lists. Lists with consist of the same element, repeated over and over again, and so on. So the library tries to find tricky data for you. And this is one of the things which the library can be excellent at or it can be basic at. It's how nasty data generated is. And what happens now when the library finds a bug for you? When it finds that during one of those thousand runs, something is not right and your test fails. jquick in this, jquick, for example, prints a lot of helpful data here. So let's look at this from the end. Here we've got this original sample. So this is actually the set of data for which jquick has found that my property was not true, that the test failed. But what is here this sample thing? Actually, jquick and some other property-based testing libraries don't stop at finding an example for which your test fails. It will actually try to simplify your example, your input data, to find the simplest example which still makes your test fail so that it's instantly visible to you what the real problem is there. It's called shrinking in the world of property-based testing. And can you see here what the problem was in terms of my implementation and my test? The problem was 29th of February. So I have actually forgotten about leap years, and my code was throwing an exception when such data was passed to it. So see, this is the exact thing where property-based testing just shines at. It finds the bugs for you. Because usually when we write code, we think about a couple of examples in mind, and we write the code thinking about those examples, we write the tests thinking about those examples, and we are biased to thinking about them. And the library, it's not biased. It's finding weird data for us. It's finding tricky data that we haven't even thought of, and this way finds bugs for you. So what happens if we fix the bug? After all, it's random data. So when we run the test again, will it be green because I have fixed the bug or because we have another set of data? Actually, what Jake Quick does, and it's really nice, is that after failure, when you run the test again, it will use the previous seed. If we have randomness, then we have seed somewhere underneath always. So it will use the same seed to make sure that we have the same set of data generated, and that you know that when your test is green, it means you have actually fixed the problem. So that's just one of the things that Jake Quick does very nice. So now let's talk about generators. Where do we get this data from? And in property-based testing, it's a very important thing to provide the right generators, and which will provide correct data for your methods. Because if your generator generates incorrect data for you, which should never happen really in your system, and then you don't have to care about, then if your test fails, it doesn't give you any value. But if you have the correct generator providing correct data for you, then if your test fails, you probably have a bug in your code. So when you're working with simple types, you have a couple of generators out of the box. For example, for strings, you can have generators out of the box. You can specify their ranges. You can specify what kind of characters you want to include, and so on. For numbers as well, you can have positive, negative numbers. You can have big decimals and requested scale, and so on. You can specify collections as well. You can specify what elements are inside the collection and also what size the collection has to be. You can work with optionals or with nulls, and you have many, many useful generators out of the box. And if that's not enough, then you can write custom generators. The way you do it in jquick is you often use this arbitrary class, and on it you have static methods that you have to call, that you can call. So for example, here you have a generator, so a method returning an arbitrary of string. So this is arbitrary is a generator in j. So this method will return to you a generator of strings with the range from A to Z, but with Q filtered out, and with the length between one to eight. And the same way you can have here, and the way you use it in your property-based test is to this for all annotation. You either provide the name of the method or the string that you have provided next to the method. And if this is still not enough, then what you can do is you can combine the generators. So let's say here you want to have a generator for valid customers. So this is our customer case class. We know that it has a name and two dates. So what we need is an arbitrary of customer. And for this, we use an arbitrary of string defined here like that, so strings, alphabetic numbers between three and 21 of length. And then for the dates, we need to provide our own generators. So for this, we write a helper method that for a given range of dates will return to you an arbitrary of a date. And the way you can work with that is first you calculate the size of the range, and then you start from an arbitrary of longs. This is something you have out of the box. And then you say that you want to have longs from the range of between zero and this number, the number of days between minimum and maximum date, and then you map it to create such a date. What you end up with here is the arbitrary or the generator of the dates from a range. And the way we would use it is you provide different start and end ranges, depending on what you want to generate. Because for the date of when the customer joined, the oldest day the customer can join is just the start of our business. And the most recent date is the date of today. But for the date of birth is different. What is the minimum date of birth? Well, I don't know, but let's ask our product owner and when we get the answer, let's put it in the constant here. And the oldest, the soonest, the latest earliest, well, the maximum number from the range, the maximum date from the range is 18 years ago because maybe we have in our terms that you have to be 18 years old to join. So this way you can have two generators defined exactly according to the business requirements and business rules that are true and valid in your system. And then we can combine those three generators. And here we end up with the generator of customers using the constructor of the customer class. And this is what you would do if you don't have a generator for your specific classes out of the box. And then what can you do if you have to specify the relationship between your types, between your instances actually? So in our case, we have this property that the discount should be proportional to membership years. So we need to have a customer who joined earlier in the past and the one who joined later in the past. What we can do is we can use assumptions. What it does is it tells to the library out of the set of data that you generate, the library, please take only those for which this assumption is true. So only take the customers who don't have a birthday, the older and the newer cannot have the birthday today and please take only those pairs when this one has joined earlier in the past than the other one. And now with that assumptions, we can then call the calculate discount method for the both of them and verify with an assertion that the discount for the older customer is greater than for the newer customer. So this is a handy, a useful thing which sometimes you may want to use. And there's another thing that I really love about jquick and some of the other libraries is that sometimes you want to say that some of the data you want to test better more frequently than others. So for example here, in terms of our discount counting, let's say that we have a property which takes the number of years, the method prepare discount is written a bit differently. And we want to say that the majority of our customers are with us for a year. So what we do do, what we do here is we define the frequencies. We want to say that we want an arbitrary or a generator which for 60% of the time returns a one here. For 30% of the time it returns a zero and for 7% of the time returns two years for three, three. This way, and combining it with other generators which provide different data in our system testing them together, we make sure that the data that is most important, most frequent in our system will also be tested more extensively in our tests. So that the generators and the data are closer to real life. So this is an excellent point and something that when you write your own generators can be very useful. So now what are some of the challenges of property-based testing? If it's so great, then why doesn't everybody in the world already use property-based testing everywhere? Do example-based tests still have to exist? Maybe we don't need them anymore? Well, of course, it's not all so perfect. You have some challenges. And I will talk about four challenges here that are most important for my point of view. The first one is that property-based testing lets you, it actually makes you ask difficult questions earlier. Because traditionally, when you write tests, what you could do is you could, for example, follow the TDD, test-driven development, and write a very simple test case first, then write a very simple production code, then write a more complicated test case, more complicated production code, then think about edge cases and so on and so on. You have this nice cycle. And here with property-based tests, the first thing you have to do before writing the first line of code is to answer the question, what is the correct data in your system? So you have to define those minimum and maximum ranges, for example. So you will end up very early in the process, going back to your product owner and asking them, see what is the longest surname a person can have? Or what is the maximum number a person can spend for magazines each month and so on? By the way, these are the real questions that I had to ask when writing property-based tests. So you have to go back to your project owner and ask them those, even though you haven't really tried, you haven't even started testing. So this is kind of different and something you need to get used when writing property-based tests. The second challenge is non-determinism. It's random data. We don't really like it, do we? So it can often happen that you write your tests, they're all green on your machine, you repeat them again and again, they are green, they are green, it's fine. But then you push it and on Jenkins, they fail, they fail. What do you do? Well, what you do not do is you don't say, this is a flaky test, let's rerun it. No, you have to actually open the Jenkins logs and see what kind of data was generated for which you're called through an exception or returned invalid result. You have to take care of that. And how do we deal with that? How do we think about it? Well, don't get mad, think about it that actually property-based testing has found a bug for you. If it didn't found it, then it's probably your client or your end users who would find it. Someone who would have a longer name than you expected or that would put different data that you didn't expect to your system. And the first challenge is documenting. Often when I am new to some part of the system, I want to know what the code is about. What I do is I open the test case and I read the simplest first test scenario, the naive happy path scenario. I have an example there and I already know what the code is supposed to be doing generally. And the way our brain works actually is it likes the examples. It likes the concrete examples. So what it's not interested in that point, it's not interested in some kind of invariance, approximations and so on. It wants to see examples. And for this kind of documenting property-based testing is not so great. So this is one of the reasons why example-based testing have to stay. And the last challenge and the most important one in my opinion is coming up with properties. This is something that is not always so easy. So right now I have shown you properties that can be for the example domain that we have, but when you try to use properties next week in your project, you may find yourself staring at a blank screen not knowing what properties to think of because you have to define a rule which is always correct, but you cannot repeat the implementation. You cannot repeat the production code because then it doesn't have any value because if there's a bug in production code you will have a bug in your test. So you have to think about it differently. And this coming up with properties is a creative process for sure. So sometimes it is easier when you talk about it with your colleagues. Sometimes it's easier if you go for a walk or take a shower, just do something else and then you have this Eureka moment, you know the perfect property which will test your code. It's a creative process, but also you will get better with experience with that. But also that's not all because smart people have come up with a list of patterns for us. We like patterns, don't we? So you can have a look at this list of properties and when you need to find an inspiration to come up with properties, read a couple of them and probably you will find something that matches your business logic and is the right property. Let me show you four examples of such patterns here. The first one is Oracle, but not as a company name, but as a person who knows everything. So if you are a lucky person who somewhere, maybe in a different language, in a different package, has an implementation which does the same thing that you have to do in your code. Maybe you are writing a super efficient algorithm for sorting, but somewhere else in the standard library, there already is an algorithm for sorting, but which is not so efficient for your kind of data. So what you can do is you can run both algorithms on the data generated for your property-based tests and compare the results. If they are the same, then it means your implementation is correct. And if you have such a Oracle or such a reference implementation, then this is a very, very nice way to test your methods with property-based tests. Another nice pattern is alternate wording. So this works best with a colleague. You talk to each other about what your system, what your method actually does. So if we have a method for sorting and we would like to rephrase it in a different way, we could say that for each input list, the output list is sorted. Well, that doesn't give us a property. Let's try again. For each input list, if you look at the output list, then every number has to be greater than or equal to the previous number. And now this is something that you can make a property of because you can take the pairs in the output and compare them. And this kind of pairwise order is a very nice property in terms of sorting. So this is a very good pattern that you can use every time you want to write a property. Just think differently, rephrase what your method does and find the property. Another useful pattern is invariance. So some things never change. For example, if you have a list, then even after your sorting algorithm works, the number of elements in the list is still the same. So we can check this invariant, this kind of safe boundaries here with properties. Another invariant could be that every element which is on the output list also had to be present in the input list. Nothing comes up from the blue and so on. This is something that is very simple. It's not very creative. Maybe sometimes it can be, but it gives you confidence, some kind of confidence that for weird data, your system, for example, doesn't throw an exception or doesn't duplicate or produce extra values and so on. And the last useful pattern is symmetry. This is something that I like a lot. For example, imagine that you have a serialization method somewhere in your code, but you also have a deserialization method. What you can do is what you can test them together. So you have to generate data, you serialize it and then you deserialize it back again and you verify that at the end, you have the same than what you had at the beginning. Another example could be writing to a database and then reading back from the database and so on. This symmetry is also something that is not very, very creative, but it can find a lot of the bugs for you. Okay, so now that I have said that there are a couple of challenges and that maybe property-based testing is not as easy or not as straightforward as traditional tests, then do you have to have property-based testing for all of your code? Of course not, but where do you apply property-based tests? I have two strategies for you that I recommend. The first one is if you write a new piece of code, maybe start traditionally. So write a simple test, then a simple implementation, a more sophisticated test, more complex implementation, then maybe think about the edge cases and then when you think you have got it right, you have nothing else to add, only then start thinking about properties. And then maybe you will have to think twice about the business requirements and the way you understand them. Maybe the property-based tests will find some edge cases for you. Maybe it will find some bugs. Then you will fix your code, provide another property, and then this way you have a bit longer feedback loop. It's a journey which can be very exciting. And the other strategy is only write the properties for the key components in your system. What does it mean, key components? For example, maybe you don't want to have property-based testing for some trivial parts of the system, but you want to have them for those complicated ones. For the ones which calculate some weird things and nobody really understands what this piece of code is about. If you try to think about properties for them and you run property-based tests for it, maybe you will discover what it's really about and it will find some subtle bugs for you. And what else it can mean, key components, it can be the core parts of your system. This part where your system is different from the other companies out there. This part of the system that you want to make it right because if there's a bug in it, maybe someone will lose money. Maybe your company will lose money, maybe your end users will lose money or somebody will be really angry. So for those, it's worth this extra investment to think about properties and to make sure they're tested as well as it can be. So to sum up, the property-based testing comes with a higher cost, it's creative, it may take you longer time, but on the other hand, what you get in return is you have the value which no other kind of test can provide to you. The thing where the property-based tests shine are with finding edge cases and misconceptions in the way you think about your code. The most challenging thing is coming up with properties but you have the list of patterns and it will come more naturally with experience. And the last tricky part is balance. I cannot tell you that, I don't know, everybody should have 40% property-based tests and 60% example-based tests because every project is different. But if you find the right balance, if you find the sweet spot between example-based tests and property-based tests, then you will know that you have made everything that you could so that your code is as correct as possible for the happy path and edge cases you already know about and for those that you haven't ever even thought of. If you'd like to know more, I encourage you to read the J-Quick documentation which talks about more features that I didn't cover here and also talks more about thinking about properties. This proper testing is a great series of posts about property-based testing and coming up with properties and it doesn't matter that it's about the Erlang library. The first link is this list of patterns that you may want to add to your bookmarks and consult it for inspiration when you are trying to think of properties and all the examples from this presentations live in a GitHub repo under the last link. Thank you. If you have any feedback or comments, you can scan the QR code below and if you have any questions, I will be happy to answer them now. So let me see the Q&A tab. Yeah, so we have a question there. With so many iterations of a given property test, does property testing take a long time to run? This is an excellent question. After all, we are running our test, for example, a thousand times. So it should be a thousand of times slower or longer. Actually, it's a bit better than that. The libraries do a good job of optimizing this but still this is a lot of the runs. And you may want to think about it when your test does something special. For example, if your test reads from a database or opens some files and so on, then maybe you want to specify it in JQQ just provide an annotation that this property test, let's just call it only a hundred times or only 10 times, not a thousand. And this is also something to take into consideration when writing all kinds of integration tests that can take longer. So that's a very good thing to watch out for. Thank you so much. Yeah, that was a great, very thorough presentation. Well, yeah, thank you so much. I'd love to share your experience for this great presentation.