 I'm going to introduce Holden that will present you a talk about distributed system testing with Apash Kafka. No, no, spark and beam. Spark. Kafka is different. I love the Kafka people. They're amazing. You could use this to test Kafka if you tried really hard. But the same techniques work. So yeah, my name is Holden. My preferred pronouns are she or her. I'm a developer advocate at Google. I'm on the Apache Spark PMC, which is nice. But I'm very recently on the Spark PMC. So if you run into spark bugs, they are definitely someone else's fault. Only things that are new are my fault. I'm a co-author of two books that O'Reilly is selling down at the end of the hall. And you can follow me on Twitter. And for the people in the back, you probably cannot read this bottom part. But the slides are already linked from the talk abstract on Faustem. So don't worry about it. If you're really excited about spark and talk about testing spark somehow gets you excited to write more spark code. I have a bunch of videos on spark as well. So in addition to being who I am professionally, I'm trans, queer, Canadian, and part of the leather community. This is not really directly related to anything I'm talking about. I just think it's important to remember that we all come from different backgrounds. And we should be nice to each other. There's no secret Canadian debugger that we've been hiding from you. We all have no clue how to debug our distributed Java code. Like, this is, we're all screwed. But if we work together, we can be, you know, get through our lives with only minimum amounts of sadness. Okay, cool. So I'm going to talk about why you should test and validate your distributed systems. And admittedly, you showed up to a talk about that. So you probably care. But a little bit of preaching to the choir never hurts. We'll look at normal unit testing in spark and beam. And then we'll talk about what happens when we actually need to go from unit testing to try and testing some of the distributed properties of our system and how that makes us sad, what we can get through it. And then we'll talk about validation because I think validation is really important. And I keep seeing people building pipelines with no validation built into it. And that just makes me sad. And there will be at least one panda, I hope. I may have actually taken the panda out by accident and several cats. And if you come to my talk tomorrow, the content is different. I do reuse cat pictures though. It takes a long time to artisanally source these. And it's all about the sustainable Flickr harvest. The Creative Commons tags just aren't refreshed as much as they used to be. So if you've got a cat, please take pictures of your cat and upload them CC license to Flickr so that I can have fresh cat pictures. So I'm hoping you're nice people. You laughed at my kind of crappy jokes. So you probably are. If you don't like silly pictures, you're sort of stuck with me, although there is an exit. And you're probably familiar with one of these languages, but apparently not all of you are. And so if there's something in an example which doesn't make sense, please raise your hand and I will explain that piece of syntax to you because it probably doesn't make sense to someone else. And it makes sense to me because I spend far too much time writing code with underscores in it. And it teaches me bad habits. How many people are familiar with Spark? That is a lot of people. How many people are familiar with Beam? How many people came here for a Kafka talk? Thank God. I'm sorry. I'm sorry. Okay, regardless, even if you're not familiar with these particular systems, I'm hoping that the same techniques can apply to other things. The specific libraries and code examples that I'm going to show, you're going to have to rewrite, but honestly it's code on the slide. You should probably rewrite it anyways. Cool. So why should we test? It makes us better people. And it avoids making our users angry. I have some funny stories, all of which are under NDA. So it's unfortunate, but they may involve companies that sell books and that small little search engine. It can help you save money. AWS or Google Cloud is expensive. And if we can catch our errors before we put them into production, we can save having to run our pipeline a second time. Repeating the mistakes which I'm not legally allowed to discuss is not a lot of fun. I didn't have to find a new job after any of these, but that's mostly because in California it's really hard to hire engineers. And honestly, you probably already care. But validation, I'm not sure if you all care about validation. So fundamentally, the idea behind validation is eventually, even if we have really good tests, we're going to get on the failboat. There is no world in which our software works. I've looked and the closet to Narnia did not have working software either. There was an emergency patch. They had to apply. The trees were down. I guess that didn't work. Regardless, when we know we're on fire or we're on the failboat, we can halt deployment, we can roll back. We can avoid recommending inappropriate products to miners. We can keep our jobs, right? And the problem is, if we don't know we're on fire, the people that find out we're on fire are our customers. And our customers sometimes have influence over our lives. And if I know I'm on fire, I can stop it before I get woken up at three o'clock in the morning. I no longer carry a pager, which was one of my key deciding factors in choosing jobs, but you may not have ridden yourself of a pager yet, so you may actually care about this. And the TLDR is some jerk on another floor is going to change the meaning of some data you depend on, and that's just going to happen. If you're lucky, there are jerks that work at the same company as you. And if you're unlucky, there are a jerk at a vendor who you have no idea how to contact. And it's just going to fail. And yeah, OK. Continued. About 39% of people take the results of their spark jobs and push them into production automatically. This is terrifying, especially when we look at the next slide. Has the output of your spark job ever caused a serious production problem? And 50% of people say no. 16% of people had to update their resume after running spark. And 30% of people did not have to update their resume after running spark, but their customers were upset. And so the moral of the story is we really want to try and keep ourselves in this half as much as we can. Also, if you disagree with these numbers, you can fill out the survey here, and I'll take it into account. Or more accurately, Google will automatically. I'm pretty lazy. So why don't we test? Faking data is hard, especially with distributed systems. I probably can't get away with three element tests. The good old, I checked null, I checked a filled in one, and I checked one with some junk data, is no longer enough. I'm testing on 100 computers. I need more examples. Making a terabyte of examples by hand is not really scalable. Our tests can get slow. If you work with Scala, you're already familiar with this. It can take a long time. Writing tests is effort. And my boss was like, well, it worked. Just put it into production. And distributed systems make our life terrible. The follow-up is somehow someone convinced people that notebooks were a great way to interact with distributed systems. And I think notebooks are amazing, but they're really shady for testing. And I keep seeing people doing like wonderful exploratory work in their notebooks and then they put them into production and they don't test them because it was too much work. And this is sad. Just because you put it in a notebook doesn't mean you don't have to test it. It just means you need to take it out of the notebook. And so why don't we validate? We already tested our code. What could go wrong? It turns out many things. The other ones are the things that we tend to do for validation, which involve collecting metrics about the number of things we've processed. Looking up the size of our outputs and all of these things get a lot harder with the distributed system. We have to aggregate this information across a whole bunch of machines. And the built-in things to aggregate these metrics are all broken. And so that's great. Yeah. Yay, question. Sorry, the distinction between the word test and validate. Can you help me? Yay, sure. Okay, so the distinction between test and validate. Test is like a thing that I can run and it tells me yay or nay on my code change. Validate is a thing which is on my pipeline and after I run my pipeline in production, it tells me this was probably OK, right? Like, I processed the number of records I was expecting. Nothing really looked weird. You should run your validation stuff while you're running your tests as well. But you probably don't want to, you know, run your tests in production. You want to look at your counters and your metrics explicitly in production and compare them to historic values. Or if you've got some ground truth about what you're expecting, you can encode these validation rules and have them baked into your pipeline. And so my tests are going to catch the errors that I can think of and hopefully, if I'm lucky, validation will catch the errors which I've never thought of processing like an extra terabyte of data or just dropping an extra terabyte of data on the floor, which I've done. That pipeline made some really bad recommendations when we only processed about two gigs of data. You know, yeah. OK, cool. Yeah, OK, so this is a cat. The cat is sleeping. And the audience appears awake. So that's surprising. The simplest unit test looks a lot like a traditional unit test, right? We say, like, this is our input. This is our expected output. And of course, because this is, you know, a distributed systems thing, we're going to use word counters as our example because that's just what everyone uses distributed systems for. That's why it's the default example. And then we assert that our tokenizer, which is part of our word count pipeline, is going to, like, take this input and produce our expected output. And this is really simple. This has the downside of really not testing a lot. It will catch bugs inside of our tokenize function, right? If I was splitting on the letter I and I wasn't expecting to, it will catch that. But it won't catch if I've done anything weird with processing empty data or empty partitions. So it's not great, but it's better than nothing. And in Beam, we can write something similar, but it's in Java. We've got a lot of extra alligators. Om nom nom types. So we can go ahead. We can say this is our expected output. I forgot to put the input. Pretend that I put it there. And we run our, like, fancy pipeline. And we say, well, we run our pipeline, but we have our assertion here. And we say, like, this is what I'm expecting to come out. This is nice in Beam because they actually thought about testing when they made Beam. So this is built into Beam. Whereas this in Spark for some reason is not built into Spark. Yay! My attempts to get it merged have not succeeded despite being a PMC member. I should try again. Anyways, but the TLDR is, we have our exciting word count pipeline. We can test it. But it's not great. So where do these tests run? They run on our local machine. That's not very good. But it's better than nothing. And Spark attempts to simulate a real cluster. The simulation breaks, right? It's not perfect. It doesn't behave the same way the real cluster does when you're adding files. It doesn't have the same properties with, like, you know, failures are difficult to simulate in a local mode environment. Beam also has a local mode. It's a little different. It does less work on the simulation and more work on Beam fast. So this is nice. The direct runner is good for writing a bunch of Beam unit tests. But it's not perfect. And another option is you can, like, say, hey, your simulation of a real cluster isn't enough. I'm just going to stand up a two-node cluster on my machine. Please talk to that. And you can do that with this very exciting shell script. You set up .sh, travels, .yaml, whatever. And you just set this shell environment variable, which is definitely the best way to configure our software to spark local hosts. Ideally, you would, like, point this to an actual, like, three-node cluster rather than, you know, two shell scripts running on your machine. But two shell scripts running on our machine will catch more errors than the simulated cluster. This gives us almost a real cluster experience. But, okay, so we've essentially covered all of the testing which isn't going to give us a lot. We're going to focus on the testing which is going to catch fun problems, the kinds of problems which I'm not going to think about. And this is because I'm used to writing software that has to work on my machine, but I'm not used to writing software that has to work on a few hundred machines. And so we need to test that we handle the partitioning correctly. We need to test that, you know, the serialization of our things is actually going to be successful. And if we're in a local simulator, it tries to serialize the things that are needed, only once you run on a real cluster that it actually, you know for sure, it's going to work again. So how do we do this? So one of the first problems is, if we think back to here, we have this like assertion that these two sets are equal, but this is just comparing the two sets locally. And if I'm going to test at scale, I probably can't bring a terabyte of data back to my machine. Otherwise, you should just run your stuff on that machine and, you know, not use any of these things because they're terrible. Right, yeah. So that's great. So now we have to figure out if our two sets are equal. Spark has built in distributed set operations. They look almost exactly like what you want, except it doesn't know the difference between coffee, coffee, panda, and panda, coffee. And if someone says these two things are equal, I say someone stole in my coffee and I will cut them. So instead, we can use RDD comparisons, or if you really like writing co-group expressions in Spark, you can just write a co-group in a zip. And then you can actually compare that your two distributed collections are equal. And in Beam, it's pretty much the same thing. You co-group, then you filter out the records where everything is already matched, and you P-assert that the result, once you've done your filter, is empty. And if it's not empty, then something has gone wrong and you can see the collections that were not empty. But where do we get our data, right? If I've made my data, if I'm testing a distributed collection, I actually want to test something that's too big for a single machine. Most people still generate their data by hand, and this is not good enough. I don't have the time to write a gigabyte of test data, let alone figure out what a gigabyte of golden set data looks like. Some people just get away with sampling production data. I think these answers are definitely skewed to the U.S., where privacy is a flexible concept. I really hope I don't get fired for saying that. And of course, not at glorious employer who can do no wrong, but definitely at other companies, they may not have the same restrictions. Video camera in the back. Right. And so that's maybe not going to work for you. But if you work maybe where it's not user privacy data, it's like sample data from cows. The cows probably aren't going to sue you. I mean, you may sue you on behalf of the cows, but you've got a good shot in court and you can probably just sample your data for your tests. Alternatively, we can try and sort of change how we think about writing our tests from golden sets to properties. Are there any like recovering Haskell users or not so recovering Haskell users? One person. Damn, I was expecting at least two. Okay, well, that's fine. So the Haskell people definitely looked at testing and said this is not a lot of fun. I'm going to take a lot of time and write some papers on this, which is what Haskell people do best. But incidentally they made some software which was really good and the rest of the world copied. So QuickCheck is amazing. And ScalaCheck is Scala is not all that obviously not a rip-off of QuickCheck. There is a Python version of ScalaCheck. It behaves pretty much the same. It's a little more difficult to work with because of that whole types thing. No offense to Python users in the house. And so there's two libraries to make this work with Spark. No one's bothered making a library to make this work with Beam yet, but if you're really excited about property testing, you could totally just take one of these libraries and switch like probably ten words around and make it run on Beam. I tried doing that on the airplane, but the United Wi-Fi was not good enough. So we can specify this exciting property that when I compute a thing with a map expression, and so here I'm just taking an input and I'm saying I want the length of everything. I'm saying that the result, the number of elements in my result should not have changed. This is not all that exciting, but we could actually think about how we could write a similar property which would be useful to us. We might say something like if we've got a data ingest pipeline and its responsibility is to kick out invalid records, the number of records that I have after I've run should be less than the number of records I had when I started. There should have been some invalid data that I kicked out and if there isn't, I didn't do my job. And if I produced more records, I did something very different than my job. And so simple property system, and then we can tell it to generate one million entries. Okay, I'm glad to see Dr. Evil did make it over here. Really, you can change this to whatever number you want. The important part is when you're doing this, you have to point it at a real cluster. It's not as if you run it on your local host. It's just going to be sad. But the nice thing is, you can actually have the same properties and you can have different generator configurations for your integration environment and your local dev environment. So when you're running this on your local machine, you'll generate a few hundred examples, but when you run on your integration environment, you generate the one million entries. Right, yeah. So this can get a bit slow. Even... Well, okay, so the property tests generate a lot of tests, right? And a lot of times, what we'll find out is that, yeah, it turns out the error was actually just really a classical error. I just screwed up inside of my length function or something, right? Like I just did something wrong with parsing the data. And for those things, we should try and catch those errors in sort of the classical ways. The only problem is that the programming model of Spark really encourages us to write code, which is difficult to test. So let's look at this, right? E.g., lambdas are not always our friend. So I love lambda expressions. I am a recovering Haskell programmer myself. Once upon a time, I was a scheme programmer. I've written a lot of lambdas. But at some point, writing all of my functions like this mean that when I want to test my difficult to tokenize RDD function, I have to provide it with a fake cluster. Alternatively, if I split this out into, you know, my thing, which is depending on Spark's distributed data sets, and the thing which really doesn't care about the data set, I can use a lot of really great auto-generated tests on this tokenized thing very quickly and very, very effectively. Because sometimes my error is inside of the lambda, right? And we should catch those more quickly. The other thing is like realistically, even beyond the fastness of this, has anyone like enjoyed reading the stack traces from Spark? Right, that's zero people. And it's not just that everyone's fallen asleep. It's that Spark is incredibly difficult to debug. And so one of the things that we want in our test is we want to be able to figure out what went wrong. And if we make our tests only around this, every time we break something inside of this, our life gets really, really sad. And for the people in the back, I just took the lambda expression and gave it a name. And now that it has a name, it, you know, feels important and we can also test it on its own. Right, okay. So we can do fun things with data sets. The TLDR is the Spark people decided they wanted to rewrite their API. And so, yeah, whatever. Rewriting APIs, all kinds of fun. We can use all of the same techniques except instead of talking about RDDs, we talk about data sets. And I'm sure that the Beam people will eventually also decide to rewrite their API because it's what we do. And then we can also probably just take the same techniques that we already talked about, change the names around and keep going. Right, here it's pretty much the same thing. We have an equality function and this checks that these two things are actually equal. I mean, in this case, they're checking that the same thing is equal to itself. So this better be true. If it's not, there's a bug in the testing code. But this is how you would do a distributed comparison there as well. So the reason we care about this is they added schema information and we can generate our generators from schema information. Streaming has all of these problems except we also don't know when our test is done. So the problem is not only do we have to figure out how to test and distribute a collection, we have to figure out when we've processed all of our data. And if we've got Kafka, I mean, we can figure out that we've processed everything that was in Kafka right now, but we don't know if our test has finished loading everything into Kafka. And then we end up like these pandas. Panda runs into panda and it's sad but also cute because our test will fail and how most people solve this is they put a sleep 100 in their test. And then you're not this cute panda, you're a very sad other animal that I don't want to hug. It's okay though. If you've ever said to yourself, I bet someone bothered to solve this problem, three different people have in different open source libraries and they essentially hide everything from you and screw around with the underlying system clock to guarantee that things have happened. And this is guaranteed with a very flexible version of the word guarantee. But essentially there's no sleeps, instead your system clock is just going to go and it'll probably work. If it doesn't work for you, they're all open source libraries. Okay, so we can put these techniques together into Spark structured streaming. We take a new API, another new API and a third API and put them together. What could go wrong? It turns out most things. But they actually, the Spark people finally realize that testing was a first class citizen and so you no longer have to make the system clock go like instead we have this nice thing called process all available. Much nicer, much more reasonable. You just write this thing, you pretend that writing Scala code is natural to you and then it just works. If you don't write Scala code, the Java version of it is pretty much the same thing with extra angles eating. So that's cool. Business logic goes here. And this is really the part we want to test, but Spark is amazing. So validation, yay! And so yeah, you should run your validation stuff during your integration testing as well. This will probably help you catch if your validation rules are wrong before you put them into production. So here we validate our jobs. Both Beam and Spark have their built-in counters. They've got things like bytes that have been written and read during each stage, the number of records which are being shuffled around, and all sorts of really wonderful system metrics that you probably don't care about. That being said, if GC time spikes a giant amount, you probably want to investigate that, but that's not really what this is about. You can add the counters for the things you care about. I care about records which I declare invalid and kick out of my pipeline. I also care about the users for whom I have no idea what to recommend and I give the default recommendation to. And accumulators in Spark are not perfect. In Beam they're called counters and they only work some of the time. But, or more specifically, their behavior depends on which execution engine Beam is running on. That's specific day. So if you run Beam on Spark, it behaves as well as Spark does, which is poorly, and if you run Beam on something else, it might work better. But that might get fixed, maybe. So we can have simple rules which are like the percentage of invalid records should not be that high. We can also have other historic rules like the number of users should be greater than the number of users I had yesterday. Or if you're at a failing company, you might not have decreased by more than 10%, or it's time to halt the job because I'm going to update my resume anyways. It turns out most people are really lazy. And the most common thing that they just do for validation is they validate that the output file sizes are reasonable, which is really easy to do, and they validate that the execution time has not spiked. Or, alternatively, also decreased because that can be a really good metric for I dumped a bunch of data on the floor if my pipeline finished in, you know, half as long time. There's some libraries to do this. You should not use any of them, even though I co-wrote some of them. But they're great concepts, and you can just reimplement all of this code yourself, which I know is what we love doing. And one thing to think about is if your property-based tests, if any of them can make sense as validation rules. For example, if we say things about the number of records coming in and out of different stages in our validation, those can be, sorry, in our property tests, those can be really good things to assert our true at runtime. Or maybe not assert because we don't want to throw an exception, but at least log. Yay, Python example! I promised there was going to be Python. So this is the Python Spark example. We keep track of the number of records that we reject. I'm dividing by zero, so I'm sort of hoping that I reject all of the records, or I have a magical CPU from Intel. So I have to make the Intel jokes. It's pretty much a requirement. I mean, this is like the really, really old Intel joke about floating point numbers. Anyone remember? Yeah, okay. Some people are nodding their heads, like shut up and move on. So we do our rejection count here, and we keep track of the number of records which we say are sad records. Roggin. And then we go ahead and we just print it out with a print statement, but ideally we would do something smarter like maybe not push the job into production. Or sorry, not push the job, not push our output into production. So you would exit one or whatever you do to indicate to your airflow not to continue running without human intervention. We can do the same thing in Scala code. Yay! Yeah, this is the same thing in Scala code. I really should have just left it in just Python code. But we can also use the internal counters, which Spark keeps track of and Beam. And so here I can say I'm expecting between three and a thousand records. This is actually true. I should probably just be using pandas because pandas are much cuter than Spark. But we can have an absolute rule that says, hey, this is some values and it's referenced to an internal counter. All right, we can also, instead of an absolute rule, we can have a historic rule where we say I want this value to be within 10% of where I was yesterday or within 10% of where I was last month, like whatever cyclic property this record might hold. If we were looking at orders, we would probably use year over year to account for seasonal trends. The same thing can be done in Beam with a lot more text because it's in Java. So we can make the same counters for the words that we've matched and the words that we've rejected. And then we can have our assertion here that these values should be reasonable after our pipeline has run. That's the number of records that are processed. Sorry, the number of records that we rejected versus kept. We probably wouldn't use words unless you're just deploying the word count pipeline again, which is great. So if this talk was not that much with the sleepy pandas for people, there's a bunch of related things. If anyone's like a really hardcore Java developer, there's a Java version of this. I should have marked it. But one of these links will take you to a Java thing. And you can't click on any of them because you're sitting in this room. But later on, when you go and look at the PDF, you can figure out which one of these is Java specific. And there is a Java blog post on how to do the same stuff. There are a bunch of GitHub projects because why not? SS Check is notable because it's one of the ones which isn't written by me, as is the Spark integration test that's written by my former employer who no longer pays me money, but they still make good software. And SS Check is not Spark or Beam specific, but it's a really interesting library and I think everyone should check out property testing. Here's how to include Spark testing base if you want to use it in your software. SBT for people who like slow build times and warm laptops. And Maven for people who remember the 90s. There are a bunch of books on Spark. I'm a co-author of many of them. This is the only one which really talks about testing because I thought people would care about testing and figure it out on their own. It turns out that was not the case. There's only like one chapter on testing and though rarely folks have it for sale on their table. And you can also buy it from Amazon if you're worried about Jeff Bezos' ability to purchase another newspaper. I worry about this greatly. That joke goes over a lot better in Seattle. Actually it goes over a lot worse in Seattle, but there's more of a reaction. I will be in the Faust M HPC room tomorrow at four. I don't remember what I'm talking about, which is super awkward, but I already made the slides. I just, yes. If anyone wants to go to Stockholm, I'll be in JFocus. If anyone wants an excuse to go to California, San Jose is a shitty place to visit, but it's close to San Francisco. And London and Brazil, those are nice places. And QCon will actually be in San Francisco. So I'm going to do questions now. Yeah, so that's okay. Thanks. I think I've got 10 minutes for questions, hopefully. Yes. Yay! 10 minutes for questions. Does anyone have questions or was everyone really zoned out working on their laptop? Either way is fine. Laptops it was. Okay. Yay! Question! Sure. So that's a great question. The question is what should we do for validation when it fails? And I think that depends a lot on your use case. Back when I first started making recommendation pipelines, I thought recommendations aren't that bad if I get them wrong. I'll push it into production and log it. Then I found out that I was wrong. So I think it's really important. If we've detected that something has gone wrong, we don't know how bad it is normally. So I think it's useful to save the output and exit with an error code so that Luigi or Airflow or whichever tool we're using to coordinate our pushes to production does not push our data to production, but I can still go and look at my output and manually validate what happened because it's possible. Our validation rules don't have to be perfect. We might have a rule in there that says our user growth is expected to be within 20% or something which happened at Amazon was I expect my orders to be in this band and sometimes you go out of that band but it turns out it's just because there's a sports game and that doesn't mean that your code is wrong but you wanted to check it before you pushed it through. You want a human to have to sit there, think about it and then be able to blame that human later. Related, you should have someone else... No, I'm just kidding. You don't actually need to blame the human afterwards. I think blaming people is bad but you want a human to be able to look at it and think like, does this warning make sense given the information I have? If I try and write validation rules which are always correct, I'm going to have to connect to the National Football League website and that's not a thing I want to do. Nothing against NFL football. Okay, any... Yay, second question. Okay, so the question is how much validation is too much? That's a good question. I think it depends on what the potential impact is of your pipeline going wrong. I think if you're building things which might kill people, like maybe self-driving cars, I would not say there is particularly an upper bound. On the other hand, you have to ask yourself essentially, is my time better spent making more validation rules or more tests? And I think often the answer is more tests and the validation rules that I tend to see people be successful with are very simple and we tend to get a lot of return on investment for the kind of validation rules we can come up with in a week or two and then sort of reverting back to just focusing on tests. But one of the things which I think is great is if we have an outage coming back and thinking how we often do, could I have caught this change with tests? Also, if it was not a thing we could have caught with tests, is it the kind of thing I could have caught with a validation rule? Was it something about my input that changed drastically and I could have measured this change and alerted on it? So I think it's very much unique to your specific problem space which is a very weasely way of not answering the question. Any other questions for me to not answer? Yay! Sure. So the question is, is there any way to validate the cost of a validation rule versus the cost of it not running? And so I think most of our validation rules the primary cost is going to be the false positives, right? Because they're not going to be perfect, right? The secondary cost is, for some of these, like keeping track of, I go back here, right? Keeping track of the records that I've rejected. Like this thing has some cost, right? This is not free. That being said, this is negligible. Both Spark and Beam and pretty much all of the systems that have systems for collecting metrics like this do a poor job of collecting the metrics because they wanted them to be very cheap to collect, right? And so they're not perfect metrics but they're very inexpensive to collect. And so I think by default like the cost is, has this validation rule stopped me more often than it saved me? And if that's the case, probably I should try and refine the rule or think of a different way because eventually what will happen is someone will just wake up, see the alarm and press go and then all of my rules are useless and I don't want to get there, right? And so if it's crossed that threshold, definitely back off depending on the people you're working with you might want to back off earlier. Cool. That was a question I could actually answer. Ask a question I can answer. The question is why do I always say recovering before Haskell developer? It's a bad habit of mine. Mostly because I tend to work in this problem space where we tend to use a lot of JVM and Python based software. So there's a lot of people who have come to us from a different community and been like I love functional programming but I also love money. And so they're like recovering because they're like types everywhere and then it's like, maybe not here, maybe not today. And it's just switching between the different problem spaces. There's nothing wrong with the people that have decided they love Haskell in their life and that's just what they want to do. But if you want to write algorithms you're not going to do these crazy hacks. You're going to do different things. I mean you'll probably do quick check but you're probably not going to do this like weird side effect thing in the middle of your code. You're going to feel bad about having a side effect in the middle of your code and you're going to not do that. That being said everyone else is just like yeah it's a side effect. Actually they don't even say that. They don't even think that. That's just a functional paradigm which I break all the time. Okay. I think that's probably it for questions. If anyone wants to talk to me I am wearing this and I will be around tomorrow. Oh right and this is Boo. That's very important. I forgot to introduce my co-presenter. Her name is Boo and she is a doogler. She has her own Google badge which was part of the contract negotiations. Boo is very happy to meet all of you too. She can answer any questions about barking. So that's it. Thank you so much for having me and listening.