 I'm really excited to be here talking to you about this at RubyConf this year. I'd just like to say thank you to all the organizers who have put in a lot of work to put this together for us so we could be here today. So again, my name is, let's see, is this working? Yep, my name is Tim Mertens. I am a software architect at a FinTech company based out of Chicago called Avant. Prior to my current role there, prior to my current role there, I led our test engineering team for about three years and I've been working in the software industry for about 10. You can find me on GitHub and on Twitter, sort of, I'm not a big Twitter user, but you can hit me up on there. And I'm gonna be posting some links later on my Twitter account, some resources for you to look at. So I'm gonna be talking today about deterministic solutions to intermittent test failures. I think we've, well, maybe not all of us, but a lot of us have experienced this problem where you have a really large test suite and eventually things start to fail irregularly and it can be a really big pain in the neck. So just to reiterate that fact, my wife made fun of me for making this slide. She says I'm a little bit of a fanboy for Elon Musk, but, you know, Elon, he's sitting in his chair and he gets his newspaper and he's reading it and it's an article about how there was another gravitational wave detected and he says, you see, it's happened again. And his reasonable friend comes over to him and says, Elon, this is base reality. It's not a simulation. And meanwhile, somewhere else in the universe, there is this old guy sitting at a computer and he says, Jesus, the build failed again. Can you rerun it? And in the other room, this younger guy says, sorry, I'm kind of busy. He's playing games on his phone. And the old guy says to him, don't make me send you back down there. And he responds, yes, father. All right, so this is something we all deal with. I'm gonna be using some code samples in my presentation today. These samples are all based on RSpec. You can take the workflow, though, that I'm gonna share with you and this can be applied to any testing framework, but my primary experience is in RSpec and it provides a lot of tooling that helps us to do what I'm gonna show you today. There are also some source code examples that I put together that are available. I'll have this link again at the end of my slide and I'll put it on Twitter. But this has examples of some of the test failures that I'm gonna talk about today. And the build in this repository is purposely failing so that you can go familiarize yourself with these things and solve the failures that are happening in that repository. So this is a common thing that happens. You feel like the build is passing consistently for a while depending on the size of your repository and how many developers you have. It might be a while for you, it could be a day, it could be three days, it could be a week. And inevitably somebody says, ah, this is awesome. The build has been passing for so long. This is great. And then what happens? The next build always, always fails. And people say, oh, well that's just a flaky test. And before we get into some of the workflow part of this, I wanna talk about the myth of flaky tests and I wanna deconstruct this for us. Because there's this common notion that there are things in our test suite that are unsolvable. And I've been doing this for long enough that I just don't buy it. So I wanna remind everybody, your tests are software too. If you have an API call that is critical to your business and it's failing every once in a while and the CEO or a business person comes to you and says, hey, why is this failing? Do you say to them, oh, well, it's just a flaky endpoint and you know, there's nothing we can do about it. No, you don't do that. It's software, you make it do what you need it to do. Test code does exactly what you tell it to do. It won't do more than you tell it to do and won't do anything less. It's gonna do exactly what you say. And when we use the term flaky, we're implying that there is some kind of unsolvable problem. We have this failure that we just can't deal with. There's nothing we can do about it. I like to take that term and replace it instead with the term non-deterministic because then what we're saying is we have behavior that's happening in our system that can be accounted for. Once we understand the root cause of a failure, we can modify our test or we can, maybe it's a production bug and we can fix the bug, but we can account for that non-determinism so that our tests themselves are deterministic. I also wanna remind everybody that when you're ignoring failures, they could be real defects. You could have legitimate defects in your production code that people are simply ignoring because it's difficult to troubleshoot. So I wanna talk a little bit about how I got here and how we came up with this and it's a story that is fraught with peril. There were a lot of failing tests that I've had to deal with in the past and this is where what I'm gonna tell you about today came from. For those of you who have not been in the development community for a long time, I'm gonna be talking a lot about continuous integration. This is where you push your code up to, let's say, GitHub and some service is going to automatically run your test for you. Some examples of that are Travis or Codeship or CircleCI or Jenkins. This is what we're talking about when we say continuous integration. So back in the day when Avant was small, we had a small number of tests and code that we could load up and put into our CI tool and it would take our build and it would run it and hopefully things would go smoothly and everybody was happy. But over time, our code base grew and our test suite grew and so our test suite was taking a lot longer and so we said, all right, well, we need to paralyze this and manage our build times. And so when I say paralyze builds, that means we're taking, let's say, a thousand tests, 10,000 tests, however many tests you have in your test suite and you're splitting them across a bunch of different containers or workers or whatever you wanna call them. And so we did this and we split our build across a bunch of containers and every once in a while, something would explode. But we were focused on building features and when you're focused on building features, you can lose sight of the fact that this is a bug that's existing in your test suite and people like to say, oh, well, that's just a flaky test and master. Rerun your test suite or run the test locally and if it passes, go ahead and merge your code or rerun master and then deploy as long as it passes. And so these things tend to just get left behind. And so as time went on, we kept building more and more things. We had much more complex dependencies and so we started using more complex things in our test suite, like factories, to help us build out these large graphs of dependencies for some complicated business logic that was getting tested. And eventually people just don't really know everything that is going into what's happening when they run a test. And so we were still running a lot of builds and running a lot of tests and we had a lot more code and things were still blowing up sometimes and people would complain about it but they wouldn't always do something about it unless it was a critical failure where it was failing all the time. Now eventually we got to the point where we realized, you know what, our provider that we're using isn't really cutting anymore, we need more capacity and so we decided to search for a new CI provider. And so we were looking for a CI provider that could give us these big machines where we could run a lot of tests on them and it would shorten our build times for us. And so we did that and we tried it, we put our test suite on this new CI provider that would run things more efficiently and all of a sudden everything is exploding all the time and what we found was that our old CI provider was breaking down our tests into these very small groups and there were some things that they were doing that just made it so there were some things that we just never even saw because certain tests would never run together and there's a couple of other different things they were doing but in any case when we put everything together and we randomized our test suite we had a really big problem. A colleague of mine and I spent the next eight to 10 months pretty much just fixing tests. And then we were finally able to switch to a new provider. So today what I'm gonna share with you is a workflow for troubleshooting these types of failures that came out of this process, right? I fixed thousands of tests and so when something fails I usually already know what probably is happening. And so I'm gonna give you this map to fix your test failures so that you can go back and you can do it yourself. So, all right, master fails. A test fails in master, woohoo. We're all really excited, right? This is the best day ever because whenever master fails you have something to do. So what do we do though, really, to debug this? First thing we wanna do is we wanna run that test by itself. Does it fail? If it fails, then okay, does it fail every time? This is the best possible solution. We have a reproducible failure. We have something that fails every time and is gonna be really easy to fix because I can just run it on my local machine and debug it until I can figure out what's going on. This is probably one of the number one challenges in solving test failures when you have an intermittent test failure is you can't reproduce it sometimes. So, how do we debug reproducible failures? I'm just gonna touch on a couple of common failures that occur but I'm not gonna linger on this because this is something that is generally relatively easy to fix. So, a couple of things. Stale branches, you might have branches in, I'm sorry, you might have someone that is working on a branch and they don't back merge for a week and then they come back from vacation and they're like, oh, I got signed off on my PR, boom, merge and all of a sudden master fails. But what happens is in that last week maybe a logical change was made that's incompatible with their branch and now the tests are failing. Business dates and times. You might have logic that depends on the day and time and sometimes you can get to a weekend or a holiday or maybe the next year and all of a sudden your test suite is failing. Mock time and system time. If you're using a gem like TimeCop or TimeOrp or others to modify the Ruby system time it's important to remember that if you are relying on something, if the code that you're testing is relying on some external process which is also doing something with date and time all of the external processes from your Ruby process are not gonna share the mocked Ruby time. Missing preconditions, this is another common one. If you're distributing tests across a bunch of different containers and you have a test that where the test code or the production code is not requiring something that it needs or maybe it's not doing the data setup that it needs and it's been relying on some other tests to do that for it you can run into situations where when that test is run on some node in your CI environment it fails because the other test is no longer running for it. And then finally, of course real bugs make sure when you're fixing a test you're not just putting a bandaid over it where you don't understand what the problem is and so you just put a bandaid on it to fix it make sure you understand the root cause and that you're not just putting a bandaid over legitimate defects. All right, so let's go back. Let's say we ran the test by itself and it doesn't fail every time, it doesn't fail or maybe it doesn't fail every time. The next thing we wanna do is we wanna take the group of tests that ran on the CI container that failed and we wanna take the seed that was used to randomize the test and run it. Again, so what is a test group? A test group is a subset of tests from our test suite which run on a specific node in your paralyzed build. So here are some examples. If you're just using the RSpec command directly this means you don't have a test group, you're just running all of the tests. Maybe you have a list of files that are being piped into the RSpec command, this is your test group or maybe you're using a tag or some other command line option of RSpec to specify your test group. Test seed is a value, it's an integer in RSpec that you can pass into RSpec and what it'll do is it'll reproduce exactly the order that the tests were run in in your CI environment. So here we have output from RSpec, the output's truncated and so at the end of the RSpec output when you run your test you'll see this line where it says randomized with seed. This is your test seed, so you're gonna take that. And what we're gonna do is we're gonna take our group of files and we're gonna take our test seed and we're going to give that to the RSpec command. And so you can see here we have our test groups, couple of different examples, we're using the seed option to pass in the test seed and I also like to use the fail fast flag because when you're running a large group of tests that take a long time to run and something fails in the middle it's really nice to have RSpec just automatically stop when it hits that failure. Okay, so we rerun our test with our group and our seed. What do we do next? All right, so if it fails the next thing we wanna do is we wanna bisect that failure. In RSpec 3.3, I believe, they added a feature that will automatically do this for you. So when we talk about bisecting tests what we're saying is we're gonna take the full group of tests that ran with this failing tests and we're gonna cut them in half and cut them in half and in half again until we get to the minimal command required to reproduce the failure. So ideally you end up with one test that's passing and then your failing test and when those two tests run together in a particular order you'll see that other test fail. So here we have, this is essentially the same command as we ran before. The only difference here is that instead of the fail fast flag we're replacing it with the bisect flag and this is gonna take a long time if your test suite, if the group of tests that you're running take a long time to run but it will reduce that set to a minimal command. All right, so now we've bisected our test failure. If our bisect succeeds then we know that we now have a test failure a test failure that is of the classification test pollution. What is test pollution? Test pollution is when you have one or more tests where the side effects of those tests are causing another test to fail. And so how do we debug that? There are a set of very common failures that we see which usually these can be attributed to and so I'm gonna talk through some of those. So data pollution is a really common type of failure that can occur in a test suite and this is when you have one test that is let's say creating a record outside of the usual test transaction. It's persisting something to the database or maybe it's persisting something to Redis or some other store and it's not resetting it afterwards. So I'm a big proponent of isolation testing and making sure that your tests are cleaning up after themselves but what we've found and what we practice now at Avant is we practice defensive testing. So the thing, what that means is it means that your tests shouldn't expect pristine conditions. So they should clean up after themselves as much as possible but they should not expect a clean slate. And let me give you some examples. So these are things that put really hard constraints on your whole test suite because of a single test. So you might have a test like this where it performs some action that has a side effect that creates a user and after that action is performed it sets an expectation for the user count to be one. What does this mean? This means now that no other tests in our test suite none of our test support code, none of our fixtures that we have can create a user without breaking this test. That's a really hard constraint to put on an entire test suite for the sake of one test. So what we can do is we can rewrite this test instead to expect change so that we're not putting that global constraint on our test and our test will still pass even in the presence of other records. Don't similarly don't expect global scopes to only return the records you created in your test. Here we have an example where we're calling an active scope on a user model and we're saying expect the results of that to match this array. And what we do is when we say match array we say all right expect the results of the scope to have exactly these users in it. But what if another active user is created by another test or by a fixture? This will fail. What we prefer to do instead and the way we can solve this is we can instead expect inclusion and exclusion. So this allows us to set the assertions that we wanna, sorry, it allows us to assert the things that we want to assert in our test without having this reliance on a super clean database when it runs. Another common failure that we see is caching. Caching is really hard. It's a very hard problem in computer science. And so one of the things here that's important to do is that if you have a test that's mutating global state or a global singleton in a breaking way, oh wait, this is gonna break other tests, ideally you wanna reset that cache after that test mutates it. Better yet, if you can, use stubs or mocks to avoid modifying the cache at all. Now if you're testing a singleton, what this means is if you are implementing singleton behavior using class methods explicitly and you don't have a singleton instance, then it's gonna be really hard not to mutate your global cache. So whenever we're implementing singleton behavior, what we do instead is we implement it using a singleton instance and then you can see here, the top example here is testing a singleton class directly. So we're adding a key to our singleton class and then we're checking that the set of keys that it's storing contains that value. Now what I prefer to do when I'm testing a singleton so that I don't mutate the global state is create a separate instance of it for the purpose of testing the singleton's behavior and perform my test on that instance. That way I'm not mutating global state. You can still test that your class level methods are properly delegating to the singleton instance but you don't wanna mutate it directly. Mutated constants. Sometimes you see this, someone overwrites a constant in their test because maybe there's a timeout value that they wanna reduce so that their test runs faster or maybe there's a feature flag that depends on a constant that has a data sign to it that turns that feature on or off. You don't wanna modify this constant directly because that value could change in the future. Even if you reset it back, maybe that value gets updated in the future and it's no longer the same thing, you're gonna be resetting it back to the wrong value. So instead, learn the tools that your testing framework gives you. Our spec mocks has a very comprehensive, it's a very comprehensive mocking framework. So you can use things like stub const to replace a constant value with something else or you can allow a class method to be received and return a stubbed value. These are things you can do to avoid constant mutation and what it's gonna do is at the end of your test when it gets torn down, it's gonna replace it with the original value. Don't rewrite class, don't rewrite production class implementations. This is something that I've seen before. I don't know what else to say. You know, it's not a good practice. It's much better to just create an instance of the thing that you wanna modify and then put a stub on it. You don't wanna modify the implementation of your classes because what happens is now you've modified the implementation of that class and that's gonna carry through to any test that run after this one. Don't mutate, sorry, test constants is another interesting one. Sometimes people do this, they think oh, I need this variable in my test so that I can assert on it in a bunch of different places and so they'll define a constant in their test and assign a value to it and then they'll use it in their assertion and what happens is three months later, somebody comes along and they copy the production class because it does something close to what they wanna do. They modify it a little bit. They copy the original test. They modify it a little bit to reflect the new behavior but they don't update this constant name and what happens is now you have two constants that are defined in your tests and people think that oh, well, this is a scope to my describe block or a scope to my context block but your describe block is not a namespace, is not a module in Ruby and so what happens when you declare a constant here, it is gonna go up to the global namespace and so when these two tests run together, whichever one gets loaded second is gonna win and so the first test that runs is gonna fail and so it's preferable to use let variables or use local variables in your tests so that you're not mutating global state and of course, always look out for real bugs especially as you start getting into intermittent failures, things can get a lot more challenging and a lot of times it can come down to a production issue. All right, so let's go back. We re-ran our tests with our group and seed. Let's say it doesn't fail or maybe we go and we try to bisect it and the bisect is unable to come to a minimal command to reproduce or can't reproduce it at all. What do we do next? So the next thing we do and this may sound really stupid and easy because it is stupid and easy but run your test in a loop. This is an example. Take your test inside of your outer level describe block, put 100 times do block and then run that test by itself. A lot of intermittent failures can be reproduced this way. When I do this, I like to use the fail fast flag again because it'll stop execution immediately when a failure occurs and I can start taking a look at it. All right, so we run our test in a loop. Does it fail sometimes? If it does, we know we have a non-deterministic failure. What that means is we have a test that is either the test itself is doing something that's non-deterministic and making an assertion on that behavior as if it is deterministic or you have production code that is non-deterministic and the test is asserting that it is. So non-deterministic failures are failures that occur seemingly randomly in our test suite for no apparent reason. And so I'm gonna give you a couple of common types of failures that we see that fall into this bucket. One of the most common failures that I've seen that this happens with is unordered queries. People assume that when they make a database query, it's gonna give them back the results in the order that they created the records. But this is simply not true depending on what type of database you're using. If you're using Postgres, which a lot of Rails users do, if you look at the Postgres documentation, it explicitly states that if you do not specify the ordering of a query, it is going to return the results in the order that they are found. And usually that's gonna be the order they're created in a test, but not always. And so what we wanna do here, there's an example on the top. Let's say we have a scope that's unordered that we're testing the results of. If we say expect the results to equal some set of records, what that's gonna, what our spec is gonna do is gonna say, okay, oh, your results, they have the right records in them. But what it's also checking for when we check for a quality on an array is it's checking that the order of the elements in that array are exactly the same. So when we're testing unordered queries, what we wanna do instead is we wanna check that the results set contains exactly the records that we expect or that it matches an array. But what these two matchers do is they perform that expectation without a dependency on the order that the results are in. Frozen time is a similar issue that's pretty interesting. So if you, again, use time cop or some other gem and you're freezing time, what happens is all of the records that are created are gonna have the same created at. And so if you perform a query that is ordered and it's ordered by the created at time, the sort order is gonna be non-deterministic because the creation time for all of them is gonna be exactly the same. So we prefer time cop travel over time cop freeze. When you say time cop dot travel, what it does is it travels to the time you give it and then time continues to roll forward. So if you create a set of records when you're traveling through time, they're going to have different created at timestamps. But if you're freezing time, they're not. Only freeze time when precise time is needed. This is almost never in my experience. All right, another common case that we see is randomized test data. If you're using the Faker gem or maybe you have a set of test data that you are sampling for the purpose of randomizing your test data, you can end up with unexpected failures. Maybe it's returning names in a format that your code doesn't handle properly or maybe the test code is asserting the results incorrectly. Maybe there's phone numbers that don't match validations in your production code or zip codes that you don't support or maybe you're querying from, or sorry, sampling from all of the United States but you only support the continental United States. Sometimes you're gonna have a failure. One of the things you can do to help with this is if you are writing tests that rely on faked data like this is to output that data with your test errors. So when the test fails, you have more context that you can use to figure out what's going wrong. All right, so we go back and what if we ran our test in a loop and that test still doesn't fail. This is what we call unreproducible failures and I put quotation marks around it because you can't reproduce it but it's breaking CI. So obviously it's reproducible somewhere. These are the most difficult ones sometimes to figure out and so we'll talk about a few common cases here. So data in time is a really common issue. Data in time is really hard to deal with especially when you're dealing with time zones. So some of the things you wanna look for in your CI builds that are failing is do your tests only fail on weekends or holidays? Do they fail at certain times of day? Maybe you have some code that relies on business time or business hours. Time cop your tests to the date and time when CI ran and see if it fails. We have a time cop RSpec gem that we open sourced. I would say go check it out. It makes it really easy to work with time cop in RSpec using RSpec metadata and we also have a cool little feature where you can set an NVAR and it'll time cop all of your tests and we use that for running time travel builds so that we can catch failures in advance. So another common failure here is UTC and local date time differences. A lot of people don't know this. Date.today uses your system time zone. Date.current, which is part of active support uses your application time zone. And so what happens? Here's an example. This is available in that repository that I shared earlier. On the top line, this is mimicking a system like a continuous integration system where the system time zone is set to UTC and our application time zone is set to central time. And what you'll see is if you run this test, this test is going to fail because it's asserting date.current is equal to date.today. And what happens is when you are for certain times of day when the UTC date and the local date don't match your test is going to fail. And a lot of times this can be a production bug. SQL date comparisons, this is another really interesting one and this is pretty much always a production bug. So it's really important to know that when you are doing SQL date comparisons, let's say you're looking for records after a certain date or before a certain date, when you pass a SQL string to active record, what it does is it takes your value that you give it and because it's injecting it into a string, it doesn't know what column you're comparing against. And so it can't infer the type of that column. And so what it does is it's going to coerce the value that you give it to the default database string that it would normally generate for it. So if this is a value that has time, like a date time object or a time object or a time with zone, what it does is it coerces it to a date with time string without zone in UTC. And so if you are passing a time object into your query and you're querying against a date column where the date columns are stored in as local dates, there are going to be certain times of day where the UTC date does not match your local date and Postgres or whatever other database you're using is not gonna know to coerce it to the current local date. Timeout's an asynchronous JavaScript. This is another really common source of errors. I know a lot of people hate doing acceptance or integration testing, whatever you wanna call it, browser testing. So when you have a failure in a browser test, it's important to keep in mind that your performance in your continuous integration environment is usually worse than your local machine. So you might never reproduce this failure locally and if it happens, if you have a timeout occasionally in CI and you can never reproduce it locally, oftentimes you'll find it's because of differences in performance. Also, performance can vary widely depending on what tests run before a particular test or how your application is set up. Maybe a test runs before it sometimes that renders some of the views that a page needs in order to finish rendering. So sometimes it'll take less time to render. When you run into these types of situations, you want to increase your timeouts for CI as needed. You don't want to have a, it's most important for you not to have failures in CI. You don't wanna use your browser test for performance testing. If you're setting low timeout values for your browser tests because you say, oh, well this page needs to load in three seconds, you should not do that in CI. You should always be testing performance against production or production-like environment. CI is not a production-like environment. So also if you have JavaScript that's executing, sometimes we'll see failures that happen because your test tries to interact with elements on the page before the page is fully loaded. So I added a feature to the Site Prism gem. If you're not familiar with it, the Site Prism gem is an implementation of the page object pattern. And if you're not familiar with that, I would just recommend Googling it so that you can learn more about it because I don't have time to talk about it right now. But what the load validation functionality does is it allows you to specify a set of preconditions that must be fulfilled before the block that you're executing in your test is actually executed. So you can say, wait for my spinner to disappear from the page or wait for this form to appear. And then when those preconditions are met, it will finish, it will continue to test. And if those preconditions aren't met, it will give you back a reasonable error. So if you still can't reproduce it, what do you do then? You need to start digging. You need to start looking for environmental differences. Compare your CI configuration to your local environment. Are there environment variables that are set in one place but not in the other? Are there differences in how your tests are being executed? Maybe you're executing the RSpec or MiniTest or whatever other commands differently than you do it on your local machine. Try what you're doing in CI and see if it fails for you locally. Maybe there's an issue with how the database is being set up in CI. There might be seeds that are missing or present that you don't have on your local machine. Or maybe someone committed a migration but in CI, a lot of times you might load from a schema or a structure file because it's faster than running migrations. And so if someone forgets to commit a migration to the schema or structure file, you might have some failures. Some other environmental differences that you might run into is library versions or inconsistencies in how specific libraries behave. There's a lot of differences between OS 10 and Linux commands where they don't always behave the same. If you can, use Docker for consistency. If you're using Docker in CI and you're using Docker on your local machine, you know that you are running your tests in the same virtual environment. And so that can ensure consistency. If you still don't know what's happening, some other things you can do is SSH into your CI environment. If you can't do this, try to find a CI environment that allows you to do it because sometimes this is the only way to continue. I did this, I actually was doing this on Friday. Use common sense. Think about the failure that's happening. Look at the stack traces you're getting. Can you come up with a hypothesis about what is possibly causing it? There was an interesting talk earlier today about using the scientific method when you're debugging. Go listen to that talk and think about how you can apply that to your test suite. If you think that there's a third party gem that might be causing this problem, go look at the repository. See if other people have had this issue. Maybe there have been bug fixes that you don't have in your application yet and all you need to do is upgrade your gem. And if it's not there, maybe you should file a ticket and contribute back to the open source community if you do find a defect. Learn to use pry and buy bug. These are really, really useful tools that can assist you in debugging your software. And finally, incrementally narrow the scope of the defect. If you're just throwing darts against the wall and you're not tracking what you're doing and doing it systematically, there is a lower likelihood of finding the root cause of the defect. You wanna be continually narrowing the scope so that you're looking at finer and finer grained possibilities of what could be causing the issue. Rule things out. Make sure you know your test support code in and out. This is important. Sometimes we have, so we have a test suite that has over 30,000 tests in it and so we have a lot of test support code. And sometimes there can be conflicts in what the test support code is doing that sometimes can cause issues with our tests. So make sure you know what your tests are doing. Look at failure trends over time. If this test is failing for a long period of time, maybe you can find, maybe there's other tests that are failing in a similar way and you can find correlations between them or you can look at the builds that have failed with this failure and look for correlations between them. You can add logging. If the test is still failing, it doesn't hurt to add some logging to that test to give you more information about it so that then the next time it fails, you now have more context. Thank you all for coming. I'm really happy that I was able to talk to you today. Just some quick takeaways. So keep your builds green to avoid sadness. It's really disheartening when you come in on Monday and the build is failing and it's not just one thing, it's 10 things that are failing and you don't even know where to start. Remember that tests are code two. They do exactly what you tell them to do and nothing more. Set realistic goals for yourselves. If you have a test week currently that suffers from this problem, you have to keep in mind that it's not necessarily gonna change overnight. It takes time to root these problems out and solve them. And finally, celebrate success. Set milestones for yourself and celebrate them. This is a picture of cupcakes that our software engineering manager got us when we hit a 95% pass rate on our build for a week straight. That was a really big accomplishment. So again, please get in touch with me. I'm gonna be posting a couple of things on Twitter for you and I also have a larger version of the workflow that was in my slide deck that I'm gonna share. Thank you for your time.