 So, a little bit of the day, next talk is, yeah, a little bit of why artists sometimes suck if you want to get your comments in. So I hope I don't step on too many people's toes. I apologize in advance. I want to give you a little bit of food for thought. If you don't agree with any of this, just throw the food at me. It's not raw tomatoes. So let's start. Well, with Congress, tests, tests are good. Of course, we want to test our code. And more tests, of course, is better, no more better. Many tests run all the time, all the time we, all the time people happy. James was running the tests for a recommendation. That's our idea, what we want to have, so that we're sure that we'll be right somewhere with her. So you don't choose the jet revisions that everything keeps working. That's the idea. The reality sometimes looks a bit different. For example, this is what happens when you commit a change together. Then Jenkins takes the New Cassian in here, because it's only all you see is that there's pre-attempts, it took four attempts to get the windows built, actually working, actually green. So we want to do these tests on multiple different platforms. The next window is smack. And whenever one of the tests fails, then Jenkins says, this is not good. But most of the time it's not your fault. It's the fault of one of these tests that happened to break for obvious reason. So what you do is you reschedule the Jenkins build then to see if it goes green. And then that's proved you didn't change anything, just leave it at the test, and now they will be so your code wasn't the fault, it was the fault of one of the tests that failed. And sometimes it takes like four iterations at that time to start to think about all the machine power wasted on this. So your carbon footprint for a guaranteed commit starts to go up, and you think it could just involve taking an airplane to Geronimo's belt instead of doing something like that. The other way that this shows up is that many people on the mail, so this is on the mail, sorry, started to have questions like, I'm doing my first build here, it fails, what can I do? I used to do builds all the time on the work and now they start to fail with dubious places and tests, and I don't understand what the issue is, how can I fix whatever I'm doing wrong. So this is a typical type of fashion that newcomers or even old hands have, like this, just today on the magazine, who claimed that something that used to work on the machine for ages now starts to fail in TV, so let's tests that are flaking and then cause trouble for people. Natural instinct balance, it doesn't work, it raves, it's software that can fix it. So let's do that. When one of these tests fails, what you typically get is a modernized message that tells you exactly where the failure is and what would need to change in the code to make it work again, or you get some cryptic, long output, and if multiple tests fail their output, you can get mixed if you don't make mine as well. So this is an example of something that failed, and you see from the simple human test, the idea there is always that you add some sort of code to the test file and something shortly of an expected outcome, which is 1 here, and then it shows the actual outcome, which is 0, and then it shows you where the test code that happened, so there's a filters test, CXX file, a line, 145, there's such a search check whether something is as expected, and that obviously failed. It gives you any one is 0 or can it be, but you can look into the test code of course to see what is actually checking there. So let's go look into the test code here, and then everything should be obvious, and we should be able to change whatever changes needed, we got that test fixed, everything will exist, the search and check that something should be expected but actually it's whatever the result is, and the result is obvious up here, either test passed or test failed, so these are enough values, 0 or 1, so apparently we didn't have a 1 there, we don't have a pass there, we have a fail there, that's good to know, but why did it fail? So looking further up, the end result is taken from the BU result, BU result says, oh that's up there, and if we are in an export case, I'll be in an export case, I have no idea, I just see the output there, so in one case we could do a load, in the other case we could do a save, and then the return value is 0 here, which says whether it worked, but apparently we did either of these, and it doesn't work, but that's all we know, little to no to actually fix this and make tests green again and make everybody else happy again, so this is not an ideal way, obviously, to write a test, I mean it's an ideal way to write protection code, because we know that things work there, and if they don't, we have a scenario that we can debug for our customer who pays us for that, but if our own tests fail, we won't be able to be breathing, I mean we get angry and we want to debug them out because I want my commit actually to pass, not this test, and my commit has to pass because of this broken test that somebody once wrote. Talk titles, of course the title of this talk was Taken Tests and Mounted, it was an album, in the 70s by Goinia, Taken Tied and Mounted, from which I ripped this title, ripped it from an even older film, and as it turns out, we know we used some sort of cards with the instructions or ideas to unbreak, to help your creativity along, and it's quite funny that when you look at these cards today, that they devised in the 70s, many of them, some of them are about making music, but many of them could also be used for writing better, writing better software tests, so we could look at these and decide what to do, like shut the door and it doesn't come from the outside, or you can continue, you can continue with the label to just carry on with this, but we want to use that to do something else, despite the title. So what I want to thank you for making these slides, there's a tool for these tests of things you could do when these tests do fail to have an easier way, so step zero on the first iteration succeeds, because for one, I mean, for one, you might accidentally don't execute your test to the unit test for example, which just happened, and then everything goes green so we actually will test everything. One more important for this talk here is that when you see that test fail, you see the output from the test that failed, and if you then think about, is this output any useful, or will it be useful for somebody else who doesn't see the test fail, we got a chance to write the test in the way of the output, a failed output, and not just a passed mistake. So there's one case or root cause and then in this case, there's some test in code, that tests some functionality, and then again, so somebody came up on the main just to drop it. Expected value is one, it gives me a zero, it's the limit, it's some building point computation, so it also gives a delta and some range of values that will be allowed, there's always rounding errors, but they're quite far off, so zero is quite different from one. The next machine, and I'll tell you a little bit, we need to build the CPU and these old instructions at a different and old floating point and you could enable the new mode of ESC right now. So if once you had actually backed this, you found out that zero was the one difference and the test result broke down to some key inf function calc and that was fair to the two values and returned the one result for not the result that was expected by the test, but the test itself and what looked at supported results into feet this function with these values was the rounding error that we would allow because there's no such thing for calc so for calc because we're going to compute that and see our mind machine with this floating point environment that I got this value so I'll snap that into the test code and if there's a different value and just as well in acceptable value or it might not be in which case you would need to stop building calc for floating to the Linux until we can get when your CPU starts to use. So if you're just looking at and I try out it gives this result, I snap that into the test document then you're you didn't break down the problem deep enough to actually test against something that will be specified what the expected outcome actually is so you hope that the outcome that we'll see is the right one and then could be fired in your test code but it doesn't help if somebody else has a machine it also would have a better result but quite different from yours and we also see that of course with many tests they use fonts so it depends on who has what fonts and how small with your system so text metrics and key check copies they differ quite pro-graphically on those books have a principle that is what dry don't repeat yourself so whenever you have something that you write twice in your code abstract from that in your fashion values into so that you don't write the same code twice which of course is the advice that is good for normal code for my test code as we've seen before there will be functions where we get the full return value which just gave us a good it gave us no information what exactly is paid so if you write that test code so do every individual step spell it out individually check all the intermittent values that you get to see where exactly is the value coming so it doesn't help if in the end you see all that didn't work out somewhere in the middle some wrong term was taken so for test code in a way the ideal piece is that instead of scaling through a dry level you don't find your way back once to pay for example you can give some message in the paid case which is a string that tells you something about the context but it's sometimes we read that that's kind of a red flag that uses some abstraction somewhere so that multiple paths to your test end up at the same search check and you need to see multiple paths there which part it was that ran into the failure so of course this is there's no solitude here so spreading the code too far indication that your test is just too much of the same over and over again but please do keep this in mind as there is too extremes and for test code prefer to go more to the wet side of things than to the ideal dry code that you want for production code or to go on a test for this code test code of course source of pain users of random even if you would always use the same seed so that the value is not always so why don't you then compute a handful of values that you then explicitly use in your test code instead of using some like some form with the random value that you see with the deterministic value of these and there's of course other sources of knowledge that isn't like races which is harder to avoid than avoiding the call to random function but yeah we use to have quite some tests at least that you know tested and tested for the different properties of all these you know objects that did use random the big random properties to change them to random values and then sometimes they fade to the proper random value supported by this one random big property so that's a fun thing to do by similar to the first to the third is that also the environment in which you run your tests should be deterministic and should always be in the same stage and for our test that mostly means that these tests often open documents the office code is so that when you open the document you're putting the log file next to it that says this file is locked by such and such person so if somebody else tries to open it then you will get an error of message box that you can't open it because somebody else is changing it so what we do for most of the tests is either open them in a real only mode so that the log file doesn't get written or move these test documents over to some safe place in the work here because every test has some place in the work that gets cleared before the test is run and afterwards the test succeeded so there's a natural place to place your document and the log file is there and the test is done the second time the log file is removed again so this was one of the why tests I ran into the other day probably unbelievable again what happened there is that I ran the test the log file was cleared and then saw that I had this type of make check I also wanted to make that screenshot make check screenshot thing so I cancelled the build the log file was still there and got off the virtual network which meant that for some reason my my host man as it probably seems had changed from A to B I then restarted the test went to bed came back to a run test that didn't do anything because this is one of the endless mode tests that when they run to initial say something they just can't display because they have no screen I think and they're just the same and the solution of course was to just not get this test this is the log file written into a source tree where it wasn't deleted properly but to first copy the test document over to this workplace for a test and we have functionality for that for all different kind of tests this was a python written UI test but there's the same way for all tests to use these functionalities reached the top of the hill come to the end of the talk for time do subscribe to this one mailing list we have a mailing list there are all of them there's changes that you submit there the regular to the boss we have this master page as well for you we also have the mailing list where you can get all the favors of all these different platforms and there's like I wouldn't want to lie but easily there's 20 minutes for a day there's not another you get a pointer into the failed log and you can see exactly where it failed because the test didn't tell you much but if you are able to fix one of these ever-recording failures then you'll be our hero because that will be one step down from these repeat a build four times until the garage commits or until Jain said that your garage commits is green it's failed in four different ways the test sometimes does so please go ahead and do that under that thank you