 or how do you identify yourself, OK? So you're interested so that you have a team that, OK. All right. Which of you are working with Java, OK? My examples are Java, but you have similar tools for other technologies, usually at least. And the tools that I'm going to show they're not really the point. The point is having a tool, not the specific tool. So I think we're fine. Just out of curiosity, what other technologies do you work with other than Java? Groovy. Groovy? Groovy. Groovy? Yeah. I'm going to get a picture of something. Well, in certain communities, apparently the way of living is that you always build your own tools. I've understood that C and C++ generally go in that direction after some fundamental tools like the compiler is in place. And some other communities like Java, perhaps the usual way of working is you take the latest popular build tool, but you never touch it. You just take what comes out of the box, but you don't really extend the tools much. Yeah. The point is the purpose, not the way you're getting there. I guess it's 30 pass, so we should get started. I'll first tell you a bit about myself, mostly about my background. So I've been earning a living doing software stuff for more than a decade. I've been writing code for longer than that. And even though for the past maybe seven years, eight years, I've been mostly sort of working around methodology things, training methods. I'm also a certified scrum trainer, but don't cut that against me. I've been doing a lot of non-programming work recently, but I also do still frequently work with teams, usually not full-time for a long time. I might be there for two weeks full-time, but then I might be away for a year. So it's somewhat different than working full-time and being mentally committed to the team. But my background is in programming and reactor as a company where I work. We do software delivery. Most of this talk is actually based on the latter, both the effective unit testing, which just came out, chapter nine, which is called speeding up test execution. Now, the title is a bit different than this talk. We're going to talk about both speeding up your test, because that's, especially in Java, a lot about your build speed has to do with how many tests are you running, what are the tests doing, how are you running those tests, and so on. So it kind of tends to revolve around test execution speed in certain technology environments. In others, it might be compilation or things like that. So if you want to learn more, you can also get the book and get a bit more elaborate description of some of these things. The book is also a collection of test smells, and I'll be talking more about that tomorrow. But we will look at a few test smells that relate to performance in terms of test execution speed. So kind of like a two-pronged approach. Here's the session outline. We'll start by first making sure that we all sort of speak or we might be speaking with the same words, but we also understand what the purpose is that I'm talking about when I talk about speeding up a build and why it might be important. Then we'll look at, quickly, simple ways of profiling your build. And once we sort of figure out where the problem is, where the hotspots are, or the bottlenecks, then we talk about optimizing test code and tweaking the infrastructure, which is more about how you're running test than what the actual tests are doing or what the actual build is doing. Good. Please ask questions on the fly. We should have time for questions. And just give me a shout or raise your hand. If I don't see your hand, then give me a shout. Let's start with the impact of build time. So I've concluded that there's no way around it. If the build is slow, it's a shitty situation. You can try to ignore it, but it will come back to you. So there's two ways to sort of try and deal with it. One is to just, well, wait for the build to finish. It's going to take 20 minutes, so I'll get coffee. I come back, I read a paper, look at the build. It's still running. OK, I'll go get another coffee. I'll check the websites, what's on. And eventually, you'll get a result. Now, that's regardless of the build result, whether it passes everything seems to be OK or it fails. There's a compilation error, there's a test failure, whatever the reason, whatever the result. In any case, all of that time has been wasted. Because you didn't actually learn anything during that time, you didn't produce anything during that time. It's kind of complete waste. So naturally, if the build is going to take 20 minutes, you're going to move on. You might trigger the build, trigger it off, and then start working on the next thing. And most of the time, that might be OK, because the odds are if you're quality aware, you have decent engineering practices, most of the time, you don't make mistakes. But then every now and then, 20 minutes later, you'll get that nasty contact switch. Something failed, and it's already been 20 minutes. So you can't quite remember what did I do, what did I change, might have to revert your workspace back to that situation, and start debugging. So even that kind of approach does has its downsides. Either option has something very wrong with it. Obviously, the best solution would be to get rid of the problem altogether, having a less slow build. So I'd like you to grab a piece of paper, any piece of paper if you have, because I'd like you to make an explicit guess about some of the things. Any piece of paper, you don't have to pass it on, it's just for yourself, so that you remember the numbers that I'm going to ask you to estimate. So here's the situation. There's a programmer, he's working on some code, he makes a mistake, and then some time later, he realizes the mistake and fixes it. Now the question is, how much in cash does fixing or making and fixing the error, how much does it cost if the delay is one minute? I make the mistake, 60 seconds later, I see, ah, I made a mistake, let me fix it. What's the cost? Write it down. Some kind of currency. For me, it would be euros, but for you, it's probably something else. How about five minutes later? What if the delay is five minutes rather than one minute? How about one day? You go home, you come back to work next day, and then roughly eight working hours after you made the mistake, you might have slept a night in between, you find the problem and you fix it. How much does it cost then? Do you have all three numbers so far? Final question, what about one week? If it takes a whole week before you notice the problem and fix it, how much does it cost then? So I'm curious, what kind of numbers did you get for the first situation? It cannot be zero. It cannot be zero. You made a mistake and you had to do rework, so it cannot be zero. It can be zero point something, but it cannot be zero. Near zero, yeah. What's the highest cost that you estimated? But if you fix it within one minute. Oh, OK. So what's the highest estimate for the one minute scenario? One euro. So we have near zero, was it one euro? $10, something around that area maybe. Too much details. That's too much. But this is an imaginary situation, of course. Well, that's an interesting perspective. I think there's a point. And let's go a bit further and see if that changes the way we talk about this. So what kind of numbers did you get for this, the five minute option? 10 euros, so 10 times your previous. What about others? Maybe three times the previous, so it's not quite linear, but roughly linear to the first one. So why isn't it the same cost? It's the same mistake you made. It's likely the same fix that you apply. Why is the cost this bigger? There's a bigger context change, yeah? Yeah, so we start having all kinds of possible consequences of making this mistake and letting it leak outside of our fingers, into the maybe version control, maybe to other people's workspace, and so forth. So just out of curiosity, what's the number for one day? 300? 100. 100 times the same. 50 times. So another magnitude more. What about one week? 300 times? Yeah, we're starting to talk about thousands of dollars, euros, something. Well, this is a general bug. Bugs are, bug is a nickname for program or mistake, right? But maybe this helps you. So Google does a lot of software development, and they've done data mining on their actual version control system and bug ticketing systems and so forth. So there's a guy called Mark Streibach, who a few years back, he dug into this thing. And he tried to figure out what's the cost of their quality or lack of it. So he looked into the delay between introducing a bug and fixing it, or introducing a mistake and fixing it. And he came up with these estimates. So the range was from $5 to $5,000, depending on how long the delay was. And some of them were quicker. Some of them were slower fixes. But this was the statistical average or a mean. I'm not sure which was it. But there's a big impact in terms of cost. And you got fairly similar estimates, at least some of you who voiced them out loud. So clearly there's something going on. At the one week level, we could imagine that other people get involved. But already within minutes, there's some kind of a difference. And within hours, there's a significant difference. So I think this might not be as big an issue for one organization or the other. For Google, they calculated that it's roughly $160 million per year if they could crunch those delays near zero. They can't prevent mistakes being happening. But if they catch it within a nanosecond, the cost is practically zero. That was a surprising number for me. But then I started thinking about the delays. Realizing that, yes, it actually it's really hard to go back and start debugging. Because I need to re-establish the context. So I need to read a lot of code again and go back and forth. So the delay is really important. I think that's why we should be relentless in trying to fix the delays, shrink them, and give developers, give myself primarily, but also the people around me, as quick feedback on the software's health as possible. So I wanted to point out the monetary aspect, that there's a very immediate impact on the cost of development when there's a delay involved. And the duration of the build, it does impact the delay, because if the build takes 20 minutes, it might be that it's the very last thing in the build that signals the failure. If we're constantly observing the log, we might notice that, oh, some test failed. Let me scroll a bit up and see which test. So you might find ways of getting that information out early. But every now and then, it will be the very last thing that fails. So talking about profiling the build, there's this kind of dilemma. We're optimistic, and we think that we can shoot in the dark and guess right. We have intuitions about why this build is slow. But unless we're very lucky, or we actually look at it, we're not quite sure. We shouldn't be sure, especially because it takes so little to verify where the time is going and what the proportions actually are. Is it compilation? Is it packaging? Is it running tests? Is it running this kind of test, or that kind of test that is slow? And how slow it actually is. So when we're profiling a build, what we want to do is basically three things. Figure out what's the total delay. We want to figure out where does that delay come from? Where does the majority of the time go? And possibly in the majority, there might be the biggest potential. Although we might end up realizing that actually it's the compilation that takes the biggest time. And we've already looked at it. We can't really figure out a way to compress that time any further. So maybe the biggest potential is actually the second biggest chunk of the build time. But anyway, we need to figure out all of these three things somehow. And I'll show you a few simple tools for the Java environment, Java community tools. So the first one is built-in. So in Ant, there's a tool called Profile Logger. So I have a bunch of short demo scripts because I can't rely on the network. Some of the tests actually use a remote server and so forth. But I made these demo scripts that essentially, if they don't have a log file from a previous run that I could just display, they run the build with certain options and capture the output for later reference. So here, for example, we're running Ant with an option Logger, Org Apache, blah, blah, Profile Logger. And so if I actually execute this script, we can see roughly what it does. So it would run the build. Now we're skipping that because we have a cast result here. And then I've picked up a few interesting lines from the result. So when you enable this option, or this Profile Logger, in your build output, you get these reports, this build step or build task in Ant and terminology. It was started here, and it finished here, and it total duration for that was something. So just by looking at these numbers, you could see that, well, JUnitTask lasted 100 seconds. And then there's a target called TestExecution that lasted 100 seconds. Well, if you know the code base, you probably know that the TestExecution target just does the JUnit task. So they're actually the same thing. But clearly, you see that this is where the biggest majority is. And also, you might reason that that's where the biggest potential is for optimization. Another tool is the Ant PerformanceListener. So similarly, you just configure or pass a different parameter to Ant. This time, you have to use the Ant Country Library, which is an open source extension library for Ant. And you basically register a listener called NetSF Ant Country Blaba Ant PerformanceListener. And the difference is basically just in the aesthetics of the output. So I've run this on an approach that I called OOP 2013. But it basically reports the target level stuff and not the target and task level things mixed with each other. So this might be more clear to you, because you don't have to do so much deduction in your head. Oh, that's a task and that's a target. And actually, those two are, that's inside that. So you don't have to do that mental toggle that much. Also, if you actually look at the Ant Output for Ant Profile Logger, there's a lot of white space in between those reports. So this is actually cleaner to read. It's easier to read. For Maven, there's also a tool that I build because I realize that there doesn't seem to be a tool available that does similar things for Maven. So you can enable a build extension in Maven. So add this bunch of XML to your massive XML. And pretty much, then, you get a similar result. I'll give you a snapshot of that as well. So at the end of the build, once the build is about to complete, it basically spills out this unified report, saying, here's the project that I ran. So if you have a multi-module project, it actually lists all of the modules separately. And within each project or module, there are different phases in Maven terminology. And within each phase, you execute plugins or modules in Maven lingo. So here, for example, we have the whole module took roughly 70 seconds. And that's 100%. Out of that 100%, process resources phase was trivial. Compile phase was trivial. Test phase took 95% of the time. And the only thing we ran inside the test phase was the Surefire plugins test module executing tests, basically. So for a more complicated build, this would be, again, slightly easier because you can look at the percentages rather than deduce the importance of numbers yourself. Maven build utils. Oh, yeah, there it is. So these are some tools. And I'm fairly certain that most technology platforms have at least the ability to get that information out of the standard toolset. You might have to code a little or script a little, but it should be fairly easy. So let's focus on the two ways of making it fast once you know where to focus. So let's imagine the test code that we want to speed up. Why do you think tests might be slow? What kind of things you've observed yourself? Set up and tear down. So you're doing non-essential, useless, or let's say, unnecessary setup. Or OK, so if you can find a way to safely do things just once, obviously that would mean less statements to execute, less work for the CPU. Might be less work for the IEO, the disk, and so forth. Anything else? So third-party servers are notorious because usually there's a long, thin pipe. And then you have to push all those zeros and ones there and wait for them to come back. It might be really slow. Yeah, so you could also turn that around. If you can run them in parallel, that would speed up your build. But yeah, sometimes it's artificially sequenced. You could easily do things more in parallel. We'll talk more about that towards the end of the talk. These are some reasons. I've collected some under two categories. One is that the tests are doing too many things. So they're doing unnecessary things. And sometimes they're doing things that are useful, but they're slow. So for example, the remote server, it might be something that's essential to the behavior that you want to test. But actually accessing the server, that's just slow. So if you can find a way to do it faster, that would be better. If you can avoid it and still retain the meaning in your tests, that would also be better. So here's a few examples. Tests doing too much, setup that's not relevant, doing too much computation when you could basically use more evolved inputs to start with. Repeating the same thing, slow objects versus less slow objects. But then there's example of slow things. Some of these, we sort of covered some things we didn't, but I think you've seen this a lot. Tests that have thread sleep calls or whatever the API is in your language. How many has written one of these? I have. That's the band-aid. If you have something that deals with threads, the easiest way to write some kind of a test is to just spin it off and wait for a long time, long enough that it's fairly likely that whatever you spin off manages to complete before you make checks or assertions. But then there's the networking stuff, doing network access, databases, and file systems. These are the three things here are usually they go together. The reason you access a file system, sometimes it's because you're accessing a database that has the data on a disk. Sometimes it's really like a file system, file system access. Also networking usually relates to some kind of a database. Sometimes it's a web service or things like that. But a lot of the times that I've seen the network connection is actually for accessing a database. So I'd like to show you examples of a few of them. And there's I think an intellectual challenge there with one of them. So let's start with an example of a setup that does too much. So this is like a made-up example, a trivialized example from something I've seen a lot. And the dirty word here is spring framework. For some reason, spring tends to come with this. So there's a bunch of abstract classes because your components, the classes you're testing, they want dependencies. And the way you get those dependencies is through this dependency injection framework. And you define kind of a context so that the framework knows how to connect these objects so that everybody's dependencies are met. And usually it's an XML file. Sometimes it's an annotation or annotations on the objects. Luckily, that's starting to be more common. But in any case, doing this stuff takes a while. Reading the XML from somewhere, parsing it, or just running through your byte code, looking for the certain annotations and then wiring things up, it takes some time. Now, here's a test that extends from the abstract class. And it actually uses the context to get some bean. Supposedly, we need it for this test. But then there are other tests that don't really need spring. So they don't extend from it. Now, if we run these, we can see that word go there. This is roughly the overhead. It seems to be roughly the overhead of reading the spring context. This is the fast test that doesn't extend the base class, one millisecond roughly, and the other is almost a second. It's not quite true because there's a class loading cost that only incurs once. So if I'd run two of these, the second would actually run much faster. So let's do that as well. Let's run this too, just to see roughly what the difference is. So it's not actually that bad. But if you compare to this, that's eternally slower. So these things would add up. And if all of your tests deal with spring, just count the number of tests you have. If it's 1,000 tests, 10,000 tests, that's starting to add up to your build. So that's interesting. But then the problem with these inheritance-based setup utilities is that at some point, you have the need to do something with a slightly different common aspect of your framework world. Let's say it's locale. And you create a nice little utility base class for locale stuff. And of course, because you need the spring context stuff in general, so you extend that from the general spring test base. And you add whatever the locale specific stuff is. But now when you want to write a test for something that deals with locale, the easiest way to do that is by extending this class, even if you don't really need the spring stuff. And then you take that into your test, even though that's not strictly necessary. But because this is the easiest way to do it, people end up doing this. So pay attention to this. It's a fairly common smell. Another example, here's, again, an imaginary socket listener test. The idea I got from some code that I wrote roughly 10 years ago, it was awful. And I just found this on my hard drive and curiously looked into it. And I find all kinds of ugliness, including these thread sleet tests. So here's the idea is that you have a socket listener that's listening on some kind of a socket connection. And then you start the thread where the listener works. And then after a while, something should happen. But you're not quite sure when. So if we want to know what the listener is doing, we need to do a lot more work than just start a thread and wait for a second. So in this case, we have a test that starts and stops the listener. We know that 0 milliseconds isn't enough. We've concluded that well, if we wait for a whole second, that at least seems to be more than enough. So let's put it at 1 second. And then after stopping, perhaps the same thing, it might not stop immediately. It might still have some tails to finish. And what we'd really want to say is something like this. We wait for the threads to start up for a maximum of 1 second. But please give me back control. Give me continue my execution as soon as you can, please. And for that reason, we might want to use some kind of utilities, like maybe create some kind of an observable thread or expose synchronization objects so that you can talk to your socket listener and say, please notify me when this step has been done. And please notify me when this other step has been done. And maybe you can then pull on those events and then do the assertions. So again, just to point out what the difference is, we're pretty sure that sleeping for a second is pretty much exactly 1 second. Since we did two of them in the test, the whole test takes 2 seconds. And here's the same test logic, the same production code we're testing, but clearly we've said almost 2 seconds by returning control as soon as we can. Now that's going to be slightly different if you run this over and over again, but it's always around this area. So just to point out that this is code that you need to write. It's not complex code, but it is an extra effort. I think it's well worth it if you're doing anything with threads. Now the final of the three examples I'd like to use is networking. This is a situation where you have some kind of widget that shows stock tickers, like the latest stock value for Apple or Google or Facebook. And there are two components. One is the stock ticker, the widget. So that's what we're actually testing here. We create two kinds of instances of that. One we create for presenting the stock value of Apple stocks in this view using the real live source from somewhere in the internet. Maybe it's in the internet, but over the network anyway. And then we do the exact same thing, except we use a stubbed outsource, which over here it's actually a mock object. Then what do you think might be the difference? Big one? So let me point out the external remote server. It's not that remote. It's running on this laptop. So it's actually created here. Let me just show you quickly that the fake stock server only does one thing. Offer one API stock quote for a certain stock name, and it renders a small piece of XML back. It's run using the JETI web server. So it's an actual production quality web server that we're running in process over a fairly quick network connection. So it's not that bad. I'd like to hear your estimate on how long will these tests last. If I run these fetching over HTTP, the starting and stopping is not part of the test. It's done as part of the setup. So we're talking about just the test logic, firing off a request, getting it back, and pushing the response to a view of some kind. So what do you think might be the duration of making that very simple HTTP call? Any guesses? Probably we're talking about milliseconds, not seconds. But how many milliseconds? Sorry? Or the difference is 1 to 10 between these two types, which is the slower one? Yes. Obviously, the server will be slower. So let's see. We have three tests just to point out that there's the class loading cost that incurs just once per JVM. So fetching the stock quote over HTTP is more than a second. Fetching stock info using a stub or a mock object is 8 milliseconds. But actually, the second fetch over HTTP is just 2 milliseconds slower. Effectively, the same performance as using a stub. Interesting. So this kind of thing is why you should profile your test before you make conclusions. Because we all know that networking is slow. We all know that file access is slow. But it's not always the case. The fact that we use the mock object library here to create the stubbed-out source for the stock info, mock object libraries use reflection. Now reflection is also slow. So unless you're doing manually writing stubs, they're probably around the same performance, of course, depending on what the service is doing. If this was an actual service somewhere, they'd do a lot more processing than our little fake stock server here. But this is interesting. Networking isn't always that slow. So let's go back to the presentation. So we have these examples of tests doing too much, tests doing too slow things. Some of them might not be that slow. But these are all things to keep in mind in that demo we just did. So I'd like to spend the rest of the talk talking about how we run these tests. So first of all, when we start optimizing, we should again know where to focus on. So we should profile where the bottleneck is. Like trivializing this a bit, it could be CPU, the bottleneck, or it could be some kind of input output. If you're running a UNIX system, you have these tools like top. It's not really made for this purpose, but it's useful enough. If you trigger off your build and you run top, the CPU usage pretty much tells you if the CPU is bottleneck. Then it's at 100% or 200% if you're using two cores and so forth. So that's typically, well, CPU is the bottleneck. If it can reduce computations, that's an immediate improvement. You could use a look at the idle time report here, which suggests that the CPU is waiting. So it's at least not all the time the bottleneck. You could look at the system calls metric, which doesn't technically mean file access or IO, but most of the time, it correlates with the code doing a lot of IO. Most of the system calls tend to relate to networking or file access. So that's rarely at 100%, but if it's high double digits, it sounds like IO. So once you know this, let's say it's the CPU is the bottleneck. Some of the obvious ways are using more hardware, but how exactly? You could add CPUs. That might mean getting a build farm and distributing it even more massively than just within one computer's course. That's clearly one option. The simplest one might be to just make sure that you're using all of the course on your computer. So one of the demos I did was this sort of parallel test thing. So the difference between POM XML and POM parallel test XML, the only difference here between the two runs is I've explicitly said, please use multiple threads. In this case, exactly two. I just throw in two because I happen to have two cores on my laptop. And the difference in the execution times between parallel and serial execution, or let's say parallel and non-parallel, is almost linear to the number of cores. Yeah? Because the JVM can say, you, Mr. Core One, please run these stuff. So what you should probably do is start with something. And the number of cores available is a pretty good guess. But then start bumping it up and down and see which number gives you the best results. It's not necessarily something you can tell up front. So that might be very easy. If you're using a tool that makes it easy, there's no reason not to. And even if you're not using such a tool, it might be easy enough that it pays off. So we can use better hardware. On one project four years ago, our build took 24 minutes, and these were fairly high-powered laptops. So we figured out a way to give every developer their personal cloud computer. Amazon comes with these different types of instances. Some are called small and some are medium. Some are high. And then there's super high, extra high, brilliantly high, and so forth. And depending on how much money you want to spend, you get more computing power. So we were basically giving every developer their private Amazon instance. And this was an eight-core computer at the time. I can't remember the name of the instance back then. But eight cores, it cost less than 10 euros a day. So very minimal compared to the potential cost of 24-minute delay before you actually find out about a problem. The first time you make a mistake, you're covered. You paid for the Amazon instance. So we made a small change to our build script that if you exported an environment variable that points to an Amazon host name, it uploads your local workspace or synchronizes it with that server and then tells the server to run the same command that you just typed in and starts pushing back the console output. So previously, it took 24 minutes to run the build. And during those 24 minutes, you couldn't do anything else, not even browse the web, because the computer was so bogged down. It was like CPUs at 100% all the time. With the Amazon instance, you send off the build. The overhead was less than two seconds to synchronize the workspaces, the local workspace with whatever happens to be on the server. And the build time came down to 3 and 1 half minutes. From 24 to 3 and 1 half, just by using a higher-powered computer, probably also faster disk, but that wasn't really a big impact, at least on our laptops. So clearly paid off. I do have an example here, but it's not quite the same same setup. I basically ran the same test on two different types of Amazon computers and then figured out that there's no difference, there's no real difference. And I didn't want to shell out for the really, really high-powered computers. So here's a difference between the cloud computer and the laptop, both parallel and sequential. And it seems that actually the difference between my laptop and the cloud instance I was using, it's not actually that big. We're talking about maybe a bit more than 10% improvement. That's already a lot, but it's not eight times less. And actually, in this case, it seems that a much bigger improvement would be just to parallelize first. And even if when you're parallelized and then you push it to the cloud, actually on this instance, in this project with this code base, it's actually slower to run in the cloud. This has a lot to do with how many cores the server had. In this case, just one. So it wasn't parallelizing. It was faster than the local sequential thing, but only because they had a better CPU, not because they were actually parallelizing. So there's a lot you could do with the cloud. But you could also do more stuff with the hardware than most people don't think about. So for example, thinking about disk speeds, you probably know that you get hard drives with different spinning rates. Some are 5,000 plus runs per minute. Some disks are 7,200 or something like that. Some server hardware might have like 15,000 RPM disks. So that makes some difference. But you also have RAM disks or SSD, solid state drives. That might be a big difference. The gotcha there is you shouldn't just believe that SSDs are faster and run to the store buying them, because modern operating systems actually do a lot of memory mapping already. So unless you do a lot of I.O. on the disk, the SSD drive might not help that much. You could also look at the stuff that you usually ignore when you're buying a hard drive, if you're even buying a hard drive nowadays. The size of the right cache, 1 megabyte, 2 megabytes, 4 megabytes, can make a big difference if you're doing a lot of disk access. So the caches are actually important in this level. You could also look at better software for disks. Are you using a 64-bit operating system? Have you checked if the operating system and its disk drive has enabled 64-bit access to the disk? Because many vendors have it disabled by default. That might be a big difference as well. Are you using a RAID, a RAID array? You can actually use software RAID on just a single disk and get an improvement in performance for a certain profile of access. So there's a lot of things that you probably could tweak. It might not be a big difference. But once you get close to the potential, any small difference is big at that point in time. Also, you could relieve the IO bottleneck by using more CPUs. Now, what do I mean by that? Well, most likely, you don't have a uniform spread of CPU versus IO access in your build. Quite probably, if you look at just your tests, some of them are accessing the disk. And some of them don't at all. So don't make all of the non-disk related tests wait just because some disk accessing test is blocking on the disk access. If you parallelize, this should happen sort of automatically, but you might want to make sure that that's happening. Your tests are actually scheduled such that one blocking test doesn't stop all the rest. You could, of course, use more disks. You might do tricks with just mounting different file systems, even if there's a single physical disk. That might make a difference. What's probably even more important is to choose which file system. On Linux, you have X1, 2, 3, 4, and how many different file systems? You have Riser FS. You have this and that. Some of them are better suited at your IO profile again. So if you have spare time, try and change the file system because that might make a difference. In summary, there's four things I think you should do if you're suffering from a slow build. First, realize that this is meaningful. If you can't, if it's not meaningful, that's actually good news, kind of it is. It means you have other bigger problems. It doesn't sound that nice. But at least you don't have to go into outside of your comfort zone and start tweaking rate arrays and file systems and the sort of things that might be foreign to most programmers. But if you do see that build time is important, profile first, make sure that your code isn't doing stupid things and take a critical look at how you're running the tests because there might be many low-hanging fruits that can give you a big boost in performance. All of the examples you saw, they're almost trivial to implement once you have the dedication to figure it out. This is all open source free software. There's no added cost on doing this. It's just the investment in the time to figure this out and implement it. Thank you for this. If you have any questions, if you have stories to share, I'd be delighted to hear some of them and maybe how this relates to your environment, which might not be Java. So I would reframe that a bit. So what I'd like to have is the same build for all environments, but I might invoke different parts of it in different environments. So for example, in some environments, I have a different risk profile than some other environments. If I run locally, anything could have happened. I might have broken pretty much anything. But there's also some things that I know that even if I've broken this, I wouldn't know just by running things locally because I don't have a cluster environment. I don't have all of the same software stack on my laptop than we have in production. So there are some things that just don't make even sense to run on a laptop. So you might think of it as turning off flags in different environments. But I'd like to keep the version control sense as the same build. So that reminds me of some things that I didn't put in this presentation. So one is test ordering, like fast test first, so they get some known type of certainty very quickly. For example, if you talk about unit tests and integration tests, running all unit tests first fairly quickly, that gives you certainty about what can I assume safely about the health of the system. And this is a clear separation. Another way to separate things and stack the moments when you get feedback so that you get some of it early is to look at functional composition rather than, let's say, technical composition. So you might want to run logistics-related stuff, including unit tests and integration tests, first, if that's what information you'd like to get first. For example, I've been working on the logistic stuff. Therefore, most likely, if anything is to break, it's within the logistic stuff. It doesn't mean that I haven't broken anything else. But doing this risk management or risk analysis type of thing and deciding what to run first might make sense in other ways as well. So on this project where we had the 24-minute build, something that we tried before that was to modularize the build differently. That actually didn't quite go through. It turned out to be so complicated. I had left the project by then, but they had another attempt at this. And this time, they had the devotion to actually take it through. And it also resulted in this particular thing. So since things were more clearly isolated from each other, you had a clearly better way to make assumptions. And you could see your risks more clearly. So separation of the modules and clear understanding of dependencies didn't help just the build system to figure out what needs to be run and what doesn't, but also help the people decide what do I want to run and what do I decide to defer. So also a very good idea. Also for people who compile a lot, this stuff is apparently one of the biggest potentials, slicing the build, because tweaking the compiler isn't that easy. You probably have to write it yourself. Well, universally, whatever you're doing, you should mock it. I wouldn't go that far, but I do find mock library is useful. If you run it very late, you get very delayed feedback. Yes. Or if something is wrong. So I think it depends also on what you consider as doing it. If it is an external service, which is not doing too much of logic, probably mocking is very good at doing it. Something which is, there is a logic in it. I've never ever tested it. I find it very scary there. So I think I agree with you. I'd like to add a small of, I'm not sure if it's any different than what you're thinking, but if there's logic in the database, most likely we want to test it and we are testing it somehow. So we have those kind of integration tests that run against the database. Some kind of a database. Maybe not the Oracle in production, but something that's similar enough that our SQL statements work or things like this. So yeah, we've been thrown out. That's fine. We're over time. So thank you and let's continue just outside the door. Thank you.