 All right, let's get this started. My name is Ryan Davis. I am known pretty much everywhere as Nspider, except for Twitter, as you can see. I am an independent consultant in Seattle, and I'm a founding member of SeattleRB, which is the first and oldest Ruby group in the world. So setting some expectations. This is an introductory talk. It's very little code in it. I'm not going to teach testing or TDD in this talk. I'm going to be talking about the what and the why, not so much the how. I have 218 slides, which puts me at just under five and a half slides a minute, so I have to go kind of fast. So let's get started. The simplest thing that we can ask is, what is Minitest? Minitest was originally an experiment to see if I could get Test Unit replaced for about 50 of my projects that I had at the time. I'm up to about 100 now, with as little code as possible. And I got that to happen in about 90 lines of code. It's currently available as a gem. We didn't have Ruby gems back then, when it was originally written. It now ships with Ruby as of 1.9.1 and up. It's meant to be small, clean, and very fast. It's now about 1,600 lines of code, which sounds like a really big increase over 90, but still, that's very small. It supports unit style, spec style, benchmark style, very basic mocking and stubbing, has a very flexible plugin system, and et cetera, as we'll see. There are six main parts of Minitest. The runner, which really kind of is nebulous now. Minitest, that should say Test, which is the TDD API, Minitest spec, which is the BDD API, Minitest mock, Pride, and Bench. I'm only gonna be talking about two parts, Minitest Test and Minitest Spec. So let's jump into Minitest Test, which is the unit testing side of things. So test cases are simple classes that subclass Minitest test, or another test case, and tests are methods that start with test, and they make assertions about your code. It's just classes, and methods, and method calls all the way down. Everything is as straightforward as you get, and it is magic free. That slide is two years old, so this wasn't my only barbit null. So Minitest Test includes the usual assertions that you would expect from XUnit, plus several added beyond what XUnit and TestUnit usually provide, methods marked with a plus are new to Minitest, and methods marked with a star do not have negative or reciprocals, as we'll see in a sec. Unlike TestUnit, Minitest provides a lot more negative assertions. It doesn't provide some that you might expect, which I'll go into later, but one of the questions is like, why are there so many pluses? I really want my code tests included to communicate to me and other readers as clearly as possible. Not only does the code communicate better, but the error messages are much more customized, so when there is something wrong, I get better information about it. And finally, AssertEqual is enhanced to do intelligent diff mode. It allows you to see what's actually changed instead of a huge blob on the left side and a huge blob on the right. You get to actually see what's different between the two. But I said that I would describe why some negative assertions are missing. This is something that I hear a lot more than I'd actually like to hear. The question of where's refute raises or assert not raised? It's the same place as a refute silent. So let's look at that. Let's highlight the key components of refute silent. So refute silent says that this block of code must print something. What it is, I don't care. That is a valueless assertion. What you should be asserting for is a specific output that you want. In the same vein, refute raises says that this block of code must do something. What it is, I don't care. It's a valueless assertion again. Instead, you should be asserting for the specific result or side effect that you actually intend. I've heard the argument, but it's useful. No, it isn't. It implies side effects and or return values have already been checked or aren't important, which is always false because you wouldn't be writing code otherwise. It falsely bumps your code coverage metrics and it gives you a false sense of security. I'm an ex-lifeguard. I have a lifeguard in my high school days at various lakes in my county. And one of the things we really feared were parents with kids with water wings because the parents think they've got flotation devices and we'll just talk to our friends and ignore our kids and watch them drown like this. So in other words, this only makes it look like something has been tested when in fact it hasn't had any tested applied to it at all. I've heard it's more expressive. No, it's not. Writing the test itself was the act of expression. It's an explicit contract in every single test framework out there that any unhandled exception is an error by definition. The test mere existence states there are no unhandled exceptions via these pathways. And I've been having these arguments for years. In fact, I had this argument last month and I know that some people will never be convinced and honestly, that's okay. You can't win them all. But it doesn't mean I can't try. So hold on your hats, stand back. I'm gonna try one more time to convince all of you. It's like if you call it, it must be okay, right? So I wrote all these extra assertions to verify that everything is okay. And if you'd like to license this code, come please see me after this talk. If only that were possible. Our jobs would be so much easier. Next up is MaytestSpec, the example testing side. In short, where MiniTestTest is a testing API, MaytestSpec is a testing DSL. Instead of defining classes and methods, you use a DSL to declare your examples. Test cases are described blocks that contain a bunch of tests. Tests are it blocks that call a bunch of expectation methods. Here's an example that is equivalent to the previous one. So we have describe instead of class, we have it instead of def test. But in reality, describe blocks really are classes and it blocks really are methods. This same example one to one transforms into this code where describe makes a class, it makes a method, there is no magic. All of your normal design code tools exist and they work as normal, which is really, really important. It means that include works, def works, everything is as expect, as you expect, you're just using a slightly different language. Similar to MaytestTest, MaytestSpec has many expectations defined and a similar set of negative expectations in the similar set of missing culprits. And all of this is gained for free because each one maps from expectation to assertion directly. Underneath MaytestTest and MaytestSpec is the infrastructure to run your tests. And it does so in a way that helps promote more advanced and robust testing. Maytest has randomization baked in and has always been on by default. It helps prevent test order dependencies and keep your tests robust and working standalone. By rule, every single one of your tests down to the lowest level should be able to run by itself and pass. If it requires another test to run before it, it's not standalone, it's not a unit and it's wrong, it's buggy. And as far as I know, Maytest was the first test framework to have randomized run order. There's an opt-in system that lets you promote a test case to be parallelized during the run and that takes randomization to a whole other level. It ensures thread safety in your libraries and absolute robustness. Cause if it can handle the parallelization, it can handle anything. The original Maytest was a tiny 90 lines of code and over time features have been added that you get to choose to help you enhance your tests. It's still really small in comparison but it's incredibly powerful. So what was my reasoning for Maytest design? Maytest is not special in any way, shape, or form. All of my usual tropes apply. If you've heard me up here, ranting and raving, Maytest is no different. First and foremost, it's just Ruby. It's classes and methods and method calls. Everything is as straightforward as you can get. I believe that less is more. If I can do something in less code, I will, absolutely. Method dispatch is always gonna be the slowest thing in Ruby. More importantly, less code is almost always more understandable than more code. So let's take a look. Here is assert in Delta, which is the equivalent of assert equal but for floats. Never ever use assert equal on floats and never use floats for money. There, I've done my usual caveats. It's as simple as possible with minor optimizations. In this case, we use a block to delay the error message rendering. In the case that we don't have an error, we shouldn't have that cost. And really, this just boils up on up to assert. So you really only need to know about 15 other lines of code to understand how this works as a whole. Indirection is the enemy. I want errors to happen as close to the real code as possible. I don't want things delayed. I don't want layers of indirection in between. I kind of feel like Noel is making my point because he just kept talking about the layers of indirection that our spec adds over and over. I want to strip all of those out as many as possible at least. I want the responsibility to lie in the right place. No managers necessary, no coordination going on. I want objects to be responsible for their own duties. This may not be the best example. I don't know how to get a good example of indirection as the enemy because it's rather hard to show what you don't do. Must equal is an expectation that directly calls assert equal on the current test context and then assert equal is three lines long if I remember right and that's it. No magic allowed. Even test discovery avoids object space. It has minimal method programming in it and it uses plain classes and methods to do all of its work. I originally wrote Minitest in part to see if I could because I was the maintainer of test unit at the time and test unit terrified me. But I also wrote it because I was working on Rubinius and I wanted Rubinius and JRuby and other implementations of Ruby that hadn't finished being a full Ruby to have the simplest implementation of a test framework possible so they could get feedback quickly. And finally, it has a thriving plugin ecosystem. I designed Minitest to be extensible so that Minitest itself could remain minimal. Here are just a small amount of the popular plugins for Minitest. Okay, so what does this have to do with Rails? Well, the official Rails stack uses Minitest. Each release, it peels back the testing on in encouraging better testing practices except that peeling onions makes you cry, right? Hopefully not in the case of Minitest. So Rails 4.0 was the first version to cut over from test units to Minitest and they did so on the Minitest 4.x line. At the time, I had already declared that I wasn't going to keep updating standard lib Minitest to Minitest 5. I was only gonna maintain it at level version 4 and that's because test unit is built on top of Minitest and has a lot of hooks into the internals and it just made it really hard for me to ever update and not break their stuff. So I put a freeze on that. So Rails updated to Minitest 4 to remove a layer of complexity and indirection but also to allow them a migration path to Minitest 5. Because test unit was already wrapping Minitest, there was basically no impact on anyone. Arguably there may have even been a almost imperceptible performance improvement but I'm not gonna claim that there was one. Rails 4.1 switched to the Minitest 5 line. This was painful for Rails itself because of crufty tests and the number of monkey patches that they had on Minitest itself. This got Rails onto the newer code base, the active development line of Minitest and it made things easier to do like exec based isolation tests. There's a number of tests in Rails or each test actually forks a process to run a separate Rails app by itself so that if they're running in parallel they're not gonna infect each other. As painful as it might have been to get Rails switched to it in passing, you hopefully never noticed them. However, Rails 4.2 turned off the alphabetical order of the testing and to remove a monkey patch and that started to run tests in random order. This is solely to improve the quality of Rails's and your tests but it may have had some impact on some people. We did get some reports that after updating people started having tests that previously passed failed. I had to isolate a number of test bugs in Rails itself because of this and I'm gonna say honestly, despite the pain that it causes, this is a good thing. Test order dependency bugs are problematic and they're incredibly hard to track down. I'm gonna talk later about a tool that can help identify these bugs and hopefully future versions of Rails, they should be tracking many tests. Aaron Patterson and I keep those things in sync and he lets me know when things are coming down the pipe. So as a Rails dev, what does all of this mean? Hopefully if I've done my job right, means nothing. You shouldn't even have to see many tests most of the time unless you wanna enhance it with some plugins and that's because your subclassing Rails's test cases like active support test case or action controller test case, there's about six of them if I remember right, there might be more now. And the basic architecture looks something like this. You write your own test class that subclasses active support test case and active support test case subclasses may test test. It provides things like per test database transactions so you don't have to clean up, you just add a bunch of records, they're gonna be gone on the next test. Aaron and I wound up adding before and after set up and tear down hooks to make it easy for Rails and other libraries or frameworks to extend many tests to do extra wrappings that they needed to do. It provides things like fixtures to load test data, declarative forms if you like that instead. If you don't like those, you can just use def. And it provides extra assertions like assert difference, assert valid keys, assert deprecated and assert nothing raised. Wait, what? Don't worry, this is the actual implementation of assert nothing raised. It's only there for compatibility's sake and personally I think that it should be deprecated and removed, maybe that should be for Rails 5. And all this means that you can write simple tests that describe your current task, things like database transactions don't have to clutter your test code and you can focus on what the test is really trying to do. This is a very simple example and I think all of these test examples come from the testing section of the Rails guide online. Action controller test case is another Rails extension except that it subclasses active support test case so you get all of those goodies layered on. And it extends it to include more like all of your usual HTTP verbs, simulate web server state, and provide assertions specific to handling requests, which lets you easily write clean functional tests like the following. I love air conditioning, I'm so dry right now. I'm just gonna leave the lid off. Active dispatch integration test provides full controller to controller integration tests. As usual it subclasses active support test case so all of the usual stuff is there. It provides a ton of assertions that I cannot fit onto one slide and allows you to write comprehensive integration tests that span multiple controllers. If you want more details about all the stuff that Rails adds on top of many tests you can get that here and it's described in pretty good detail. It's actually a really good read. So Rails' approach to subclass main test test leads to a very simple setup that remains very powerful providing you with everything you need for unit tests all the way up to integrations. It leverages main test power, providing randomization, optional parallelization to provide better testing, make your tests that you write better and more robust over time. But what happens when something goes wrong? Perhaps you want to use spec style. Turns out the DHH disapproves of our spec so much that he wouldn't allow us to switch the test framework in Rails to main test spec. He reverted that commit. As I understand it he simply doesn't want people to submit patches using spec style so he didn't want it easily available. But that doesn't mean that you're stuck. If you prefer spec style that's totally fine. You can use Mike Moore's main test Rails or Ken Collins' main test spec Rails to do varying degrees of this type of code. The two libraries are not the same. They suggest slightly different styles. One tracks our specs style a bit more than the other. Or perhaps you upgraded to Rails 4.2 and now you have failures. Also known as Ryan you broke all my shit. I'm sorry. Unfortunately that's not as simple to deal with. It is a bit harder but again, it is a good thing to catch and fix this sooner rather than later. Why are my buttons not working? It is a test order dependency bug. And quite simply, that simply means that tests will pass when they're run in a particular order. Say A before B but not when B runs before A. And if it was only three tests that wouldn't be a problem. It would be pretty easy to find and fix. But you probably have hundreds and hundreds of tests. That's not so easy until a few months ago. I wrote MiniTest bisect to try to isolate the problems that we were having getting Rails onto MiniTest randomization. It helps you isolate and debug random test failures. In short, it intelligently runs and reruns your tests, whittling it down until the culprits are minimized. And here we're gonna see a simple example of it running. Ignore the fact that I'm pre-specifying the seed. Pretend that I'm running this anew and we get this failure. We get the failure. We grab the random seed and we rerun the test using MiniTest bisect instead. That's gonna rerun the entire test suite to ensure that it is reproducible. And once it is, it starts to bisect the culprits space down to the minimal subset. And you can see that it speeds up dramatically as it goes on. Getting it down to two tests run in a particular order that will cause the repro every time. I see I'm not alone with this problem. Or maybe you're just used to kitchen sink development. For starters, just try it. It might work. Things like Mocha and a lot of testing libraries are ready to work in MiniTest and they're just fine. Otherwise, you're not alone. And someone probably beats you to it. So look for the existing plugins that are listed in MiniTest. Read me or search on rubydems.org or Stack Overflow or whatever. But my suggestion, and I know this is going against the flow here, try less complicated testing. Only bring in plugins once you've decided that you really, really need them. Otherwise, start fresh and clean. Try it. You might like it. But the problem is that change takes time. Just remember that you might want to measure the before and after in order to be more objective about this change. I've only got anecdotes of projects speeding up when they switch to MiniTest. I would love more data and if you would submit that to me, that'd be great. I have heard that people have halved their test times by switching from R-Spec to MiniTest, but again, I don't have anything objective. So, all of this MiniTest stuff sounds interesting, but why should I bother? The first argument is this. I'm not gonna bother with that. To me, you're a lost cause. There's plenty of data showing that the benefits of testing, and if you can't get past that, I'd rather help other people. I'm gonna pick my battles, thanks. Like this one. This is a battle I wanna pick every time. Obviously, not everyone uses it. The official rail stack uses it, which means DHH uses it. Tenderlove uses it. Jeff Kazemir and his cohort teach MiniTest at the Touring School. Nokogiri, Hamelgod, New Relic, SQLite, and a bunch of other very popular gems use it. In fact, there's more than 4,000 gems that declare dependencies on MiniTest, and unfortunately, because MiniTest ships in StandardLib, there are plenty of gems that don't declare their dependency on it. So I'm sure there's plenty more. So what are the real functional differences between MiniTest and RSpec? Being test frameworks, there's plenty of overlap, so I'm not gonna go over that. Where they are unique, though, is where it gets interesting. To be fair, RSpec provides a lot more than MiniTest. Things like test metadata and metadata filtering, more hooks like before and around. Implicit subject described class, fancier mocking, et cetera, et cetera. Basically, it's fancier. MiniTest by definition doesn't offer as much. Some of the stuff that made it unique has been adopted like randomization, but benchmarking, parallelization, and speed are the main distinguishing features. Basically, it's simpler and more pragmatic and snarky. So it's the cognitive differences with RSpec is where things really start to diverge, and this is where I think that Noel's talk actually proves my point quite a bit. Myron Marston, a few years back, wrote a really great response on Stack Overflow comparing RSpec and MiniTest. It was a bit biased, but honestly, I think that it was rather fair. The problem is that it's really long, and the meat of it is in this first paragraph here. But even it's pretty long, and apparently I've been working with a tender lab for too long, whose attention span is that of a pherodon methamphetamine. So there's so much there that I have a hard time dealing with it all at once, so let's focus on that one paragraph. I'm gonna color code it red for the RSpec points and blue for MiniTest. I'm not gonna read that crap. I've read it too many times. So Myron thinks this is why RSpec is great, and I think that this is everything that's wrong with RSpec, and we're both right. Philosophically, we're both right. We have different goals, and we have different perspectives on what good is. So back to that paragraph. Let's try to boil this down even more for the ADD. Even that's pretty long, so let's break it down to a simple table. Example groups. MiniTest compiles blocks down to simple classes where RSpec, and I'm paraphrasing this here because he didn't say what it did, reifies testing concepts into first-class objects, and that first word I think is the big red flag for me with regard to RSpec. If you're using that word, you should be coding in Haskell. Examples, MiniTest compiles, it blocks down into simple methods, whereas they use first-class objects. MiniTest uses inheritance or mixins for reuse, where they use shared behaviors for first-class constructs, and simple methods for making assertions for expectations versus first-class matcher objects. What even that's pretty long? So let's boil it down further. MiniTest uses a class, they use a first-class object. MiniTest uses a method, they use a first-class object. MiniTest uses subclasses or include, they use a first-class object, and MiniTest uses method calls, they use a first-class object. First-class simply means that you can assign something to a variable, and you can use it the way you can use any other value, and it just so happens that everything that MiniTest uses is Ruby, and nearly everything in Ruby is first-class, so that's not a good distinction. Everywhere that I could use Ruby's mechanisms, I did, and everywhere that RSpec could reinvent, they did. So let's try to take a look at this and see where this starts to get more cognitively complex. I think this is best illustrated by examining how RSpec works. Here we have two nested describes. Each one has a before block, and each one has a single example. Yet the before blocks seem to be inherited, and the examples are not. There'll be exactly two runs here, where the first one uses one before, and the second one uses two befores. So our A and B classes is nesting like subclassing. What's the analogy we can use to understand this? Well, if their classes, and nesting is like subclassing, then we need this undeath method to ensure that we don't inherit any tests from our superclasses. This is the approach that MiniTest uses, and it sucks. But that's the runtime behavior that RSpec users expected out of MiniTest spec, and that was something that I wound up having to put in. What about this analogy? Are before and after included modules? If that's the case, then we don't have to undeath any methods, but instead we need to generate a bunch of inner modules and have a bunch of includes, and setups need to intelligently call super, and because of the complexity, this sucks, too. This is basically another object model. To effectively use RSpec, you need to learn a whole separate object model that sits on top of Ruby's object model. This is doubly confusing if you haven't already learned the Ruby object model, or if you're just trying to learn both at the same time, you're gonna get overwhelmed. What winds up happening is it encourages users to hand wave. Noobs are just using, just learning Ruby. They don't have the time or the ability to dig in and learn both object models, and it encourages users to hand wave the oddities away. What else are you gonna do? So it encourages them to not know what describe and it or actually are or do, and if you didn't see the previous talk, there's a lot of stuff that goes on inside those described knits. Basically says, here's the magic incantation to do X. So when you're a beginner, any sufficiently advanced technology is indistinguishable from magic, thanks to Arthur C. Clark. So now I need to rename my talk. Sorry, Noel, but RSpec is magic. So from that previous post from Myron, he said that many people find it to be overkill, and there is an added cognitive cost to these extra abstractions. Indeed, here are the raw numbers of that added cognitive cost. Don't bother grokking these numbers, we're gonna visualize them in the next slide for both the FLOG line and the comments plus code line. FLOG is a complexity metric proportional to how hard something is to test, to debug, or even to understand, and here RSpec is 6.6 times bigger, and this is the combined meat of the project. This is code plus comments, and it's basically how much you're gonna have to read to understand each library. At 8.5 times bigger, it's akin to reading Dr. Seuss versus James Joyce. Do you like my Dr. Seuss font? I think I faked that really well. So back to that added cognitive cost. As we're gonna see in a second, it's not just cognitive cost. There are performance differences as well. All those abstractions reinventing the wheel. It has a real cost. Here we have some fairly complex plots. This shows the runtime and solid lines and the amount of memory allocated in dashed. The green lines are 100% passing, the red lines are 100% failing, and the others are in between that. And as you may notice, there's a slight problem to these charts. Can anyone guess? They're not using the same scale. So now you can see that there's a severe and painful difference between RSpec and main test. Failures have exponential growth in RSpec. Runtime is always near zero in many tests, memory is linear, and it's always lower. Brian, who cares? Passing RSpec is fast enough. Indeed. I actually agree with that. Everything is bread and roses as long as everything plays nice. But what happens when it isn't? You have a bug or you refactor or anything else goes wrong. Oftentimes when I'm refactoring stuff, I'm gonna make a change and I'm gonna have 100 tests fail. All of a sudden you pay. For completeness, the speed of the actual assertions in both systems are purely linear. The speed of running those tests at the method level is also linear. And because they're linear, please do not try to regain any speed by reducing examples or expectations or otherwise reducing the quality of your tests. If you want to speed up, Minitest will always be faster than RSpec pretty much by definition. So you can switch to Minitest or you can never ever refactor or have any bugs. So in summary, at the end of the day, as long as you test, I don't actually care what you use. Use the best tool for your job. Hopefully I've shown the technical merits of Minitest. Choose what works for you. Not because it seems popular. Oftentimes I hear that they chose RSpec because there's more documentation available for it. But maybe there's fewer articles about Minitest because there's less need for them. Minitest is much easier to understand. You can read it in a couple hours and understand it head to toe. So maybe Minitest users aren't missing. Maybe they're just busy getting stuff done. Choose what works for you. Who knows? Try it. You might like it. After all, it's just Ruby. Thank you and hire me.