 Good morning, folks. Am I good on the mic? Yes, OK. Now I can hear it. So when I proposed this talk, I was thinking a lot about how we were going to handle the fact that microservices make testing hard. But I didn't put testing in the title because sometimes we avoid those talks because they make us feel bad about the testing that we're not doing. But lots of us aren't developing tidy, discrete features that are easy to manage. So how are you going to move from a tangle of interconnected monolithic features into something that you can test and deploy each part of? How do you manage that combinatorial complexity? It's OK. When I was 14 or so, I got this bad boy for Christmas because my parents hate me, possibly. If you love any children, get them the easy version of this because this version was terrible. I spent days trying to get it to work because, first, you had to put all the things in exactly the right pillars in the right place. And then the little clips that hold the space rails on are small in plastic and fragile. Go ahead and move my stuff, sorry. And then you had to tweak the rails just right because otherwise the marble would fly off into space and be in your sister, hypothetically. And it was impossible for me to do. I just didn't have the patience or the fine motor skills or the engineering discipline to make it happen. I never did get it built entirely by myself. I ended up begging our family friend, who is literally a robotics PhD who works at JPL to help me with this. And he and I took two days to put it together. It was very nice of him. Building this toy was one of the most frustrating experiences of my life, but it turns out to have been valuable because this is what software is like, right? We've all had that moment where you're like, I only changed one parameter. Why did everything stop working? Software is the same way. This is a relatively straightforward architecture diagram for software. When we think about software and draw it out and talk to each other about it, what happens is we do some small part of the system that we understand or we abstract it out to a way that we can conceptualize it because modern software is so complicated, we really literally, not even the best of us, can hold it all in our heads. We have to have abstractions to have a handle on these things. That gives us a lot of room to play because the more abstraction layers we have, the more we can do what DHH calls conceptual compression. If you haven't seen his RailsConf talk, you should definitely go watch it because not only doesn't have this super valuable conceptual compression concept, it also has quotes from Piketty, which is a really interesting mashup of economics and software. But the state of the now, let's talk about where we are right now as an industry. We're moving away from monoliths and toward microservices and edge servers and cloud and distributed software. Some of us are moving faster than others, but if you're here, you are part of the people who are hauling the system along toward this bright future. But today, we have some kind of horrible mashup, and that's just where we're at. In the end, we're going to end up with lots of nodes that have multiple channels and dependencies. We're not really going to have server admins anymore because servers are someone else's problem, right? How many of you used to be sys admins, right? Yeah, do you ever miss having 60 servers that you know? No, most of you don't. Yeah, so that's going to be somebody else's problem, and we're OK with that. But when we break things down into smaller elements to manage and automate, it gives us more free time to think about interesting things. And a friend who talked about doing only interesting problems. And I think that's a really great way to think about what we want to be doing. Why do we test? Well, before we can answer that, we have to think, what does test coverage mean? We're trying to avoid failure, and we used to do this by testing all the possible combinations. Mathematics says that's going to get real unpleasant, real fast right now. In the new world of microservice architecture, that is mathematically nightmarish. We used to talk about test coverage, but then we used to use uptime of our servers as a metric for availability. Who here had a server that stayed up for more than a year? Yeah, were you proud, right? That's not really how we're operating anymore, and we also cannot talk about our percentage of test coverage. What do we need to test for? This is the thing that I want you to remember. Charity said this, and it doesn't get less true every time I put it on the slide. Nines don't matter if your user is unhappy. Nines don't matter if your user is unhappy. So whatever we're testing for, one of the things we have to test for is user happiness. And that's a really amorphous and complicated concept, like are users happy about throughput? Are users happy about data security? Do users care about security? They don't know they care, but they'll care eventually. The trouble with what we're doing is toil. And this is something that appears in the Google SRE book. Not the Google SRE book, but the Google SRE book. And Lizfong Jones defines toil as work that is manual, repetitive, automatable, tactical, has no enduring value, and costs you to scale. So when we're testing, are we creating tests that are toil? Are we creating tests that don't scale? Are we creating tests that are only tactical? Or are we taking a little time to think about tests that are not toil? Automate a way as much as you can. We have better things to do with our time. So what's a useful test? I sat down and did some research and thought about it some. And here are some things I think might actually be useful for most systems. End-to-end message time. Like, how fast does it take a message to get through your system? Is your system available? And by your system, I mean the parts of your system? Like, can you ping all your services? Is your request performance good? And that seems like the same as the end-to-end time, but it isn't. Like, how fast can somebody get a response out of your system? Are the responses that they get accurate? Because there's a lot of, like, fuzziness in what we return to people sometimes. And that's OK if that's what they're expecting. Accuracy is not necessarily about correctness, right? You remember this from science class? Accuracy is about how finally you can measure something, whether or not it's right. Is it usable? I know that we don't usually put usability and accessibility in our testing metrics, but why the hell not? Why the hell not? Because I will say this because I say this in as many talks as I can make it fit in. We are all only temporarily able-bodied. I assure you there will come a time when you look at a phone or whatever smart device we have in 10 years and just feel overwhelmed and crushed because you cannot figure out what's going on. You are going to come on this time. Three years ago, I was looking at the back of a conference badge and I realized I couldn't read the schedule because I'm that age, right, where my eyes are fine but my arms are insufficiently long. And, like, everybody knows that's a thing that's going to happen. But when you think about usability, remember that you're not always going to have perfect wrist and you're not always going to have perfect eyes and you're not always going to have the cognitive quickness to catch up to what kids these days are thinking about. So please, please add usability and accessibility as a required part of your test suite now because retrofitting it is a pain in the ass. So what are bad tests? Bad tests are facades. Bad tests are like the Potemkin villages of testing, right? They're code coverage. It's a very nice-looking percentage. Thank you. Yes, I have 98% code coverage. And what does that even mean? How does that do me any good? Or if you test only the happy path, you have to test the happy path. That's certainly something I want you to do. But I will also tell you that the happy path is well trodden, and the not-so-happy path is the one that needs testing. We test only clean data. We write a script that generates data so that we can test. And it doesn't occur to us to write the kind of script that a tester might write. Testers are seriously twisted individuals who perceive the world through a veil of cynicism and brilliance. And they come up with things to test that I would never think of. So if you have a tester, treasure them. And if you don't, sanitize some data a little bit, like take out the PII and use real data, because it is messy in a way that you need to be testing against. And we test on staging. I have a lot of feelings about this because staging is definitely a best practice, right? Except who here can afford to replicate their entire production environment on staging? So a couple of you, that's good. Either you have small environments or a lot of money. Most of us cannot afford to replicate production. So what is staging but a Potemkin test, a false test that serves to make us feel better without actually showing us anything real? So let's talk about the beautiful future. Where are we heading? What are we going to do with this whole microservice architecture world? It's pretty exciting. Well, I think it needs to be self-healing. We can route messages around problems in the internet. This is like what the internet was invented for, right? When DARPANet came up, we were thinking about the Cold War and the fact that we might lose entire cities. So the internet is invented to route around problems. We need to build that same kind of resilience and self-healing into our microservice ecologies. Hubs and spokes break. Let's make it less catastrophic when they do. It's also a really robust way to think about microservices, not as dependency stacks, which when you pull up pictures of microservices, that's what you see is like, here's the database, and here are the microservices, and here, no. Think about databases as like several things. And your connections between your microservices is possibly something you could repurpose. Like, what if there's a problem and you could start feeding data back the other way through your pipe? That's not how we really think about it, but I think we could make that happen in the beautiful future. Ottawa got hit by a tornado this week, right? Did you all hear about this? No. Ottawa got hit by a tornado, which is news because Ottawa used to be north of the tornado zone, but that was then. And what happened was the worst possible thing that could happen for a power grid. The tornado ripped the roof off their main transmission station, the station that gets two nuclear plants worth of power through a big, fat pipe and distributes it out to substations, which then distributed out to people. They lost half of Ottawa's transmission capacity in this one tornado. They are already almost all back up, because what they could do was use a little extra capacity from this station, a little extra capacity from that station. They still don't have the big pipe. They still don't have the transmission station. But because an electrical grid is a grid, they can route around the damage. It's pretty impressive. And they are running things way past their operational norms, and they are overloading things, and they are reducing the service life. But people in Ottawa can keep their insulin cold, and their beer, however Canadians drink it. Sometimes I give these talks in Europe, and I say things about cold beer, and they're all just like localization. The beautiful feature is modular and full of microsurfaces, and the line between yours, mine, and ours is blurring rapidly. When you think about microservice architecture, you probably think about your microservices, right? Except I work for a company that provides software as a service, and I know that we happen to use vulnerability scanning tools that are a microservice that connects to our stuff. And we happen to use monitoring tools and continuous implementation or continuous integration tools. Whose microservice is that? If CircleCI gives me a service to do my integration, is that my microservice? Is that their microservice? Well, I think it's something that I should test. I'm pretty sure they test it, but I think I should test it in my environment with my pipes. Because otherwise, how do I know what's going to go on? I went and looked up a definition of microservices, and I really like this one from SmartBear. They say, a microservice is a distinctive method of developing software systems that tries to focus on building single-function modules with well-defined interfaces and operations. SmartBear is an API company. It makes sense that they talk about the well-defined interfaces, and that's something I see a lot of people skip when they talk about microservices. They're like, microservices do one thing and one thing only, and that's great. I'm like, yeah, that's great, but how do you talk to it? How does it talk to you? That is the API core of what we're trying to do. In the beautiful future, a lot of our work has been taken over by robots or automation. They're not smarter than we are. In fact, anything we can program can only be slightly dumber than we are, but they never accidentally add a space to Python. Or, hmm, yammel. Knowing laugh, yeah. As we climb up layers of abstraction from machine language, we're getting closer to telling programs what to do instead of how to do it. After all, we can only tell computers to be kind of smart. This is the airbag construction from the Pathfinder mission. And remember my friend who helped me build the roller coaster? He worked on this project, too. He said, maybe less frustrating. We didn't have to teach Pathfinder how to bounce. That was physics. We built something that could bounce, and we let physics take over. And that's something that's really interesting about games right now. We have these physics engines that are not defined rules every pixel goes here. They're a set of guidelines for if your goat licks a pole and then the pole falls over, yeah, now I know who has kids that play Goat Simulator. That's a really weird game. It's super distressing. The physics on the goat neck are not OK. But we're not telling things exactly what to do. We're telling things roughly what the physics are doing. And that's where we're going in the beautiful future. It's easy for us to think of tests and test results as real, but they aren't. They're symptoms of the living and evolving organism that is software. We have so many layers of abstraction that we don't actually know what's going on. We just believe we do. And the sooner that we change our thinking about this, the sooner we'll begin to understand that tests are about symptoms and not about causes. Doctors don't ask us how our thyroid feels, because we don't know. We know that our skin is weird, or our nails are weird, or we have a really fast heartbeat, or we feel really sluggish, or something like that. And we'll go to the doctor and complain. And then they'll take a test that tests the thyroid stimulating hormone, because you still can't directly test all of the things that the thyroid produces. So we're like, well, your thyroid appears to be trying to work hard, and it doesn't seem to be working hard. So there's a mismatch there. We can't really tell what's going on. We know roughly how to treat it, but we can't get to the root cause, because bodies are super complicated, but so is software. So I promised you Tinker Toys. Why are microservices like Tinker Toys? They're like hubs and spokes. If you took a whole bunch of the Tinker Toy hubs and piled them up, it wouldn't be very exciting. And if you took a bunch of the spokes and tried to do something with them, you couldn't get very far, right? We have to have both. The hubs or the microservices, the sticks are the channels between them. And without both of them, we can't build anything interesting. We're coming from a world where the channels weren't a problem because we were essentially building with wooden blocks. You could build something interesting with wooden blocks, but microservices aren't wooden blocks. They're just hubs. They're just the little round things. That's all you can do. So we have to be aware and cognizant that the channels are the thing that we're driving toward. The APIs are the important part. When people build with Legos, they have a specific terminology that describes the number and depth of the studs so that other people can reliably duplicate the build. I love that there's a whole build method called SNOT, studs not on top, Lego people. APIs are the same thing for your microservices. If you don't define exactly what's going out and what's going in, you're not doing anything useful. So I'm gonna let you take a picture of this. Let me just say, API users have certain consistent requirements, whether they're human or machine, right? We want to connect securely. We want to handle data consistently. We want to get confirmations, pass through states accurately, spend the least possible amount of effort doing all of that, right? A good API is easy to play with. It gives you a sandbox. It makes it possible for you to inject things without necessarily disrupting the flow. A good API is easy to integrate with. When I first started using Swagger, I spent two super annoying weeks trying to figure out why it wouldn't work. And the answer is, at that point, you had to have a website host set up. I wasn't really covered in the early documentation, I think because they assumed that who would try and run an API service without web hosting? They have a self-hosted solution now, it's much better. You have to be able to integrate with something in order to use it. An API exists to hide the mess. This is the functional equivalent of shoving your dirty dishes in the oven before your guests come over. I don't care what your dirty dishes look like. I care that they're somewhere to set my drinks, right? So when we design an API, it doesn't have to be describing a beautiful back. It has to be describing what people can reliably predict from you. And we don't want to add things for no good reason. It's important that all of our features have function as well as niftiness. So why do they fail? Why do APIs fail? Well, security is hard to do. We design from an egocentric point of view instead of thinking about people who might be using an API in a way that is different than the way we planned it. We offer inconsistent endpoints. If you don't have a style guide for naming your endpoints, please go home and make one because your camel case and underscore mixing is just making all of us sad. We're missing information. There's insufficient documentation or there's stealth things that nobody has talked about that you need to make happen. So we have these APIs and everything is going okay, but multiplying combinations of microservices grow quickly. These are the orbits of Earth and Venus over the course of a year. Is that actually going? Yes, okay. Are you all properly hypnotized? Yeah, so that's like two bodies. Imagine how much worse it gets every time you add a new microservice. You're not gonna be able to test all that. The best thing we can do in the face of this complexity is stop trying to test everything and use our brains. Test that the microservice works. Test that the channel works. Test that the user flow works. And then every time you add something new, we're gonna test it in these four ways. Test it with everything on. Don't show this to people because it looks really terrible, but test it with everything on to make sure things are stepping on each other. Test it with everything off to make sure that you are not dependent on something. Like it's really easy to break if you haven't tested with everything off. There's unseen dependencies. Test the status quo. Like before you turn your new feature on, test to make sure that everything was working before because it's really hard to troubleshoot. I deployed something to an already broken system and it seems broken. Nobody's ever done that though. Then test it. And this is where I say feature flags are your friend. Test it by turning things on and off. And if you have feature flags, this is super easy. You can just flip them. Use the feature flag to say, does it work when it's on? Does it work when it's off? How about different circumstances? Different users? Different services. If you have a flag, that's a much easier thing to test. Do all the things in production. Kill staging. Kill staging. It is a lie and you're only spending money on it. We had a client who said, we were spending about $10,000 a month on maintaining this very large staging thing and we killed it and we're spending the money on something else because we can test in production. If you can test in production, then you know that you're getting weird data and you know that you're getting user data and you know that you're getting spikes. And you don't have to show users what you're testing in production. Release and deployment, they need to be different but you could kill staging and save all that money for something awesome like donuts. Production can have an experiment layer. Production can have an integration layer and production can have a stable layer and different users can see different things. So this is a roadmap. It may not be your roadmap but I want you to think about whether there are parts that you could take away with you. Start with small victories. Estimate what's possible for you to do. Eat the whale one bite at a time. You don't have to go all in with everything. And when it goes well, testing your elements and the connections is probably sufficient. The isolation holds and your predictions work out. If there are failures, they're probably not disasters because you have firewalls. When it's bumpy, when it's rough, you discover disconnects that you weren't aware of. Testing these through lines tells you that there were breakages that either you have just introduced or existed before and that's really important to know. So even if something fails, it's a successful failure, right? So if this was too long and you read Twitter instead, monoliths are the past, monorepos are the future. Full test coverage is not a thing anymore. Please get over it. What you test is more important than how you test it. And test all your things in production. And if you would like a free t-shirt because I never carry them around, you can take a picture of this slide and we will send it to you. If you would like to talk more about feature flags, canaries and testing and deployment, I've proposed an open space this afternoon. Also, if you wanna talk about how to do documentation in a DevOps environment, I have an open space for that. Thank you all.