 Thanks, y'all. Can everyone hear me okay? Awesome. Good morning, Texas. Let's talk about software testing, shall we? Whoo! So my name is Andy Knight. I am known as the Automation Panda. I love software testing, automation, behavior-driven development, and Python. You can follow me on Twitter and my blog at Automation Panda. I'd like to thank my company for sending me here today. I work at Precision Lender. I've been there for just a year as of this week. Precision Lender does applied banking insights, and a couple of us will be at PyCon this coming May. So thank you, Precision Lender. Here's a question for everybody. Raise your hand if you have ever hit a software bug. Everybody, please put your hands up. Okay, thank you. Great. We all have experienced software bugs. Do we like software bugs? No, they're bad. That's why they're bugs. That's why we want to squish them. So here's another question. Raise your hand if you always test your code. Wow. Yeah, that's about what I expect. I've given this talk a couple of times, and that's usually what happens. Now, I'm not here to do shaming or anything. It's not bad. There's reasons. It's okay. But I want us to stop and pause for a moment and think, time out. Why is there just this discrepancy? We know we have all these software bugs, but we're not always testing our code. Well, why is that? And furthermore, why don't we automate the tests so that the code gets tested automatically so we can spend more time making the code to begin with? This is not an easy question to answer because there are some tough points we have to face. First of all, especially with test automation, we know deep down this is something we should do. For small projects, it's not such a big deal. But for big projects, things that scale, things that have huge teams working on them, we have to move fast, we have to move continuously, and there's lots of moving parts. There's lots of danger areas that things could just break and all those bugs just rush in like cockroaches. We know that we also don't have enough time in the day or even in the universe to test everything manually. We can't repeat the full testing of these giant systems over and over again ourselves. There's just not enough time for that. We need some solution to do things automatically. And that's where test automation is supposed to help us. Unfortunately, though, we also know that test automation is hard, and that's one of the barriers of entry of why I think so many people didn't quite raise their hand when I asked, do you always test your code? Test automation is not an easy thing. I do this day to day. I know firsthand it's not easy. I fight the problems in the test automation every day. It requires strong developer skills. It has its own technologies. It has its own software stack. Just because you're a good developer doesn't mean you're necessarily going to be a good test automation engineer because there are certain aspects to that domain that are tricky. And I could spend all day talking about that. But ultimately what I want you to recognize is that test automation is not just about writing test scripts. Test automation itself is software and deserves the same practices and tools and attention that we give the product code, so to speak. And with that in mind, I've also, from talking with a lot of people in different companies and teams, I found that test automation can be intimidating for a lot of folks. Many teams really don't know how to start. They know they need it. They know they have bugs, but they don't know how to approach the test automation challenge, so to speak. And so I see two things happening. Either teams will keep punting the football down the core and just keep avoiding test automation altogether, and they'll either skip test coverage or they'll hire teams just to do manual testing, and that's not a really good situation because that's unscalable. Or a lot of times what happens is they'll try. But because they don't train themselves appropriately, they don't get the information they need, they don't really try, or they don't take the careful steps to begin with, next thing you know, a year later they've got thousands of tests that are failing every night, and they're not getting value out of them, and they just want to nuke the whole thing and say, we waste the time, let's start over. That's not good. Both of those situations are not having anything or having stuff that is detracting from your value very bad. And we hit this moment where we are, ah, eekad! What do we do with this? Because we know we need this, but we're not doing it well. And that's what I'm here to talk about today. And in fact, I've got a little story to go along with how we can approach the test automation challenge. And I want you to meet a friend of mine. Let's meet someone who can help. I'd like you to meet my friend, Amanda the Panda. Amanda is a software engineer who loves using Python. Amanda works at a company called Bamboozle. Yup, Bamboozle is a small but growing tech company. Bamboozle provides an online platform for Panda commerce. As a full stack developer, Amanda has a good understanding of Bamboozle's web front end, REST API service layer and infrastructure needs. Recently, Bamboozle had a couple severe bugs escaped to production. Oops, eekad! Management decided it was finally time to start automating tests and Amanda volunteered to lead the effort. Since this is the first time she's done test automation, Amanda does her homework. She starts learning about different testing types, tools and frameworks. However, her web research feels more like a game of buzzword bingo because there are so many buzzwords when it comes to testing and automation. And everyone seems to give different definitions to the same words. Anyone have that problem before? True story. So, like, what are the differences between unit integration and to end testing? What are all the packages I need? What's this cucumber thing? The Selenium web driver thing? Do they work together? They separate frameworks. I don't know. We need help. So, Amanda calls her friend Andy. That's me. Software engineer and test. There's so much info out there, Amanda says, that I can't discern what is the right approach to take. We want to do things right. How do we do it right? Let me ask you a question, Andy says. What is your main goal with testing? What is your main goal? Well, Amanda replies, I guess I want to make sure that the critical parts of the bamboozles app are working correctly. Anytime someone introduces a bug, tests should catch it immediately. Great. It sounds like you need a functional test automation solution. Functional tests make sure that the features work correctly. As you said, functional tests serve two main purposes, validating goodness and identifying badness. I'd like to share with you the test automation pyramid. Has anybody seen this before? Some people? Okay, cool. Let's talk about this. The pyramid is a rough guideline for how an automation solution should be structured. All layers are functional tests from unit to integration to end to end. At the bottom, we have unit tests, which cover individual units of code, like functions, methods, and classes. Unit tests are white box, in that they interact with the product code directly. They're making calls to your functions and classes. Above that are what we call feature tests, which cover black box behaviors of real live instances of the product under test. So you don't directly access the code, you access whatever the deployed version of that product is. Integration tests cover components where things meet, usually like a one-two hop, maybe a service API. And to end tests cover paths through the entire system, typically like a user would exercise through the web UI. Each layer of the pyramid mitigates risk at its optimal return on investment. That's why we have the layers. We want a strong foundation of unit tests because they are small, quick, easy to maintain, and will catch things very quickly. Integration and especially end-to-end tests are far more costly to build and maintain. And they also take more time to run. I want you all to consider the rule of ones. A unit test would take about one millisecond to run. An integration test usually would take about one second to run. And an end-to-end test order of magnitude will take about one minute to run. So there's this issue of scale as we go up the pyramid. Hold on, Andy. Amanda interjects. Why do we even need feature tests? They sound like such a burden when compared to unit tests. I also read that web tests can really be flaky with Selenium WebDriver. Andy replies. Well, hold on now. Unit tests don't catch all problems. Here's an example. Back in 1999, NASA lost the Mars climate orbiter when it disintegrated into the Martian atmosphere. Why? Ground-based software did not convert measurements into the metric system. This is a real story. The calculation modules worked individually, but they did not integrate properly. Had NASA checked software with proper integration tests, they would not have lost $655 million. Wow! Amanda explains. I wouldn't want that to happen to my company. It sounds like I need to take a pyramid approach. So, what skills do I need to get started? Andy smiles. Good news. You already have the skills. Test automation is software. And the same development principles apply. Just pick a core framework and get rolling. You may also want to take some online courses about frameworks and test packages that you'll need. Check out Test Automation University, which offers a bunch of courses and experts for free. So, there's another question Amanda asks. What test framework should we use? Bamboozle does a lot of Python on the back end. So, I looked at frameworks like unit tests, PyTest, and Behave. Python is one of the best languages for test automation, Andy replies. And you can't go wrong with PyTest. PyTest can handle tests at all layers of the pyramid. Markers can categorize tests, as functions and fixtures are lightweight. And plugins can be added for reports, code coverage, and even parallel execution. Among all Python test frameworks, PyTest is also arguably the most pythonic. So, that could be up to debate. Now, a test automation solution is more than a core framework though. So, be cautioned. Successful automation projects bring together readily available packages, rather than developing everything from scratch. For example, you may want to use Python's standard logging library for logging messages. Requests would be great for testing REST APIs. Selenium WebDriver and Splinter would be great for automating WebUI interactions. And there are also tons of PyTest plugins for different frameworks like Flask or Django. So, be sure to look before you leave to see what packages will be useful for you. I can also recommend a few books to get you started. Python Testing with PyTest, by Brian Aachen, is an excellent resource on the PyTest framework. Also, by Bruno Oliveira, is PyTest Quick Start Guide, which is another take on how to get started with PyTest. And for general automation concerns in Python, Handsome Enterprise Automation with Python by Bassam Alley is another great resource. So, check these books out if you want to learn more. Create a device, Andy, Amanda says. This will help me get started. So, Amanda goes back to Bamboozle and starts writing tests in Python using PyTest. She found it was fairly easy to get started, even though she's new with test automation. Taking a pyramid approach, she starts by beefing up the existing unit tests. And then she moves on to using requests for REST API testing and Selenium WebDriver for WebUI testing. She puts code in the same repository as the product code so that they can be version controlled together. However, after writing a number of tests, Amanda discovers some frustrations. She meets up with Andy for bubble tea to pick his mind. Andy, she says, how do I avoid duplicating so much test code? So many of these test steps are the same. It's very repetitive. I just need to make minor tweaks here and there. Code duplication is code cancer, Andy replies. Have you heard about property testing with hypothesis? It could help you with your unit testing. Hypothesis automatically runs tests against a wide range of scenarios so you don't need to hard code variations. Also, to avoid duplication with feature level tests, try using Gherkin with the PyTest BDD plugin. You can write test scenarios in plain language using given-when-then scenarios. And those will be glued to Python functions that will execute like a test script. Steps can be reused by different scenarios. And writing in Gherkin first helps you mentally separate test cases from test code. So you can focus on writing good tests. I'll definitely check out hypothesis in PyTest BDD, Amanda says. I think there were talks about them at PyCon 2018. Now, here's another question. How have you handled test data in the past? My unit tests used PyTest's monkey patch fixture to mock data. But I'm struggling to find good ways to manage data for bamboozle systems, for black box testing. Test data is definitely hard to maintain. Try to keep test data to a minimum. For most tests, data needs are actually pretty small. Tests should set up and clean up whenever they need, such as database records or custom settings. Some data, however, like user accounts, are expensive to repair and should be handled outside of the tests and treated as a writing state. It may also be good to periodically set up and regenerate that data to make sure your systems are clean after any unexpected damage could happen. That makes sense, Amanda says. I've also been storing inputs like URLs and usernames and local config files so that they aren't hard-coded into the test code. Awesome, Andy says. That's great. You may also want to consider using some sort of key management service like AWS KMS or Azure Key Vault. That way your passwords will be secure and your config files will not go stale. Amanda goes back to bamboozle again and she uses what she learned to improve her test automation project. It becomes her full-time work and she transitions unit test work to other developers so she can focus exclusively on the future tests. She does so well that her team keeps asking for more and more tests. Despite training other developers to help write tests, she becomes buried by the backlog. Amanda takes a step back and asks herself, what tests should we automate? Her answer is simple. Automation is expensive, so focus on the highest return on investment. Amanda writes guidelines for her team to avoid getting crushed by an impractical 100% test automation goal. She recommends automating happy paths for core features that will be run repeatedly. She also recommends avoiding edge cases, one-off checks, and visuals because those would be much easier for an expert to log in and try manually. Amanda also recognizes that these are simply guidelines for her team and her project at bamboozle and they may not necessarily apply universally to all other circumstances. However, even after narrowing the scope of the test made, the number of tests is still quite large. Amanda's team has hundreds of feature tests that will balloon to thousands in the upcoming year. The service level tests now take about ten minutes to run and web tests take over an hour. It's no longer practical for developers to run the test on their local machine. So Amanda calls a meeting with her friend, Otto the Panda, another developer at bamboozle and she asks, how can we reduce the burden of automating tests, or how can we reduce the burden that automating tests puts on our development team? How can we reduce this burden? Otto says, it would be really nice if we could run these tests as part of a continuous integration and delivery pipeline. Then, every code change I make as a developer would become automatically tested once it gets pushed. The only problem we still have then would be that the tests take a long time to run. What if we run the tests in parallel using PyTest ExTest, Amanda asks? It could plug directly into our existing framework. And parallel testing, even though it would be hard on our laptops, would be much easier to scale in a CI environment with shared compute resources. I can make sure that the test code is thread safe and would have no collisions such as web logins. And thankfully, PyTest fixtures can help us set up and tear down everything cleanly and safely. If necessary, we could even set up Selenium Grid to scale our test automation execution across multiple nodes with multiple platforms and multiple browser versions. Or we could also look into platform as a service providers, such as Sauce Labs or Browser Stack. Otto replies, that would be awesome. We also need to make sure feature owners are automatically notified of test failures. That way, they can triage any failures immediately, and we get that fast feedback cycle. Amanda and Otto team up to make it happen. Amanda updates the automation code to use PyTest ExTest and Otto adds new jobs to their existing Jenkins build server for running test suites. They also set up a report portal instance to their test results on the dashboard. Tests now complete and a fraction of the time and developers get failure notifications immediately. Unfortunately though, adopting true CI brings unforeseen consequences of scale. More test runs means more issues are uncovered. While most test runs are fine, some start failing intermittently. Ouch! And that is one of the classic problems of test automation. There are four common reasons we see this. It could be that the feature under test has a really tricky bug. It could be the feature under test was... the feature itself was updated without also updating the test code. It could be the automation didn't handle an unexpected condition, such as a timeout. Or, EGAD the worst, the automation itself has a bug. None of these though are good outcomes. Unfortunately as well, the failures leave a bad impression on the team. Some of Amanda's coworkers stopped triaging test failures and they even suggest skipping the tests all together. Amanda doubles down. How can we make tests more reliable? She asks herself. First, she realizes that continuous integration is the production environment for tests and should be treated as such. And I'm going to repeat that again. CI is production for test automation. Test code should undergo the same best practices as product code. Including coding standards and review. Share test framework components should also have unit tests because they're shared by multiple things. Secondly, Amanda discovers that the most common points of failure are remote calls and race conditions. She tunes the framework to have appropriate timeouts, immediate retries, and helpful failure messages when problems arise. She makes it correct what it can and safely abort when it cannot. She also looks into something called contract testing which uses mocks for services to avoid intermittent network problems. Third, Amanda reaches out to her team members for better collaboration because it starts with the team. They institute better processes for flagging feature changes that will affect tests and also for being gracious when failures happen on both sides. As a result of these three improvements, automation becomes more reliable and confidence in the whole test solution has been restored. By this point in our story, the automation solution is quite mature. New tests are added every sprint and results provide fast feedback to the development team. Other developers like Otto join Amanda in automating tests at all layers of the pyramid as part of their natural development workflow. Quality becomes centric to the team's vision. And that is Amanda's test automation journey. There's so much more I could say about all of this. A half hour slot is just not enough time. But I do want to share some resources. If this talk was meaningful to you and you want to go learn some more, this is the slide to take the pictures of. Just disclaimer, boom. If you want to learn more, I have my blog at Automation Panda. Yes, I'm biased towards that one because it's mine. I also recommend checking out Test Automation University as well as the Ministry of Testing. The three books and their authors are listed here as well. Those are the ones I mentioned earlier in the slides. And at the bottom I've also listed all the different frameworks and packages and such that I recommended as part of Amanda's journey. So you can learn more about those. And so yeah, thank you so much for inviting me here to Texas. This is my second time in Austin. If you have any other questions or want to talk more, please come see me whenever I'll be here both days. You can reach out to me on Twitter at Automation Panda. You can contact me through my blog. Thank you so much. I appreciate it.