 Hello everybody. I'm calling in from Chicago, Illinois. Today you're here for setting your team up for success so anybody can write tests. Let me tell you a little bit about me. So my name is Zach. All right, these are my links actually. So I've got my GitHub and Twitter's up there, my LinkedIn. You should be posted in the slides that I'll share after the presentation if you want to follow what I do. So I'm the lead software development engineer and test at review trackers in Chicago. I'm also the organizer of the Selenium Meetup Group here in Chicago. I speak at some of these kinds of things. I love to cook and I'm trying to learn piano. That's kind of one of my quarantine goals. I'll also say I've been to travel to India before. I was at the last Selenium conference in 2018 and would love to be back over there in 2020. God willing. So yeah, that's a little bit about me. So here's today's agenda. The presentation will be divided into two areas. To set yourselves up for success. First, I'll talk about failing transparently. What it means to fail in a way that people know what's going on. People understand enough about the system under test to act on it. And the tests are transparent in how they're failing. So that's the first section. The second section will I'll be talking about creating a culture of testing at your organization so that anybody can write tests, contribute to tests, triage tests, etc. And I'll also say yes please type your questions in the Q&A section. I will be monitoring that. And as my presentation is divided into two parts, I will stop in the middle and answer questions about each of those parts. So please ask those questions. So before I talk about failing transparently, I'm going to tell a little horror story. I hope you can all relate. So in this scenario, it's late on a Friday. Our developer is actually developing a new feature for the land authority. And it's not the space authority. They're developing a feature for the land authority. They run their continuous deployment tests, the ones right before going to production, and they see this error. They are not touching the space authority. So they're a little bit confused why their land authority changes is causing the space authority error. They also don't really know what's going on. They're a little bit confused about the error. They've actually seen it before and didn't understand it then and they don't understand it now. So they ask the group, hey, is it good to move forward? Can I deploy the land authority? The rest of the organization gives the thumbs up. They've seen it before. They don't really know what's going on. The test engineers are on vacation. The rest of the QA staff wasn't asked about it. So they move forward with the land authority change despite the failed tests. They get all the thumbs up as SpongeBob alludes to. And then, oh no, it turns out the land authority was deeply important for the space authority. That failed test was a real true failure and the plan blew up. So we've all been there, right? You know, it happens. So I think what really happens there is like a failure of the tests. The tests failed to communicate like the importance of what was failing. They were hard to understand in terms of what was failing. And because they weren't easy to interpret, people felt complacent. They felt okay to move on. And there's that kind of institutional buy and they got the thumbs up from the other developers saying, oh, it's cool to ignore the test. But it turns out that the test was was showing something important. So this talk will be about avoiding this doomsday scenario. Hopefully your software testing scenarios aren't as dramatic. But okay, so here are the kind of the five sections they'll talk about with regards to failing transparently. So the first way to fail transparently and clearly is to be a master of failure messages. What do I mean by that? So each of you at your organizations, you have a suite of tests, and you're probably using assertion library. And what I would suggest is I would suggest that you become an expert in what your assertion library makes failures look like. This this image is a picture of what a Python high test error looks like or assertion error looks like. So it's pretty good here. It shows you exactly where the assertion fails. And it kind of shows you how for is meant equal five and it doesn't. So I think this one's decent. I've got a couple others. This is a J unit failure. It has a little bit of English expected six but was negative six, not bad, not bad. But I would and then let's do one more. Okay, so this is a Ruby failure, I program in Ruby. So I'm very used to looking at these are spec assertion failures. They kind of show what you expect and we got and there's some nice white space in between and it shows you the comparison operator, which I was like. So again, I would posit to become an expert in what your assertion libraries, what their failures look like, you know, explore all of them explore every kind of failure that your search and library can make and become an expert at reading those to know what are wrong. Here's some other examples of like the frameworks making errors look different. So these are screenshots from the site prism framework, which is a framework that we use our view trackers with with Ruby. And they're expected has text returns true got false. That comes from the specific assertion library. It does not come from the testing library. So yeah, you know, tests can be complicated, you know, you have a framework, you have Selenium, you have an assertion library, you have a framework to interact with Selenium. And like, all these different things can raise different assertion failures. So yeah, be familiar with not only your testing frameworks, assertion errors, be familiar with the custom ones that like whatever Selenium framework using can can put forth. This is another one from a Java testing package. I can't remember the name of it at the moment. But it looks like this gives you the little snippet of the HTML. So that's pretty cool. So I've talked a little bit about the assertion library errors and becoming an expert in those but an additional thing to be aware of is knowing the difference between code errors and assertion errors. So for example, in this screenshot, this is a no method error. So that's different from an assertion error. I think that if you have a code error, it's more likely that it's a problem with the test itself and not the software under test. But that's not always true. So but it's good to know, like at a glance, if you see something like this, it's it's different than it's a different flavor of error than the assertion type of error. Here's another example of a code error. It's a little bit complicated here. But basically, we were looking for one user row with that find chunk of code, and then a link within the user row. And what happened is the user row find block didn't return any user rows. So then I got this error. So in this case, it's kind of like, yes, I'm getting a code error, but I'm getting a code error because it wasn't able to find any user rows. So it's kind of related to Selenium related to the front end. But you kind of got to read between the lines of the code error to see what's going on. So something I'd also suggest on your journey to becoming to failing transparently is if any of those kinds of failure messages don't cut it, make your own. A lot of testing libraries provide ways to allow for custom failure messages. On the on the left here is Python's way of doing that. It's like the second argument. And on the right are spec expect array to be empty. And then after being empty, you can provide a custom failure message. That's really great. I would suggest to use that. In the message. Okay. Yeah, so I suggest user custom failure messages. And if if those custom failure messages, the thing about them is they should really add to the to the existing failure messages. So I would say these two aren't even great examples. But a great example of custom failure messages is this one. This is one I created at review trackers. So this test is expecting a user to be indexed. And if the user is not indexed, it returns this kind of error. And this error is really great because it tells me how long the test waited for the user to be indexed. It tells me what resource it was. It gives me the ID of the user it expects to be indexed. And then I can go and take that ID and explore my software under test and see what happens. I can, you know, I have some data to like investigate the the situation. And it also shows the raw response from the indexing service. So I would suggest when you know, when you make these failure messages, provide as much information as you can, make them actionable so that you can go dig under the hood of your software under test and find out what we're on. So that is this section. Let's go to the next one. Alright. So another thing I would posit that you need to do to fail transparently is use status checks for the services your test depends on. So what do I mean by that? So to describe status checks, I'm going to talk about this little sample app. So let's pretend we have a shopping cart here where it gives you your item subtotal and the tax and the grand total. This is your shopping cart front end application. So under the hood, in this situation, let's say the shopping cart, your front end hits the shopping cart API, which is tax API to get all of the data that it needs. And then the shopping cart API assembles all the data it needs from the tax API and various databases and then sends it to the front end. So there's kind of two different service dependencies to get the data that you need. So cool. So our tests normally pass for this shopping cart. However, let's say one day you get into work and you see this failure, expectation not met error. And instead of $97, the test is getting $95.45. That's not great. You'd be like, what's going on? And you'd be like, is the error with the shopping cart API? Is the error with the tax API? What's actually going on under the hood? And it's not clear. You'd have to dig into that because because of your infrastructure, you know that there are kind of two dependencies to get this data in the right state. So the fail itself isn't immediately actionable. But if you had status checks, you could create status tricks for them points your test depends on and then run those in the before hook of each test. Status tricks are great. You know, there's there's even entire companies built around this idea of uptime monitoring and status checks. I don't know if any of you have heard of the company Pingdom, but one of Pingdom's main services is just just pangs your website and just make sure that it's up. It'll ping it like every five minutes. So what I'd posit is let's say if you know your test depends on shopping cart API and tax API, set up a status check right before your test runs to ensure that shopping cart is up and tax API is up. So if either of them is not up, you won't even bother with running the test because it wouldn't make sense to even run the test if the service is down. And then in this scenario, ideally, instead of getting that cryptic error where you get 9545, you'd get an error that says tax API is down. So then you can go investigate that instead of your cryptic test failure. So third thing I'll mention with failing transparently is configuring nice test reports. Without nice test reports, there's lots of scrolling, there's no syntax highlighting, there's a lot of clicks to see an associated screenshot and all of that leads to slower triage. That gives me a big headache. I don't like scrolling. I really enjoy syntax highlighting. I'm getting a headache reading that error on the left here. And it takes a more cognitive load to to associate this failure with this screenshot in another browser tab, something like that. I'll have to do some mental work to associate those two things together, especially if they're multiple fails. Now I'll be juggling tabs. I'll have one tab that has the test failure and then a couple other tabs with the screenshots open. So that's kind of hard to balance in my brain. And it's already a stressful situation. So I would posit that to fail transparently, you need to have nice test reports. They're worth spending your time configuring. Why are they worth spending your time on? They create a comment was here for those investigating. Those that are investigating can can look, you know, at at once and see the screenshot, they can see the area of the test code where the test fails, they can see the assertion failure, it'll have nice syntax highlighting, it'll have all the information that they need to triage it out of glance. And it's like I said, it's already a stressful scenario when you're when you're triaging test fails. So I would recommend to get these test reports configured. If your boss is telling you that it's not worth the time to configure nice test reports, because they're like, Oh, well, you got to write all these tests first. And then we'll do we'll, you know, we'll do this in Q4. We'll do this next year. I'll say, no, do it now, because you'll save yourself time triaging tests, like they're they're worth so much more than the time spent configuring them. And you can always tell your boss to talk to me, and I will convince them to to use these nice test reports. At review trackers, we use the RSpec HTML reporter gem. And it's really great. And it does the work of associating that that screenshot with the the syntax and the assertion error and everything in the stack trace. It's all all in one report. It's really great. So in addition to those test reports, we have some of your review trackers that enables us to analyze test health across test runs. And we use a software called report portal, and it's completely open source. I highly recommend it. I really, really love report portal. The screenshot right here is this is a list of tests that failed in the last 30 rounds. And I can see our our failure rate is kind of at a consistent 6.67% not not ideal. We're still, you know, tests are failing two out of 30 times, not great. But, you know, if I ever see a test, you know, fail five out of 30 times or 10 out of 30 times, then I'll be like, All right, something's going on here. And then that'll be my day we'll be investigating why that test is being more flaky. So yeah, I would encourage you to configure something like this so you can get that that analysis across test runs. It's very valuable to have. So in addition to those three things, next topic is using a test logger. I would suggest that it's really important to log what your test is doing as it's doing it so that you can again fail transparently. So what do I mean by a test logger? You know how your software under test has logs? And the developers are always combing through those logs to determine, you know, why there was a server outage on Friday or why why the data isn't present and stuff like that. People are always combing through logs. Well, what I would suggest is that the tests themselves also create logs that log and porn steps in a test case. So that can help you triage the test fail when it happens because you'll know like what the test achieved what what it did and what didn't do and stuff like that. So some examples here of things that your test log can log are it actually successfully logging into the front end. So I have that message right there highlighted. And then another example of something you might want to log is verifying the data is present that a test needs. So why might you want to do that? I review trackers for our ends to ends tests. Sometimes we use an API to do setup for the test, and we'll do that API call in the very beginning of the test. And then that's just set up and the the data that's fetched is happens in an asynchronous way. So in order to be transparent about what's going on in the test, then we we log a message when that API call is done and the data is done fetching in the async way. So if that log message doesn't happen, we know that the data wasn't there. We also assert on that data being there as well. But this is helpful as well because you can see how long it took to get that data. If if one day we see the tests that normally takes 30 seconds is taking three minutes, we can look into those log messages and see, hey, how long did it actually take to to fetch the data in the beginning? How long did it take to do that setup? And if we see those seconds go up in time from the log messages, we'll know that something's up with the with that API and that indirect dependency. So that's one of the reasons I really like using these these test loggers. Some other reasons to use test loggers is when you're investigating a fail that only happens when it's run in parallel or run around CI and stuff like that. In those situations test logs are super helpful. So I can describe a scenario where we use test loggers that are retracers. So as you can see in this GIF, this is our competitors tool and here the end users changing the filter from 12 months to last two years. And we experienced an issue where it's not really shown in the GIF, but we experienced an issue where you'd switch from 12 months to two years and the data actually wouldn't change. And we were kind of a little bit baffled. Like sometimes the data was changing, sometimes the data wasn't changing. We were trying to investigate why the data would change or wouldn't change. And one of the things we did was implement some test logging. So all that we did is we just logged what the test saw as it went. And what that does is it just like provides a bigger picture of the data that the test is seeing. So here's some examples of the test was grabbing all the data from the table before the filter, like before the change was applied. So here it grabs all the data for the 12 months and then it grabs the data for the two years. And then by grabbing all the data we can see what the test had saw and then when if the assertion were to fail we can dig into that more and see what did actually change or didn't change. So that's the really cool way to do triage. So test logs also mean that they're time stamps and you can correlate those time stamps with a message from the actual software under test logs. That's usually valuable. If here I'm highlighting a time stamp of a scorecard that's a functionality of review trackers generating in 18.7 seconds, if I ever wanted to see that scorecard generating in the software's under test logs I can like look at that time stamp and then compare. And yeah, it's another great way to see what's going on when a test fails. Look at the time stamps from the log messages and correlate those with your software under test. So one last thing that we do with our test log review trackers is we log the credentials of the test account used for each test. That is incredibly helpful because then you can log in and see the state of the system when the test failed. I do this on a regular basis. It's pretty much the first thing I ever do if I see a test failure is I will look at the logs and I will look at the admin username and the password and the account ID and then investigate the state of the system when it failed. I'll log in and I'll be I'll explore what was going on. So in the case that I described earlier this case when we were seeing that the filter change wasn't reflecting I went and logged into that account and filtered for myself and I saw it yeah and I saw yeah the data isn't changing when I changed from 12 months to two years and I actually was able to see it with my own eyes. So that's always really great to do. The last way that I want to posit to fail transparently is to save test artifacts. What do I mean by test artifacts? If your test grabs emails, saves PDFs, CSVs, any kind of if your test grabs any kind of data I would save them somewhere save them to like a file server. Save them somewhere where those files are accessible after the test run is finished. Save them somewhere that persists. And the reason why you would want to do that is that because that data can help you investigate fails. And not only that sometimes it's just helpful to see the data that the test uses. So here's a screenshot from our our Slack test runner and although this has passed we even put a link to the test artifacts and if you were to click that link that would take you to a place like this where you have just a collection of all the CSVs PDFs everything that the test had downloaded and made assertions on in the test life. So here's like the entire body of the email that it checked. So yeah so this is great if that email assertion were to fail you can actually look at the email and see what the test saw. If the PDF assertion were to fail you can open the PDF and see what the test saw. So I think test artifacts are really great save as many of them as you can. Data never hurt anybody. Storage is cheap these days to get an Amazon file server is really cheap or an EC2 instance that just stores files is really cheap. So I would suggest you to store any kind of data that you can because you'll never know when you'll need it. You'll never know when you need it to investigate those kinds of files. So that is the first half of my presentation. Let me see if there are any questions in the Q&A. Okay it doesn't look like there's anything in the Q&A. Okay since there's no questions I'll move on to the second part of the presentation. Oh sorry can you say that one more time? Last 15 minutes. Okay thanks Alright so creating that that culture of success and culture testing here's how you do it well before I talk here's how you do it I'm gonna describe a story. So it's early on a Monday and I'm a developer and this is our feedback form and it has five it has one to five stars and management has said make it have one to ten stars we want to have the ability to to log feedback from once ten and you're like all right management I'll do it sounds easy so you spend your time really on a Monday doing the development for that feature and it's really easy you just change a couple things yada yada yada you're done you did all your code changes but now you're like hmm I got to update the tests don't I so you go in to look at the tests and the tests are more confusing than the code that describes the behavior. This is a screenshot from an actual test that reviewed trackers before I started and so there's a complicated sequel going on you're like wait where do I edit this to make it be ten stars and check that there's ten stars this is the area isn't it. Here it's doing something with five stars how would I change it to do things for ten stars. I'm really confused I've lost all motivation to contribute to the tests I have to wait till the test engineer is online and that made me angry you know I've had a really great start to my day making the change in the code but it should never be hard to make a change in the tests associated with the code that's just an annoyance so we want to avoid the scenario we want to create a culture of testing we want to make it easy for developers to to write those tests so how are we going to do it first way to do it is easy factories so why is the factories UI engineers to do the setup in a uniform way have an easy time doing it and they factories I'll posit the factory should have default arguments and be easily configurable so that anybody can jump in and create the data that they need in like a second to do the test they shouldn't have to know like the vagaries or the details of the system to to create that data to to become test so we have a factory review trackers called accounts with various users and this one method call creates a new account seven users 30 views all this data it's just one method and it creates all the stuff you might need for a test and it also accepts keyword arguments so that you know if you wanted to only create three users you can pass in users three and then it only create three if you don't want it to create reviews you can say reviews false and it won't create reviews so it's infinitely configurable and each resource itself has its own factory tags his own templates his own groups his own yada yada so here are some pros of doing it that way then it's easy to configure the data to test one method automatically configures all these associated records and in terms of maintenance you can change the definitions for your database for the factory all in one place to that according to the database definitions you don't have to go into multiple so like if you change whatever review looks like you add a new column to the reviews table you don't have to change your test code everywhere you just change it in one place the cons of doing these factories is it does create extra data you might not need all these associated records for your test but we're okay with that area checkers because of the time saving that that method of call makes and yet factory still depends on database definitions so like if you if you change your views table you still need to update your factory to reflect the change in their views table so you that is not removed but it's still easier to to have one method to create all the data so the sign of a good factory what is the sign of good factory it's when a dev or a qa engineer uses it to create data for manual testing so we had a piece of development error view trackers where we were changing the api that would be used under the hood only if your account had over a million reviews and uh in order to test this uh in ci and on your local computer what the developer did was they use the factory and just created a for loop uh looping a million times using the factory to create a million reviews and that took five minutes so now they had the setup for those million reviews so that they can easily test the code uh so that's when you know your factory is doing good work um so to create some more of that testing culture uh would posit to be damp not dry what does that mean uh the phrase damp means descriptive and meaningful phrases um not dry which is don't repeat yourself and dry is kind of a tenant of developers um to um refactor your code and you know if you see the same code doing things over and over again um refactor that code to only be one method and have arguments and and and uh i would say since tests don't have tests um it's important to to uh be more descriptive and don't worry about dry um don't refactor your code so like here's an example so this is a dry test and why is it dry it's dry because um that method register all users um you can reuse that across tests because what it does is it just takes in the users objects and it just registers them um but when you do it this way the method register all users it's not clear which users it's registering where i guess it's registering all of them but like um it's not as readable uh because now if i want to know which users are being registered i have to go into the setup block and read that users uh read the users up there um so it's not as reasonable to to make it dry so i'll suggest to make your test damp and uh this is what a damp version of that test would look like uh i'm explicitly creating the users i'm explicitly registering them i'm not using uh methods that do those actions um but since it's important to the test case we're registering them right there um and i would posit that this is the the better way to write tests um you don't have to go into another file and then understand what that helper method does um i'm sure roval heard of the phrase copy paste programming and i'll posit that that's okay to do in test repose and uh really what i like to do is copy paste tweak programming and here's an example of me doing copy paste tweak programming like i took um that case from before where we're um we're registering multiple users and now the new test case is the test can't register multiple users with the same email so it's a little tweak on the last test case and um i'm creating two users with the same email both of them have the email alice they're both being registered and then the assertions are changed a little bit instead of asserting that both are registered i'm asserting that the one of the duplicates isn't registered so here i have copied and pasted and tweaked um so creating culture testing so i suggest create it's really important to create documentation for contributing to the test repository um why is important to create documentation uh selenium can do a lot but as an api it has its quirks we all know that it's a little bit quirky um it's not the easiest to use if you don't have experience with it um i'm sure you've all seen these kinds of stack overflow posts there's hundreds of them uh and just to show that selenium is a little bit complicated under the hood um this is from the w3c specification for like how a selenium click works and there's a lot of steps in there uh selenium does a lot of stuff so um what i'm trying to say is selenium can be a little bit hard to use at times so what i would suggest is you providing documentation that makes contributing to your tests easier so here's some examples of documentation we have a review trackers for for making contributing to tests easier um if developers see the documentation they'll be able to speed through it and write those tests because uh the docs will help them do it um it's just kind of a good treasure trove of knowledge to have um i would also pause it to create test helpers that makes selenium less daunting uh it wouldn't be ideal if each engineer wrote their own code to scroll to a specific place or fill in a field so here are some examples of of um tests uh helpers that i've made for review trackers to to do those kind of simple selenium actions because selenium can be a little challenging um here's another example of fill and input this is a custom method we have here review trackers and then um i'll also suggest that you all use common coding practices like doc strings and comments so engineers can understand your tests easily i would say that the documentation doesn't only need to be in document form it can also be in your code so yeah use those doc strings use the commenting functionality their developers are familiar with uh to so they can use your utilities um so in addition to how to contribute i would suggest to have documents on the services the tests use and how to investigate their outages um why should we have that kind of documentation isn't that kind of overkill i would say no because when a services down it's high pressure uh we don't want that to be the time to learn about a service and a doc can really help a treasurer feel less stressed when they're investigating things so um in uh the doc the status checks in your test service documentation for like what services that test use so that's really great so if you then have documentation for how to investigate each of those services um for example like where to check the logs where to see the uptime how to connect to the database uh how is the service different in each environment you know whatever makes investigating easier like put that in a document for each of those services and then um have that documentation accessible so that if the service is down if the service is having a test issue people can investigate it um our view trackers we even have general steps uh like this uh for just triaging test failures in in the first place and i kind of got this idea from the book the checklist manifesto uh like even so when you're when you're doing stressful tasks it's helpful to have checklists because it helps you stay on task when they when there's a stressful situation um without a checklist you you have a tendency to get off track or um not stay focused but the the checklists in a in a high pressure scenario the checklists help you stay focused um so yeah i would suggest creating a checklist like this this to um to help in with those investigators so more about the culture creating the culture testing i would suggest that we have strong programming paradigms in our test code um treat the formatting in your test code the same as production code you know run your linters have good naming conventions file structure design patterns um use all those things don't don't tell people oh it's just test code we can do it however we want um i would say have the exact same standards as you do your production code yeah um okay two more slides cool i'm like near in the end um i would suggest you configure the test similar to the production code itself if possible um so that there's less context switching for the developer who's contributing to the tests um that's really helpful um context switching reduces productivity so if you've got javascript uh code javascript production code make the tests written in javascript etc and use the same file structure stuff like that um other people should be able to read a test and know what's going on test code is code blah blah blah um avoid conditional logic in your test cases um i would posit that instead of doing the test case like this where it handles city search and google and facebook and yelp in one test uh do it like this where um the test knows that it's handling a google review and it's only asserting about that google review um conditional logic in test cases makes them harder to interpret and harder to follow the flow so yeah make sure that you eliminate that kind of conditional logic um i'll also say if there's a test that does a different thing in a different environment um separated into two different tests and skip the irrelevant tests in the irrelevant environment so here is an example of a test where in docker and local it skips an assertion and um i would suggest instead of doing it like that do it like this where now i have two versions of that test and um it does different things based on the different environment that it's in cool um i'll stop there i'm sorry i didn't get to all of the content but um the other slides are really good so you'll be able to get through it i hope this is good so i got a comment from samar who said saving passwords in logs is not a good practice and what i would say to samar is you're 100 correct uh it is not good practice uh however um for these tests these are only run in lower environments and each test is segmented to um a single account and that account only has certain kinds of permissions with our access control scheme so um there's no risk in us providing those uh passwords no attacker could do anything with those passwords because all that they do is they give access to that one account and that's only present in our lower environments and all the data for that account was also created by that test itself anyways so like thank you jack for sharing your experience with us today