 poll here. How many people work in some sort of QA or test capacity? S, dead, set, something. Okay, we have a few folks here. How many people work in, like, operations, SRE kind of thing? Okay, much more. Good. I've tailored the talk correctly then. Developers? I'm sorry. And last, but not least management, anybody here? A couple folks. There's some stuff for you. All right, let's get this rolling. Why your next QA job actually will be in operations? So who am I? I've been in the tech scene for 20-plus years, kind of all around the shop. Mostly in QA, that's where I started, but other things in delivery platforms. I work at a small clinical research organization called Quintel's IMS. Only about 60,000 people. That's kind of my new gig. Before that, some startups, some IoT stuff, 12 years at Red Hat, and I began my career at Rockwell Automation doing industrial PLC programming software. I'm really actually passionate about engineering processes in Kanban. I have a background in philosophy, so I did not get a CS degree. I do occasionally DJ, and I love whiskey, especially Japanese whiskey, some of my info there. So the preface, anything I'm going to say is my opinion and does not represent employers past, present, or future. So this talk really came to me after, like, Jeff Susno's continuous quality, what DevOps means for QA. That was done in 2013. I highly, highly recommend it. Go check out the videos of it, and if you're actually more of a QA person and see, okay, what does this really mean for me, and how does it do it, he really says it much better than I can. That being said, I kind of build off of that and kind of talk about the metatrends, if you will, of where I see these things going. So what I'm going to posit is today's pipelines, infrastructure has killed dead traditional testing in organizations, and if you still do it, you won't be for long. The flip side of that, which should be obvious to folks, this whole new infrastructure as code thing, that's code. Probably should be tested. And lastly, for those of us who are still in the QA mindset, there's a space for us in this future. Let me talk a little bit about who, like, testers are at core, because we're a little bit different. We're a little bit strange. So this chapter in beautiful testing by Linda Wilkinson called, was it good for you? It kind of goes out and it's really kind of lovely. As a tester, it warmed my heart that somebody actually got me. So really, that's kind of abstracted into some of the qualities of people who like to break things. So a lot of times, since we see broken software all the time, we end up having a really kind of twisted sense of humor about breakages. You know, personally, seven ones make me laugh. It's great. It's a production software. It's a way to pull up. That is freaking hilarious. And, you know, we in general like breaking things. It's like, look, look, really, it's broken. We especially like taking that back to Devs and say, hey, guess what? Doesn't work. We're very curious about things, especially about how we can break things and like doing experimental things to, well, break things and, you know, get more data back on how the system or product is working. We often have to pick up things very quickly. And we're not really political animals. We're not going to say, you know, that code really, I know you tried hard. It really isn't good. We're most likely going to say, hey, it's busted here, here and here. And let's move on. So we're not really going to sugarcoat, you know, where our release is or something like that for management. Management really wants this release to go out on time. We have this really important customer. And it's like, guys, it's just busted. I'm sorry. I didn't write the code. It's broken. So we're going to be kind of truth tellers. And we question everything. We don't trust anybody. I like to say software sucks. We try to make it suck less. So what really is QA's job? And for QA folks in here, you can recognize that this is not your job. You're not a quality cop. We don't have any rest powers. We're not a release, but we don't bless releases. It's like, yes, this release may go out. We're not a gatekeeper. We don't have the business context to say, yes, we should release this or not. That's the business's job. And we definitely, definitely are not the people who insure, we're not quality insurance. So we don't make sure that the release ships with no bugs. What is our job then? If it's not that. And people think sometimes that's what we do. But what we really do is this. Jeff Susser puts it as QA as a mirror, as a boundary spanning mirror. The kind of the function that has the holistic view of the system as a whole and its parts, right? So it's trying to reflect and focus things about the system to all the other parts of the business. I kind of boil it down a little bit. And it's to inform the business that the state, what the state of the product is, in order for it to make intelligent business decisions. So it's like, you come to your testers, you come to QA, and QA tells you what's up. So do that by designing, running tests and experiments. That's really important test design. No one likes to talk about that as well as results interpretation, right? So you don't get bad information back. And the biggest thing folks can do is actually fight confirmation bias. So how does it that job relate to ops? Like traditionally, like QA and ops should be like besties totally, right? You know, QA is like, is this thing correct? Did the defect get out and affect a customer? Ops looking at production, right? Is this thing going to go down? Is this thing going to be stable? Am I going to get paged at 4am? Right? If it breaks in production, it affects both groups, right? Usually QA gets hammered. Why didn't you catch this? Ops is up at, you know, 3am pagers going off. Oh, production is on fire, right? Both of us, both of the groups distrust development. That's like, hey, yeah, just push us out. No problem. No problem. Did pass test? No. No, but but it's fine. It's fine. Just hot fix it, hot fix it. Hot fix goes beyond both groups and we're just like, what the hell is this? And both in a lot of organizations are undervalued by the business until something goes horribly, horribly wrong. Way back in the time of dinosaurs, the 90s. If you think of like waterfall and like traditional QA and silos, you your QA group, you're just sitting here, you get a bunch of requirements. If you're lucky, code via trebuchet over the wall, slams on you. It's like, oh, we're supposed to get this three weeks ago and we only have a week now. Consider less than development or worse yet. Hey, that developer guy that came in for an interview, didn't do so well. Put him on the test team. Terrible, terrible anti pattern. A lot of manual testing here. Before the days of like really widespread automation tools and frameworks. But really part of the job was to proxy the customer inside of your engineering group. So your testers and your QA department had to think like, oh, what is the actual customer going to do? We're really optimizing for meantime between failure. So why are we doing that? Because the cost of failure is so high in these days. We're doing physical products. You have like supply chains. If you have to make a change, it's going to cost a lot of time and money. So you optimize for like, okay, make sure that defect isn't there, is not there. So you break things inside so your customer doesn't have to. So long days agile comes along, it's like, oh, somebody writes a manifesto. Of course, there's going to be a revolution. And at that time, you start to get online distribution, right? You move away from CDs and floppy disks and you go to digital downloads. Test automation at this time really kind of gets more and more widespread. So you have like a old SQA robot, which is now rational IBM, and like, all these things start to come together, right? So the idea comes up just like, okay, we're not going to have separate things, we're going to put QA on the scrum team. That's going to be great. And you'll just sit and write the test. Maybe you're doing in cucumber or something like that. Or the other thing, let's just outsource it, right? It's not super important. So we'll just throw it away and have it done over there. And you know, it'll be great. That's a brilliant business decision. But another important thing is the customer gets closer to the engineering department. At this time, operations, you ops people come out of the back office and you become more important, right? So you know, that you're important to the business because you're delivering online and you're taking care of the systems that do that. The platform itself is the product, right? So especially like in e-commerce or something, your platform is the product, the thing that you're selling. So the ability to deploy those changes, shorten those maintenance windows, not have a lot of downtime, affect that change becomes easier and has a lot more business impact. It's visible. And at the time, you guys got it more important, we kind of got less in the test end of things. 2005, that came out, right? 2005. That's how long this kind of change came, you know, has been. So people start doing MVPs, right? So you don't have to have this fully baked thing done. And you're trying to learn as you go. So you don't have to, it's a process. You don't have to have it done all up front. So you're moving away from that kind of waterfall and spec and say, yeah, we can move faster, we can iterate. Meantime of remediation becomes the most important thing. You don't care that it breaks as much if you can fix the problem. Cost of failure is lower. No real physical product or supply chains to worry about. To turn something, you're just kind of pushing code to a server. It's very easy. You get more customer feedback. At this time, everything kind of gets reduced to automation. In the QA space, it's like, oh wait, QA test, you guys just write Selenium code. That's all you do. And we kind of internalize that too. We kind of bought into that, right? So the thing that comes after that, wait, if we're just writing code, why don't we just let the developers do it? We don't need you guys. You're done. And then we get trapped in our tools. We stop thinking about the system. We're only, here's a spec, here's a page object model. We write that. It gets worse in cloud, right? You know, all those constraints start to be peeled off. Time to market becomes the most important thing. Correctness kind of goes, okay, we can fix that later. You know, deployment automation starts to come in. More code. We need more code. We need more speed. This leads to where I think we are today. I like to call this the automation apocalypse. Companies like here, Mabel, there's also some other companies, Functionize, Appletools. Machine learning and computer vision has gotten to the point where any kind of rote testing can be done. Like 20 years ago when I was doing this, originally, you know, it's like, man, you always need a human because you'll never be able, you'll have to off by one pixel screenshot comparisons, that sort of thing. Right now you can like upload your swagger into something. It'll generate tests for you. And then you can hook that up to your load testing automation. Or just upload your gold image. A third-party service is going to come in, like hit your things and see your changes and you're done. So for all of us QA folks who are like, yeah, we just do Selenium, obsolete. But apps, man, it's your golden time right now. You have all this stuff, that pipeline that you have, all that automation, that is your product. Reliable, repeatable delivery, core business capability. Down to, you know, are we down? No, downtime maintenance window? No, no, we don't do that anymore. Right. But you're using so many different third party systems and frameworks. You're you're looking a lot more like a software system integrator than a sysadmin. Right. So you have to do APIs over here. Here's Jenkins and you know, you have all this AWS and blah blah blah. All of these things are spread out. Do we need something with the mindset of experimental curious? Do we still need that? Is there a place for that here? I think there is. Because the business right now doesn't know what it's doing. It's starving for information. They throw out an MVP and they're like, I need feedback. I don't know. That should we use this, should we use that? So that information is critical. So the business can either pivot or get the next iteration of its product out. So I think that mindset, that curious experimental mindset, it's going to be valuable. But it's not going to be QA. It might even be called test. Right. All this infra needs to be tested. So how many of your men's here, how many people test your pipeline, right, as well as your product have like CICD on your infrastructure code? A few, a few. So how many people think you should actually do that? Right. So there's work to be done there. Performance and security become much, much more important. And complex distributed systems, like in your cloud providers, that you need to confirm that behavior and you can't really model it inside. I'll say it again because it's really important. Test, test, test infrastructure code. That resilience, actually the resilience of your distribution pipeline should be better than your product. Right. Because that is what makes it so you can go fast on the product side. Software supply chain also very important. So from a perf point of view, we're like throwing things all over the place and we need to know that not just on how fast we can reply to some request, but really for rightsizing infrastructure, right, to actually do performance, you know, profiling, because that'll cost real money. How many people in a cloud provider ever had something that was overinstanced? Yeah, you see that and it's like money on fire. It's just like crazy. So yeah, but how do you how do you get the right size instances? You test that. Exactly. That was my security slide. Didn't come through. But do we need security testing? Equifax. Enough said. Remember, pen testing has testing in the name. So there is more testing happening over there. This will become more and more important. God forbid you work in a compliance regime. The results of not how much how much trouble do you think Equifax is in, right, with its regulators? You know, you're gonna you're gonna probably see some very large fines, didn't take care of the PCI or, you know, you work in HIPAA, you start leaking patient information, very bad things for you, but it can literally kill your business. Maybe you should test that that audit in compliance, what you're going to be checked on is clean. So your business doesn't die. The business also wants more and more information from users, right? So that's, you know, your business analysts, you have all these things to proxy it up front, you know, back in the old days. But now you can actually do tests on your users, right? That's A-B testing. How many folks are in shops that do A-B? A few. That's where test design becomes important. So you get good information on A-B. The same thing with real user monitoring, like rum. So chaos, chaos, that's supposed to say chaos engineering, monkeys for the monkey god. Chaos engineering. All right. Do you think Netflix can actually model the internet, like in staging to test things? No, it's like too big and too complex. I think even for some of our things, it's very hard to mimic now with so many third party services and so much variability. What's really, what's really going on? So you're going to actually have to test in production. QA people, you love this. It's like, I get to break production? That is awesome. But from the upside, that resiliency and recoverability is very important for you to be actually able to do this. Otherwise you're just burning down your own house. So it's evolution on the path. So those destructive experiments, right, they prove the resiliency of your system. The exception is the rule. So I said everything moved from meantime between failure out to, you know, recovery and we don't really care, things can fail, we're resilient, blah, blah, blah. IoT kind of flips it on its head. You're dealing with hardware. You have supply chain users might not update things. You probably want to test that better and have go a little bit more on MBTF before it goes out. Because your cost of failure is higher. Try to get replacements out. You have fiscal products, security, people aren't going to update. You don't control the update, the user has to get it. So in conclusion, more code we get in ops related stuff, more code needs to be tested. So we should know that QA people, we should start to learn more operations things. That mindset is going to be super, super helpful. And really for the business folks, what that cost of failure is should guide how much you do. And that is all. So your talk is actually really timely for me personally, because I just recently made a switch from QA to DevOps like a month and a half ago, which I've really been enjoying. It's been fantastic. One comment and then another question. And I found out yesterday that all of the QA team that I used to work with, they've been let go. Yep. And so to your point, this is actually happening. It's real. So I've inherited a bunch of we do all our infrastructure in Terraform. Yep. And the idea of testing that is just such a cool idea. I think I'd like to go to my boss and get that going. How would you approach beginning something like that? Usually our testing processes, we do a Terraform reply. And if things Terraform plan, yeah, things look like what we want them to look like, it's tested. And but obviously that's not as resilient or as thorough as it could be. You look to sound like you have an answer. So being here and walking around the room, seeing all the different vendor stuff, a lot of it is all focused from a DevOps perspective on the delivery pipelines, the monitoring, the coming from a transition from that QA background to trying to fill out all the pieces in the pie. Where do you see there's not a whole lot of automated testing people out there kind of filling in that group? Do you see some of those traditional, you know, cucumbers or, you know, selenium or being carried forward from an automated testing perspective to do faster do that? Or are those going the way of the dinosaur as well? Or yeah, I really think for some of that, it will as machine learning and computer vision solutions for that type of testing really start to get more mature. Right now I'm just seeing the start of it. I, you know, I was out at Vlase San Jose, and I just ran it to these, you know, companies, I'm like, Oh, my God. This is this is incredible. This actually works. You know, so but I think from if you think about these are just frameworks in if you understand frameworks and you understand test design, you can easily jump to another framework or another domain. Right. I think a lot of QA folks, a lot of testing folks that do kind of functional automate functional automation now should probably if they're really like that really kind of think about specializing, right? Security are our lost brothers, right? Let's let's help those guys out. Let's, you know, do more performance work. You know, these are translatable skills. You just have to learn a bit different domain. That that actually kind of tied into my question, which could hear because one of the things we have a lot of folks, you know, on the traditional QA side, and they're looking, they're seeing, you know, kind of landscape evolve in front of them. But right now, I feel like we're in a kind of a situation where experience is needed in order for your next gig to happen, right? Like everyone is looking for somebody who's done that, not evolved into. So which what are the suggestions you have for people who have been in their traditional QA format sees the landscape? Like is it self learning and experimentation in their current gig to be able to then find a new one? Or like, how do you make that leap? Having made that leap, it's never, it's never really easy. I think one of the big things is to go to your local DevOps meetup. And they're very welcoming. And you can start to learn that side of the house. It's really easy to bootstrap some self learning to if you if you like that, even better if you have, if you're on the QA team or the test team, if you take more control of your environments at the company you are like so so you can like self service, deploy your own environments, let's say out in the cloud, and really get more deeply involved in how that pipeline works. That will take you steps on the path. Yeah, this is just another thing on the trend I've been seeing of it in the DevOps space is DevOps becoming more jack of all trades and even expert of all trades and kind of a lot of these special specialist roles like QA in DBA roles are getting rolled into and expect the point where they're expecting or a lot of companies are expecting one role in one person to do a lot of different things which require deep expertise. So what are your thoughts on that and how you know, how do we deal with that? Yeah, so that's really kind of a management thing is it's really about team formation. So, you know, you're like have T shaped people. So you have a broad skill set and maybe one or two, one go deep or like pie because there's two and then there's one over thing. So your team you want a what I like to call a Tetris shaped team, right? So you have all the different blocks and specialties. And they just kind of fit together into that deep, solid square. But to actually do that, you have to have management that understands that's how you build build the team. I think these are kind of all trailing into each other from a traditional Dev and test standpoint where you have an organization where it's hard to sell QA to management because nobody understands that you actually need to test software, right? And then transitioning into estates from a standard QA or STE, right? And now we have this DevOps thing, which is a difficult sell into management already. Exactly. How do we, how do we approach selling QA for for the pipeline or QA for the infrastructure? You know, if I had figured that out, I would have a consultancy dig gig somewhere and make it a lot. I think we'd all we'd all work together. Yeah, ready for experiments. Is that it one more? That's it. Catch me on the hallway. I love to just talk about this. Thank you.