 about QA and, like, why you need QA people and why you need them now. First of all, like, welcome to Drupal South and it's great to be here. It's great to have you guys. Let's see how it goes and how you guys are going to feel about the talk. First of all, my name is Daniel Carvalinho and, yeah, any variations of that, so don't worry about pronouncing my name. I'm from Brazil and even Brazilians don't get it right. So, yeah, whatever. I'm living in Australia, like, for the last around two years and I've been working with Aqua as a technical aqua manager for, like, two months now. On Drupal, I'm DSCL, contributing, translating for Portuguese and reviewing stuff. I've been working with PHP for around 20 years and Drupal 10 years this year and all of this is just to say that I've seen, like, oops, I've seen things and that you wouldn't believe. First of all, we are going to go through a little bit of history, like, a little story on the QA thing. So, once upon a time, there was digital agents and these digital agents was having, like, a normal day and they received the RFP. So, yeah, RFP is great, let's see the budget. Yes, let's get this money. And so what they do is basically they put, like, maybe, I don't know, sometimes the managing director doing stuff or maybe they will call just a digital producer or a project manager to look at that and lots of times they will create the proposal just asking the developers over his leg on the technical bits that they'd never seen before and, yes, they will get a proposal ready and they will send the proposal and discuss and et cetera and they will get a contract. And once they get a contract, they sign and et cetera, they go, oh, now that we have a contract, now that we have the deal done, now we are going to think about people. No, sorry. And so we are going to discover, like, thinking about people, what do we need to do, like, what we are really doing in this proposal? So now I will get, like, a technical lead or somebody involved to really look into the RFP again and make sure that the guy that sent the proposal didn't mess up. Right? So, yeah, they go over a little bit and then they get to... ...estimated schedule and they think about who are they going to involve. So team allocation. Ah, what about we just get, like, one backend developer and then we get one front-end developer and what about, like, the technical lead or the architect or the project manager? No worries, they are juggling, like, 10 projects. Now they can deal with another one. That's not a problem. But, okay, and who is going to test it? Nobody thinks about testing. Usually they think and then in the proposal, testing is, like, 20% extra over everything. Not really a QA person. So, yeah, the backend guy will test whatever he does by himself and, yeah, it's cool. And the front-end guy is going to test what he has done and it's all good. And then whenever everything is put together, that PM that has other 10 projects is going to go over everything and make sure that everything is working because he has a lot of time to do that. Okay, and what about code review? Who is doing code review? Do we need that? And UAT, do you have time for user acceptance tests? Are you involving the client at some point? No. Yeah, site goes live. All good, perfect. And you guys, like, it's something that happens and happens a lot. I don't know if you guys say that you've never seen it happening. You are probably lying because it did happen and you didn't see it. And the thing is that whenever it gets online, the client comes back and starts to say, oh, there are many things wrong in this layout. How did it pass? And then there are key things in the site that they are not working. Who did test this? And this is, like, extremely slow. So we need to do something. So you have a last chance. You need to get this site ready in four days. All good, all bugs fixed, and everything running perfectly. Or there will be consequences. Probably legal, probably financial only, whatever. So yeah, what do the PM or whoever believes that he's in charge now do? He goes there and gather a lot of resources. So they prepare the team for madness. So everybody is working the full four days over and over. Over time doesn't matter if you want to rest or rest on your table and just do it because we need to get it done. Yeah, but there is light at the end of the tunnel. The thing is that three things that get the client very unhappy are the delays on the deliverer, which usually lead to extra costs, or usually when, depending on the contract, right, there are contracts that the extra time actually leads to costs for the agents, not for the client. But in the end, the agencies to meet the time and the cost, they will end up with lots of bugs. So you are going to get a site live, but there will be a lot of stuff that is going wrong. So what do you do for avoiding this? You need to prepare a QA strategy. And things that are going to usually fail on a project and lead to that issues are things like estimates done wrong and lots of times estimates are done back in the proposal. And whenever it gets to the discovery phase or whatever you want to call that part, what happens is that the team try to meet the estimate instead of reviewing the estimate and saying, yes, this was wrong, you need to go back to the client and say, we didn't expect to do this, this and that, and you need to change that. And I'm assuming that most of people here are working on some sort of agile process, not any waterfall or things like that. I don't believe there is still a lot of agencies doing that. I know there are, but not a lot of them. And the second thing is like I said, whenever the guys are doing the proposal, they think about adding QA as a percentage of the project instead of adding QA as a person that is going to be available for testing it. So in the end, the strategy doesn't really exist because you add a percentage and whenever the client come back and say, oh, this is too expensive, you go there and take the 20% becomes 10%, that 10% becomes 5%, and you don't need QA anymore. So the first part, the first point that loses budget and loses time is the QA because, yes, everybody can test it. So you don't need the thing to be specific for that. And then this leads to the fact that you have insufficient allocation of people. You don't have anybody there for really taking, like really caring about the testing of the project itself. And too many changes, but too many changes I just added there, but in reality, if you think about agile, too many changes are hard to define because you are supposed to be able to adjust. You just need to make sure that the client understands that whenever they add something, something is going to go out and you need to manage that. So, yeah, if they change it, yeah, whatever, it will take more time, but something else is not going to be done. So, yeah, maybe too many changes is not like something that you need to consider. And how usually there are three ways that agencies deal with this. Agencies is basically the managers, right? So, there isn't the viable culture. So, the idea of the viable culture is that quality is just a viable there. They don't really care. It's not really their responsibility to care about the quality of the project. Their responsibility as a manager is like just deliver the project, not with quality really. So, this is one way of dealing with that. And all the problems that arise, they will be like just solved as they come. Nobody will really track that. Maybe sometimes problems will come and nobody is going to track it. So, it will get lost and then it will get to production. And then the client is probably going to find it later on. The second one is just routine. So, the managers basically agree that quality is important, but they don't have the budget. Because they probably cut it. They have cut it whenever the client had like asking for discounts. So, yeah. There is no more budget for dealing with QA. So, we just move on and go for it. And whenever things happen, they will track it, they will just act for real in the critical parts. And so, whenever they act on the critical ones, they will be, oh, okay. So, critical is good. All these smaller ones, we just leave it. And whenever we have the time, yeah, we will get there. But who knows when. And the third way of dealing with it is really acting as a manager for real with the problems where the managers consider quality part of everything. So, quality is in the process, not out of the process. It's not something that you add. It's not an add-on to a project. Quality is part of what you are doing. So, you need to care about the quality. You need to care about assuring that quality is there. So, and yeah. And the problems, every time the problems are found, they would highlight it. They would track it. They would discuss about it. And they will find the best way or the best time to deal with that. And they will be handled. So, you are going to get to the end of the project with all the problems that you found fixed. And if there is anything that you couldn't deal with or you didn't know how to fix it, this is going to be taken back to the customer or to the client. And like this we cannot handle now because there is no time enough to handle it and etc. And this is where you guys want to be. This is where everybody should be to deliver the project properly, right? So, and there is something like the cost of the conformance and nonconformance. What do I mean by conformance and nonconformance? By conformance, I mean, when the company considers the quality to be part of the process, they will spend money on prevention, on conformance, on being prepared to handle the quality and to handle the issues. So, they will spend the money investing on training people and not like in something else or not thinking about it later. And also, they will think about evaluating everything. So, when I say like investing time on planning tests, like it's going back to the RFP phase or discovery phase and have somebody at that point already thinking about what is going to be tested in the project. Because whenever you get an RFP, if it is well-retained, of course, you are going to easily spot what are the critical points or critical paths of that site or that application that you are building. So, at that point, you may be able already to be working on that. And whenever you say we talk about nonconformance, we are talking about failures. So, internal failures is whenever you fail internally to execute the tests. And whenever you execute the tests, you find lots of issues because you didn't handle it properly in the process. It's more about tests in the end of everything and not tests during the process. So, this is one thing that instead of spending money in the prevention, instead of spending money in before to be prepared for the issues and be quicker, like to fix, you are now spending money on fixing it after everything is done. So, this is one thing. And the other thing is the external failures is that whenever the things get to the client, so the client is going to find the issues that you guys let pass. So, this has the cost of getting it back and fixing it, but may also have costs that are tied to legal actions. Like, let's say that the issue that you guys that passed was like something that was letting PII appear like personal identifiable information appearing like in the site playing clear. And this could lead to a massive legal issue. So, this is where the nonconformacy, the fact that you didn't care about the quality is going to bite you. So, and talking about like points when you find the, sorry, where you find the bugs, the thing is that there is a huge debate actually if the cost of the fact really exists. Like, people say that it doesn't matter when you find the bug, the bug tends to have the same cost. I don't really agree with that. And if you think about the process of developing like the SDLC of any software, like anything that you are building, there are specific phases that if you find the bugs earlier or later, they have different costs for you to handle it because of many things. So, if you go back, and whenever you are analyzing the requirements, you find something that is potentially a bug because people really have like, whoever wrote that requirement didn't think about some cases and you find that case back then, it's the cheapest one because you haven't really started to develop anything. You are just still discussing it. So, there is basically no cost. The only cost that exists is like a little bit more time to write a requirement that fits whatever needs to be delivered. But as you go on, whenever you are on the development phase already, if there is an issue, you have developer hours already counting. So, the cost of having to fix a bug in the development phase is getting a developer to go back to that code and do whatever is necessary there. And then, let's say that you've developed everything and now you are integrating with other systems or even another site that you have developed before and etc. And the cost of finding issues related to the integration include the developer hours and then include the integration hours like whatever time you spend developing the issue and whatever time you spend with the other team or anything like that trying to integrate the things, these two things, teams or people are going to be involved on that so it's more things, more people to pay more cost. And as whenever you find this, later on on the testing, when I say testing, it's like the testing for the tickets passed but whatever happened later on it got to somebody that is going to test like in the case of my story, the story before, the PM, whenever the PM got the issue to test it involves all the above. So, it involves the developer hours and both the integration hours and then the hours of the person that tested because they tested and they need to send it back and then it needs to go down again, back to them so there is more cost. And whenever it gets to UAT, if the issue gets to UAT what is the cost there? So, I will go a bit faster here. When you think about it you have hours from the developer hours from the integration if there is an integration and then you have hours from the PM or also the QA guide that is working on that and then you have time from the client and time from the client can be tricky because sometimes clients don't understand the user acceptance test as part of the process and they think that everything should be done whenever it gets to them. They don't understand that they are part of everything and they need to help us testing if that is what they want. And then you have the worst one that is finding the issue on production. So, if you let it pass the whole development phase the whole testing phase and it gets to production you get everything that we talked about before plus a lot of stress. So, yes. I believe that there is cost there are differences related to when you find the bug. It's not like something like oh, whenever I find the bug I just need to go back to that guy and fix it. No, there is a lot of people involved if there is a process. Sometimes there is no process you just go back to the guy and watch production, whatever and then you are in the same ship forever. So, yeah. What do you need to do then? Get somebody. Get a QA person or get a team to work on QA. So, they need to be dedicated to QA it's not something like just go there and train your developer to do it better. No. It's somebody that is going to really take care of that and that only. But what actually mean QA? What does QA stands for? There is a discussion on what QA like the usual thing if you think about QA you would always define it as quality assurance. So, there is a debate talking about quality assurance and quality assistance which is what does it mean? There are two actually two ways of seeing it one thing is that quality assurance is not really right if you think about it because a tester is not the guy that is making sure that the quality exists the guy that should care about the quality is the developer. The developer is the one that knows how to do that the developer is the one that knows how to test practices and etc. So, he is assuring the quality he is the one making sure that the quality exists but the tester is more like quality assistance. Why assistance? Because he is actually assisting the developer to know how to test and after the developer does everything that they need to do they will come and test all over again. So, when you think about the process itself when your quality assurance would be something more like everybody does whatever they want to do and in the end the tester gets in and tests everything that was done and pray for everything to be ok. And the quality assistance is more like like I was saying before the guy gets involved in the very beginning so the tester the QA person knows what is going to come into the process the project. So, they are always touching point with the developer they are always talking about how to test things and they are always testing it. So, when should you involve a tester or a QA or whatever should we involve them in the RFP? For most of the cases it is too early to involve somebody in the RFP because yeah, sometimes the guy is probably involved in the project so lots of times it makes no sense to stop somebody that is working on something to pull the guy to a proposal which is uncertain. We don't know if that is going to get some money or not. So, it is better to not involve the guy in most of this in most of them because they are usually simple quote unquote. But, whenever you spot something that is very complex like everybody that work with that and deal with RFPs you can feel when it is way too complex. Whenever it is way too complex or something like that it is good to involve a guy from QA early at that point because you probably need to plan what are you going to do for testing because you may need new tools to test you may need new people to test you may need to hire like an external partner to test that. So, you may not have the ability to test it but you need to involve somebody that can identify it quicker like it makes no sense for you to wait for later. But, yeah and in the discovery phase we got the if you go back to the story that I said before you got to the point where you got the contract you are in the discovery phase you need to how can I say to understand whatever you are testing whatever you are developing and yes, that is the point where you need to for sure involve somebody that will test it because the guy is going to say this is how we are going to test everything and like defining user acceptance criteria and etc. So, that is a good point. So, whenever you are planning the project the tester is going to be working closely with the technical lead to the project manager and the role is to define the user acceptance test and the criteria for the stories and he may or may not he may or may not be responsible for creating the tests. There may be I have worked with teams that the developer would create the used cases as part of the delivery of the ticket. We need to provide the step-by-step test for the specific ticket and then the QA person would get that one, two person, I had two people so one would be testing that manually and the other one would be automating that to test with everything. So, yeah, that this is why the QA person may or may not be the one holding it. And then what do the developer do? The developer whenever used case is the test cases are available the developer should be using the test case and the and whenever they need to provide the necessary data for the tickets to be executed and make sure that all the best practices are applied. So, basically what I'm saying is like test early and test often. Whenever you need to test start to test as early as you can and test it as many times as you can because this is what will make your project successful. So, whenever you are thinking about testing the types of testing that we usually talk about is functional testing that is the usual one just go through the functions and click around and etc make sure that everything is doing whatever they are supposed to do then you think about the regression test that is whenever you get a ticket done you need to get that ticket into the everything that was done already and test everything again to make sure that that ticket didn't break another part of the site and then you have the smoke test. The smoke test is something that you are going to stress some critical parts of the project not necessarily every single part of the project. And just a funny thing that I didn't know before but the smoke test comes from hardware testing. So, it's basically you would stress the hardware and check if there is smoke coming out of that. So, you stress the critical parts and see if anything is going to break. And then I have a bunch of tools that you would use for that. So, yeah, a lot of them, I will go quickly over all of them. So, basically one thing is lots of issues happen like because the site just have broken links. Something simple, something preview and something that you could get from, I don't know, if you use a CDN you would see it. If you use analytics you would see it but there are tools that could test, like just crawl through the site and click all the links for you and find it before you even deploy it. So, it could scan your local. So, that first one Xenolink, it's very, very old but very, very useful. I don't believe it's maintained anymore if anything but it's very useful. This Creaming Frog is actually I believe is a service. So, I know it's a tool but I believe they have a paid version of it. And then the second thing that we need to care about is testing, browser testing, but not just devices, not just desktop but also mobile. So, yeah, browser stack, source labs, IPM, all of them. Now, before browser stack, like long ago browser stack didn't have the devices now they have all of them. But the Appium is actually not only for mobile but it's not only for sites but also for apps. So, if as part of their project you have a mobile app being built, you can use that to test the app. Like, you can run the app and run tests over it. The other thing is code quality. One thing that I've been using, like I've used in the later project that I had was GrandPHP, which is like a GrandP guy for PHP. And then it's like usually a pre-commit hook that will run lots of other tools that you want. So, you would configure that to run like mesh detector like a more technical thing, right? So, mesh detector, copy and paste detector and etc. To scan the code before the guy can commit the code. So, it won't let the developer even push things that are not supposed to be pushed. So, this is a good one. There is a lot of resistance about pre-commit hooks because, oh, it is going to slow down my development. I don't care. Quality before. So, and then site speed is something that, like, you can test with other ways, but web page test, Pingdom and Lighthouse all of them are very easy to use and you can, like, with web page test, you can test from many points in the world. So, you can say like, I want to test this page from Japan or from US or from wherever. And you can even create some small scripts for doing that. Clicking and filling forms and et cetera. And it's free. So, it's a nice one. Lighthouse is something that everybody has on their Chrome browser. So, just go there in the inspector, audit, and it's there available so you can just do it. And Pingdom is similar to web page test. You can test lots of different things there. The other thing that you need to test and it's usually hard to test is email. So, you are sending email to people who are not testing email everywhere. So, these guys help you to test emails. How are the emails going to look like in different clients? So, you can use these guys to say, oh, how does it look in Outlook or on Gmail or whatever. So, these guys can help you with that. Yeah. The other thing is behavior testing. So, what is behavior testing? It's basically you can script whatever the user will do in that page, right? So, you go there using Behead. I've seen even QA people creating, like, create people that were not technical, not PHP developers creating Behead tests because it's way too simple. It's basically plain English somehow. So, you can create the Behead tests, like, go to this page, fill this field, click this button and etc. And then you can add this even in, like, in your build process and have this executing every time or have this on your local and execute every time before you send it to QA for real. And it's a good thing. Talking about Drupal, one thing, whoops, one thing that we have is Ritony. Ritony is basically an audit tool. So, you can use in Drupal. There are a few policies that come with it. It's actually created by Josh. And he's there. But I won't put him in the spot. And it actually you can create small policies for the site in general and create profiles also. One policy would be like test if there are modules to update. Test if, I don't know, there is pages missing. How many pages do I have accessible for anonymous person? And also you can pull all these policies together and create a profile that would export a HTML report for you with a menu and etc. So it's very good for internally, for you to show people, but it's also good to define the site readiness to be launched. So you can use that as a deliverable in the project, saying like, yes, the site is fully ready for being deployed or going to production. So it's load testing. Everybody needs to load test. Whenever you are creating a site, you need to care about how many, like what is the traffic that you are going to get in that site. And you need to test for that and above. So you make sure that whenever the site goes online, like goes live for real, it's not going to go down the minute after. Right. So, yeah, there are like jmeter is the number one. Blazing meter is basically jmeter as a service and has, you are more, like, it's more powerful to use jmeter directly instead of using blazing meter. And A B is a very simple one. A B testing, like a benchmarking, like a partial benchmarking, which you can, most of the developers would have in their own machines and they can test, like, just like accessing tons of hits to that page or to that site and check what is happening in the back end. So, basically, in the end you all need to have a good testing strategy. And everything that is going to happen, like in your project, the success of your project depends on a good approach on quality. So, yeah, thank you. Thank you so much, Daniel. Did anyone have any questions? We've actually got time. Yeah, yeah. Thanks so much. That was a great talk. I think we might have all in our career had one time where a client said, why do I have to pay for testing? Shouldn't the developer just do it right? How would you respond to that? Yeah. It's a tricky one, right? Like, yeah, the developer is going to do it right, but he's doing everything. He doesn't know exactly what you want. Maybe the test that he did acts perfectly, but whatever you were expecting of the functionality is not really what he tested. So, this is why we need to have somebody that is going to understand the business, because we don't have time enough for the developer to understand the business and the functionality itself. The best testing is not only about the functionality, but about the business itself. So, whenever I say that the QA people usually acts like an internal project owner, product owner, sorry. So, they would understand what is going to be built on Drupal or whatever, but they are going to understand also the business view of that. So, they know exactly what the business want. And these are the people that are going to create the test cases. So, if we don't have that person, we don't have, like, so, okay, we are going to, if we don't have the people that are going to understand the business and create the test cases, should we charge more for the developer so the developer has time enough to understand the business and then create the proper tests? Yeah. Not really a question, but to add to what you're saying is, how intuitive it is to complete the task or the function, right? And often for a developer, it's a lot more intuitive for them to do something that they just wrote, the way that they thought it was supposed to be done. And then a QA person is going to think about it completely different. And that's kind of the value of that testing process of having somebody else do it or having more people involved in that testing process, because developers never really just deliver what they think needed to be done. When they do test it, right, but it just is the way they expected it to work. Yeah, because they are always filling out the forms with the right things and then filling out the forms with the wrong things whenever they want these to fail. But the tester is going to feel it would whatever because they have a crazy imagination. As someone who's mostly on the client side of these things, what are some good signs, I guess, in a proposal that a vendor's got a good handle on this? Yeah, usually it's hard to say from the proposal I would say that it's more about meeting and talking because proposal like paper accepts anything. You can just get something that like a random test, text and add there, because yes, this is what everybody's talking about in QA. I've seen that before, like people getting other company's proposals, looking at whatever the guys are writing and write the same thing or write something similar. The proposal itself like the document, I don't believe it's a way to see it. It's more about talking to the person, like to the team and say like what are you guys really doing? Because they are going to say or not they are going to show if they know what they are talking about. Probably last question. Thanks. What do you think makes a good quality assistance person like if you are hiring someone for QA what are the key things that you would look for? One thing is that the person should not be afraid of making other people cry. Because yes, lots of times I use to fight with our QA people and pick on them. But I believe that the QA person should be somebody that can do whatever they want inside the project. Because they are like the guys that are acting throughout the project to make sure that everything is going right. So even more than the PM because the PM is, yeah he is there or he is there but they are also thinking about business and thinking about budget and thinking about other things. The QA person is only thinking about quality. So you need to make sure that this person is able to go to somebody and call that person and say this is wrong and we need to do it like this and not like that but also in a respectful way because there are guys that are very not a people person. And if you don't mind me adding I see QA as being a responsibility of the entire team. So even though the QA person has their responsibilities it should be top of mind for everyone in the team. We have to end it there. Thank you so much everyone. Thank you all.