 So, welcome to my presentation about automation in the world of project. Maybe at the beginning, a few words about me. I started my career about 15 years ago, or even later. I started in a big corporation, which is called Motorola, probably everyone knows. Then I switched to a medium-sized company, Widow. And after that, I joined a very small startup Cognified. It was about 12 years ago, and I'm still there. We transformed from a small startup to mature organization, which they were projects for the biggest companies on the world, like Ford, for example. And most of the years, I was there on the road, head of QA practice. But the last few years, I spent as a development manager. Outside the work, together with a few of my friends, we started to organize post-net testing and quality group. It's the oldest meet-up in Poland about testing. And we organized almost 60 meetings. And the last few years, we recording podcast, testing parod. Unfortunately, it's in Polish, so sorry for that. Okay, so it was about me and what I'm going to talk about. First, I would like to introduce you to my world, the world of projects, because Cognified is a software company. And I'm going to show you a little about our experience with selling test automation. So, how we start to approach to this, how we talk with clients and where we are now. Then, if you start selling, you do more and more test automation, you start building your frameworks tools and thinking how to reuse them to be faster and cheaper, of course. And when you start doing a lot, you have your own tools, frameworks, everything, how to deliver it inside the projects, how it changes in Cognified against through those about 10 years. And then we just summarize and I hope there will be some questions so I could answer for them for you. So, a little about Cognified. As I said, we are a project-based company. What does it mean? We deliver about several projects, totally different projects about sales or time. It can be projects that are 50 engineers or you can have a project which you have four engineers, just example that are totally different size. And we're working mostly on web, where we use CMSS or content management systems, mostly done by Adobe or Sidecore. And we customize it and integrate with almost everything what it can be. So, we mostly deliver it to a stop level if you think about different systems. Cool. So, short intro. And let's go to our main subject, so, selling test automation. How we start? So, we start to talk with our clients about theory. It was quite strange, especially as we had... We don't have test scripts. After a few years we decide no, we don't want to go with typical approach with test scripts, we go to exploratory testing. And there was another small changes inside our projects and our approach to quality. Because we stop using just test for testing. We invite developers to that work. So, in this case it was very, very difficult for us to deliver this story. Because we weren't able to count it. Because mostly you just count. You have 20 test scripts. Execute of this time is 20 minutes. So, we have some costs. And if you automate and you run those 20, 30 times you can just very simply calculate the error. But we weren't able to do that. So, another approach that we try with our clients was the coverage. So, we started to discuss with clients how many test cases we are going to delight or how many things we are going to cover by our test automation. So, we start talking about code coverage. So, then at the beginning we tried when we had about test cases coverage too. Then we come to requirement coverage. But in agile way requirement it's a quite white word and it's very difficult to define. So, we start talking about feature coverage. But when you start to talk about coverage. Okay, so what coverage do we have? 10%, 20%, 80%. And what is a good answer? Especially when you're talking more with marketing guys it was very difficult for them or even for most of our people. So, this way they stopped working or didn't work even at the beginning. As a result of those talks we just got a question even like can you just write a good quality code from the beginning so you don't need any testing or any test automation. It's the question that you heard from our clients. So, it's cause that we weren't able to say our test automation. And then we thinking what to do and how to approach that. And we ask for ask questions, very important one question. Does test automation bring business value for our client? And it's very important question that you can ask yourself when you think about test automation. So, when you ask those questions just don't want to answer yes, no. We start to find what the business value is for our clients. So, let's look about the business values that we have. So, one of the main business value was the delivery approach. So, how number of releases that our clients would like to have from us? Or how often? Does they wanted to have continuous delivery approach? Does they need a release that we deliver production ready? Because in fact not each our client wanted from us production ready release. So, accept a lot of issues, another not. Another question was how do you want, how fast you want to change your website? Or what do you want to change on the website? Another is time to release. So, can I accept on some issues? Some issues on our software, on our application or not? Or another is lead time. So, how fast I need to deliver you a new feature? Because there are challenges from their business. Again, another question was availability. And just example, one of the answer was a bugging deployments. Another talks that we understand by a client as a business was extendability and mentality. Because it's very important to ask clients what do you want to do with those applications? Do you want to extend it? So, adding new features? Do you want to maintain it in very easy way? Or do you just don't want to make any changes on that? Just deliver and forget. And those was some business values for our clients, which clients say that is very important for them that we heard. And not always, we never had this automation. Of course, there are other things like business critical areas that clients always say that just guys, this business logic like reservations has to always work. All others, in fact, if it will not work 10 minutes or even one day, I'm really fine with that. Or there was another client changes like A-B testing. So, I would like just to test some kind of the journeys just for a few of our clients just to see if it makes sense or not. So, where we are and what we are understanding by that. If we look about this business value, like availability, for example, the solution for our clients, because I have almost 100% availability, wasn't this automation, but we start talking about blue-green deployments, which gives you the possibility to be always up. Or, for example, when our client says, if we just read them to the previous slide, that very important for them are quick changes on the website. But what kind of the changes? It was just images or adding new block or whatever, or changing some menu. So, let's give him CMS or some components, auto-rable, and this was the answer. So, we couldn't find as an answer test automation. Okay, what caused that? That we are understanding that our test automations does not bring business value for our client. So, we stopped talking about this automation with clients because they wanted to talk about business values. So, something which allowed them to be better in their business. Not just about this automation because it's just one of the solutions, but it's the solution that not always makes sense or even is the best, the better one. As I said previously, there are other solutions like blue-green deployments, some changes in architecture using CMS or whatever, which shows the problems, not just improve the test automation. I can even say one tip from one client that I consult. They have an issue with quality, and they invest a lot of money to solve this issue in test automation. And after five months, the metric which they defined the quality was exactly the same. The metric was number of issues raised by the users on some service portal. So, again, it shows that this automation doesn't bring values, does not solve that. But we are understanding that test automation doesn't bring value. And here are just the questions. After my presentation, I set up a pool. You should have this on your panel on the right. And if you just mark, if you agree with me or not after the presentation, it will be great. Okay. But we have to propose that we are the expert. So, we have to propose a solution. Even if we propose this automation, we start understanding that then when we know what exactly client needs, we can go and say and even say that we guys will deliver you 80% of coverage on this module. But other modules are not important. So, there will be just 20, just to be sure that it's working and something that we need from technical point of view. Okay. And what we learn not to ask with our clients, because it does not make sense or doesn't give you anyone, is, as I said, we never ask now, do you need this automation? We ask what you have with the business values, what business issues you would like to solve, and then we just come in with some solution. And as I said, we don't talk about coverage and things like other two. So, our suggest test automation as a remedy for all quality or delivering issues, because it's not and it's something that you lack and talk. Okay. So, when we change our approach to test automation, we're starting to do it more and more in our project. So, when you're starting to do more and more test automation, we start to beat our frameworks and tools. And of course, we're understanding very quickly that building tools and frameworks when you're building, it's cool to reuse them across different projects, especially as we specialize on these Adobe or Sitecore solutions. So, they're quite similar, our projects, or they are common parts. Thanks to this, there is an area to reuse. And what are the advantages of reuse? Of course, it's cheaper, yeah? Because you beat something once. It's faster because you have a framework tools, we can just start in writing, you don't need to beat it at the beginning. And it allows us to use the same already existing infrastructure because it's prepared. And our teams are familiar with these tools. So, when people just switch from project to project, if he knows the tools, he can give any value to the team or starting working at the first day and just bring the value. And our tools are specialized. If you use specialized tools, of course, they are better than the generic one for your solution. When we started with our client to talk about reusability, and I think this part is quite or very important for project companies, like software houses, rather not for the product one. Because it's very specific connected with projects when they deliver projects for the clients. So, in our case, client is the owner of code. When you do the projects, in most cases, your client would like to be the owner of the code. So, if you reuse the internal framework, your client becomes the owner of this framework and you just can't give it away because you can't use it in another project. It's about the low, yeah? Again, if you start talking with companies from Fortune 500 or banks or assurance company, they have some list of approved tools frameworks. And it's not surprise our internal frameworks wasn't on their allow list, yeah? Which we are able to do that. It's cause us months, even sometimes to add that. Second, our client asking. So, you start using your internal frameworks. Are you going to share the code with us, yeah? Because we would like to know the code even if we accept that you are going to use those frameworks. So, what's about the sharing the code, yeah? Again, it's our IP. Who covers the maintenance cost? It's another questions from our client because one is to use it, yeah? Second, if you have a software, you have application to maintenance, yeah? And after the maintenance cost, there was questions which was very important for our clients, yeah? How do you ensure us that we not be able to get it off your frameworks, yeah? Because if you're using internal frameworks, our clients was afraid that they have to use just us because just to... Other our competitors doesn't know those frameworks, yeah? So, then we just get off and then we have to invest a lot to beat new frameworks, things like that. So, it's a problem for them too. And I got another questions from our business value, business side. How much our internal IP are worth, yeah? So, what's the question? So, what's the question? So, okay guys, you're going to use your internal frameworks. So, we are cheaper, we can deliver faster, things like that. So, it's speed up. So, how much additional value we can get from our clients? Because we're using internal tools, yeah? And those calls that we always talk, talk and talk with clients. And we finalize that sometimes when we agree tools or we get some acceptance for our tools, we're almost finishing sometimes project. So, we start it very light, produce a lot of technical depth, which we're trying to eliminate at the end almost of the projects, yeah? And the cost was very, very high and it wasn't great weight of delivery, yeah, software. So, it's where we was with our framework tools use a bit. That was great, but there was a lot of issues to use them inside the projects. So, what we decide, what we decide? And the answer was quite strange, maybe for you. But after analyzing everything, we say, there's no sense to have internal tools in this case. Let's open source our solution. Okay, it shows that we show for our competitors or detours even, yeah? Our IP. Yes, it's now, after this open sourcing, we read a blog how our open source tools are fine and really shows or allow to solve problems on the clients on our competitors' blogs, yeah, when they're writing about our open source tools. So, we lost our IP. We share our IP with our competitors' tool, yeah? But what we have for us as a bonus, yeah? So, we solved our client's concern because the first concern was client is the owner of the code and we can lost our internal framework, yeah? It's not true because it's open source, so it's fine. Amounts, sometimes when we try to adding our internal tools to allow this, yeah? In our clients, again solved because I don't know why, but all those procurement securities departments accept very fast open source tools. It was just sometimes a week where if we compare this week of waiting till months when we're trying to convince them to use and add to this our internal tools, so it's a big win for us, yeah? You can wait a week, but you can't wait a few months. And again, the questions about maintenance cost. Still, it wasn't the issue because, you know, we're trying to beat some communities around that. And even, you know, clients just start using our competitors. They have access to those code, they could maintenance it, yeah? So, first the maintenance cost was lower and everyone could do that because the code was shared, yeah, and open source. So open sourcing allow us to solve all those problems and start this automation here. But, okay, here benefits, yeah? So we got some internal benefits, of course. Our people are proud now because, you know, they are famous, they do something which I can say about and can be recognized. And in fact, it's just our values. We always talk a lot about knowledge sharing, beating communities and things like that. So we got some internal benefits, especially for our engineering, yeah? But what caused? I had a chat with one of Cognified owners and he asked, okay, Zbyszek, you went open source. So how much money do you think we lost? Because we just got them from a client, yeah, like additional money for a project, yeah? And then when I start trying to answer for them that, okay, and trying to calculate how much the number on the money, I realized that it's not true because what we achieved? We think more test automation, in fact. We try, we're building the test automation infrastructure, yeah? Because it's just not a code. You need to beat infrastructure and everything. So, and then you have to maintenance it and things like that. In fact, we're understanding that our project scopes are bigger and bigger. Thanks for test automation and the wide using of them. And we get more money for those projects. So as you see, the answer is that we don't lost money. Maybe we will not earn those bigger number, but, you know, the revenue is almost on the same level, but we deliver the better code. We can start automation very fast at the beginning of the projects. So we deliver better projects in those case. And yeah, why just few bullets just to summarize what I talk about. And yeah, additionally, we're selling the trainings, yeah? Because if you are the specialized in something like open sourcing tools and you're starting to using, your clients would like to have a team too. And then you can, you have to train your clients. So we're selling the trainings too. And as you see, open sourcing allows us to deliver something better and earn more money, which at the beginning it wasn't, it was questioned by our business. Okay, so we finalized the two parts about open sourcing. And it's the last part now. It's about delivering. So we deliver now mortise automation at the beginning and things like that and how we transform to deliver it, yeah? Because if you deliver, and there is a continuous delivery, of course, something around. So let's go to this part. I hope you're okay. And I would like to start this part with something, with a square. This square I created nine years ago. And in my article in testing experience, maybe some of you remember that, this newspaper. And as you see, our very old approach, especially when it created last, was to tell you on testers. So inside the team, we say the testers which working in the project team, together with developers, just do the maintenance. So if you need something change, yeah, they just switching some X path or things like that. But test case, automated test was created mostly on test cases by external team of testers. So which weren't part of the project, or even weren't part of a scrum team. So they were just working independently. And it's something that we work nine years ago. But when we start to deliver more test automations, something change. And in fact, our environment change too. So we work a lot to improve quality of feature delivery by devs. Our developers interested in test automations. They come to us and say, guys, I would like to write few test cases for automated test cases for my code, for my feature. How I can do that? Can you just do that? We change our definition of done. First, our definition of done was just to deliver feature. So feature is somewhere on production or staging, whatever. But in many cases, test automation was written later. So it's something like you have your deliver feature in a sprint third and test automation in sprint four. Something like that. So we change that and we say, no, the feature is ready when we are able to maintenance it. So we're adding this maintenance ability to our definition of done, which cause that we can't work in those cases. That test cases are written later. We switch inside from testing to quality assurance. So our test engineers become rather quality assurance engineers, which will cooperate with devs to help them in testing or thinking about the quality rather than testing. And we start to introduce test pyramid. So we understanding about this unit test integration test and the end-to-end testing as less as possible. It caused that we starting to moving more and more test automation to the lower levels. And of course, continuous delivery appeared. So when we were eight years ago, we were here. So there was a test test who do automations. And let's say writing new test was outside the projects by external testing team. Now, after all those changes where we are, everything is done internally. So inside the project, as you see, we move almost all creation and maintenance of test automation to developers. And QA is just ownership and give some support for them. What it means, we don't have test automation engineers. We don't have our software developer in test. It's because we change our ways of working. And we believe that it's better and it's work for us. So how it works now when we're thinking from delivering test automation. So QA is the owner of quality. So in our case is the owner of test automation on different level and things like that. So he takes care that it has to be executed on something that we agree or believe that it's needed for those projects. So you can think that QA is something like a screen master who works with self-organized team. So QA in this case working with developers or supporting them or just challenge them. How to write or take care that those test automation tests are written on something that we needed on the level. We change the definition of ready. So in definition of ready, we are adding the test automation level. So we say what we would like to automate and what and on which level. And I believe that it's very difficult to do for a test test. So developers know very well what we can cover on unit tests, what on integration tests and things like that. Thanks to this, we're adding the cost of test automations as a story point or whatever. And Dev QA do code reviews in both sites. So yeah, sometimes QA wrote some tests. So then they do the review. If developer wrote test automation scripts in our QA do code review even. We have something like QA demo. So when the feature is ready, developers shows how it works for test, project manager, whatever. And it doesn't show just only how this feature works, but shows how test automation works. So what kind of test automation he wrote on which level and things like that. So he ensures that we are able to maintenance this feature on an expected level. DoD I mentioned before. And automation is a part of the backlog and technical depth. Yeah, I can say that there are a few times that we are not. We say, okay, we deliver this automation later because there are some client ask to deliver it faster things like that. But it means that when we do that, if we do that is exception, which is very important. It's just exception. We just creating a task brought some test automated scripts for this feature. We're putting it to the backlog to the developer and backlog. Yeah. So it's we process it at the same as the new features and assigned to the sprint and whatever, but adding the priority. Yeah. Okay. So what I would like to that you take, you know, that you take away from my presentation. First is very important for me that what we learned that test automation is not a business value. It don't bring any business value. It's just one of solution, but not the preferred one in many cases. Sometimes there are better solution from a detector point of view or others that it's worth to use, not just writing this automation more and more. Yeah. Second is about open sourcing. So it's worth to do that. Even if you are software house, whatever, open source, open doors, it's really help you to deliver better projects to talk with clients about that. And yeah, it's not, it's not losing revenue. In many cases, you can take even additional revenue for your company. And it's very strange when you're talking about on a test testing conferences, yeah, especially on conference when there are a lot of people test automation engineers. What we would like to propose it's moving test automation from tester to the whole team approach. Yeah. And the developers wrote this automation to when they are able to wrote a feature then run all those automation scripts. See how his feature works or what breaks fix this. But in fact, if his feature brings some changes, makes him responsible to implement those changes in test automation scripts too. Yeah. And thank you. So let's move to the question. Okay. Hi. Thank you so much for sharing your experience with us today. I would like to thank you and I would like to also encourage people if they have questions, please post it in the Q&A section. I do see there was one question which was asked. Perhaps I missed a point, but that IP stands for stands for and before. Okay. Yeah. It's an individual property. So it means that it's something that you own. Yeah. You are the owner of a code. You are the owner of solution or something like that. If you would like to use, you have to pay for you or you just keep it as your advantages or a customer. Yeah. So just thinking like intellectual property, something like that.