 Hear me now? Okay. So good afternoon. I'm excited to be here. I hope that you're joining the conference. I've been here since the last day. I made a long journey to come here only for those 45 minutes but actually we had very good meetings and discussions and so I'm really happy to be here. It's my first time in India. So first of all I would like to present myself and to tell you who I am and why I came here and what I'm going to talk with you about and how that correlates to what I'm doing on the day-to-day work. So my name is Ronald Riel. I'm located in Israel. I don't know if any of you have been there but although it's a small country we're doing quite a lot of software development. It's a different kind of software hub rather than India but those are two major I think important places for software development. I'm with HP software. I've been in Mercury. Any of you know Mercury? Okay. So I've been in Mercury a few years ago and then we were acquired by HP and I used to BQA manager for several years and then I moved to the product management. I'm working very close with the R&D and I'm working and defining what they should do to some of our products. Probably some of them you do know like QTP, loadrunner, quality center and in the last two years I moved to lead our ALM SaaS offering which maybe some of you know we have a new solution for ALM for agile management sorry and I encourage you to watch it. Before I start so I made the I'm not here to talk about I'm not doing a marketing pitch. Okay I'm not going to talk about HP products. I'm not going to talk about how you take and implement them in-house. I'm only going to share with you basically what we're doing and what practices we adopted internally. The only point that you will see slides of our product that will be that slide. Basically the product line that I'm leading is agile manager, our SaaS solution for agile management and a new product that will be available probably by the end of the year which is bringing a new capabilities for quality that are more targeted for agile projects. Stay tuned it's going to be a fascinating product. I hope you would find it interesting and relevant for you. If you want to hear more are you more than welcome to come after that session to hear. So the main point that I want to reach is that you will get I don't know any one of you who is either doing continuous delivery or is planning to move to continuous delivery to take one or two takeaways from that session and tomorrow morning to see I mean I guess tomorrow morning is your weekend right so two days from now in Israel we're working from Sunday to Thursday so in our case it's a bit different but early next week to see what you can take and change and modify the environment that you're working on. We are lucky I think also the guy for Microsoft they're also lucky because we are developing tools that our customer basically are doing things that are more or less the same as we're doing so we have the luxury to talk with a lot of customers to know the market and to see how we can take all those inputs and make sure that what we're doing is really is really the right thing to do. You don't need to write anything I upload everything into Google Docs so you can find it online if you can't scan it you can come to me later on I will give you my contact information and you can send me email and you can send me email and I will send you a link to the to the case study. Before we start I want to know more or less who is in here in the audience so how many people over here are project managers okay how many people over here are consultants okay how many people over here this with quality in any way on your daily work okay very good so before we start to drill down into into the journey that we had I want to share with you into the details I want to share with you the journey that we had and and more or less like the guy that talked before me we released products in a one-year cycle we had we thought that we're doing agile we also call that an agile I can tell you that I've been in conferences a few years ago and I asked people in the audience how many of you are doing agile how many of you are doing agile okay so that's pretty good the question five years ago by the way was more or less the same we also raised our hand when people ask us if we're doing agile honestly we were not doing agile we did something in between to waterfall to agile we had milestones we used to do stand-up meetings we call them stand-up meetings we also have few events that are as the people who are doing agile but at the end of the day if you're looking at the essence of agile if we really were able to modify and change stuff that we are working on along the way we weren't capable to do that when we move to develop our new product line we knew we knew that we're going to develop that as a sus solution we want to make sure that we are doing things right we decided to want to we want to move to continuous delivery and that was a big change I mean from releasing a product in a one-year list to to release it as an ongoing release that's a big change it's a big change not only to the R&D by the way the R&D it's the easiest area for you to do that it's a big challenge for the entire ecosystem it starts internally in R&D like QA documentation product marketing all those are pretty affected by that change but I will focus only around what we did internally on our teams we started by targeting to release every week and and we didn't you know why we changed that we currently release every every month to production do you know why so in our case it's basically our customers ask us that it's too frequent for them they ask us to reduce the amount of changes in order to them to have the time to digest the change so basically our pipeline today can provide content to production every month but we do that on every week sorry but we do that on a on a one month iteration our sprints are two weeks long and and that was part of the change that we had yeah so I covered I covered most of the thing we have two weeks sprints one month is a delivery to production and we have a major release which is every quarter which is basically all all the content that we released on the last three month so that's basically one lesson that we learned about continuous delivery and we're not going to to elaborate on that we are releasing content on every month and it's not changing what it means that if a feature if a feature is ready so it goes on the train I can tell you now all of our releases in the next year except except if they fall on the weekends or holiday we do a minor modifications but basically we release every month which means that if a feature is ready it can go to the release if it's not ready basically it should wait until the next month and that's tune the entire system and and the developers the QA and everyone who are involved in that product in that project in order to make sure that we are ready we define the release criteria those release criteria are well defined for every feature and if you don't achieve those release criteria and the done the definition of done basically this feature is out and it will have to wait for for another month and that's basically it says that that we failed in our mission to deliver that content so I want to talk with you what we are doing now I talked with you what we did what we are doing now and I also want to share with you some of the challenges with that we are looking down the road so we as I said we knew that we are going to move to a continuous delivery we started to do that but we knew that we are not doing things right we felt that we need to do a bigger change rather than just to start to change the milestones and the events that we are working on and basically we started to understand and that's what I'm telling you now it's to understand that that's a cultural change before that's a process change okay what it means that all of us need to you had to understand that delivering contents in such a short time frame really change all the processes and the paradigm of what we did so far it means that for example if we are looking at the QA organization that had a time to build and plan their test activities they had a time to take content I don't know how many of you that have a QA background how many of you came to the R&D and said folks this build is not ready for QA are there any people over here like that so I've been a QA manager that was one of my pleasures to get a new build to run my automation suite and come to say to the R&D folks unfortunately my testing suite didn't pass please take it back it's not ready for QA when you have two weeks you don't have the luxury to do things like that and that really changed the way that we had to look on how is our on how our QA organization is working and and to change our organization structure basically what we did is something I don't know how many people over you here are changed their organization to work in in feature team structure meaning that all the stakeholders are inside the team can you please raise your hands so at least for us although in many cases people say well it doesn't really it's not really important who is the manager I can tell you that in our case it made a big change once you change the structure of the organization that the QA is not a separate organization it's part of the feature team so the QA task really changed from being the QA problem to our entire team problem and that's a big change I can tell you that one of the amazing thing that we saw was the reduction of the amount of defects that were reported I know if you've seen or I'm sure that you can you can understand what I'm talking about basically the number of defects that were reported were reduced the main reason is that most of the developers they don't open defect for themselves right they just go and fix that the same thing happened in our case the QA folks which will talk about them in a second they they worked very close with the with the R&D and they when they saw issues they came they came and talked with them and they fixed the issue right away I'm sure that all of you know the price of solving defect in the in the time curve I mean if you find defect too long too late in the cycle it means that it will cost you much more to fix that right so in our case we really pushed that very early in the stages and most of the issues were detected very early early in the stage and as I said in many cases we didn't even reported defect so the quality task started to became not the QA problem but the entire team problem or our entire organization challenge and that's something that we really saw a big change in how it impact the alignment of everyone off to the QA activities as part of that we changed the role of the QA we don't call them QA anymore we call them dev testers in our organization the teams that I'm working with we don't have QA those are the people I mean those are not the exact people that I'm working with but they are dev tester what does it mean did anyone over here we are not the only one but we are not the only ones by the way that change the term did anyone of you also saw the term of DevTester okay so basically we wanted to change the name because that was important for us to to show that we really made a change over here and that we changed the definition of the role from a QA guy that that was its only problem to another player in the team who is another developer in the team but he's more focused on testing activities basically that change had several implications and I will talk about about them right now the first one is that we went to our QA guys and we looked on on the toolset that they're working with if you come to if we used to come to our developers before that and we we would come with our toolset that we had the developers don't like to use those tools they are working with their source controls with their development environment the idea is they know how to work with with continuous integration they know how to use open source and frameworks and those were the ecosystem or the environment they had to use and they wanted to work with and we knew that we need to change that and to transform what we are doing in order to align that to our processes so those guys and new dev testers had to to be familiar with those tools in addition they had to be able to have the technical skills to work with those tools they need to know databases they need to know they had to know how to work or I code they had to know how to integrate into the CI and write CI jobs etc so that was one of the challenges that we faced how to look on our QA guys and to see how we shift them and I will I will talk about it in a second but basically on our case it was pretty easy because as a software organization most of the people that we had are coming from a computer science background but that's a change that I can tell you that we also so happen in other company in other companies that they're there guys over there were with less kids and they went through an assessment process that I will I will talk about it in a way in a way what we really looked for is people with a background of developers or capabilities to become a developer but they can look at the application from the quality aspect which is a bit different to be able to see the end-to-end cases to understand the end users and and and our customers and that was a bit challenging so we ran through an assessment an assessment process when we took all of our QA guys we ran them through a training of three days training where they got all the technical background they got the skills to work with the tools and at the end of that process we really looked toward the people that are capable to become our dev testers I can tell you that some of the people didn't want to the last technical guys didn't want to become dev testers and we look for another opportunities for them in the organization but most of them I don't know if that also happened in your case also had the inspirations to become developers so that really bring them more central into the developers world and that was and they were quite excited about that so after that we look at the people at the end of the day we had around 30 to 40 develop QA guys we ended with around 25 and we had to recruit few more than we knew that we they have to to come with those technical skills so I'm talk I want to talk about the activities that we're doing as part of that what are the things that we are doing and the way that we're trying to divide that is things that we are doing during the sprint or or as I usually call them in cycle and what we are doing out of cycle and that also activities that that we're doing in the past we used to have those center of excellence I know for you over here how many of you have internally center of excellence automation center of excellence performance center of excellence security center of excellence all of those are really strong but but we need we need to see how we put them early in the process in order to come and to be ready to release in such a quick pace so manual testing versus automation so maybe maybe you can relate to that how many I mean what's the difference what do you do what do you automate what do you do manual testing maybe can someone I can't hear you sorry okay okay so I will relate to that it's pretty correct anything else okay that's another point what you can't automate do manual basically we saw many many customers and actually we did the same we started with manual activities and then we took those scripts and we automated them basically we knew and we understood that that's wrong right the challenges as you said at the back that and the same here is that those you cannot compare them it's not apple to apple it's basically apple to oranges everything that you can repeat over and over again or you want or you want to get the main flaw flows of your application your regressions those are things that we automated manual activities start to become more the exploratory testing and all of those were fully integrated into our two-week sprints developers were involved in that and took part on those activities and and developers don't like to do qa tasks right but when it's the entire team goal that everyone are relying on that and and they do exploratory testing and since that our new automation framework is supporting their environments then they are able to execute those tests look at the results and be part of that and I will relate to that when I would talk about the CI a few more words about automation so unit testing is part of our definition of done API testing or lower level testing are also part of what we look in order to make sure that we can run fast and continue and to get those safety nets the day the unit testing is the first safety net the API is the second safety net and they agree and they the font and is the last safety net usually takes longer time and then those kind of test run on a nightly job rather than on our continuous integration job that run faster that's a slide that I took from one of our people who is driving the new qa methodology for that team that basically describe how they work and implement their automation so they first start in the when we do our planning they start to discover the user story and to look at that and design the test scenario okay usually they're working very close with the developers to understand how they should build that they code it and then they map it to a product tree I want to talk a few to say a few words about the product tree basically the product tree part of the challenge that we had is that user stories last for a sprint but when we're talking about the quality we usually want to look from the entire product right and we don't have the correlation because after the sprint and you don't look anymore those user stories are not relevant anymore so what we did we've built what we call internally a product tree basically this is a map of the product in the eyes of our users and every test is linked into a place in this product tree and when we're looking at our quality we we look at the map of the product tree and this is how we know the quality of the product so before they they move and validate that test they map it into the right place in the product tree they run a test validation and once it's ready they push it into the source control okay basically they are working in the same practices and methodologies like the developers are doing and then it's integrated into our CI and CD we usually have rules to define to which job that test should be joined to for example a GUI test usually will be joined to the nightly build that has a longer time to run API test probably will join depends on the time it takes to the CI test etc and then we have the report we see that everything is ready and it's an ongoing cycle how many of you yes sure no no no that's I didn't mention that at the beginning but it's it's I'm from Israel we are usually interrupt people in during their speaks so feel free to interrupt in the anyone who is joking over here probably worked with Israelis in the past yes so you are totally I will repeat the question if you didn't hear it so basically is asking when I mean automate GUI automation usually can happen pretty late in in the process right and then how you you take that into account when you you build your test automation so usually since our teams are a feature teams we have people that are front end development and backend development the UI in many cases by the way is ready even before the the backend is ready usually over there things are more complex but there are cases that you cannot run the test because it's not linked to the the backend that really enabled the functionality behind that I'm not talking about that in this session I'm I'm also running the UX we have I have several UXs in my group that working with the team the way that we are building more cups etc is a bit different from what we used to do but you are right basically the things that the case that we move to automate the GUI is pretty late in in the sprints okay so usually it's pretty either we automate the GUI for the previous sprint in case that we didn't reach that or we were able to push that I don't know three to four days before the sprint and so if your tests are data dependent you can run into the issues where the test may pass into one environment and it fell into the other environment or do you also have to take the overhead of managing the data or you know freeze the data set because a lot of financial applications what happens is that you're dependent on market data you're dependent on holding sex sector with changes on a day-to-day basis so did you also come across scenarios like that yes so definitely basically data is a very important piece when you are looking at what the validation you're doing okay so when they develop those tests sometimes they have different data sets that are for the continuous integration on our daily continuous integration that are running and some data sets that are different for our nightly job or for the job that we are executing during weekends that provision also multiple environment sometimes more complex environments that you want to simulate more complex cases okay all the environments by the way this is a sus product but all of them are deployed on the same place that we will later on I mean there's kind of staging environments later on those pretty easy can become our production environments it's the same environment overall so how many of you over here are doing continuous integration today by continuous integration I mean have an ongoing build that are running deploying and have testing activities to that can you raise your hand okay that's that's good I mean I presented in a session I think two years ago and not many people really did that I I was part of the people who were developed QTP or automation solution and and many of them didn't integrate the automation suits into their CI and they didn't leverage that so probably this is not new to most of you but when we have a build that is running deployed we are executing the test we are getting the results at the end at that point if something failed we are stopping we want to see what happened in case that that's a test case that happened that failed we are analyzing it if that's a problem in the test or if it's a code that change we know who are the people who push that change into their into the CI and and they need to either fix it or make sure that the owner of the test will fix that the challenge is to see for example you talked about a great testing in many cases the tests are ready before the back end is ready okay so we have some kind of a way to to talk those tests that they are they shouldn't influence on the build status they are failing and when we change that tag in case that the test fail we know that and it's because of the functionality is not ready or it's a they implemented the functionality a few adjustment need to happen in the test itself I want to talk I want to talk about security about security and performance so what we we started to do and that's a practice that start to become more and more common as the guy that talked before me I mentioned the challenge around performance okay usually the performance issue that are detected late it's usually you start to test performance because you need a system to be ready to test performance and you do that pretty late in the process and well then when you find the issue usually they are at the back end or very core capabilities and a change is very risky and to do those risky changes late in the cycle it's something that you don't want to do what we started to do is that our dev testers are running a low level or low scale low testing as part of our CI so for example we are executing a 100 virtual users and in a way we have some kind of a bench a benchmark that we can monitor and to see if that's going up or down and we have a sale if this is changing we are looking into it and we're trying to understand what happens the same goes for security we do some kind of security checks as part of our CI processes when we commit a change they are very basic a very basic validations but in top on top of that we still have our center of excellence that they are involved they are guiding that our people what needs to be done and in many cases when a deeper or more complex low testing is required or security testing is required they are doing that and they're checking what is the earliest point that they can do that so that's a combination of taking the the need to test those early in the in the process and leveraging the fact we still have our center of excellence that are testing that later after out of the cycle as I said few words about the move to SAS any questions so far or I can move on of the sparing up on the project budget and the team so sorry from the beginning because okay my question is you said a dev tester was paired with a developer right so what is the impact of this on the project budget and the team size did it increase the number of people in the team and the project how much it increased the number of I mean the ratio stayed more or less the same it means that our teams are usually around seven people and we have more or less one and a half QA dev testers in that team a product owner in that team so that's nearly nine to ten people team okay that's more or less an autonomy that can run by themselves on the functionality that they should deliver so is it what what is the optimal ratio that you would recommend so so it's I think at least more or less now since also the queue the developers are involved in the in the some of the QA activities it's nearly one and a half to six that's more or less the ratio that we have now and and what we did so is is what we did see is is that our quality raised up mainly because we detect and find the issues much early so we can fix them our cycles are much faster faster and I would also talk about SAS that really can help us to to find production issues pretty fast and solve them so we can take more risks and reduce some of the overhead that the QA used to have in the past so one more related question so many times you know the people in QA offer a different perspective as opposed to a developer but when you know dev testers are closer to developers like you know you call them a dev tester would you not be missing out on that perspective like testers starting to think like developers so would some some perspectives not be missing in that case I'm not sure that I got your question so the essence of the question is testers think differently as opposed to a developer yeah yeah so now I got it yeah so basically that was part of the reason that I showed them like superheroes okay because their job is pretty complex they need to be able to look at the application as testers used to look in the past okay to look it as an end-to-end to understand all the functionality to know how the customers are using the application they're using some of the capabilities that we are leveraging from the fact that we we are a SAS product okay but that's part that's a very tough it's very hard to find very good dev testers overall I think that the ones that can do that I mean it's very hard to find them and they have a job security I think for the next couple of years that's that's for sure so a few words about SAS one thing that we are doing is that we are leveraging the fact that this is a SAS product and we are doing gradual exposure basically what it gives us is the ability to take more risks we are usually we are using our own product for our project management or agile project management and and we are the first one that expose new functionality usually on a weekly base after that we leverage on we extend that into the entire HP software after that to HP and only after two weeks we all of our customers have experienced that that functionality I can tell you that another benefit that we got from that which is not related to QA is that customers that we have that want to have insight to what is coming can take those environments and integrate into them and to see what is coming soon to their environments we usually take those two weeks to fix all the problems that that our customers might might introduce if we would release that before that it really gave the QA or us as the people who are looking at the entire quality piece to be much more peaceful and to be willing or or to be okay with the fact that we are taking more risks okay we know that if there are issues we will see them before our customers will experience them and this is actually part of the reasons that we we have higher quality one of the challenges that I had as a QA manager was to know what are the what our customers doing to understand what are the main flows that we need to test and validate them we do use Google Analytics on our environments and you know what are the flows that our customers are doing those flows are being used also by us as the people who define the product and know if what we we wanted to push to the product really happen but also by the QA you know what are the main flow where we should invest our time and how we should do that and another place that we are starting to push stronger lately is the ability for the A.B. testing capability basically it's not it's not around testing it's more around validating what the customers needs and to see how that works but it really helped us to to understand the case the test cases and what need to be validated what what really the customers would do at the end we are doing that on a very small percentage of our customers and it really helped us to know and to get feedback early and to know if things we need to invest more in testing on that area so as I said that's a that's a journey that we went to we still have places that we need to improve ourselves I think that there are mainly around the security and performance we started the change nearly a quarter ago when we do see how it impact and the value in that and we need to do that more the A.B. testing it's another place that we are extending the usage of that but basically those are the lessons that we learned along that journey at the end of that basically what it enabled us to deliver new functionality much faster in higher quality and I mean I'm not talking about all the benefits of agile to make sure that what you provide is really what your customer needs but when I'm talking from the quality perspective it really helped us to focus and to know what we are delivering and to make sure that that's on the right quality that challenge to do that on a short cycle and moving fast that's a big challenge for for a QA organization and as most of people over here said before me that's really add a lot of fun to the process people are much more aligned and understand what we are doing and they are much more engaged to the process and to the product that we are developing. So just to summarize we talked about culture change and the organization alignment we talked about dev tester the assessment process that we went through in order to make sure or to see that we who are the people who can be our dev testers we talked a little bit about manual testing versus automation continuous integration performing the security in the cycle and outside of the cycle and what are the implications basically you can find we have a blog where we post not only that story but also other other lessons that we learned that you can look at and this is a link to our agile manager solution that you are more than welcome to take a look at basically that was it I want to thank you we have a raffle right now of the HP slate to anyone who filled the information in our booth so we can do it now Manish do we have that here yes okay I'll keep that if you if you need the links any questions before we do the raffle so yeah I didn't talk about metrics over here but basically we do look at traceability metrics of use cases and features to test that's part of the definition of done that we were talking about and to see that we are covering all of that we do have metrics that we look also on what the developers are really doing okay the code changes I didn't drill down into that because that's part of the capabilities that we have is to link code change into user story and test and basically to take all the metrics out of that so you can know I made a change I know which environments are affected by that and to see which test is I need to to execute and to look at the those kind of measurements on the process I didn't answer you according to your your face it looks like I didn't answer you so if you if you want I'm here so we can talk about that okay thank you very much