 Hello everyone, this is Misut. I'm very happy to be here and to be a part of this amazing conference and I'm very excited to share my experiences about some monitoring activities and some metrics to monitor in terms of the quality maturity or the maturity of our quality teams and we will talk about lots of details but first of all what I can say is I have worked in lots of different domains like defense industry, the industrial automation, IoT and robotics but what I can say is whatever we are doing we are generating some data and there are lots of metrics to continuously track to get some information about our progress and our maturity even if without the job that we are doing even if without what we are doing in our companies even when we are sitting at our homes while watching tv or doing something in our smartphones we are continuously generating data because while sometimes I'm doing something in my smartphone I'm getting some notifications like the screen time was higher than one hour higher than the last time the last week so which means we are continuously generating data, raw data is always over there but this raw data may be somehow converted to some available information so data can be put into valuable information and by these terms we can collect some ideas and get some insights about what we are doing about about our activities because yeah we are doing something we are making some progress and we are completing some tasks but to analyze this to interpret this what we are doing or in what pace we are doing this we can track this information track this data and by interpreting them we can convert this data raw data into valuable information there are lots of different data that we can track and combine and then do some calculations and eventually we can reach to some valuable information to get some ideas about our decision-making activities as well in these terms we can see how much progress we have made until that time and then in the same pace what what else we can do more in the future so if at that time if we realize that we need something more or we have to stop doing something then we can decide these kind of action items so we will discuss lots of details and here is the brief agenda about the talk we will start by first of all discussing why this is important what kind of advantages we can get from these monitoring activities and why it is essential for the quality or seeing the maturity of the quality what else we can do what other activities we can add into our workflows or our lifecycle our processes and then we will discuss the optimization of the metrics that we track and then customization of the environments because we may customize our environments to make our workflows and our lifecycle as transparent and as visible as possible and in this way by adding some custom fields into our processes into our transitions we will be able to see all the transitions all the state transitions on our tasks and on our progress we will be making our environments as visible as possible and then we will discuss lots of different metrics and different categories of metrics what kind of metrics we can track to decide about the maturity of our quality and then finally at the end of the presentation I will share you a small little practical application in which I automatically continuously track our quality metrics so let's start with the first one the first of all why we are doing this why is it essential because what advantages we can get of these monitoring activities or by trying to analyze the maturity of quality what can we do by these terms first of all we can see our or we can evaluate our current situation if we are good enough to proceed to the next steps or if we should change something we can decide this first of all we can see our current progress for example how much what percentage of the task we have completed so far or the covered portion of the requirements by the test cases we can see this maybe half of the requirements are covered by the test cases and still the second half our rating to be covered by the test cases we should define more test cases or we can see finished number of executions or how many bugs do we have or how many critical bugs do we have how many low priority bugs do we have we can see all these kind of numbers and all this kind of progress and all the situation we are in we can observe and track with this with the help of these metrics and on the other hand after seeing our current situation and our progress then we can make a feature estimation with a linear projection because for example in the sprint in the middle of the sprint if I monitor our my progress and if I see that I could not finish maybe even one third of the task in my backlog then for the second week for the second half I can decide that if I continue with the same pace then maybe I can finish another one third so it means eventually it will be maybe two-thirds of the whole task in the sprint backlog so it is not sufficient normally after my planning I was supposed to complete all the task in my sprint planning so after doing this I can take some action items maybe after I see that I already see that I will not be able to complete all of them maybe I will continue with the high priority ones actually I would already do that I should already have done it but somehow if I was not able to complete my actual tasks and I had some other tasks then maybe I can continue to concentrate on the most essential tasks in my sprint I can do this kind of decisions and take some action items if I need the thing is first of all see the current progress and then with the same speed or with the same conditions under the same conditions how much more I can do I can do this feature projection it will help me to take some action items in this way I can answer lots of questions like about regarding the number of the bugs in the product or the different aspects of test cases how major my test cases are or what kind of test cases I have or on the functional or some non-functional test cases as well or about the requirements or about the maybe the use cases or regarding the time issues like how much time I am spending on different types of tasks or how much time I need to complete some different tasks I will be able to answer lots of different questions because otherwise if I don't have some numbers or if I don't have some facts then the the things that I claim are just only some sayings without some evidence I will say that whenever the project managers ask me that how much time do you need to complete your test cases then I can say one week two week but based on what I am saying this if I do this monitoring activities then this will be an evidence for me in the past cycle for this number of test cases it took this much time to complete all the execution so this time for the next upcoming cycle for this number of test cases I need this time I can do this regarding the previous monitoring activities it will be as an evidence for me and as I thought of course we will get some insights collecting lots of different information help us to see the weaknesses in our processes for example I can see okay I have 10 bucks 10 major bucks all in the performance type then I can conclude that maybe in the next sprint I can more concentrate on the performance issues or performance test cases so these kind of weaknesses or the points to improve can be revealed by this kind of tracking the metrics and different monitoring activities so let's see and discuss what we can do to perform these monitoring activities in the best way okay monitoring activities is very essential and helps a lot and we can get lots of different advantages we can see our progress and our current situation and we can do a feature estimation as well but in what way we can do it in the best way one of the best practices is the first one maybe is to conclude on which metrics we have to cover or we would like to cover because there are dozens of metrics that we can track there are hundreds of thousands of metrics that we can track but going on with all of them is not possible covering all every individual single metric is not possible so what we can do is decide to a subset which can serve to our goals in the best way then continue with this subset optimize the metrics and decide which one is the best which one is the most suitable for me serving to my goals for example if my goal is to reduce the average test execution duration then maybe what what metrics I should concentrate on are the ones which are somehow related to the execution duration direct lower indirectly first of all I can track the individual average execution duration for each test case I can calculate this and continue to track if it is going high or if it is reducing in time because my goal is to reduce the average execution duration for each test case so I can directly track this but I can track some other metrics which can participate into this execution duration but of course I have to decide which one are the relevant ones because if I concentrate on some irrelevant metrics then I will lose some time so the first thing is to decide for the best subset to continue the track because otherwise it is not easy it is maybe impossible to track all of them and I will lose a lot of time but how can I decide which subset is the best one I can have some checklist like the correlation of the different metrics that I continue to track or the consistency of each metric they are reliable or they are predictable in time or they have discriminative power or not I can take these parameters as my decision factors and then I can decide to my subset in the best way maybe a second best practice is the customization of the environment because whatever tool or platform I'm using then it is representing my progress for example for the issue tracking system if I'm using JIRA then there are some ways to customize the workflows customize the state transitions for examples the for example the default workflow is starting from the open status or new status and then in progress and then done but for me it was not sufficient so what we have done in our project is to add lots of different custom fields like for example for our we designed a life cycle a custom life cycle for our test case test cases and then we added all the relevant fields over there like after and the test case is created after it is in new status then it goes into in design because first of all the test steps are designed and then after the design is finished it is reviewed by another colleague who maybe who will execute the test case so we set it under review status and then if it has decided that okay it is executable either manual or with the help of automated scripts then we set it into executable state and then if it is possible to automate the test code we start the implementation tasks and then finally the code implementation the developed code is done it is also reviewed and finally we set the test case to finalize state so in this way I can see all the transitions and in each step for example whenever it was designing it was being designed I can see how much time was consumed at this stage and then because on JIRA I can track that at what time it was set to in design and at what time it was set to designed so subtracting this time from each other then I can measure the time in between these two state transition points and I can understand that how much time I need to design a test case in average and we added some other custom fields for tracking and in this way we were able to get lots of different insights about for example the brands before being merged how much time was consumed for each branch when the PR was created and when it was merged or about the execution duration for each test case we were adding in this execution it was like 10 seconds and each time in after each execution I was setting this value 10 seconds or 15 seconds whatever and eventually I was being I was able to measure or calculate the average duration I will show you how we can do this in an automated manner because after each execution setting the execution duration for each single test case is almost impossible it's very difficult after these best practices let's see what kind of the categories of metrics we might have in our monitoring activities first of all the time related metrics because time is precious and very important and one of the pillars of the project management most of the time we are trying to schedule our activities and by what time we can start or by what time we can finish so mainly how much time do we need to perform some specific tasks we are trying to decide and we are trying to schedule them in the best way because otherwise we may be late and we may be behind of the schedule so we may have to pay some penalties due to the delays so that's not a desired situation so we are trying to schedule in the best way so maybe the way to do that is to continuously track the metrics and according to the lessons learned like how much time we consumed in the past cycles for the execution of the full test set or what was the time in between two different activities if we calculate them or measure them in our past samples and past experiences then we may be deciding them for the next cycles in a very healthy and consistent way another category of the metrics is the cost-related metrics because another pillar of the project management is cost because money matters everyone is trying to minimize or reduce the costs so we are continuously tracking how much resources we consume or how much cost or budget we are consuming for training activities or we are calculating the headcounts all the costs related including the human cost or the tools platforms or the activities in the technical matters that we perform so we are trying everything to reduce in terms of the cost so in this way what we can do is to calculate or track all the costs and see if some unneeded budget is consumed on some specific activities then maybe we can get rid of them if it is not really needed in terms of the quality of the project or the product so to do that we may be observing all the costs and deciding if they are really needed or serving to the ultimate goal of the project or not if it is something that is not really serving to the ultimate goal which is the quality or the satisfaction of the customers then maybe we can maybe postpone it or totally cancel it we may take some action items regarding it and eventually the third or maybe the and maybe the most important pillar of the project management is the quality pillar because yeah we are trying to reduce the time needed to complete our fulfill our activities and we are trying to reduce the cost to fulfill our activities yeah we minimize them but what about quality yeah we minimize the time we minimize the cost but we don't have quality so it doesn't mean anything it doesn't make a lot of sense so eventually we need some quality right because otherwise the customers will not be satisfied with our products they will not be using it so it doesn't make sense to reduce the cost or the time if we don't have any quality so how we can measure the quality it is not easy but still we may track some metrics and decide at least how major our quality processes are so one of the maybe the most essential one is some metrics collected over bugs because bugs are representing some aspects of the product or the project because it is showing some different aspects like the functional or the non-functional aspects or not only the product but the process as well how good we are performing our testing activities or the quality related related activities like we can show or monitor the distribution of bugs among their priority or severity levels how much impact those issues have for example if we find some issues we have a great impact which may even result in the loss of some customers then it means that we have some really major issues so in this way we can understand the quality or the level of the quality of the product or we may monitor the distribution of bugs among the component of the system on which they heap together on for example most of the bugs if most of the bugs heap together on a specific component then we can understand that we have a quality issue on that specific component or unit or subsystem whatever it is so then we may take some action items to improve the quality on that specific part of the system or maybe we can monitor the distribution of bugs according to their ages for example how old these bugs are for how long they are open still not resolved so in this way again we can understand that how old bugs we have in our system or in what pace we are resolving them how capable we are to immediately resolve those issues in the system we can get lots of insights analyzing the bugs there are lots of valuable information underlying the bugs we may somehow reveal them and convert these raw data into real valuable information and in this way we can collect lots of insights both in terms of the product and our processes as well so we can improve them but specifically on bugs what we can track is maybe the number of escape bugs because escape bugs are the bugs which were not found in the development activities or the testing activities but were found and reported by the customers from the real production environments so we may concentrate on this because why if they were not found in the early stages if they were found in the later stages it means that the fix would be much more difficult because the code is already in the production environment changing the design would be really difficult at much costy whenever the bugs are found by the customers in the real environment they will they will get in contact with the business team or operations team to take some support and they will talk to development teams or the leader of the development team and team leader will distribute the tasks among the team so it will require a lot of time we will lose a lot of time and this cycle will result in some extra costs so of course the ideal solution is to find the issues or weaknesses or vulnerabilities of the product in the development activities before releasing it into production environment so we may concentrate on a number of escape bugs and try to understand the root causes why we are having some escape bugs normally we were already supposed to cover all the use scenarios and find the weaknesses but if we have somehow escape bugs then we may try to understand why our test cases do not fully cover the use cases we may go over these kind of issues and try to minimize the number of escape bugs in the later cycles and eventually we have some inner quality aspects as well I mean this this as inner quality because normally they are not seen by the end users for example we have a code complexity issue in the product maybe end user is not aware of this and maybe it is still maybe even it is okay to have this code complexity because in terms of the usage of the end user whatever algorithm you are using is not really important you may use anything if you somehow succeed to develop the behavior but of course considering the maintainability or the long-term quality it is really important it is of great significance in terms of the recoverability or maintainability because if the code is not understandable or it is not meeting the standards of the quality and the other best practices for developing the code then it means that to fix the issues in the later stages will be much more difficult because the code is not understandable and whenever we have some issues it will not be easy to understand to change the algorithm or apply the fix so in these terms even though the end users are not aware of these kind of issues it may still have some impact on our quality processes and finally in addition to all these pillars of different categories of metrics like time-related, codes-related and quality-related metrics we may have some hybrid issues what I mean is we may combine the time-related issues with some other metrics or the number of test cases with the number of requirements we may merge lots of different metrics to decide about our efficiency or about our maturity like not only measuring the number of test cases but maybe measuring the number of test cases per requirement or maybe not only measuring the test execution duration but maybe measure the test execution duration per test suite so we may merge different metrics and in this way we can generate some hybrid metrics not only one aspect but regarding lots of different aspects something per some other metric regarding another aspect of the product in this way we may have some clues and we get some insights about our efficiency how efficient we are performing our activities and what percentages we have in this way we can get this valuable information and finally in addition to all these besides all these technical aspects we have another aspect which is the emotional metrics which is mostly related to the satisfaction of our employees our teammates because we are always talking about the aspects of the product and always talking about the technical aspects of the project management but how about the people because the satisfaction of the people or how delighted they are to be in this project to be part of this project it is really essential how proud they are to be a part of this project we should have some core values and we should continuously track this satisfaction of all our teammates because this is the culture of our organization otherwise the project can be somehow fulfilled or the product can be somehow developed but if we are not managing the satisfaction of the people well then after some time they will start to look something different look something new they will start to leave our projects and products product development activities so even though we are somehow developing our product it is very important to continuously track this metrics for the long-term success because not only the short-term success but if we aim a long-term success then the satisfaction of people the emotional metrics are very essential as well not only the technical ones but after considering all these both the technical aspects and the emotional aspects all the metrics maybe sometimes we have to be careful about some of them because some of the metrics maybe hazardous which means they may be misleading like only concentrating on the story points it may be misleading or only concentrated on the back numbers created by different people like if we evaluating the performance of each individuals by the number of bugs created by those people then it may be misleading because all bugs are not in the same criticality level or everyone is not concentrating on the same module so some people are concentrating on more developing the automation code or some other people maybe concentrating on only the execution of the test cases so maybe if we try to make a performance evaluation only considering the number of bugs found by the people it may be misleading so speaking in general if we are trying to track something strictly related to some specific numbers it may be a hazardous matrix please be careful about not strictly concentrating only on some specific numbers or specific metrics we should be really careful about considering the whole situation we should be observing the big picture not only a specific metric and finally at the end of my presentation I will share a practical application in which we will see how we can perform these continuous monitoring activities because continuously performing these activities manually is not easy so what we can do is to develop some automated code automation scripts and then continuously or periodically run this over some jabs or some maybe CI platforms so what I will demonstrate in this part is first of all I developed a code which pulls all the information that I need from JIRA the issue tracking system that I use in my project so I was performing some requests I was pulling some queries and then getting after getting the response I was parsing the response and was reaching to the relevant information that I need for example when was an issue created or when the issue was closed so after getting these two I was calculating the duration between the creation time and the resolution time so in this way I was measuring the duration for the resolution time because I already know the creation time I already know the resolution time so I can calculate the resolution duration so after collecting all this information eventually I was making some categories like the performance regarding tasks or the functional regarding related tasks the functionality related tasks and then for each category the average resolution duration I was calculating and if some specific issues were resolved in a very unexpected time like much faster than the average duration or much slower than the average duration then I was investigating what was the root cause for that because normally there was a supposed estimated average duration but it was out of that range so I was trying to understand what was wrong what went wrong to resolve that specific issue and after collecting all these numbers and all this information of course I have to store it somewhere so trying to keep them locally is not easy so what I was doing is to upload them into our cloud platforms in my case I was using Amazon CloudWatch services and calling those services I was uploading my data and it was continuously stored over there and finally I collect all the numbers I store them continuously so this is great numbers are perfect they are telling a lot of information to me they are representing a lot of story this is perfect but having only numbers is still not easy to interpret okay lots of numbers numbers but so what we have to interpret it we have to analyze it so to do it in an easier way is to transform them into some charts or graphs in this way we can easily see if the numbers are going high or they are reducing in time and if we have some limits we can set some boundaries or threshold values and if they are violated then we can immediately see if the limits are violated or not so having graphs or charts makes life much more easier for us to understand what the numbers are saying so we can use lots of different dashboards and monitoring tools like graphana to do these kind of graphs and eventually we can immediately see what's going on in our environments if the trend is going high or low in time we can immediately see so to wrap up what we have discussed throughout the whole talk is we went over some different metrics to track to decide about the maturity of our quality activities and we discussed the importance of these monitoring activities what kind of advantages we can get from them and then what we can do to perform these monitoring activities in the best way first of all we may optimize the subset of the metric and then we can customize the environment that we are using and then we can categorize them we can track some different categories for metrics like time related, cost related or quality related metrics and then we can hybridize them right instead of concentrating only on one aspect of the metrics maybe we can combine them, merge them and generate some hybrid metrics and last but not least of course we have to interpret them okay we are collecting some information we are getting some numbers we are collecting all this data but eventually what this data says this we have to understand we have to analyze and we have to interpret and take the needed action items because otherwise doing the monitoring activities does not make a lot of sense we will have only some numbers but eventually these numbers should serve should contribute in the improvement of the quality so if we do not interpret what they are saying to us then maybe it will not be a lot of advantage that we will get from them so this was the summary of what we have discussed in the session so I hope you have get some insights and wish you to enjoy the rest of the great talks in this great conference thank you so much for listening and do not hesitate to reach out to me or feel free to get connected over these contact channels thank you so much