 గ్రరక్తోలన్లి, షంవసిక్రి. వవటీట్యకంవి ఇందానూ యా. ఆనోమీందాడ్లుఇడిస్ల్. ఇపంవికాపైతాలులు. మెవిక్లిక్. నంికితిఉర్లిబిరిమంరూ పలూక్ మార్లు స్రిక్లు ప్ల్లిలులి నిలులాం. ఇప్ in the last class, so we go నిన్లిరికిక్లానిక్లిలిమారికర్లిత్లిత్లి. what they have studied in the last class so we going through the coding rules is one of the important static analysis of course so there are 10 golden rules suggested by Michael net double so those need to be applied reviewed and analyzed against the embedded software code and it is also important to know what we should be reviewing it at what stage so all this will be as per the plan against what we need to review there are guidelines and rules and those rules are all against each of these artifacts in the embedded software life cycle so requirements, functions, specifications, rules, specifications have their own guidelines and rules and checklists also will be different and these life cycles will have a transition criteria as I said in earlier class in terms of entry exit so the transition criteria needs to be satisfied in order to say that product is ready for going for the next life cycle output so we have how much to review also is one of the important factor so there are different reviewing body like peer level self-review offline inspection likewise there are many types of reviews we will have some guidelines and rules that is again subjective depending on the complexity of the embedded software system so that will decide how much we need to review the next one we studied about is the performance work-throughs and reviews so basically there will be a team involved while doing the program inspection they will off-hand will be given a set of documents to read and visually inspect before the meeting and the participants will gather in the con meeting the participants can be geographically more that's what we see in the industry so basically they meet each other the objective of the meeting is to find error but they don't have to bother about solutioning of the errors that is how to debug how to fix that all that is not part of this previous it just identifies the errors so in a walk-through group typically a group of developers with three or four being optimal number will perform the review again subjective depending on the complexity so one participant in this will be an author so the others will review against artifacts and will have a conclusion on the error in consent with the author of the program so there is a group code reading that means the entire code set will be divided into group and that group they will go through the code and they will discuss about the code issues and the inspections whatever they have done in terms of procedures and all that so they will fill up a checklist the checklist is also called as a tool so what checklist will have is a inspection in terms of declarations comparisons control flow the data flow the input output program interfaces all these need to be checked basically that's what they do and when they do the inspection there will be typically four to five people one of them will be called as a moderator so he is primarily responsible for all the activity and he also has to ensure the errors that are being corrected against what is been found as a inspection meeting peer reviews is something like the equal programmers or testers involved for the evaluation that means the group of people are at the same level in terms of the development or testing against each other artifacts they will do a review and of course here also checklists and guidelines are same as that is used in the other review methods any improvement also can be corrected on the updates optimization etc by the peers so that will be taken care as a bug fix along with improvement so it also gives an insight into the approach and methods at a higher level so the types of peer review process what we have studied is offline peer review walkthrough peer review inspection peer preview so basically the objective of all these will be to evaluate the deliverables for products to verify its completeness compliance and find the defects scope is to conduct the offline review of the different products which say example I have put it could be any of the embedded software life cycle product contract documents it could be a project management plan document or a test artifacts etc and deliverables as part of the offline peer review is level design test cases test plans anything that is comprehensive for this offline peer review and there is a four important process aspects entry criteria inputs outputs and exit criteria each has its own condition mechanism in terms of deliverables or the outputs so next type of peer review is walkthrough the objective scope entry input output exit criteria basically remove same but that needs to be identified for this so next one is inspection peer review where as I said in the earlier slide a group of people will gather they go through the artifacts in terms of the programs code the SRS HLD LLD and try to come up with the errors that are there in the existing program also suggest to improve in terms of efficiency of the program so this also has an entry input outputs and exit criteria okay so that with that we move to the next one test matrix basically test matrix is not just based on the lines of code it is basically based on the complexity and the tests how it is going to be conducted and how it is going to be reported basically customer how is going to be satisfied all the artifacts need to be collated and identified in terms of its importance to drive what is the matrix that is good for a particular embedded system program okay so we will move to the next session with that background so I will start with the test matrix see here the objective is the process defines the matrix and analysis analysis process for a projects support services lines of the line of business organization basically this defines the process this process also defines performance model for statistical management of processes objectives of this process are establish organizations metrics objectives aligned with the business objectives and apply them to the line of business process projects and support functions level specifying the matrix data collection so basically it needs to define the or establish the various metrics that organization needs for that particular embedded project or the embedded project that is falling within the LOB LOB centimeter line of business called in the embedded industry it could be aerospace the automotive or it could be a sub systems part of the the main domain so conduct statistical analysis of data that sub process projects support functions LOB and organization levels see here sub process are something like which are derived process from the main process projects you know entire project and support functions something like we have a project and the project needs support functions in terms of administrative resources skills HR could be involved anything all these matters in terms of dependencies or criticality for a successful testing program testing project I would say the main software project also has a life cycle of testing project so that is very important to have all this organizational aspects in terms of sub processes projects support functions LOB and organizational levels where we are going to make some key decisions to support the smooth functioning of the testing so all this will be articulated in the test matrix so that is one of the objective then the next one is establish and maintain a quantitative understanding of the performance of the organizational process and to provide the process performance data base layers and models establish feedback mechanisms and provide reports to relevant stakeholders that means for each of the process that we have established this would be a good feedback mechanism that will help in terms of correcting any issues that itself can have that means we have a review process and review process itself has some issues or some corrections are required so for that how we are going to evaluate it based on the feedback that is the feedback will bring out some of the improvements in the particular case and the particular process that is being followed so that will be reported to the relevant stakeholders such as test manager or project manager or test lead for ease the main stakeholders so the last objective is to provide data for process improvement so it is not only enough to have I just identifying the this mechanism symptoms of metrics it is also important to have some sort of a data which can help improving the project processes so that is also one of the important test matrix okay so test matrix again I will go through this very important quantification so quantification of the data is nothing but the matrix but the matrix is not just the lines of code of course lines of code is also one of the metrics here lines of code is size so after size is not based on the lines of code but size should be subjective based on the complexity structure of the program that is being used so basically attention is focused on control flow and data flow complexity structural metrics are based on the properties of flow graph models of the program is very important to have the complexity of the embedded system program and embedded system testing aspects so with the help of that the size of the actual matrix that is going to be evaluated in the reported will be done so test matrix importance we know that very important to have a matrix reported to customer and how much bugs have been fixed how many are passed how many are failed how those fails are addressed how they are going to be justified how they are fixed in different phases so what is the action taken and what is the trend so trend will see in the next slide today so basically the trend will give a clear picture of where the testing is going ahead so that is what it gives a picture of the program so very important to have a defect matrix trend as a monitoring of the aspect of the test matrix also it provides matrix will provide improvement for the current process in terms of how much we can optimize how much we can improve etc okay the next one being test matrix lifecycle so test matrix itself has a lifecycle it again depends on the particular project so this has been defined in one of the ISN journal I have put the reference in the bottom so what it says is there are four test matrix lifecycle aspects that will be delivered or that will be used so analysis, communique, evaluating and reporting okay test matrix lifecycle analysis has identified the matrix what is going to be used define the matrix identified define the parameters for identifying the matrix which are identified so these are basically the first phase of the test matrix lifecycle that is analysis lifecycle analysis phase so what we do first is identify the matrix to use so what are the matrix we are going to have just identify those it could be a pass fail count it could be total number of test cases mapping to the requirements it could be a defects fixed with a category like high category low category or defects of criticality defects of minor issues likewise so those these to be identified first what I am going to capture then once I identify that matrix then I need to define what that matrix is going to have suppose pass fail count what is that pass fail count things so we need to have a definition of pass means when the test is being passed when it is verified against certain expected results and the actual results in the expected results are matching it is defined as pass and those pass matrix are identified and similarly fail matrix also can be identified we need to define them the next one being define parameters for identifying the matrix identified that means the various parameters that are used for identifying in terms of the inputs or the transition criteria that is being followed the next phase of the life cycle test matrix life cycle is the come in it here what we do is explain the need of matrix to stakeholder and testing team basically who are going to develop this test matrix who are going to deliver this test matrix who are going to use it basically the testing team and the relevant stakeholders surrounding the testing team so those need to be clearly explained about what we have done in the analysis we have collected all the defects we have collected all the templates information inputs of the testing program and that needs to be communicated that needs to be explained to the relevant stakeholders because they are going to develop it or use it whatever has been defined educate the testing team about the data points need to be captured for processing the matrix so we have defined it we have identified it how are we going to do is all that how will need to be clearly entered or told to the testing team that is very important so that is what we do in the communication life cycle part of the test matrix the next one is the evaluating so what we have done in the community is we are told and start the activity of the matrix collection the matrix usage and all that so once we have the matrix available that needs to be verified that needs to be reported appropriately I mean report is the next life cycle part of course for reporting before that we need to have appropriate calculation in terms of percentage percentage calculations it could be trend updates any formula that needs to be used all this have to be done because usually this will be done using automated reporting automated way so how they do usually is excel sheet so excel sheet is a very good tool in terms of reporting the artifacts so all this will be part of the capture verify and calculate the matrix so verify is also one of the important aspects why because either it is been captured rightly and correctly without any errors so that is also important aspects basically what we do is we evaluate the captured data and verify against the data has been correctly captured and we do a calculations which will be which will be which will help basically reporting the matrix so is different tools for doing the capture and calculation the final life cycle phrase is the reporting so what we do in reporting is develop the report with effective conclusion that means we need to present it appropriately so that the customer it could be or to be higher managers or it could be a program management team so they need to be knowing what is going to happen what is going on and what is the defects how it is being captured how they have been evaluated all these aspects have to be presented appropriately so we need to develop a very good report in terms of objective scope description then we should have a evaluating mechanism what are the issues that are found in the testing and how they have been reported all this also we should end the report with a conclusion and summary saying that this product is fit for use or this product has some issues why those issues are there is those issues have been fixed or justified what is good to use and product has this many major issues this many minor issues all from the correctly reported that is what it means then it is to be once the report is done in to distribute to the stakeholders and respective representatives so we can also have distributed to development team representative who can take a look into this and try to improve or fix the issues in that particular life cycle also we need to this very important that whatever the report we have developed and reported is rightly understood and rightly being reached to the stakeholders so how do we go to do how we are going to do is by taking the inputs the feedback from the relevant stakeholders so that is what we do with the reporting life cycle okay so software testing matrix types so what are the testing matrix types that we have today for a embedded system project is we know that we are going to have different types of testing so they are all broadly categorized into automated manual white box black box all this static analysis could be review artifacts all this will be part of one of these three what are those manual testing matrix performance testing matrix automation testing matrix so we will study each one what is manual what is performance we what is automation so we know that testing can be done manually like analysis some of the assembly level debug or testing may not be possible to do the dynamic testing automated and all that so we need to use environment such that the environment is driven and verified manually so those manual testing matrix also need to be captured also we know that static analysis is also part of manual testing where we do the testing without executing the program so that will be part of this manual testing performance testing performance of the execution execution summary execution data it could be on a client side server side and efficiency all this like timing performance and memory all this will be separately reported using the performance testing matrix automation testing matrix so what we do is we will report the we will capture and report the matrix for the automation how much is been done what are the artifacts what are the failures passes during that automation of the tests let us try to study software testing matrix detail okay so manual testing matrix will have a productivity test case productivity test execution summary defect acceptance defect rejection bad fix defect test execution productivity test efficiency defect severity index so these are some of the testing matrix so some of the important matrix in this manual we will try to explain in the next slide so basically what it does when we go through the testing life cycle where we do a test case development test procedure development test script development test execution test execution refix regression testing all this so we will definitely we will have different types of artifacts accumulated in terms of test cases itself and the program underneath which it is been tested also can be used so test case productivity basically how much productive the test of team has in terms of developing test cases and test execution summary that means in terms of execution how it went well in terms of how much time it took for a manual test execution so what is what are the results printed all this will be part of this summary so that also will be metasized then defect acceptance whether the defects hundred defects are reported how many of them are really defects how many of them are defected defects that also need to be captured bad fix detect means the program has issues and the issues have been fixed but the fix is not correct then the test execution productivity how much productive the test team or the execution it could be a resource or it could be a resource from human whatever it is so some tests may take a few hours some tests may take few minutes depends on the complexity on the type of test design that has been done then the test efficiency how efficient the testing is the manual testing is then defect severity index that means how severe those defect are coming to the next one performance testing metrics we have a performance scripting productivity we have performance execution summary performance execution data in terms of both side and server side you can assume this as target likewise you can apply it is based in an example that has been taken then the testing efficiency of the performance testing then the performance severity index all the performance type of testing will have to be captured separately using the testing metrics guidelines so that is what the software testing metrics types we will pick up a couple of good examples of this metrics test case productivity so what is the test case productivity or total case productivity so the formula is this case or TCP is called as productivity is nothing but total number of test steps divided by efforts into step hours or inter steps are so that will give total number of sorry the total productivity of the test case development that means there are number of steps say 100 and effort took for that is few hours that will be divided against that and that will be nothing but the step hour steps per hour that is what is the test case productivity so example is given here so test cases x y z 1 2 3 4 5 so each one has its own number of steps the total number of steps are 183 so for example the effort that is taken for developing these steps is 8 hours so what is going to happen is the test case productivity is going to be 183 divided by 8 what is been spoken here in this formula which is going to be 22 that means the test case productivity is 22 run out of 23 steps per hour so 23 steps per hour is the productivity so to what way this is going to be useful what way it is getting used so we need to understand that so there are different aspects that are important why this test case productivity is used productivity is important in terms of improvement estimation and confidence this year these three aspects are basically the driving factors for identifying the test case productivity so if the test case productivity is good in the next cycle suppose from the previous cycle that means we have improved and if we take a new project or there is a testing project has come with so-and-so complexity so-and-so lines of course so-and-so type of testing required so we know we have taken this much to develop the tests and we will apply for estimating that new project that is the factor that we use it it again depends on the complexity but subjectively it has to be taken care based on this test steps how they are written whether it is similar to that or it is different than what we have done in the earlier project likewise so this test matrix is very important and the last one being confidence basically it gives a confidence for the program management and the test management by seeing the productivity how much property the testing team in terms of developing test cases so that is what the importance of test case productivity the next type of matrix is the defective the defect acceptance basically so we know that the tester will test and report the defects whether those defects are valid or those defects are invalid that also need to be identified from the developers or the tester themselves so he has written a test pops he has executed the test steps and it has failed let us say the failure could be there in two ways failure could be there in the software the failure could be there in the requirement or failure could be there in the design whatever it is on the other side also is important to know whether the failures are real failures acceptable failures or there is a issue with the invalid test case design or test steps for example suppose we have you know that in test case design we have a expected result column and we have actual result and expected result is spoken of true and actual result has become false so we report this as a failure suppose instead of mentioning the true the tester has wrongly mentioned as what will happen the actual report actual result and expected result will match and we report as non-failure so what will happen is it is falsely been reported as a pass but there is a issue so we need to correct that as a non accepted failure so this is very important in terms of defect acceptance so defect acceptance formalize like this total number of defects we have and how much of them are really valid for example we have 40 valid defects against total number of 50 180 percent right so that is the percentage that we are going to so this will give me 80 percent of defects that have been reported are valid and 20 percent there is a issue 20 percent are invalid defects that is defects are not really genuine or there is a issue with the test case or the reporting whatever it could be or a documentation so it is very important to understand the defect acceptance as one of the key matrix that has to be reported the next one defect rejection is also one of the important test matrix this will determine the number of defects rejected during the execution that means the reported defects are not justifiable or the issues that is found out in the requirement or design are not accepted by the development team those acceptance to rejecting rejection ratio is nothing but rejection percentage or dr percentage so defect rejection is nothing but number of defects rejected here by total number of defects into 100 that is the percentage so here is a valid defects here it is a defect rejected so that is what the percentage next one is being test execution productivity is also very important testing matrix because this will give a good estimation of how much it is going to give how much time it is going to take for a given product having so and so steps having so much so and so test cases so basically this test metric gives the test cases execution productivity which on further analysis can give inclusive result that means we have the test case productivity in terms of hours basically or days it could be so there is a formula here test case test execution productivity nothing but total number of test cases executed here by hours total hours it takes into 8 so 8 is what typically the allocated time in a working hours of the day basically so that is what the optimal value they use so where T is the thing but T is calculated as test cases so number of test cases executed at least once so number of test T of one is nothing but T of one is said to be number of test cases retested with 71 percent to 100 percent of total test case steps T 0.66 is equal to number of test case retested with 41 percent of 41 to 70 percent of total test case steps P 0.33 will give a matrix as number of test case retested with 1 percent to 40 percent of total TC steps okay so basically here idea is we should have executed the test cases at least once it could take multiple iterations that is fine but typically productivity is based based on the one time execution so that in the regression we do not have to repeat the executions as we did in the primary stage so that is what it means so retesting aspects will be defined like this T of one T of 0.66 T of 0.33 that means that is a percentage 33 percent 66 percent likewise so about one to 40 percent of total test case steps will take retesting and 40 to 70 percent of total test cases will take 0.66 as the test execution productivity and ideally 71 to 100 percent of total test cases will take the productivity as one so that is what it says so how productive we are in terms of executing against the hours is what test execution productivity the next one test efficiency how efficient we are in terms of executing the tests and reporting the effects this metric define determine the efficiency on the efficiency of the testing team in identifying the defects it also indicates the defects missed out during testing phase which migrated to the next phase so test efficiency is nothing but dt divided by dt plus du into 100 this use a percentage value so where dt is nothing but number of valid effects identified during testing d is nothing but number of valid defects identified by user after lease of application in other words post testing means during testing we have number of valid defects and the defects that are identified still after the release of the application is nothing but the the post release effects suppose we have detected from few errors and few defects in the pre-release phase of the perfect testing and that has been fixed retested and it is passed all this has closed and we have released still there are issues which are uncovered during the test execution so that uncovered portion of that is nothing but the efficiency of the tester and the efficiency of the testing lifecycle it shows so basically the number of valid defects if it is suppose dt is equal to so suppose 10 b defects have been identified and they are fixed and they are released and there is number of defects found out during the post testing on the field on the user side if it is say suppose 10 so what will happen is dt plus du equals 30 and we have dt by dt plus du dt is 20 divided by 30 into 100 basically so something like 60 percent whatever it is that means two-third of the things 60 percent of the defects efficiency we have that means we are efficient 60 percent in terms of testing the product still we have a gap of 40 percent it is not 66 equal to approximately so still we have issue with the testing that means we are not efficient so in the embarrassing industry typically they expect 95 percent this is what the metrics they follow because nowadays or it has to be very stringent and the companies are not offered to reproduce we test and all that because is going to add a cost a lot so we cannot afford to have any books or post release issues we have to be 90 or above percentage in terms of efficiency that is one of the important methods. The next one is defects severity index I will tell you overall so you do not have to bother much about the formula and all is basically identifying the severity of the defects that means we have about 10 defects so all defects need not be very severe it could be a major issue or minor issue or a trivial issue like this we can have a categories severity in terms of how much it is going to cost in terms of damage to the product or it is going to have an impact on the product functionality or the behavior so it could be a major it could be a minor so each one will have a its own weightage basically so that weightage is what will give you the index the defect severity index DSA is also one of the important shortcut testing methods that is there in the industry today and the formula is like this defect severity index is nothing but the sum of severity index into number of defects for that particular severity divided by total number of various defects so one can divide the defects severity index into two parts DSA for all status defects that means how the status have been reported this value gives the product quality under test and the next one is a DSA for open status defects this value gives the product quality at the time of release that means when we are going to deliver the product at that time we are going to have the open status defects so those severity will be reported for calculation of DSA for such open status defects so only open status defects must be configured so the formula is for that DSA is sum of severity index into number of open levels for this severity you have a total number of various defects so that is what DSA open severity index will indicate the next one is automation coverage so is also one of the important metrics we have so this metrics gives the percentage of normal test cases which are automated that means we know the testing cannot be done 100% automation so we need to do a subjective automation and subjective manual testing so there is a mix of both so how much percentage we can able to automate because automation we do not have a human dependency it is done by the machine and we know the productivity and we know the coverage so coverage is basically total number of test cases automated there will be total number of manual test cases into 100 that will give the automation coverage example if there are 100 manual test cases and one has automated 60 test cases then the coverage is 60% so that is what the meaning of automation coverage the next one is effort variance this is one of the important metrics that usually managers they do hello we know that how much of the test cases are automated against the manual test cases the next type of metrics is the effort variance this is one of the important metrics that is been used by the program management of the higher up basically so effort variance is the actual effort minus estimated effort divided by estimated efforts and multiplied by the 100 that gives the percentage so basically what we do is we know how much time how much effort it took for a particular testing it could be say 200 hours this is what is estimated basically that means start of the program we know that 200 hours is much is going to take for a suppose so much of test cases and actual if it has happened is some omitting let us say so what is the variance how much it is having variance nothing but actual effort minus estimated effort which will give you minus 20 usually it will be more depending on the type of projects appropriately it will be done so you have to by 100 sorry you have to by 200 will give you 10% basically minus 10% so similarly if we have taken the actual value as 220 hours suppose in that case how much it will become 220 minus estimated is 200 this will become 20 you have by 200 that time it is 10% so apart variance that means whatever the variance we have from the estimated efforts is 10% in a typical embedded industry the variance accepted is minus 10 to plus 10 so you may ask question why minus 10 sometimes what will happen it will be overestimated because of lack of knowledge or something so but we do not take that much effort so to be on the safer side what they will do they will add some 5 to 10% of overestimation then do the testing so what will happen is over a period that will spill over here and there but maximum allowed threshold is minus 10 to 10 less than % is what the variance that they can offer so that is what the effort variance is about the next type of matrix is schedule variance so what do you mean by schedule variance so we know here also estimation is there so estimation is done in 2 ways we know that typical estimation I am talking about there are different types in terms of efforts in terms of duration both are very very important in terms of schedule this is all schedule aspects okay so efforts variance will talk about efforts and duration variance will talk about schedule variance so schedule variance is nothing but the actual number of days minus estimated number of days you had estimated number of days in terms of percentage suppose for 10 days or 20 days estimated days is nothing but 20 days and actually it took 25 days so what is the schedule variance this also called as actual number of days 25 we have to negate that with the estimated days estimated is 20 so divided by estimated 20 how much will become 25 minus 20 is nothing but 5 why do you add 20 nothing but 1 fourth or 0.25 correct 25% so we have a variance of 25% usually typical it is not allowed to have too much of a variance schedule variance so better to have a limited or controlled schedule variance so that is why it is suggested to have SV and EV calculated regularly so that we know how much we are progressing so very important aspects of the embedded software testing so usually 5% is what schedule variance depends again on the duration and the complexity of the projects etc so 5% is what they allowed but this is very important testing matrix they use across the project throughout the project between as well as the beginning and all the cases so that is what SV and EV effort variance schedule variance is the variance of the estimated effort schedule variance is the variance of the estimated schedule last one being scope change these are the important matrix also the matrix this one indicates how stable the scope of testing is that means we know what is our scope so we need to be knowing how much it has changed from the original scope that is been defined and the scope whether it has increased or decreased during the testing or after the testing whatever it is so how it is calculated is scope change basically that is the matrix we are going to produce scope change is the thing that total scope minus previous scope divided by previous scope into 100 in terms of percentage where total scope is previous scope less new scope if scope increases total scope is previous scope minus new scope if scope decreases for example scope change so previous scope is suppose 10 that means 10 is my scope here 10 scope means I have 10 requirements to cover and total scope is 20 so scope change how is going to be calculated we have total scope minus previous scope 10 divided by previous scope in terms of percentage how much it is become now 10 minus 10 divided by 10 equal to 1 percentage 100 percent yes 1 into 100 100 percent of the scope got changed something like that so scope is increased because previous scope was 10 and total scope at the end of testing or due to testing it has changed it has increased 100 percent if scope is going to decrease then it will be decremented scope calculation previous scope minus previous scope is nothing but scope decreases that is what the total scope this originally I started 100 requirements as a scope to cover in the testing and during the period of testing that has changed scope of the testing added new scope requirements or added new test cases or decreased or decremented test cases or scope so the scope decreases or the scope increases so that is what the scope change this one the important test matrix they use across okay so that is what the software testing matrix is about so in the next session we will try to conclude about the atomization test matrix to manage how we are going to report it so what are the trends burn down charts and with that we will try to complete the unit 3 session in the unit 3 session in the next lecture