 థ్లివా మాస్లిటిియాస్లికూ రెసి ఆర్లివాసి. ప్లిందాదాచా వౕంట్లి. ఆదాటిస్ంస్లులు సింది ప్టంత్ంనై ప్స్లిది. so where we will go through some of the software matrix and the tools that are used it is part of the static testing and this unit is about static code use and matrix and we know that in the last session we talked about the matrix the importance of the matrix so what is the basis on which we decide the matrix especially for the software size so we know that we are not deciding the size by the code not only the code but the complexity and structure of the code is also important I mean the program that is underneath the embedded software that is how we calculate the matrix so basically these matrix are required on a day-to-day basis or week-on-week basis or importantly basis or based on the customer inputs is to make sure that how well we have done the program testing how well we are on track how well the program is progressing likewise so we also have a test matrix on a cycle analysis communicating the reporting so we basically identify define then we will communicate the identifier matrix how it should be entered how it should be used then with the help of that we will evaluate capturing the data and the matrix calculated using form formula on the captured data then we will try to report it effectively with various formats either using the tool or with the help of any documentation that is useful for the stakeholders similarly we will take the feedback from the stakeholders about the complete report of the test matrix then we have studied about the software testing matrix types manual performance and automation testing matrix so manual what we do is test case productivity execution productivity defect and execution efficiency severity all this manual type of matrix we collect in performance in terms of execution some of the scripting and that especially the performance of the whole environment and the technicality of the tested program will be reported and some of the important matrix also we have studied like test case productivity so code number of raw steps divided by the number of hours that took to develop that will give the steps per hour is what the test case productivity that means these many steps can be developed per hour so based on this this will be used for estimation and the improvement over the period similarly we have defect acceptance in terms of how many defects are valid and what is the total number of defects that gives the percentage of defect acceptance similarly we have defect rejection total number of defects that are rejected divided by total number of defects will give a percentage of the rejection ratio similarly test case execution productivity number of test cases executed divided by the execution efforts in terms of hours will give the productivity of the execution so it is more productive if it is 100% so it is based on hours next one is the efficiency so efficiency is basically important metric to determine the efficiency of the testing team in identifying the defects and execution of the software testing then we came to defect severity index in terms of how senior of the defects are so this is calculated based on formula which is severity index into number of valid defects for that severity we have a total number of weld defects that will give the severity index and that can be divided into two parts severity index for all status defects severity defects severity index for open status defects similarly the automation coverage in terms of total number of coverage percentage we will report the example there are 10 test cases that are automated out of 100 tests then the coverage is 60% if 60 test cases are automated automation covered the total number of percentage is 60% then there are two important metrics in terms of progress on the embedded software testing or schedule variant and effort variance so actual effort divided by estimated efforts into 100 is what effort variance so this should not be more than 5% usually the embedded industry variance should not be more than plus or minus 5% if it is minus also it is not good if it is plus also it is not good but this we have underestimated or we would have worked that shows the index basically on how well the estimation and how well the program coped up with those estimation the next one is being the schedule variance here as for the duration actual number of days minus estimated number of days we will be 100% so this also cannot be more than plus or minus 5% of variance because we cannot afford too much of overrun this called as two important terms schedule overrun that means if it is minus greater than minus 5% schedule underrun if it is less than 5% means schedule is so fast that it is well ahead of what is being estimated that is what it gives an indication so that is what is about schedule variance we have the scope change so scope change indicates how stable the system is and how much it has scope change from the original scope scope can be increased as well as decreased depending on the complexity of the project and how well the program went ahead in terms of development and testing so that is what we have studied in lectures by today we will continue on the six which details about the matrix for automation, automation scripting productivity automation test execution productivity, automation coverage cost compression all this will be automated so that is sort of a thing we generate is typically based on some tools like X and C for our own vehicle so we know that common matrix are used as a variance should be variance scope changes okay so basically for testing we need to deal with the facts in terms of managing what are the facts that we have considered for generating the test matrix basically so for RBT so we need to have a matrix particularly done in terms of reporting for example how many requirements are completely tested without how many requirements have failures how many requirements are untested so these are very important matrix it is not just enough to report a pass fail count failure count invalid count not applicable count there are different sorts of counts which are a result of either re execution or fix in the software code or fix in the requirements it is also important to highlight how many requirements in terms of what are the sort of requirements of functionality that are executed without any failures that is what this indicates similarly how many requirements really have failures that means what is the functionality that is failing that is what should be indicated using the matrix and the other one is the are the requirements being uncovered or untested is also another important element of the software matrix which needs to be factor basically this will be a building for the components in terms of how well how solid the testing strategy is and how well the product is basically both are complemented each other and as a whole it should be very well supported with this sort of a matrix so how do we achieve the competencies basically before release the software to our customers we will build all this matrix in terms of reporting so what are the fixes we have fixed what are the issues we had and how well the programs basically you will ask actually how do you test what is the strategy what is the report that you have all this will be reported and coverage is also one of the matrix so coverage will cover the requirements as I said in terms of the failures non-failures uncovered etc so basically more thoroughly tested products will have more confident outputs in terms of matrix so there is no surprise at large so definitely coverage is a strict concept because it has multiple dimensions including code coverage, design coverage, composition coverage all this test design techniques coverage also requirements coverage likewise we have many aspects factor for the reporting so basically at the end of testing we will report in terms of percentage the goal is to achieve 100% testing requirements or 100% requirement and in end whatever regression you do there should not be any failures so always the percentage should be decreasing that is what aim of the matrix report while we are reporting at the end of our testing in the software testing so let us look at some of the examples and the various charts that organization uses in terms of reporting the facts and the software metrics you can see an example basically being provided by ex black consulting services for example so nicely it is presented in terms of requirements in one column and the current progress is showing in terms of coverage that is this table talks about requirements coverage by area currently untested, tested and failed tested and passed so altogether the left most column should be 100% that is what it aims for so what are the types of requirements or categorization of the requirements that it looks for when the functionality that means requirements in terms of functionality how are they tested what are the failures what are the passes the next aspects are something like usability how useful the product is in terms of usability aspects in reliability requirements then we have performance requirements then we have installability requirements so all this categorization of the requirements we cannot any more like performance already there performance again we can subcategorize timing, speed, memory likewise communication also one of the requirement functionality or requirements category that we can have for all this you need to report on a total saying that how many are currently being untested how many are failed how many are passed here in the first one you see function there are 7% of the total number of tests that are not being tested there is 93% that have been tested in that 3% are failed 90% is passed similarly next one 25% untested 50% passed so basically all the facts in terms of requirements it will bring out so very nice way of providing the report this way the matrix can be factored and presented so that the customer and the higher program management will be clear of what is going on in the software testing next one is again from the same website very interesting way of presenting the representation in terms of trend chart also called as a burn down chart that means how well the program is promising where it is going to end so what is the trend in terms of various matrix that are going to be that are being captured during the test execution this is reported using an excel sheet or there are several tools also can be used for reporting the matrix you can see so it can be represented anyway but here example this figure talks about the bugs open and the relation how it is being trended for the particular duration so we know that end date is here how are we going to reach it by the current trend so 95 2009 was the last date and the program is being reported every week starting from 719 the test program was started on 792 and till 830 august 30 this was reported and it was ending 95 so you can see there are two colored graph all the dotted lines the red one being total opened bug opens open and the green being total resolved that means how many bugs on the particular day are opened we should aim for the complete bugs closure that means the green should meet the red so that is what the trend should end with when we end the testing or end the program so here there are total number of 110 total bugs are predicted and start of the program so there is a zero bug and next week we have about you have to put a cursor here and move on the right hand side about 30 bugs are opened out of which still it is zero no resolution is there bugs are still open and during the second week we have about 50 bugs that are opened and at the same time there are about 10 or 15 bugs have been resolved similarly we have 1 2 3 4th week about 70 bugs have been reported and about 60 to 70 50 to 60 bugs have been resolved and in the 6th week we see about 90 bugs have been opened and about 80 bugs have been closed or resolved those bugs are resolved and in 6th week we have about 110 bugs and bugs are already opened that means testing is done but it is not fixed because not all the bugs have been resolved so about 90 bugs are there still and in the end when we reach the milestone there are been resolved this is how the trend set or trend chat will be drawn you can key in this details using an excel sheet excel sheet we need total number of open bugs total number of world and we need a date and the upper limit of the right hand side so with the help of that excel sheet there are formulas we can apply and draw the chart of course there are different types of charts it is up to the user or the test manager or the test lead how we can present it so that it is convinceable to the customer and the higher program management okay so this is the trend chart the first one the next one being the burn down chart that means this basically talks about the burn down of the program how I am going to meet on a given deadline you can see an example here take tasks as tests so there are say hours something like 250 hours are remaining and 25 tasks are to be executed how I am going to reach it so from the I will start from the left hand side with a total number of allocated hours and as I progress my hours are going to decrease accordingly the tasks also expected to be reduced so that many tasks we have reached and when we are left with near 0 our task also should be reducing on the downside that is what this burn down chart will be prepared this also being developed using an excel sheet here you can see on the horizontal axis from day 0 to day 21 how the burn down chart is being projected in terms of day and on the column side we have two columns one remaining and on the right hand side we have the number of tasks remaining and completed tasks so you can add as many as metrics into this so there are about three metrics you can see one being the green one which is nothing but the ideal burn down that means from the beginning how are you going to burn down towards the end that means how are you going to complete the program on 21st day ideal burn down chart should look like and this one the orange sort of color is the completed tasks so on each given day or maybe alternate days or a week there are number of tasks that are going to be completed and that will be highlighted in this color so altogether those tasks should accumulate as 25 and the blue ones are remaining reports that means we have total allocated 250 hours and you can see the variation finally we are left with almost zero that should be done well before the day 21 and light blue one is the remaining tasks so from the day one on the 4th day we have all the tasks remaining and we have the full time available as we progress the tasks should reduce as well as the day or the remaining time will also getting reduced that's how the ideal burn down you can see for example we take on day 8 so on day 0 250 hours are left and tasks remaining are about 25 so this is how we are going to develop the burn down chart and you see an example day 8 we have completed about how many if I draw on the right hand side 1, 2, 3 so about 3 tasks plus we already completed on day 5 one task and day 3 one task so 3 plus 4 plus 5 so that means about 5 tasks have been completed this date, day 8 so you can see the total number of tasks from 25 to 20 it has come down similarly on day 15 suppose on day 11 5 plus another 4 9 so total number of tasks should become about 9 so this is what you can see 9 minus 25 remaining so meanwhile for executing or doing a task we have also consumed consumed some effort so the effort also accordingly has come down so remaining effort on each day and each reporting day you can see it is getting reduced it again depends on the resources how much effort they have put on location and overall program we can see the up and down are common but at the end of the program we should make sure that we are going to reach the tasks completed as well as the hours well within that the effort hours whatever we are going to going to consume should be well within the range here in this example we should not consume more than 250 hours we should make sure that when we reach 21 days we are done with 25 tasks so 3 parameters are important the duration the effort and the number of tasks with these 3 parameters we are going to report the burn down chart so this is basically being done with the help of several tools or MTP or Microsoft project plan so developing that and scope of the embedded software testing but typically these parameters are used for reporting the metrics of sample burn down chart all you have to remember is 2 things we will report it for embedded software testing trend chart and burn down chart these are important metrics this will give a clear picture of where are we heading okay the next one being the metrics capture and tracking tool what are the tools in general they use as I said there are several tools exist in the embedded software industry that could be excel sheet MTP or any other capturing and tracking tool like we have RTRT which gives a percentage of how much tests have been passed and failed we have LDRA and we have vector cast likewise there are many tools cantata like this lets look into some of the common web based tool easy to use and maintain in general they follow for the embedded applications those 2 are test link and bugzilla test link is for test management in terms of articulating the test model and bugzilla is used for defect tracking and defect management purposes these 2 tools are common tools in embedded applications and any applications you take but these tools are not for testing or testing execution but they use for capturing and reporting the metrics also for tracking so test link is from sourceforge.net you can go to the web website you can download and use it is an open source as long as you are not using for commercial purpose you are free to use that tool and download and use it for all test cases test strategy execution details of course it has configurations internal configuration management and you can use it for plugging into the external configuration also and you can find the tool as per your need that is also one of the good feature of that may be in the future session i will try to go through the sample test link and the bug tracking tool bugzilla basically it is a web based test management tool you need a server and all that if you want to install and use that in house we will try to see the basics of test link how it will capture and track so it is an illustration of the sample project test project that is the basic component in the test link so test project will have this many parts before that so definition of the test link what it says about different things that we need to use for the tool test case we need so basically test case describe certain via steps that means steps can be actions scenario and expected results will be part of that so test cases are the fundamental things of the test link test suit we need to have test case we need to have a test suit i will explain you each of this for the definition of the test link tool test plan sorry test project so these are the basic elements of the testing tool and so with that framework we are going to have so this framework we can so test case describe so through steps steps are actions and scenario test suit so basically test case suit organizes test cases to organize multiple test cases as a test suit it structures specification into logical parts the next one being the test plan so test plan is created when you would like to execute test cases so test plans can be made up of the test cases from the test project test plan includes builds, milestones and any of the binary is executable environment all this can be part of the test plan user assignments, test results all will be part of the test plan the next one being the test project test project is something that will exist till the end in the test link so till the project is closed the test project will be created in the testing test project will undergo many different versions throughout the testing lifetime because we are going to configure and reconfigure again and again inbuilt version mechanism so we can do that test project includes a test specification with test cases, requirements and all the key words users within the project have different roles so this framework will be assigned to different users so users will have their own roles and as per the role they will key in, use, update and all that activities they do in the extra so that they are integrated all together so next one being the user what is the user so each test link user has a role that defines available testing features the user can be administration or a normal user likewise depending on the role we will have a project team using the test link so this is a test link 1.6 example it is told here so this is from the source forge so test project has a test specification, requirements test plan and any other customised fields and reports on the other will be used and we can import and export the compatible elements into this framework depending on the type of import items we want to export the existing one then the users of this framework will be using the test link tool with this framework and he will play around all these elements like test specification requirements, test plan, custom fields and reports and he can also attach additional items like in his scripts part of the test cases he can attach within that so all this will be part of the framework that will be used in the test link so that is the important of matrix capturing tool and matrix tracking tool also this is basically used for test test matrix test cases and test plan and reporting so the next one being snapshot of the functionality that it has its illustration of the test link example so what are the different roles and functions that the people they will use the test link as a core team so users can be a guest, user can be a test executor user can be an analyst user can be a test lead or a administrator so in the end of this complete testing project the administrator basically backs up the complete database of the test link also is responsible for creating the framework adding the rights to the users modifying the contents and managing the users all this will be responsible responsibilities of the administrator and test leader will define the test plan and test suits to that test plan and he can create builds define the ownership define the priority all this will be part of the test leader and also he can write requirements or import requirements from the external entity and in parallel to test leader there is a test analyst who can also write and modify the requirements or import the requirements because test analyst and test leads are very important who can decide what test cases need to be used for what sort of requirements and test analyst also will help in terms of describing the test results and reporting it is also responsible for test specification and assigning the keywords test analyst is also can be called as test case writer or test case developer, test engineer it is up to you how you want to define in your project test executor is an independent person according to this you can have the same testers executing the test so basically he will execute and test results and guest will have some sort of read access who can browse test results he can basically analyze identify the complete project details basically this guest can be customer or PM project manager or any other interface people such as QA quality assurance quality people QA also can be part of the test link depending on the project nature it can be done so test writer creates the test project test lead will import the requirements from any work document or any tool produced such as doors from IBM or rectify and the tester describes the test scenario and context of the test cases and with the help of the test specification and he can also the test lead also can create regression testing aspects and tester can key in all the test cases and test lead will create this plan and you can add build any script execution mechanisms now the builders have developed the build the test lead will add those build and the manager can have a guest role as well to see the results and test cases mocking also the another important aspect of this thing is that traceability that is missing here so very very very important term that is being used in the embedded software industry traceability from design requirements is a must to be reported when we are done with the testing so there are several steps that are involved for testing usage once all these details are keyed in the execution is carried out and results are logged in and the project is being reported so this tool testing is used the next one being bug villa you can see an example sorry it is testing only the tool is snapshot how it looks you can see whatever we have seen the roles the previous use case diagram or the functionality where you can see different elements that are displayed here on the left hand side you can see test specification requirements keywords, test project management user administration on the right hand side you can see test execution test plan contents, test plan management each one has its own folders that folders can be customizable and these are the basic framework elements part of the test link tool so upon login these elements can be inspected, modified or reported or used so here you can see test link 1.7.4 when admin has logged in we can view this we can see this and add or delete whatever it is because here is the admin security activities so first step is already here is logged in and created the project then the second step you can see the steps also being specified here as an example test project management in terms of creating test project, editing, editing test project, assigning the user roles step 3 is the specification all the test cases will be yielding and addition printing all that will be done step 4 will have requirements specification document assigning the requirements to the test cases as step 5, step 6 will be keywords management in terms of traceability and all that test plan management so test project management is different than test plan management test plan needs should be having the user roles and environment and all that how builds are managed all this will be part of this step 8 is test plan contents will be keyed in step 9 is test cases execution step 10 is milestone management or build management step 11 into 12 executing and reporting the test so this is a simple snapshot of a sample program used in a test link like this there are several tools and those tools can be used depending on the project and its complexity that is being applied so accordingly the reports can be done for the defect it is the test cases and test management and the next one being the defect tracker or defect management it is also important along with the test case of management to identify the defects, track the defects for closure and report the defects to the appropriate stakeholders so that is why we use the metrics for capturing the defects and tracking the defects so it is called defect management so the test manager keeps track of following metrics as part of the defect tracker number of open defects for severity category at times number of salt defects in a period for severity category you know the severity of the defects depending on that you will assign the priority and you will report it and you will assign for the next fixes total number of raise defects for defects how many migration or retest re-inscussion of them for defect to close it or report it total number of retest so all this will be part of the defect tracker so test manager basically keeps track of this defect tracking okay so defect tracking is done with the help of several tools those tools are basically tied up test cases tool because primarily because it can be imported exported with help of tools such as a test link and bugs are also can be reported accordingly so one good tool is also open source tool it is called bugzilla this can be referred and used from bugzilla.org this is also basically web based general purpose tracker web based program to report update and generate the bugs and this has a server client link and all that with the help of web based application this report will be generated to capture the metrics so that is how bugzilla being used this is very similar to the test link so both can be used in conjunction to generate and report the testing metrics you can see an example of bugzilla is a nicely nicely reporting tool so the way it is being used for capturing the metrics and it is also used for tracking on a day to day basis or depending on the type of tracking and reporting mechanism this also web based general purpose bug tracker based on the html and the apache server and this is open source tool as long as the commercially is not being used it can be used as an open source and you can see in the next page a snapshot of bugzilla tool there are 4 windows that has been shown on my website that I have referred here and you can choose the way how you want to report it that is a type of image how you want to report window is there horizontal axis what you want to report vertical axis what you want to plot more line chart being used in this day to day progress being used for reporting the number of bugs and vertical axis the same report is being displayed reported with the help of a bar chart you can see on each day how many bugs so there is a uptrend so on this day you can see 110 bugs have been reported on the 6th on the other day it is 261 likewise similarly the bugs pie chart where you can see a pie symbol having a slice of resolved in pink color and new bugs in blue color reopened bugs in yellow closed in green and assigned verified all this can be reported with help of bugzilla tracking tool so this can be effectively used on a day to day in terms of reporting so users are the test lead for the people who wants to report will collect the data from individuals or individuals also keen as a user in the bugzilla tool so that there is no manual intervention much required yes they have to open the tool and the key in the data and the report will be generated automatically with couple of clicks so that is what bugzilla will do so with that we come to the end of test metrics capturing now we will go through some of the glossary that we were using so far or we are going to use this again from the software test book embedded software testing book test set collection of test cases test design technique test method test strategy we know that description of the relative importance of the system parts and quality attributes leading to decisions about desired coverage techniques to be applied and resources are to be located testing group of people responsible for executing all the activities described in the test plan testing technique standard description of how to execute a certain test activity test tool an automated aid that supports one or more test activities such as planning and control specification build initial plans and data test execution and test analysis test type a group of test activities checking the system on number of quality caltistics test unit a set of processes transactions and functions which are tested collectively testability review the detailed evaluation of the testability of a test basis test type we see this we can see testing a process of planning preparation execution analysis aimed at establishing the quality level of a system testware all products produced as a result of a test project testware unit testing testing of individual software components white box testing the test design techniques that derive test cases from internal properties of an object using knowledge of the internal structure of the object so with that we have completed the unit 3 test matrix test analysis and code reviews we will before we end we will just recap of unit 3 so what are the three items that we have covered so we have covered static testing this is definition static versus dynamic testing static analysis reviews inspection and test processes control coupling and flow data coupling is also nothing but flow control flow analysis then we had gone through software complexity the formula using mackabay complexity mackabay software complexity also we took few examples with the help of static analysis to understand what is the space or MISRA etc we generated software complexity and static analysis reporting parts then we had gone through some of the worst case execution analysis aspects and tools that are used in terms of reporting the deficit worst case execution time and we had seen stack analysis, stack workflow coding standards, common objectives, guidelines reviews inspection process program inspection program work through and review process peer review processes what are the types of peer reviews offline work through inspection and test matrix we saw in today's and yesterday's class lifecycle matrix types all types of different matrix types and matrix lifecycle test matrix and reporting it and we also touched based important matrix details test case productivity, defect acceptance defect rejection test execution productivity, test efficiency defect severity index, automation coverage effort variance, schedule variance so these are some of the important matrix that we had so with that we come to the end of all this so we will start the next unit which is nothing but software integration in the next session