 we will come you to the next session of software testing unit 2 testing methods lecture 18 series of this unit 2 today we will go through some of the testing techniques in terms of white box and in our yesterday session we had discussed about model based testing is another software testing technique advantages we had seen and we took a sample model and how we can generate the test cases for each of the model representative elements and then the testing and we also understood about the tasks that are required for doing the MIT model based testing and we had seen some of the models in software testing simplistic approach we have studied in terms of a calculator as an example then the flow that can have for the model calculator also we understood and the feature wise or the functionality wise testing of the MBT also we listed out in terms of scenarios as well as the functionality as aggregation application mode display etc taking in a calculator as an example then we listed out the classes that are required for the test model like model itself test generation test evaluation and the categories of those classes like in terms of the selection criteria technology execution options specification and test technology for testing evaluation of the tests that is been carried out and what are the options that are there for doing all these activities that are listed out under the categories that also we had seen okay so okay so today we will discuss about white box test white box testing technique before that I will just touch base some of the testing aspects which we had a session sometime okay so dynamic testing basically the testing done at real time or during the execution of the testing MS software testing is called dynamic testing it is basically divided into two groups white box and black box so there are various techniques for black box there are various techniques for white box okay so why we need black box and white box testing techniques because the software under test or the unit under test needs to be tested at different levels so we need to test as a function as a complete black box running on the field or at the real time or from the user perspective that is why we use the black box approach where the internal details will not be known whereas in the case of white box testing technique we will go through the internal structure algorithm code or implementation we can for the software under test then we will do the testing technique okay so basically we need to have a strategy for testing what is the purpose of testing the goal how to reach that goal all this have to be laid out then once we have these details we are going to have a test case selection methods that means how we can pick the test cases for each of the requirements of the function app which scenarios need to be combined together so that those scenarios can be executed in one shot so all test cases representing inside the group also need to be done while doing the test case selection then we are going to come up with the coverage how much of the functionality requirements we have covered so in terms of black box testing we go for the requirements coverage in terms of all the functionality coverage whereas in white box testing we will go by the code coverage okay so black box and white box testing test case selection methods black box functional or basically based on the requirements function specifications interfaces system specification as per the need that we have for the software under test or the system under test embedded system under test black box test design techniques are based on the functional behavior of systems without having any exclusive knowledge of the implementation of the code or the structure of the logic inside in black box testing the component is subjected to input and the resulting output is analyzed whether it corresponds to the expected behavior so we have all the external input in terms of a signal could be a discrete analog whatever it could be so all these will be driven and we see the behavior of the system resulting in an output and that output will be analyzed whether it is conforming to the expected behavior as per the conditions that is specified in the specification in the white box or the structural testing is also called logic driven testing it is purely based on the structure of the code or the implementation white box test design techniques are based on the knowledge of components internal structure and uses all the information about how the inside of an unit works that means we will be knowing all the detail that is there as part of the embedded software each and every components and units will be understood in terms of doing the test code this information might be a code and design of course for understanding the code it just may not be enough to go through the code so we need to have a design aspects as well understanding the power design aspects is very important because without understanding the design we need to understand the code so there are different types of design like high level design and low level design is most likely to be very linear and mapping implementation of the code so that will be studied and understood before we do the white box testing so white box test ensure that each implemented statement is run at least once and tested against correct behavior so we will study about what is the statement based testing decision based testing coverage and all that aspects in the today's and next classes so basically white box testing the internal details are testing in terms of statements structure and all that here is a diagram which use a fair idea about how a system can be segregated so we have component test integration test system test accepting test as one side and we go along for the testing at a higher level we do system test and acceptance test and at the intermediate level we do the testing and at the component or the unit level unit testing so the one that we do the component of the code level is nothing but white box of course integration test can also take credit of this white box or white box testing or unit testing can take some credit of integration test where there is a challenge in terms of white box where which we cannot cover with the help of white box so integration test results will be considered in that case black boxes of course purely based on the system test and partially based on the integration also so so these two are a basic techniques of course there is a one called a grey box testing which combines both white box and black box techniques so mostly grey box testing there is an application domain systems of complex systems but it depends on how you want to take up mostly they deal with white box and black box as I said the coverage is very important because we need to report the testing along with the results with a coverage aspect like we need to cover the functionality and we need to cover the code so white box testing deals with the code coverage so what is code coverage code coverage is nutshell is as simple as total number of executed code divided by total number of code that is there so we measure this in percentage similarly for black box testing we use the requirements coverage in terms of total number of requirements tested versus total number of requirements available so that is the percentage it gives for coverage okay so differences between black box and white box testing we just glance through this it is also called as input output driven testing or specification based testing here it is also called as logic driven or implementation based testing implementation is nothing but the code software is viewed as a black box without bothering about the internal behavior and structure of the program here permits to analyze the internal behavior and structure of the program software is tested against its specification whereas software is tested against low level specification or low level design with knowledge of code that means code and low level design both are used for doing the white box testing the data is derived solely from the specification that is the inputs and the outputs are derived purely based on the specification whatever its specific here test data is driven or derived from the logic of the program how the flow is also called as a control flow and another term they use it as data flow so these two are basically use here also there is a control flow but purely from spec or the requirement or the functionality here the flow is purely from the structure or the logical blocks I would say of the intended code intended system behavior whatever you would like to call so these aspects will be then in white box testing the control flow and the data flow whereas the control flow and the signals are used for testing the system with the help of the requirement from specifications is nothing but the black box testing okay so black box has its own advantage the black box tester has no connection to the codes we do not have to care about the code so always he look from the perspective that code should have some issue or the bug so in that way he will test it test cases are designed as soon as the specifications are complete so no need to wait for the implementation to be done so you can go ahead with the test plan test cases and all that based on the requirements the implementation could be still in progress but he can go with the test planning test case scenarios generation and whatever the high level testing is needed so there is a need of high detailed functional knowledge or system to the tester and its behavior that means we should have a sound knowledge of the system it is very important especially doing the black box testing this will be done from a end users point of view because end user should accept the system and this is the reason sometimes testing technique is also called as acceptance that means from the user's perspective or the customer's perspective it is very important to understand the whole system and accordingly the tests have to be written and tested so that the product can be released as a defect free product and black box testing helps to identify the ambiguity and contradictions in function specifications that means while writing the test cases or executing it we will know there are issues in terms of the system specification itself or the function specifications because there will be some requirements which are contrary to the other requirements some requirements says this way it should be triggered whereas the logic that is detailed out in other requirements may be contrary to the primary requirements so all this will be brought out because especially this will happen when we deal with the complex systems where the multiple teams are involved and there is some sort of understanding or some sort of a errors that have been injected when writing the spec. Efficient is the black box testing is very efficient especially on a larger and complex system purely because we approach the testing technique in a larger and complex system and it is better because just testing the code may not be enough or it is really not enough because some of the end product issues or errors or bugs it will not be or it will not be found with the help of white box testing okay black box testing disadvantage is testing with every possible inputs is very difficult because all signals and ranges of the values is very difficult and it is tedious also and all test cases we may not have be possible to execute we have seen that partition equivalence and boundary value analysis and all that in our previous classes so all we need not have to do but we should have a nominal range in between which is enough so that is subject to the availability and applicability so black box testing do not care about the structure and design coverage no knowledge of code basically but it is advantage as well as it is sometimes some of the low level details like target based hardware registers timing performance etc he may not be having a proper knowledge so it may be very important to understand about code for doing the black box but white box will definitely bring out all those aspects another problem is that when doing a black box testing if an error or fault is found it may not be possible to isolate where the problem is it may be anywhere in the code which is difficult while doing the black box testing whereas in terms of a white box testing we know what statement has fallen into an issue and for isolating this white box testing somewhat credit we may have to pay or some sort of a code inspection analysis may have to be done for detecting the fault code reasons for intermittent failures of the code cannot be found that means sometimes a failure comes sometimes the failure will not come while doing the testing is very challenging testing aspect it is called point-to-point testing in just very very nice whether user required functionality increment exactly or not so while doing that intermittent failure could be there very difficult to isolate due to what that failures are coming though on a nutshell while doing testing at black box level it appears to be a simple failure but that should be able to reproduce this is one important term that is used reproducibility that means we should be able to reproduce the issues that we found during the black box testing so that reproducibility is an important aspect and all the cases mostly should be reproducible but one or two cases or few cases it may be difficult those are called intermittent failures so those things are a bit challenging but with the help of white box we may overcome those problems some of the errors may not be able to discover until white box testing is done so we know that there are issues but until we complete the white box we will not be able to discover it and test environment is very important and we have a heavy dependency on that and the system should be stable then only the black box testing can be done effectively because the black box testing will as a whole it will come on the entire system and the entire system should be running under all the conditions that you implemented and those conditions are having a dependency for carrying out the tests and you should be stable and running all the time then only we can conduct the black box test okay white box testing advances as the knowledge of internal coding structure is it becomes very easy to find out what type of input data can be used in testing the application effectively where the problems are and each statement or lines of the tester will have a knowledge so with the help of that he should be able to churn out where the problems helps in removing the unwanted length of code which can bring in hidden defects that means there are certain lines of code which can be optimized or which can be removed as a defect because it will be hidden inside and we can improve the implementation with the help of white box testing helps in determining the fault code segments that means as I said the lines of code when we do a statement with the black box testing if the user has implemented wrong we should have equal to if he has used W equal to the meaning is entirely different whereas the first one is for assignment the second one is for comparison so the purpose of that line varies and fault is all day existing that is how white box testing will bring out all this call decode which is implemented discrepancies between the code and the requirements can be identified accurately that means white box testers will have a knowledge of requirement also and while going through the code and testing it you will have a in back of his mind requirements and the back of his mind having requirements he can definitely know where the problems are and discrepancies he can find out easily with the help of a code and the requirements understanding within accurately with an accuracy and no much system dependency unless there are signals that are derived from the external system that means he doesn't have to care about the external signals suppose some unit he is trying to do there are few parameters which are derived from the external system he don't have to trigger the external system rather he can feed the values for example whatever the expected values that signals are supposed to take so those are some of the white box testing advantages and we go through the white box testing as knowledge of code and internal structure is a skill tester is needed to carry out this type of testing which increases the cost in general that means we cannot afford to have a shufflement of the team again and again person should have a thorough understanding of the code we shall have a good knowledge of the code as well so that is the prerequisite and a skilled tester only can do this where he has a knowledge of the underneath the language and the underneath the compiler or the target where the code is being used along with the system as well but more emphasis is on the internal structure and the logic of the code and it is time consuming as the test all the aspects of the code that means if there is a one lakh lines of code is there all the lines of the code need to be tested so it's a bit time consuming as well as a lot of effort required and the cost is required in terms of ramping up the ramping of the team on putting the right effort right schedule and all that. Covering structure of code and its implementation as preset standards at time there is a need of system knowledge for the internal details of the implementable specially the tolerance of states etc that means it is not enough to have just the the internal structure and the code knowledge you should have a knowledge of the external signals in the system and the states at which the system behaves and the transitions all these details you need to have okay so that is the background of white box and black box system and we focus on white box because black box testing we have covered like partition equivalence coverage analysis nominal range and all that okay now we will catch base on the coverage testing okay so having understood black box and white box testing we will go through white box testing technique the internal structure logic and all the aspects of testing we will go through so the basic idea of white box testing is to test all the internal structure and the logic and the implementation that is there in the system so there are different types of testing available let's try to understand that okay coverage testing the weakness of functional testing that is the black box testing is that it rarely excites all the code coverage tests attempt to avoid this weakness by ensuring that each core statement decision point or decision path is excised at least once coverage testing also can show how much of your data space has been accessed that means we have a flow logic and decisions logic having decision on controls so all this may not have been excised in black box testing or the functional testing so how do we do so with the help of code we will do that white box testing all the aspects of that code that is there in the statement and the decision points like if else while do yes us all those will be tested the decision points and the condition points and the decision paths all this will be called coverage testing that decision logic can have basically derived from the charts such as if else while do processing processing we have then case I am taking example it doesn't matter which language we are using see at a C plus plus whatever it is we come across all this decision points statements and logics right so these blocks will be basically excised to make sure that all these blocks are covered and the statements underneath these blocks or covering all this will be excised so that is where we have a weakness in the black box testing or the functional testing because we test the whole function not addressing every path that is underneath that also known as white box test or glass box test some people say glass boxes but we call it as white boxes coverage tests are devised with full knowledge of how the software is implemented that means we should have a full knowledge in the control of the code that is with the permission to look inside the box means purely it is internal structure of the software it is also called structural coverage very important term where the complete structure of the implemented code is excised to make sure that there are no defects so that is why it is called as coverage testing and why white box testing white box test depend on specific implementation they can't be designed until after the code is written means testing with design technique we need to have after the code is implemented so those are some of the coverage testing aspects that need to be done in white box test okay so what are the broad level coverage testing elements that are there as I said three types of coverage we do in terms of coverage testing they are statement coverage where test cases selected because they execute every statement in the program at least once minimum one time the statement have to be covered so that is where we use statement coverage and other broad level categories decision or branch coverage this test cases test the every branch that means we have a decision right like if else so if else can have true path as well as false path so both have to be excised in that false path can have again some old decisions for example let me see if I can this is a decision box and suppose statement is there compare something compare a signal against its range suppose a temperature so we will do the decision taking care here whether it is signal is true or false or signal less than some value if it is true this path will be there if it is false the most statements are excised it could be another decision box further down the line again we will have to excise leading to multiple paths it could be another true other false so basically we need to excise this this this one path this this this this this this this again likewise it will go the flow will go so all this because there are number of statements for true number of statements implemented for false and again we have a decision underneath that that decision also will lead to couple of paths in terms of true and false so all these have to be executed at least one while doing the statements covering that means we do the statement coverage each of the statements along with that we are covering the decision or branch coverage another one is conditional coverage what we do here is test cases chosen to force each commitment in a decision to take on all possible logic suppose this this is instead of for this less than 30 one more condition also suppose signal less than 30 and greater than 20 so there are number of cases which we can do so that could take another path is of true false though this statement one here statement two statement three likewise we may have different conditions that is existing instead of a decision in branch all this have to be excised and all the logic values that means to be taken care like less than 30 greater than 20 greater than 30 less than 20 like we have seen boundary how we do for black boxes same way applicable for here also where all possible logic values are taken so this is another type of coverage we have so basically three broad categories white box as statement decision condition and to achieve statement coverage every statement in the program is involved at least once during the software testing achieving statement coverage it shows that all core statements are reachable this especially very important because there are standard view on sanity that mandates this that means 100% coverage is mandatory statement coverage 100% is very much required may be I will brief one of the session about the ones which is an aerospace standard 100% statement coverage of course there are different safety levels AB CBE very safety levels are there in the world based on the level of safety the decision is taken at to see what sort of a coverage we need to have for testing what it box testing especially that is how one of the standards is similarly we have auto sr and auto muti so basically to cover the statements that are getting executed so white box testing in other words what are types of white box testing that we have so we have statements testing where all the statements are to be executed and we have a branch decision testing where each decision path and long stop that are need to be tested and we have a data flow testing where the data that flows within the embedded system need to be tested in terms of all possible values and the branch condition testing where the branch leading to different conditions need to be tested then we have branch condition combination I will put the different slides and we will detail about all this in a separate session and we have another one called modified condition testing it is also called as MCDC where there is a truth table they write it for all the possible conditions sometimes we may not be able to test all of them but it is again subjective to the level of safety and level of justification based on that we may have to do the last one is a linear core sequence and jump testing these are the technique that is being used so we have seen black box white box testing in terms of white box testing different methods statement testing branch decision testing data flow branch condition branch condition combination modified condition and you will see SCHG having said this we will take the first one as a statement coverage so what is a statement coverage a testing the idea with the statement coverage is to create enough test cases so that every statement in the source code has been executed at least once that means execute every statement in the code at least once during the test case execution when we do the test case execution all the statements in that module or the unit has to be executed or you should touch that whatever the test path that it is going to execute requires use of tools because manually it may be difficult to do it there are a lot of tools with the help of that we can do this statement coverage testing the tools I will list out the tools specify that what those tools are used in the industry such as LDRA vector cast RTRT etc basically what they do is they test hooks added to the existing software that mechanism is called instrumentation instrumentation skills perform also means instrumentation of the existing software in such a way that all the statements should be executed at least once with the help of that tool we will be able to do the workflow when using the statement coverage used to first execute our existing black box test cases that means basic normal in terms of the functionality that are created while monitoring this before this monitoring is all but the simplest test cases performed with tool support that means normal test behavior with the basic flow of the functionality will be tested when all the black box test cases have been executed don't get confused with the black box and all that what I am trying to say is normal cases where we excise the functionality of the particular module or the unit that will be excised first so the tool same tool will be used where being used for white box to generate the test and execute it so the tool will report which parts of the code that are not tested so then we are going to add those additional untested area in terms of the lines of code so the idea is now to construct the new test cases that will cover as many of the remaining statement as much possible so start with the part of the code that should be reached walk backward in the code to determine the values of the input variables required to reach the desired part of the code with the specified values of input variables check the specification for expected and execute the new test case while monitoring that means we may have to do back and forth in terms of reaching the code so how we can reach the code may be good with the help of the inputs that are required so the inputs to reach all the code will be through the data driven within the code that data driven aspects could be a parameter because we use a function which has parameters so these parameters will be triggered such a way all the statements with the possible flow of the executable statements which are not reached with the normal functionality will be covered so that is why we use statement coverage and the expected result how to expect is basically there are functionality wise requirements for that module what it is supposed to execute and what is the output of that so based on that we will set the expectation and the expected results those expected results will be monitored while doing the statement execution with the help of this instrumentation instrumentation is done by the tool for the white box but the white box the tool we need to configure it or modifying such a way that all these aspects in terms of normal functionality as well as complete statement coverage needs to be done one problem that I have seen you know the expected result should not be from the code that means we should not see the core in terms of the expected result we should see the functionality of the code that is being executed that should be taken or considered as an expected result while executing that basically of course if it is a individual statement having the local localization aspects then we can use the code itself as expected results but most of the case we may not be able to achieve because the code is supposed to deliver the required output based on the functionality and you see or dig more into the code the test one may get biased so first thing is to understand the system with the help of a specification then that specification he should match with the connected models that are underneath each of the functionality and that each functionality he should try to drive the different inputs and each functionality he needs to provide a expected result of the behavior in terms of its functionality so that is how we will be doing the statement testing and statement testing is done with the help of instrumentation and instrumentation is mostly done with the help of tools and coverage how we going to achieve statement coverage how much we have touched that means how much we are able to execute statement and against that total number of statements this will be in percentage so we are supposed to have a 100 percent sometimes it may not be able possible to reach 100 percent so it could be 90 or 95 percentage etc so whatever that 5 percent leading to not covered statement so we may have to do additional testing so additional testing could be with the help of code inspection or analysis whatever it is so those will be used for achieving the 100 percent sometimes say some statement in the code may lead to an interrupt and that interrupt may not be able to achieve with the help of instrumentation or some register checking or register operations those things may not be able to do with the help of tool or off-hand with the model itself there is a dependency on other things those aspects may not be able to cover those aspects we should be able to do with the help of debugger analysis or inspection whatever the possible alternates that we have for the statement test so that is about our statement coverage probably I will try to explain with an example C code in the next class and there are other white box testing technique and there is a C code you see here example code we will try to see that there is a white statement there is a E-statement so all these inputs need to be triggered for statement coverage so that all the statements are executed suppose if the count takes less than iterations and iterations is here how much 750 so we should be able to execute all possible 750 this will not be done manually but with the help of tool it will be done okay now coming to statement coverage testing or the coverage testing we use the term called instrumentation or the software instrumentation so what is instrumentation means so software only measurement methods are all based on some form of execution logging means we measure the software in terms of execution only how much software is executed is what we measure in terms of software testing the implication is that after the block is entered every statement is executed so by placing a simple trace statement such as a printf at the beginning of every basic block you can track the block and by implication all the statements in the block are executed that means we have suppose 10 statements in a block or a functionality or a model function this suppose has statement one or by semicolon you know semicolon executable terminator so three statements are there how do we know that each statement is executed so we can put a trace the trace could be anything but the tool uses a different tracing mechanism as a simplistic approach we can use a printf so that by seeing the report if i have a three printf printing some values for each of the statements i know that three statements have been executed so that is so the funda about the implication of each of the statements in this functionality or the block and if the functionality has several hundred for thousands of lines it is not practical to have inserted the printf by other terms that is why i told this process is called instrumentation this instrumentation is done with the help of tool there are a lot of tools which does all these places if the application code is running under the authors the authors might supply a low instruction logging service so sometimes authors itself will provide for trace in terms of the statements so if that is there then a trace code can tell authors at the entry point of each basic block the authors can log the call in the memory buffer the target system or report it to the host that means authors will have a function to see what are the statements that can be traced or not traced etc to feature or the functionality is available within the authors that can be used for tracing out all the statements within the block but mostly non-author standard systems they use instrumentation aspects with the help of a tool and even less instructive intrusive form of execution logging might be called low instruction low intrusion printf a simple memory write is used in place of printf it need not be printf printf basically depending on the i o that if we have it will print the value or we can use a memory write where we will write into some sort of a memory and that memory can be dumped to analyze or instead of a memory we can take a report in terms of printing it into a r s to 32 or any output that is connected so that the less intrusive form of execution from the taken care at each basic block entry point logging function marks a unique spot in the in excess data memory after the tests are complete external software correlates these marks to the appropriate sections of the code alternately the same kind of logging call can write to a single memory cell and a logic analyzer we will speak about logic analyzer I will tell what it is there is a hardware interface as I said r s to 32 or can or any of the interfaces we can use it for logging the traces for covering all the statements if upon entry to the basic block and the logging writes the current value of the program on both the ethics location memory then the analyzer set to trigger only on a write to that address can capture the address of every logging that is executed after the test suite is completed the logic analyzer trace buffer can be uploaded to the host computer for all this is what I told the logic analyzer sort of a tool is used to check the traces and log the execution so this method or the process is called software instrumentation if the system being tested is ROM based and the ROM capacity is close to the limit the instrumented code image may not fit so this is an interesting thing I will explain if the embedded software fits to the flash or the memory in the target base right so what will happen is sometimes after the instrumentation code that size may exceed so what we can do is we will reduce the functionality and test only the pieces of the blocks and we will do multiple iterations to covering so what we can do is you can improve your statement code using two more rigorous coverage techniques decision coverage and modified the condition of the next technique basically okay so there are other aspects of testing ray box testing and coverage testing tools and all that we will see that in the next session we will go through some of the words that we have used additionally I think after MBT we have coverage statement lines of code instrumentation etc so this need to be studied of course we will have some of the glossary initial state system state in which the first event is accepted input input domain input value inspection inspection we will study in the next week a group of review quality improvement for material so we will be inspection of the existing thing it's a process integration we know combining different components in the hardware software or software integration strategy will work out addition about how the different combinations from the integrated integration testing is performing the integration iterative development and iterative life cycle based on few enlargement of and refinement of a system through multiple development cycles of analysis design implementation testing so we have seen in the life cycles V model is used similarly iterative development is also about life cycle model that is being used known errors that is defects that have been found but not solved that means we already know the problem with the understanding that this known with the known defects we will go ahead with other known other unknown issues life cycle life cycle structure process by dividing into multiple process LITO we have seen it's a TM method where four core stones of structure testing LITO means life cycle infrastructure techniques and organization so that is about the class lecture 18 as on it we will continue the white box testing next class