 Hi, and welcome back to Program Analysis. This is video number four in this lecture on symbolic and concoctic execution. And what I want to do in this short fourth video is to talk about one large-scale application of this idea of symbolic and concoctic testing in practice. There are many, many other applications of these ideas in practice. So this is just one example. But I think it's pretty nice to show that these ideas do not just work on these tiny examples that we use here in the lecture. But actually also can be used to find bugs in popular and very large programs. So this example of a large-scale use of concoctic testing is based on a tool called SAGE. So SAGE is an implementation of concoctic testing basically as we've seen it here in the lecture and was developed a couple of years ago at Microsoft Research. So what SAGE is used for is to test the robustness of a large set of programs against unexpected inputs that are read from files. So you can, for example, think of a media player or maybe an office-like application that is opening files, maybe files it does not trust. And where you want to check whether anything unexpected in these files could, for example, lead the program to a crash or maybe trigger some behavior that attackers could exploit by sending some malicious file to someone who then opens maybe a Word file and then lets the attacker exploit vulnerability in the underlying code. So in order to test for these behaviors, what SAGE is doing is to start with a few known input files. So for example, it takes known audio files, which are just regular audio files or maybe known office documents that are known to be good documents and then handles each byte in these input files as a symbolic input, which means that it's basically trying to change every byte, or at least it's trying to change some bytes in these input files, in order to trigger different paths that have not yet been taken while executing the tested programs. And in order to find out which bytes to modify and how to modify them so that you will then reach some new behavior that you haven't seen before, it's using this idea of concoct execution. So it's executing these programs, for example, MS Office programs, gathers path constraints while doing this and based on the path constraints, then asks the solver which will tell SAGE how to change the bytes in these input files in order to trigger some paths that have not been taken before. So this SAGE tool has been applied to many, many different applications that are relevant for Microsoft. So it has been applied to hundreds of applications and in this period of five years between 2007 and 2012, actually more than 400 machine years of computation have been used to run this concoct testing. So it's actually pretty expensive to run it, but at the same time it also has been worthwhile, which is why a company like Microsoft has decided to invest that much computational power into it, because this SAGE tool has found hundreds of bugs in these programs, including many security vulnerabilities. Just to give you a feeling for how this compares to maybe some other techniques that are used to find this kind of bug, one-third of all the bugs that have been discovered by any kind of file fuzzing during the development of Microsoft Windows 7 have been found using this SAGE tool. So it actually has had a significant input. If you're interested in more details, there's a nice paper that has appeared at XE a couple of years ago that describes this whole experience of running concoct execution on a large set of real-world applications and how the different bugs have been discovered by doing that. So let me finally summarize this lecture on symbolic and concoct execution. So we've seen that these two techniques both solver-supported forms of white box testing, where white box means that the test generation is supported by a relatively expensive program analysis that looks inside the program and tries to find out what happens if particular inputs are used and how we can learn from these inputs and the behaviors they trigger to generate new inputs that then trigger new behavior. This is done by symbolically reasoning about some of the inputs or maybe all of the inputs, and by creating new inputs that will cover not yet explored paths. In contrast to the techniques that we've seen in the previous lecture where we looked into random testing and fuzzing, these techniques are more systematic because they actually look into the program and they know exactly, okay, there's this path that I have not yet covered. So let's do this little change of the input in order to now cover this path that I haven't covered before. Whereas with random and fuss testing, it was a much less systematic way of generating new inputs by basically hoping that you would find or that you would trigger some new behavior. There's a trade-off between being systematic and being more probabilistic because the systematic approaches like this example of symbolic and concordly testing that we've seen here also typically are more expensive. So there's no free lunch and in practice, each of them can work well depending on what kind of program you're analyzing and what kind of behavior you try to actually trigger. There are also a couple of interesting open research challenges around this whole idea of symbolic and concordly testing. For example, it's still a not completely solved challenge how to effectively and efficiently explore a huge search space that is given by this execution tree. So we've seen that these execution trees can be infinitely large and even if they aren't, they can be very, very large and picking the right branches to explore further is still an ongoing challenge that people are looking into. And then of course, there are many, many other interesting applications of this constraint-based reasoning about programs. So for example, in the space of debugging or automated program repair where people are using these ideas of symbolic and concordly testing to for example, find out why a program is buggy or maybe even how to fix it so that the bug goes away. And this is already the end of this lecture on symbolic and concordly testing. I hope you now have a better idea of how this relatively systematic way of automated test generation works and also how it maybe compares to what we've seen in the previous lecture where we talked about random and fast testing. Thank you very much for listening and see you next time.