 To evaluate the verify approach to wireless protocol validation We performed what is common which is taking a reference implementation first verifying that the The system doesn't find bugs in an implementation that we don't think has bugs And then systematically introducing various types of bugs into the system and making sure that the system can detect them as Violations and so there were several different bugs that we introduced into an ns3 implementation of the 802.11 protocol A couple of them had to do with sequence numbers So in certain case the device under test would not increase the sequence number properly in other cases it would increase it too much And the second a second class of bugs were more semantic bugs So for example the device under test would retransmit a packet even when it received a knack or not Retransmit a packet when it didn't receive a knack and these are bugs that you would hope that a valve a validation approach like this would be able to catch The loss rates between the device under test and the sniffer and the endpoint in the sniffer were set at 10% Which means that the sniffer was receiving about 90% of the packets headed in either direction And we varied the loss rate between the endpoint and the device under test to stress sort of these components of the protocol the Acknowledgement and retry and the sequence number issues And what we see here is is that so that we look both at precision and recall as is typical So the precision gets quite good once and so the the parameter here that was used K determines how many packets the sniffer can miss in a sliding window So remember we in order to make the search feasible We have to bound the search space and so what K says is that if a particular mutated trace would require the The search to miss more than this many packets in a particular sliding window We just abandon that part of the search and report a violation And so for this particular protocol what we can see is that the precision Gets quite good once we get up to about K equals 25 so we're allowing the protocol to drop 25 packets in a particular window and Recall at that at that rate is also pretty good And so there is a combination of parameters that produces very good precision and recall meaning that the validation approach is able to find Bugs that we introduced in the protocol with very high probability And so this is a nice riddle it just you know indicates that for this particular protocol There's a combination of parameters for the particular heuristic search that work well And you know and this was like a you know a nice result There's also some interesting that the fact that the recall doesn't vary as much according to K Is on some level a feature of this particular protocol because this protocol sends Acknowledgements or sorry this protocol sends retry packets that are identical to the packet before them And that causes this sort of difference in how the recall responds to K We also actually found a real bug in NS3 while we were experimenting with verify There was a bug in rate control algorithm that caused that the pro to the implementation to get stuck at a lower rate And so this is kind of cool when you go out looking for fake bugs, and you actually find a real bug in the system It's kind of a nice sign that then maybe you're doing something right This work was presented at the runtime verification conference a few months ago won the best paper awards That's pretty exciting And we've continued this work and a new paper on this is under submission and in the new paper What's pretty exciting is that we're actually extending the original framework in several ways but probably one of the most interesting is Trying to improve the coverage of the validation approach so let me give you this example where I have an Xbox and an Xbox controller and Let's say that for this particular protocol, which is again a toy protocol. That's very simple there are two states that the control that the Xbox controller can read reach s1 and s2 and S2 is reached when I receive a particular packet of type P1 from the from the Xbox console and S1 is reached if there's a timeout So if I don't receive that packet when I'm an S0 and what I might want to do is I might want to force The protocol into state s1 for some reason. I want to see what happens. I don't have coverage of this Maybe the reference implementation is good enough that it's always sending P1 And so I want to try to explore all the states in The state machine in order to validate that the implementation works correct in all possible cases And so rather than allowing the controller to send P1 across the air The sniffer is actually going to shoot that packet down In real time So I'm going to jam that packet on the air to make sure that the controller never receives it And the controller is now going to end up in state s1 And so we've taken the sniffer from this purely passive part of the system and now we're giving it an active role And there's a couple of cool challenges that go along with this One is that we actually have to be able to figure out which packets to jam So we're using the verification framework to build what are called jamming policies that then get implemented at runtime And those policies are designed to try to make sure that we get good coverage of all possible states that the device under test could Reach the second challenge is more technical and that's just how do we recognize that the packet meets the jamming policy and Jump in and jam it fast enough to actually prevent it from reaching the device under test And that's sort of a fun technical challenge Jing Hao who's leading this project has been doing some really nice work at getting this to work on a USRP2 software radio On the actual FPGA that runs on that radio so that this can be done fast enough to Sort of snipe the packet out of the air. So this is the extension of this work It's very cool. There's some again sort of a nice mix of sort of algorithmic and technical challenges That are going to push this work forward