 No, he doesn't. Okay, hi everyone, welcome to the next seminar. So today's speaker is Hakim Dardospoom from Cornell University. Hakim, he is a professor there at the computer science department. He got a work from NSF, DARPA, IBM, Intel, NetApp. I don't know whether I forget anything. He got his bachelor in the University of Washington. And then his PhD from Parkway. When he sent his bio, he even put an exclamation mark in Berkeley. I don't know whether you want to talk about Sonic and some very interesting work they did. And they presented in NSDI in 2013 and this year as well. Thanks for coming. Great, excellent, thank you. So I was saying that it is great to be back in the Bay Area. I'm in upstate New York now, and it was actually in fact snowing yesterday. So there was snow on the ground when I left and it's nice to be here in the sunshine, the golden light. So as Yance was saying, I did graduate from Berkeley. I have actually been to Stanford a number of times. We actually had a course back in the day where we went back and forth each week from Berkeley to Stanford. So it's nice to be back. So today I'm going to talk about the cloud from a, really from a network perspective. And not just a network, but from the lowest layer of the network. So how we can actually improve the security and performance of the cloud. Sonic stands for Software Network Interface Card. So I'll get into that. So the key here is that, you know, we've all heard that the cloud is supposed to save the day. It's supposed to come in, it's supposed to be like a commodity, just like electricity, whatnot. You use it for the government, the military, industry use it, it reduces costs. It's always available. It just does whatever we need to do. And if you ever have a problem, it's almost like that commercial one, they have that easy button through the cloud button. So this cloud is going to come in and save the day. What is this cloud? The cloud is essentially network access to seemingly infinite amount of computing storage. Okay, so this is according to the National Institute of Science and Technology. But underneath it, it's really a distributed system. Compute storage and network access. And at heart, I'm really a distributed systems person. My PhD was based on a system called Ocean Store. Ocean Store was a system designed to store the world's information forever. My thesis, if you're ever to download and look at it, it says that there was a challenge question that essentially said, how do you store a mole of bytes for a thousand years without losing a single bit? So that was the goal of Ocean Store, and that was what my thesis was based on. So it's really a distributed system, but at the heart of any distributed system is the network. And so as a result, much of my research touches network, compute, and storage. I have a system called Zen Blanket. It's essentially nested virtualization, and it was described in Eurosis. And what it does is it allows you to take an instance live and migrate it from one provider to another. For example, you can take an instance and migrate it from Azure to Amazon and then to Cornell and then to IBM. And the way, like I said, the way it works is that second layer, instead of starting a OS, we start a hypervisor. And with that second layer hypervisor, you can then do anything for the provider. And it does actually work. So we have a paper that describes that. Not only can you migrate an instance live, but now you can actually encapsulate some of your communication. So layer three and layer two is so you can actually emulate the entire topology. So you can actually migrate an entire system. It sounds like picking up the city and migrating it from one provider to another. What motivates this work is really trying to prevent vendor lock-in and allow you to use any provider interchangeable. And so that was described in a paper called VirtualWire. And also we have a system called Rats, redundant-related cloud storage that then essentially... You can view it as RAID amongst the cloud, so it then allows you to use any cloud provider interchangeably. And it reduces the cost of switching from one provider to another. One provider goes out of business as a nirvanic spit, or changes their pricing scheme, or you can maybe take advantage of some other lower prices like Google Compute for example. So the key here is that a lot of my research is motivated by distributed system that is hard to do. And today that's cloud computing. And I have quite a bit of work in networking, storage, and computation. Some of the interesting points here are in networking we have investigated a completely wireless data center. So using 60 gigahertz transceivers and assuming that you get rid of all liners, except for power. How would you architect a data center? How would it before? What would be the fault tolerance characteristics? This was a best paper that described that kind of paper design. We went a little bit further than just a paper design, but it was very interesting and novel. So there's quite a bit of work here in each one of those areas. I graduated students in each one of these areas. So the student in networking is at Google now, one is at IBM, and another one is at Facebook. But today, like I said, I'm going to focus on the network, on improving the performance and the security of our network and looking at the lowest layer. So really, when you get all down to and take all the cruft away, the network is the center. If you are any business, if you are a government, if you are a Department of Defense, if the network goes down, you're in trouble. So if you're in the military, the network has to work, and it has to work reliably, and it has to perform. And it turns out that there are quite a few situations where it doesn't work as expected. Here is a map of the National Lambda Rail, and what I did with the National Lambda Rail was I asked them to create me my own path. It's a $100 million network. It was designed for network researchers and people in the sciences, and it was interesting not many people were using it at the time. So when I came to them with this request, I was surprised they actually did it. So they created this loop, and the reason why I wanted to do that was I wanted both ends of the network to be at Cornell. I was doing some studies of primary backup storage systems from data center A to data center B, and it was just easier for me to have both data centers physically in the same place. This is described in final storage technology 2009. So what we wanted to do is it was a 10 gigabit network. We wanted to backup at 10 gigs and whatnot. It was a paper called Smoke and Mirrors File System that talked about this intermediate consistency state that once it's in the network, you can say that it's safe almost similar to a synchronous backup. So we actually had a real system that then tested this out. But what we found out was performance was terrible. And it wasn't because it was congested. Nobody else was using it, and this is almost a completely isolated link at the time. So that wasn't an issue. The packets were actually making it to the other end. So what was going on? And so we see here is a number of papers that then investigated what was going on with the performance. Here's a DSM paper where we instrumented all types of part of the end host and the network itself. It wasn't until this internet measurement conference paper in 2010, that we actually figured out what was going on. And what we did there, the problem was alluding us. But what we did there was essentially I worked with a physicist, Dan Friedman. This is, you guys know Mike Friedman at Princeton. This is his brother, his older brother actually, who's also a Iraq War veteran. He learned to fly helicopters and stuff like that. But anyways, he's a trained physicist. And so I was describing this problem and he became interested. And so he said, let me build a neck out of laser and oscilloscope, which was kind of crazy, so we used a lucid oscilloscope to capture these wave forms. But essentially what we wanted to do was hook this laser up to our router. We hooked it up to our campus router. We modulated this laser at 10.3125 gigabit. And we had to obey the 10 gigabit Ethernet standards. And the reason we wanted to do this was we wanted to control the spacing. So we wanted to test what was going on with the packets, spacing lines. And so we introduced these packets with uniform spacing. They were exact, no deviation whatsoever. We controlled the timing to the add-ons, which was a little bit insane. And then we captured at the other end, which was, again, a Cornell and an oscilloscope, what actually happened. And it turned out that even though we may have been sending at a low rate, the packets were being batched together within the network. And we received short bursts at 10 gigabits per second. And every once in a while, the ad-hose would have some scheduling delays. You would drop some packets and performance would go through four with a long band of delay products. But the key here is that all of this was completely hidden from us because it was below layer three. The space between packets are idle symbols and they're thrown away from the network interface pattern. So we couldn't observe the pattern here. So from everything we did beforehand, this pattern looked equivalent to this pattern. And we couldn't see that the traffic was actually arriving very burst, very short bursts of 10 gigabits. I mean, why is this a problem? As long as you have adequate buffering, you shouldn't drop packets, right? I mean, you're sending at the same rate. Yeah, so it shouldn't be a problem. In fact, we are sending at a low rate. So it almost didn't matter. This shouldn't be a problem. Yeah, it was just that the machines were using at the time, at the end-hose, every once in a while, between the Nick and the host, we would drop packets. What we surmise is that there was probably a scheduling delay of servicing the buffer. You'd actually overrun the buffer and you actually would drop packets. But the packets were not dropped within the network. They were dropped within the network. Yeah, so it was basically mugged in the end-hose. It was at the time we were using a machine that had a uniform memory architecture. So the bus was actually a bottleneck as well. And those things that were growing up. So you could maybe call it a bug. It actually was not as much of a problem once we went to a more newer interface and also had more cores. So the key here is that it may not necessarily have been a fundamental problem because end-to-end everything was 10. It's basically an unbalanced system where the person is provided by one piece of hardware saturated in the other bus. As such, I didn't know that these are working. Yeah. Also this phenomenon is pretty well known, right? I mean, even in what you were early 90s, there were papers by Rod Jain and others that talked about packet trains. Yeah, so that was... But that was with different either-net hosts on the network and end-of-being person. This was actually... There was no one else on the network. It was just our traffic. It was actually some routing elements. We actually have another pit for that kid. Mathematically describe how this could occur. I agree. That's not really too much of the issue. One of the issues was we couldn't really observe the problem in any easy manner. I mean, good Lord. I had to work with a physicist in order to figure out what was going on in layer 1 and 2. So there was no really easy way to do it. My trade is really more CS. So I'm coming from the background where anything below layer 3 to layer 1 and 2 is more of a black box. So for one, more of a systems person, what we want to do is be able to have some view into what's going on with layer 1 and 2 without being a trade physicist or hardware engineer, for example. So this actually shows our setup there with a laser and a oscilloscope. And what we've done now is we've essentially taken this entire setup, which was hardware, but everything else was, we actually recreated the entire network stack of software. We took this setup and shrunk it down to a PCI board that you can plug in, and then you can actually interact with the up, down to layer 1 and software in real time. And this is a system that we call SONIC. So SONIC is a software network interface different than SDN, software-defined network. And so the goal here is to give a systems person view into the entire network stack. Or this is software-defined networking just in a different layer. Right. So I guess what I meant is that it's kind of worth talking to what people think about as SDN when you're talking about separating the controller from the data. Yeah, often people just talk about the control plane, but clearly you can do things in the data plane in the layer 1. Things like software radio do something similar. Exactly. So this is analogous to software-defined radio. In fact, that's, there's a paper called SORA. And our design for SONIC actually doesn't exactly copy, but it is inspired a lot by the hardware design in SORA. And so what we want to do is essentially then take layer 1 and 2 and instead of having that be in black box allow software access to that entire stack. This would then allow you to do very, it turns out, allows you to do very precise network measurements by having access to layer 1. It turns out you're always setting a 10-ticket bit. And if I can touch every bit, then I can have access to 100-pico-second granularity. The NSDI paper this year described a covert channel. I'll talk about that a little bit, but we can essentially modulate packets at the sub-nanosecond level and send covert ones and zeroes and they actually work and can recover those. So that's quite interesting. Well, we can profile network elements. It's all described, each one of those. So the key here is that now we have view into the entire network stack. We can use this to increase the performance, at least understand the security, the availability of our network in a self-sense, our daily lives. So Sonic is a software network interface card and we can use that to improve things. Like I said, the key here is separating what is set so we're actually going to have the software through any manipulation of the bits. And the hardware is just then going to send the bits. So the hardware is just going to send or receive and any interpretation or manipulation is going to have the software. And here is just my note that this is not exactly the same as software defined networking. I do actually have some stuff in software routers and I do have some collaboration with Nate Foster as well as some software defined network. We have a Gates Hall at Cornell. We just got it, just moved in in January. And we're putting in a software defined network in there as well. That should be ready to go pretty soon. But anyway, I'm not going to talk about that exactly. But, excuse me, so what I am going to talk about is how Sonic is built and some of those application network research applications a little bit more in depth. So, first of all, this is our general network stack. We have the application layer and whatever you want to send is broken up into segments or whatnot and then gets encapsulated into the transport layer, TCP segments or whatnot, the header put on. Then gets encapsulated more within the network layer IP packets with another header put on. Then gets encapsulated even more into layer two, the data leak layer to be an example with the headers, some preamble and a cyclical redundancy check. We want to make sure that all the bits are the same. So, so far, so good. This is just general networking one-on-one. And so what happens below this, a lot of people don't really concern themselves, especially if you're a systems person. But it turns out what happens is that each Ethernet frame is then broken up. This is pretending. It's then broken up into 64-bit blocks. Those 64-bit blocks then have a two-bit header put on it where you have a transition from a zero to one or one to zero. Each one of those 64-bit blocks that are then scrapping so that you have a mix of zeros and ones so you don't burn out your network interface card. Furthermore, each one of these blocks before they're scrambled, they were encoded to have a startup Ethernet frame. Was that S there or an end of Ethernet frame block? The T there for termination or a data Ethernet frame block or an idle Ethernet frame block. So these are now where you get 64-bit encoding. And that's also why it's not running at 10 gigabit per second, but it's at 10.3125 gigabit on the wire. And then we have the little bit lower in the physical layer. We then take the bits and change to the bit width down to 60 bits with the gearbox, and then essentially that gets serialized and then sent out to the network on the wire. So that's the 10 gigabit Ethernet stack in one minute. And the key here is that between every Ethernet frame are inserted some number of idle symbols. So if the start of Ethernet frame depending on where that frame starts you'll have some idle symbols and at the end or in between packets you'll have some number of idle symbols. An idle symbol is 7 to 8 bits and like I said since it's always sending each bit that is about 97 picoseconds wide or around that 200 picoseconds. So in some sense if you can just have access to these idle symbols and just count them then you have a very accurate sense of time, at least time relative to yourself. So the inner packing spaces. Can you put a belt patch What's that? A different protocol to add some gaps to the packet. Yes, this is the Ethernet puzzle. Yes. And so it's always sending. So if you're not actually sending data you'll send idle symbols. So you're always sending at 10 gigabit. That's why you have this very accurate notion of time. So all these photos was treated as black box as we said before, right? Yeah, versus this person. Yes. And this is all clogged with the, basically clogged with the sender and you sort of recover the timing of the receiver. Yes. And you're able to do that with essentially that. That's with that PMA and PND. And so the key here then is that all of this happened below this software hardware boundary which the network interface card did everything that I just showed you in terms of the encoding and then, you know, parts of layer three and up were done in software. Okay. And so as a result then all the timing information between packets would then be lost once you get to your application. So your application then has no way to know what the actual spacing was between packets because we lost some of that very accurate time information. So what we want to do then the goal of Sonic was then to push that boundary low so that you could actually access layer one in software. And so in software we could then have that very accurate notion of time. We could just count the idle symbols and we know that 700, 700, 7 nano seconds have passed. Or 7 nano seconds. You just literally just count the bits. And so we're able to do that and we're able to do that in software writing just C code. The paper that was described this year literally was 51 items. So now we can talk about the inner packet gap that's what the IPG is or the inner packet delay which is the same thing except you add in the size of the packet. This is different than the net FPGA project which looks at subsets, routers and some of that functionality in hardware. So we actually have a board from the same company and they're actually very similar boards and one key difference is that we actually have that physical layer in the software. It wasn't possible to do that with the net FPGA 10G board because they actually have a physical layer in the chip that prevented us from doing it over the line. So anyway, so this just shows the difference. We had a question related to that. So then with SOC presumably you need more FPGA space or some FPGA space to handle the software later on which is offloaded to hardware and then FPGA. So for, yeah, that's actually a comment. So this is actually our design for Sonic. Everything below that red line is in hardware and everything above this red line is in software. These rectangular boxes are cores. So we actually do the encoding and scrambling for descending and receiving with separate cores. We do the creating of the checksum and the verifying of checksum with the separate core and then we even have some documentation for that kit that implements some functionality that we want. And we separate this. There's actually some shared memory and a DMA ring and everything where the hardware communicates with the software and within the FPGA that's when we change the bit bits and do all the stuff we need to do and then the transceiver which is right over here then what takes the serialized bits and sends that out on the wire or receiving. So this is the actual entire system and we do actually have these are a kernel threads as well and we also have some user level threads that you can write user level secrets. In fact this slide right here shows a packet generator and capture application where you may define where the end hosts are so the source and destination IP address and max and stuff like that of course but the key here is that you can also talk about how many idle symbols to put in between every packet. So this right here for the packet generator right here we're going to sit and loop and create packets. It'll insert exactly 12 idles between every packet. And you can give it a trace or you can give it some type of algorithm as well so that that can vary but the key is if you said 12 it'll generate exactly 12 idles and then it'll be exactly whatever 12 times 100 picoseconds is without any deviation. So it will be exact and then whatever we capture this just shows example of a capture loop right there then you can actually capture what the inner packet spacing was exactly. In fact in another slide I'll show you an example of that Sir tell me something I'm missing sir you said even in your original experiment those transmitters were doing its job perfectly right it was sending packets with regular interval and all of that. The receiver side the physical layer was doing its job and above that I was missing it out so doing all of that physically how does it help to solve the original problem does it? Yeah so with our design we actually took the scheduler away so there's actually no scheduler it's just sitting there in a loop and it's always servicing these DMA reads that we have what I talked about before there was a scheduler sitting there in a tight loop and you can imagine taking some type of delay due to a schedule and as a result that DMA ring may fill out so these are kernel threads and they are in a fairly tight loop we also have some very efficient FIFO FIFO buffers between them so that we can communicate so between the hardware and software of the DMA ring we also have a FIFO between core as well this shows, what I wanted to show here was what I was talking about which is that if we generate packets with exactly 12 idols in between them or some number of idols then this is an example of that type of experiment we have on the X axis here this inner packet gap measured in bits so for this particular setup running at 1 gigabit per second with 1500 byte packets we could put exactly 113,340 bits between every packet and the Y axis then would be the PDF we could do a CDF as well but the key here is that there is no deviation no deviation whatsoever we would then send packets into one port of a switch and then via another port of a switch receive it at another end for Sonic and then measure what we get out and then you could see how much the spacing between packets were perturbed so this shows again on the X axis that inner packet spacing and then the frequency this is the log axis for the Y axis it shows that there is some randomness within each routing element and in fact if we were to go over some number of routing elements you can then start to describe how batching packed these packet trains may occur fairly naturally so the key here is that in software and fairly easily the application I wrote wasn't too complicated I can now fairly accurately describe a sub-network element in fact I can even describe a network path I can create some fingerprint of a network I can also do the experiment that I talked about before with the physicist so Dan Friedman I can now do it fairly easily just write a nice simple experiment 12 idles or whatever the data rate should be however many idles there should be between every packet I write that and I can capture I showed you the code for that in the previous slide that one took us 10 months to do beforehand now essentially underground and have fairly accurate control and measurements and so now we can start to understand what's going on with the network a little bit easier we've also done this with the same experiment right here with some other network elements so this shows we have a Cisco 6500 router which was our production router on campus we have some other IBM or Cisco routers where we perform the same experiment where we generated packets at some data this is 6 gigabit per second 1500 by size packets and we measure how much that routing element perturbs the spacing and so this then shows essentially a different fingerprint if you will for a different routing element and how it responds this is a net FPGA so you couldn't actually perform the same experiment the far reason is that there is this level of arbitration within the net FPGA which progress you from having the same level of control that we have with this sign but so we can describe a net FPGA as well so the packet size is really bad this right because there are lots of packet sizes and less gaps yeah so packet size data rate matter so do you also have these kinds of fingerprint if you go over a multi hop part so that's a good question yeah so if you have multiple hop paths you could then and we do have this in our paper this year for NSDI we actually use that in order to create you know talk about this next a covert channel so we use this to understand over a network path what's the distribution that packet spacing gets inner packet spacing gets perturbed and then we use that information to then encode a zero and one in a timing channel so we then change the width of the packet so that we could actually measure how much it would be perturbed and that it would actually retain that level of perturbation going to the other end and we could say oh I set a one because it's far apart or I set a zero because it's close enough so I'll show you that in a second but the answer is yes we can use this to understand an entire network path but in this if there are certain boxes for which you don't have this fingerprint kind of a transfer function then does that yeah so we haven't gone very far in this you know in terms of figuring out how they these signals combine together we've got as far as what I just described a second ago for cool channels but to actually understand okay there's on this path you know there's ten routers and two Cisco and IBM and the type of Cisco routers we haven't gone that far to actually describe that so is there some live traffic no congestion or is there any other cross traffic I think is what you're asking so that would change the patterns a bit as well because that adds more yeah so actually let me let me talk about just a little bit about the you have a question? yeah have you considered beacons in other words is tracers through this occasionally so that you can actually profile on that as opposed to the other traffic so like I said we haven't gotten that far too so what we have started doing is just profiling different routing elements that we have and seeing how they combine together so one other concern is the supply chain management so if you had you know you don't have control of the entire manufacturing process of the switcher router and you know with the VOD for example there are concerns could there actually be some code or functionality in there where you can now exfiltrate information and so we are looking at that question with covert channels could you router some network elements exfiltrate data via a covert channel and so a covert channel you know you're hiding communication within the obvious communication so at some level there's two levels of covert channels a storage channel and a timing channel a storage channel you're changing bits within the protocol so the TCP or IP header you're changing perhaps unused bits to then encoded zeros and ones those are fairly easy to to do is also detect or prevent with the timing channel you're changing the spacing between packets and like I said you have a one maybe four apart so it is closer and usually this is done at layer three and higher so even timing channels are pretty easy to detect there's statistical way to try and hide it and then statistical ways to actually find them as well so what we want to do is create a storage or timing channel using the physical layer so we did this and if you were to look at the 10 gigabit ethernet standard you don't need to understand all of this this is a encoded idle block or starter frame block but the key here is that there are parts of the standard that aren't used so you can actually then use those unused parts of the standard to put in a zero or one we did this it works the problem is it all works for one hop two elements that are directly connected doesn't really work transfer because as soon as it goes it's that outer it doesn't retain that it actually decodes and then re-encodes so what we then looked at was a timing channel and it turns out that that actually does work so what we did here is that we actually have a packet that we then make perhaps 100 nanoseconds further apart or 100 nanoseconds closer together so just slightly further apart or closer together so that most elements wouldn't even notice and it would be completely invisible to any software ant coast and to see if that actually works perhaps even to our surprise it not only doesn't work but actually works quite well with a people ask what type of capacity can you have with this covert channel so with a covert channel it depends on the packet per second rate if you're sending at a gigabit per second with 1500 size packets then that packet per second rate translates to about 80 kilobits per second if the overt rate is about a gigabit per second then I can have a covert rate of about 80 kilobits per second and this is directly related to the packet size and the data rate and it actually works it works with cross traffic so we did this on the NLR at this point the NLR had one to two gigabits of cross traffic per hop that perturbed the spacing a little bit and it also worked with less than a 10% of the error rate so we're actually able to recover more than 90% of the bits and if you use poor error correction you can get that much much higher and like I said this would be completely invisible to any software ant coast so this is what was described in NSDI this year how high capacity this is so usually when you talk about timing channels in order to hide them they're usually talking tens to hundreds of bits per second so this is three words in magnitude faster and more effective and also like I said invisible to to an ant coast how does this change as your link utilization increases because often on the real internet at some point you're going to get a puddle of that with the neck length which is 100% or you know kind of fully used to the extent that it works so the question is how does this change as you increase utilization or approach congestion so the key there is that you have to increase the amount of space that you put in to encode 80 or one so for NLR with about two gigabytes of cross traffic for hosts we needed about a microsecond if there was no cross traffic at all it was about 100 nanoseconds that means that that's like 20% that you try higher like 89 because if you think of being what you want as a network operator you sort of don't want over provision basically you want to be at the point where you have a little bit of congestion the answer is that yesterday we tried various amount in our paper and at some level it doesn't work so with the two gigabit cross traffic and one to two microsecond of modulations it actually worked with less than 10% of the area as you increase the amount of cross traffic than three or four on a 10 gigabit network path then it was no longer effective in a sense that we didn't get that below the 10% of the area and I do actually have that information but it's on my other laptop I was going to be able to flip to I have some graphs but I brought an HDMI interface so sorry not able to show that but in any case it is in the paper and it does show as you go we did experiments as you go more hops as you add more cross traffic where is that point that you can get it below the 10% of the area what percent utilization does that 2 gigabits per second represent 20% so I mean it is 10 gigabit 10 gigabit is there some knee in the curve where it becomes pretty impossible quickly for there were so we did experiments and right about 2 to 3 is where it got expensive that just means that so if it was 2 gigabit cross traffic then we needed to modulate the spacing by about a microsecond if it was 3 or 4 then you have to go to 4 microseconds and part of what we wanted to do was had a little bit error rate the other part we wanted to do was have it essentially invisible to a software entrance at 4 microseconds it is still pretty much invisible but much beyond that you can start to see it at a software entrance and so this shows that for example sending at a gigabit per second then with 1500 bytes packets then you have 3,000, 3,500 idles between every packet so if I had it at 1 gigabit then I would have exactly 3,562 idles between every packet now if I want to encode a 0 or 1 and I had 100 microsecond modulations then I would subtract about 128 idles to make a 0 or I would add about 128 idles to make a 1 ok so this is I could send a signal exactly like that and then for the receiver what I want to do is then measure what the interactive gap is and so this is a CDF and it shows that for the most part I was able to tell what it was a 0 or 1 whether the spacing was about 100 microseconds less than that average interpaculate or about 100 microseconds more so with Sonic I was able to recover more than 90% of the bits sent in fact this was over 90% now if I have some software ed-host this then shows that you know I wasn't able to accurately recover the inner packet gap in fact in order for a lot of software ed-hosts to operate at 10 gigabits per second there is the batch that goes on between the NIC so that's why most of the packets have fairly small inner packet spacing and packet delays and so there is no way essentially this is completely invisible to some software ed-host they can't see this at all ok so now as I just given on the time let me go ahead and head towards wrapping this up so one question is how would you actually prevent a covert channel there's a couple things that you could do one you could renormalize the inner packet spacing so that's probably the most effective way I don't know if you would ever be able to always detect if there's a covert channel but you can't suspect if there's one I don't have the graph on this laptop but you can actually see that we separated two peaks than a couple of peaks and that may have you suspicious that there may be some covert traffic may not be you could just have natural occurrence you have two modes or what not but you could renormalize using something like Sonic renormalize the spacing and make sure that everything has a uniform spacing ok so some of the research that I've done I actually funded by the DOD so I have a half million dollar of DERUP so that's the DOD University research instrumentation program which I got more than about 16 Sonic boards and two clusters that we call our own mini cloud so we're now able to do research using this platform we're now able to quite a few other people the other thing that we've done is make Sonic make Sonic available be a genie so you can actually if you want to now use UCSD, UC Davis or Rensi nodes you can actually from your own desktop do some of these similar experiments as well we have Sonic enabled exogenes as well so the key here then is that now all of a sudden we have a more accessible way to understand our networks we're able to understand the performance and actually enhance the performance and security of that you can now start to talk about a software defined network measurement platform so you can actually and we're looking into this integrating Sonic into a software defined network so that you can have access to control of these lower level layers some of the things we're looking at for example are available bandwidth estimation so it turns out that the algorithms so available bandwidth usually performs very terribly it doesn't give you a very accurate measure of the amount of available bandwidth but the key reason is that you're not able to control that inner packet spacing very well so if I can actually generate packets with the DAC spacing and I can measure how much they were perturbed then I can actually estimate that amount of available bandwidth much better so we've actually been able to demonstrate that with Sonic that a lot of those algorithms pastured for example do actually work quite well when you use Sonic so with that let me go ahead and conclude everything I've talked about so far is published already so between NSDI last year and this year there's other papers you can see all the work that I've done alright good so what is the incremental cost of Sonic versus doing do you think this is for experimentation I can see it's very useful right and you've learned a lot and all of this is very that's actually the question so for for research this is a great tool it is costly because I mean when you look when I showed you that design those were cores that we were using so for one channel we're using four to five cores or what not two channels we're using eight to ten cores which is quite expensive some of the things that we're looking at if you're to actually scale this up is integrating this functionality into line cards for example so that you can have more programmatic access to that layer one but you would not put this in a production network you know it's too expensive this is research though so I think that's the thing so I still don't understand I guess the answer to Greta's question which is that going back to your original issue it seems that the core problem was at the end host between the packet that the NIC was getting and the packets that the application was getting or not getting in this case and it seems to me that it could have been diagnosed by like looking at something like the packet counter to the NIC how many packets are the NIC going versus what's the application getting and then sort of trying to figure out where they're dropped actually the DSN paper describes that the packets were dropped between the NIC and the host so we look at the counter system we didn't know why they were being dropped so this actually the IMC paper with Dan Femian and whatnot that helped us to understand that we really didn't understand because we were sending at 100 megabits per second or a gigabit and here we had a gigabit infrastructure and 10 gigabit NIC and you know these pretty beefy servers that should have been able to handle that so we were a little bit surprised that we were dropping packets between that NIC and the host but the interesting thing is that the network is doing its job right the packet counter on NIC A is the same as the packet counter on NIC B unfortunately the packet counter on application A is not the same as the packet counter on application B I think Sonic is super awesome for a variety of reasons but it seems that it's not actually it didn't actually it told you some interesting things but the problem was actually in the N-host and probably could have been diagnosed and ultimately had to be diagnosed by looking at the N-host for behavior and not actually considering not actually needing the problem wasn't in the Sonic side it was in the N-host we couldn't agree with that I guess the key though is that we couldn't see what the NIC was seeing the interesting thing is obviously you have to look at the bits that are coming into the NIC but what you could have seen for example is look at the packet counter rate of NIC over time and probably you had to look at that and then you would see this burstiness and say oh wait we've got this strange bursty they're a fairly short burst over time the average rate was 180 so they're a very short burst and it actually so we tried but at the end of the day finding the root cause isn't actually using Sonic because that's not where the problem is the problem is actually in the N-host so you sort of need a different tool for doing that sure I guess so what we did is we tried to use that and motivate that we couldn't see what was going on today we wanted some ability without using a laser and a oscilloscope I think we actually had a mirror so we had the fiber coming into the N-host we had a mirror that then had the same signal go to the oscilloscope and went to the N-host and they're just they're kind of extravagant set up and you know the guy I was working with talking about the eye the diagram talking about what an ocean is and here he is excited about the eye of the diagram so anyways real quick by the way great work is this only true when you own the fiber you know it's because if you overlaid this over sonnet it kind of normalizes everything so I know this so in terms of I don't know what this is but in terms of the covert channels for example you do not have to own the entire network at all so if you go over sonnet which most of these networks do then they're going to change the impact of rabbit so that's a very good question NLR used to have a sonnet path between Chicago and Atlanta they then changed that from where I understand we started on the five round so we actually had four loops I showed you one long loop and we weren't very matched with the names but we had a short, medium and long for the names but anyways our medium loop went over that Chicago to Atlanta link which was sonnet and we actually in IMC paper do talk about some anomalies that we saw with that they changed they changed the underlying medium and they also no longer use the same CSR1 Cisco routers so once we got sonnet to work we weren't able to repeat the same exact experience in fact NLR has done that as well so you can't even use NLR we're talking with internet 2 to see if we can set some stuff up but the key is that at that time which was 2010 that environment doesn't exist because when you go into OC 1536 and things like that then you're subject to these things they're going to inject in a packet of rival times and then they're going to inject routes so you'll never be able to see them so this did work on NLR and there was a time that NLR had a sort of path and by the time we did this experiment they did any of these have been applied to using the wireless links using any applications that are beneficial so like I said a lot of this was motivated by wireless software-defined radio and just the control of the medium access player I don't know if this transfers or not because this is a lot of this does actually depend on the high rate actually so for the covert channel if you actually send it at a lower rate the channel was actually a little less effective in a sense that you couldn't send that 80 kilobits per second I'm talking about like a Wi-Fi environment we have wanted to connect the same I guess what do you try to do the same? I think one thing would be very useful in Wi-Fi would be from a measurement perspective to understand better contention right now the way you content for the channel it's hidden within the so that would be like very useful yeah so a lot of the cross-layer research is done on a wireless domain we try to do cross-layer research here as well I guess getting the reliable fingerprints over the wireless and the wireless access point would be quite difficult right because of all the interference and all the changes for your other building problem like a Cisco different types of Cisco those fingerprints are very what is the prominent or you could distinct but I wonder whether you will have those kinds of distinct fingerprints for wireless yeah I don't know I mean the nice thing about wireless and something like SORA is that you now have you know find great access to that actual analog signal so you perhaps could find different access points I don't know I think you've opened up a really interesting issue of how we can improve existing and get more interesting useful information out of them and my suspicion is probably you could just add a little bit of hardware to the NIC or the FI and get a lot of really interesting information to it but the question is a little bit you should add and we looked you know DAX and stuff like that and it just didn't give us the ability to answer the question that we want to answer and when we looked at it further we didn't have that programatability so it seems like there is kind of a cycle that now you have something like Sonic you could say oh there are some you could figure out what's useful and like well gosh if we had these other you know transistors are pretty cheap you know the optics are kind of what's expensive and you know although things like 10G base T I actually have a lot of transistors but that's over covered so I actually have a grant with NSF I have a submission with NSF now that would potentially look at you know closing that loop and the same thing applies to Wi-Fi getting more introspection into Wi-Fi but I was really excited by kind of the software defined sort of side of things like with software defined for a year we can actually use DSP to generate a wireless signal and I really like the idea that normally I can't change the encoding of 10Gb Ethernet but with something like Sonic it seems like maybe I could I could do new encodings I mean that would be a really, I mean as researchers that's very exciting. Yeah so you absolutely can change the encoding of the filer with something like Sonic our original name was actually SDNA which started to be confusing with SDNA so okay for the introspection we're gonna let people go and we hope everyone wants to stay more