 Well, thank you for being here. My name is Alan Samuels, and I'm an engineering fellow at Sandisk and I work in the Systems and Software Solutions division. And title of my talk is Apparent. We're having, there we go. Yeah, lawyer stuff. You can read that later. So what do I mean by infinite bandwidth? I was in the office the other day and was thinking about the evolution of storage network hardware and CPUs. I sat down with my counterpart on the hardware side and we plotted out the evolution of the speeds of networking and SSDs, storage since I worked for Sandisk, and CPUs. And here's that chart. The top line there, I think is in red. I apologize, I'm colorblind so you'll have to live with me if I get that wrong. Is the evolution of network bandwidth, and the middle chart is SSDs, and the chart on the bottom is sort of proxy for CPU speed, which is the amount of DRAM bandwidth you get in one CPU core. Now these numbers are prodded a little bit into the future. The good news is things in the industry are fairly predictable a couple of years out. We know what's happening with the DRAM standards. We kind of know what's happening with the CPU sockets, the networking, and the SSDs. So the crystal ball is pretty good for the rest of this decade, and of course we'll see what happens after that. You might argue if you look at the DRAM bandwidth that that curve is flattening out from the line that I draw. But the real takeaway from this is that if you look at the y-axis, you'll notice this is a log graph. And I redrew this for the log impaired. There we go. And the gap, which is the difference in the slope, is what I mean by infinite storage bandwidth. So essentially what's happening today is CPUs aren't getting a lot faster, networking and storage is. So I took this data, and there we go, restructured it. It's the same data, but in this case what I've done is I plot the number of SSDs you get per CPU socket. And except for the first few years of the millennium, you can see the trend for this is down, which is the same consequence as what we saw, which is that DRAM is not getting a lot faster SSDs and networking are. And it doesn't look too bad today because it looks like this is leveling out somewhere around, I don't know, I'm going to guess maybe 10 SSDs per CPU that doesn't sound so bad, except that this graph is using 100% of the bandwidth in the CPU socket, which means that you've turned your CPU into something that can't ever talk to memory, and that's not much of a CPU. Server class CPU typically cannot have its memory subsystem more than about 20% busy, or the CPU performance drops off dramatically. So I've redrawn this, there we go, with that number, that thought in mind, and what you can see here is that around the end of this decade, you're looking at a situation where for each CPU, you're only going to be able to put maybe one or two SSDs next to it. Which if you're an Intel shareholder is great, and if you're a SanDisk shareholder, it's not so great. I think that there are probably a number of other people besides the SanDisk shareholders that don't want to see that in their future. So that's kind of what I want to talk about, which is what does this trend mean? So I'll do a little bit of speculation. Basically, what's going to happen is we get closer to this sort of DRAM bandwidth limit. So the first thing we always do with technology when you get close to the end is you put a Band-Aid on it. And the Band-Aid for this problem is new form factors. If you have a one-use server with 12 storage slots in it, and you only can put one or two SSDs before it maxes out, that's just an awful lot of white space in there. And that's a fairly easy thing to take care of. You just change the form factor. And there are plenty of these out there today. I've got some versions of it here. It's a pretty good short-term solution, but as we'll see in a minute, it's just a short-term answer to the problem. So the real effects as you get closer and closer to the bottleneck is best looked at from an economic perspective. So if you think about the cost of your storage subsystem, it's really the cost of the media, the cost to access the media, which you can think of as like a controller, a hard drive, or an SSD controller, plus the cost to manage that, which is storage software and the CPUs to run it. And essentially what's happening is the shared nothing architecture that is the darling of the current thrust in systems really conflates the access and the management. In other words, you basically have to go through the management component in order to get to your storage. And as we've seen before, you're just making the problem worse relative to the DRAM bandwidth limit. So as long as you stay on that architecture, over time your costs are going to become your management costs, which is essentially CPU costs. And increasingly the cost of your storage system will be driven totally by the management software and the media will be free. And once again, my employers get very unhappy. So the real way to deal with a problem when you encounter it, first you put the band-aid on it, but then you have to really embrace the problem and really rethink it. So the question is what do we have to do to the storage management? And the answer is you have to pull the management out of that system and move that into what you would think of as an upper layer. And then go back and rethink your media access and what do you want? Well, you want it to be simple enough so it can be done by hardware because I can't have a CPU in there because then you're back in the same hole that you're in today. Or you will be soon. You want to make this storage accessible all the way across your network because storage that's not shared is just back in the same problem. The only way you can get at it is through the CPU. And storage that isn't shared isn't terribly useful. So you want just enough machinery in there to get some course level grain of sharing that's simple enough that you can be done in hardware. And if you think through this problem, you'll realize we've seen this before. It's a sand. OK. Or I could call it fabric-connected storage if I was in marketing. And really, it's just everything old is new again. But it's not your father's sand. If you rethink the problem with modern technology, you realize that the old style sand has several problems that need to be updated to be useful at current speeds. One thing, Fiber Channel itself has always been a problem. It's extremely brittle from an operational perspective. It's always been expensive. A lot of work has been done on NVME, and I'll come back to that in a minute. But SCSI initiators, SCSI was really conceived for hard drives. You really see that in terms of performance and functionality. And there really isn't any kind of robust sub-drive allocation. When your drives were a few hundred gigabytes, that wasn't really a problem. Dedicating a whole drive was fine. Drives today are multi-terabytes going to double-digit terabytes in the very near future. And you really don't want to be throwing those at your applications as the unit of allocation. So what is the next generation sand going to look like? Well, I think NVME over fabrics is really what the industry is working on to address that problem. The spec is out for review. It should be done next month. It has the requisite characteristics. It's simple enough. You can do it directly in hardware. It goes back to the client side and rethinks the interface. So the kernel interface on the other side is shrunk. And again, CPU is what is the dominant performance. The namespaces capability give you the drive allocations. The only problem is it's not really mature enough for enterprise deployment. But that's going to get fixed. And this is a talk about futures. So the real question becomes what is the network going to be? There's a plethora of candidates. The incumbent is Fiber Channel. We've already talked about that. And really what everybody wants is Ethernet, because what we've learned over the last 20-plus years of internet working is that Ethernet has the best economics over the long haul. But moving storage in a big way over Ethernet really not a solved problem today. Okay. For RDMA access, you really have two competing protocols today. And the NVMe protocol is built on top of RDMA. You have Rocky or iWarp. Okay. And what you're really is you're trading off whether you make the edge easy or the networking easy. And the reality is neither one of these is a solved problem ready for scale at scale deployment in a general purpose situation. There are, there's one large scale deployment of Rocky that I know about. And these people have gone in and very finely tuned their networking. They have custom microcode in their controllers, in their switches, and applications. It's not really appropriate for a general purpose environment. The iWarp solves all the networking issues except that there's almost no implementations of iWarp. And certainly the ones that are out there are not as cost effective as what you'd like to see. That could also be a way to solve this problem. And you won't see storage overnight over IP really deployed in a big way until this problem is solved. But let's assume that it does get solved and take a look at what are the kinds of systems that we can deploy on that. So the first thing that people always do is take the off the shelf stuff and let's see what we can do with a little bit of software and some components sort of found lying around. And we've seen some prototype implementations of NVMe over fabrics as sort of a direct sand replacement. People have been showing four to five million iOPS through Xeon class CPUs. Seems like 10 million iOPS isn't unreasonable to get to maybe in a couple of years. Still nowhere near the capabilities of the storage subsystem on the back end because basically all of your bandwidth is still being funneled through the CPU. The next generation of that is actually under development today. And what you're going to see is a movement of the software part of that data path. In other words, we're going to get the CPU out of the way by having the NIC actually crack the operations and submit them directly to the SSDs. So now your CPU isn't tied up basically handling the storage commands but it's still going to have a CPU to move the data because the architectures of the NICs and the SSD and the PCIe basically require you to still have the physical diagram from the previous slide. It's just that now the CPU is sort of an innocent bystander and he's not really doing anything for the most part. Now your iOPS are going to get a lot better because the implementation in the NIC will be tuned and dedicated, won't have to deal with a lot of issues like interrupts and things that just burn up CPU time. But the total bandwidth in the system isn't changed. Your CPU is freed up to do applications processing, which is great, okay, except you've stolen a bunch of bandwidth from it so the CPU is kind of drugged down. So the net of it is this is great but you're closer to the wall, you haven't gotten rid of the wall, okay. So to get rid of the wall, and this is where we get the real speculative imagination on my part, is I think what you're going to see is a new generation of SSDs and NICs that rethink the interface, maybe even including the flash chips themselves so that you can basically get rid of the DRAM buffering in the middle, because that's really where the problem is, and stream the data directly out of the network right into the flash itself, okay. And essentially you can think of this conceptually as an ether drive instead of SAS or PCIe on the front of your drive you've got Ethernet, there's no CPU to be found anywhere in this thing, okay. And where this really works well is with rack scale architecture, so I don't have time to go into that in too much detail, but basically what rack scale architecture is is the idea that we're going to take all the components there inside of a rack, all the CPUs, all the SSDs, all the networking, all the DRAMs, and what we're going to do is put like with like. So bunch all the storage together in one place, bunch all the CPUs together, bunch all the NICs, and then create a really fast fabric to interconnect them. And the real advantage of the rack scale, or some people call it a disaggregated architecture, is that you can now independently scale the amount of networking and compute and storage that you deploy in your rack without any kind of physical limitations on it at all. That's really where the current servers cause the problem for you is that your enclosures sort of fix the ratios, you know, you get only two sockets of server and only 12 drive bays. And well gee, what if I need nine and three? Well, we don't make that, okay. So instead of having a fairly coarse grain setting on the knobs, which is what you have today, with a rack scale architecture you have a much finer grain setting on that, and you can do things like statistically multiplex the consumption of the storage across a much bigger client pool and you find that it's much more economically efficient. And you've actually, there are a couple of implementations out there, Intel's been talking a lot about that, and this is really where you rip the Band-Aid off. So, you know, this talk is really targeted, my message is to the open source developers, the open stack people, you really want to start thinking about this as a model of how we manage these things. There really aren't a lot of good tools for managing essentially sand at a cluster level or a data center level. That part of it is really kind of punted today. There aren't any really good solutions to it. We've got lots of upper level software that'll deploy on that just fine, but we really don't have the stuff at the bottom. And you know, that's really a call to action to do that. And just to sum it all up, because I'm almost out of time, what does it all mean? It really means new form factors are in everybody's future. You can pick which camp you like to be in, but those form factors that we've known and loved for years, they're really getting near the end of their lifetime. The things have just changed fundamentally. You want to get your storage out from behind a CPU. Mostly because you don't want to be paying, you know, as long as it's behind a CPU, CPU is going to dominate the cost. And I believe that rack storage architecture is essentially what we're going to see at the physical level to get to that. And the software defined storage will deploy on top of that layer. That's really all I have. Thank you very much for the time.