 Welcome to another edition of RCE. I'm your host Brock Palin and again I have Jeff Squire some Cisco systems and the open MPI project Jeff. I hope you had a good holiday weekend Hey, Brock. Yep sure did and certainly can tell that your microphone is on the fritz We're gonna have to apologize to our listeners in advance today that you you're using your laptop mic instead of your regular one Yes, I'm having strange USB issues where I cannot get my good microphone to work. So I'm on my backup We have today Peter Honeyman and Bruce Fields both from the University of Michigan's Center for Information Technology integration better known as city. I should point out Michigan is my alma mater So I've actually kind of been familiar with city for a while. So Peter Bruce. Welcome to the show Yeah, hi. Thanks a lot How about you guys introduce yourself and say your names people to connect and give a little background Yeah, okay. Yeah, I'm Peter Honeyman I'm a research professor of computer science at the University of Michigan and I've been here for a long long time measured in decades All of the time I've spent at Michigan. I've been at city as more or less the staff scientist And At this point I kind of run the place So that's my gig And I'm Bruce Fields and I'm an assistant research scientist at city. I came to Michigan to study mathematics but after getting a getting a doctorate in 2000 came to city and since then I've been working on a Linux and NFS and I maintain the Linux kernels NFS server Okay, so we're here to talk about the NFS v4 effort specifically but for anybody who might not know Can you give us a quick description of? What NFS is and then get on to What NFS v4 brings verse v3? Yeah, sure So NFS stands for network file system Most people are familiar with NFS. It has been a standard component of all Unix and Unix like operating systems Since the early 80s It was developed by Sun Microsystems. Well, it was kind of developed by Bill Joy when he was at when he was still a grad student at Berkeley, but it was really matured at at Sun and became Universal among Unix Unix based operating systems because it was An open system in the sense that Sun published the protocol for NFS and allowed All of the other vendors to do their own implementations So NFS really was one of the very first open systems Around in fact Sun went beyond just publishing the the specs for the protocol, but they made Open source implementations available. Well, they were nearly open sourced. They were quite freely available To universities NFS is a protocol More than more than anything else I mean people think of it as a file system, but really it's a protocol a client server based protocol that allows Servers to distribute files to to multiple clients within, you know, a strict Unix like namespace it Supports the POSIX interface and looks in every way to a client like, you know, a regular local file system Except when the network screws up. Anyway NFS was designed in a time When you know local networks prevailed when networks had lower bandwidth then then it's available today and latency Rarely became much of an issue because everything was was done on the local level It was also designed in an environment. It was really designed before that, you know, the explosion of the internet Which I guess dates to like 1990 or the early 90s So security was was less of an issue for for for NFS. It started out as version two No one really knows that what happened to version one. I mean, maybe it's like, you know, Oracle It wanted to sound mature. So they started at version two There was a some some small changes were made to it along the way in the 90s to improve its ability to cash files and to improve performance but NFS seemed to be stuck at version three for a long long time and there was some some some problems with version three That made it made it difficult to extend it to scale it out to an internet scale with the scale of the number of users on the internet as well as the the the latencies That that are common on the internet It was mostly Implemented as a UDP based protocol and that's just not going to cut it on internet scale But because it was a protocol that belonged to Sun Microsystems It was difficult to to push it forward Sun had the incentive but couldn't really develop the consensus Other folks that were involved in NFS development had the consensus, but they didn't really have the Incentive to improve what was you know, essentially something that was owned lock stock and bail by their competitor so Sun actually in a very mature way found a way out of this problem by dedicating the protocol to the internet engineering task force, which is where internet protocols are born and the IETF formed a working group to Develop the next version of NFS that that's where NFS v4 came from by about 1999 or 2000 that It was pretty well understood what NFS v4 would look like and how it would be different from From from earlier versions and that's kind of when city got involved when some of the some of the vendors Sun Microsystems network appliance were looking to promote this the new standard or that you know the incipient standard by commissioning city to build the The Linux based open-source reference implementation of NFS v4 In that way there could be multiple independent Implementations from from a common specification, which if shown to interoperate would satisfy the IETF's requirements for for for new standards So that's kind of the early history v3 and v4 Okay, so what are some of the advantages of v4 over v3? Sounds like v3 actually while I had some performance improvements over v2 really didn't have anything to address the scale like our focus here on RCE tends to be high-performance computing people running large compute clusters or clouds or similar kinds of Applications maybe with thousands of clients NFS has never really been something that's been described to go on thousands of clients. Is this something being addressed with v4? Yeah, that's some of the some of the later work that that we're doing Let me tell you a little bit about v4 and and how it is different from v3 First of all some of the problems of v3 were addressed With you know essentially with a hammer security in in in NFS v4 is mandatory and v4 is required to support Kerberos based Security mechanism as well as a public key based security mechanism This requires some changes To the remote procedure call mechanism that's used by NFS. There's a an internet standard called GSS generic security services And by melding The RPC and GSS you get this RPC sec GSS Which is the the remote procedure called package with the GSS? security mechanism Layed in as as one of the you know as one of the security mechanisms for the RPC now I Kind of think that actually that could be done with v3. I mean that could have been done with 3-3. Don't you think Bruce? No, it has been I mean yeah v3 supports all v3 implementations modern ones at this point support RPC sec GSS as well but it was partly the v4 Implementation effort that actually made that happen Yeah, so So it so so it gave it gave the right push At that point. I mean NFS security prior to RPC sec GSS was pretty much you just kind of throw your user ID on the wire, you know Oh, I'm 101 and the server believes you so it was a you know a trivial exercise for undergraduates Throughout the in the late 80s and throughout the 90s to build little shells that could you know wander around through NFS space and pretty much You know lie about their identity and and access Files file servers Anywhere okay, so that was kind of it, you know security was was really kind of a joke That's now fixed the that mechanism that kind of security mechanism is Optional now in in NFS v4, but it's required that it support the stronger security mechanisms. That's one major change a second major change is UDP is pretty much prohibited in In NFS v4 mean the language of the spec doesn't come out and say it what it says is that the transport layer is required to support, you know some some some features that More or less say yeah, it's got to run on TCP so compliant NFS v4 implementation is must run on TCP and that solves a lot of the the long-haul problems the wide area problems that that NFS v3 had which you know based on UDP It had all the retransmission issues that that arise with with with UDP And the you know fragmentation and and you know all the other ugliness More or less the reason why we abandoned UDP for for most purposes Let me think what else does people have well another historical okay? No, I'm sorry as a networking guy. I just I'm a little curious what kind of features are required That require TCP but sides, you know Built-in retransmits and things for like that for long haul. What oh my obvious questions, you know jump to like all right Well, UDP was great for scalability But you lose at least a little bit of that with TCP because it consumes a bit more resources and things like that Could you talk about that a little bit? Yeah, I mean, I don't think that UDP was great for scalability Unless you unless what you mean is It didn't require any cycles Which I don't think of as being a scalability issue the You know, I can't quote you the actual language of RFC 3530, but what is it? It requires control and The word reliable is in there. Yeah, so yeah, so Yeah, reliable flow controlled. I don't think it requires sequencing But you know if you want if you need reliable flow controlled you And you want to do that with UDP then you pretty much have to build up the entire TCP of infrastructure around UDP Which is more or less what was done in v3 you know With the in the in the RPC level but that's a messy place to do it and it It leaves you with an implementation that kind of doesn't advance with TCP, you know And TCP advances everything that runs on TCP gets the advantage of it Anyway Yes, that's that's answering your question about UDP And I think Bruce was gonna say a little bit more about some of the advantages of v4 some of the differences Well as historical annoyance with v3 has to do with the way it developed Which was when when a new feature need was needed It was often added by adding another protocol on the side to implement that feature So for example when people wanted File locking What what they did was to create another a separate file locking protocol So there's a separate protocol for file locking there's also a separate protocol for ACLs There's another one that's used by the initial mount program to get the root file handle And this becomes somewhat of a nightmare again for the especially for the long haul If you need to traverse any firewalls So one change in v4 is that all of the functionality is put into one single protocol it was also a real problem because While while vendors paid very close attention to NFS interoperability lock D Interoperability was less, you know less assured so that Became the case that if you if you thought you might be working in in a multi-vendor environment Then you couldn't rely on the existence of a locking protocol, you know that that that worked across all vendors So yeah, so locking is now built-in. So that tells you something real important right there about v4 as opposed to v3 v3 is is known as a stateless protocol in the sense that the server doesn't maintain State on behalf of the client every request has all of the information in it That's that's needed to service the request v4 is extremely stateful the server keeps a lot of state on behalf of each client And and you know that I mean that's a mixed bag there that that affects the reboot and recovery mechanisms for for v4, but it also You know it frees the the the client server protocol from from You know having to have on the one hand this kind of arid environment in which you can't really do much. You can't have locks You you can't Negotiate Special features of the connection for a specific client At the same time the open call in in NFS v4 I mean NFS v3 didn't even have an open call v4 has this open call that is really pretty expensive to run and it requires a lot of negotiation between the client and server To to pick out all of the the options that that they want to you know to control that could their connection but AFS proved long ago that having a stateful Server Can have real performance benefits in terms of scalability now when I talk about scalability. I'm talking about You know how many clients can a server can hook up to a server without washing that server out or Can we scale to long haul or to you know to high speed or low speed scale up scale down Having both you know a Reliable and robust transport and a stateful server Allows You know it's kind of a funny thing because what AFS proved is that With with with with the server maintaining state on behalf of the client The clients could be expected to do a lot more of the work And that's why AFS has better scaling properties than NFS v3 You know back in the day when AFS was was being developed You could you could go up to maybe a couple of dozen clients per server with NFS and AFS was shooting for a thousand clients per server They did that with the you know the strong caching On on the clients, but you can't do client caching You can't do you know consistent client caching without the server getting involved and you know keeping track of what you know What's in each client cache? NFS Still doesn't really have That kind of you know robust and reliable Mechanism for client caching but one thing that v4 does have is a mechanism called delegation Which kind of came out of the windows world What it what it comes down to is if if a client is the only client with an interest in a file Then the server Can delegate all of the operations on that file to that client mostly what we're talking about is is stateful operations like locking So if I'm a client and you know, I I start accessing a file And the server Delegates the file to me grant grants me a delegation on the file then I can Run locks and and stuff like that on the file and it becomes local operations I use my local lock manager, and I don't have to talk to the to the servers lock manager at all now Obviously if two clients want to want to use the same file if there is in fact concurrent sharing Then the delegations have to be revoked which is very similar to an AFS callback operation and then subsequent locking takes place on the server with the servers lock manager and things get a little bit slower as They have to when you you know when you have to negotiate moderate this kind of concurrent sharing So that's another feature that that That's in there. It also has as Bruce mentioned ACLs built in and the ACLs These are access control lists You know v3 was very much of a Unix like File system so it it the access model was more or less mode bits v4 has You know these access control list that that really kind of came out of the windows world again because people anticipated that v4 could extend to Windows clients And the easiest way to do that was to kind of pick the union of features Among Unix and Windows clients so so the ACL model is is something that's more familiar maybe to to Windows folks in the sense that it is not the POSIX. It's not POSIX ACLs Well, I mean it's my personal opinion that POSIX ACLs are totally broken because they strive so hard to emulate Unix mode bits It's my personal opinion that Unix mode bits are totally broken So it does, you know, so it has it has These ACLs built in it also has other windowsy kinds of things like you know windows file system when you when a when an application opens a file it can open open it and deny access To other opens at the at the instant of open. It's kind of this atomic, you know open deny operation And v4 has a mechanism for for that as well, which is really difficult to support on Unix systems but Makes it possible to have multi-protocol servers that that could serve up both, you know, SMB and and NFS to Windows clients because it has the same you know has a common semantics for for sharing See is there other stuff? Yeah, there's other little things like it has a compound RPC The the remote procedure called mechanism in v3 is Kind of a command at a time you issue a command you get a response And it's very you know very synchronous like that in v4 the The RPC there really only is there's really only two RPCs the null RPC And then the compound and the compound contains a bunch of other remote procedure calls So you can you know load up in a single RPC an open and a Read of the first 4k in a single request And I'm not really sure if the It's the designers of the protocol thought that they would get a Performance benefit out of it. It turns out that you kind of don't And I think what it I mean my opinion is it cleans up the protocol a lot in in v3 Some of the some of the responses that you got back from the server would include extraneous information like stat information about a file that you That you modified Which you know strictly speaking is not really part of the expected response to an RPC But it was something that you would you know the designers of the v3 RPC the v2 RPC They knew you know quote new that you would always want to have this information anyway in v4 you can now sort of isolate the A request in response and then if you think that a request like a like a right request is going to require the return return of attributes Then you just throw the the get attribute request Into the compound RPC along with the right and you know it cleans up the semantics makes it easier to Reason about about what the protocols doing? All right, so you threw a whole bunch of features at us here I wonder if I could go back and touch on it at least one of them here you mentioned or Bruce mentioned earlier actually that In v3 every time you want to do something new you you made a new protocol for it in v4 You've kind of slurped all these protocols together into one So there's one protocol and one of the benefits you mentioned was you know vendor interoperability because it's all one thing now Is a new protocol itself Extensible so that you can you know do the Unanticipated things in the future without doing you know the v3 syn of creating yet another protocol on top or or are you gonna have to go that route? Yeah, it's accessible in new ways for two reasons one is as Peter mentioned it has this compound operation so the operations that you send to the To the server are built up out of lots of little operations and It also added a minor versioning Capability so you can While saying still staying within NFS before you can add additional capabilities The and the most common way to do that is by adding additional operations that can be strung together into these compounds and The latest effort there has been NFS v4 point one Which has an RFC that's just coming out now Which adds a bunch of new operations that support new features Yeah, this extensibility mechanism is actually really important When we were talking before you know before we went on the air There was a question, you know, what would end up what would we expect an NFS v5? The answer is there there should be no NFS v5. We are now Testing interoperability and completing our implementations of v4 point one now that the IETF has endorsed that the The specification for it and within the IETF we're talking about what will we want in NFS v4 point two So this minor minor versioning extension gives us a way to to move forward without having these, you know, sort of cataclysmic change At every at every every time someone has a new idea. I bet you're wondering what's in v4 point one so We've been talking a lot about Security and a lot of cleanups and fixes from v3 What specifically do you expect v4 to be able to handle a very large number of clients per server? Like you mentioned the AFS could do a lot better than NFS v3 could What kinds of things have been done for like in the high performance computing space where we may have 10,000 clients actively using the server at one time that exists in v4 that will enable that verse v3 Yeah, a lot of the motivation for the development of v4 has come out of the high performance computing community especially the the national labs and Of course what what what we learned having Designed and implemented v4 was that it wasn't enough There are things that are done in in large clusters that Are so unusual and irregular that special special mechanisms had to be invented for them things like you know 10,000 10,000 nodes opening the same file at the same time or 10,000 nodes creating a file in the same you know creating 10,000 different files, but in the same directory those kind of metadata operations Were found to be you know really slow and in fact Affecting the performance of large computations even though those kinds of operations are relatively rare I Mean they are but but those guys run on on pretty pretty bizarre machines like the Roadrunner guys that have a you know A mean time to failure of less than 24 hours, so they have to checkpoint for it fairly frequently and a checkpoint operation Involves this kind of you know massive metadata massive scale Metadata kinds of operations We're actually still working on that on those kinds of problems and The real problems that we're finding there are with the POSIX API And we may we you know we may figure out how to how to solve those kinds of problems but one of the things that we have figured out how to solve is something that has been endemic to the to NFS and indeed to any Conventional client server protocol from the start of time that is to say six to eighties And that is the single server bottleneck To the extent that a whole bunch of clients want to get access to a single file That file lives in an in a namespace the namespace dictates what server That the clients all have to go to and so everyone's got it, you know kind of get lined up behind the same server There are ways to have cluster file systems that that spread a namespace across a Bunch of servers and now there are products out there that that that do this fairly well IBM's got GPFS Poly serve which is now a part of Hewlett Packard had a pretty cool cluster file system What what those things have in common is behind the scenes they have a distributed lock manager of some sort that is Negotiating, you know the access from the various cluster nodes when when sharing takes place But presumably those guys are getting it right. They're doing a good job of it and building high-performance systems Then the question is well, how can we you know, how can we extend this into the into the NFS space so that? you know Servers can take advantage of the fact that that there are multiple cluster nodes Serving up the same namespace so that we can distribute the load across them This was actually pretty easy to do an NFS v3 because it was stateless So that all of the action took place in the background in the distributed lock manager But now that NFS v4 is a stateful protocol some Some negotiation or some some arrangement has to be made for the For the the cluster node heads that are acting as servers to you know to kind of keep things straight the obvious way to do that is to push the Consensus management and the concurrency control back into the Into the back end into the into the lock manager of the cluster file system so we found ways in in In our implementation of the protocol to support that and so now now we support cluster file systems fairly well There's another issue that arises and that is a Client trying to get access to to files that You know exceed the The servers bandwidth Ability so normally what you do here is is you stripe a file across multiple servers and You know get access to it that way now you can do this manually and In fact most operating systems have a have a way of doing that but That becomes you know kind of a an implementation that is specific to a High-performance file system a proprietary high-performance file system that will have its specific mechanisms for for arranging for striping or for you know Object storage access or or or sand access or whatever and then the vendor is is stuck with this problem of supporting You know I need now I need to get the striping mechanism running on Linux and on net BSD and on windows You know all over the place The HPC guys at the labs The folks at the labs Saw an opportunity to build support to extend the NFS protocol to build support in for this kind of Access this extended kind of access Striping and and similar things Into the protocol so that's one of the things that NFS B 4.1 has in it something called PNFS or parallel NFS which I don't know. I'm talking too much. I'm gonna let Bruce describe PNFS The idea is very simple. It just gives the Server a way to all to advertise alternate channels to a file's data So at the time that a client opens a file it can ask the server. Okay, where's the data for this file actually located? The server hands something back to the client called a layout Which tells the client? Where to go and the client goes there And there are different alternate alternatives for what there could be the client could talk directly to Disk if it happened to be attached to the same sand that the server was Client could Client could be sent to another NFS server and could just do regular NFS reads and writes to that server and there's also support for object-based protocol and the The benefit of that is that it gives you a way to offload the IO bandwidth so that you can get higher aggregate bandwidth to the to the storage Yeah, or the layout could specify, you know the the stride of the striping and the location of the striped servers It's actually a very generic and general kind of protocol It involved a small extension to NFS just involving the ability to pass layouts back and forth between the client server But you know, we try to use our imagination here at city and think of you know, like a associative search of images and stuff like that You know using layouts for that kind of crazy stuff but mostly what we're interested in is is scalability and performance using pnfs to support High, you know very high speed access to To files striped across multiple servers So what is city's involvement specifically you mentioned that you were thinking of these Like real-world kind of cases of how pnfs would be used. What does city actually work on with NFS before? Well at the moment we have two or three major sponsored projects We are part of a Department of Energy sponsored projects out of the the PsyDAC program that scientific PsyDAC stands for Dance for funding source Yeah, well, they they they've it's comes out of the DOE office of science The the pedascale data storage Institute is a group of five national labs and three universities that is Trying to push forward in in this the space in which in which DOE office of science does it work? which is you know a lot of accelerator work and and simulations and You know, they're mostly what they said what what that office supports in computing is is You know computation-oriented Projects we're on the storage side. We're like kind of unusual in that way. Anyway, so that that's one of the major projects another another is sponsored sponsorship from EMC to develop the the Layout the the NFS the pnfs client for for block layout that is to say The NFS metadata server will give a layout that gives the the the LUN, you know, the the LUN vector For a for a file Or a file system and then The client makes access to the sand Using iSCSI, but really kind of a local local this kind of stuff that parallel parallelizes pretty nicely And the the third major project that we're working on is is a native Windows NFS v4.1 client Most of the work that's been done to date in NFS v4 has been on Unix Linux You know Unix like systems. There is one NFS v4 project from a company called hummingbird, but Microsoft was interested in working with us to support pnfs on the Windows client So we're doing something that's kind of unusual at city. We're hacking in the Windows kernel to to develop a High performance, you know, that is to say, you know fast and slick client implementation for NFS v4.1 Right on top of Windows So, you know, we do sponsored research. We're a university city is a little unusual in that We have full-time staff working here Most most university research labs are you know have grad students and You know one one one faculty running a little empire of grad students here We have me one faculty running a little empire of full-time staff, you know Bruce is like the assistant director of city We have a couple of Folks that you know, they they're glued to the screen all day and their programmers their excellent programmers City really is like the technological jewel of the University of Michigan and has been for a long time We also have graduate students doing doctoral research and support undergraduates With our research sponsorships kind of as you know internships and to help develop their Their education to give them opportunities to work with professionals on research projects But as cities, you know, it's a university research lab No obvious bias in your in your qualifications there that you give it all either I think Well, you know, we kind of do like working here We get the you know the psychic rewards of being academics Let's let me ask you this you mentioned a whole pile of other organizations who are involved in NFS V4 Mostly with whom you're collaborating. Are there any other notable ones like you didn't actually mention son in there at all? I'm sorry. I meant Oracle Are they still involved in the NFS development and or you know who owns does anybody own the intellectual property or is this IETF owned or how does all that stuff kind of work? I'll answer the funding question and I'll let Bruce talk about the IP stuff The early funding for cities NFS before development came primarily from Sun Microsystems and Network Appliance. I mean Oracle and NetApp There their support You know, it was absolutely critical and it was millions of dollars of support because it was a really big job their support tailed off over the years as we As the implementation became mature and we started to kind of fill in the corners that that that needed to be filled in things like the You know, like the windows stuff and that the the block-based PNFS client So we're so we're working on in areas That are not central to protocol, but that happens son We're absolutely instrumental in the early days IBM also provided a Great deal of support you know sponsored research support In our work in developing the cluster that support for cluster file systems NetApp continues to support city and I'm imagining that there will be other other folks That that provide financial support We we got support from polyserve Anyone else? I don't know a whole bunch of you know What's what's cool about it is is that there are a lot of vendors involved in the development of v4 both in the Specification, but it also in their in their own implementations that they hope to make a lot of money You know selling products City has been kind of this neutral meeting ground For for the vendors where they can you know, they can all come and work with us and know that that You know, we don't have a bias in favor or against any one of them and You know, we are cities a vendor neutral place That supports this kind of multi-vendor activity Which is important because you know what we're trying to provide is Interoperability for a protocol so that the vendors can all go out and make the best Possible implementation and make the most possible money off of it. So now let me ask Bruce to talk about the IP stuff IP stuff The the protocols are all you know entirely neutral they're all controlled by the ITF The work that we do is is released under a Sort of BSD like open source license. That's very liberal In practice the work that's incorporated into the Linux kernel by necessity ends up being a GPL more or less So everything that city does is is open source the other implementations of course aren't all although Solaris at least should mostly be I don't know does that answer the question? Yeah, yeah, would you say that any one of these like is the work that you do that ends up going into Linux Would you consider that say, you know a reference implementation like do other people derive? You know vendors derive from that or do they write their own either from scratch or at least parts of it from scratch or? You know, how do how do they typically do that? right I expect that they write their code from scratch But we are a reference implementation in that in the sense that we provide an example example of an implementation of the protocol and We're often one of the First testing targets are the first testing target And we also provide some other testing tools We have a you know complete set of regression tests and pie and fest that's widely used by server Yeah, we wrote a complete implementation of NFS v4 in Python Which oh wow yeah, and it's it's it's fabulous for for testing because you can build scripts that do the sort of thing that you know it'd be difficult to To make exactly the you know some some specific behavior occur In a file system by manipulating the file system externally But because we have all of the components of NFS v4 In written in Python, you can build these bizarro scripts that test wild error conditions or Performance performance kinds of things Interoperability is is you know is key when you're developing a protocol like this. So yeah, we have we The intent all along was for our our work to be a reference implementation That vendors would use to look over our shoulder But also for testing correctness of their own implementation In this to the extent that they interoperate with us then their implementation is correct That's kind of arrogant really if if they're if the interoperability If interoperability fails Then you know we both sit down and go over the spec Line by line and go over our code line by line figure out, you know, who's who is broken? Sure, it's kind of the transitivity of of interoperability, right? Yeah. Yeah, but we have like three We've been having three meetings per year for years now for over five years bake-a-thons and and the annual connect-a-thon which our interoperability bake-off kinds of Testing We have one coming up in a few weeks, right first week in October will all be in Austin You know a bunch of geeks sitting there with their computers pointing them at each other and looking for problems. Yeah It's really pretty funny. It's a long table with a lot of geeks Outstanding so so what's the what's kind of the uptake of NFS v4? I mean, what's what's the status our vendor shipping products? You know, can I go out and get an NFS v4 clients and servers and put them on my cluster today? Or you know, where where are we in the adoption process? Yeah, I'm guessing that whatever it is that you're running has v4 today And what do you have? I've got red hat version 4 so probably not Yeah modern versions of modern be the keyword there someday I'll get around updating the red hat 5 but I'm I'm several thousand miles away from my cluster So I don't have an easy luxury to do that yeah v4 First appeared in 2.6 and it has matured with the with the numerous releases of 2.6 and with bug fixes and performance enhancements But anyone running Linux anyone running a you know recent version of Linux has v4 now whether they're using v4 is another question because The client-servant negotiate which version they're gonna of NFS are gonna use v2 v3 v4 and I'm not I'm not sure that a lot of people are using v4 at this point. I Know that you know a lot of Sun customers are using it but You know honestly the you know the driving force for v4 in the early days was expected to be security and and long-haul performance But those turn out not to be huge market drivers You know as usual nobody wants to pay for security So nobody and even though they have v4 They don't want to pay for the administrative overhead of having to learn You know how to set up your mounts and your exports for v4 Everything seems to be working okay for v3. So let's just stick with that and the long-haul stuff Turns out not to have Been you know big enough market to to to really drive A cut over we're thinking that PNFS in 4.1 will be the thing that pushes HPC and some other enterprise scale folks You know the petrochemical industry the financial industries Into demanding the features that that that that 4.1 offers Features offer And So, you know, I guess we're kind of looking at the market the v4 market really exploding like in 2011 time frame, you know at the moment 4.1 is still being You know that the protocol is is specked out the RFC will be out any day. It was approved by the Internet Activities board One of the one of these, you know, IETF guru kind of things They approved the the final specification of it last December. It's been in the hands of the RFC editors To dot all the eyes and to get all the witches and that's right So we're awaiting the actual release of the RFC, but the the spec that was approved is the specification of the protocol And we're spending the the current candle calendar year Completing all of the implementations Which they really are pretty much complete But then we have to you know, there's this process of getting things into the Linux kernel that involves a Consent consensus management among developers and maintainers now Bruce is one of the co-maintenors of the NFS server The the client maintainer also lives in Ann Arbor and we work very closely with him John Mikkelbust But those guys are you know And it's funny. I mean Bruce is sitting right next to me and you know, I I pay his salary But in a way he works for Linus, you know He is addressing some concerns that You know that the that the folks that are sponsoring the research here the R&D here They're not as all that interested in getting everything right. They really want to get things done Whereas Bruce has a vested interest in getting things right. I stop speaking for you Well, there's a process and there's been a long prototyping process going on in 4.1 And people have demonstrated that the 4.1 can be made to work between, you know, two independently implemented Clients and servers at this point and Have demonstrated that they can get some of the performance characteristics out of it that are driving it But there's some work that needs to be done to go from those prototypes to something that Handles all the kernel the all the sort of corner cases and Walt eat your data when something goes a little bit wrong and Has code that's just easy to understand and that can be maintained in the future by the kernel community Yeah, I mean tainers care about maintenance. I don't know what the deal is with that so there's a process of negotiation there and you know the people who are working on the prototype trying to try and make it a you know more correct and cleaner and then submitting it to to the kernel maintainers and And it's a show to the kernel community and it's a big change so that shoveling that much change into the kernel It has to be refactored in ways that it can be you know understood and digested by the developers that have a stake in all of this and at some point consensus happens and the maintainers Allow things to you know to slip through the sieve so our our hope is that And here I'm speaking for me not for Bruce is that we can get PNFS into the Linux kernel in the current calendar year and then spend 2010 Soaking it and working on performance issues and doing instrumentation and scalability testing So that by the end of 2010 it really is ready for for enterprise. So the enterprise on the of the of the you know our hell kind of kind of customer we expect that There will be early adopters in the HPC community Especially at the at the national labs that start using PNFS that are well they're using PNFS now in some instances, but You know for those guys It's really important that they're the kernel that they get from kernel org have everything they they need and they did not do special kinds of patching to build the kernels that that they need so You know, so that's the you know, that's the kind of the the road map That that what that we're looking at and some of that you know some of the constraints and the difficulties In the this is a social process of developing Software in this multi-vendor kind of environment Okay guys, so is there Like the tools for anything like PNFS if someone did want to try it Things that are not shipped with the links kernel right now like an NFS v4 website anything like that where are those located? Okay, yeah, we do have software repositories with our experimental code we have a mailing list which people are welcome to ask questions on and the Best thing to do will just be to Google for PNFS and Linux and that will ultimately get you to a git tree and Then you just you know kind of whatever it is that you do to a git tree Get it you get it And then you you know and you build it and yeah, I mean it's it is ready and waiting for for people interested in In using it in in tests kind of scenarios in fact Brock and I were talking this morning about you know What could we do between city and and and his lab? to you know, maybe play around with with some of the you know scalability characteristics In that are that are developing in storage protocols As an adjunct to the kind of stuff that they do, you know kind of on a day-by-day basis So that's that's kind of where it's at now is is interested People, you know geeks with with some time on their hands and kind of you know a little bit of craziness Are playing around with it and and benefiting from it? But it won't be until it's You know integrated in into the kernel or kernel into the mainline kernel That you know ordinary people Ordinary geeks Well, we'll start getting their hands on it, but early early adopters now Are welcome and and a bunch of people are playing with it At you're right at this moment Okay, thanks a lot Peter and Bruce for your time talking about NFS before The 4.1 standard is hopefully something we see soon And you can get this show at www.rce-cast.com There's an RSS feed and iTunes link or you can always just download the MP3s or at least every other Thanks Thanks a lot. Thanks guys. Thank you