 I'm Charles Forsythe. I've got a company called, one of the founding directors of a company called Betanova based in York, and we were doing a lot of work with very early distributed systems. Bell Labs came out with Plan 9. That was under a restricted licence, but I got some friends interested and we formed the company, and later, Bell Labs came out with another offering system called Inferno. In a roundabout sort of way we ended up acquiring that from Lucent and looking after it for a good many years. Lastly, open source being what it is, I've been doing contracting for partly because for tax reasons and because it gives me a bit more flexibility from doing things within the company. Nevertheless, the company is still there and both Plan 9 and Inferno are still there and being used. The structure is something like this, so I'm going to give a brief overview of the systems, very brief information in the background because I might otherwise run out of time. I'm then going to focus on the aspect of the system that encourages us still to use it despite practical problems that get in the way such as having to write device drivers, having to interface with things, having to make do without a whole host of ready made software that you get free of charge with all these other systems. Nevertheless, we still persist in doing things with it and others do too. Not very many, but enough that I've been able to work pretty much full time for well over a decade doing things with just these systems for the most part and occasionally dipping for commercial reasons into other systems and then retreating in some pain. Because when I was much younger I did a lot of work with Unix when the alternative was a big complicated bloated horrendous system and used Unix because it was lean and interesting and mean and small and easily understandable that you could change any bits of it and it was great. And latterly, some of these others are not the ones that have been described tonight because obviously people have gone to the effort of making them lean enough to work while still not losing the advantages of having access to a lot of software. Erlang for example would be a good example of something you might want to lose. But in the system it's just not currently available. There is an aspect of the system which has encouraged us to continue to use it and to build applications and appliances and other things using it. So that's what we're going to talk about and I'll go back if there's a bit of time left and perhaps mention a few practical things to do with tool chains and other stuff. But I suspect I'll probably run out of time. Now I mentioned I'd be working in this area for using these systems for a couple of decades now. I started using it at the University of York when I was still working there. And one of the side effects is that I spent so much time in that that as these other systems have evolved and changed and added complexity upon complexity. And there's a whole host of things that I just never use in the system I hardly understand it. And I go to conferences, the UK UUG conference in March last year and I feel as though I've arrived from another planet or perhaps an alternative universe. And so now you're going to come with me into this alternative universe where history took a different turning and things look sort of similar but there are differences. And that's Pete's world from Doctor Who. So Plan 9 and Inferno, they're both operating systems. They have different characteristics. The interesting thing about both of them is that they were designed from the start to be part, to build and be part of a distributed system. And the idea was to use a collection of specialized services to build one larger system. Since we're in the Wilkes room, I should point out that much of this was inspired by the Bell Labs people looking at systems like the Cambridge Distributed System. And thinking that this would be a good way of building a Unix-like system. But rethinking a lot of the things that had changed since Unix was first done in the late 60s and early 70s. I mean there's a big difference in the sort of capability, the most obvious one being networking. And the system is intended to include network security from an early stage. They wanted to be highly portable. They had enough of the FDF hell that had arrived in Unix and so on. So in the alternative universe none of that stuff exists. And instead the labs people have gone on and they've developed something and it actually took off and people are actually using it. So I don't have to justify the absence of this one thing or another. The idea was to keep the structure quite simple. One reason was to be able to use it in experiments with different kinds of hardware and they wanted some systems offered to drive it. So the idea is to keep it malleable and simple. It's more than just a language. I mean it's particularly through Inferno. I think a pair it with say the Java approach where the idea is that the language is the means for doing everything on this distributed system using an abstract machine to have it running on it. And all the nodes in the network. Some of those ideas appear here. But there is this idea of there's still an operating system level in that environment. So the primary things in the system that you see repeated if you've ever looked at the plan line in Inferno papers are these three things that function as the primary driver for the organizations of the system. The representation of all resources as file like things. The observation there was that UNIX had this idea of you open a device. Devices unlike some contemporary opening systems. Which had specialized operation structures each kind of device. There was one for tape. There was one for mag disc. There was one for this. Instead there was just a name in the file system. You opened it and use the normal read and write system calls. And then the sector thing called IO control. Well originally S G T T Y and S T T Y and G T T Y. To specialize control operations on them. But this idea that the resources you had on the machine were accessible in the name space in this hierarchical name space. The same as the files that were on disk. The idea was in plan lines to take all that idea and use it much more effectively much more intensively to do to represent all resources in the system. That you ever want to distribute to some other node. Combined with that there's this idea of a computable name space. So the name space is not a directory structure that's physically stored on a disk somewhere. The name space is a more abstract thing. It's just the name of something and then what it refers to is sort of a second stage depending on how you built the name space and how you have connected things into that name space. And the idea of computable means that there are operations that actually manipulate name spaces. And the final thing is connected with the idea of distribution is the idea of having a file service protocol. Because you're representing all resources as files you've got some way of assembling name spaces. If you have a protocol that will export make the name space visible on the network and accessible through the network. Then you've made all your resources because all the resources represented in these file like things. All your resources distributed on the network without having to do anything special. So this is the set of system calls in the plan line system. I've divided them into three categories here and really there might be one or two specialized system calls that I haven't mentioned here. But by and large these are the ones that you use and that's it. It's not 250 of them. There's no problem about sewing. And they fall into three categories. The first category is the things to do operations. Sorry we've got line and a slightly strange place but first thing to manipulate files in a Unix like ways. You've got open rewrite close create with an E. You remove stats. So those are probably you can guess what those do. On the bottom you've got the primitives that manipulate processes and process images. And those again are similar to the Unix ones with a few additions. So you've got a variant of fork to create a new process. You've got wait to wait for a process. Exits is like exit but it takes a string as an exit status. And exec replaces the process image by the contents of a file in a very Unix like way. Then there's some extra ones that are different from a few Unix. In fact even the fork is given a different name because it functions rather differently. So rendezvous is a way for two processes to synchronize and exchange values. That was the original primitive that was provided to allow concurrent programming. For various reasons specifically for real time applications, semaphores were applied to avoid problems similar to priority inversion. Where you end up with a lower priority process blocking a higher priority process. So semaphores were added. And in many cases those have been used to replace some of the uses of rendezvous. Alarm and sleep. Notifying noted is similar to Unix signals. And seg functions manipulate memory segments. And finally there's ERSTRA which sets and retrieves an error string, a per process error string. Which is a textual thing similar to ERNO, but it's a string rather than a number. One advantage of that is that when you start looking at programs running in different environments, you don't have to have a globally unique error number mean that you have to maintain. And across a variety of administrative boundaries and also application boundaries. And then the middle function here bind, mount and non-mount. Mount and non-mount look familiar from Unix, but they actually do something different. But bind, mount and non-mount manipulate the namespace. So if you think of a process running in a tree of names, then bind, mount and non-mount allow the process to change what names are available and what those names refer to. Actually this refers to a couple of things, I'll just back up a bit. Explain those a bit more clearly because I'm going to get into trouble with the next slide otherwise. So the basic idea is that bind takes an existing name and puts it somewhere else in the namespace. It binds it to another point in the namespace. Mount takes a file descriptor which is connected to a file server and I'll get on shortly as to what that means. And puts a connection to that file server into a namespace at a particular point. And un-mount undoes the effects of those previous operations. So those three are enough. There are probably some interesting operators that could also be provided. But those three are enough for practical purposes to do what's needed. So here are some examples of file servers. Some kernel services provided in a traditional way similar to Unix, except that instead of having special files these names are actually generated on the fly if you like. The pure names that refer to things inside the kernel and the kernel itself generates those names. So it's probably closer to the sys and prox stuff in the Linux environment. And there's a sort of convention of handling things. Instead of having a control request instead devices that need device control tend to come in pairs. You've got a data, the read and write to do the data of it and the cuddle file associated with that does the control function. That way you can have ordinary read and write requests being directed. You open the control file name, use ordinary read and write requests to control the operation of the data side of things. What advantage of that is that unlike, say, IO control is that you can export the pair and then you can give a remote machine the ability to control a local device. Without having to worry about, well, what's the bit pattern that's needed in this particular thing? What is the set of values that is associated with IO control and so on? And multiplex is represented by file trees and then there's a big collection of user services. There's some fairly boring examples, user services that provide conventional file storage for DOS file systems, 96 CD file systems and so on. The interesting ones are the ones that provide services. So, for example, DNS is provided through a file server. There's a program called the Connection Service, CS, that provides the ability, translation of symbolic names. I'll show an example shortly. Mail file system, UPASSFS, that represents mail messages as hierarchies of names with the contents broken out, so the from, the two lines and so on are broken out. The graphic subsystem is done using namespaces. And there are various functions like IO stats, which puts itself in the namespace between a process and its original namespace, so you can track all the operations within that namespace and provide a little table showing which files have been accessed and how much IO has been done to each of them. Factotom is similar to PAM, I suppose, in Unix, it's a security agent, but it's the only one that knows all the security algorithms and it uses the namespace, it accesses a file server to allow other applications to make secure connections without having access to any of the keys and so on. And there are operations that work on the protocols to provide caching. Networks are implemented as hierarchies of names, so that you've got the ARP table, the connection service, the domain name service, to translate a name, you open that writer conventional string into it and read back a description of what the translation might be. The ethernet, if you want to know what the ethernet address is, you just catch that file. That's the thing, because it's in the namespace, it's accessible to the shell from the command line, so that you can easily wander about and explore a system. Again, some of these ideas have turned up in Unix laterally with CIS. Originally a lot of those things were kind of view onto things, but they weren't the real things, whereas these are views onto the real, there is only that in the system. So these things speak directly to the device in perhaps a deeper way, in that the only way to make a network call on the ethernet is to open up one of these files and read and write it. IP interfaces are available, it's actually collected instead of having jails or separate socket subsystems. You simply mount a different instance of the IP stack to get a fresh set of names that aren't connected to anything, and then you connect them to different network interfaces by opening one of these things and writing some requests into it. So here's our domain name service works. As you open that DNS, you write the domain name that you want to translate into it, and it writes back, sorry, when you read it, you get back a list of text strings that give you the various translations for IP addresses, and there's convention for doing the reverse translation and the rest of it. This also means that because net DNS is a file server, it's an application running in its own domain, in its own space, it automatically is able to cache things for all applications that use it. Furthermore, if you're on a tiny machine, you can import slash net slash DNS from another remote machine and do name translations using its cache and its ability to communicate with the outside world without having to run the domain name server in your local machine. Well, here's an example of how you can translate different other kinds of requests. I'm going to run out of time so I'm going to zip along. To provide network independent, so the system doesn't assume that everything is internet. To provide network independent translations, there's another service that's used by applications instead of using DNS directly, and it translates a peculiar form of symbolic reference for a network type, a host on the network under service. The idea is that you write the thing you want to translate and you read back a set of recipes, each line representing, well, if you want to get to this abstract thing, you can get to it by doing this or doing this. The interpretation of this is that the way you get there is you open the thing with that name that I've just given you and you write this string preceded by the word connect into the file that you've opened, and that will give you a connection in your process, a file descriptor, to the thing that you wanted to get to there. If that doesn't work, you can try this one and so on. I mentioned a computational namespace is some amount connected to a file server through a file descriptor. Bind takes an existing name and puts it over another existing name so you can remap things. It's important to note that the granularity of this is per process. I mean a group of processes can share a namespace but in the limit a process can have its own private namespace with just the names it needs to work with. So the ability to use a name and what it refers to acts as a kind of capability as well. So you can jail a process by simply not building a namespace that doesn't include say the network devices. A lot of it depends on naming conventions so you might have noticed in the early one there's all this is a clone files, there's a standard naming structure for multiplexers and a standing naming structure for the data and control file names and so on. Union mount is sort of a multiple mount really because it doesn't do the union of the entire namespace. It does a union along one dimension. So if you put a directory onto another directory you can choose where it goes before or after it and create a kind of search path and also if the names are different then beneath that then you can also create a kind of collection of possibilities on a single point in the namespace. So that's used to replace the shell's search path by simply binding the things that you want in the right order on slash bin. Slash dev is actually an empty directory when the system starts up and things are simply bound union bound onto that so it puts a collection of kernel devices onto it. Net similarly is empty and then the interfaces are added onto it. I'm going to run a time so I'll just do this next bit which is the protocol itself. So the protocol is called 9p. In Infernet it's sometimes called sticks but it's now the same as 9p. There were originally some minor differences between the two. It's a file service protocol unlike NFS. It's deliberately stateful because the things you want to talk to have state and so it's connection oriented and there is an application in a service that will provide a persistent 9p connection subject to certain constraints of what you're connecting to to recover from errors and things but that's a separate component. A fundamental thing is that from the start user applications, user mode programs are intended to be act as file servers. So it's not like fuse, so it's like fuse being added to Linux later. Here it's built in from the start and you can serve 9p on any file descriptor so it might be a network descriptor or a pipe. And you use the month system call to take a file descriptor and say the thing on the other end is an 9p service and from then on below the point where you attach it in the namespace the kernel translates all the file system operations you make into 9p requests on that connection. And the protocol itself has a very simple structure you've got just these operations. Users and groups are represented as strings not numbers so you have much less trouble doing mapping across administrative boundaries. You start to do something but it's considerably easy to manage. Diagnostics are also transmitted by error strings. You can use any byte sequence protocol so we've used it over infrared links to a Lego robot brick running a specialized program a simple 9p server that serves up control and data files for the server motors and various other things on the device. So this is the extent of the protocol and the implementation of the protocol. This is, if I explain roughly what these things mean, this is actually the physical representation of the protocol as well. You've got a four byte size, one byte with a particular value that represents the operation, a two byte tag that identifies a particular operation. So if you've got several read requests active at once on a particular file then they have different tags and several can be in flight at once. The FIDD corresponds roughly to an open file instance although it doesn't have to be open because FIDDs are also used to walk through a namespace. And as you see there's a fairly close correspondence between those operations and system calls. There is a special mechanism in there to do authentication as well so that the idea, again using reads and writes, if you get one of these it gives you access to a file that is connected to an authenticator for the file server which in turn turns around and uses the Factorim agent that I mentioned earlier. And through the combination you end up authenticating securely access to a particular part of a file tree as a particular user within that file server. So I'll stop in about a minute and take some questions that don't run over time. I'll do this one anyway because this gets the heart of it. So now you've got resources as files and I've given very quickly some idea that networks and services and so on will represent as files. You've got the 9p service, you've got the ability to make namespaces out of these things. So when you implement the distributed service the first thing you think about is what should the namespace look like for this service or a collection of services. Once you've done that and you've implemented the file server to do it, you have the ability to put things together. So for example in plan 9 if you do network graphics you do it by importing dev draw. There isn't a special network oriented graphics protocol. Of course on dev draw the files inside dev draw there are special graphic like operations that you do. But the distribution of it is all handled by 9p. Network audio, so if you've got a machine that hasn't got an audio device and another one that hasn't then you import the audio device into your local namespace and you can use it as if it were local. The sound will come up from over there of course. And if you want a network gateway so you've got a remote machine that's connected to the outside world and one that isn't then you can import that slash net. And then when you make network operations it's as if you were making them directly from that machine because you are. It's being transmitted by 9p across to that machine where it does the network operations there. That has the advantage. So for example at home it turns out that my virgin media line has gotten addresses in the blacklist so I can't actually use SNTP directly to connect to anything to do my own outgoing mail. But it turns out that the virtual server that I've got at Byte Mark in York isn't on the blacklist and so I run an inferno instance on that machine. It's running a virtual Linux. I run an inferno instance on that machine that presents a slash net just the same as plan 9s would and I import that slash net into slash net on my home plan 9 machine and then I can do SNTP calls. And it's fine because it's actually that other machine that's presenting the interface to the outside world. You know sometimes there are problems of efficiency and you begin to look more closely at ways of improving it or be able to do streaming and various other things. Nevertheless especially when developing this gets you a good long way without having to do anything special. In many cases network audio is an example network graphics especially with increasing speeds of networks things are just fine. So the CPU service which is equivalent to in a way to remote shell is done by it connects the so we call the thing that connects to the CPU server the terminal. So the terminal you connect to the CPU server the CPU you export your slash dev from the terminal subset of the slash dev from the terminal to the CPU so which binds which imports the slash that and puts it on its own slash dev. So now dev draw dev cons which is equivalent dev TTY dev cons refers to the keyboard and the mouse and the graphics device of the terminal. The window system similarly works as a file server so that you can do things like you can in the window system you can create a window CPU into that and then every window and then you can run the window system inside the window. And then within that window every window you create is actually automatically on the CPU server so in terms of the commands you execute because so there's kind of a recursive structure that you can you can take apart. So again in the embedded space the advantage is that you've got small devices that do tiny to do something big but you can program the 9p on them so they can export their things or they can import bigger services bigger machines and you can put together quite interesting internet of useful things. Internet of names. I would say internet I was going to use internet of names as a title of the talk but it turns out that's being used for something stupid. And internet of namespaces just doesn't have the same sort of ring to it. Okay. And so yeah so the system plan I also has got built in from the start so my support concurrency. This is the sort of stuff that was that wasn't in Unix in the first place because on a PDP 11 you were hard pressed to do too many things at once. And so unlike bigger mainframe operating systems at the time it didn't really have deep support for it's a concurrent use of shared memory. And the programming model allowed by the system includes both shared memory with usual synchronization primitives and also there's a library lib thread that provides CSP like channels. Inferno has its own programming language. If you like it's similar older but in terms of the languages you might know it's similar to go except perhaps with a slightly prettier syntax. Okay and in fact again compared to the development of Unix from the start most of the non-trivial file servers are concurrent programs. So the Windows system is concurrent. The export of Fasts which is a command that exports a file system namespace to a remote machine is a concurrent program. Acme is a funny kind of editor similar to Beart's Oberon editor in sort of style and design but it's actually a file server. And each text window you have on it corresponds to a set of a little hierarchy of names. The Windows system is a file server it serves cons which is the equivalent of TTY and that's the conventional name for the place where you get keyboard input. And it serves dev mouse and it serves a different so each so each window has got cons and mouse or pointer in Inferno. But in each window that cons and mouse refers to that local Windows virtual console and virtual mouse. So when applications read and write that the window manager turns around the multiplexes that use across the real across the cons and mouse that it got from its own slash dev. But of course that's how you can run the window manager inside the window manager is because within the window cons and mouse refer to just that localized instance and so the whole thing is recursively virtual. Oh and also important for there is real time support in terms of in the shape of a of an EDF scheduler. And the Inferno I mentioned it takes the plan 9 ideas it does it runs as a native operating system and a variety of machines like ARM, PowerPC, Ax86 similar to plan 9. There's a list of machines that can do but it also runs hosted so you have this operating system plan 9 like that runs as an application on Solaris Mac OS X Linux Windows. And from the point of view of the application written in this safe concurrent language limbo which is said to sort of go like in terms of style. And as you've got CSP like channels that you send and receive type to messages on. So it's a type to CSP. It's actually closer to the pie calculus in terms of the things you can do with it. But yeah so it looks like the same Inferno operating system that appears every while. So that's how I can run it on my virtual server on ByteMark and it virtualizes some of the host operating systems interfaces. So it makes the Linux socket stuff look like slash net. Or if you like the slash net names connect down through the Linux socket stuff. So the application doesn't have to do with socket calls it just opens slash net and does the usual things with it. And Inferno even more than plan 9 was consciously intended to be used for embedded devices. It was originally done developed to go in an AT&T set top box. And it was latterly then they went on to use it or a variant of it as the management section of an IP telephone switch. But plan 9 has been used in better applications. I will stop here actually which is that. The idea here is that these are ideas that could come out of the alternative universe and be applied perhaps in the mainstream of the current universe's development of Unix. Although it's tricky to do because so much of it has to be POSIX compatible. And the moment you say POSIX compatible that means you end up with in C say lots of if deaths and a ton of header files. I mean who knows what's in any of the header files. Well nobody does that's why you that's why they all set up so they include each other and then have checked to see whether they've included each others because nobody can remember which order they're in because there are too many of them. Which things are in stood. Is that stood in to any stood. Yes quite. OK so the idea in both systems design was to aim to be minimalist. To make a lot of use of compositions you have a few clear mechanisms. Ideally home to do things precisely and then you compose them to build something bigger out of it something more capable out of it something capable out of it. Inspiring well there are enough examples within the system that you start looking at particular applications say the example I was going to use if I hadn't run out of time was a grid scheduler to serve to serve jobs up to to a collection of work of slave computers that are doing computation. And you farm out the work. Well this is done by we it's an actual inferno appliance we built and that was done by thinking first about what namespace would use to represent the schedule model you would have and then developing it by splitting the work amongst three people one of whom had used namespace to present graphical interface to control it. Another one who did the work on the on the worker the client side the computational side to access the namespace to find out what work to do. And then finally there's the person who did the schedule of work and by using the namespace as the as the as the design basis the agreement between the various components. It would allowed the development of each of those things to proceed independently because if you want to test the scheduler you can just catch some files. If you want to test the graphics thing you can just create some certain some files in that real files or you can just echo things into a simple file server. And so on so when we put the thing together it it it worked. And since that was the day of the demonstration it was just as well. So it's inspiring the sense that you look for examples and you think you look at things a little differently in terms of how to represent these things and also in terms of how to split the problem up differently. And then looking at other applications in other operating systems and thinking trying to do similar sorts of things. And then finally you said the important thing about the thing is that it's bigger on the outside. So instead of having a vast thing inside this operating system which is huge and then you have to fuss trying to squeeze it down. You start something is actually fairly it's not tiny but it's reasonably compact by comparison especially by comparison some. It's probably a fraction the size of system D for example. And it hasn't got tentacles into the entire world either. And that's a result of being minimalist in design but also focusing on having a few primitives and then some operators to allow you to compose those rather than trying to build everything into everything from the start. Right so I'll stop there and are there any questions?