 So we're here at the Lunara Connect and who are you? My name is James Bartomew, this is my badge. And so what do you do? I work for IBM Research as a distinguished engineer. My primary day job is actually focusing on container technologies in the Linux kernel that assist us with the IBM Cloud. But actually at Lunara Connect I'm talking about some of my more hobby projects, which is a code that I've been working on for a long time that assists me in my day-to-day job working with Linux doing other things, but it's not actually part of the kernel code, which I have tried over various different attempts to get upstream into some of the various projects. So I'll be introducing some of what I've done, some of the things it enables, my trials and tribulations trying to get it upstream, and the person who persuaded me to do the talk thinks it might be encouraging for people just starting out in the open source community, especially at the Maro, to see that even if you have 20 years of experience as a kernel developer once you switch projects, it's just as difficult for you as it is for everybody else. So what's the problem? The problem is that open source is a very open process, as the name implies, but once a project becomes mature there's a sort of inbuilt barrier of resistance to taking code that actually comes from outside the project. And it doesn't seem to be anything conscious, it seems to be more a set of unconscious biases, but that makes it very difficult for people who have a mission which is just one particular patch set. And the fundamental basis of open source is supposed to be scratching your own edge, which means that as a person coming to an open source project you pull it in, adopt it, think it's wonderful for a while, then you find that one rough edge that really annoys you and as time goes by it's the thing you keep on rubbing up against. And eventually you get so annoyed by it that you pull the source code apart because being open source you can, you find what the problem is and you fix it. And then you put it back together and for you the rough edge has gone away and for a lot of people that's the end of this interaction. But there's another step in that interaction which is where you try to inform upstream of what this particular problem was, why it bothered you and how you fixed it. And a lot of times that makes you what's called a drive-by coder. Perhaps the term drive-by, especially with the shootings in the US is becoming a bit cliched and we should perhaps use the term something like single-issue coder instead. But what it means is that you don't necessarily, you care about the project as a user, but as a coder you only care about this one particular thing that you had to fix and now you're intent on getting it upstream. And a lot of times when you come to a project with just one patch they look at you and go but you have no history with the project, you don't know what our processes are, chances are you submitted the patch or something. So you get this barrage of things that you did wrong that you have to try and correct and it becomes a fairly monumental hill to try and climb just to get your one patch in. And I'm not really providing a solution or a panacea for this problem because there isn't one, but I'm calling attention to it because there are a lot of people who are in the same boat and want them to know that they're not actually alone. So there's no solution from you? Is it possible that somebody can come with a solution? There's no generic solution. Part of the solution is trying to interest the project in what you're doing. So if you can interest the project in what you're doing and they see it as valuable to them, everything becomes a lot easier for you. So I'm actually in my talk giving example of this. So some of my hobby projects enabling of what's called a TPM, a trusted platform module, it allows you to perform secure functions with keys that cannot actually be exfiltrated from the platform which is a very useful way of storing your secure shell keys and your GPG keys and everything else. So it turned out that last year I met up with Werner Koch who is the GPG maintainer. He was actually very interested in some of the stuff I've been doing with TPM and encouraged me to do it for GPG with the result that in his tree I actually have a drive-by branch from my TPM code. But that's basically one success and pretty much a sea of failures. So it's not a generic how you do this thing. It's just how I managed to do it in one particular case. I still haven't found the generic use case that works for everybody. I suspect because of the diversity of open source projects a generic use case doesn't exist. So what do you do with IBM? At IBM my day job is actually, well, I'm a distinguished engineer. So a distinguished engineer is the engineer who's actually on the lowest executive run. So your job is to care for projects from an executive standpoint which means giving advice and counsel and trying to move others forwards within IBM research. So right at the moment what I'm looking at are container security technologies. The one you may have heard of but if you haven't heard of it you can easily Google for it is NABLA containers which is a secure runtime experiment for containers that actually produces a container that is more secure than a hypervisor which we think is a very useful property given some of the propaganda that's going on in the industry. More secure than a hypervisor? Yes, at least by the benchmarks that we evolved. Part of running this project was actually trying to give an objective measure to what we think of as security. So that was the first part of the project and the second part was actually creating a container description that would have a highly secure profile as measured by this measurement that we actually gave. So is that a big deal for everybody in the world if there was something more secure in terms of what you're doing? I think in terms of what I'm doing this is more a research project but that doesn't mean we can't productize it and there does seem to be a lot of desire in the world for a container which has security characteristics that both match and exceed that of a hypervisor so I'm hopeful this will actually be useful in the future, yes. And what have you done before? Before that, you mean a long time before that, before I got into containers. I still am technically a Linux SCSI subsystem maintainer so before that I worked mostly on storage devices and I also helped found a company that did high availability for Linux a long time ago. High availability? Yes. What does that mean? High availability is where you run two programs in parallel and if one goes down the other one takes over. All right. So when you think about the Linaro, have you had some involvement within the Linaro people? I have to confess that I haven't actually been to Linaro Connect since I gave the keynote in Dublin in 2013. That was one of the first ones, right? Yes. The keynote was about... it was comparing AH64 to the failure Intel had with Atanium and worrying that they were committing some of the same mistakes so it was a slightly unpopular keynote from Alms Point of view if I remember rightly. Did they do any of the same mistakes or did they listen to you and they didn't? I don't know if they listened to me. I think they didn't commit all of the same... I mean, Atanium is definitely a dead architecture whereas AH64 is still viable so it's still... it's sort of not in the Atanium bucket but I think AH64 is still struggling, yes. Still struggling? Yes. So what do they need to do? It's actually very hard to say. I mean, Grant and a lot of the other people are talking about things that they really need to do which is get a standard description. It's enormously helpful that Intel, the PC, has a standard description that spans sort of laptops through to service and I know AH64 tried to do this but the community seems to be slightly fragmenting again when I look at what's going on with the boot projects and the boot descriptions and everything else so I think pulling that back together and making it as easy as possible on the OEMs would be a great step in the right direction. The other thing really was back in 2013 when I was doing this we thought that ARM would take over from Intel it was really based on the power consumption and what happened there is what usually happens in the industry when you specifically call out one feature in your competition the competition immediately ups their game so Intel up their game with the power descriptions for Skylake and the architectures beyond that so the power footprints are actually very comparable ARM doesn't really have the power advantage anymore so the next thing you need to do is find another advantage and exploit that instead you can't treat your one advantage as something that you'll have for all time you have to treat your advantage as sort of a temporary thing which will be replaced by something else yet and I haven't actually heard ARM articulate coherently what its next greatest advantage over Intel is now that the power one is very exhaustive Maybe it's something to do with customization doing something different somewhere Well customization is an interesting one but it's also an Achilles heel if you've heard about the Linux desktop it's often thought that the reason no Linux desktop really exists in mass market form if you exclude sort of tablets and smartphones everybody was thinking of laptops it's basically because there was so much choice in terms of a desktop consumers do not like choice therefore customization as an advantage is a double-edged sword and you were mentioning power and IBM is famous for the power what does IBM do with the power? Power is still one of our most profitable centers at IBM it's still the architecture that runs the enterprise but obviously power customers have an eye to the future so part of what I do as my day job at IBM research is look at opportunities to get power into the cloud and one of the great things about having your own CPU group which we do at IBM is that if you look at what happened with the virtualization technologies like hypervisors it was actually CPU assistance to hypervisor functions that really helped them break into the mainstream because before they broke into the mainstream everybody was on with the ZAN power virtual descriptions where the actual kernel that booted on your hypervisor had to be a different kernel from what you'd actually boot on your physical system once hardware acceleration came along that was no longer a problem so one of the things that I can actually look at in IBM research from Blue Sky point of view is what are the CPU enhancements that are actually needed for containers to get the same advantage that hypervisors got from the CPU it turns out that's actually much harder than what it was for hypervisors but we're still looking at it actively All this stuff that I mean I'm not an expert I was just trying to do videos right but all this stuff that has to do with virtualization and containers and all that doesn't it mean that maybe the architecture doesn't matter as much? Well containers is definitely the architecture doesn't matter that much because hypervisors were based on a hardware description emulation so you required the hypervisor itself to be embedded in the hardware containers lifts that up a level if you think about it in terms of ABI the unbreakable ABI for the hypervisor was the hardware description it was the VMM, the virtual machine monitor that emulates the hardware the unbreakable description for containers is the kernel syscall ABI it's actually that there are two interesting things to ask about this number one is is that as high up the stack can we go is there another higher ABI that we could actually make the basis of some future virtualization that goes beyond containers so that's one interesting thing that I'm looking at and obviously if you're wondering what that ABI might be it would probably be the G-Lib C ABI so there's a question of whether we can raise containers up to that and then there's another question of once you've raised the ABI up as you alluded to, what the hell goes below it so with containers honestly Linux is the only thing that really goes below containers but if we lift it up to the Lib C ABI it's not entirely clear that Linux will be the only thing that goes below it and so there might actually be an interesting opportunity for alternative architectures in this sort of future container world and what could that be? I honestly don't know because nobody succeeded in producing this future container description yet it's merely speculation that it can exist there are reasons why it might not be possible containers if you look at them was an accident of the fact that Linus Torvalds was so dead hot on Linux not breaking the system call ABI if he hadn't done that we'd be in the same difficulty producing container descriptions as Windows is and in fact containers would probably not have been a thing today if he hadn't concentrated with a laser-like focus on keeping the kernel ABI backwards compatible so it's not entirely clear to me that I can make the same thing happen with the Lib C ABI or there is some ABI above the kernel perhaps not Lib C, perhaps in some scripting language that we can also make the same promise for but it seems to be this backwards compatibility this hardness of the ABI is what's required to bring up the next generation of container descriptions Why is it so hard to get this done? Is it like some kind of magic algorithm nobody has figured out yet? Don't you know exactly what the steps to do to get this done? Not really, the problem is that the ABI itself is not as hard as people think it is and if you look at why Windows failed at this, it's a very good reason engineers like workarounds they think that when they see a problem the first instinct is to work around it so in Windows what they did was first work around for something that didn't work in user space was to fix the kernel to make it work and that then ties the kernel very tightly to the user space and means that what we can do in Linux like bringing up an Ubuntu Docker image on top of a Red Hat kernel can't be done in Windows you can't bring up a Windows 7 Docker image on a Windows NT or a Windows 9 kernel because the actual ties between the software the user space and the kernel are too tight so we know what the problem is but once you've got all that history you go back and fix it Who's going to be able to find the solution? Are you doing it? somebody else? Well I'm looking at it from at least in part of my research time at IBM and I imagine there are lots of other people looking into this as well because the potential rewards from actually fixing it would be interesting but it's not clear to me that it can be done yet we haven't actually, in IBM research managed to produce a coherent description of the libc level for a container and virtualization and all that stuff is there a lot of optimization that goes into that like you want to take advantage of the full CPU and all that hardware that's down there right or is that part of it? That is part of it I mean CPUs exist primarily as agents for executing the will of the program and as far as most CPUs are concerned the will of the programmer is written in machine code but the number of people who program in machine code nowadays is very very tiny so we've evolved this entire sequence of ecosystems that sort of do all of these translation layers above it but obviously as we try and lift the ABI up that separates us enormously from the CPU so either we do something to bridge the gap from the software point of view which could be done but it's not very likely or the CPU people wake up and do something to bridge the gap from their point of view which would be additional CPU enhancements that would actually support container descriptions Cool, and I guess there's a lot of other where are you based? I'm based in Seattle Are there a lot of cool things happening with your colleagues around there? Most of my colleagues are in Yorktown Heights which is where the TJ Watson research lab is so I spend quite a bit of my time either communicating with them on a video or actually going to visit the TJ Watson lab Which is at the other side of the US? Yes, it's in New York State near Yorktown Heights So there's a lot of people there doing interesting things IBM has several thousand people in IBM research doing all sorts of interesting things not just cloud we do things like quantum computing whether the cognitive the deep thought programs are also coming out of there we do an amazing amount of research It's a fun place to do it It's also IBM designing the next generation ARM processors, right? In partnership with ARM doing a 5nm, 7nm, all this stuff sometimes I've done videos before with somebody from IBM was like partnering and doing lower nanometers, you know helping the fabs get better Well, I've heard that people are doing that I honestly didn't know that we had a partner with ARM to assist them It is well known that going beyond sort of the 14 nanometers to 10 nanometers it's becoming steadily more difficult I've got to say that this is really far away from my area of expertise so I shouldn't be pontificating on it but, yeah IBM does lots of interesting stuff Come enjoy