 Hello everyone Welcome, I Guess we're ready to start So I said that already welcome. I hope you enjoy the conference. I hope you enjoy Sophie and Bulgaria First of all, I want to say big thanks to our sponsors without them the event would not be possible Of course the freebies the foundation our biggest sponsors They've been supporting us every year and here they are again Existence also a long-term sponsor big thanks to them as well And we're lucky to get trivago on board for the first year this time And we hope they join us next year again Google of course with us every year and Google are actually giving away an acer homebook There will be a raffle Anyone who wants to join in I should open the link filling a quiz and Maybe they'll win an acer homebook The link is also in the programs Are again with us for the last couple of years Madison Gurka as the narrow networks PC engines again with us Eurogame technology, it's Bulgarian company my employer actually The net bestie foundation So a couple of quick announcements before we start with the keynote the social event will be tonight at Sofia hotel Balkan and we'll start at 7 30 p.m. Oh so we had to Make a few couple a few last-minute program changes There will be no freebies the death summit reports But we have three news talks on the schedule this afternoon and those are the talks There will be on the website in 15 minutes Yeah, and of course we are live streaming the event and those are the links to the web streams Okay, so enjoy the conference and I now give you Jordan Hubbard who will give the first key note. Hi I'm Jordan Hubbard. I'm currently the CTO of IX systems one of the sponsors of this conference and Obviously freebies D and I go back a long ways In fact, I probably should have called this talk looking forward to another 20 years to be symmetrical But I can't see more than 10 years ahead and in 20 years. I plan to be dead so So I'm gonna talk a little bit about the future and the title of my talk is a little bit of a double entendre I am looking forward to another 10 years. I think there's some great things ahead of us but I'm also going to try to make some Suggestions and and and points about what we need to do to be relevant in 10 more years so I'm going to predict I can't win this either way because if you Disagree with me you will say that things don't work this way in the open source community It's a big hobby and we do what we want and if you agree with me You'll say things don't work this way in the open source community and we do whatever the heck we want But I hope to change your mind a little bit about that because I think for many of us here Especially myself, but certainly not limited to myself. This is not a hobby anymore. This has become More than that and we have we have achieved more than than what you would achieve with the mere hobby And I think we're we're certainly All in personal positions where freebies D is a lot more relevant to our daily lives So but let me first look at let us first look at where we came from so Freebies T 1.0 November 1st or November 2nd depending on what time zone you're in 1993 Just over 20 years ago. I did the first as the release engineer, but obviously we're working with many people 1.0 release of freebies D and this was Our this is our 20th anniversary So we're almost 21 years old now and as I said we can we can drink legally in the US in just a couple more months Yes, we'll we'll touch more on the drinking theme in a little bit So this was our first distribution media Okay, I'm lying just a little bit But not by much. This was our first distribution media. I remember installing freebies D a number of times over a 1.2 megabyte floppy and That's of course when I couldn't do it over a parallel port because that was faster. That was really high-tech And this had ramifications which which well we'll kind of touch on ramifications throughout this talk You'll notice for example that all of the files in the Bendist and this was true until comparatively recently is about just over 230k and that's because that was the size that we could break up the media and fit into both a 1.44 megabyte floppy and a 1.2 megabyte floppy so in case you've ever wondered why we picked that specific size it divided evenly into the installation media of the day and BSD turns out as full of these little Easter eggs where you wonder why did they do it like that? Well Let me take you 20 25 years back as Kirk can certainly tell us many stories about This was the 1.0 ports collection back when it was manageable You can see bash is there so It might have been We don't know can we scooch the Projector over a little bit. I think we're clipping clipping off the left edge. I don't know if that's possible but in any case Obviously some humble beginnings and this was my first bill machine that actually is not an actual picture of my first bill machine I don't think I ever photographed it, but it's pretty much what it looked like and it had a Absolutely blistering 50 megahertz Intel DX2 processor in it and I paid No kidding $1,600 for the first one gigabyte hard drive I remember when they first came out and I went and paid cash for it so that I could do builds of free BSD This was the first dream laptop many of us had This is Warner Losh and I in Japan. That's me touching of Sony vio Very lustfully because I couldn't actually afford one at the time, but Yeah, I was obviously a little lighter back then we'll touch on that too So And this was our very first conference As you can see not much has changed That is Kirk hoisting a a pitcher of beer and as David O'Brien in the background bringing two more pictures of beer To the to the podium at the Radisson. I think this was you giving your BSD talk Where you yeah, and you got progressively more inebriated as the talk went on, but you remain Remarkably lucid. I think we actually understood you for pretty much 99% of that talk but this was at the Radisson hotel in Berkeley and it was It's really interesting just think about how humble our beginnings were I mean the first morning the The water lines of the hotel was cut so nobody could take a shower and everybody showed up with their hair all plastered to their heads and looking and smelling not great and But we had a great time and we managed to pull it off and here we are More than 20 years later still having these conferences and having them internationally. So I think that's really awesome So the other thing that certainly changed is that 20 years ago Our targets in the commercial space and we did want to be relevant and have have commercial users were ISPs pretty much that was at least 90% of Free BSD's user base We also did a lot of Small and home office web servers and routers people got a 386 out of the closet with a couple of ethernet ports and made a router out of it Very do it yourself remember from all the Traffic on questions, and if we had any any major companies at all That was yahoo which remains to this day a free BSD shop and hotmail which does not We also obviously were used on a lot of basic developer desktop machines You know x11 plus 50 terminal windows was considered an acceptable developer desktop again, not much has changed And the majority of free BSD machines were physical PCs that ran on AC power And that may seem like a pretty obvious statement to make but again this is something that that will come back to later because that things have changed and The key takeaway from this slide is that free BSD's role in the world was pretty overt The the brand meant something when you talked about free BSD at least in the network services community and whatnot it had a had a certain Cache and there was a there was a certain brand value around it and everyone was very upfront about using free BSD So here. We are 20 years later. That's a 64 gigabyte USB thumb drive 64 times larger than the hard drive I spent $1,600 for and now most of you have one dangling from your key chains and Probably got it free at some conference as a giveaway It just amazes me and what you can obviously put on one is It's just astounding. We don't actually put much more than cat pictures and small installation images on them, but we could do a lot more with these and This little $35 device I rounded up to 50 because you have won an SD card obviously has more power by Probably almost an order of magnitude than my first 486 DX2 so That is a game-changer and it also means that a totally different part of the world can Can do this now, right? This is something you can sell into into much poorer countries. I paid a lot of money for my first PC Again many orders of magnitude more than $35 and the fact that you can now do meaningful development on something like that is again It's a it's a game-changer and obviously my dream laptop has evolved somewhat as well I don't have to stroke it anymore. I do own one and again this machine has Far more power almost a terabyte of flash storage huge display density Very powerful multi-core processor is just you know, I can I can run Five or six instances of free BST under virtualization Faster than I could ever have run it on a physical machine before Poor Sony vio just fell by the wayside somewhere and the other thing I want to point out Which I think is pretty relevant is that this has BST included You may not consider OS 10 to be BSD in the in the most classic sense of the word But there's a lot of BSD in there and more importantly There's unix in there and when we bought our sony vio's way back when when we could finally afford them you basically got windows on it and or OS to if it was that long ago and That those were your only choices and you had to to rip off some other operating system to get a unix desktop so the fact that we actually have Unix desktop this many decades after Microsoft did their very very best to kill Unix dead I think says a lot about the the staying power of Unix So obviously the current release version is 10.0 working on 10.1 is I believe we speak They're now 24,000 ports Somebody asked me if I had known then what I know now if I would have done it There are hundreds of committers from both act. I'm not gonna answer that question. There are hundreds of committers and A lot more we still have a lot of strong showing in academia But as I mentioned earlier a lot more of us are in the commercial world a lot of us have actually been paid to come here and that's a very interesting and very Telling development that again we've grown up We also for the first time Have long-term sources of funding the free BST foundation to broke a million in donations last year That's that's that's real money now, and it's not they're not just standing by themselves There there were a lot of sponsors We saw on that slide a minute ago And a lot of people putting putting real sums of money into free BST and sponsoring the development of it Which I think is a really great thing because there are there's always going to be a category of really annoying things that Nobody really wants to do that much for free Or just in their spare time and so the the funding makes tackling those sorts of challenges possible But what's also interesting is that the markets have really changed, you know We're not aiming at where we were 20 years ago or at least we should not be and one of the things that's really changed is that The GPL has gone to version 3 and it has created a renaissance in interest In the BSD license in the commercial space I use that particular animation because I really you know you think of kangaroos as nice cuddly animals And then we kicks you into the water Which is how a lot of people felt when in the commercial world when they followed the GPL Bandwagon and then suddenly v3 came out and a bunch of technologies that they were reliant on went v3 And there was a whole bunch of new scary languages Scary language in there that most lawyers Scares the crap out of most lawyers I mean I've been in a lot of meetings with a lot of lawyers where they're going in line by line through this thing and going Take it out. So That's actually helped us quite a bit In roles that I'll touch more on in a minute So one of the areas in which it's helped us is that you're seeing free BSD and other variants of BSD or pieces of BSD technology being used in a lot of appliances routers load balancers security appliances monitors file servers obviously And it's also the basis for a lot of software appliances, which you can put on more generic hardware Obviously, I work on free as so I had to put it on there But it's by no means the only one and if you look at the commercial page at the previous D website You'll see a bunch of different things there And as I was surprised to see last year, it's even the base OS for a popular gaming console so What's Really becoming more evident than anything else though is that what we used to consider the embedded market was engine computers and cars and Avionics computers things that were really really small and special purpose had very limited capabilities Not much memory and some sort of our toss that ran on them but that is really changing overnight and If there's a key takeaway to kind of set against my previous takeaway It's that free BSD is now becoming much more covert in its usage people rebranded right juniper calls it to Junos the Company whose name I can't remember right now up in Seattle calls it 1fs EMC and then their subsidiary Again, if you look at the website, there are probably a dozen different names for free BSD now People just I don't even know what Sony calls their operating system. They probably don't even refer to it It's just the operating system that runs in the PlayStation 4 What's that? Orbit, okay. Well, there you go Interesting name But it's not called free BSD and you know, is that a bad thing? Not really what it is is it just says that free BSD has reached a certain commodity value That it's useful as a brand name to know what technology with where the technology source is But you don't have to call your product free BSD in fact You probably don't want to because it you're gonna you're gonna do different branding exercise around whatever your product is If you're a security appliance, you're gonna try and put the word security in there somewhere if you're a gaming company I guess space or you know and so Again, that's that's a very interesting development for me at least because it really does mean we've reached the come the OS commoditization era where Free BSD is a brand is both meaningful and not meaningful And in fact, I've had a lot of arguments with people where they've said well You really need to change the name of it because free BSD. What is that now free has weird connotations in BSD? Well, that was 25 years ago at some obscure place in California and My rebuttal is always it doesn't matter. It's just a just a brand right when when we came up with the iPad at Apple There were a lot of people talking about the connotations of that word Nobody thinks about it anymore. It is just a brand and free BSD has reached the same place We don't need to change the name. It's it's not it's not a problem so Looking at trends and I just grabbed this slide from from Gartner and I don't really care for it very much because it kind of tries to put all of us somehow at the iPad's doorstep, and I don't think that's Perfectly accurate. Obviously mobile devices in general are what have caused that curve to happen It isn't just Apple releasing the iPad on April 3rd it was probably the the match or the match that lit the gunpowder, but we're really seeing mobile devices take over the world and That's not going to stop and In fact, if we look at OS deployments today by sheer numbers or sorry Unix OS deployments phones are obviously the biggest tablets Watches that's an artist conception by the way. I have no idea what the actual I watch looks like and Cables what? Unix runs in a cable. Yes as a matter of fact it does There are a number of peripherals out there that do video transcoding DRM I hate to say Various tasks that require a pretty robust software stack You don't want to really take an mpeg for decoder stack, which is half a million lines of code and Put it on top of an RTOS because it turns out that that code was written on some sort of Unix operating system And it makes all kinds of system calls and relies on a number of libraries So as you start kind of pulling on that piece of string you very quickly find out geez I need a Unix in here. Well, it turns out that's not a big deal because there's enough space in those little connectors to put a tiny little arm chip at each end and so one of the first thing that cable does is boot and Yeah, what could go wrong, you know our cables gonna have heart bleed bugs or you know It's it's an it It's an interesting question so So yes, as I said the embedded market is really morphing what you think of in bed is embedded is So embedded you don't probably even realize it and when we get fully into the internet of things whatever the hell that means You're gonna see even more tiny devices that In these various ubiquitous roles that you never imagined a Unix operating system being in but people are gonna reach for Unix and stick it in there because it's the easiest and cheapest thing to do and The the hardware is powerful enough all the reasons that that you know Restricted you from doing it like having only 4k of memory or a one-bit processor that is so yesterday So, you know talking a lot about mobile here But obviously the enterprise space is kind of the other end of the barbell And I think it deserves its own own chapter so to speak And even there it's kind of interesting You're seeing a lot of pressure at what we kind of considered the classic enterprise desktop space with people bringing their own devices in That's that's the BYOD movement people talk about and it drives IT administrators insane because they have to take this Huge of array of devices that people insist and usually people by people. I mean the CEO Insist on bringing to work and insist on it Operating interoperating with their software and working with Salesforce and all the various things and so For about three or four years. They fought tooth and nail against it and finally they just gave up and so You're seeing that what what is considered sort of the classic desktop deployment even in the enterprise And I remember an era when people like GM and Unilever had 10,000 20,000 desktops They're they're going the way of the Dodo bird Why right just take your iPad with you or your your tablet of whatever flavor you like or your phablet your large phone and Use it during the meeting and that's what people are doing and they can project from it and do a lot of other things The other thing that's squeezing it from the other end is this whole cloud computing thing is becoming a little less nebulous And a bit more specific in that if you want to do computation and you want to do storage and you want to do a certain known set of things the the economics of off-hosting to the cloud and I hate that word because it means some so little and so much is actually becoming pretty compelling and Plus you're seeing a lot of software vendors a lot of ISV's saying I won't sell you a copy of my software any other way I don't want you hosting this yourself We're hosting it in the cloud So you you sign up and you pay your your monthly fee and pretty soon Why why do you need anything more than a web browser to run the software? The other thing that's kind of interesting too, and I've been noticing this I had a Older friend of mine even older than me Say that he just got a job at one of those San Francisco startups that are kind of going under the the collective description of two guys with two laptops and a coffee shop and He was amazed that they were able to do An entire office's worth of work from their laptops in the coffee shop because every time they committed a change to the code It fired off this huge chef driven set of automation which would spin up machine virtual machines and Amazon EC to and Start various tools running and they're due regression tests and the whole bunch of things and some more VMs would fire up and it would go to a staging server and then they would connect to their little URL for the staging server and they'd look at it and say you know click around use it Yep, looks pretty good They click a ship it button and then boom it would go and deploy and During the peak of the automation and they would run a lot of tests I mean this wasn't just a fly by the city or pants kind of operation they probably had 50 or 60 VMs running and So they didn't need a data center. They didn't need a dedicated IT guy they just they just had somebody who understood chef or salt stack or one of those automation frameworks and Spend a few days or maybe a few weeks or a few months even Spinning up all of the automation infrastructure they needed and then they were done and they just use that so again This has ramifications, which I'll get back to So No one's no one argues that Unix is basically Lego, right? It's it's a bunch of composable pieces you can put together in interesting ways and one of the one of the aspects of Lego is that It's a great general-purpose toolkit and we we kind of designed Unix the same way It's very composable and we see it as this elegant awesome ass kicking kind of thing The world sees us more like this Which is just a pile of Lego man We just we're gonna take the legos that we want and please try to make them all square And we'll put them together it's for these automation tasks and these sort of mobile and embedded development things But of course they're not all square because this is really more what it looks like It's a lot of square pieces and some very specialized pieces that you can't do anything, but one thing with so what What's what's hitting us hard in our in our insufficiently abstracted Lego universe is Again the fact that we have a we're carrying around a lot of weight to support physical hardware that really no longer exists in quantity The most most I think I don't have any metrics to make up on the spot But I would say that probably the majority of OS deployments today are virtual If you just counted sheer sheer number of seats And of course the devices attached to those are also virtual So the quality of your virtual device drivers is probably more important or becoming more important than the ones for your physical device drivers Now there are exceptions of course 10 base 10 gigabit Knicks and whatnot or 40 gigabit or soon a hundred gigabit They have a very specific demands that they place on physical hardware So at some point you have to run your your copy of ESXi or Zen Center or whatever on some physical hardware That has to work, but everything from there on up is virtualized and Again to talk about the the two guys with two laptops in the San Francisco coffee shop scenario Most of what their automation tools actually do is do their best to erase the notion of a host personality There to take all the configuration information and take it from some other place and splat it on top of a running OS instance So a lot of the the beautiful abstractions that we've built For for configuring and for sort of personalizing a box or even the the workflow that we're used to where we get a copy of free BST on a CD-ROM or thumb drive if we're a little bit more progressive and we install it on a physical machine and we go through all the configuration steps and Lovingly tailor it to our needs. That's dead It really is I mean we do it because we're used to it. It's really really familiar to it I mean we kept using system stall for how long so But that's not how the rest of the world wants to do it. They want to do it in an automated fashion They want to script it so getting back to mobile Again, I feel very comfortable in saying that the number of Unix machines running on a battery are now several orders of magnitude above those running off of AC and They're not plugged I mean I'm physically plugged into a few things because I have to be but again You're talking to radios now. You're talking to Bluetooth. You're talking to Wi-Fi. You're talking to near field communications devices your is a radio is an RFI world and That creates some interesting design pressures on you obviously power consumption becomes absolutely key if your Watch or your phone goes dead after just 24 hours of use you consider that a very unusable device or at least You're pretty pissed off that you have to go plug that thing and every night or it's dead the next morning and Because radios tend to to make very Sporadically That's what's word. I'm trying to find here the quality is variable in a radio signal and the things like interference and whatnot become real problems things like channel a Congestion so all of these things put back pressure on the operating system to do things like become more agile and Hop around or change the the radios configuration to to dynamically seek out the best possible connection opportunities and obviously if you're a truly mobile device you might be doing a hundred miles an hour down the autobahn and Continuously disconnecting and reconnecting to different cell towers. So that that puts some some really interesting constraints on the OS so one of the less obvious points to make is that trying to debug this and Actually diagnose problems in the field also put some some real challenges on you and our telemetry sucks Our ability to do remote debugging sucks. I'm not gonna not gonna mince words We have some work to do there because again if you're trying to make a genuinely mobile device You need those things you will live and die on the strength of those technologies So I wanted to point out something else. It was interesting because if we look at our open source brethren One of the biggest desktop providers in the Linux space that I think we can point to now is Ubuntu There they're a huge success story Mark shuttle worth opened his pocketbook and you know basically wrote them a lot of blank checks and they were able to do pretty much anything they wanted and If you look at the the evolution of their website over the last year They used to be a desktop OS and now if you look very carefully you see cloud server desktop Phone tablet TV management and all of the large panels down there are about how they're ready for smartphones And they're number one for cloud They they are they are not skating to where the puck is going, but they're chasing it they saw the puck go by and they went whoa what happened that way and We have to do the same so What that means in technical terms is we need to be Really open to fundamentally new approaches one of the really one of the the best things that I got to do when I went to Apple in 2001 was get given essentially a blank check on the engineering side I was told look we already pissed off our installed base of Mac OS 9 users There's really nothing more you can do to them that hasn't already been done your your starting fresh with OS 10 and Do whatever has to be done to make it the the best OS for the kinds of products that we want to make and We I took them up on that that we all collectively took them up on that and we we went I won't say we went nuts that that would be that would be too too strong a word But but we took full advantage of it. We we ripped cron out. We ripped. I net D out We ripped a bunch of other weird legacy services for mock out and we consulted it all at launch D We wrote all kinds of different mechanisms for dealing with dynamic Device arrival and removal and power management and we just we just went to town and we knew that we were going to be able to abstract all the system administration Issues through away through the GUI or later with iOS with almost no UI at all at least not for system administration and So we were able to fundamentally rearchitect big big chunks of the system and that was actually a wonderful thing because For the first few years we dealt with all the same reactions that you would expect in a classic Unix environment You did what you changed what you can have my I net D when you pry it for my cold dead fingers, right? We had all the same reactions. It wasn't like everybody was like oh sure But a couple of years down the line whenever one saw a G You know I can just launch this by dropping this file into this directory And I can specify in a very flexible fashion how to scrub the environment and how to launch it in a secure fashion I don't have to worry about people Pre-pending stuff to my command line, you know There were a bunch of security ramifications and just ease of use ramifications that came along with the rearchitecture that In a couple of years it was like of course you do it that way. Why wouldn't you? so Again, that's really hard and open-source We have the the famous bike shed effect that we like to talk about in free BSD where you know any proposal that That threatens to change the status quo gets bike-shedded to death and I think that's really Not as charming as it used to be I think that needs to change because to do all the things that I'm pointing at and to move into these markets means Some things have got to change. We've got it. We've got a Fundamentally, we architect some things We also need to be willing to shamelessly, I mean adopt and learn from things that are that other people are doing I mean, we don't have to do this all ourselves. There are a lot of technologies out there that have been Practically offered on a plate to the free BSD community in the past and they've kind of said It's not really our flavor. We didn't we didn't architect that ourselves Yeah, we'll think about it and then years go past and opportunities are missed So I think we need to be a lot less proud about that because again, we are a commodity OS now There's not a lot of point in being truly unique and Nishi anymore And the the last thing is I think we need to be willing to take on some big picture challenges Back back in the 90s when I first started doing this I used to you know stand up and stages like this and talk about Finding a mountain and my big analogy was you know when you're going on a mountain climbing expedition It has a wonderfully focusing effect on everybody involved You got a pack for it You got to get all the right supplies and now it's kind of dangerous and you're climbing up a peak and it's hard But it's also incredibly rewarding and you're doing something that you've never tried to do before at least if it's the first time you climb that mountain and It it brings you all on the same page, right? You're gonna climb that mountain and try not to die and I think I think we may have lost sight of that a little bit I think we need to find some more mountains to climb We need to pick some big picture challenges that will be hard because you know when I first Got involved with this. I was I was 30 years old and 30 pounds lighter and I'm I'm getting old and A lot of us in this room are getting old and we need to find something We need to find challenges that will appeal to the next generation of of hackers And that's something we actually did pretty well in in the 90s and the first part of this millennium It it shocks me sometimes to realize that some of the movers and shakers in this project that the leaders the grand old men We're just starting college when they first came to free BSD. They were the young snotty-nose kids, right? and Now they're not they they they have they have become the new generation and the leaders Well, that's gonna happen again again if we're trying to get 10 20 years further down the line we have to think about that now and Thank God for the Google Summer of Code by the way because I think without it. We wouldn't even know what interns were So again, I want to get specific. I don't want to just keep waving my hands up here but one obvious project idea that we need to take on is To automate a machine to to really run it through all of its paces cradle to grave you need to be able to munch its configuration data and Slash Etsy is a septic tank It is just a whole bunch of weird-ass formats and things that have been there forever and ever and ever It's really really time to just toss that out. That's one of these re-architecture goals. I'm talking about And I don't care about the format. It can be YAML. It can be JSON. It can be XML but there should be a single way of accessing it reading it writing it finding it and You know, it's really kind of sad that even in this day and age in the year 2014 if I write a new utility that's going to be a new demon that runs on my system and I need to have a Configuration file format for it. There is no API. I need to write my own bison parser and Invent something and put it in slash Etsy because why not everything else is there That's and let's not even talk about the dot files in your home directory, right? This is just architecturally Yes, and obviously we need to have the the cojones to go back and alter all the existing tools to use it That's the thing that everybody always stops at. Well, yeah But if I come up with the library and I come up with the single unified format that I gotta go change these things and change them It's it's more trouble to talk about it than it is to just to do it. It really is I've been there done that It's not as hard as it looks Oh and Just to point that out that it's not as hard as it looks to use this little machine here as an example Everything in OS 10 whether it's system configuration data per app configuration data per service configuration data It's all an XML P list in a couple of known locations And that's it and we went we hacked the utilities We did everything required to to unify on that. Is it the best format that ever walked the earth? No Is it good enough? Yes What's that? No, you just look at core foundation light and you can again steal the ideas You don't have to take the implementation just I mean there's not a lot to it There really isn't and they're there are probably 50 XML parsers It's you know, you don't have to write this yourself So another thing that needs to be done Especially if you want to work in the mobile space is you need a notification centralized notification system You need to be able to register for notifications. You need to be able to post notifications Right now on your average free BST based appliance if I even just changed my host name a bunch of stuff stops working That's really silly. It's one of the first notifications that OS 10 added was calm dot system dot host name You subscribe to that anybody changes it from any place boom You know, it's changed go do whatever you have to do to your internal state adapt move on Obviously radio is powering up and down at all times like I talked about that were configuration changing because you're moving between cells Your IP address is changing Chef or puppet has gone and whacked a new set of configuration data on to you entirely and everybody now needs to go and you know The world has fundamentally changed and right now that is done so crudely It's just you run around here everything with a hammer that you think might have been affected and restart it And there's just no technology there. It's just brute force And it doesn't need to be it really doesn't what you also see is a result of not having a distributed notification system is there's a bunch of things like active directory libraries and whatnot that cash information and Since they have no way to do cash invalidation, you know, they break the first rule of computer science never cash anything You don't know how to invalidate. Well, what they do instead is they run a cron job once every hour or whatever to shoot the cash In the head so they do a bunch of work wasting a bunch of power and you know And possibly doing needless network traffic to recreate a cash that didn't need to be recreated or During that one-hour period when that user got added over there and now people are going I can't chone something's wrong Well that cash is still screwing you up So you really we really need it and again because I want to talk about real examples Here's the notify API Notify if I API was done probably in all of two weeks and a simple notify demon If you have an OS 10 box You can send it a sig user one and it will dump all the subscribers Into var run notify and the name of the PID status and you'll see there's just a huge number of notifications And when we first did this we probably had five six in fact I think we first did it to do with time zone changes because we found out there was code and libc that was statting Etsy local time Thousands of times a second. Why well because the time zone might have changed So it's one of the first things we did is make that a notification And you'll see if you can read the slide that some of those well, it's to slid over too far But some of them are you know direct point-to-point notifications. Some of them are using a little shared memory cells So if you want to pull Periodically just see if your caches and melody you can do it that way it'll send you a signal It'll do any number of things. It's open source So again either use the implementation or just use the ideas so as I keep mentioning the whole system startup and service wrangling stuff is there There's an old saying you you you can't polish a turd, but you can paint it and Etsy RC dot D is a painted Etsy RC So, you know, I was one of the architects of this stuff I I think I probably checked in the first copy of RC dot local RC dot cough actually because hey, let's add some knobs What harm can that do? And it's it's evolved. It's incredibly sophisticated I've actually looked at the Etsy RC dot code and seen all the little provides and you know It requires stuff in there and it's seeing how it builds a directed acyclic graph and runs all the things in the right order And it's like wow, this is kind of a triumph of of brute force over adversity But it's also wrong It's just not how you want to do it. You want to do things dynamically you want to essentially just register and say hey when something who tries to talk to me start me up and I don't want to make any dependencies explicit anywhere because the problem with explicit dependencies is they keep changing and You keep having to chase that dragon And anyone adding a new service doesn't even necessarily know which what they depend on they find that out by trial and error so again you know, I'm I'll talk about that in a second But you know, I'm not gonna I'm not gonna suggest launchy because I've beaten that drum enough times But even the Linux die-hards have gone with system D. They've refactored their system And it's you know again just having a broker that you can register with and you know on-demand with IPC or timed events Or hardware arrival removal will do the right thing using some metadata to tell it what the service needs to do and what it needs to run is really the right architecture and To that last slide if you really want to do power management You also fundamentally need this because you can't let things run too long You you do need you do need an adult in the room right something that says you have been running for too long and I can see your prior consumption going up you've run away and You know people write crappy code It's a fact of life and you're not always going to be able to even control what crappy code gets installed on the system Because there are these app writers who write these apps and they stick them on there and they run them in the background Or they run them from Cron or whatever and when one of those things runs them up on your mobile device All you know is that the mobile device which should have had a battery life of five or six days had a battery life of two hours And now you're pissed off and you blame the device you blame the crappy operating system on that device You don't blame the errant app because you don't even know that it was errant Where where do you get that information from what what telemetry tells you that it was errant? So you need to shoot it down and or you need to at least have something that's in charge to report on it Say hey, this guy is acting weird. You should probably do something about that So again, I keep coming back to this But yeah, we need we need telemetry and speaking with my free NAS hat on we're gonna do it at iX systems we're gonna create a telemetry a demon and some API's for it and we're going to pretty much instrument the crap out of everything because We we are at the pointy end of the spear in terms of dealing with enterprise customers who have file servers Which are one of the most critical pieces of infrastructure for people and when those things go go hinky They rely on us to figure out why and far too often we have no idea We we come we come to the scene of the accident an hour or two after it's already occurred and all we do is see bodies lying all over the road and We don't even have things to measure the skid marks with because there are no skid marks, right? These are virtual bodies and it's a virtual accident and so We need to we need better than syslog. We need some way to say alright Let's look at your telemetry. What what was going on in the hour prior to this event? And maybe even what was going on from a couple of different daily collection points because guess what people break their networks people have clients that go completely insane at three o'clock in the morning usually on Saturday night and Start flooding, you know the server with with with NFS requests or just otherwise acting bizarre And you can't even always detect that from the server. All you know is you get a lot of traffic. Was that good? Was it bad? I don't know so Usually when you talk about telemetry and remote debugging Certain three-letter agency gets mentioned and everybody starts throwing things Let me tell you something the NSA does not need our help They do their data collection much further upstream where it's already pre aggregated for them They do not need to attack the individual boxes. That would be a waste of their time but the the users do need the telemetry because again to keep her turning to this point maybe harping on a little We're running in a lot of appliances now and those appliances the users just want them to work and You've probably used a hundred products just in the last year where there's a little question Do you want to help make this product better? Unless you're really paranoid you probably said sure guess what that meant? You signed up for telemetry So finally on the debugging comment, which I think should be totally uncontroversial We do need a way of doing IP debugging and that's another thing that I existence is going to sponsor We have a rough prototype already working, but I'll tell you I could never have worked on this or the phone Without some way of debugging it remotely. It was just just insane trying to do anything else You know you're not going to always have a serial port or or a Serial over IP or you know any of those sort of Clujie ways of getting at debugging and of course sadly firewire is going the way of the dodo to and that was really cool Because you could do firewire DMA But it's leaving so we need to be able to deal with panicked boxes in the field over IP And that means basically a very simple IP stack that that gets that gets used in the in the times of last resort and says All right, debug me now And guess what when you have that feature you can also start doing other interesting things which I've used as well Which is a little web portal things that do nothing But say give me an IP address and I will go connect for you and I will scrape that box And I'll run a bunch of GDP scripts on it and I'll grab all the relevant information and I'll file a ticket And I'll attach this stuff to it and I'll even tell the triage team Hey, it's still there at the debugger prompt if you want to connect and do different things And if you don't well, I got you everything that most people need to debug a kernel panic so Also being able to NMI a box when it's stuck or hung or really really slow Or you know there's some weird lock contention problem going on it can make the difference between finding a really pernicious problem That's been annoying people for years and and not So One other thing I've raved about on the the project mail is a little bit is I think I Think the SBC is the raspberry pies the beagle bones all of that as porting platforms have been really valuable I don't want to even sound like I'm disparaging them. I think they're really cool. They're really cheap They're really useful and we should keep porting to them But I think I think we've reached the level of stability and usefulness with the arm port that it's time to actually move to real stuff due to to a reference phone to a reference tablet and I'm not even sure what that reference phone or tablet is yet There's a plethora of them right now if you go to Singapore or Hong Kong you can go to a market and buy Five tablets for for a hundred bucks. I mean, you know, they're they're really generic now And they usually run some weird concocted version of Android or something but they are getting cheap enough now that at some point we're going to be able to kind of look at the the horrible The not so bad and the the great and then pick something in the middle because the greater probably going to be expensive and Say that's going to be our first rest reference platform for a phone port or for a tablet port And and the main reason for that is is is multi fault They're multiple reasons for this one is as I point out here You're never going to know how to deal with real radios real devices like accelerometers Cameras etc. And so forth until you actually run on a platform that has those things And you want to be able to support those things because they're part of what make a phone or a tablet an interesting device to use Also being able to carry around a copy of you know free BST running on one of these reference things means we'll be able to actually Validate that our telemetry and our debugging stuff works because if you don't use it you don't live on it so to speak then you are not Validating that that an end user in that same role is going to have a good experience And finally again using those sort of things will require that we think of a little higher up We're not just a kernel We are a collection of services and utilities and at some point we'll have something for dealing with Bluetooth and dealing with audio and dealing with displays and all those things are going to have to be part of the Software stack and I think it's time that we we tackle some of those things because again those are the interesting challenges It's it's previous to you should be more than just hacking on kernels So to sum it all up I think we still have some rough edges and not being sufficiently legal like again OS is a commodity now It's it's been out for decades and decades, so we should make it More usable for the people who want to automate it virtualize it stick it in a cable and and not have anything that stands in the way of doing that or Essentially force them to do the subtractive exercise of pulling stuff out It's it's one of the first things that people do is is have to have to yank code out of free BSD and that's kind of silly We we should make it possible just to conditionally compile it in the first place And again just to reinforce my previous point. We need to pick hardware platforms that are that are real and relevant phones tablets watches Cables let's let's find some reference ones and actually start porting to those and see how it works and Because we're going to be kind of late to the party We should be willing to look over the fence to people who've already done all this the androids and the apples of the world Have already been down this path, and they've done a lot of cool technology some of which is open source Some of which you can just look at a header file and say all right come and see kind of what they did in this design Or just watch it work, and you can learn a lot from that and clone it You don't have to necessarily take the same code. You can you can create your own independent? Implementation, but but skipping a lot of the hard work and figure out what it should look like So and let me just say that we do have a bit of a rep in being resistant to this kind of thing that you know If you go to the BSD folks, oh, they'll bike should eat a death. They're not going to take your ideas on board They're not interested that that's a really unfortunate reputation, and I don't think it's entirely merited But to the degree that it is we should shake it off and change that reputation So in summary the future is ours if we want it. Thank you