 Introducing from defer panic we have Ian Eiberg who will be talking about adventures and building unicolonial clouds. Ian? Thanks, everybody hear me? All right, sweet. So yeah, I'm an engineer over at defer pack. We build out unicolonial cloud situations and so that's basically what we're going to talk about today. Real quick just to show you on the list of what we're not covering in this talk. This isn't like a high-level evangelism type of talk. We're gonna actually talk about how to implement this stuff rather than go ra-ra. And we're not really going to spend any time on defining unicolonials. I'm sure everybody who's interested can type into Google and figure out a brief definition of it. Nothing about IoT there's another great set of sessions on IoT. If that's your thing, I would go there. Nothing about NFV. There's also quite a few sessions on NFV. I personally know the Ericsson guys that work on this stuff so it's but we're not going to talk about it. What we're really talking about and I would imagine many people in this room are working with our web services and services like that and so within that category there's kind of another big split. You have your compilation of unicolonials and then your kind of orchestration side of the things. We're not talking about compilation whatsoever because that's its own like little rat's ball of bullshit. We're focusing on the orchestration side. Lastly, there's lots of people that talk about unicolonials online and there's been more and more speakers about them and so forth. But there's also like another group of people that like write articles about them yet they've never even like booted them up or done anything with them. So we're just talking out of their ass. Don't listen to them. And if you want to, you can go to a slash defer panic slash Virgo on GitHub. Go ahead and clone this and we'll play around with this app like later on. This is basically like a teaching utility just to show you kind of how to orchestrate certain tasks with unicolonials because at the end of the day it's not just about running things in prod. There's a whole host of other concerns that we'll get into. But yeah, you can download that and kind of follow along with it. So let's begin the question. I'll ask a few pointed questions. First off, what is an operating system? Right? I think we all kind of have this vision in our head of what operating system is. But you know, on one hand we have OSX, right? So this is running on this Mac here. And so we can say, oh, this is an operating system. But yet I'm willing to bet none of you host actual production software on this operating system, right? Most of you probably use this guy, Linux. And so there's lots of reasons for that. And it's something to think about when we're talking about serving, especially like web services and so forth. Now, I would ask, is Linux the appropriate system to be doing that in 2016? You know, maybe this style of systems worked back in 1971, 1969 on the PDP 11. But in 2017 now, I would argue that it's inappropriate. Brings us to our next question. What is an application? Once again, it seems like a pretty, pretty easy thing to answer. I just, you know, I'm not a big gamer. There's probably some gamers in the room, but I just typed in most popular 2017 game or whatever. I got this image of this kind of cool looking sci-fi thing. And I'm just gonna guess, looking at this game, that there's so much crap going on in it. Like there's all this video stuff going on. There's all this probably super advanced sound stuff going on. And you think about, like, all the things that have to happen inside the source of this game. And then you think about how dramatically different this application is from this application and what all it needs to go into making engine X work versus the prior application. And you start thinking, what is appropriate to be putting on my server? So why is this important? You know, like, I've asked these questions, but why is this really important? So there's this guy named Peter Levine. He's pretty famous in the valley. He works for A16 now. And he has this video that he did about a month ago called the end of cloud computing. If you've got like 30, 40 minutes to blow in the next week, I would just watch this. It's kind of cool. And he kind of talks about these shifts in computing that we've been taking. And he's been, he's talking about like last 10, 15 years, we had our cloud computing, but now we're like switching away because all of compute is moving way, way further away from the data centers. And they're gotten closer and closer to so called edge computing. And so what's really interesting about this is this is going to force us, the developers to deploy and manage our computing resources in a different manner than what we've been used to for a while. And I think this is really going to start driving a lot of things in terms of infrastructure and so forth. And this is why I think this is one of the reasons why I think this matters. So gets to the point of like, why are unique kernel cloud special versus, you know, the stuff that we use day to day in and out right now? If we look at something like OpenStack, this is like the canonical, you know, if you want to roll your own private cloud solution, I mean, OpenStack is probably like where you're starting, right? But you have to think about like what this was built for, you know, is built for VMs, classic VMs that we've been used to for the past 10, 15 years. It's not really built for the VMs of the size and the shape and the change of unique kernels. So I think it's a little bit inappropriate to use something like OpenStack for this. And then, you know, other people will be like, well, we got Docker and so forth. And, you know, Docker just seems to be like the MongoDB of DevOps. And it's honestly completely inappropriate for unique kernel land for lots of reasons, not to mention that it's layers away. But then the next question is like, well, can we use Kubernetes? And, you know, I've mentioned all three of these OpenStack, Docker, Kubernetes, and you can find code on GitHub that interacts with unique kernels with all three of these systems. But once again, it's not appropriate. You know, by the very fact of the matter that pretty much all the implementations for Kubernetes control containers, and they themselves reside already on VMs kind of mixes that fax. Yeah, they're just like stack netties and some other solutions out there. But it's not a great fit. So let's kind of dive into what are we doing and how we're going about doing this. In this slide and also the subtitles, I mentioned L2 through L7 that we'd be traversing. But the reality is, is that, you know, we've personally had to rack around hardware as well. So we're at the physical as well. And there's there's reasons for that, you know, it probably has to deal with this guy. You know, as soon as we found out that some of the big public clouds weren't great solutions for unique kernels, you know, for one, the volume size, minimum volume sizes like a gigabyte, the default one is 10. And so there's all these other things, no, no lack of true L2 networking and so forth. We started hanging down dedicated providers out there. And, you know, I think what we found out was AWS and Google cloud and all those other kind of big guys that really killed the dedicated provider market. So we're like running into network issues, power issues. And we didn't really want to spend like a ton of money each month on our stuff. So we just kind of threw it together ourselves. But I would imagine if you walk down this path, you'll run into the same sort of issues. Because what we're really getting out here is this notion of programmatic hypervisor usage, you know, and why you know, I said it's not just pushing the prod, right? Because we have debugging issues, we have networking stuff, we have volume manipulation, and we have hyperbubble service orchestration. All these are extremely different in unicolonial land than they are in regular VM land. So if you've ever kind of like gone on to the Rump Run Repose or the Garage Repose or wherever, you know, you'll see in the tutorial saying just use Q me, you know, you you plop out your VM, you string it in, you push the right mouse button, you, you know, crank the wheel, you do some jumping jacks and you have it running. But honestly, this isn't programmatic, right? We need something a little bit more than that. Because, you know, we have, we have many different environments, right? We got our dev environment, we got our testing environment, staging prod, you know, any of these, we need it to be automated, repeatable, fast, painless. This all screams it must be software running this stuff. And not to mention that we have different hypervisories to paint on what we're using, you know, Q me, and so forth, virtual box, X, I have and they all have kind of their ups and downs and plus and minuses, depending on what you're doing and what environment you're using. In this talk, we'll, we'll mostly focus on Linux and OS X. But what it really comes down to is this concept of manageability, right? Manageability to me is kind of what really defined modern cloud computing environment. It was that ability to just like, instantly create 100 VMs and then trash them or, or clone them and then push them to somewhere else and have it be sort of reliable. And I, I think that's a, that's a key thing that we, we don't even think about anymore. I mean, like, to, to bitch about when like AWS region goes down for two minutes out of the year, you know, and then compare that to what things were like maybe 10, 15 years ago. But then we get into things like higher level orchestration, right? So, so what is that? Well, there's things like your service discovery. So Zookeeper and SCD and all that sort of stuff, which, which a lot of people, I think a lot of shops don't really need this level of services, but, you know, some do load balancing, which other people are probably more familiar with, like with their reverse proxies on engine X, that sort of thing. And then, you know, your resource aware scheduling, a la mesos and so forth. Once again, this is not something that I would expect most small shops to ever even like look at. But it's, it's something that all these examples that I just showed you work differently in unicornal land. And then deploying. So something as simple as deploying. You think, well, I just like hit the start button and let the VM spin up. And, and in your mind, you're thinking, oh, well, this is how it works. But until you actually do this, you don't realize that in most cases, most compiled languages, you're actually going to be creating a new VM each and every single time you hit that deploy button. And you're going to push that image up to wherever it is. In case of an interpreted language, you maybe don't have to recompile the current each time. But that rolls into rollback, right? So like if you're doing a rollback on a modern VM or with a container system, and you think about how that process works, you're probably typically not trashing that AMI each time. Maybe you are, but I don't think a lot of shops actually do. Well, in unicornal land, you actually are because once again, you're removing that VM from that load balancing pool or wherever it's residing. And so you have to think about how that works, the mechanics of it. Same thing with fellover. So in unicornals, it's pretty typical to have more virtual memory. It's just not really a concept that seems to work in unicornal land. And so if I have an app that just keeps on requesting more and more memory and then all of a sudden it runs out of memory, and it stops responding to requests and so forth, how do you handle that? Do you kill it? Do you switch it out? There's the software that we've written to handle this for today's VM-based orchestrations, that software doesn't quite work in unicornal land because of these reasons. And then you start thinking about things like Lambda and all the serverless and functionalist type of paradigms on top and how they interact with containers and VMs and so forth and how that would interact differently. So at the end of the day, like most unicorals will come in the flavor of VMs. You can provision a bare metal or it's your Raspberry Pi or whatever, but in most cases, most people are going to be working with VMs and so you might have ones that are multi-boot compliant and that's great because then you can do lots of tricks, but not all of them are and so some you'll just get like a straight up old disk image to fly off of. And as I mentioned, we are going to be working primarily with regular hardware, so every CPU in the past, like what, eight, nine years has the extensions necessary, but they're not always turned on. Obviously, if you're on Google or Amazon, you're not gonna want to do hypervisor, hypervisor, you're gonna have to actually burn your own AMI. There is software out there that does that. It's painful not that it doesn't work, it's painful that the big clouds weren't really meant for it. And of course, the hypervisor situation on OSX sucks. Once again, it's one of these things that as developers, we haven't really had to walk down this path before and so there's so much software that's kind of missing that can be written hopefully by people in this audience. Again, change gears real quick. I had the joke about people talking out of their ass on things about the Unicernals earlier, so it's one of the things we're gonna discuss real quick was debugging Unicernals. This is actually a video up on YouTube of us where we were debugging a Go Unicernal and of course we're catching the panics and the stack traces and the GC and everything. Not to mention, GDB, obviously the most, maybe well-known debugger out there, besides MS Visual Studio works perfectly fine. Here's another example of GDB debugging a Go Unicernal. This is actually on Rump Runs, GitHub, Wiki. And the different implementations have their own interfaces, so this is include OSs here, which looks a little bit different now, this is a little bit older, and then one from OSB as well. So the whole notion that they're not debugable is kind of bullshit. Onto the next topic, networking. So once again, working with VMs here, so and in this talk we're focusing on Linux and OS X. On OS X, you don't get tunnels and apps for free, like you do on Linux and so forth, but you can download this 10-tap tar ball here and install it, and then you can see the extensions in there. So if you're gonna do that, that's a great way to get going. Also on OS X, their firewall stuff is using PF control. I think prior to that, they're using IPFW. They seem to switch it up every other release or so, but simple NAT rule that they use over there. And this kind of gets into one of those things of like, using IP advisors in your dev environment today is not the greatest thing, right? Because we have that whole problem of not being able to bridge on wireless devices because you can't pass the frames through. And so we have to do neat little tricks with NAT and so forth to afford our traffic appropriately. So I said we're gonna jump from 01 to LT all the way to L7 and so forth, so it's no joke. This is a common example of what you're going to run into when you're rolling your own unique kernels. You're looking at the SSL locations in the Go code. So we had a Go program that was making outgoing connections, but you're getting like the X509 error, right? And so it was a pretty clear indication that it wasn't finding the cert. So like, okay, well, let's just pop the cert in wherever it needs to be, figure out where that is. And of course, every single distro out there and every single different system not only has a different file name for the cert, but stores it in a completely different location, which is just completely batshit insane that this is even a thing. But it kind of, this is the type of stuff that you normally probably don't have to do when you're on like Ubuntu or whatever, but you do have to kind of think about and deal with in unique kernel land. And of course, you get to play with DNS and so forth as well. DNS mask here is probably gonna be your friend, although most people just kind of like pass that on through whatever gateway they're serving their traffic through. But you can also use this for like DHCP. So if you have like a multi-tenant situation, then you can just like set up DNS mask on that bridge that you're serving DHCP through and go through there. Bring us the volumes. So volumes is actually kind of a large subject because there's so many different things that you can do with them in this environment. And with volumes, we also have file systems, obviously. And so on that note, once again, if we go back to some of these tutorials on the wiki, a lot of them like the point at using immutable file systems, which is great for some things, like if you have a static website and there's absolutely no mutation going on, that's great to use that. But in reality, I can't think of one pod system that is like inherently totally immutable because name any web, popular web framework on any popular language out there. I guarantee you, you have some sort of temp files being read and obviously the database is mutable. You got logs going somewhere, you got uploads doing something. I mean, there's so much crap going on, like maybe some of those can be immutable, but I would say most of them won't be. And so it's something to think about, certain applications that make sense and many applications makes no sense in. And so when you're choosing your file system, you wanna think about read and write and which one just makes sense. Which brings us to the other issue of do you want one volume or do you want multiple volumes? Because once again, this gives you many different things to kind of think about. I talked about how compiled languages usually produce one kernel at the end of the day and you have to do that every time you hit the deploy button. It's not necessarily the case in interpreted languages. So if you're running like Ruby or PHP or JavaScript or something like that, chances are you're not adding in another extension, like another native dependency to the system every single time you deploy. It's probably just deploying new JavaScript code. It's probably just deploying new Ruby code. And if that's the case, you don't necessarily need to redeploy the actual kernel part. Maybe you just deploy a new file system instead, new volume instead. So it's an interesting thing and same thing for configuration. There's lots of cases where people need to do configuration when they're provisioning systems or when they're doing deploys and maths and so forth. And a lot of those things, whether they're Chef or Puppet or Ansible or whatever, they all have like their own little thing going on there. So you can use this trick there as well, which gets us into like base volumes and volume copies. So obviously you probably want persistence for some of these things for backups and so forth. And of course, for live migration and so forth, the volume copies kind of come in handy too. So if I have a database running and I don't really want to take it offline, but I want that copy of it, it's a great way to go about doing it. So on OSX, a lot of these operations can be done with like E2FS or OSX, we use HDIutils probably like the, I think that tool comes on any Mac right away. And so you can do most of those operations there. MakeFS I think it's found in brew, but once again, this is one of those tools that you find in brew that is radically different from the tools on the other BSDs, because all the flags are completely different. I don't know why authors do that, but so yeah, this will help you out with the file systems. Bring us the logs, right? So lots of, at the end of the day, we probably want some standard out, we probably want some standard error. Where do we actually ship these things to? You know, a lot of people choose like the remote syslog of right off the bat. So if it's like your regular web service that tends to work, other people, especially in like a multi-tenant situation, you might just mount another block device and log out to there. And then if you're just kind of like playing around with QMeter or whatever, you can always kind of fall back on regular serial output. And we'll actually kind of show something here. So testing is, if somebody asks me like what's the worst thing in unicolonial land today and like what's the most needed improvement, I would say testing like is a complete, it sucks. I'll be honest. And just to show you how crazy this actually is, we had a problem with the Go fork of, so we maintained the Go Rump Run fork and basically garbage collection wasn't happening for a particular, it wasn't happening. So we had a batch come in, it fixed it and we're like, well we should make a regression test to make sure that it actually is fixed, right? And so this test actually comes after it's done compiling the fork of Go, which takes a little bit, it goes ahead and compiles this other Go program as a unicolonial, it boots it up into QMU and then we purposely try to allocate like a pretty large object that we know is going to guarantee to run garbage collection. We output the stats, we screen scrape it and then we make sure that our test was there. And so that's like one test and it's such a simple thing, but yet all the operations that had to do for that. So it kind of shows you like the level of testing is at in the ecosystem and like how much room for improvement there is. So workflows, as I alluded to, can be pretty different and allow you to think differently as well, which I think is important. Configuration management, for instance, right? So one of the things that people will run to immediately is unicolonials are a single process in most all cases by design. That's a security feature, not a bug. So single process basically means SSH is out the window, right? Once again, a security feature. But how do you config systems? Well, there's lots of different things. A lot of people will just be like, well, we're just gonna bake it into the kernel and what you do is what you get. But you don't necessarily have to do this and I think this little tin line, this is literally like 10 lines of PHP or something. And it's not necessarily supposed to be production or anything, it's just to show you that if you think outside of the box, you can easily do the exact same thing a thousand different ways. And here you can kind of see like editing the host's file through like a tin line. Something with volume deploy workflows. So this is a really interesting thing that once again really only works with unicolonials. You would never do this with a big VM because it just does not make any sense. But if I wanna spend up like a hundred unicolonials and then like push out changes to them without spending them down, this change might work. And so basically the trick here is as you can kind of see in this little pre-vote concept thing. So you have this little PHP app says test and then it includes like your other code on a different partition and says test, right? And so obviously it's not there. But if we just create our new file and so forth and then we create a new file system off of that and then go ahead and mount that, we obviously had it. So what we're looking at here is deploys that are only containing the code because once again this is an interpreted language and so we don't actually have to recompile the actual kernel itself. We're just mounting a new partition and boom. And so this is like one of those examples like you would never ever do in a regular VM. It's just kind of stupid. But the question is like, what will you build? So I was gonna show you this tool real quick. Once again, I'm not sure if let me unfold screen this. So we'll show this and is this gonna work? Yeah. All right, so basically this is a tool I told you you can download. This will allow you to run the new curls locally on OSX and also Linux. And so it looks and feels a lot like Docker for instance. But we have like this HTML one. And I gotta drag this window as well because I apparently am not so great with the projector. But basically, we have a HTML example and then we can run like a PHP example doing the same thing. All right, so this one is at four dot four dot six dot. Gotcha, you can also search for other ones. You can kind of see that other people have been kind of playing around with them. Once again, this doesn't do any compilation or anything. It just runs them locally. But I hope it gives you ideas on how to kind of improve certain workflows and certain tooling around the ecosystem. Anyways, questions? Thanks for the talk. Actually, lots of problems you were describing are actually solved by Kubernetes and Mesos today. They are not scheduling or managing unique nodes but they're managing Docker containers. Why not build on these primitives to like have the management layer and just get a unicorns on a bare metal provider like packet for example. So you mentioned Mesos? Yeah, Mesos but more Kubernetes. I know there's Dell EMC, there's a unique. Yeah, so yeah, good question. So I really glossed over that but there's quite a few problems. Mesos is a great example because they kind of, they were working on this before kind of the whole container hoopla. Kubernetes is not a great example. Unfortunately, a lot of like Kubernetes installations are already provisioned on top of VMs to begin with and so having two layers of provisioning like that don't really work too well. You really need the orchestration layer at the hypervisor level, not on top of that VM. If you do, it just needs to be kind of like out of band and so forth. So yeah, unique for instance, out of VMC, they do have integration with Kubernetes but once again, like trying to do that on Google Cloud, it's gonna suck. Trying to do that anywhere else is gonna suck. You really need the hardware acceleration and in order to do that scheduling that needs to be kind of built at a lower layer. And as for like Mesos, that's another interesting thing because that's where scheduling layers kind of compete. So Mesos is more for that process level, right? Because now you can subdivide those resources. So like that scheduling part where I showed you but when you have that VM, that scheduling has effectively moved down into the hypervisor and it's probably not like the best place for it. At least we don't have software that like really takes advantage of it very well yet but Mesos, that much works a lot better up here on top of that VM layer. Yeah, so like packet.net for instance, they do a lot of bare metal provisioning and so forth. So yeah, that's what I'm saying though is there's definitely frameworks out there like OpenStack for instance, obviously allows you to do that. But those frameworks are built for kind of classic VMs whereas some of the include OS for instance, you run some of their demos and it's a megabyte tops. So that's a very different VM than a gigabyte image. I have to say that I don't have experience with this but I have experience on the old type of computing word. How far are we from a production system? Yeah, so I mean there's production users today. You know I talked about not talking about IoT and not talking about NFV but NFV is probably where like the most production stuff is running today. Big telcos are starting to utilize this. Basically a lot of the telcos old hardware stuff is being eaten alive by software and so like a lot of the NFV stuff is actually utilizing Unicurals. Erickson in particular I know is using that stuff. On the website, there's definitely some people that are twirling around with it. There's a lot of twirling yet to be written though, so.