 All right, I guess we'll go ahead and get started. I'm Flynn. I'm a tech evangelist with Buoyant. I mostly work on the open source side of Lincordie. In a previous life, I was the original author of the Emissary Ingress Gateway, but now I work in marketing. So everybody is filled the room to listen to a marketer talk about a programming language. I am gonna talk about why Rust is the future of cloud native, but to do that, I'm gonna start in the past, in 1996 at the Operating Systems Design and Implementation Conference in Seattle, where in the vendors hall at one point, I ended up having a conversation for about half an hour with a guy with white hair and a white beard and a name tag that said Dennis Richie. If you are not familiar with this guy, and many of you may not be, he was the inventor of the C programming language. He co-invented UNIX, which means that he pretty much kick started what we're doing here in this industry today. And in the process, he brought us decades of buffer overruns and security violations and CVEs and countless hours of dollars lost ripping our collective hair out, trying to figure out what was going on with our programs. That's honestly the biggest thing I remember about that conversation. He was very charming. He was clearly brilliant, nice guy, great conversationalist. And it was very, very difficult to reconcile in my head all the amazing stuff he'd done and all the incredible pain that had gone along with it. So this talk is pretty much about how we do better and about why that's important. I'm starting with C because it has been massively influential on all the tools that we use today. And that influence should not be understated. But it was written for a very different time. When it was written in 1972, the industry believed that programmers could write code that was correct and that they would not write code that was malicious. And we know that both of these things are false at this point. That we as people working in programming language will make every single mistake the language will allow us to make eventually. And we know that there are some people who are going to write malicious code. That was not the world that C was written in. So the end result is that C and C++ are fast, but they have no safety in them. Some of you may be thinking, oh, C++ is safer than C. No, it is not. And we'll come back to that. I will fight you. There are a bunch of other languages that make different trade-offs, right? We have this interpreted languages that tend to be fairly safe, but usually quite slow. Go is kind of an interesting place, all its own, where it's fairly safe and pretty fast. But it's interesting to note that at this point, the only choice we have for both very safe and very fast is REST. To understand why we need to talk about pointers, we need to talk about ownership, we need to talk about lifetimes. And we also need to talk a little bit about static analysis versus dynamic analysis. Python down there I show is less safe than its interpreted siblings because of its type system. Everything in Python is dynamically typed. Those of you who have worked with me in the past know that I'm quite fond of Python for lots of things. But the fact of the matter is that the dynamic type system means that it is not possible to read Python source code and know for certain that it is going to work. You have to actually run it to find out if it's gonna work. And you have to put in a lot of effort to try to make sure that you follow all of the code paths so that you can know that they will all work. This is an example of the kind of difference where static analysis does a better job because static analysis, which will not work for things like Python type checking, but static analysis lets you look at the entirety of the source code all at once and know for certain that you've covered everything which dynamic typing cannot do. Dynamic analysis cannot do. So that's our first axiom for safety. Static things are better than dynamic things. You would much rather have the compiler check everything for correctness than have to follow every path and trust that you got that right. That brings us to the pointer. Or as I like to call it, the root of all evil. Conceptually pointers are not all that evil. They're just an implementation of the concept of a reference where you have some chunk of data and you want a way to access it that does not involve referring specifically to that thing. You want something that's a little bit indirect. That's a reference. These things underlie pretty much any data structure more advanced than an array. We use them all over the place. They're very important. Pointers are the simplest implementation of references where you go, oh, hey, my datum foo lives at this particular address and you take that address like the numerical value and you say, great, that's my pointer. That's how I refer to foo. This is very simple. It's very powerful and it is extremely dangerous. Usually people start off talking about the dangling pointer problem for why pointers are dangerous. Suppose we have to move foo for some reason. It changes its address when we move it but the pointer didn't get updated and then when you try to follow the pointer, bad things happen. This is the least bad thing that can happen with pointers. There's actually such a thing as a dangling reference not involving pointers at all but this is a thing that happens a lot. The worst thing that happens with pointers immediately, you run across with the languages like C and C++ where we can do pointer arithmetic where we get to alter the number, the numeric value of the pointer. The whole concept behind pointer arithmetic is that we take advantage of the fact that memory is laid out in this linear way and then if we have a pointer pointing to one of these values, then you figure, oh, we should just be able to go and add to that pointer itself, not the value it points to, add to the pointer and then it will go point to something different and you can do a bunch of things very quickly with this and it has a couple of huge problems. The first one is that your program can scribble all over memory. As soon as you get to set the value of the address in the pointer, everything the runtime can do to protect you from yourself goes out the window because this is an intentional thing that was added to the language. You are supposed to be able to go and modify things at arbitrary addresses in memory and that means that you can make mistakes and stomp on stack frames and memory management metadata and other executable code sometimes and all this horrible stuff. A slightly more subtle thing is also as soon as you can do this, the type system goes out the window as well because even if our datum there is an integer and we have a pointer that is typed as an integer pointer, there's nothing preventing us from creating another pointer that has the same numeric value but a different type and then when you go through that second pointer, you've just changed the type of your datum as far as the runtime is concerned. Again, this is not something it can protect against because this is designed into the language. This is also why I say that C++ is not any safer than C. C was built to do this particular operation because when you're writing operating systems, which is the thing that C was designed to do, when you're writing operating systems, you often need to be able to go and manipulate data structures at specific places in memory and C++ decided it wanted to be a superset of C and it had to do everything that C would allow you. So even though C++ has smart pointers and stuff, the language by design allows you to go and always use raw pointers everywhere so you can bypass any safety thing that C++ does and the language has to allow you to do this. Another thing that's a little bit interesting about pointers is the concept of null pointers. These exist in languages beyond C. Go has null pointers, for example. A null pointer is just an explicit pointer to nowhere so that you can explicitly say that this pointer doesn't point to anything. On the face of it, this seems like a pretty reasonable thing. The problem is first, it's another thing you have to check for at runtime and I'm sure none of us new programmers would ever forget to check null pointers at runtime. Certainly I've never forgotten that check. But the other thing that's a little more subtle about this is that null is a value, it's not a type. You can't do static analysis to make sure that you never do reference a null pointer. It's just not a thing that can happen. So it is much better to use a different system so that you can do static checking for it. I am realizing right now that I forgot to mention a particular feature in Rust that makes this particular thing go away. So I'll try to remember to mention that when I get to it. Sorry, forgot to put it in the slides. So yeah, since you can't do static checking for null, you have to actually run the program and find out if you hit any null pointer errors. Not great, we're back to that first axiom where dynamic is not as safe as static. Another interesting bit is that null gets worse in languages like C and C++ where you can only return one value at a time. If you do things like this in go and in Rust, you can just return multiple values, which is great. If you wanna do the same thing in C, you have to introduce a structure and then allocate the structure and put stuff in the thing that you have allocated and then return a pointer to the thing that you have just allocated. And everybody has to check for null and it can be really bad. The fine print down there is that yes, technically you can return structures in C. If you actually do it, it ends up abusing the stack frame if you're talking about large things and so people don't do that. They tend to allocate and use pointers. So yeah, not great. C++ also has the standard tuple type but guess what, that's allocating stuff under the hood. Ultimately, that kind of thing gives you more chances to make mistakes. It gives you more ways that you can make mistakes. And like I pointed out earlier, we now know that eventually when writing programs, we will make every mistake that the language allows us to make. So introducing more ways that the language allows us to make mistakes is not a great idea. Second axiom with apologies to Richard Bach from whom I stole this line sort of. In order to code freely and easily, you must first give up pointers. I should probably also point out, yes, those of you who are already Rustations are probably thinking, but wait, Rust has pointers. No, Rust's normal pointers are actual references and unsafe pointers are unsafe so we don't have to talk about them right now. Might come back to that if we have time. Let's talk a little bit about ownership. Ownership is a little bit of an abstract sort of thing. It refers to who is, it really refers to who is responsible for dealing with a particular piece of data. And in languages where you have to do dynamic, or have to do manual memory management, excuse me, say that three times fast, then ownership is often described as who has to free the thing when you're done with it? A better way to look at it is who gets to decide you're done with it? And this is implicit in almost all of the languages that we use. I don't think I know of a language other than Rust that talks explicitly about ownership. Objective C kind of weasels in on it and stuff like that. Some of the automatic reference check, offer automatic reference counting stuff touches on it, but I don't know of a language other than Rust that explicitly talks about ownership and lifetimes. Ownership is explicit in Rust. The compiler checks it. The compiler makes certain that you have not gotten it wrong. This is a quote from the Wikipedia page on Rust. Every value must be attached to a variable called its owner. Every value must have exactly one owner. We're gonna come back to that in a minute because this was one of the things that caused me a fair amount of pain while messing with some Rust code not too long. Well, I guess it wasn't that long ago. But the other bit is that Rust is explicit about when ownership gets transferred, both by assignment, but also by making a function call. So if we do this in Go, this program will run fine and it will print hello world twice. If we do exactly the same thing in Rust, it will not compile. And the reason is that when you call printer S on the line above the compilation error, that transfers ownership away from main and once main doesn't own that S variable anymore, you don't get to do anything else with it. So the compiler says, nope, you can't do that. Cut it out. The compiler errors, I'd like to take a moment as well to point out because the compiler errors in Rust are actually useful. This one going, hey, look, you have made a mistake around ownership and this is what you might do about it and here's where to go for more information. I'm a huge fan of this. More languages should really do this. It turns out to be really hard, which is why more languages don't do it. The easiest way to solve this problem is instead of passing the string itself, pass a reference to the string and then ownership doesn't transfer and then you can do it twice and everything is great. I should point out that the compiler is still checking to make sure that you haven't done horrible things with the references, which we'll get to shortly. And the compiler also is checking to make sure that you don't try to use an immutable reference to modify something because it's immutable. So there's another large class of errors where the compiler will protect you from making these mistakes. Here's a more subtle example that deals with ownership. Oh man, I missed an opportunity. I should have left off the explanation of what was wrong and asked you to guess what it was. If you run this Go code, it will say found one, found two, found three, all the way up to found five and then it will print the array at the end and it will go one through six instead of one through five because this Go code modifies our array while we're in the middle of iterating over it. Basically, everybody knows that this is a bad idea and you shouldn't do this. But even though it's a bad idea and you shouldn't do it, the Go compiler will not protect you from that error because it can't. The REST compiler will. The REST compiler on that line that says error will come back and say no. Calling numbers.push here requires a mutable reference and the for loop already grabbed an immutable reference and you don't get to do those two things at the same time. So you have to go through and restructure this code instead of just falling quietly into this trap. And again, I want to emphasize this is a compile time check. If you manage to somehow have a mutable reference in one function and then try to use it as an immutable reference in another function then the compiler will still find it. It actually turns out to be fairly difficult to write a concise example of the compiler doing that across multiple functions. So I didn't do it. Third axiom, which is a thing that REST kind of does a lot of, is requiring you to be explicit. Making things explicit is harder up front. You have to go to more work. It will cost you more effort, but then it will pay dividends for the rest of your life. And possibly more importantly, it will pay dividends for the rest of your code's life. It would be nice if we outlived all our code, but who knows? All right, let's talk a little bit about lifetimes. This is another concept that's implicit in a lot of languages but explicit in REST. This go code will actually run. Even though up there in the create pointer function, we are allocating and creating a local variable and then we are returning a reference to that local variable before the function returns. So we've now returned a reference to a thing that has not really, it shouldn't really exist at that point. But that works okay in Go because Go has a garbage collector and the garbage collector will recognize that there's an outstanding reference to our local variable V. And so the garbage collector will keep it around just in case you're still using that reference. If we try the same thing in C, it will compile but it will not work. Bonus points, anybody have guesses for what this will actually do if you run it? I guess I should probably be happy that nobody's been doing enough C that they want a hazard to guess here. I was expecting this to crash when I ran it actually. It turns out it doesn't crash. It just prints 42 the first time and then prints garbage the second call. What's actually happening here is that when you call create pointer, then a stack frame gets allocated that includes space for V. It then returns a pointer to that location on the stack which is not active anymore in the program but the memory is still there and unless you get really, really unlucky and cross to page boundary at exactly the same time that the operating system wanted to release that page, you'll still be able to access it. But then the first call to printf will copy the 42 out of the pointer, push it onto the stack frame for the printf call in the process smashing the original value of V and then the second printf will dereference the pointer to the same spot on the stack that V used to be but now that's just some garbage from the call to printf the first time. Isn't it fun when we get to deal with pointers? Don't answer that. I don't want to know if you think it's fun. Rust will detect that this sort of thing is wrong and it will refuse to compile because it will come back and demand that you just don't get to do that. You have to explicitly mark the lifetime of your variable so that Rust knows what to do with it. In the second example here, we've used that static, there we go. We've used that static lifetime marker to tell the compiler, hey, create value is gonna return something that is required to live for the entire life of the program and so this is safe, this will work. Here's a much more subtle example of a lifetime bug. This one from C++. When I run this on my machine, it compiles but it does not say hello world, it just says hello. Well, okay, it says hello comma. Again, anybody want to raise a hand? Hazard to guess for what's actually going on here? Fear of commitment. So what's actually happening here is that here's where the error is. That may kill a world function. By the time it returns, the world string goes out of scope and is destroyed and it so happens that at least with my C++ compiler when it gets destroyed, the compiler is at least polite enough to go and stick a zero in the first byte and then it ends up not crashing but that's kind of implementation specific. That's not a thing you can rely on. If you try this in Rust, it will not compile. It will come back and say, no, no, no, you have to be explicit about the lifetime of this string in your greeter structure and so to make it work correct, we have to sprinkle these lifetime specifiers around so that the compiler knows how long the thing has to live and so that it knows that it can actually rely on that thing coming back. This example will compile and run and correctly say hello world. This is probably as good a time as any to mention the thing in Rust that I forgot or to put on the slide, which is instead of null pointers, Rust has a concept called an option that can have either a value that is specified as some with a real value or none and so the compiler can keep track of that because it's a type and so if you get an option back, you are required to do the right things to make sure that it will work properly rather than just hopefully forgetting or hopefully not forgetting to check for null and again, all of these things get checked at compile time across the entire program. On the one hand, this is great because it prevents you as a programmer from making a mistake but on the other hand, it's great because it means the runtime gets to go very, very fast because the compiler has already done all the hard work. This is a good thing. It also means that Rust gets to get rid of both null pointers and the garbage collector, which is pretty cool because as soon as you get rid of the garbage collector, you do not have to deal with unpredictable delays during which the garbage collector is collecting the garbage. It gets to just run quickly and predictably. There's another axiom there. If you make lifetimes explicit in particular, you get to get rid of pointers and you get to get rid of the garbage collector. So all of this is why I stick Rust alone by itself up in the very fast and very safe category where we don't have null pointers, we don't have manual memory management, we don't have dangling pointers, we don't have garbage collector and predictability, we don't have data races. I didn't really talk about that very much but the Rust compiler can detect when you are trying to do things like modify the same piece of data from multiple asynchronous things at the same time and then refuse to let you do that, which is amazing. There is absolutely a learning curve for all this though. All those things that I was talking about that you have to be explicit about in Rust that you don't have to be explicit about in other languages. Yeah, you have to learn how to do that. It takes some doing. It's a little weird. Actually, it's a lot weird in the beginning but you know, it's okay. The learning curve is worth it. In particular for the cloud native world, at this point I think it's fair to say that the ecosystem is ready for the things that we're doing with cloud native. There are, I called these out as crates, you know, Rust packages that get used a lot in the cloud native world that are really, really helpful. Tokyo for asynchronous hyper. Somebody should donate a logo to the hyper project by the way. Hyper is a very fast and correct HTTP library. There are other things built on top of it that are probably worth looking into if you want a very user friendly HTTP library but all of these work really nicely together to give you really, you know, pretty elegant asynchronous network protocol and Kubernetes stuff. KuberT is the main set of Kubernetes bindings for Rust. KuberT is a thing that one of the LincorD maintainers wrote that sort of a more opinionated controller runtime based on KuberT. So KuberT is the thing that the LincorD Rust controllers are based on. All of these things have made dramatic leaps and bounds since we started at LincorD in 2018. Our first Rust controller didn't come along until 2022 with, you know, more support for Kubernetes bindings and such. Anybody starting today with Rust, congratulations. You get to benefit from our suffering at Voyant. Enjoy, because, man, there was a lot of suffering back then. Earlier today, my colleague Matei actually gave a talk about the specifics of writing Kubernetes controllers in Rust. If you were not able to make that one, I would encourage you to go and check it out on video if you're interested in this sort of thing. If you're in this room, you're probably interested in this sort of thing, so. Finally, one more thing to wrap all this up. Backing up again to the mid-90s, I was at, I think it was the Usnick Security Conference. I ran across a talk by a guy named Marcus Rainham who did securityforwhitehouse.gov in the beginning. And he pointed out that in the 1950s in the United States, we had cars that could do 100 miles per hour, 160 kilometers per hour. But we didn't mandate seat belts until the 1980s, 30 years later, which means that there was an entire generation of human beings who lived just believing that, well, yeah, you get into an accident and you die. That's just the way it goes. That was accepted. That was a normal thing. We've been writing unsafe code with unsafe tools now for more than 50 years. C came out in 1972. That's a generation and a half of humans. I would be willing to bet that it spans more than the career of everyone in this room. And for all that time, it's just been accepted that, yeah, eventually we make mistakes and get horrible security problems and that's just the way of the world. This is not okay. We do not write software that does nothing. We don't write software just for the sake of writing software. We have a purpose in mind for it. A lot of the infrastructure that we write in this world is handling things that are very critical. Picking on LinkerD for a minute, LinkerD gets used in emergency dispatch call centers in the US and probably elsewhere. We get used in healthcare organizations where patient data is wrangling around and at risk. We get used in financial organizations where your money is at risk. All of the stuff that we're doing here, people rely on it. It has to be safe, it has to be reliable, it has to be fast and that order is deliberate. Safety first. No matter what we do, writing safe software is really hard so we should be using the tools that make it the most easy it can be even though that's not gonna be very easy. There's no point in making it harder than it has to be. In other words, if you wanna remember just one thing from this talk, it's that it's time to put on your seatbelt. Throw away this unsafe crap and write in rest. Thank you. I think we have a couple minutes for questions if anybody has any or I'll be up after. Do we have a mic for questions in the audience? Thank you. So I really enjoyed this marketing for Rust. It was really fair to say that there's a fairly high learning curve. As a Go developer who has now spent quite some time with Rust, there's another thing. What do you think about compile time? So the time it needs to compile your Rust program. Controversial opinion, possibly. I don't care. And I'm being slightly snide there, but only slightly. On the one hand, I've worked with projects that literally took more than 24 hours to compile. So basically everything we do in this world is great but what I also find is that Rust seems to do a pretty good job for me personally, at least, of not rebuilding the world and re-downloading the internet and things like that. So when I'm doing development in Rust and trying to iterate fairly quickly, it seems to work out okay. And if I'm talking about I'm gonna go and release something and I'm running it in CI, whatever. I don't really care all that much. So I guess the short answer is that in my experience, it has been good enough. Not perfect, but good enough. Does that answer the question? Sure. Yeah. When Go does not have to download the internet, I think Go is faster, probably, but it's not enough to make a big difference for me. Also, Go downloads the internet a lot. It's really annoying. I think there's somebody over here. Yeah, hi. Yeah, thank you on the refresher for pointers and their challenges. I mean, there are already like hundreds of projects in the CNCF landscape. Are you aware of any project that in hindsight would have go better with Rust or other way around? Are there any projects where you would consider a rewrite in Rust? Maybe not you personally, but. Rewrites are hard. So let me take those two questions separately. Are there any projects that in hindsight might have thought Rust was better? There's actually one I know of right now, BlixT, which is, I believe, going to be rewritten in Rust. So what Shane says is that they originally wrote the data plane in C, oh dear God, and then rewrote that in Rust and then had the control plane in Go and now the whole thing is gonna be in Rust. I, congratulations, Shane. That's literally the first cloud native infrastructure project I've heard of using C. That's, wow. LinkerD also, LinkerD version one was written in Scala for the most part, and then LinkerD version two, the data planes in Rust, the control plane had Go and the control plane is making its way to Rust as well. LinkerD did this by taking pieces that needed to be created new and doing them in Rust rather than going back and rewriting the world all the time. So the second part of that would be rewrites. Rewrites are hard, rewrites are expensive. Rewrites mostly are a question of trading an existing known set of bugs for a different set of bugs that might be unknown. And so I would tend to caution people about rewrites in general, but yeah, maybe there's a way you can start doing new work in Rust and do the migration that way. Anyone else? Going once, going twice, deliberately ignoring Shane, trying to cause trouble? All right, thank you very much. I appreciate it.