 Thank you so much. So hi, this is five years of Cloud Native Rust. So this talk was originally prepared and was gonna be given by Eliza Westman. She is one of the Lincority maintainers. Maintainer's the Lincority project who works at Boyant. She's a Rust enthusiast and very, very cool person but unfortunately she couldn't be here for illness so I have taken over. My name is Alex. I'm also a Lincority maintainer. I also work at Boyant. I'm also a Rust enthusiast. No comment on if I'm cool. But what I wanna talk about today is kind of, I wanna give some background on kind of how the last five years have gone for the Lincority project with respect to us adopting Rust. So we decided about five years ago to adopt Rust as the language to write the Lincority proxy in and that was a very kind of controversial or big decision at the time. So I wanna talk about why we did that and talk about how that went and kind of what the outcomes of that were beat would have been. And I wanna talk about why that kind of matters to people who are not Lincority maintainers and what it means kind of for the ecosystem in general. So number one, why did we decide to use Rust in the first place? So first of all, I've been talking about being a Lincority maintainer. Lincority is the open source service mesh part of the CNCF. Is everyone familiar with Lincority? Have people heard of this? A few hands go up, yeah. So Lincority is a service mesh after having been at KubeCon for a few years now. Very good at explaining what a service mesh is very quickly to people whether they like it or not. So the idea here is that you have a proxy that acts as a sidecar container. You put one of those in every pod in your cluster and then all of the network traffic for that pod that's going in or out goes through that proxy. And so because that proxy can detect what protocol it is, it can see whether it's HGP or TCP and it can do all kinds of intelligent things with it like it can transparently add mutually authenticated TLS onto that connection without the application needing to kind of do any work to make that happen. It can do intelligent latency aware load balancing based on the actual observed latencies that it sees for real request data. It can add observability, circle breakers, all kinds of really cool stuff. And so Lincority is infrastructure and there's two different things I mean by this. Number one, Lincority is network infrastructure for your application. So anytime your application makes any kind of network call to anything else in the cluster, it's gonna go through Lincority kind of transparently. And so just like the operating system is infrastructure or your network card is infrastructure, Lincority is infrastructure to your application. But Lincority is also used by infrastructure for humans. So 911 call centers use Lincority, healthcare organizations use Lincority, financial institutions use Lincority. These are all very critical pieces of infrastructure that if they work or they don't work, this is a big deal, right? It's not just like, if it's low, who cares? No, these are very, very important applications. And people rely on these pieces of infrastructure so they need to be secure, they need to be reliable and they need to be fast in that order and it's not negotiable. We cannot compromise on any of these three things. So when we're talking about Lincority being secure, what are the types of things we're worried about? What types of things cause security vulnerabilities? Well, for example, there's buffer underruns, if you try to access a piece of memory outside the valid bounds for that memory, it's gonna be a problem. There's things like dangling pointers or use after free, where if you try to dereference a pointer after that pointer is no longer valid, it's gonna be a problem. And reading uninitialized memory, if you try to read a piece of memory before it's valid, also a problem. So these are all examples of memory on safety, trying to use a piece of data but without, while assuming it's one thing, when it's actually something else. So it turns out that a surprising majority of security vulnerabilities are due to memory on safety and 90% of Android vulnerabilities, 60 to 70% of iOS and Mac OS CVs, 70% of CVs and Chromium all do to memory on safety in one form or another. And so if you wanna avoid these kind of memory on safety issues, well, one option is to go to a managed language. So if you're using something like Scala or Java or Go, C sharp, Ruby Python, these all do run time checking to make sure that you're using memory appropriately. You're not accessing things outside the valid bounds, you're not using pointers that, oh, that's not good. All right, we're okay. Yeah, I should have written it in Rust. So all these languages, they have run time checks that make sure that to protect you against these kind of issues, but those run time checks come with a cost. There's a performance cost associated with doing them. And so sometimes you don't typically get the same performance in these languages that you might get in something like C++, which is very, very fast, but is susceptible to these kind of memory issues and it's very easy for these kind of memory related bugs to slip into your code. So when we're talking about Linkerty's data plan, this is the proxy where all of your network traffic is flowing through, memory on safety, in this case is completely unacceptable, right? That data that's going through the proxy might contain personally identifiable information, credit cards, health care information, financial records, you know, any kind of sensitive data. And that's not something that we can take any chances with. But if we're using a managed language and we're subject to things like garbage collection pauses, for example, that's not great either because all of your application data is going through that proxy. If that proxy, you know, does a garbage collection pause and waits for 100 milliseconds, 200 milliseconds, that's slowing down your entire application. You know, that's in the hot path. So we really want something that can give us memory safety, but at the same time, like real-time latencies. And so what if there was something that could give us kind of the best of both worlds? Ta-da. So Rust is a system programming language. And the big idea behind Rust is that it gives us these memory safety guarantees at compile time. So the compiler is able to statically check your program and is able to look for all kinds of memory issues at compile time, so it doesn't have to do these checks at runtime and can guarantee that the code that it produces is free from large classes of bugs. So how does it do this? So one thing that Rust does is it ensures unique ownership. And that means that if you have a piece of data in a Rust program, there is one place that that data is owned. You don't have two pieces of code that both think they own a piece of data and one tries to delete it or they try to modify it at the same time. The language itself guarantees unique ownership. It also has this concept of checked boroughs. So if you're trying to access a piece of data that you don't own, there's a system in place for deciding when that is okay, when it's not okay, and when you can use this data and make sure that it's still valid when you're trying to access it. And it has this concept of exclusive mutability. If you wanna change a piece of data, Rust ensures at compile time that you're the only person or the only piece of code is looking at that. So you don't have to worry about pieces of data being changed out from under you or being mutated at a time that you don't expect. This is a little bit abstract. Let's take a look at an example. Is that visible? Yeah, it's pretty visible. Okay, so this is a program written in C++. We have a vector v, so that's a global array of strings. And what we do is we push a string into that vector, hello world. Then we create a pointer, which is our pointer to the first element in that vector. We print it out, and so hopefully that should print hello world. Then we push a whole bunch more strings in there, hello Chicago, hello KubeCon, hello San Francisco, et cetera, et cetera, et cetera. And then we print out that first pointer again that's supposed to be pointing to the first element of that vector. Right, so pretty simple program hopefully, but let's try to run it and see what happens. SegFall, core dump, it crashes. So that's surprising, right? Unless you have an eagle eye, you might not have expected this program to have any problems. So what is happening here? Well v is a vector, it's a growable array. And what that means is that it's backed by a static length array, a fixed length array. And if that ever grows past the bounds of that array, it's gonna allocate a larger array somewhere else in memory, and it's gonna copy its contents into that larger array. So when we just had one element hello world, it was fine, we took a pointer to that first element, but then as we pushed more and more data into it, it eventually got big enough that it needed to be reallocated somewhere else. And so it moved that data to a larger array somewhere else in memory, but that pointer was still pointing not at v anymore, but at where v used to be. And so that's why we get this SegFall, because we're trying to access this piece of memory which is no longer valid. This array has moved out from under us. So not great, and kind of a subtle bug. So how does Rust help us here? Well let's look at what this program would look like if it were written in Rust. So basically the exact same program. We have a vector called v, we push hello world into it, we take a pointer to the first element in the vector, we print it out, push a whole bunch more strings in, and then print that pointer again. So almost identical program. Who thinks if I run this it's gonna work? No one. Who thinks it's not gonna work? A few people. You're half right, it doesn't even compile. The compiler's gonna tell us that this is not a valid program. And so why is that? What's it saying? It says cannot borrow v as mutable because it is already borrowed, or is also borrowed as immutable. What does that mean? It means that when we took a pointer to that first element of v, we were creating a reference into it. We were saying we would like to have access to this data. And then we start mutating that vector. So this is that concept of exclusive mutability. Rust guarantees that if you are going to mutate someone, something, no one else is looking at it at the same time. And so the compiler has caught here that we're trying to change a piece of data while someone else is looking at it. And that's not allowed. So how do we fix that? Well, it's very simple. If we just create a new pointer down here at the bottom, that's going to solve our problems. And the reason for that is that that first pointer is now used only once. It's used for that first print line. And then it's never used again. So it can be dropped there. And the compiler is smart enough to notice that it's never used again. And can say, okay, well now these edits, now these v.pushes are okay because no one else is using that pointer. So we have exclusive access to it. We can do that mutation. And then when we're done, we can create a new pointer and we can look at that data and it's gonna work. And if we run the program now, it works. Hello world, hello world. So very cool little way that Rust can protect you from these subtle bugs where things kind of change out from under you. Let's look at one more example. Okay, so here's another C++ program. We have a class called Greeter. It has a reference to a string in it called who, there's a constructor, and then a greet method which says hello and then the contents of that who reference. We have a function called make hello world which creates a string called world and then passes that reference into a greeter and returns it. And then we have a main method which calls make hello world and then greeter.greet. Okay, does anyone see the bug? Is anyone eagle-eyed enough to notice what's going wrong? A few people, okay. Let's try running it. Let's see what happens. So this does print hello, but it doesn't print hello world. It prints hello something else, something bad. It says hello to something very, very different. So what's going on here? Well, when you call string world equals world, you're allocating a string on the stack, a local stack variable. And then you're taking a reference, uh-oh. Sorry. Where did my slides go? So many tabs, I know. We have a local stack variable called world and we're taking a reference to that stack variable and we're passing it into greeter and returning it. And, where am I? Here we are. And then when we return from make hello world, all those local stack variables get deallocated. And so now you have this pointer to the stack variable that no longer exists. And so that's why we get kind of this garbage printed out because we have a pointer which has outlived its valid lifetime. That pointer was kind of only valid for as long as that stack was, as long as we were in that stack frame. And once we pop that off, that reference is no longer valid, but we still have it around because it's kind of been returned and we're still gonna use it even though it's not valid anymore. And that's why we have this issue. So, how does Rust help us? Well, here's the same program in Rust. We have a struct called greeter again. It's got a reference to a string. It's got a greet method, which prints hello and then the who. We have a make hello world, which creates a string called world and passes a reference to it into greeter. And then we have a main which calls make hello world and then greeter.greet. Who thinks it's gonna work? Who thinks it's not gonna work? Who thinks it's not even gonna compile? A few people, you're right, it does not compile. But the compiler error here is kind of interesting. It says missing lifetime specifier. And so Rust has this concept of lifetimes, where if you have a reference in Rust, you can also specify a lifetime that that reference is valid for. And so that lets you make sure that if you're ever going to use reference, you're using it at a time when that reference is valid. And if you try to use that reference at a time when it's not valid, it's not gonna work. The other cool thing here is that the compiler has not only told us that there's a problem, it's also given us a suggestion on how to fix it. It says if you add this little tick A annotation to your reference, that's gonna work. So it's helped consider introducing a named lifetime parameter. So let's try doing what the compiler tells us. And now we've added these little tick A references. A can be anything, it's just a name. But it's a name that specifies the lifetime of this reference. And so now Greeter is parameterized on that A. So it says this is not just any Greeter, this is specifically a Greeter whose reference lives for some amount of time. And this is actually gonna work here because the Rust compiler is also able to tell that world here is a static string. And so it doesn't need to be allocated on the stack. Locally in that method, it can just be allocated in static memory. And therefore the lifetime of that reference is the entire lifetime of that program. And so since it's able to infer that, all of these lifetimes are gonna work out. And ta-da, it's gonna work. So this is a little bit of a trivial example because the static lifetime is just the whole lifetime of the program. But you can have much more sophisticated uses of lifetimes in Rust where you have references that are valid for a certain amount of time. And when they go away, they're not valid anymore. But because these lifetime parameters are passed through, you can guarantee that you're never going to use a reference outside of its lifetime. Hooray. So that's kind of some of the benefits of Rust and some of the problems that it can help protect you against. I wanna talk now kind of about how it's been for the Linkerty project, specifically adopting Rust way back in 2018. So when we were first working on Linkerty 2.0, we did a big rewrite of Linkerty from Linkerty 1.0 to Linkerty 2.0. This is where we transitioned from writing Linkerty in Scala to writing it in Rust. And we had three design principles that we really wanted to stick to. Keep it simple, minimize resource requirements, and it should just work out of the box. And I wanna focus on design principle number two, minimize resource requirements. Because yes, this has to do with CPU usage. We're writing it in Rust, which is very low level, so it's very fast and doesn't use a lot of CPU compared to other managed languages. So minimize CPU usage, definitely. Minimize memory usage, also Rust is very good about memory management and being very explicit about when you're allocating memory and when you're not and how to do that. So we were able to keep the memory footprint of the Linkerty proxy very, very small. But this also has to do with human resources. So if you need an entire team of Linkerty experts in order to run Linkerty, that's using a lot of resources in terms of engineers and brain power and operational cost. And the same thing on the security side. If you have a security team that needs to be constantly putting out fires and doing remediations and that kind of thing because of security vulnerabilities in your service mesh, well that's also very expensive and uses a lot of resources. So I think Rust helps with all of those things. So Linkerty as a service mesh has two different pieces. There's the control plane and the data plane. The data plane is the proxies where all of your network traffic is flowing through. The control plane is the piece that sits kind of outside of the data path and controls those proxies by pushing configuration down to them, by giving them certificates and doing rotation and that kind of thing. And for those data plane proxies, this is where that requirement that they need to be secure, fast and efficient comes in. They're in the hot path of your application. Any kind of slowness in the proxies will manifest a slowness in your application. Any security vulnerabilities in the proxies will result in security vulnerabilities in your application so that we cannot compromise on being secure, fast or efficient. And so for us, Rust was kind of the only viable choice. It was the only language that we felt like we could accomplish all three of these in. But this was back in 2018 and the state of Rust back then was a little bit rough for doing this kind of thing. There were no production ready HTTP2 or GRPC implementations at that time. So we helped build them. So HyperH2 and Hypertonic, those libraries that we helped to bring to existence. And Async Rust in general was still kind of in its infancy. So we invested a lot in projects in the ecosystem to kind of make the foundations on which Linkerty was built. So for example, we invested heavily in Tokyo, which is Rust's Async runtime for doing Async I.O. In Hyper, which is the fast, safe and correct HTTP implementation for HTTP1 and HTTP2. And Tracing, which is an Async aware logging library which was created before Linkerty but has since kind of gained popularity and is used widely throughout the ecosystem. And so this choice to adopt Rust and Linkerty has paid huge dividends for us in terms of latency. So this is a benchmark of Linkerty against Istia that was done back in 2021. If you're interested in seeing kind of the methodology or how to reproduce these results, there's a link and I'll share more links at the end. But you see at the top there in the blue is the baseline. So that's just the latency of the application by itself. In the teal or C foam, that's the latency of the application with Linkerty added. And then in gray, it's the latency of the application with Istio added. And you can see kind of at the P50 latency, Linkerty is a little bit faster, but as you go out to the longer and longer tail latencies, that difference becomes more and more stark. So adopting Rust has also paid dividends in resource efficiency. So this is CPU and memory consumption of the Linkerty proxy versus the Envoy proxy in Istio. And you can see again, like a huge, huge difference here. And it's also paid dividends in security. So these are Envoy vulnerabilities. And what you might notice in this table is that almost all of these, or a lot of these anyway, are in that memory corruption column. So these are the types of errors that Rust specifically prevents and specifically helps you to avoid. So this is all talking about the data plane. Those proxies where all the network traffic is going through. What about the control plane? Well, the control plane is not in the application's data path, right? All of your requests aren't flowing through the control plane, it's just responsible for managing the control plane, or managing the data plane. So if a configuration change takes a few more milliseconds to propagate to all the proxies, that's not the end of the world. And so because of that, using a managed language for the control plane is actually acceptable. It's fine. So we used Go to build the control plane and the CLI of Linkerty. And this was a really great decision because it let us very easily integrate with the rest of the Kubernetes ecosystem. We got access to a lot of libraries for interacting with the Kubernetes API. And it made that process really, really easy. But with that said, in 2022, we introduced a new controller into the control plane called the policy controller. And this was written in Rust. And there's a lot more detail about that in Eliza's KubeCon in North America talk from last year, which is linked there. And it used a library called kubrs, which is a Rust library for interacting with the Kubernetes API, which is great. But it doesn't kind of have the same maturity or widespread adoption that ClientGo does, which is the Go equivalent. And there were some things that were really difficult to do in kubrs, specifically around kind of watching Kubernetes resources for state changes and doing reconciliation loops and that kind of thing. So we developed something called kubrt. This is a library for writing controllers using kubrs. And this made it a lot easier. It just kind of bridged that gap and gave us that ClientGo-like functionality in Rust. So if you're at all interested in writing a Kubernetes controller in Rust, I highly recommend you check out kubrt. It's very cool, made it very easy. So why does this matter? Well, if you're doing anything with financial data, health data, any sensitive data whatsoever, you owe it to your users to avoid memory unsafe languages. This is kind of not a risk you really wanna be taking. And if performance also matters, if you need the best of both worlds, Rust might be the right option for you. So if you're thinking about it, if this sounds appealing to you, if this sounds like similar to your requirements, give it a try, try Rust. It's been incredible for us. And we really think that that deal like it. This is a quote from Alejandro, who's another one of the Lingerty maintainers who didn't have any Rust experience prior to working on Lingerty. And he says, it doesn't cease to amaze me that when something finally compiles in Rust, there's a high chance the program will simply work as intended. Love it. And then our advice to the CNCF ecosystem is fund Rust open source projects. I mean, if security matters to you, invest in it. So we believe that the future of the cloud will be built in Rust. Future of Lingerty certainly will be. Thank you very much. Yeah, two minutes, yeah, let's do it. So one question. So I'm just curious, I was looking at the performance numbers compare, I mean, I can understand the memory issues that Onway has with Istio, but is there an explanation of why Lingerty with Rust is doing much better than Istio even in terms of performance? Considering that C++ and Rust are comparable. Yeah, yeah, good question. I think a lot of that discrepancy comes from the fact that Onway is a very general purpose piece of software. It's a very powerful proxy that can do a lot of different things. The Lingerty 2 proxy was written specifically for Lingerty, it does nothing else. So it's very purpose built and it can kind of avoid a lot of overhead because it's purpose built. Well, thank you very much. I'll be at the buoyant booth. If you have more questions, you wanna talk to me, come find me, I'll be there for the rest of the conference.