 If you don't know what this little creature is in the bottom right when when Lincardie graduated we had to Pick a mascot, you know, because they have all those cute little animals for Kubernetes and for go and all that and so we tried to pick we tried to think of like what was the least cute animal that we could pick and lobster was the one that we ended on and It's actually a blue lobster, which is very rare, you know So if you ever see one of those in the wild don't don't eat it And then we had to answer a bunch of questions about like it does a slobster what what gender is a lobster and I did a Lot of research and it gets complicated so I'm gonna leave that for for you all to research at home how to tell, you know The gender of a lobster all right. I changed the subtitle just now those of you who were in the audience Saw me do this live. I promise it being here does not make you a boring person despite It's very boring title and the fact that I am boring. I'm gonna try and make this really exciting This is our Lincardie maintainer kind of project update that we get to do every kube con So I really appreciate you coming out here and listening to me and you know feel free to Raise your hand if you have a question at any point I'd love to make it more interactive and not just me mon logging, but I may get Lost in the in the beauty of what I'm saying. So, you know, we'll see how it goes All right, this was supposed to be delivered by Alex Unfortunately, Alex is filling in for another talk So I'm going to be delivering this in her stead But Alex will be delivering a talk that probably will be much more interesting than this one called five years of cloud native Rust that's today at 230 on level three. I think you can if I did my qr code right You should be able to just scan that otherwise you can just type in one r2 qn One of the things I'm only going to mention very briefly in this talk But Alex is really going to go into is the investment in rust that we've made so, you know It's been as you can see five years And a lot of what makes Lincardie great today comes from this very early decision And you know a lot of kind of the pain and suffering that came along with that to adopt rust and and to adopt This very particular approach where you know every aspect of the of Lincardie's data plane is You know it's kind of informed by not just rust but around the asynchronous network ecosystem That's been built up in rust. So please do attend this talk This me William Morian. Thank you for being here. I used to do this talk a lot and I actually haven't done it for a couple years So, you know, we'll see how it goes. This is a picture of me fixing a toilet in the very early days of Lincardie development This was basically my best contribution to the project Arguably at least that's what some people have told me. All right I'm gonna do a very brief intro into what Lincardie is and you know if you've seen this a thousand times before then You know just bear with me. It's a service mesh. I'll spend one slide describing what that is It's known for being light for being fast for being security first and it's also known for being Simple and that's really the focus for us created by a company called buoyant, which I love And please, you know pay us a lot of money Seven over seven years in production 9,500 slack channel members We're we're close to hitting 10,000 So if you haven't gone to slack.linkardie.io, you don't have to do anything just sign up so I can get this number to 10,000 If I see anyone in here who's you know at the next talk at Kubcon Paris next year and we're not at 10,000 I'm gonna be upset at you for not joining Make sock puppet accounts if you need to everything's fair in the you know in love and cloud native Lots of github stars lots of contributors Weekly Edge releases and so on we were the first service mesh to graduate the CNTF a couple years ago And our job, you know the job of Lincardie is to give every platform engineer in the world the tools that they need to Create secure and reliable and observable cloud native applications. So hopefully that resonates with you If not, then it's may be a very boring talk. All right, it's a link Lincardie is a service mesh I promised I'd spend one slide on this You know so the way I think about this is a service mesh is an infrastructure layer, right? So it's not something that the developers have to interact with directly It's something that should be kind of provided to them as part of the infrastructure It's at the platform layer and you know in a way It's kind of like this modern L7 capable network So if you come from a networking background you're very used to you know what I would call L4 Stuff, which is you got packets and you know you've got IP addresses and you're trying to deliver packets from this IP address to that IP address I know I'm like summarizing a lot of stuff in you know in a really short way And then for but for Lincardie we kind of think we're thinking about things in L7 versus L4 which means we're thinking about rather than IP addresses. We're thinking about workloads, you know Or rather than and workload identities You know rather than TCP connections and TCP packets. We're thinking about HTTP to connections and gRPC messages and things like that and and so it's it's kind of a different level of Networking and you know the goal for us is this should be uniform across your entire application So all these benefits, you know you get kind of regardless of what the application was written in and what team wrote it You shouldn't have to change your application for the most part. That's true There's always one or two little asterisks that we have to navigate through And then our implementation as I mentioned is this idea called a micro proxy So we build it in rust try and make it as small and as lightweight and as single purpose as possible So if you're familiar with Envoy, it's a general purpose proxy. The Lincardie proxy is not general purpose You can't use it in any context other than Lincardie And that is part of what allows us to make it slim and trim And then we've got a control plane off to the side that kind of manages as proxies for you and gives you the Interaction points, you know through CRDs or elsewhere To interact with Lincardie as a whole this makes sense so far to everyone Okay, I see a lot of nodding heads. Thank you. Come on in. You're not too late. I'm just going through the intro stuff All right, what makes Lincardie so darn good So for us, we got a couple kind of design goals One is we want to make it just work so we want to make it so if you have a functioning Kubernetes application Runs today and you add Lincardie the application continues to function and you know 99% of the time we can do that Of course, there's always one or two asterisks And it's also a very good way of finding out whether your developers have obeyed The standards when it comes to things like HTTP calls have they made Assumptions about header ordering or about header casing and things like that, you know, that's that'll be a fun Fun discovery for you as you add Lincardie Because we do require those things we do require the applications to conform to those standards Ultralight, of course, I've talked about simple to operate You know, that's kind of our promise to you is you are the poor souls who are going to be on the hook For waking up at three in the morning if this thing goes down, you know We want to minimize that as much as possible and we want to give you the ability to just have in your head Like is there a model of how this thing works and when things are behaving, you know Contrary to that model you can ask a really informed question rather than just believing it's this black box of magic You know complicated stuff And then of course we want to make it secure You know and and treat that as kind of like the default behavior Not as something that you have to enable or configure to the extent that's possible Okay control plane data plane, etc These slides are basically Online so you know if anything I'm saying goes by too fast you'll you'll be able to to find them All right the data plane that I mentioned rust microproxies not using envoy Okay, this gives us a whole bunch of interesting kind of characteristics You know part of this is it's part of our security first approach So rust allows us if you're not familiar with rust really interesting language You know Alex's talk later today will be we'll have a lot more examples Actually, I think she's got examples of what things you can do in C++ that are terrible that the rust Compiler will not let you do so there's some fun the kind of code side-by-side stuff there But it allows us to avoid an entire class of memory vulnerabilities that are basically endemic to you know C and C++ code turns out no human being in the world can actually write safe C++ code Beyond hello world maybe ultra light ultra fast, you know, so we compile a native code State of the art networking stack all sorts of fun stuff in here You know and really we want to we want to make this thing an implementation detail Right, so if you are a service mesh operator, you should not and you become an expert in linker D You should not have to become also an expert in the linker D proxy All right now let's get into the controversial stuff. This is my opinion. Well, okay. Yeah, that's my opinion So there's kind of three, you know, if you're very familiar with the service mesh landscape There's kind of three approaches people have taken there's node proxies Which actually came first because of the very first version of linker D was node proxies There's sidecars, which is what we are today and then there's this newer idea of ambient Ebf ebpf comes up a lot in some of these conversations It's basically a red herring because ebpf is great for l4 features Right, we talked about the network and packets and things like that. It's great It's amazing for that, but it can't really handle l7 features Not the way that we that that the service mesh needs it to handle Because the way that the thing works is it is limited by design because you have to run these bytecodes in the kernel Right and so you have to be very very strict on what you're allowing to run in the kernel. So that's fine, right? It's not a knock on ebpf. It's good for some things not good for other things But what happens is when you make a sidecar free ebpf service mesh You end up using node proxies and that actually is bad, right? Because node proxies mean that you share all the TLS information It means you share all of the requests from very disparate workloads all going through that single proxy We saw this live with linker D in firsthand with linker D one dot X which was node proxies and You know a lot of the initial adopters from of linker D one dot o Told us how painful this was so that was part of the kind of reinvention of of linker D You know circa 2018 to move into the sidecar world Ambien is better. I'm not going to talk too much about that. It is very complicated But at least has better security properties We did do a long exploration of ebpf. I think the TLDR and if you read this Blog post that I wrote about it. I think it's likely will add ebpf at some point to linker D But I don't think it really is going to change a lot for it Sidecars and micro proxies are the the best solution after doing a thorough analysis They continue to be the best solution For for serious users. They have a really clear operational and security boundary, right? That's like you there's a huge benefit to that you've got a really clear implementation of things like zero trust If you're an operator, you know, you have a model in your head That is very straightforward and there's a lot to be said for that minimal resource usage Of course, if you're gonna have sidecars like you got to make them small because if they're large then you know That's not that great and There are some warts around sidecars and kubernetes like if any of you have tried to mesh You know cron jobs and sir jobs things like that You know, it's it's annoying and you have to do some extra work. Those are slowly being Fixed at the kubernetes level. So there's some really cool stuff happening with the sidecar Sidecar containers and stuff like that. I wrote a little blog post about this 1.28 was when they when those first came into alpha and 1.29 and they're gonna I don't know They're getting better refined in some way. So that's all to say at the end of all that You know ebpf probably is on the roadmap for linkerty, but I don't think it's gonna change a whole lot It's gonna have a pretty minor effect. Okay I'm gonna jump with that. Let's see. Oh, we're doing okay for time But I'll jump into Kind of a little history of some of the recent releases and then I'm gonna end with kind of what we're building in the in the next release So 2.12, which is gosh, I don't even remember when this was a couple couple releases ago Introduced security policies that were per Route so the the security model for linkerty is that we can enforce remember them because we have the sidecar And we're kind of in you know at the at the pod boundary Right and all traffic before it can touch the application on that pod has to go through the proxy The security model is at that point we can actually enforce You know what kinds of requests are allowed to talk to the application and you can describe those requests You can describe them in any way that you want You know you can describe them based on IP addresses But you know we're trying to move to the security model of future where we're talking about workload identities and things like that So you can describe them in terms of workload identities and you can describe them in terms of routes So you can say I am service B your service a and in order for a to talk to be You know it is allowed to talk to be you know on the slash foo endpoint But not on the slash bar endpoint things like that Right, so this is what we're talking about workload identity and kind of per route or per g rpc method granularity The way that we implemented this is using this thing called the gateway API and I'm gonna talk a little bit about the gateway API because it's kind of a Recurring theme for linkerty and it's kind of this future direction we're headed So this is an example of an HP route that permits certain requests to a server Which is a you know another type of object called the author's server And so you know the goal here is we want to give you kind of this really granular level of control But we also want to do it in as standard a way as possible So rather than implementing, you know linkerty specific CRDs We try and do it with the gateway API again. There's one or two asterisks to that statement But by and large that's that's a goal right and then 2.13 Okay after that so that 2 to 12 is very security focused right giving you those policies 2.13 now We're getting into Kind of back to the world of routing away from security back to the world of routing and giving you a similar capability with request routing So giving you the ability to say I want to route traffic, you know Not just based on the kind of the DNS Name but based on headers based on verbs. We don't look at the body for a variety of reasons But this is a very fine grained way of routing requests kind of dynamically based on these routes Again configured with the gateway API Not with SMI components if you've been a linkerty user for a while You're probably familiar with traffic splits and things like that. Those were based on this SMI API in the modern world. We're trying to move off of SMI on to HTTP route And so we use that same object that was introduced in the previous release Great So now we've you know now we're starting to have like a really uniform Configuration space and there's all sorts of interesting examples of how you can build upon this to do things like You know per user canaries and stuff like that The other thing we introduced here was circuit breaking So this is something you know I think came a little late to to the linkerty probably should have done earlier And the idea here is like if an endpoint is failing, right? We should just stop delivering traffic to it because we know it's failing and because Reducing traffic to it might allow it to recover right it might be dying because there's too much traffic So if you've got too many consecutive failures if the success rate is too low Then we're going to turn traffic off and then as it recovers So we're going to ping it every once in a while when it recovers. We'll start sending traffic back to it So that was 213 and then finally 214 is the most recent release So this is Flat network that's where we introduced flat network multicluster so in 2.13 and earlier so linkerty has had multicluster Capabilities for a long time and by what multicluster means in linkerty's cases the ability to send traffic from one cluster To another kubernetes cluster in a way that's transparent To the application so the application, you know Again, I'm service a and I'm talking to service B All I have to do is connect to be like the normal way that I do and I don't have to know whether B is on the same cluster It's on a different cluster whether it's like halfway in between these Clusters because I'm splitting traffic across them whether I'm the middle of some kind of failover operation. That's all independent you know from From from the application's point of view right you just talk to the service and so that gives you a lot of flexibility You know under the hood the way we implemented this was with a with a gateway. I'm sorry. Is there supposed to be a gateway? Picture here. Okay. There we go. This makes a lot more sense, right? Yes under the hood we have a gateway on the destination side, right? So if you're workload one and you're talking to workload to it goes through this gateway object, right? And you don't get application doesn't know about that That's kind of happening under the hood the reason we did that is because the primary use cases that we saw for multi-cluster and Kind of the early days of linker D We're kind of these ad hoc use cases where I have a cluster over here and now you know separately We add this other cluster over there and it might be on a different cloud or it's in a different zone Or it's like, you know owned by a different team And so this gateway based approach was really nice because all you had to do the only l4 requirement that you had here was You need to be able to establish a TCP connection from the cluster on the left to that gateway But that was it right we didn't make any other assumptions about the underlying networking now what we've seen you know more recently is If you underline network allows pods right? Yeah, great So right so if you actually do your network setup, you know in in kind of a more forward-thinking way And you can actually route the pods, you know pods can route traffic to each other Well, then we can get rid of the gateway right and then you end up with a very Very boring diagram that looks like this, you know again workload one doesn't know where workload two is So linker D is kind of taking care of that under the hood for you But then you can you know, you can get rid of that gateway component This has a couple other nice benefits. Of course, you're not going through a gateway. So that's one fewer hop and You know, you also preserve the identity on the from the left-hand side. So that's kind of nice Yeah, sure Yeah, so the question was what do you mean by an underlying network that allows you to do this So yeah, this is basically a shared flat network So you have all the pods and my network, you know engineering knowledge is about one level deep So I'll describe this as best I can but it means that all the pods are can route traffic to each other So they're on the same shared flat network. They have IP addresses that are unique across all Clusters not just unique within the Yeah, that's right Yeah, that's right. That's right. And if anyone wants to correct me on what that means Raise your hand Don't nap between the pods go straight from pod sider to pod sider Okay Thank you. Yeah So, you know, it's an alternative mode. We didn't throw away the first mode This is just if you're in this situation. Hey great, you know, we can save you a component And you know all sorts of interesting engineering challenges that I was kind of you know Half exposed to here around services covering and things like that But the details are all in the you know the docs and blog posts and stuff around that The other thing we did in 2.14 is we officially became conformant with the Gateway API, which is you know Again, I keep using those words. Basically the Gateway API was or is a Set of API is that was originally designed to replace or to You know provide a successor to the ingress resource so the ingress resource which we all know and love You know was something that we often had to work around because it was so limited the Gateway API was designed originally To have this really expressive set of CRD is that you could use you know to describe the full set of you know behavior you want on on ingress and they did a really good job in designing this API they did such a you know So good of a job that actually a lot of it applies to the service mesh. So at some point You know that that the Gateway API team said well, gosh Let's actually make this you know work for service meshes as well as for ingress And so they added this idea of different profiles and there's a mesh profile Linkerd was a very first service mesh to be fully conformant with the Gateway API mesh profile It's really exciting if you're like a deep Kubernetes nerd if you're a user I don't know that it really is that interesting except to say that It is a step towards a goal that I'm kind of excited about which is we have a fully uniform set of configuration for linkerd and for ingress with the same set of API's same set of you know CRDs and types and things like that and I think that will be a big benefit because then You know a you have some you know You have some flexibility in terms of like you know Here's the API for controlling my service mesh and maybe I swap out the implementation Underneath without having to change any of that and be you have one set of things that you're thinking about when you're configuring ingress And for measure one set of tools that you're working with I guess when you're configuring those two things and to me That kind of makes sense because a lot of the constraints, you know and and goals are the same There was you know things got a little the this standard is still evolving So there are one or two warts where when you're actually trying to do this But so this is a kind of I would say this is a stepping stone But maybe not the you know, we're not at the final end state yet All right, so 214 was our most recent release now finally we're going to get to 215 So this is an exciting kind of announcement time So the goal for the next release is to add mesh expansion just the ability to add Off cluster non-cubanet ease things into the mesh so that means we can run the data plane outside of Kubernetes so on the one hand that's really really easy because you know these rust Microproxies will run anywhere like we can compile them for any architecture and they run You know, it's just a binary. They're actually you know We did one clever thing early on which is we did not make those proxies Kubernetes specific at all They're linkardy specific so I need to talk to the linkardy API But they don't know anything about Kubernetes They don't have any assumptions in there that they're running on a Kubernetes cluster or even in a in a container, right? They're just regular old-fashioned binaries But you know, that's the one simple aspect every other aspect of this is really really hard There's a whole set of challenges around well. How do we do service discovery? How do we do you know network connectivity when we're running within Kubernetes? You know, we have all these assumptions we can make Kubernetes gives us all these primitives that we can build upon we can You know, oh, you need to get the you know the sidecar You know into every pod. Well sure you just use a mutating admission web book controller and you know there you go When you're out in the world the VMs well, you don't have any of those primitives So you have to decide okay, which of these you know, are we gonna actually rebuild and which of these are we gonna leave as a burden for you? right the user one that I want to talk about briefly here because You know this one became really interesting with this idea of workload identity So all the policies that I talked about earlier and in linkardy 2.12, right? All of our kind of like authorization, you know framework is based on this concept of workload identity We don't want to use the IP address. We don't want to trust the network, right? We want to trust cryptographic proof of you know of strongly attested workload identity in Kubernetes land we can use your service account as a workload identity because that you know for purposes security is basically Tantamount your identity once you're outside the cluster. Well, we don't have that anymore, right? So now we have to come up with a way of Providing identity for like an arbitrary application on an arbitrary VM turns out there actually is a project That's basically solved this so that's great for us. We don't have to reinvent the wheel. It's called spiffy inspire Does anyone here used spiffy or is using it today raise your hand don't be shy I see one half-hearted hand on the far right Anybody else That's it. It's a graduated project Is anybody too embarrassed to admit that they use it raise your hand. Oh, there we go. Okay two hands. Yeah All right. Well, I don't know what them. I mean gosh, okay Maybe this was a bad idea Well, mostly I just don't want to have to rebuild all this stuff. So, you know Right. So, you know and kind of the division here Is that spiffy is a standard, you know and inspire is effectively a reference implementation And all that spiffy does is it says hey if you take a you know It's called an x509 certificate and you format, you know these fields in this way Then you now have a spiffy ID and you know, there's more complicated in that There's like a jot version a JWT version and things like that But it turns out those those, you know those x509 certificates are exactly what you use for TLS and including for linker D's mutual TLS so that integration is very very natural now the question is okay if I have a standard for you know Presenting identity that's kind of already implemented in in in linker D Well, how do I actually get the identity for an arbitrary machine and that's where spire comes in so you run spire spire gives you a spiffy certificate for you know a workload or you know, or whatever you're trying to attest So we're basically a late prototyping stage with this it's going along really really well I expect to have a stable release. I should never do this I should never put a date on a slide in a recorded fashion So editors just beep out the next section, please but early next year. It's looking really good And I just wrote a little blog post about this a couple days ago if you want if you want to learn more Yes, sir Yeah Great question. So question was does spiffy inspire for anyone who's using it out there Does that interact with certain manager? You know, does certain manager do anything with spiffy inspire? What about the one half hearted hand over there? You want to you want to tackle that? Okay, so he believes spire would be doing the rotation and management anybody have a different opinion. Yes over there Sir manager does interact with spiffy inspire. Okay, great Okay, there's a combination you can use Yeah, okay, great great. Yeah, great great question though Any other questions on this yes, sir? Yes, that's right. So you would run the product question was do you run the proxy on the VM? Yes, yeah, you run the proxy on the VM What you know, how much help we can give you in Delivering that proxy to the VM and getting everything set up and all that is it's kind of what we're exploring now But ultimately yes that proxy sits there it gets an identity not from Linkardy CA sitting on the cluster but from from spire and then it establishes a connection to the control plane Any other questions on this? I'm really excited about this because you know finally gets back to the In linkardy 1.x for the elderly amongst us here remember that we actually were able to run on VMs and you know on Mesos and on nomad and like it you know is totally orchestrated Independent and I'm excited to be able to at least get partially back that way control plane still has to run on Kubernetes So we're really talking about the data plane that gets to expand And this is also a good lesson why you should never raise your hand when someone asks a question because then you'll be called on to you know Answer a follow-up questions Okay, and that's really it we've got six and a half minutes left So we can do a little more questions than answering if you want to get involved of course you can talk to you know You can talk to me all the development is happening on github. We have slack linkardy.io, which please everyone joins we can get to 10,000 people there We've got mailing lists if you want the formal announcements. We do the CNCF funds formal third-party Security audits. I think we have a pretty friendly and welcoming Community I do have two quick ads and then we'll jump into questions Boyant itself runs these free engineer focused training things called service mesh Academy So if you are interested in learning more about linkardy Myself and a couple other of the linkardy folks do deliver this educational content I think it's pretty good. We get pretty good reviews. It's all free. So you don't have to do it We also have an enterprise distribution if you're running multi az clusters, especially and spending lots of money Please come talk to me afterwards And that's really it so five and a half minutes exactly remaining any questions Let me I'm gonna come to you in a second Amir. Let me start over here. Yes, sir Yeah, great question. So the question was you know linkardy the data plane leases written in rust What has the experience been like? You know, what kind of advantages what's been bad? The the right place to learn more about that is at Alex's talk at 2 30 Because she's gonna dive into all of that stuff. I'd say for my perspective It was a real gamble early on because Rusty in 2018 was like You know the language was there, but the ecosystem was like barely there It's really paid off though like almost everything that I think is really powerful and unique and and and amazing about linkardy is Because of that choice Yeah, great question, but do do go to that talk All right, Amir I do have some updates about point cloud But I don't want to talk about them here because I want to keep it as open source as possible But thank you. I appreciate that question. All right. Yes, sir networking expert in the middle So the question was when you're doing mesh expansion, there's configuration Does that configuration happen on the VMs or does it happen on the cluster? I actually don't know the answer to that question I would my my guess and you know Flynn back there, please correct me if I'm wrong My guess would be there's gonna be an amount of configuration that has to happen on the VM because you have to tell if nothing else You have to tell the VM Where you know where linkardy is? All right. Yeah, come come find Flynn or find myself Afterwards and we'll figure out the right answer. But yeah, great great question. All right Anyone else another question? We're gonna sit in silence lock the doors. Yes, Joe, please Okay, so the question was Joe here is a cheapskate and has GPUs And you want to know can you can use mesh expansion to bridge the GPU and your and your local Kubernetes cluster? Yeah, my understanding is yes as long as those GPUs can run like, you know, a Binary Linux, but you know I have enough Linux primitives that we can do like the networking stuff that we need to do and can run a binary Then yeah, that should be just fine Yes, sir Yeah, great question. So the question was multi cluster. Is that primarily targeted having money many clusters in the same region or? Could that be used for you know clusters in different regions or different clouds the gateway approach kind of the original approach Doesn't care. So yes, you can use that in multiple clouds We have adopters today who you know are doing a multi cloud approach and tie these things across You know from one cloud to the next the that gateway connection is MTLS, you know So it's secure you can put it over the open internet and you can feel confident in that the flat network approach Doesn't work across regions. You need to have like the the L4 L3 L4 capabilities, you know sort it out there. I Believe please green hat correct me if I'm wrong All right, we got time for one last question Yes Yeah, yeah, okay. So the question was we've talked a lot about the linker deep proxy But linker D also has an init container. It has an optional CNI like plug-in thing. How does eBPF play with all of that? I so the way that I see eBPF working with linker D and Probably I'm gonna be corrected on this But my guess is the thing that we can do there That would be most useful is if you're doing a pure TCP proxing through linker D and you're asking us not to do any L7 metrics at all because this is an application initiated TLS connection For example where we don't want to man in the middle it or it's you know some TCP Stream where you're just like I don't care just get it to the other side We could use eBPF to bypass the proxy and just like wire those two things together Would that be done at you know through the init container through the CNI level? I don't I don't know I'm not sure but that's that's the intersection of the all of those words that you said That I'm aware of all right. Thank you folks really appreciate your time here today I'll be here so come up and ask questions and use linker D