 Cool. Thanks, Zane. So as I mentioned, I'm going to be talking today about some new tracing capabilities that we've added to Pixi. So we've been focusing on adding more protocols so you get more out of the box observability using Pixi. And we've added two new protocols that we wanted to announce today. So I'm super excited to say that we now have support for Kafka and NATS. Both of these protocols are essentially message-based systems. So you can post messages kind of in a pub sub kind of way and then you have consumers which can read the messages off. And these two protocols work the same way that we trace all the other protocols that we have inside Pixi. So this is part of our EBPF-based approach. You don't have to instrument anything. You don't have to change any lines of code. You don't need even to redeploy anything. We just automatically EBPF to snoop all the traffic and then detect whether that traffic is Kafka or NATS. And when we detect that, we automatically start tracing it for you. And then we surface it up through the Pixi platform so that you can see all the good data. And I'll be showing a demo in a second so you can kind of see how that feels. Before I get into the demo, I just wanted to talk a little bit about what it is that we're actually tracing under the hood. So at the bottom of this slide here, what I have is a screenshot of some of the raw data that we're actually capturing. So this is the data table that drives everything else inside of Pixi. And so you can see that each row is kind of an event that we've traced. And for every event, we have a timestamp of when that message occurred. We have a source and destination. So which pod did it come from? Which pod was it going to? And in this case, since it's Kafka traffic, we have a command. So you can see that there's some fetch commands here happening. Fetch is when it's going when there's a consumer trying to get data from Kafka. There's some produce commands, which is when a producer is writing data to Kafka. And so you can see all the kind of traffic that's going on. And in addition to that, because of the EBF based approach, you can actually get all of the content of the messages as well. So we grab a bunch of information, metadata and actually the body. So you can see some of that in here. Like if you look at the produce request, which is the third record, you can see that there's in the response, there's a topics order. And I know that this particular produce request was just published something to the topic called order. So Kafka organizes all of its messages into topics and this one was posted, something was posted to the order topic. You can also see when you click on them, you can click on any record, you can see more details about it. So same information just zooming in here. You can see, for example, this is a different example, we see a request command that was a fetch. So we've kind of automatically traced that some consumer dispatched a fetch to the Kafka server. And then again here we see it's also to the same topic. So if you look down near the bottom you see name order that was the order topic and it gives us all the information like the partition and the message number and all sorts of other goodies in terms of what's happening so you can debug your applications. On the right, we have the request latency this is the latency at the protocol level so it's like individual messages from the client to the server. And so if there's any issues with that as well you can monitor the latencies. Now, while that it looks really like that's that's great we can trace all the messages the real question is, how can you use this stuff to actually debug real issues with Kafka or any sort of message bus system. So focus on Kafka here. And so the, we're going to use a demo and I've got a slide here showing what the demo setup is so we have a simple Kafka broker in the middle. That's our message bus. And we have a, this is an e-commerce kind of website where you can place orders so there's an order service kind of that's our front end it takes orders from the website. So an order is made it's just going to publish something or produce something to the Kafka broker into the order topic. And so it'll, it'll push that there. And then we have these two consumers on the right hand side. One's called the shipping service the others the invoicing service, and these two services are just kind of constantly polling to see if there's any new orders inside of the the message bus inside Kafka. So they're constantly trying to fetch from Kafka. And when they get something new, what they're going to do is they're going to take some action. So the shipping service would, you know initiate the shipping process, and then the invoicing service will, you know, take the order take that information and generate an invoice out of that. Now, when you have a Kafka broker when you have an application that has something like a message bus in it. One important question that often arises is, what's the, are my consumers keeping up with the data that's being produced is there any lag in my system. So for example, if the producer creates a message with offset seven offset is Kafka's terminology so just think of it like a message number seven, and it pushes it to the Kafka broker. Right. Ideally, the shipping service and invoicing service, you immediately pull that same order number and start working on it. Right, they pull it out of Kafka and start working on this order number seven. But if there's any issue in your application then you might see a lag and it's important to know if any of your services are running behind. And so this is often a question when you have a message buses is, are my consumers lagging are they running behind. And this is sort of, so the demo we're going to go into is going to try to answer this question we'll see how we can use pixie to answer questions like this. So with that I'm going to switch, so we'll move switch go to go to pixie. So I'm going to go to the homepage here for pixie. And then, here's my cluster. So we have the main page so I'm coming here to the main page of pixie. And what you see is generally a service map of everything inside your cluster and some other information, but in this case we're interested in Kafka rate. So there's a number of Kafka scripts so if you click on the script button here you can search for those by typing Kafka. And where I usually like to start with, like when I'm trying to just understand what's happening in the application is we have these flow graph scripts for various different protocols. In this case we're looking going to look at Kafka flow graph, but the flow graph script kind of gives you a nice overview of what's happening inside of your system. And so when you click on that the first thing it's going to tell you is oh you require a namespace so it says what namespace are you trying to look at the flow graph for. In this case our demo is in a namespace called Kafka demo that's our application. So I'm going to click that. And so immediately what you see is kind of all the Kafka traffic in your, in your namespace in your application. And so I'm going to, so on the right hand right hand side here we have is the Kafka server itself, and I'm going to move that here into the middle. And then on these other circles are different pods so we have an order pod that was our producer I'm going to drag this over to the right to just make it a little bit more clear. And then we have our two consumers which are the shipping service and the invoicing service, and they're the ones fetching from Kafka so on the left we have order it's, it's writing to Kafka and then on the right we have our two consumers which are reading So this matches exactly what we had in the diagram on the slide, and we're able to see that instantly with Pixie that like okay who's talking to who and what's my setup what are all the things that are reading and writing to the Kafka message bus. So I find it a really useful place to start. And then at the bottom there's some various different metrics there to that you can monitor and kind of see in terms of latencies and throughput and such. But again the question we were trying to answer if you remember is, is, are my consumers keeping up with the producer so as we produce orders, are we able to keep up. And so there's a different script dedicated for that. I'm going to type Kafka here again, and then we have a dedicated script called Kafka producer consumer latency. So I'm going to click on this one. And so when we come here. First off it shows you the list of producers that it knows about the list of consumers that knows about. And what we want to do immediately is we want to focus on a particular topic. So I'm just going to populate that right now and I'm going to say we're interested in the order topic. So we know that from our application that that our consumers and producers are writing to the order topic. So we want to say between any producer consumer combination we want to check for a lag. So we'll say we only have one producer in this case so I'll just put producer one that was the name I got from the table here. And then on the right we see we have our two consumers. So let's just check consumer shifting one right, and we'll enter that in. And now we have a graph at the bottom. So we look at this and it doesn't look all that interesting honestly right it's, is there's actually data there if you look carefully, it's a blue line at the bottom. And so what this is actually telling us is that the lag is zero all the time. Now this is good news, like if you don't have a lag that means as soon as the producer made an order and pushed it into a Kafka. The consumer was able to fetch it immediately and start working on it immediately. And so it's telling us everything is healthy with our Kafka setup and our applications doing just fine. Right. Now I'm going to switch over to this other service that we have which was the invoicing service I'm going to type consumer invoicing one here. And we're going to check that one. And so here, we see something different. It's not at zero. I'm going to extend the time window here a little bit. So we can see what's going on. And so what we see here is, whoa, it's not zero. So this should be sending off alarm bells right. It started off at zero, but it's gradually been creeping up over time. Now, what's happened is as I've been talking through this demo, my colleague Hannah has actually started some traffic onto our web application so she started generating some traffic to our website. It started placing orders. And the shipping service that we looked at initially is efficient so it's able to kind of keep up with the rate that Hannah set up and it was able to process all the orders that were coming in and being posted to Kafka. But this other service or invoicing service seems to have some sort of performance issue and it's falling further and further behind right the rate of orders that are coming in it just can't keep up with them. It started off kind of zero lag, but then it went up to like two seconds then four seconds and eight seconds and 16 seconds and it's, it's trying it's fetching data starting to work on it is trying to work on them work on work on them. And then, you know, it finishes that working goes back to Kafka and says what else do you have for me, and then coffee overloads it with a bunch of more work because it's falling further and further behind. So by the end of this time window over this 10 minute window it's already fallen like 60 seconds behind in its work and this is super alarming in this case right, it means we have a problem with our application. If we actually come to the application itself. So in this tab I have the application. It's a very simple application just meant for demo so it's nothing fancy but if you clip up click on shipping, what you'll see is all the shipments that have been made, and I'm going to scroll to the bottom here, and we see it has the last thing it knows about is shipment number 1253. So if I go to the invoicing page and we look at it. It thinks the last order that was made is 811. So we can actually see this problem manifested in the web application as well. We were able to catch it in pixie but we actually see the effect in the website it's running behind. And if I refresh this page, you know it is the invoicing service is trying really hard to kind of keep processing these things it's now gone up to 827. It's really not just able to keep up with the rate and it's falling behind and this can be a frustrating experience if imagine you were a customer and you were using this website, you wouldn't be happy because you're seeing conflicting information in the shipping and invoicing pages. So what do you do in a situation like this right so once you kind of catch this sort of issue with pixie you would want to go back and see what's wrong with your invoicing service right there's clearly it's not able to keep up. So you probably want to go back and maybe spread it so it can process more orders per second, or maybe there's some actual performance performance issue in there that you want to go fix you maybe want to go study it is there some bottleneck somewhere that you need to relieve. You want to go fix it and make sure that this line goes back to zero. So that's it for the demo. I'm going to leave it there. Coming back to the slides. We, we have a bunch of different scripts for Kafka so we covered the flow graph we covered the last one which is the producer consumer latency. There's other ones that on your own time you can go take a look at and see if they're useful for you and sorts the sorts of questions that you can answer with them. And then this is going to be the last slide though I want to talk about so we're switching gears a little bit here. So we're done with the Kafka demo, but I wanted to take a moment and talk about the sterling roadmap sterling is our EBPF based data collector. And it's what's getting all the data that feeds up into the rest of pixie. And we have a number of tracks inside of sterling that we work on, we have multiple tracks but two of the major ones are the protocol tracing track and the CPU profiling track. The protocol tracing track is the one that automatically grabs all your protocol traffic without you requiring any instrumentation. And so it currently supports things like HTTP gRPC for go my sequel postgres bunch of different databases we have a bunch of protocols. And today we have now Kafka and that's which are in beta, which we encourage you to play around with and see, you know, what sort of information you can get out of them and obviously if you run into any issues please do report them to us. And so that's those are the ones that are kind of done today and where we want to go in the future with our protocol tracing is to kind of expand this, this set. So in no particular order, things that we're looking at are things like gRPC for C++ and that would cover interpreted languages to gRPC for Java. And then there's AMQP, MongoDB and there's others, right. And if there's any other protocols communities interested in as well please do get in touch with us because we're always trying to keep a pulse on what's important to the community. And then on the other track the CPU profiling track this is the feature that provides you with the popular flame graphs. It's been it's been a kind of a popular feature of ours. Today we support profiles, CPU profiles for Go, Rust and C++ so you can see the nice looking flame graphs for those languages if your application is written in Go, Rust or C++. And we're looking to add support for other languages so things like Java and maybe Python and Ruby or things are, Python and Ruby are a little bit further out on the horizon but where top priority for us is actually Java next is to take a look at Java. And so that's just give you I just want to give you a little bit of a taste of where we've been and where we're going. And so that's it for Kafka and the Sterling roadmap. And with that I'm going to pass the baton to Vihang who's going to talk about end to end encryption. We'll keep the questions for the end. Alright. So yeah, hi I'm Vihang. I'm one of the engineers at Pixie and I'm here to sort of talk to you about a new feature we just we recently added to Pixie. However, before we get into some of the very nitty gritty details about end to end encryption, I want to start by quickly doing a recap of the data model and how how your data is collected and stored in Pixie. So this is like a very high level rough architecture of what how your data flows through Pixie. So data collected by Pixie what the work that the Sterling team is doing of that data is stored just locally in your cluster. Our script execution framework also runs within your own cluster. So the data processing itself is also happening within the cluster itself. So when, when the Pixie client, let's say you're running the UI as omit should or maybe you're using a Pixie CLI or something like that. When the client needs some data to render what happens is the client spends the script to the cluster. And then all of the data processing is done in the cluster. And then the script results are sent back to the client. So in one of the models, what happens is that the client will directly connect to the cluster to fetch the process data. So this is what we call direct mode and this is what we are indicating on the right side of the graph you're seeing. In this situation, you know, your client is directly fetching the result data from the cluster. So the disadvantage of direct mode that we had to address here at Pixie when we're testing our own product and also we have noticed elsewhere is that in this scenario your clusters might actually be behind some firewalls. And then now you have to like punch some holes in these firewalls for all of the Pixie clients to be able to access these clusters and get the script results from. So to address that challenge and simplify sort of the firewall configuration. We built a way for the Pixie cloud to proxy the script results. This is what we call password mode. And so this password mode is a convenience method where Pixie cloud is just proxying the data between the cluster and the clients. The data isn't persisted anywhere in the cloud it's just a proxy for for the data. And then just a quick reminder like Pixie cloud itself is also open source so when we open source Pixie it's not just the vizier parts of Pixie but also cloud. So folks can run folks in the community can run Pixie cloud themselves. And so in past remote it would still be flowing through an instance of Pixie cloud that is controlled by you. No matter which of these two data transfer modes you use all of these connections do still use TLS. So the data is always point to point encrypted in transit. So even in this scenario it is still possible that you might want to use Pixie cloud for the convenience, but you might be running a Pixie cloud instance in an environment you don't fully trust. So we really wanted to address that scenario where you're using past remote with Pixie cloud you like that convenience, but you still want to sort of protect your data and transit. So to address that we decided to work on end to end encryption. So into an encryption will provide an additional layer of security, where the data itself gets encrypted at the Pixie vizier and will stay encrypted during transit and only the client will decrypt the data. So we can dive into some of the details behind how we implemented this into encryption. So, thankfully, there was, there is an IETF proposed standard for data encryption. It's called a JOSC or I don't know what the pronunciation is I didn't see any guidelines for it. But basically this was a new standard that was proposed for encryption of JavaScript data. Even though this is not a completely accepted internet standard yet it's just a proposed standard. There's already client libraries and many different languages. So we were able to use the cryptography API which is supported by all major browsers to to do this on the client side when you're using the UI. So we have a CLI, which we've written in Golang, we use JWX, and then the Golang and Python APIs use JWX and offload respectively. So basically you can you can get into an encryption no matter which of the Pixie clients you're using. And then let's walk through, like, example of how enter an encryption would work from the UI. What happens is before we can make any requests to Vizier for data, the browser first has to generate a key pair. So, as I mentioned earlier we use the web crypto API for this, that all major browsers support. And the first thing that the browser does is generates a key pair. This key pair is then the public key from this key pair is then sent to Viziers on every request to run a script. So, anytime you want to get script results, the UI, the browser will send this public key along with the rest of the request to the Viziers. As before, all of the data processing is done in the Vizier side. However, now there's an additional step before Vizier responds with the data. The Vizier will use this public key and encrypt the table data before sending any response out. And now the response that Vizier responds with on a successful script execution will be encrypted with this public key. And this encrypted table data then is sent back to the clients either through Pixie Cloud if you're using pass through mode or directly if you're using a direct mode connection. And then the browser which generated the key pair to begin with has access to the private key of the key pair, which is still controlled by the browser it wasn't sent to anyone else. And so the browser can use the private key to decrypt the results and render the UI and so on and so forth. So yeah, reminder that the private key will never leave the browser. In fact, the private key is scoped to the session storage on the browser so the same user using Pixie on different computers or different browsers will have different key pairs. And these private keys will be local to their browser store session storage. As far as the CLI and the API's go they behave the same way you just replace the browser with either a goal based binary, or the go or Python API clients that are equal and instead of using the web crypto API, they will use the libraries and go and Python, the crypto libraries there. So that's sort of a very quick view on how into an encryption words. And this basically now gives you this additional guarantee that if you're using pass through mode, your data is still encrypted even when it's proxying through Pixie Cloud. And that sort of gives you the freedom of having full guarantee that no one can sort of look at this data in the cloud, no matter what environment Pixie Cloud is running in. So that's, that's all I have. Happy to take questions. Okay, well, thanks everyone for coming to the meeting. I just have a few quick updates about the community. And then what we're going to do moving moving forward so Pixie was officially accepted as a CNCF sandbox project. So we're currently in the process of actually transferring ownership of all of our assets including our GitHub repo to the CNCF. And as part of this process, once our repo is moved out to CNCF will also be opening up all of our issue tracking and roadmap. In addition, I wanted to say we plan to have a live event at KubeCon North America, assuming that that event still happens in person. And that is planned for the week of October 11. And please stay tuned for the details. And we also plan to have a Pixie governance board meeting in October. So stay tuned for all those and, you know, we look forward to seeing everyone on future calls and events. Thanks for coming. And if you have any questions, you know, please feel free to hang on. Well, we'll stay on for a few more minutes. Thanks everybody. I have a question. Is other ways for the community to contribute to protocol tracing. Yeah, that's a great question. We have, we've gotten a number of requests about that. There's a how to contribute protocols. There's a lot of people out in the community who want to have a protocol of interest that has not been supported yet. Obviously we're working hard to get those out, but it's great to have community contributions as well. And so one of our goals for a future Pixie not meeting is to actually outline the steps that's required to implement your own protocol parser. So that you can just plug it in. So it's the way that sterling structured is there is a protocols directory where you can go just implement your own protocol parser but we're trying to make it a little bit easier to use and document the exact steps that you need to go through and falls and things that we've learned along the way. And once we have that document ready, we'll release that out to the community. And we highly encourage people to, to contribute to be super awesome and there have been some people out there who have already taken the initiative to try to implement a number of protocol parsers and we're super excited about that. And we're just trying to make it easier for everyone who wants to contribute. And we'll be tuned. We'll be publishing a doc for that. Another question that we've gotten on slack lot is, can you share how pixie traces encrypted traffic. Like what are the, what are the limitations for that. Yeah, so, let's see, I think we have three stuff that backup slide. Let me just go to here. So we can use the slide here to take a look so the way that pixie, like the sterling engine of pixie traces things as we typically trace things at using ebpf k probes at the Linux API and there were able to trace all of the kind of read and encrypted traffic. So anytime a network call is made to like send, send packets or receive packets or send a message receive a message, we trace that information. And then we're able to reconstruct what was happening. And so our protocol parsers and then surface that up into the rest of pixie with encrypted traffic that we can't capture at the kernel interface anymore. So we capture at the kernel interface, all the data is going to look encrypted to us and that's kind of useless. So what we do in that case is we actually use ebpf again, but we add different sorts of probes we add probes on the SSL and TLS libraries themselves. And we trace the data right before it gets encrypted. So we combine it with all the other information that we have and kind of plug it into the rest of the sterling framework to parse the protocols and do all the rest of the magic. But essentially, you know, we just have additional probes before right before the data gets encrypted at the interface of the TLS SSL library. And by capturing that data that's how we're able to see the encrypted data. So it's for open SSL and goes crypto TLS library currently. So the yes, the traffic has to go through one of those two libraries if it goes through some other custom library then we don't currently trace it. But but those two libraries cover a good I mean with open SSL alone we cover a really large swath of the traffic is a lot of people are using open SSL. Well, thanks for that on it. And yeah, everyone please feel free to then join stuff on our Slack community and ask questions there or file get up issues. And thanks again for for joining.