 is Alexis here, don't see him. Brendan Burns, don't see him. Brian Grant, then Michelle, so we got five out of six. Cool, let's get started. Liz, you wanna go lead, and we'll get the communities to present. Okay, yeah, I can have a go, my voice holds out. All right, I don't think we have very much other than leaping straight into the community presentations, but let's do it. So, Fanos, who do we have from Fanos? Yeah, it's me, Bartek and Frederic. Awesome, take it away. Thank you, so hello everyone. My name is Bartek Plotka, and I work for Improble. Nice one, I can hear anything. Hello, hello, can you hear? So yeah, I work for Improble, and I'm one of the maintainers and initial authors of the Thanos project. And today together with Frederic who would love to present it to the CNCF. So first of all, what is Thanos? Thanos is a monitoring system, or in other words, set of cloud native components that you can install on top of the Prometheus, which is a graduated CNCF project. You can easily add Thanos to your Prometheus servers that are continuously collecting metrics potentially from multiple communities clusters. Thanks to that setup, you can solve four main drawbacks of running Prometheus on scale, which is, first of all, querying your metrics from the single place. So what we call a global view, making Prometheus highly available. So finally, allowing zero loss for rolling restarts of Prometheus or gracefully handling failover scenarios. And finally, supporting cheap and easy to operate way to store virtually unlimited retention for your metrics with efficient support for long-time range queries thanks to built in the sampling. Next slide please. Thanks, Thanos project is quite unique in making sure you can deploy and experiment without major changes to your existing monitoring Prometheus setup. We can distinguish three main deployment models for Thanos without going into much details in a very basic one presented on this slide. You don't need to set up any separate resources, nodes or clusters for Thanos. You just add the sidecar to each of your Prometheus server and a fully stateless query on top of it. Query then is able to execute PromQL query on a global level fetching the metrics from required Prometheus sources. It's also capable of transparent deduplication which allows to run Prometheus in HA groups. In this form, it does not allow long-term metrics but it is quite obvious option if you want to start with Thanos project. And in fact, some production users we know successfully stays with this option because it's already matching their requirements. Next slide please. If you want to add long-term metrics attention, you can easily set up Thanos to upload Prometheus files in a native TSDP format to object storage of your choice. Then you can connect global query to the long-term object storage using exactly the same common single streaming JPC API. This allows for relatively cheap storage without worrying about high band with low latency sample streaming and complex replicated ingestion path in some cases. Currently, Thanos supports a variety of object storages. We have GCS, S3, Azure, OpenStack Swift, Tencent COS and Alibaba OSS which is in progress. All of those except GCS were added thanks to our external contributors. Next slide please. Last deployment option is still experimental but Thanos also will allow sample streaming which eventually will be able to immediately transfer metrics out of the Prometheus to the separate Thanos cluster in real time. So you don't need to access Prometheus in query time essentially. This option is a bit more complex to operate. So it's not for everyone but it adds more options to choose from when running Thanos, especially in the case when you don't have direct access to Prometheus servers for example. Next slide please. Why you can say that Thanos is simple and flexible? So first of all, all those three deployments we mentioned kind of showcase ability of the Thanos project to be shaped and tailored to your needs. In fact, you can have a mix of those three deployment models under a single centralized system which many of production users are doing. Furthermore, it is simple because you can incrementally deploy Thanos once you or your business model progress. This also really help avoiding scope script during the immigration process and help with initial experimenting as well. One of the Thanos goals is also to make sure we keep low maintenance costs. This aims to have simplicity and level of reliability close to the Prometheus where you don't need a full team to maintain it and operate. Essentially, you only want to access it when you need it. We are also focusing on having very simple and unified APIs. The query layer does not need to understand from what place the metrics are. What place that the metrics come from. They can be fetched from Prometheus directly or object storage or another query layer or from totally different solution. For example, some company built and maintain OpenTSTP integration. This simplifies things a lot and allows further customizations. We also built Thanos while reusing and contributing as much as possible to the Prometheus code base. Thanos is not meant to reinvent things like query language. We are here really to make those pieces more distributed, a bit more cloud native and scalable. And finally, it has essentially single and optional dependency, which is an object storage. Next slide, please. So how we look in terms of community. We are fully open source under Apache to license. We establish code of conduct. And to be honest, we are quite surprised with the grow of the community and the wide adoption. We are quite popular project on GitHub. We are extremely grateful for a really large amount of external contributions. We hit 117, I think unique contributors mark. We have a Thanos website with roughly 100 daily viewers. We have a Twitter account for announcements and very active Slack channels with more than 600 users. We are also excited that we were able to build quite large maintainers base. There are five maintainers and three official users that helps us to triage the GitHub issues. We believe this is a quite big responsibility and hard work. So it shows a big commitment from everyone involved. Also the unique thing about our team is that almost all of us are from the different company, which is nice. Finally, it's worth to mention that although we are an improvable organization right now on GitHub, we already reserve Thanos IO organization and we are slowly moving to that. I think donation to the CNCF will definitely accelerate this process. Now I believe it's time for Frideric to tell us more about Thanos adopters and production users. Thank you, Baatek. Next slide, please. Thank you. So we have a number of production adopters in our call here. So Shang from Alibaba and someone from Monzo and Ben from GitLab as well. So this is just a selection of companies that have been asked or have asked us to be on this presentation. And some of those are also in this call. Should you have any questions for them? So I think one thing that's very interesting about the production adopters is the kind of model in which they run Thanos. So in comparison to many of the other projects out there in a similar space, Thanos is almost exclusively used to satisfy these companies, the metrics needs for these companies themselves as opposed to like a software as a service model, for example. So I think this is kind of unique to the Thanos project. And we think why, or it's a reason why a lot of companies are using it because Thanos works really well in this kind of case. Next slide, please. So a little bit about the history. So Baatek who just presented and Fabian Reinhardt were the original creators at Improbable in late 2017. And in early 2018, the project was first publicly announced at the London Prometheus user group. And while it was launched there first, or announced first, I think what really kicked it off for the community was the S3 object storage support which was added in early March, 2018. And I think this is kind of a milestone for the project because Improbable nominally didn't need S3 support and this was entirely driven by the community. So like two months after the project was first announced, the community started contributing to this in a really meaningful way. So I think this is really awesome. And then in March, 2019, first maintainers outside of Improbable were introduced and now we're here proposing the project to the CNCF sandbox. Next slide, please. So there are a couple of alternatives and there are of course, a lot more monitoring systems out there. We decided to keep the list to those that are most closely related to Thanos. So there is Cortex, which is already a CNCF project. Cortex is quite similar to the third deployment option that Bartek showed us where essentially Prometheus replicates its database to a remote place and Cortex ingests that. And M3DB created by Uber has a distinct integration mechanism with Prometheus, but it is able to handle those Prometheus data as well. Victoria metrics is kind of similar to Cortex in the sense that it has a similar API and a similar deployment model. And InflexDB, which nominally wasn't born in the Prometheus world, but also accepts the remote write protocol, which is the protocol that the other projects on this list here use to replicate this data. Next slide, please. So how does Thanos relate to the other CNCF projects? So we've already mentioned it a couple of times, but I think it's worth calling out again. Thanos makes use of absolute vanilla upstream Prometheus. So there's absolutely no modification. You can take the released hardwall, it's created by the, or released by the Prometheus project or the container image is published by the Prometheus project and put the Thanos components next to this and essentially grow all of this from your vanilla Prometheus setup that you already have and grow your setup into this distributed Prometheus, one could say. And everything is built around this as opposed to modifying or forking. Then Thanos makes heavy use of GRPC and this is not just because GRPC is CNCF thing, but we actually make heavy use of screaming, for example, and this heavily benefits to the performance of the project in a very positive way. All of Thanos communication internally and externally is instrumented with open tracing and Yeager and Google Cloud Tracer are the two adapters that are currently available for this. And while not tied to Kubernetes, there's, Thanos was kind of born into the world of Kubernetes. So a lot of the examples are for Kubernetes. There's a direct integration with the Prometheus operator for deploying Thanos with the Prometheus operator. There are helm charts, customized templates and a bunch of blog posts describing how to run Thanos on Kubernetes. But I think even though there is all of this relationship, I just want to point out, Thanos is in no way tied to Kubernetes, but it's just thriving within that ecosystem. Next slide, please. So we make heavy use of Prometheus, right? So how does Thanos actually stand in terms of a relationship with the Prometheus project? So a number of pieces are actually literally rendered. So in order to be able to compatible with the Prometheus project, we don't actually need to spend much time on that. We literally vendor the same code for the PromQL engine, for example. And the time series database is also exactly the same as what Prometheus uses. And so we're not here to reinvent the wheel, we're here to make a distributed version of Prometheus. And so like the time series database, the query engine and the alerting engine, all are exactly the same. And as a matter of fact, Bartek and I are actually both also Prometheus 14 members and Thanos maintainers. And aside from that, other maintainers of Thanos are also heavily contributing to the Prometheus ecosystem meetups. These are online meetings that take place every month where the community meets and can ask questions and discuss things. And this is also led by one of the other Thanos maintainers among others. Next slide, please. So why do we think the CNCF sandbox is the right thing for Thanos? So I think first and foremost is the neutral ground. So a number of companies have approached us and have said we would really like to contribute to the Thanos project, but we want it to be on neutral ground for us to, for that to happen. In the same way people want to talk about how they're using Thanos within their company, but they want a neutral ground to be able to do that. And a further one is there have been discussions with other projects in the CNCF about potentially merging efforts. And this would be a safe space for our projects to explore that. Now, whether that will ever happen, we don't know, but it's a safe space for us to be able to explore those ideas. And as we've already said, we have a really strong relationship with already graduated Prometheus project. And even there, we've had discussions that maybe one day these things could be folded back into the Prometheus project. And again, we don't know if that's going to happen, but for that to be a possibility, we would need to be part of the CNCF and be on a neutral ground together. And last but not least, we think Thanos fits really well into the portfolio of the CloudNet of Computing Foundation project. As I think we've already shown, we have a use of these technologies and just I think it's a benefit to the ecosystem to extend this portfolio with Thanos. And next slide, that's it. We are seeking one more TOC sponsor. Shang generously has already offered to sponsor. So we are looking for at least one more. Of course, we would be very happy with more, but that is the end of the presentation. And if you have any questions, we're more than happy to answer those. Thank you. Thanks. I'm actually curious to know why Sandbox and why not incubation level. What do you think you don't currently have that meets the incubation criteria? That was my question, Liz. Oh, we think the same, Joe. I don't know. I don't think we've actually discussed it that much. I think we just thought that that's the entry model. So I think we could discuss incubation more if you think that would be more appropriate. I think there's certainly a large user base already out there. Yeah, I think the user, certainly brand names is really impressive. That's really nice to see. So yeah, I think we'd be more than happy to already see it in incubation. I think you have sort of the activity. You have the diversity of contributors. You have the production usage. That ticks a lot of the boxes. I was just going to say I'm happy to sponsor also. I know Alexis just put it on the chat. I would be as well. I mean, I think it's pretty much a clear, definitely going to get into Sandbox and perhaps we should have a conversation about whether there are any criteria that need looking at before incubation. But I think Sandbox, any questions, any thoughts of why we shouldn't? All right, congratulations, Thanos. You are in the Sandbox. Thank you. Thank you. All right, is it Kiefer to our next? Should be. Oh, there's some append slides, but we'll skip those. Go ahead, Fabian. What was that? We missed, was it? Okay, whatever. Hey, hello, this is Fabian from Reddit. Could I kindly ask you to reload the slides? I just added some additional comments here and there. Awesome, great. So here we go. All right. So hello, I'm Fabian from the Kiefer project. And that is the second slide. Please go one slide back. Backwards, please. That's you're going forward. Yeah, there we go. All right, so Kiefer, Kiefer was started in 2016, so it's already quite old. The main driver back then for us to start with Kiefer was that we wanted to have a single orchestrator for both compute form factors, right? We knew that containers were coming up but we also saw that VMs are still around. And with Kiefer, we said, right, that's fear, but we don't see that all VMs are going over to containers. So we need to provide a platform to run both. And this platform should acknowledge the differences between these different form factors, right? Both form factors have different properties and different life cycles, different workflows. And we want to be able to acknowledge both and properly handle both. In general, the pragmatic statement we came up with back then was to say, even if we want to run virtual machines inside of Kubernetes, it is important to us that we maintain the Kubernetes or cloud native look and feel when working with those virtual machines over having all the virtualization features that exist in the virtualization world today. For example, the CPU memory hotplugging is one of the examples which might land soon in Kubernetes or something like that, but which wasn't clear back then. We said, so we don't provide those features in order to not conflict with Kubernetes too much. One outcome of saying we want to adjust both workloads was also that we could unify the stack, right? So we said, we are interested in bare metal deployments and having VMs and containers on the same platform on Kubernetes would allow us to also unify the underlying infrastructure. So storage networking and the supporting technologies like authentication logging, metrics, and so on and so forth. After a small research period in 2016, we actually opened SourceCupid in January of 2017. We moved along and we did releases from the beginning. We saw a little bit of community attention which we'll get to in a moment. All of this was released under the Apache 2 or is still released under the Apache 2 license. Next slide, please. So what does Kuber do? For most, Kuber provides a comprehensive API to run virtual machines on Kubernetes and those are virtual machines as you know them, right? So if you're running VirtualBox or VirtualVertManager on Linux or VMware Workstations, you know, these kinds of VMs are what we want to run. It's also the same VMs you would be running in OpenStack, for example. We have different properties or we are inheriting the properties of scale, for example, from Kubernetes. So if I mentioned OpenStack, it doesn't mean that we want to meet all of the OpenStack or other projects requirements, but we say we want to run virtual machines in the constraints that Kubernetes is providing to us. The API today is supporting to define virtual devices as you can do with other virtual machines. We can do live migration in a Kubernetes-friendly way so we don't provide live migration. We do not provide live migration for PODs, but we do provide live migration for virtual machines. We allow to have multiple NICs, which is actually backed by multis. We would just support putting from raw disk images and we can put range of Linux distributions and actually Microsoft Windows, which was also one of our goals back then. In general, this is based on LibreD and KVM and QEMO, so a well-proven set of technologies, which is also used in other projects like OpenStack and Overt. And this also means that we inherit that robustness which was brought to these projects over the years. Qvert is also Kubernetes-native application. What does it mean? That Qvert itself tries to integrate with the cluster, so we are an add-on to Kubernetes, right? So you deploy Qvert on top of Kubernetes and then you can right away reuse the cluster resources like storage, network services, but also the compute resources, right? So wherever PODs can run. In addition, we can leverage node-level features like the CPU manager work or multi-network, huge pages, empty-deer and block storage. All of these have been Kubernetes features which came into Kubernetes over the past years and we were improving a Qvert in order to allow us using those features as well for VMs. Sometimes this requires some small glue in order to make a feature usable for virtual machines, but often it's also just a path through. For example, huge pages. It's really simple to address or to provide that feature to virtual machines because it's so simple to adopt the virtual machines to the feature provided by Kubernetes. Other features are more difficult to consume, for example, CPU manager, and that is where we actually still have it on our list or we were already engaging in the upstream discussion to also see that we solve the problem this technology is addressing for both PODs and for virtual machines. And furthermore, we're just not on the integration with Kubernetes, but we also try to see that we integrate into the ecosystem. So as Prometheus was also mentioned early on, Kubernetes is also providing metrics, virtual machine specific metrics in Prometheus compatible endpoints. On the other hand, it also means that the VMs can actually also be seen through the regular Kubernetes metric endpoints. If we then look at further how it's being implemented and how we extend Kubernetes, that is by using the all state of the art technologies like CRDs, the custom API servers, administration, webhooks to do validation and mutation. We use client code GRPC, which is also a CNCF project. So there's a range of projects we pick up. We actually try to see what we can pick up before we build it ourselves. All of this leads to a situation where we can really do a simple cube CT to apply minus F or three times at least. And then you extended your Kubernetes cluster to be able to run virtual machines. This actually was proven because we see that in our community we have a range of Kubernetes distributions and for them the approach we took is working. So we are seeing very few issues. For example, if you try to run Kuber on a cluster which is running on Ubuntu or some other Linux distribution. And over time we were able to, we think we were able to maintain the Kubernetes native look and feel and behavior you're expecting from pods and other entities in Kubernetes itself. All right, please move along to the next slide. This is a great architecture diagram. I don't wanna go too much into it. So what we're seeing here is basically that we have the usual operator pattern. So Core is coined as operator pattern. But in the end it's really the controller pattern that Kubernetes is using for all of its controllers. So if a custom resource which is watched by controller which is shown in the lower left and speaking to a note agent called the vert handler which is pretty much in the middle. And then interesting is that we use that we ultimately run the VM which is a little bit on the right of the middle is that the VM is running inside a pod. So that makes it actually transparent for a lot of projects which deal with Kubernetes and provide additional features around Kubernetes to those projects it's transparent if it's a VM or if it's a pod because to them it just looks like a pod. Yeah, next slide please. So the use cases, what do we wanna do? I think the simple one is to run virtual machines to support the change. So we saw that people always assume that people want to go to containers. Containers have the use case after all, right? But the reality is that all the VMs which were built over the last 20 years or so not all of them can be moved to containers. So we need to provide a way to support the people in order to do the change where it's possible to move to containers to increase their efficiency and to lower the footprint. But we need to give them an escape hatch for stuff that cannot be moved to containers, right? So very old code or a code that cannot be moved due to licensing issues. This is actually the two use cases you're seeing at the top. The other I also mentioned that before is to have a single stack, right? That we want to have a single control plane and a single operational model to manage both virtual machines and pods. It's true that we provide different APIs for pods and virtual machines but nevertheless, all the surrounding a look and feel and tooling can be used to control both. So Kube CTL is the or Kube Cuddle, depending on who you ask. It's a tool of choice to manage both. We maintain the stateless approach, state declarative approach which is also used for all the other entities and Kubernetes itself. All right, next slide please. What was surprising and what we didn't see initially was that Kube is now also used to run Kubernetes actually for a hard multi-tenancy use case. We saw actually a couple of community members who started to use Kube in such a way to run Kube on a bare metal cluster and then use it to provide additional Kubernetes cluster to their tenants with an hard isolation because these layered Kubernetes clusters are then run in virtual machines. This also led to external projects like the Kube Cloud provider which now actually allows a tenant to introspect the underlying Kube cluster that's shown on the right hand side. On the left hand side on this slide, we also see another use case for Kube which has been over discussion and is partially used by community members in-house to use Kube to run virtual network functions. So we know that VNFs have a long history when it comes to OpenStack for example and the people are under pressure to see how are those VNFs are used in a container native world? How can they be run? We know that CNFs are coming up to move stuff to containers but they have their own challenges. In containers, it's hard to have your own kernel modules or then you're very constrained. So there's a problem for VNFs but also CNFs. And here, Kube can also help because you can take your existing VNF, move it into a Kube VM and run on top of Kubernetes and then integrate with stuff like the network service mesh, multis or for example, Calico. We do not have all those integrations yet but we surely have them inside. Next slide please. So far we talked about the features and how Kube is used. Now let's take a look at other projects in that space in the Kubernetes ecosystem. This is the small comparison of them. So if we look at Kube, what does Kube characterize best? And I think two things. First, the API, how do you deal with the workload? And the second is, what are you intended to do with that API? For Kube, we say we haven't dedicated API, that's why we say a custom resource. So we have custom resources to work with the workloads and the purpose is to run VMs. If we look at Cata containers, device of Firecracker and friends, here the purpose is a different one, right? These projects look at isolating pods or containers in order to isolate them from the underlying node but the API users working with is a pod. So the workflow and the use case are still containers. The next project I want to compare us to is Vertlet. This is the purpose of that project is also to run VMs. They're specifically focusing on cloud workloads but here the drawback is that they're using a pod API. The issue is that you cannot really use the pod API for example to provide virtualization specific features like connecting to the graphical console or supporting live migration. That is much uncleaner to solve if we're reusing the pod API. The last project I want to mention is Rancher VM which is today also providing custom resource to run VMs which is smaller in scope but the problem we're seeing with the Rancher VM approach is that they're currently not focused on integrating with all of the Kubernetes features but rather focused on a more streamlined experience with certain storage back ends and for example, networking plugins. Yeah, that's a short comparison to other projects. Next slide please. I mentioned the Kuvert community and a few more words on those. So first there's a CNCF Sandbox PR app on GitHub. It's quite fresh but please take a look. We've got about 1,300 GitHub stars continuing to increase. We have a lot of contributors from RedEd that is because we are interested and we're going to, yeah, we want to see how we can use Kuvert and some of our products but we also have external contributors like 17. All of these contributors created about 1,500 pull requests, 270 GitHub forks and we have actually now 19 releases with the July release. All of the community can usually be found in the community meetings or in virtualization channel on Slack and on the Kuvert mailing list which I forgot to add here. Some of our existing users and contributors are Akamai, Apple Cloud for Cisco Loots, OSI, RedEd, SAP and StackPath. The checkmarks indicate from whom we know how they're using it. So we see that we've actually got a lot of users and quite many of them are also contributing to it. All of them are using it for different use cases. Loots and SAP for example, they worked more on the hard multi-tenancy model with Kuvert and did some adjacent work in other projects. That's why the checkmarks are in brackets. Next slide please. So why do we want to donate Kuvert to the CNCF? We see that we already have a wide community for Kuvert and we would like to have a neutral ground for that community to continue to work on Kuvert and make sure that there's not the impression that RedEd is steering it to its own likes, but rather have a more neutral ground. We also want to give Kuvert more exposure and hope that if it becomes the CNCF centerpiece project that the visibility will be higher. And ultimately Kuvert itself can also be seen as a building block. So it can be used to run traditional VMs. We actually have our, we are developing a UI for our use case to be able to run classical VMs on top of Kubernetes using that UI and the CLI approaches. But with the hard multi-tenancy use case, we also see that Kuvert can be a building block for other use cases. And we hope that by making Kuvert a Sandbox project, we can highlight that can be a building block for example, also for the VNF use case. Next slide please. So that's it from my side. Kuvert is still looking for TUC sponsors. And yeah, I would be thrilled to hear what you think about that and if there are any volunteers. Awesome, any questions from folks? I had one, it's Quentin here. Presumably these virtual machines that Kuvert runs require a lot of the surrounding infrastructure that pods have. For example, in a replication controller as a replica sets, demon sets, jobs or all of the sort of life cycle management stuff that pods have. Do you sort of implement a parallel set of different controllers or do you reuse the existing Kubernetes ones? How does that work? Yeah, so we had that question in the past and we've played a little bit with that. But to be honest, so far our focus was not so much on enabling the use of those controllers. We have a few ideas how that can be implemented but currently we don't support the direct use of the high-level workload controllers. There are some, again, there are some workarounds how you can enable that and we have some preliminary work to enable the native controllers and we actually have a VM replica set which is replicating what the replica set is doing for pods. Again, I think the story is not answered is not concluded but mainly because it's currently not in the focus of one of the community members. Sorry, just to flip the question around, perhaps the other way is could you explain why you didn't just expand pods to be able to contain VMs as well as containers or take that approach in which case you would have kind of inherited all of the additional Kubernetes infrastructure. So by the way, I would like to see that. So first, the VMs are running pods. So that's not the problem. That problem is, we technically have is that the entry points to define VMs are custom resources. And if you look at the diplomas and other high-level workload controllers to the Kubernetes, then the issue is that you can only provide templates for pods. So that means the entry point is the pod, it's a pod entity, right? But in order to work with Kuber, you would need the VM entity to be the entry point that's currently not possible. The thing we discussed back then was to say, couldn't we create or couldn't we allow the high-level workload controllers to template other entities as well? For example, virtual machine power case, that did not fly. I would actually look if we can re-raise it in these times again, because we've seen, for example, with the garbage collection or the eviction API, that there are now examples where contracts exist, for example, the eviction API from the garbage collection, for example, which also works with custom resources. So maybe the timing when we asked to have templating for custom resources was not the right time. And maybe that question would be answered differently if we get back to it today. Does it answer it or was it too or both? Yeah, sort of. I mean, it seems to me like if you could rewind the clock and do it again, you might actually put the VMs inside pods and then you would get all the other stuff. But that both said a long time ago. Thank you. Sure. Do we have any users of Keeper on the call? Let me check. I see some from our side, right? I didn't advertise it too much in the community yet. So I did not invite them and they were not aware that this is taking place right now. They were aware that we are going to sandbox it, but I did not advertise the specific date and time. I was just curious to see if there was a groundswell of support from the user base saying, yeah, bring it in. Yeah, so, yeah, that's obviously, I mean, that's really unfortunate that I didn't share that. So we obviously asked them and we have, we can, I mean, we definitely have to support, for example, from Lutz and there are a few others, but I don't want to say names now because I don't have the emails in front of me. So I can bring up the names which are supporting to bring Keeper to the CNCF sandbox, but there's definitely support there. Otherwise we would have not done the step to propose it here. I'd recommend they just comment on the PR so we could do it offline. Okay? Yeah, definitely. I think seeing that I would be, you know, interested. So, yeah. Cool, I think. Right, we are getting short of time. Can we get through in total in 10 minutes? Yeah, we've got 15 minutes for the last. 15. Cool, thank you Fabian. Hello, can you guys hear me? Yeah. Okay, I'm Santiago, Justin is here with me and we're going to talk about the incubation application for Intodo. Next slide please. For those who are not familiar, Intodo is the first framework secure software supply chains as a whole. It boards in and outside of the cloud, but given that the cloud is probably one that stress tests this environment the most, there are diverse environments with like multi-tenant hosts and different components working loosely in loose connections. Intodo allows you to create a policy that can give you security assurance and compliance checks and not its trail that's cryptographically verifiable. I'll explain a little bit how this works because I think it's very important to, in the context of why Intodo is important and how it can be very powerful to secure the cloud-medium environment when creating artifacts. Next slide please. Intodo basically is formed by three components. One of them is what's called a layout. You can think of it as similar to a Jenkins file. It specifies all of the steps that need to be in place and then it also specifies the interrelationship of the artifacts as they flow through the supply chain. For example, in this case we have a version control system that creates some sources and then it sends them to a CI-CD system and also to a build farm. The build farm is blessed to create the final binary that's eventually going to be packaged into a devian package. The layout also says who is allowed to perform all of the operations in the supply chain. For example, in this case, we have the public keys of Bob assigned to the DCS step and the public key of Carol and Erin for the building and packaging, respectively. Next slide please. Now, while this layout is created, sign that the stations as evidence of each step they can place are also created by these functionaries. That's how we call them. There are basically a piece of metadata that says, hey, I did this operation. Here's my signature to rubber stamp it and these are the artifacts that I created. This is important because that's how we compare the policy that was created earlier with what actually happened. Next slide please. This is actually used when you were verifying. The end user takes in the layout, it takes the final product, the final artifact and the series of rubber stamp at stations and it walks down the paper trail to see that all of the operations that were meant to happen, they actually happened. Next slide please. Now, I think it's easier to understand how Intodo works with some integration case studies. The first one that I wanted to talk about is the reproducible builds project which is a part of the Devian and Arch. We are using Intodo to verify that all packages created in the Devian packaging infrastructure are reproducible. And not only that, it's possible to reproduce these packages in different environments and verify that upon installation. So we have a, in this case, for example, the project owner, the one that creates the layout is the Devian developers. They sign a layout saying, hey, this build step must be reproducible and there needs to be a threshold of n signatures. Right now it's two, but we're trying to bump it to a higher number as we get more rebuilder organizations in. And then these rebuilders are the functionaries, the one that will be rebuilding everything and creating rubber stamped at stations with their private keys saying, hey, I rebuilt this package. This is the link metadata for it. And finally, the client in this case is the app get installer. We have an app to transport that you can use today to download these packages and actually verify that all of the packages that were installed were reproducible and they were actually reproduced in different infrastructures. Next slide, please. Another example is the control plane people. I don't know if they're on the call, if you want to say hi. I invited them a little bit late, but as part of the lift and shift operations, they're moving legacy applications into the cloud and as an add-on, they're also using in total to verify that all of the operations that exist during this like Jenkins pipeline plugins, this is a very cloud-native setup, are actually a testable using in total metadata. So in this case, for example, a release manager within an organization can create the total layout, sign it with their private key and then send it to the in total Kubernetes and mission controller. Then the Jenkins plugin will start asking all of the slaves to create in total metadata and then relay it to the admission controller. Once new image is about to come into the cloud, the admission controller will verify all of the satisfactions and the layout and make sure that everything's working correctly. And if everything's okay, then the container makes it in, otherwise it doesn't. The third integration case study that I wanted to talk about next slide, please is a data dog. I also wanted to have Trishank, which is the one that made this happen, talk a little bit more about it from the data dog side. He's a security solutions engineer at DataDog and I think you're under the call, right, Trishank? Yep, hi guys, can you hear me? Yeah, yes. I wanna try to share my screen, but unfortunately I can't. Got like nine minutes, let me try to, here's the blog post so that everyone can see. Is it possible for me to share my screen? You should be able to do it now. I think we like it. Okay, one minute, share. Okay, great. Okay, let's do this. So this is the blog post, I just posted it. We wrote a paper about it that's been accepted to YouthNIC, so all the nitty gritty details, but let me quickly walk through everyone with how this roughly works, right? This is gonna be great, it's eight minutes. So okay, but basically we use in-toto along with another project called Tough and we'll talk about how they both relate each other to build what we think is the industry's first untrusted CI CD system, meaning, you know, we don't use any special trusted hardware. We can use any generic CI CD system in the cloud and we don't care if any part of it is compromised between our developers and end users, right, we get this security feature we call Compromise Resilience. I'll talk about how and what we did this for. So Datadog is a monitoring company, we collect your metrics, your app performance, your logs and so on, and you use the agent installing your hosts and containers to do this, right? And so integrations are just add-ons of plugins that you install and give the agent superpower so that you can monitor now Kafka, for example, or you can monitor NGINX and so on, right? So, and we've got hundreds of distincts that come out of the box. A challenge was that we bundled all of these hundreds of integrations with the agent every six weeks and we wanted to decouple them so that people can install new integrations, just new add-ons or plugins out of cycle, right, out of the agent cycle so that they can test new features. But the problem is that the state of the art, basically everyone uses what we call online keys, basically something like DLS, where you've got robots, what I call robots, signing your code for you, automatically building and signing your code and distributing it to users instantly, right? This is great, your developers don't have to worry about reproducibility, handling code signing keys and so on, but the downside is that what can go wrong? Well, if someone compromises the developer keys or they compromise your GitHub repository, let's say, or they compromise into your CI CD, your container image registry, you get the ideas, basically instant compromise. There's so many of these pieces and all it takes is one piece and your robots automatically build, sign, and distribute malware to your users. We don't want that. We want a feature that we call compromise resilience, which means that even if our CI CD is infiltrated, we're okay, our users won't install malicious software. And so we do this using Intoto, which gives us end-to-end guarantees about a software supply chain between our developers to our CI CD, we use this stuff, which is the CNCF project, which is the relationship between them, you can think of a metaphor that Santiago used, which I quite like, which is that Intoto is like, so if you think about a bottle of drugs, Intoto is the thing that tells you, oh, this person made this ingredient, this person made this ingredient, and so on and so on, composing it. And tough is the plastic seal around it, making sure that things are not tampered with. I don't want to talk about all the details, but basically a rough security analysis is that, what can go wrong? Now that we've installed Intoto production, this is actually being used today. I'm going to show you a quick demo. If our developer keys get compromised, yes, theoretically people can cost malicious software to be built, but we use UV keys, which are this hardware tokens of trust that basically store our developers' GPG keys so that even if there's malware compromising the user, we have a secret pin and we have touch authentication to make sure that it's not instant, you would know something funny is going on, and you can actually trade the signing keys. Furthermore, you can use threshold schemes like requiring more than one, so you can say three developers need to sign off source code before it is trusted by any users, right? But the interesting thing is that now, we don't care if our GitHub repository is compromised because signatures won't match, wouldn't match whatever developers sign, right? And the keys are stored on hardware tokens called UV keys. We don't care if our CI CD is compromised because we would know signatures wouldn't match. Same thing, continue image registry, file service and key servers, you get the idea. Let me show you a quick demo of what I would, and four minutes. Yeah, I think it might be wise to add to Santiago. Yeah, I would just go back and finish up. Sure. It's just a pleasure to meet you. Hi. Sorry, that was... We'll have only three minutes, so we'll probably skip the roadmap. Can we go to the community and other information slides? Yes. So Michelle Morales is sponsoring us. We want to be in incubation because we've seen enough production used to know that this is not an experiment. We are looking for visibility because we come from academia, so compared to other projects, we don't have as much exposure with an industry, but well, that's a conversation that we can have a little bit more in detail. We also, we have this URLs for the website. We have three releases per year. We have 100 plus stars now. We're having a lot of exposure now that the supply chain security compromises are becoming more prevalent. And actually, these numbers are on contributors are a little bit wrong. The last two or three days, we've had two more contributors come in, but all in all, we're having a, I think we're having a very vibrant community that we're building up from different places across academia and industry and open source communities like Debian, Arch Linux, Open Susie. We're also contributors from kids, reproducible builds, so on and so forth. I wanted to just, next slide please, just give a quick snapshot of all of the places that you can contribute if you would like to. And I also wanted to give a quick plug on the next slide please. We were the first project that had the security assessment. It's been a year long project process. We went through many iterations and it was actually a very amazing experience which we were able to tighten and verify and review all of our security practices. I think this is very important for our security project. They, this is a slide that they created that basically says what they think about the state of the project. They think the design is straightforward and adaptable. That was like a paramount for us since the beginning. They, I think they approved and they liked our security analysis. We also have a spin-off project came out of it, of cataloging and tracking software supply chain compromises that probably will kick off soon and under the six security working group and they made recommendations on have the CNCF assign a US researcher and have the CNCF help us with resources on certain areas. Well, one many ways. And one last thing is that on this slide if you go to the actual deck I've added a link to the full directory with the longer assessments if anyone wants to see the longer version of the one slide summary here. Right. And that's about it. I'm going to open the floor for questions and discussion because we're on the 12 PM mark. I'm just gonna add that I know this kind of supply chain assurance is really important part of the puzzle. So I'm supportive of this. I think we need to do a little bit more digging. I think the space and the project looks like it's on the right track. I think sandbox versus incubation for me hinges on some of the... I'd like to understand more about the governance and sort of the depth of the bench in terms of like, is this all hanging on a couple of folks or is it something that will be sustainable? And I think those are some of the things that we look, you know, at the... At least I'm looking at it across different levels here. So, but we can take that up offline on the issue for sure. Okay. Dozen. Well, should we call that a day and take in total sandbox or incubation discussion offline? Oh, to the PR. All right. Thanks everyone. Thank you everybody. Thanks everyone. Bye. Thank you, bye. Thank you.