 Okay, let's go ahead and get started. Can you full screen that, expand the slides? Welcome everyone, another CNCS CI Work Group meeting. This meeting is being recorded. And if you would like to, again, add any notes, feel free to put those in. Let's go ahead and see the first bit here. CrossCloud CI updates. So this is on the CNCS CI dashboard. Next slide. We've added a new project in Envoy. It's been added to CNCS CI. There's gonna be a talk about this one, adding new projects. Oracle Cloud Integration is in progress. Right now working on integrating the storage components and other parts for Oracle. And then some minor updates for various versions, projects that continue to be in progress. So saying adding new projects will be presented at Shanghai QCon and trying to get more projects to participate in adding themselves and contributing. We'll also be updating the docs to, this is part of getting a quick start for using CNCS CI. And we're planning on some updates for the overall dashboard, the UX. One of the things that is rolling in place and should be live soon is showing NA whenever it be an earlier step fell, then we'll show that status badge would be NA. Next slide. Also working with several groups on using the various technology behind the CNCS CI, the network service mesh working group is using the provisioner, the Kubernetes provisioner called CrossCloud. That's what does the multi-cloud provisioning and starting to use that for testing. And we've also been helping the network service group directly on some of their other CI issues and testing for the project across different network cards and other things. So it's not just cloud specific, but some hardware specific things that the group has still have. Taylor, until we get brown integration, VMware is using a version of the provisioner to do our out of tree cloud provider testing, setting up the cluster or the fork of that. Oh, that's cool. Is that using the version that can? It's not directly using it. We forked it and then made some changes to be more specific to us, but it's the provisioner's the basis of it. Nice. Yes, it's nice to see more groups starting to use the underlying tech fork or not. Next slide. So this is outlining some of the thoughts for next goals, probably a big focus on the UI and having some more screens about projects, the clouds, and then looking at some things with user scenarios where we actually do integrations between the projects. This lists some other things as well, including switching over to Sonaboy and some other items like QBDM. Can I make a remark about ignition? It's funny because I'm the one that pushed y'all to look at that. But with everything happening with CoreOS and Red Hat, and I'm not sure what the direction is there as an outsider, funny enough, I switched to cloud and knit because ignition only exists on CoreOS and freight car, whatever the open fork of that is or the non-CoreOS fork is. So it would lock y'all into it. I didn't realize at the time, it was locking people into a distro that everybody was using CoreOS. So I'm not sure what the future of that is and if that's something that should be considered with respect to what tool to use. Yeah, thanks for that, Andrew. I think what we're thinking is it would be an additional support. So we have cloud in it and then optionally have direct ignition support. Gotcha. And there's some other things, but this is kind of what are the parts of the tech that we may adjust putting time QBADM in place for the actual bootstrapping of Kubernetes would probably be something on the sooner than later list. The ignition or anything else is probably further out since like you say, it's a CoreOS specific. Now on the QBADM is that's, you're saying that's not reliant on the future of the cluster API provider, right? You're looking at it to just bootstrap Kubernetes, but eventually maybe if there are enough cluster providers, it could also bootstrap the platform as well. Potentially we could go as far as using the cluster API and with QBADM if when and if it gets to that point and it looks like that. Right now what we're looking at is the after the Terraform provisions, the resources, how do you get Kubernetes installed and brought up and have the cluster connected. So probably be looking at time QBADM at that point and then as new features make it to production as far as in the QBADM space, we start looking at if it is a good match. Okay, next slide. Working with a lot of groups like the cluster API, just mentioning OpenCI, a lot of the other CI CD groups that are in CNCF and Linux Foundation. We are, some of us that are on this team are actually at ONS and Amsterdam. So we'll be meeting with more folks that are here. If you're here then reach out. Some of the time we'll probably be at the CNCF booth. Otherwise you can hang us on Slack if you're in Amsterdam. I'll be there in the morning. Awesome, we'll be at the booth every day for some period of time. You can catch us on Slack otherwise. So upcoming events, we are attending regularly the network service mesh meetings, as that's gonna be a big feature added into Kubernetes. And we're helping with some of the testing on CNS. The Kubernetes conformance working group trying to keep track on that. So we're part of that. We will be QCon China and doing an intro and deep dive that's a focus on the project, adding projects. And if anyone's interested and you're there, please join us. And then we'll also be given a talk in Seattle. Potentially the same, but we have some other interest and then we'll be going to EnvoyCon as well if possible, which is right around the same time in Seattle. That's it for us. Any other questions before we're gone? Okay, and the rest of this is kind of an overview of the dashboard for anyone that's not familiar. If you are, we can just move on. CNCF dashboard builds and goes through the builds and then provisioning the cloud providers and then running tests for all the apps that are supported like Flounder, Accord, DNS and ONAP, which is a Linux on this project actually. That's it. Thanks everyone who helped put this together. So, Andrew, are you ready for your demo? Unfortunately, yeah, I'm in the middle of trying to get stuff working, an interquarter, right? So yeah, let me go ahead and show my screen. Sounds good. Hold on, let me zoom this being zoom. If I share my screen well. So when you go full screen, my zoom automatically goes full screen and then since I'm on a Mac, it takes me to a desktop, a new desktop and then can't share it from there. So that's working. I don't know why this isn't. Anyway, all right. So I have created two things, one of which is probably unnecessary, but I've been learning. So, yeah, zoom, thanks. I've been learning. So it's been a good process for me, but so the two things I've created are Yaketi and Yakitui. Yakitui is formerly known as VK8 conformance. Yaketi stands for yet another Kubernetes installer thingy and it very closely follows Kelsey's hard way, right? So for the deep dirty secret about me is I worked on Docker as a developer for a year without ever running Docker. I worked on the storage and didn't have to touch Docker. I just wrote the components. I've been involved in Kubernetes and recently wrote my first job spec. So I'm still learning and it's a lot to learn. So part of this was learning, but what it does, what it came out of it, I think it's something interesting that could be useful for other people. It is essentially a giant shell script and that's okay in my opinion because I use shell check to ensure that it's both positive compliant as well as there are no errors. And so what does this shell script do? It stands up Kubernetes. Now, why is that interesting? Well, in the example here, you can take two completely disparate Amazon instances that are in different networks. And if ahead of running this, and this assumes you've got the hardware up, Yakitui is for standing up the hardware. This is the VM already exists or the instance exists. You get a at CD discovery URL with the size of the number of nodes in your, sorry, the number of controllers in your control plane, number of control plane nodes. So at CD discovery, I'm gonna get a size of one. That's the number of control plane nodes, right? And then you basically run Yakitui on both of your nodes. One is the controller, number of controllers and it says it's a controller and it says, but the total number of nodes is two. On the second one, you say, hey, you're a worker, there's one controller and there are two nodes. So basically the same on each, except one's a controller, one's the worker. So what happens when you do that? I actually could take you through it. Takes a little, yeah, let me move some zoom stuff around. Do I have something? Yeah, so Yakitui, yeah, so if I do that, will that work? So, and now it needs to be the name first. Andrew, hopefully that doesn't blow up. So that's running Yakitui, which bootstraps the hardware. But what makes all this interesting is that it synchronizes the communication between the nodes. And what I mean by that is that they discover one another. So when you stand up Kubernetes normally, if you're using CUBE ADM or you're using any other tool, you need to start standing these nodes up in a serial order, you see if one up and then you can start to stand the others up. With this, you could stand them up all at once. Oh crap, I used the load balancer, should have disabled that. You could stand them up all at once. And what happens is, here I can just take you through the walkthrough down here while it's standing up. All right, downloads the Kubernetes binaries, but creates the admin group, IP tables. Sorry, I didn't have a better presentation plan. So it discovers the SED cluster members from that discovery URL. And because it knows how many controller nodes are supposed to exist, it can, every node can wait until they all appear, until they've all joined the SED cluster. Then the node configures itself with SED control and it uploads information about itself to the SED host. Then all of the nodes wait for all the nodes to appear in SED. Once all the nodes have appeared, they, everything kind of proceeds as normal. DNS entries get created, routes get created. And the CRI is installed, Kubernetes is installed. And that's sort of it. But again, I think the interesting thing about it is the self-discovery aspect of it. It allows you to stand up 100 nodes really quickly because they can all just self-discover one another. And again, on different networks as well. So we've discussed it totally like, how can we divorce this from the SED discovery services or a way to do that? And we're kicking around some ideas, but I think it's a pretty, at least it's a way to do it. Let's say nifty, but it's a way to do it so that the nodes can all self-discover one another. I haven't seen anything like that, but maybe there is. Again, I'm still sort of nascent to it all. And it's still creating the load balancer. Oh my gosh, I filed an issue with HashiCore that their load balancer creator takes forever because it waits for it to be ready. So that's Yakitie, but what Yakitie does is it makes taking Yakitie and standing up a cluster and running the conformance tests on it, very easy for vSphere. It could be modified for anything. It's kind of similar to cross CloudX if they didn't use module. So it's not modular right now, it's vSphere only. But essentially, and you saw me run this, I just do a Docker run, give it some config information, give it the name of the image and just do the name of the cluster and then say up. And then once the cluster is up, you can just run the image again and say test and it schedules the E2E tests. And it does that because Yakitie has the ability to install the Kubernetes testing binaries on all of the nodes that have cubelets, where all the worker nodes. And so this schedules a job on the cluster that maps the E2E test binaries into the job and runs the conformance tests. And then you can just follow the tests with the T log and then you can just turn down the cluster. It also has the ability to download and the test results and upload them to GCS. As Eric, and I will never say his name, his last name correctly because I've not been told and it looks like Vegeta or Farida, I don't know. But as Eric Vegeta said, essentially, I recreate parts of prow here and that this will probably be what we move or migrate into prow. We'll still under the covers use Yakitie to stand up the clusters because there's still not a great solution for that for us. But for actually running the tests, once the clusters are up, we'll probably be migrated to cube testing and prow at some point in the future. Without the load balancer, by the way, this takes about a minute and a half. So it's pretty quick. It has the option to use an Amazon load balancer in order to make the cluster accessible externally, which is useful for things like Travis CI. So I don't have a, I can show you what the end result of this looks like. Unfortunately, it's not great at the moment. In fact, I'd even have to go over here. Where would I have to go to show you? Have to go over here because it's been failing forever because we, I don't know how many of you in SIG testing, but I mentioned that we hit a timeout. It turns out that, yeah, it turns out that Travis is limited to 50 minutes for free builds. Yeah, it's not active. So I'm not gonna, am I even gonna get a build history? Okay. Yeah, there we go. So where is, do I have a 50 minute builds? So show more. All right. I'm just seeing if I had the 50 minute one. Oh, you know what? I sent that link to the group. Sorry. I'm sorry again, when I'm apologizing so much and two that I don't have it ready. Yeah, here we go. I was calm. I don't know why I didn't find it. Probably because I re-ran. Oh, it was marked as an error. That's why I was looking for green. But when it's working, it just, it died because of the timeout. But I mean, it ran through 50 minutes of the conformance tests for anyone who, you know, familiar with them. And it would eventually have uploaded them if it had been able to. Luckily, and I love Travis CI I've used it for years. I emailed them this morning and they're gonna increase the time limit for us for a few weeks until we can figure out how to get to proud. So this stood up the cluster over here. It waits for it to come online. So long. Thanks for all the fish. So this is the load balancer. A couple of interesting things about it. It uses an engine, engine X proxy at the front so that it can answer on pure HTTP. So that the Amazon load balancer can actually access the backend cluster. It also includes some nice information like what are the artifact that was used to stand this thing up? So if you wanna know that or job, yeah, I'm all, you know, here's the job that you can use, job spec that you can use to actually run the test. So you could schedule that. So if I do a, where is it? Yeah, cube, cuddle, cube config, data, what did I just stand up? Yeah, CI, latest, cube config. So that's not the one I stood up. I stood up Andrew, sorry. Cube config and then do gets, and yeah. So, I mean, that's, like I said, I should probably, I need to put together some slides, but I just been so busy trying to wrap up the end of the quarter that I didn't wanna punt this demo again. Let me, last thing, let me SSH into one of the boxes so that I can show you, actually this one's up, so I can show you this, pseudo journal control, and see you, why, that didn't look like, oh, did I kill that ahead of time? Failed to install. Oh, son of a gun. That's an old one. So if I exit that, I can still show you that if there's an error there, except I'm trying to get this to work, I wanna see if it shows the waiting. Now, just too much in the way. Let me, let me go to a controller node. That will show it, 141. So this does not throw the keyboard on the controller nodes. This spans everything up a system deservices. The keyboard only exists on the workers. So, and I'm finding a lot of people don't do that. That a lot of people expect the cubelet to be up on the worker node, which I find interesting, I'm sorry, on the control plane node. I find interesting because to me, that's just like a security surface issue. Like don't put the cubelet there if you don't need to. So one of the aspects of this that might be interesting to you, Taylor, is it doesn't rely on shared DNS because it actually runs core DNS on all the control plane nodes. And it's not using core DNS for the service DNS, although it could, it has the option to use that instead of cube DNS. Although when I enable it, the DNS tests fail, but DNS works for service. I don't understand that, I haven't tried to debug it. What it does is it installs core DNS in all the control plane nodes. And so once all of the nodes are aware of each other, they all register themselves with DNS. And what they do is there's a round robin A record so that you can just do host K8S is the name of the cluster. And all the nodes know about Ks.vmware.ci, even though that doesn't exist outside the cluster. And CO1, oh, do I not have, what is this, 192, oh, it's 168, 3141, I'm making too many changes, host CO1.vmware.ci. Yeah, I guess I didn't have the suffix set up correctly. Yeah, so then they wait on all the hosts to appear and I believe I actually got this part working the other day. Yeah, so it also sets up a C name for the load balancer so that it doesn't paperclip to try to access it if you're trying to access the load balancer. So it doesn't, yeah, again, it doesn't need the shared DNS because it just stands up DNS inside. Oh, I keep forgetting, sorry. Because they all discover one another, all of the certs have both the IP addresses and the host names in it. So if I do open SSLX509, no ounce text, what did I do wrong? So you can see that it has the addresses of the controller nodes even though they weren't known, they wouldn't have been known ahead of time because this is actually using DHCP. None of these are using static IP addresses. What you do is you can either pass any certificates to Yakitie or Yakitie however you wanna look at it. And if you pass in a CA, it will sign everything using that CA. You actually can run an up command like I just did without a CA. And although Terraform isn't great at being modular and the aspect that you don't get to say only use Amazon if these environment variables are enabled, I tried that with CrossCloud and it just doesn't work because if you've got the provider enabled, it wants to configure itself. So what we actually do is the entry point of the Docker image, if it detects the AWS access keys, it actually copies in the load balancer config and the external cube config generator into the project inside the container. So it'll generate the CA for you. So this whole thing is using TLS but with essentially a CA that gets generated for this cluster. And you can access it outside of Docker as well obviously because the cube config file just gets dropped here. I mean, Taylor, this should look very similar to you. There's the cube config file. And when you actually, if you actually run the tests, so I won't, I'm not gonna run a new one. I'm gonna use an old one here. T log, this will actually show you what happens when you do a log against an existing test that's run. There, I just ran one of the tests to make it quick. But yeah, so that would tail the logs until they were done. And then you could do T put to upload them to GCS. So that, I mean, that's it. It, the self-discovery, you don't need shared DNS. All sorts of IPs and domain names. And then the Yakitui driver stands up vSphere and makes some, you know, makes running the Ede tests against vSphere pretty easy. But I think Yakity is the more interesting part than most people, probably. Any questions? All right, well, thank you very much. Awesome. I really do like the DNS. For quick start, getting anyone going, being able to not worry about DNS is nice. Yeah, that was like that we're gonna want to make. Yeah, and the not, and having certs with both IPs and names was nice. Although in fairness, the fact that cross clouds didn't help to catch a bug in the Kubernetes, in my opinion. Like they changed the default setting and all of a sudden they preferred IPs over domain names. But I like that it can support both. And I like that it can just, you know, you can connect two nodes, one on GCS and one on Amazon. So it supports reverse lookups. And yeah, it just kind of works, hopefully, when it, you know, except when it doesn't. But yeah, so if anyone has any other questions, or want to take a look at it, please feel free to hit the GitHub page. It's linked in the docs. Thank you. Thanks, Sandra. I think the next person, HH, are you available to talk a little bit about APR Snoop and Kubernetes Test Infrastructure CI? Yes. Yes, please. Let me see if I can share. Okay. My screen. It's not always, Okay, okay. Let me know if you see anything. If not, I'll drop links. Do you see anything? I can see. And if you can drop links, I think that'd be good too. All right. Unfortunately, I can no longer see the window that I'm sharing. It's all black. So I'm going to stop sharing and do the link dropping. Oh, really? Yeah, unfortunately. Something around my X setup and various things. So I'm going to use the chat area, or should I use the, maybe if I use the CI working group. Yes. Can I drop them in the chats? Fine. And if you want to add them to the notes, that would be, if you put them in the chat, I'll put them in the notes. Okay, that's fair. So where do you see where the notes went? Oh, there they are. Yeah. So first of all, there's the API Snoop interface. And if you'll click on that little bit there, and one of the things that we wanted to be able to do is show our change in coverage over time to see how well our community is doing. And at a minimum, hitting the various endpoints that are provided by the Kubernetes interface. And we spent a lot of time trying to figure out, spinning up the clusters and getting the data in a place where it can be consumed is a problem. And as we dug into the various tool chains that are available within our community, we found the CI systems provided by test infra and SIG testing are kind of where they're coalescing for Kubernetes for sure. And if you go to the, there's a dropdown, it's not quite obvious it's a dropdown, we're going to need to make some UI changes to that. But it says Kubernetes EDE conformance CUBE test because we're running CUBE test, I'm not Sonaboy yet, but there's dev19, dev110, dev111, dev112 and our original manual import of stuff from May 30th this year. If you mouse down over those, we've brought in the audit logs now, this is all generated from audit logs rather than spreadsheets that gets pulled into the visualization. In this particular instance at this moment, it's still somewhat manual, but we have found a much better place than trying to generate the data manually ourselves. And I'm going to drop a link here. This is test grid. Test grid, and I'll drop two links, test grid is where all of the jobs, some of the jobs, proud jobs get published in a way that we can group them together in meaningful ways. The second link there is specifically for our conformance proud jobs that get grouped together underneath that you'll see we have GCE, for master 112, 111, 110, 19, for dev and release and the same thing for kind and digital ocean and OpenStack and Baidu, and it's a process to get there. But because of the work we've done to integrate the audit logs into the main process that's now available for every job, that at least it's part of conformance. And we found that I think one of the things we're going to be working within API Snoop is to take, automatically pull in what's available via test grid and make that render it in a way where we have one rendering of the API Snoop Starburst, but other really, we're going to allow anybody to spin up the stuff and have a different view and provide their input. And that's one of the, like this week we're doing some refactoring of the UI so you can spin it up and point it to these different repos and start working on visualizations so that you have access to the same data that the API Snoop team is working with. So the next thing I want to show you, if we're in the conformance group and I'm going to click on GCE112Dev and it's passing and so that takes us to here. And then if I click on about and I go see these results in gubernator it's kind of a little dance to get down to it. But that takes us to this particular test and all of its jobs that have popped out of it. I'm going to click on job 139 so we see yay it was a beautiful test and all the things passed. And inside beyond the logs and I'm going to go click through a couple steps we can get through and pull down all of the wonderful data that's available on the master node. And the best thing there for API Snoop anyway is the API server audit log. So I'm going to copy that link location. This is the piece of data that we'll be able to go all the way from the different groupings of our conformance groups to pulling in the raw audit log possibly directly into the browser using node and some other other fun bits. This wouldn't be possible without proud. So if there's any questions I feel free to stop. I'm not I'm just kind of free forming here. I don't have a Prezi. Proud's been really great in that it does a couple of things. One it manages the merging so that our communities particularly the many, many, many repos inside Kubernetes can be merged in a safe way. It also provides some plugins for managing the conversations and things. And I'll go into that a little bit later but I wanted to acknowledge that this is a really great tool that would be great to use beyond just Kubernetes possibly in the CNCF community at large. That's probably enough for API Snoop. When we need more data and I think we need data beyond what's available in just the audit logs. I know that Catherine did some really beautiful work to modify the go test infrastructure to when you create a binary augmenting the binary so that it generates every second an HTML file with test coverage. It looks really cool but we need to integrate that so that the artifacts are available inside the gubernator artifacts. And so that's one thing we'll be working on soon. But one of the API Snoop issues I'm gonna now kind of step away from the so first is there any questions about API Snoop and the thoughts and processes that we're doing right now before I go on to the next topic? I'll take that silence as I go ahead. I'm gonna drop a link to something, a thought. A thought as the API Snoop team is dug in deep into the test infra provided by SIG testing and the Kubernetes community. We found there's some really useful tooling there that might be useful within the CNCF as a whole but it requires some technical things and I thought it might be interesting to see what would it be like if we automated some of the interactions with our community to allow them to connect and benefit from some of this infrastructure. So we're gonna do this in the form of a short skit. I'm gonna step out of the way and become a robot. I will play the part of the CNCF CI bot and my friend Bernal here is gonna play the part of hippie hacker submitting a PR. You can follow along at the issue that I posted in the channel. Hi, as a CNCF project admin, I would like to request some CI automation from CNCF in order to benefit from the CI infrastructure maintained by our community. Given that the repo that is included in CNCF landscape when I create a ticket within the repo and tag it at CNCF CI and I grant CNCF CI admin to my repo then CNCF CI will automatically configure my project for Proud and add other services. So at CNCF that's CI. Did you ask me something? Oh, hi, hi HH. I'm not a bot yet, but I can be. In order for the CNCF to provide the CI services you need, I'm gonna need you to grant me admin access. You can go to this little link here and add me. I'll message you when I've accepted that and we can continue. At that point, I'll be able to configure your web hooks, change your labels, merge your PRs and everything that passed your CI test and work with a lot of stuff. There'll be some interesting slash commands that are available here and you can read about them. Just let me know and add me to your admin and I'll let me know. All right, CNCF CI, how do you choose an admin? Awesome, while I was away and as you added me as an admin, I configured the slash commands for you but there's some other things that I could configure for you including Proud and all its various plugins. Here's a list of all the things that you could configure but in addition to those helping to manage your PRs and your repos and conversations, you can also help add you to test grid and for example, adding your project to the conformance for the test grid. What's really cool about being added in this way is your results, past and future will be available via gubernator including your logs and your artifacts and what that allows is for some really cool integration with the rest of our community including projects such as API Snoop and the CNCF cross-cloud dashboard. If you need any help beyond this, please join the CNCF CI Slack or attend meetings like this public CNCF to CI working group meeting or join the mailing list. All right, thanks, it looks brilliant. Let's enable some simple plugins. Can we do cats and dog maybe? Done, you now have the cat and dog plugin. Why don't you give the cat commands meow and wolf a try? All right, slash meow. Ah, kitty. One cat, okay. So that's our demo but I just think this might encourage some conversation around how we can enable our communities to start getting data together in a way that we can get some real insight across the CNCF community and their artifacts and their particularly, when we're looking at audit logs, for example, in the way of API Snoop and the way of Catherine's code coverage tool, we'll be able to use some machine learning and some other pattern analysis to see how our community is using Kubernetes and create some really meaningful conformance tests based on that insight. So this is all tied together with API Snoop but it's really about CNCF, CI for our entire community and how we can provide data for our projects as a whole. So I'm gonna hand opening up for questions. Yeah, that's it. Thanks, HH. Crickets. I'd like to see the visuals on that, especially if you can get maybe some slides together. The last part reminded me of something, Ed, from the network service mesh fit together for intro and it's kind of a skit. It's all drawings and slides and they can go through the interactions between a user and someone coming in to use it maybe taking what you have and having it written up so that you can get the visuals to tie along with the story would be good. Okay, let's say who's next on the agenda. I think, Ben, are you there for the Prometheus side? Yeah, so I just wanna say hi. I'm on the Prometheus team and I work at GitLab and I just wanna bring up like, we've been having our own discussion for the Prometheus pipeline or for the Prometheus team for all of our different CI pipelines because we have a bit of inconsistency in our configurations and I linked to our internal discussion on that and wanted to find out more about what the CNCFCI project was all about and see if there's anything that we could use or you could use or what that collaboration was like. So I just wanted to give a quick intro and say hi. I don't know if you want to have any specific questions for me yet or if I should just keep rambling about what we're doing. Rambling? If you wanna, I know you shared a link. If you wanna run through anything there, that's fine. As far as Cloud team goes, I'm happy to talk. If, like I said, if you have time and can meet us at the CNCF booth or shoot a slack message. Yeah, I'll definitely, I'll be there tomorrow. I'm giving a talk on monitoring stuff in the morning and then I'll be around. So basically, Prometheus project, we also have a whole bunch of different repos that run different CI's. Some stuff uses Travis, some stuff uses Circle. One project uses BuildKite and all of this has different requirements for different things. And some stuff is historical because we added Circle because we couldn't, we originally couldn't publish our Docker images and our GitHub releases with Travis. So we ran tests in Travis but ran our release builds in Circle. And so we're just like, we wanna consolidate this and wanna see what the CNCF was actually doing about testing and builds. And see what you've been working on and what you're planning on providing. What else? Yeah, and we have one project that we use BuildKite because we need to be able to do building and testing on non-X86 platforms and also testing on non-Linux platforms. So we have some BSD builds and some power PC builds because the architecture specific compiles need to be tested. Awesome. Okay, we've brought up this document that you've shared in the notes and as far as CrossCloud CI goes, I think we may have spoken either I spoke with you, Ben or someone else many, many months back a little bit about what we're doing. From a CNCF support site for helping with CIC the CrossCloud dashboard and the tools underneath would be one of the projects. There's a lot of other projects. HH is talking, I was just talking about some of them. I think probably be good to just maybe chat and go through some of these and look at what the big needs are and maybe HH should be interested in taking a look at this. Sure, we could do that tomorrow or any other time. Okay. And then HH, if, I mean, if this is something you have time for and wanna check out, know that you're involved with a lot of the different groups Kubernetes-wise is on the testing side and across CNCF. Ben, the open CI, that was, I think, did we have a link earlier in the slides? I think we may have, trying to go back. The open CI group might be something to get involved with because there's a lot of different groups on that that are doing a lot of different systems that they're using and that could be helpful for Prometheus. Sure. And then I think y'all have access, at least I recall right, the CNCF lab with packet.net for running, so basically resources. Yeah, we're using packet for running our build kite builds. Okay. Yeah, see if you can catch us while we're at the booth and then maybe hop on Slack on the cloud native Slack and the CNCF CI channel. Yeah, I think I already hopped on that. Okay, great. If you wanna drop the link and maybe tag HH and myself and let's see, I don't know. Anyone else is interested? Melvin, you may be interested. I know that there's a lot of different things that each of us are doing and we're trying to get the CI working group itself any of the projects that would be beneficial for others. So ideally if there's something that Prometheus needs where we can try to have something in place that's reusable by other CNCF projects. Cool, let's see. I think the only other item on the agenda was we were having some build failures with Prometheus actually on the CNCF CI dashboard in a CI Dev environment. And it looks like it could be something with dependencies that we need to add to the Docker container. So it's something I think we need to defer and maybe reach out, but we'll have a public ticket and maybe tag y'all if we need some help system on that. Yeah, you're always welcome to contact the Prometheus-team. Okay. Email. Sounds good. And then we noticed this actually started in 2.4.0. So it's, we've been doing various changes on our end to support the new clouds and various things. So it could be tied into that, dependencies, don't know, but we'll take a look and then reach out to the rest. I think that's it, unless someone else has anything. Stop here. So, okay. Slide 28. Okay. So this is monthly, again, next meeting on October 23rd. Besides this Slack, Cloud Native Slack, CNCSCI, there's a mailing list that you can subscribe to. And that's it. Thanks a lot, folks. Have a good one.