 Good morning, good afternoon, good evening. Wherever you're handling from, welcome to a brand new show here on the channel, Cloud Native Thursdays. Name's still kind of in flux, but I'd like to introduce you to the two hosts of the show, Josh Burkist and Amy Marich. Amy, you're the closest one to me in the Zoom window. Why don't you go first? Alrighty, my name is Amy Marich. As Chris already mentioned, I'm a Principal Technical Marketing Manager at Red Hat, especially an open stack. I serve on the Open Infrastructure Foundation Board of Directors and I'm core on a couple different open stack projects. Awesome. And I'm Josh Burkist. I work in the Red Hat open source program office. I've also been contributing to Kubernetes for almost as long as it's been in Kubernetes and have been involved in a bunch of different areas in the project. And I am very into everything Cloud Native. And so as our show unfolds over the next few months, you're going to see Amy and I are going to present a number of speakers and a number of topics around the full cloud stacks that people are implementing by all the way up from, in some cases, the individual CPU and GPU chips to application automation, depending on the speaker and the week. But this week, we are kicking things off with Luis De La Santas, who is a lead in the CLARE project for container security, container image security scanning. Before I add Luis, I know your position in the CLARE project, it just occurred to me. I didn't actually even look up your position. You work for Red Hat. I do, yeah, yeah. Principal engineer at Red Hat, creator of CLARE V4 and CLARE Core. Cool. Wow. Before we get this started, is there any administ review we have to take care of, Chris? No, just folks, please feel free to ask questions as we're going along, kind of treat this like an office hours if you want, right? And we'll engage the speakers as the questions come in. Yeah, and do ask questions. We can take your questions at any time. So go ahead and ask them in any of the mini chat streams and we will see them and read them out to the speaker who can field them. But in the meantime, Luis, do you want to get started explaining some basics of CLARE for anybody and the audience who is new to it? Yeah, definitely. So as you said, I'm a principal engineer on CLARE. I started the CLARE Core project which is actually the engine that powers the newest version of CLARE V4. As in this presentation that I have with me right now, we can cover, you know, what is CLARE? What's new in CLARE? Why should we even care about it, right? The internals of how CLARE is going to work, where it's being used and then some contribution information just because we're always looking for new contributions. So really like what is CLARE? CLARE is a scalable set of services that provides information to both developers and operations about your container's security. It's an open source project and it is community-developed. So why exactly should we care about CLARE, right? So as you know, we're all moving to Docker containers as the default way to ship and deploy applications. With this, we start migrating the attack vectors. Before Docker, we probably had Ansible setting up their application on servers, deploying dependencies. The splash damage from there was pretty confined, right? If you found out that, okay, I have a dependency with affected vulnerability, you're able to go to your own servers and change them. Now that we're moving into deploying applications via containers, that attack vector migrates everywhere that container may run. So it makes it a little bit obvious that your organization needs to care about its container security posture. So if you're familiar with CLARE v2 or if you're not, and you're just getting started with CLARE v4, these are the new things that we've added recently. There's now a manifest-focused API. Previously, in CLARE v2, it was a layer-based API. It was a little bit cumbersome. What had to happen is the person or the client who was making a request to CLARE needed to define the child-parent relationship of the layers before posting. If you're familiar with OCI manifests or any of the even Docker manifests, that child-parent relationship is actually comprised inside the list of layers that are in the manifest. It's a very obvious thing to do, but it does actually make a big difference in usability with CLARE v4. Content addressability is now a first-class citizen in CLARE v4. So I first heard this term in the storage industry as a way to take a unique identifier and make it always associate with a unique resource. I put a little star there because it's pretty important to how CLARE performs its work. So keep this in mind. We've updated all security advisory data sources. What this really entailed was no longer using ad hoc data dumps, not going out to GitHub. We standardized mostly on Oval where it was appropriate. There's still a few cases with some custom advisory information. AWS has their custom format. Alpine has a JSON dump on GitHub, maybe GitLab now, I have to check, but those places we can't really avoid it, but across the board elsewhere, we just went with Oval 2, which is great because we were able to standardize our parsing and it's a well-understanding specification. We now have a native CLI tool. So CLARE v2, previous versions of CLARE, dependent on an external repository, community-driven CLI tool, we brought that inside. We're now going to support that on our own. We completely rewrote the notification subsystem. We're going to dig into that a little bit more when I go into CLARE internals. We now have language support baked into the data model. This is a big plus for us. Previous versions of CLARE, they didn't really understand packages on a file system. They understood like, okay, this is like a RPM database. You can take a look at packages in there, but they didn't really understand, okay, like we have PIP and we have like sites, packages, directories. So now we built that into the data model of CLARE v4. So we want to begin adding a lot of package support within the next iterations, and we currently support Python. And then the big application architecture changes is that we went with a modern microservice design, which puts an emphasis on scale and performance. So as I'll demonstrate in just a bit, CLARE has broken up into microservices that can now asymmetrically scale depending on your performance profiles of the day. So if you're pushing a lot of containers, you can actually scale the set of nodes that deal with that, or if you're pooling a lot of containers and you want advisories, vulnerability reports, as we call them, a list of advisories that are affecting that container, you can scale that asymmetrically. So if there's no questions, yeah, I'm gonna go into that. No, actually there are in fact, a couple of questions already. Let's go. Number one, JP Dade wanted to confirm that CLARE is built into Quay. CLARE is, so historically speaking, CLARE is a Quay product. It is completely integrated into Quay. As we're going to talk a bit more about in a little bit later in the talk, it is being used quite extensively outside of Quay now too. So there's a little bit of flux in there, but yes, it's completely integrated into Quay. And most of the access of CLARE, you'll probably see online, if you're looking at tutorials, if you're looking at how to use it, you'll see it coupled with Quay. Yeah, and that's actually a great segue into the second question from Tanyway3. Can companies by support and run this on-prem and as much as you said it can be run separately? At this time, from a Red Hat product standpoint, it is bundled with Quay. So we don't have a service contracts. You can't run CLARE isolated as a Red Hat product, but there's nothing that stops you from doing that upstream, right? Just grab the upstream from us. There's abundant tutorials on how to get it up and running. And our documentation is pretty good. So always looking for feedback there too. So if you do want to give that a try, you'll have our emails to poke at us if you can't get it to work. Tanyway3, does that answer your question? I'm gonna assume so for now, but we'll ask again if we need to. Cool, okay. Yeah, so let's dig into a little bit of how CLARE works internally, right? So this is the 30,000-foot view of what CLARE accomplishes. This is the microservice architecture that I said earlier that we split CLARE up into. From a very high level, the indexer is responsible for taking a look at a container manifest. When it does this, it generates an index report. This is a report containing all the contents that was able to be found in that manifest, along with distribution information, whether it's a rel container, whether it's a Ubuntu container, and any repositories that we find, PIP repositories, MPM. I don't wanna allude too much on language support. Like I said, we only support Python right now. So we would say that we only find PIP repositories at this time. Once that index report is generated, you have the matcher service, which is responsible for having a database of vulnerabilities, accessing that index report, flagging the content in it, and then generating a vulnerability report. And then you have the notifier who just kind of hangs out down there. He watches the matcher and the indexer, and he's able to generate notifications to clients when new updates come into the system and inform those clients which manifests are affected by these new updates. So a new vulnerability comes in. We know that manifest X has this package that's associated with that vulnerability. The notifier will go and say, okay, let me notify the downstream that manifest X is now has this new vulnerability. You might wanna patch it. You might wanna push a new container to fix that. So to really dig into indexing a little bit further, indexing is a term we use for extracting the contents of a container. So to even give that more of a context and background, I'm gonna tell you what Claire sees as a container. What is a container in the context of Claire, right? So how Claire views this is a, there's a top level hash here. You'll notice that it's 604A. That denotes the manifest as a whole. Then there's an, and that's a content addressable hash. So I was talking about that a little bit ago, right? So this is a unique identifier that uniquely identifies this manifest as a whole. You'll see that the subsequent layers inside this container, E84B, DF08, CC40, those are layers and they have their own content addressable hashes. To obtain that manifest hash 604AC, it is a hash of all the layer hashes, right? So this builds somewhat of a hierarchy in the fact that Claire can understand if it's seen in manifests, it's also seen its layers. That's a little bit dense of a subject, but if you can understand that the manifest hash is computed over the layer hashes, then you can understand that the manifest as a whole can be identified by that. So now how does indexing actually work, right? And how does Claire go about extracting the contents of a manifest? First, it's gonna determine if the manifest should be scanned. In this case, Claire has never seen this manifest, right? It says 604AC, I've never seen you before. We're gonna go ahead and we're gonna actually put you in the scanning pipeline. So now it determines the layers to be scanned. So I express a common case here, right? That's the base layer. Claire might've seen that base layer a million times, right? Could be a UBI8 base layer. So it determines, hey, I don't need to scan this. I have a content addressable hash here, right? I see E8 for B in my database. I know the contents are the same. It says, hey, I'm not gonna scan that, but these two subsequent layers, okay, I do need to scan those. I've never seen them before. So now it scans only the necessary layers. That base layer goes right out to the database. It says, hey, give me the content that we've cached because this content addressable hash has not changed. I know the contents hasn't changed. Just give me what I know. And then for DF08 and CC40, that actually performs the scanning process. So why this is helpful is because Claire can do the least amount of work necessary, right? To scan a layer, we have to go out. We have to download it. We have to decompress it. We have to actually look through the tar, the tar bowl, and then find any kind of databases. So there's a lot of work there. So we want to minimize that amount of work. And luckily, the container specifications have this content addressability notion built in that allows us to just piggyback right on top of that. And then finally, it just generates an index report for the manifest itself. Any questions here? I have one. How long does this take to scan everything and give a report back? Yeah, so it definitely depends on the image size, right? Like if you were to go and pull your bunch of latest relatively small image to take a matter of seconds, if you're to go pull like a larger image about one gig, you're looking at maybe like 30 seconds around there. So it really depends on how large that image is. We utilize Golang as the software that Claire is built in. So as you know with Golang, concurrency is a paradigm. So we're able to do a lot of things in parallel. And where we do set, there's concurrency limits on things just so we don't run away with that. But it's significantly quicker because we are doing it in Go and the parallelism is just lightweight. We can spawn off Go Routines pretty quickly. But yeah, that would be something, if you're an operator and you're using Claire, then that's probably something you actually do wanna spend some time tweaking, right? We give you a little bit of knobs. We try not to go knob crazy because when you go knob crazy, people like to go crazy with them. So, but there might be some flags there that allow you to change, how many layers are pulled down in parallels, something like that. And if there's not, then you can definitely raise a ticket with us to see if you'd like to pull those things. And you've mentioned Docker support. We've had the question, does Claire support LXD? Claire just takes tarbles, right? So if you can place the layers in its manifest schema and we get a tarble and then we can extract it and then find RPM databases, I don't think the LXD format would stop you. I'm not actually super familiar with LXD. So that's a good follow-up question, but as long as it has a concept of layers in tarbles, you can take those layers and you can provide, you can basically type out or create a notion of our manifest structure, place those layers inside it and Claire can go and download them. There shouldn't be much stopping you from doing that with other types of container formats. Just a quick question actually from me, which is the hatches for the layers Are these being calculated by Claire or are we just getting them from the image build metadata? That happens on image build. Yeah, so whatever runtime you're using or image building software, whether it's Scopio or whether it's just Docker build, that's generating those hashes. We do check them though. So we'll actually do, you know, like usually it's a shot 256, I think, we'll like, we'll hash the content when we get it and make sure it matches. If it doesn't, we'll reject it because that's not really to spec, right? Cool. Okay, so now you have an index report, right? You have a report that demonstrates all the contents that we found inside the container. So now matching, so matching is a term we use for identifying vulnerable content inside a container. There's two major components with the matching infrastructure. You have what we call matchers. These are implementations of business code that understand how to match packages to vulnerabilities. So a good example is, you know, if we're looking at a RPM database, we most likely have a rel matcher that knows what RPM looks like and then also knows what the vulnerabilities that the pulp and rel exports and is able to match them together. And then updateers are, what you would suspect they are, there are these pieces of business logic and they run on intervals and they basically go and they find new content, new security databases, updates or just new ones in general and then pools in the advisories into their database. There's some smarts around there as far as, you know, doing less work, we try to see if any content has changed and we keep a fingerprint, but you could probably already imagine, you know, how updaters work, it's not incredibly complicated. So how does matching work? So matching is modeled after a data flow application. So when the metric goes and grabs that index report, right? The index report has all the contents that we found. It actually makes a content stream and it instantiates these matchers in parallel. This is another great thing about code, right? We're able to just really quickly and lightweight, in a lightweight manner, create these concurrent processes. So each matcher now gets this content stream and they just run through it. They just flag things as they see it. And once they're flagged, the matcher takes all those results, merges it and then gives back a final vulnerability report. So this is very nice because we can do things pretty quickly and also in the common case, only one of these matches is really doing work, right? Because it's not like we upload hybrid containers very often. It's not like you see something that shows characteristics of UBI-8 or rel instance along with Ubuntu. So, but technically, if you did send that our way, we would identify things correctly because of this parallel identification mechanism. So hybrid containers, it happens. It's very strange and it shouldn't happen, but if you put an OS release file and a LSB release, I think, Ubuntu's one, they will show up and it's like, hey, we identified this is some kind of hybrid container. So that's matching. Any questions here? Cool. Okay, so how do updaters work? Again, this is fairly easy to conceptualize and nothing too deep here. We basically have some business logic. It knows where security databases are. Usually net addresses, HTTP, FTP, whatever it might be. And just goes out on an interval, tries to find new vulnerability databases, new updates to them. And then if it does find them, it just downloads them, parses them, puts them in the database in our format that we can easily search against. And now there's notifications. So notifications make clients aware of changes to vulnerable manifests. So let's dig into how the Notifier service actually works. So Notifier discovers an update by pulling the matcher, right? So the matcher, if to reiterate the matcher is running these updaters and they're constantly pulling new security databases and they log or they create a database entry and when they find new content, when they've downloaded a new security database. So the Notifier is sitting there and it's asking the matcher saying, hey, do you have any new content? Did you just download any new content? In this case, it has, right? It has found a new update. So now it says, okay, yes. Okay, so this is an update of security information, not an update of container images. This is an update of security information, yes. So the Notifier is sitting there watching the matcher and the matcher just grabbed some new security database. Let's say it's like a new pull manifest came online. It downloaded something for Ansible. So the Notifier says, okay, I just saw this. I want to create a notification now, right? I want to tell the client that, hey, if your containers had an Ansible version, XYZ, let me know who has that and I'm gonna inform you guys. I wanna tell the downstream owners of those manifest and those containers to go patch your container. So the Notifier now asks the matcher again. He says, let me get the diff, right? So he asks the matcher, he says, we diff the old database with the new database, right? And we're able to find out what was added and what was removed. So this new database that just came in, it might have removed a couple of things too, right? And we should be aware of that. We shouldn't just be add. So we get a complete diff understanding that in this case, you know, CVE123 was added, 456 was added, 789 was removed. With that diff in hand, it goes off to the indexer and it says, okay, who's actually affected by this? The indexer is responsible for what? Quiz time, anyone know? Okay, I'll just tell you. Yeah, so the indexer is responsible for understanding the contents of the container, right? So it has a nice big index table that says, these packages exist with these manifests, right? So when the Notifier asks the indexer, hey, we have these new vulnerabilities. Vulnerability123 is affecting package X, right? The nthexer can look up saying and can say, okay, these manifests, these containers with package X, are vulnerable, right? So they give back those manifests to the Notifier. So now Notifier has a nice big list of all the containers which are affected by this new update that has entered the system. And then finally it creates a notification to any subscribed clients, right? It says, it just gives notification that says, these manifests, a new security update has came into the system and now these manifests are affected by them. You should probably go patch them, you know? Cool, any questions from here? Yeah, so when one is removed from this CVE list, if you're on a later newer package, you're not gonna get notified that it's an issue. But if you're on an earlier package before it was removed, you are gonna get notified. That's correct, yeah. Okay. Yep, cool. My other question actually is more in terms of implementation, right? Because obviously in a large development team, managing these notifications is gonna become its own thing. What notification ops frameworks have people plugged Claire into? Yeah, yeah. In order to handle this. That was definitely a good question because we do have a couple of different use cases now. So when we built this notification system at first, we only really supported web hooks and pagination. But then when we started integrating with EXD cloud, which powers the Red Hat Container Catalog, they influenced us to actually support AMPQ and Stomp, which are message brokers. So that becomes a little bit more of a scalable way. If you did have like a lot of notifications coming through the system, we'll actually deliver them to like a rabbit and queue or an active MQ. And then you can build a set of workers off that to scale the amount of notifications that you guys are handling. It's a good question. It's a good question because on big teams, I mean, it can get kind of crazy, especially really big repositories or registries rather. You kind of want more than just, you know, a single node paginating results back. So yeah, supporting message queues gives you that functionality, right? You can just have a cluster of workers that are listening off that queue and just eating up notifications and doing whatever you want from that point on. Yeah, I was actually thinking more in the realm of sort of ops tools, like, you know, pager duty, something like that. Because the notification eventually has to go to a person who has to know to tell the relevant dev team or a person on the relevant dev team that they know that they actually have to rebuild their images and run the tests again. Yep. Yeah, so I haven't, I don't know too much about actual, like, you know, DevOps tooling around those notifications. Quay itself has this functionality, right? Because we wrote these notifications kind of like for Quay's consumption. But outside upstream, we don't have anything off the bat. It is a cool thing to think about and to maybe bring up, you know, in my community development meetings about whether we should be providing some type of integration layer, you know, that just kind of feeds off our notifications and pushes them to popular alerting systems like pager duty. It sounds like a good idea. It's definitely something that's worth considering. So, but yeah, as far as today, we don't really have any kind of integrations into alerting or monitoring systems right now. We would expect the users of Quay to kind of build that out. Well, obviously, you've, go ahead, Amy. Okay, from an ops side, personally, if I'm gonna get these notifications, have it run overnight or something and then just send me the information in the morning versus a pager duty that's gonna go, oh no, you have a vulnerability every five minutes. So I think that would also be a lot of the different operators' taste and how often they wanna be notified. Yeah, pager duty is actually not a good example. I was actually trying to think of one of the other sort of ops management frameworks that aggregates notifications, right? Because obviously, you know, particularly like if there's a CVE in a major package, like there's a CVE in, say, Golang, then you're gonna get a million notifications about it because everywhere you use Golang in your enterprise, you're going to get a flag. And so you wanna be using some tool that's gonna get aggregated in those notifications so that you know, you know, so that you're not getting a million separate ones. And there are a bunch of tools like that. I was just blanking on a name for one. Gotcha, gotcha, I see what you're saying. Yeah, but if you're providing the data to come brokers like, you know, AMQ, then presumably most of those brokers have the ability to consume information off of those Qs. Presumably, yes, yes. And then, you know, we do kind of punt roll up logic like you're talking about to those Q consumers. But yeah, I mean, there are good points. The roll up is an interesting point because yeah, we still need to demonstrate that each individual manifest, each individual container was affected by this. But yeah, I mean, there's no reason that it couldn't need to come as like, you know, packetized events, it could come as a roll up event. So the aggregation thing is cool. It's definitely something to think about further. There is a question over on YouTube. Can we send clear security notifications to an SIEM tool of some sort? Uh, what is that acronym? So this would be like these scene tools, SIEM. They're like aggregating security management tools into one kind of single pane of glass. If you, depends on the level of customization you have there. So we, we provide a webhook mechanism. So we'll fire a webhook and we show you, we document that actual structure of that webhook. So if the tools have a webhook integration feature and they can call back, then they can pool in data from that webhook. The, you'd have to understand the clear schema and exactly what those notifications look like to make use of it. So there's no integration as far as I know into those systems as of today. I mean, I do have to say as a, as a former ops, I would want a tool like clear to be supplying raw data rather than trying to do aggregation or something else itself, because usually things like a SIEM tool is already going to have a fairly sophisticated rules and aggregation engine. And, and I don't want to have clear trying to replace any of that. Because the clear team is simply not going to put as much work into rules and aggregation as a rules and aggregation tool. That's a good point. Yeah, so. It is a good point. Yeah, those policies usually come from internal needs. So there is some cross-cutting concerns there that do kind of bubble up. Cool. Anything else? No. All right. Well, actually wait, you know, can you do something like create a JIRA ticket with a web hook or something like that? No, no, we don't integrate directly with JIRA. Okay, cool. Yeah, again. This is a scanning tool. This isn't your all security encompassing tool. Yeah, so the integration points are really meant for extensibility, assuming that, you know, even small scripts are written against it, you can do whatever you like, you know. So if you can even write, you know, Python scripts that can respond to web hooks, then you're free to do anything you like with those notifications. So creating a JIRA ticket would just be creating some small glue scripts or some small glue code that sits in between Claire and your ticketing system. And that's also part of being open source. You can contribute these scripts back to the project and to the community. Absolutely. Yeah. And I was gonna say the, I just had a truly horrific thought, which is be clear to IFTTT broker. Oh, Lord. Someone will do it. Eventually. It's only a matter of time. Yeah. Cool. So that's cool. So that's notifications so far. So now let's go over, you know, moving from Claire V to Claire V4. As I touched upon a little bit, Claire V4 is just a complete rewrite. So to get from Claire V2 to Claire V4, you have to backfill your containers, which shouldn't be as bad as it sounds, most likely because a lot of your repositories or your containers have shared layers, as we discussed before, Claire differs a lot of work. So that rescan isn't as scary as it sounds, but it does need to occur. And just to reiterate the point, anyone using Claire V2 currently, the layer-based API is gonna go away. And I think most people are probably clapping their hands at that now if they are Claire V2 users that didn't directly use it with Quay, but will be using the manifest driven API from this point on. Also, we're kind of bringing up to community discussion pretty soon, whether we should just go full OCI. I don't think anything really stops Claire from utilizing the OCI manifest specification. And we basically kind of just wanna get a community opinion about that, whether we do find it actually easier to build tools against Claire if we do standardize on OCI. But as far as an implementation, it's pretty fairly easy for us to get there. So that's something I'm kind of excited about community conversation, bubbling up around that one. Cool. So a little bit of information about Quay and Claire. So Claire V4 was introduced in Quay 33, and it's now the default container scanning security tool moving forward. And Claire V2 is now deprecated in 34. So 34 is now out the door. So Claire V4 is what Quay will be using moving forward and Quay IO will follow suit. We're not quite at the end of March that got pushed back just a bit, some internal details that I won't dig into too much. So Quay IO will be running Claire V4. We just need to basically procure assets to do the DevOps and infrastructure work necessary to get it kind of forklift into Quay IO. And then this is a cool, the cool part. I think it's really cool is that we work directly with the EXD cloud. When I started at Red Hat, taking a look at Claire was mostly focused on Quay. And one of the first things I really want to do was get further integrated into other use cases. So we started having collaboration meetings with EXD cloud. So EXD cloud runs the Red Hat Container Catalog. So if you were to go to Red Hat Container Catalog and get any of the security ratings and gratings right now that's all powered by Claire V4 head, they're running latest and greatest, they're fearless. So they, but it's been working pretty well. And they've been using it at scale with about, I think it was 500,000 container images indexed at this time. So if you do have time, just check that out. It's the Red Hat Container Catalog is great. It's where you go and grab your UBI images, which I'm a big fan of just because they're easy to use and stay along with the Red Hat Linux distributions. And then this is just roadmap. So Claire for one, what do we even expect when we have coming down the line? We definitely want to expand on language support. So Maven is coming in as a internal Red Hat contribution. This is coming from Code Ready Containers. So the Java support will be there. Golang, we're exploring right now. Definitely want some JavaScript experts to come on and really get language support into Claire in that fashion. My coworker Hank is currently working on a Kubernetes operator. That will be open developed as well. And this should make it pretty trivial to deploy Claire in the various architectures that we support. Like I said, because Claire is broken up into microservices, you can actually deploy it in many different ways in pretty flexible manner. So depending on if you do want a truly distributed deployment, you can run Claire with each service, the index or maturing notifier talking to their own databases, or you can run what we call canonically as combo mode. And that just runs all the services on a single node. And then you can scale out that node. So depending on how advanced your infrastructure team is or how acquainted with Claire you are, there's a couple of options in how you actually can go about running it. And the performance profile of your infrastructure is what's really going to determine how you deploy Claire. Then we're gonna spend some time doing performance and scale analysis, have some time booked just to do low testing, find any kind of bottlenecks that we might be having and squash those ASAP, it's a big initiative right now. And then reliability efforts. This is everything as far as using the least amount of database connections as possible, optimizing any distributed blocks so that they don't take too long to acquire or unlock. And then just general profiling of Claire. So that's gonna be our big efforts within the next four or one periods to basically some performance doing a lot of mechanic work. Cool, and then. One quick question we've got asked, how frequently do you update Claire security information database? Yeah, so that's configurable by the config, but by default and what we recommend is every half hour. So if you don't place a configuration at all, every half hour, we're gonna go and look at all the data sources and find out if there's anything new. And if there is, we're gonna pull it in. Half hour is just a nice number. It keeps everything, there's not a ton of load created from constantly going out and trying, but you can change that interval up to a minute, I believe. I think we limit you just for safety reasons. Cool, so then that's the roadmap. And then finally, I'm just gonna end with some contribution information. So we're beginning official community development meetings March 9th, March 9th, yep. So right around the corner, the first meeting will be about just the introduction, not too dissimilar to what we do today. And then some of the work that we have coming down to pipeline as far as what we're calling enrichment. So enrichment is basically going out to metadata sources such as NVD and allowing you to get further information on vulnerabilities. So that's initiative that's underway right now. So that'll work its way into that first community development meeting. And then the second one will be pinging out to anyone that's interested to actually add their own agenda items onto that meeting. We'll always have something just because Claire is still, Claire before is still pretty young. There's just, will be a ton of development coming down the line. So there'll always be something there, but if you don't have anything on the agenda to contribute for the foreseeable future. So then there's a Claire mailing list. This is a great place to join if you are interested in just following development or getting involved with our community development meetings. We'll probably do a post there, probably a couple of days before each meeting and we'll ask for agenda items to be added to the public Google doc. And then just here's some contact. So if you need to contact me, that's my email address, Luz De Los Santos. It's Lvallosa at redhat.com. Hank is my coworker. He is also a principal engineer on Claire. Aless is the engineer who really backs the EXD cloud work. He did the integration into RHEL's pull manifest and the RHEL support and the Red Hat Container Catalog. So he's a good person to have a contact to. And then there's just a general email for the Quay team. So if you just need to talk about Quay things or just get, hey, I don't know who you've talked to, that's a good place to just throw an email to. Cool, so open for questions. Yeah, let's stop sharing, I guess. Sure, by the way, Luis, your camera froze at some point in time Oh, did it really? Is it still frozen now? It is very black now. Oh no, okay. I would just stop and start your camera share and zoom and that might fix it. Once the white balance kicks in, hopefully. It's coming. Oh, there we go. Okay, there we go. There we go, perfect, okay. So, let's see, I don't know if we have anything from the stream. I have some questions, both technical and non-technical. My first question is, the obvious one, which is what happened to Claire 3? So, that's a great question. I mean, why is there no three, right? So when I came to Red Hat, I came directly to work on Claire. And at the time, Claire B3 was a project that went through a lot of individuals, kind of lost steam. And there wasn't a lot of active development on it. So, luckily, I was able to work with my manager and I said, okay, let's take a new look at this, right? Let's look at what's been done. Let's find any flaws because Claire B3 wasn't conceptually too different from Claire B2, it's mostly just feature ad. But there were some issues with Claire B2 just implicit to its design. So I worked with my manager when I got to Red Hat to approach it in more of a design analysis instead of heads down, okay, like just start fixing it, which I appreciated quite a bit because it's one of my favorite things to do as an engineer. So I got to sit down and really pinpoint, where does this thing work? Where does it not work? And we got the okay to go ahead with more of a green field project. So that's where Claire Core came in. So now Claire Core is a repository under the Quay organization, which is the new engine for container scanning. So Claire Core is an interesting project because it's agnostic to Claire, right? So technically, if you wanted to write your own container scanning application, you can use Claire Core as a base engine to do those things, right? So you can use our matching technology, you can use our indexing technology and you can build your own apparatus around it, all internal API. So with that base layer built, we then took a relook at Claire and then we just kind of jumped over Claire V3 as a marketing decision and then called it Claire V4 backed by Claire Core. So that's kind of the history lesson. But as far as, you know, it was mostly just when I got here there wasn't a lot of people working on it and we really wanted to rejuvenate the ecosystem in the community. So a green field approach was, I think beneficial more than anything. I guess so my next question and interrupt me Amy, if you have something, I would have to do, I've got a couple of questions that have to do with CI CD integration, would you look, you know, number one, whether or not there's any, you know, because now that you have the CLI, right? It's possible to integrate it into a CI CD framework, even if you don't use Quay. So my question is, has anybody done that? So has anyone done that? I am not exact. So AWS is currently integrating with Claire V4. I'm not actually sure whether they're putting that into their CI CD pipeline internally or if that's actually powering, what is it, ACR, Amazon's Container Registry? Yeah, ECR, I think you're right. I think it's ECR, yeah. I'm not sure if they're just powering their registry. So historically, Claire has been a registry focused tool and it's been designed for high volume registries, right? Because Quay.io is a large beast, right? It's a big application and it has a lot of volume. So we've always worked with Claire as in the service context, right? It's got to be fast service, got to scale, got to be made of a couple of nodes that can, you know, asymmetry the scale in, you know, elastically, so there hasn't been a huge emphasis on CI CD from internal. External, I would imagine that there has to be at least a couple of people who are piping Claire into their CI CD pipeline. I don't know of them firsthand, but I do imagine that they're going to start springing up. And I also, me personally, from a design aspect, it's one of the things I see as the next big step for Claire. Other than just pumping in language support, one of the big initiatives I want to undertake is making this CI CD integrable, right? Putting this directly in your pipeline. The way we go about that is the interesting part because, you know, we do have security databases. They're large. You really don't want to run those pools in an ad hoc manner during pipeline creation and pull down. So there has to be this some discussion about, you know, do we host the databases online now? Historically, Claire isn't a SaaS, right? Claire is a software project. So if we go into CI CD, we need to start discussing internally, does Claire become a SaaS, right? Does Claire start hosting security databases? How big are those databases? Like, say file size. There can be pretty big 30 to 60 gigs, yeah? Okay, so not something you want to be running on your laptop for pre-tests. Yeah, so testing, if you're setting up like a local development, I mean, you got to have some space on your hard drive. It's not, there's no way around that. We pull, I can't remember the number off the top of my head, but it is in the 200 million vulnerabilities. So you could have a centralized database. Exactly. That your CI CD has the manifest, sends to Claire even before you build your containers in your CI CD. Exactly. So that would give you the extra security before you even start testing anything. Yep, yep, and that's kind of the things we've been throwing back and forth. We're also playing around with ideas of making the indexer completely localized, which is a much more achievable situation because the indexer database, on the other hand, can stay pretty small, right? It's pretty condensed. It's just numbers and strings for the most part. So now that's another option, right? Is that you take the indexer, you hook it up to SQLite or some kind of embedded data store, you run that locally, and then you just pump out that index report as either like a CI CD artifact, and then your CI CD system just asks the matcher, a hosted matcher somewhere in the cloud or maybe a SaaS we provide, to analyze that index report that became an artifact of your CI CD pipeline. So that's another way of going about it. Finding that bright fit, I think it's just gonna take a little bit of talking as a community and understanding what use cases we're really after and what makes sense, right? Because we can internally as a team, we can come up with something, but we would like to actually do this with all the players involved. And you said the community is meeting next week, March 9th. March 9th, yeah. Do you have any information or should everyone join the mailing list and it'll be sent out there? So you can grab all the information right now just on our GitHub. So if you go to github.com quay Claire and then go to discussions, in, we've been using discussions for a lot. We've been using discussions for Q and A. So if you have support issues and we've been using discussions in GitHub for open design, you'll see, we just recently did the Prometheus metrics. So you can just read exactly how we're going to implement Prometheus metrics and we'll be doing the enrichment specifications there too. So open designs are there too, but there's also a discussion topic for community development meetings, which has the calendar, it has the dates and the times. So that's probably where I would go to grab them. I would also join the mailing list though because it's more of a, you know, a pingsy, right? It's like, hey, we're gonna do agenda enrollment now. If you go to the doc right now, fill out your agenda stuff, we'll go and review it. So it's a little bit nice to be on both. GitHub, as far as I know, maybe GitHub notifies you if you subscribe to discussions, but it's not the same as Google groups, which just kind of shoves that email in your face, doesn't ask you. Yeah. Do we have time for one more question from the audience here? Yeah? Yeah. Okay, there's no clear documentation on trigger only on branches and tags matching a regular expression for Quay Get Trigger Builds. I think my first question is, does that have anything to do with Claire? And if so, is that something you can answer? That is not something I can answer because that is a very Quay related question and I don't know that, I'm sorry. I dropped a couple of links in there to help out. Yeah. Narendra, let me know if you need any help. You know how to reach out to me. Okay, are we at time? We got five minutes. Five minutes, okay. So one of the other things I'm gonna do because partly because it lets me plug something going on next week is part of, you know, with Claire and everything we're having, we've revived a discussion we were having in the industry over a year ago before everybody kind of forgot about it for a while, which is what we call secure supply chain. Thanks to the good folks at SolarWinds. And obviously, Claire can be a chunk of secure supply chain in a container application stack. What other pieces would you need that Claire doesn't do? Yeah, yeah. So it's a really cool subject, you know, despite the negativity that revolves around it, it's something that the industry definitely has to do. But you know, to really, as far as I understand, to really get a good supply chain mitigation strategy, you need image providence, right? You need to understand what's in an image, where it was built, where it has transferred, you know, you need these kinds of change of integrity. And that's something that, you know, could happen from the registry layer up because when you're pushing containers there, you are able to understand the build system, especially if you're doing builds at Quay itself. If Quay grew some kind of verification system, you know, maybe notary, something similar to a Docker content trust, then you start to start filling out that chain, right? The only problem with like, notary is only good for that last step, right? It's only good to say, okay, this container was definitely created by this user, right? I know there's some projects too, which give you like granularity into those policies, like IBM has one, I forget the exact name, but you're able to say like, I will only pool containers from the author, Tom, who pushed it with this key. It's based on the tough spec, but that's not enough for supply chain mitigation, right? It's that image province, which I don't think there's a full solution to today, right? But it really needs to happen that, one, you have to get to a point where you're having reproducible builds, right? That's so easy to mistreat, right? As soon as you put, you know, from Ubuntu, even from Debian, you know, like stable, that's a moving target. You're not gonna get reproducible builds, you can't hash the containers content and get the consistent, similar container. So step one, I think that, you know, some tooling around creating consistent build environments, which don't allow you to kind of get these, these drifting dependencies that come into what looks like the same container build. From there, that has to be verified. And then the CI CD system that actually created it has to be identified and verifiable. So I think there is one project that I know that's really trying to do this end to end. I think it's called in Toto, in Toto. If you check that out, it's a framework. I've been trying to see, you know, whether we can leverage that, but they seem to be the only solution that I know of today. I believe it's actually from the tough individuals, the update framework. I believe the same individuals are now doing in Tutu, which is really targeting that end to end. I'm going to create, you know, tough keys, which verify the CI CD system that verifies the literally the git convict this code came from, chain them all together and then provide this as like an artifact. So yes, Claire can do a portion of that, right? Claire can tell you, okay, what's in this container? And then coupled with Quay with some notary magic, we can say, okay, we trust what's inside this container, but there's still missing aspects there, right? You really need that end to end to get supply chain mitigation, which is also why it's, you know, it's a hard problem. It's a really tough problem. It's also, again, it's there, I think that's going to be the next big explosion of like security tools that we're going to see is this, especially with the SolarWinds debacle, you know, like we're definitely going to see like end to end encryption mechanisms and verifiability for the entire build and contents of the container, which is something that we need to get there, right? Yeah, I mean, totally. Okay, go ahead. Because we're running out of time, we have one question off of YouTube. Are there any plans to integrate Claire and StackRocks and will they complement each other? Fantastic question. So I'm allowed to talk a little bit more about it now that they're actually bedged in and completely merged into Red Hat. So StackRocks uses Claire V2, which is great because their underpinning technology is already the same. Moving forward, there's no official talks, but I cannot see a world where we're not collaborating on Claire V4 together and migrating into a single solution, which does, yes, complement each other, most likely does not conflict with each other. So it's nice about, you know, why I love my team and the position that I'm in is because we work more on Claire as a upstream project and there isn't room for a lot of competition there when other products like this, especially if they leverage Claire come into the equation, we're working together to either, you know, work with their Claire V2 foundation and then to definitely get them trying out and integrating Claire V4 into their platform too. So I'm actually pretty excited about that. One, because it looks like we have a group of people who know Claire very well just plopped into our laps, which is amazing, right? That's like super helpful for everyone involved. Yeah, exactly. So I'm pretty excited to work with all of them and I'm excited about, you know, what comes from this integration and also excited about just the stance that Red Hat has been taking on security in general. And, you know, it shows that that acquisition kind of shows like, okay, we need to make a commitment to this. Okay, so coming up, one is if people were interested in that discussion about secure supply chain, we're going to have an hour long discussion about that at next week's container plumbing days. We dropped a link in the chat to register for that event. So that'll be on the 10th, so you can go to the Claire community meeting on the 9th and then go discuss container security on the 10th. And then for the next Cloud Tech Thursday, Amy, you've lined something up. Amy, yeah, go ahead. I was muted, I had to find the button. Yeah, so in two weeks, we're hoping to have service telemetry framework, which is a monitoring system that runs on Kubernetes and monitors OpenStack. Specifically, it's been installed on OKD and OpenShift and then monitors over to OpenStack using Prometheus. So I've included a link to their documentation. So unless something changes, that's what we'll be discussing in two weeks. So thank you so much, Lewis. You're welcome, my pleasure. And thank you to my co-hosts, Amy and Chris. And really cool, sort of in-depth look at Claire and everything. And thank you, everyone, for attending and for your questions. Thanks, everyone. Thank you for your questions.