 Okay, so my name is Diego. I'm not a security specialist. Okay, but I am Conan co-founder So I'm going to tell here my story here. Please be be kind to me because I'm trying to explain how we are modest approach to secure Conan Center Please read your hands who knows what is Conan read your hand if you know it one person to people. Okay Okay, perfect. So Conan is the missing link. It's something that we have been missing in C++ for decades It's a package manager for C++. It is open source. It is decentralized. It is a Multi-platform it works in every every operating system with any build system and it's very mature and reliable and stable I'm going to throw some some numbers here For example, we have been designated as one of the 1% critical by PI projects You know that what that means we got we it is downloaded a ton of times Because Conan itself is a Python tool. So we put it in the pie in the pie by Python package index and In the C++ Language a slack team. We are consistently a rank among the top most active channels in the whole in the whole place We get thousands of pull requests per year to our repos We have a dedicated team 10 people Sponsored by JFrog as maintainers of the conan project And there are many thousands of companies using Conan in production This is the number of artifact resurface that we know those that are not behind a firewall Those that we get that the telemetry its ranking last year was ranking around 8,000 Companies or teams using Conan that means of course a lot of support tickets We get a lot of feedback from them hundreds of video calls and a top of supporting general From these users, we selected a small group of them and we made also a group of We call it the tribe the tribe to see there's a group that has been giving feedback For the next big thing which is Conan to see that which is coming in a fully in a in a couple of months What I'm explaining this Besides the same less self-promotion Yes, I'm doing this is because you need to be aware that Conan is becoming a very important piece in the cnc-plus-plus ecosystem It's a piece that needs to be secure and today. I'm going to explain what we have been doing so far to try to Secure the supply chain for cnc-plus-plus packages with Conan First important thing the technology we manage the binaries So we have this concept of a package and we have a recipe And the recipe one single recipe can create any number of binaries for every different configuration every different operating system Everything compiler compiler version debug release configurations static or cert libraries That's that every one of those things is a new different binary and Conan can manage all of them This is the way that these binaries are created We have a recipe is a DSL on top of Python that describes how a package is built from sources And what does a package contain and together with the recipe? It goes to Conan data.jammel that can be used to put some data like the URLs and the and the checksums of the Turbles that is going to download to to build those those packages And from the architecture point of view Conan is decentralized So you can easily have your own server. We have an Artifactory Community Edition, which is completely free for c++ Okay, so you can have the whole stack the Conan client and the server For free and you can put your own private packages in your Artifactory server and get them from there and Besides that we also have central repository as npm For example, which we call Conan Center Conan Center is the place where we host the third-party open source packages Those are contributed by the community, but we maintain the infrastructure and we run the repo To give also some ideas about what is Conan Center Conan Center contains nowadays 1400 recipes for each one of these recipes it can contain in a number of versions for every different versions It can contains different revisions that happens with you do a change to the recipe but you don't actually bump the the version of the of the underlying library and For every single revisions We are building up to 130 different binaries for Windows Linux Mac different flavors of the compilers as you can see This is a big infrastructure That means that more or less we are maintaining 1.4 million packages binary packages in Conan Center which accounts for 20 terabytes of storage we are getting close to 1 million package download per month and We that means also like 25 terabytes of transfer per month So this is basically what is Conan Center content that contains two parts the first one is the package system It's a system when we are creating the packages from source everything start from a github to request We build the package we store the packages and we serve the packages so the Conan client can do a Conan install and get the packages from the from the servers and This is goes together with a web with a web application that we can use to navigate and explore and discover these these packages The package system is is a CI We are we are using Jenkins here and we are using the different agents Windows Linux and Mac agents Here to build the things we put the binaries in an artifact to repo And then we put a Google CDN on front of that for for scaling And the web system is a typical web application We are getting the metadata from the packages that are already in the artifact server We get them we crawl them and then we extract the license the version all the things that we need And we put them in some storage basically Google storage and a postgres database And we serve them through a fast API and next is a very standard web application From that application we we have this this is the front end of of Conan Center as you can see is relatively simple It displays the packages one important thing here is that it doesn't have any authentication Users don't need to log into this system. It's a read-only. It's a read-only page So that means from the security point of view the web part of Conan Center is very secure We are not concerned about it at all Maybe you can get into it, but even if you do it the information there the database information It's absolutely public It's something that you can get from the packages that you can get from Conan Center So it's not that we are hosting any sensitive information or anything like that As you can imagine our major concern is about the package creation process Because yes, we have the Jenkins. This is the first liability here. So we put the Jenkins absolutely In a VPN you cannot access anything from from Jenkins from the outside But you are still processing them the pull request from github and this is the main attack vector for for for Conan Center, of course So what we did here is we put in place a very strict Contribution process this is outlining our process several stages here and now I'm going to explain a little bit about them The first one is a user check. So the users when they do a pull request to Conan Center repo They first they need to sign the CLI. They are actually transferring the copyright of their contributions and then Jenkins the first thing we'll do is check the github username It contains an allow list if the user is not in this allow list the pull request will not be processed at all We don't we don't build anything and the users can request access For this for this allow list in doing also a like a pull request that is processed like two times per week And it's manually reviewed. So we get the users That that has a contribute permissions to Conan Center accepted manually as well Then the second stage even if you are an outright user the second stage is the static analysis so we get the diff from the pull request and we we have several tools there that analyze that diff and We check things for example that if the pull request is doing a version bump because it only changed the Conan data Jamel file with the specific URLs we check the URLs we check the checksums We check everything this is looking good, and then we annotate it as a version bump We can detect also things like a this is a dependencies bump Which can check the consistency of the current it about Jamel So everything that we can check with a with a diff analyzing the files there before building anything is done We check that the users are not touching in any folder for example the configuration folder that we have we have very little But we have something they are not allowed of course of modify anything They are touching the configuration folder then the pull request stops Then the next stage are some quality checks. We have here. We are using two tools here We are using first the github actions, and we are using some lint in there Okay, it's not a strict security measure But if you are doing something weird it is likely that these these quality checks might raise an area You are doing I don't know trying to obfuscate something or something a linter can can easily say hey You are doing something weird here, and then we will have some some insights about that And also in the Jenkins side we are running some of our quality checks That they contain for example they check that the recipe itself doesn't contain any URL For example, or it's not trying to do a W get or it's not trying to do something that is trying to connect to the to the outside So we run also some quality and security checks there Then we get to the bills when we are actually sending the job to the to the bill agents And they need to start building and this is the first thing that is a bit more more delicate because so far We have using temporary posts So everything that has been running so far runs in a pot that is is created on the fly and is destroyed after so even if you Get access to that pot it will be completely isolated It doesn't have any permissions and and whatever you do there will stay there and will be destroyed But here we also need to build with Windows and Mac And of course as you can imagine that that is also a problem So for that the best that we we are able to do so far is we we have automated Those machines so we are using ansible and we can periodically rebuild those machines in case something goes wrong Not only because of security just for our own sanity because you can get maintaining this infrastructure a scale for Windows and OS X is not that as nice and the Kubernetes So we do it, but we also use it as a measure to a let's let's build these machines from time to time So they are fresh and they are and they are clean In that step we also do some some binary checks This bill is actually building the binaries So we have some hooks there that will check a you said that you were building a search library Is there an actual DLL library here are the I'll say I'll say the magic numbers of the DLL Whatever we can check about these binaries created. We also we also check the artifact that has been built and finally Everything that is built at this stage is put in an Independent isolated Artifactory repo that is created on the fly for this specific pull request So even if you get access to this to this stage and you create some malicious packages and they are uploaded by this process They will not be uploaded to the production repo. They are uploaded to a clean independent repo for from independent also from every other pull request and finally the manual review We require this for anything to be merged into Conan Center and we require in this case to positive reviews one review from the from our official reviewers these are people that we have Selected from the community like heavy contributors to Conan Center people that we have been talking to them We have been doing video conferences with them and we meet them So we put them in an allow list and we made them official reviewers So they can do a positive review and that counts and then it also requires a positive review from a maintainer This is the stage where where the the reviewers will will care for a let's do the due diligence of what is happening here And they will do checks that are otherwise difficult to automate for example Is the URL that is being used the real URL of the project is the checksum there is everything correct is using a Some recipes might be using mirrors for example because the network might be flaky or the original source is also a bit a bit unstable So if there are mirrors, we need to check the mirrors as well So everything that a developer will will do is done at this stage And finally when everything is good when everything is clean only at this stage the merge and promote mechanism kicks in Here again, we are in a in a new in a new temporary pod because this this pod is the one that has access Right access is the only one in the whole system that has right access to the final Artifactory repo that is the production repo for for Conan Center So this is basically a product is in the Artifactory API to copy to promote packages from the pull request repo To the final production repo, which is typical with practice for package management at a scale in any in any case in any case besides all these review process and Contribution process we try to follow also other best practices So everything that we can restrict permissions we do we use two factor on the indication everywhere for every service that you We can do Every single piece of the system is as isolated as possible It was mentioned it before but our images if we can do something in an image that doesn't have a cell at all And it's just running some command or something we do it we try to maintain things as as minimal as possible We store logs of everything give have us a record. This sounds Easy, but it is actually cool It doesn't matter once you are using github pull request as a mechanism you do a pull request and it stays there You cannot remove it you can remove the source code But the pull request itself stay there So you cannot do something try to remove because it can't maintain that that information there And we try to maintain of course everything all the pieces there updated to the latest versions But let's say that and at some point you got access into our system Right and then you manage to create some malicious packages of popular C++ libraries Let's say that you infected boosts or open SSL or set leave there And you put them in artifactually Conan Center in the actual production repo. Okay, this is really bad news But probably is not as bad news as you expected if you if you thought a I'm gaining access to all those cool Companies fortune 100 companies using Conan in production that Diego was telling before No, this is not the case and as you can get a C++ It's probably different to all the other ecosystems out there Before you got being commenting about other other languages and other package managers and Conan was not there It hasn't been a concerted farm C++ is different The community has been doing different things for decades because they didn't even have a package manager in this case 80% 80% of the users are not using Conan Center binaries in production at all Okay, there are and this is goes proportionally also with the size and the value of the company if you are one of those big companies That are using Conan I can tell you that they are not using the Conan Center binaries in production. None of them are using them Okay, so the metric that we saw before they are just from the 20% of users that are actually using those binaries in production typically hobby users startups small companies small teams Exploration typically trying out things but not in production So what are these companies doing is something that should be done for every package manager? It was commented before in a previous talk So one of the approaches that are personally 20% of the users are doing is they are copying the binaries from Conan Center They do a copy a control copy typically a developer does some some out this process They check a I'm getting this version and then I'm copying it into my own Artifactory instance And they they cut access to the to the Conan Center completely and they even have many of them You probably don't imagine because in other words is is unthinkable But many of these companies they have a process they have a meeting to be able to upgrade boost from one version to the other Okay, so it's not something that happened. Hey, let's just get the latest version of the boost library from package So they have this control copy from time to time and that time might be two months Some maybe it's not best to get the latest and newest version for a library But it's very secure from the supply chain attack point of view because those binaries They are not easily getting into their organization And even though that was only 20% of the users 60% of the users and all of the important ones. They are doing this They are building the packages from sources They are actually doing the due diligence over the Conan recipes and they are building the binaries from source And we are very happy about this only in your own binaries. It's a good thing to do. So hey So all of these for nothing. What is Conan Center worth them the knowledge base? Containing the Conan Center repo is huge building C++ is a challenge building the third-party libraries out there It's very complicated Conan Center contains the recipes It instead of a read me that it automates how a library is built from source for Windows for Linux for Mac And that is super valuable and then Conan can automate a you create the binaries I sort the binaries and then you don't need to rebuild them from source I will be serving you your own binary that you created for all the next build as you that you are doing Okay, since also since JFrog acquired a video the last year We have been also working with the security team at JFrog and we have been doing several Improvements, so I'm going to tell a little bit about our ongoing work First we have been checking a are the Conan Center binaries The best they can they can be for example because GCC is not Activating by default some some compiler flags that are considered to be secured and are considered to be a good A good practice for security and by default they are not active So we have been checking these things and we are starting we implement that way to inject These compiler flags to any build system because Conan Center contains Packages that are built with CMake or with Meson or without the tools So Conan now now knows how to inject compiler flags for every project that it contains So in this way we can inject the F stack protectors strong flag In an easy way that both Conan Center can do it and the users that are building from source the binaries They can also do it and start to use these these Practices by default It's not always possible because some things change the ABI the binary compatibility in this case We are not doing that, but we have investigated which ones that's on what not We have also been implemented package signing Not only because it's important to know the authority and having the package integrity But also because the ecosystem is also becoming a bit better There are other providers that they want to start and even companies that they are starting to Distribute Conan packages to their own customers at the same time. So if the system is becoming more federated Then it's becoming more important to know the origin who who is this package from it is this package signed So this is what we have been adding so in this case for Conan to zero because it's a breaking change We couldn't do it in one on one, which is the current version We added a package signing plugin a way that anyone can sign their packages with the tools that they want It's basically implementing those two functions there One the signing function is called when you are uploading packages because developers can create packages locally all the time You don't want to be signing all those packages. So packages are signed at a upload time Packages are verified when you are downloading them from the servers. So we have these two Extension points you implement them and then it works We sign everything we sign not only the binaries But if you recall I was in a the recipe itself is an artifact that this is stored in in the server as well So even if for some reason you didn't upload the binaries the recipe is also signed as well And we besides the plug-in which is built-in. We are also providing a reference implementation with six store Okay, we are using the record and it's an implementation that is very easy to configure You only need to call Conan config is tall and then a URL of the repo and then you get this Secret store sign in plug-in We will use it in Conan center Conan center packages once Conan to zero is finally out and an stable They will start signing the packages Okay, and this implementation will also be public so anyone can sign their own packages providing their own their own signature their own keys Okay to finalize My presentation today. I would like to to discuss a little bit about the solution Because yeah, you can say okay. That was easy Conan only only has 1400 recipes different different packets Plus the number of versions plus the number of binaries is not is not that huge compared with the number of different packages that the NPM can contain for example But it is also true that we also have for because from the full Conan team only four people are working in Conan center Okay, and the number of official reviewers is 15 official reviewers So it's also a small numbers so far and this is not a cold to be hacked Okay, we have had no security incidents Okay, it is true that some of the easy security the incidents that other other Systems have had in the past like for example the color. Yes even if the the package creator if they in the library owner Wants to hijack their own library. It is not possible Because one thing is the library and one thing is the recipe to the poor request that you need to do to Conan Center to have Your recipe include that process by the way probably it was not that clear before But that is also running a test of the package if for some reason you are putting the the colored Yes in an infinite loop then it's going to block the the construction of the package will fail Okay, it's something that our pipeline would easily capture and would easily block even even for the library author library Thor cannot do that okay the same for for For other attacks if even if someone is able to gain access to the github user of some Contributor that has already access their poor request are going to be reviewed No matter if they contributed originally get originally some package No matter if they are the library author that created that that specific C++ library That is going through the process So the question here is is this something that is worth to be considered for all the technologies? because honestly Some other technologies has been super great. I'm a happy users of them But let's be honest is they are not very secure. So something has to be done I think by PI has started to do something with a 1% of critical projects and asking us to use to factor authentication to be able to upload our own packages So we cannot be hijacked and someone can upload a Conan client that would be a disaster on our behalf Because they gain access to our account. Okay, so this is a first step But is the review process something that is is doable at a scale? I think Maven has doing something like related but the other maybe it is as Also has been commented before. I think the docker images this morning All the central package manager they they follow up right to distribution in our case I can give I can tell you that From that fourteen hundred recipes only one hundred recipes they account for 80% of all the downloads from Conan Center Okay, so let's say let's move that to the other package manager Can we review maybe the 1% or 5% of the most popular packages in npm or pi PI? Maybe we can maybe we can scale the number of reviews to 300. Okay, and to finalize How what has been our approach for Conan Center security is just a simple one is a very constrained process We only accept packages from sources from pull requests. We have a lot of automation. We do lots of automated checks And of course best practices around our system to be secure and we have a manual review process with due diligence of what is happening there Okay, we keep constantly working on that In our future is we are going to deploy it has been already implemented by the package signing and robustifying the banners It's already there. It will be part of after to zero sometimes soon We will build more user gates checking who is doing pull request to Conan Center the activating accounts when they are not active doing I'll say how many Months old is a github account before gain Forgiving them access. We are going to automate those checks that sometimes are manual to We are creating s bombs. We have been creating artifact reveal info which is proprietary for for Artifactory, but the s bomb is a standard and we will create them so it can be also integrated with other security tools and it has been working for Conan so far great because Yeah, the ecosystem scale is manageable for us Okay And also because building C++ is so challenging that if you want to hack you want to gain reputation in our system You are going to have to work a lot before you you you become a popular contributor to Conan Center You are going to have to build a few C++ libraries their party libraries and that is challenging It's way more challenging that throwing an attack to npm or pi pi And within that this is something that can be a scale and is something to be considered for other communities It can be done is working great and something has to be done in our opinion And thank you if you want any more information about this you can go to the Conan dot IO page If you find something we also have a back bounty program So that's the the URL as well. Please submit your your findings there And if someone wants to reach me or talk to me I will be here after the session as well and that is my Twitter handle don't hesitate to write to me too. Thank you Time for questions One question only. Hi there a very interesting talk Thank you. So It's too bad I have a two-part question, but I'll just ask the first I'll make a comment first about the first one that'll move the second one which is a While ago I pointed out a bunch of problems with Linux package managers, which led to them using a different signing model and One community that didn't do this was slack ware because they said everyone could just download and read the source for themselves And that did not turn out well for them. So that's my first aside The second question is have you heard of the underhanded C code contests? And Do you understand how easy it is for people to write code that looks legitimate? Especially in C and C++ and get it past code review because there's been a lot of work that's shown that that's not particularly difficult to do Yeah, yeah, exactly. So that's that's a good that's a good question I've been a C++ developer for four decades. I know how it is. So The thing is we are not doing any due diligence on the C++ code itself If the library author is doing something that is really nuts in the C++ code They are going to be able to offer skate out we can control. Let's say from the origin the URL This is the official review The official point of source of that URL and to the final point It is like if they are going something really weird. They are trying to escape They are trying to do like remote calls to some place it's possible that Something will break in the build because we are running it actually It's not the same as running in a developer machine, which those chains will be typically targeted to So if you try to do a pull request that adds to our system It is possible that that we will be there But yeah, if the source is obfuscated in the library by the library author themselves and the is that the official URL for the project then it's going to be challenging, but that's more a source Attack than a supply chain to the package binaries, which is our our scope here. Okay. Thank you Thank you