 So, hi everyone, so my name is Michael and this is a session about Secure Software Supply Chain and I will cover the kind of basic concepts of how Secure Software Supply Chain works, some of the history and what we do there on the operating system level and with me I also have Stacy and she will talk more about the management part of what we do on the Secure Software Supply Chain and what tools do we have there and if you have any questions feel free to ask during the session or at the end we will have some space and those who will have like good questions and every question is good they can get a nice t-shirt so write books down, prepare the questions and you will have a chance to get your answers as well hopefully. All right, so supply chain what exactly it is and why we are talking about it so the supply chain attacks have been recorded fairly recently only so there is an European agency which is called Anisa which has been doing monitoring of supply chain attacks since 2021 and you can see on the picture that the amount of these attacks has been on a rise so in 2020 actually 2020 was the start when it started to monitor these attacks so it got like four of them in 2020 or four to eight and then it raised to four times the amount in 2021 and this has been on the rise since then and since this I mean this is not a new problem but only recently it was it's something what people started to pay attention to and supply chain attacks have also been something that's became easier as the resources to run these attacks are getting more and more available it used to require like state nations to to run these attacks but lately it's so easier it's very easier so that even like small groups or even individuals can can pull off these these attacks so be aware of those and prepare for for them because it's something that's given what's not going to go away one of the first actually the most famous attack which started all this was the Swarovans hack so it's a company which has been working on the which has a software for infrastructure monitoring and network monitoring and the that software has been in use by many customers including government agencies worldwide and the software is called Orion and sometime in early 2020 a group of hackers injected a malicious code into into that software so they broke in to the company tools and premises and injected malicious code to the source code of their Orion software and the Swarovans didn't know about that hack back then so being unaware they actually spread that hack to their customers and at the end exposed their customers environments to the to these hackers so it was pretty serious it affected a lot of companies and this was one of the first attacks which popularized the supply chain supply chain security needs and attacks and made like most population aware of their their existence so what's supply chain what's the secure supply chain so we talk about supply chain security and the definition is about securing documenting and tracing so you know that's not about security itself it's also about the option the possibilities to document and trace the all the details regarding sources binaries dependencies updates and all of that regarding the specific workload and for each workload you are running you should be able to document and trace all these details and not only upon its initial deployment but also throughout its its lifecycle because the application is deployed and when it's running it's being updated and throughout this this lifecycle at each of these times it can get get attacked and exposed to to something malicious and why do we care it's easier as I have said the number of attacks has been increasing but also there are other aspects you should keep in mind because software nowadays it's not only data centers right so you get software in cars you get software in appliances you can software in whatever you can think of and each of these software components is a possible target for this supply chain attacks and you probably don't want to sit in a car which suddenly gets a malicious code injected because somebody forgot to verify one of the dependencies because these cases can become pretty harmful at the end and the other reason why we should think about supply chain attacks is are actually human errors because social engineering is a thing nowadays it's actually easy to get hold of like contacts within a specific company and convince those contacts to like somehow get some parts of a code replaced by something malicious or put a USB stick somewhere and get the environment infected so these are the reasons why supply chain security is is important supply chain as you know is a chain meaning it's as weak as its weakest link so we need to take care that all of the links in that chain are actually secure so from the start to the end you need to take care of all the components of that chain so there are different threats which apply to different parts of the chain you probably can't read it but this picture illustrates the lifecycle of a piece of software so it starts with development which creates some source code the source code gets built packaged and distributed to the consumer and in the build process and in the development process it's consuming some dependencies from the outside and there are multiple threats which can target different parts of this process meaning source code threats you can get like malicious code in the source code you can get compromised source code versioning system you can get build threats meaning some source code gets injected in the build process which was not controlled by the source code control system you can get like compromised packages used during the build process you can get like bad packages so you expect some packages being used but at the end those packages might be compromised already and you can have dependency threats and that's something what many people do not realize today and actually they are fine if you are developing application they are fine getting very dependencies from a public repository so if you are developing in python you can get pipi packages same applies to Node.js, Go, Rust all these programming languages have their ecosystems publicly available and obviously not all of these dependencies are strictly monitored for malicious code so this is one of the frequent reasons when and how malicious code can get to the application and that applies also to not only source code attacks but also nowadays with containers which became very popular people or developers are very used to just grab their dependencies from repositories like docker hub and like half of the docker hub is not maintained for like people just injected their containers there and left them running and others are happily consuming these containers despite having a security hole so that's another point of view you need to keep an eye on what you are consuming from the outside and while it's very easier for developers to just grab stuff from third-party repositories be there that you better take care of what's being consumed so you had a question perfect so great question and I will actually cover some of these parts in my next slide for the presentation but the simple answer is you need to make sure that the source they are getting your software from is secure and that can mean multiple things so ideally you have a trusted vendor who is providing these these packages to you and they are doing the audits or you can have your own team which is doing the audits on the packages you are consuming so you can do it yourself you can have a vendor but you need to have kind of a process in place to make sure that there is no third party like interfering with the code you are actually consuming so let me carry on with the presentation I think it will answer some of these questions if not just raise those again and I will I will carry on I've answered thank you so that's actually what I want to talk about right now how what are the ways how to how to prevent secure so how to prevent supply chain attacks using some of these methods so security is about like multiple targets and I'm going into detail here so the primary targets are confidentiality meaning that information is exposed only to those who are or who should be able to access that information the other target is integrity so that information is not changed or altered on the way to the to the target consumer and obviously availability meaning that when the target consumer needs that information it's readily available and ideally not at other times for security purposes and there are multiple ways how to make sure that security that these targets are applied to the software's supply chain processes and tools and one of the recent methodologies is called salza it's a google's specification and also kind of a certification of supply chain security and it specifies how to address some of these some of these threats how to avoid them actually and how to make sure that your supply chain is secure and it mandates some some measures so if you want to have your supply chain deliver delivery certified you know what are the measures you need to apply so for instance for a source code it mandates like two person review so upon any code commit there needs to be like two people there that review those specific commits and the other source code related is a version control system so there are measures like about the the machine security when the verbal control system is running there are measures how the control system should be set up and that applies to the other parts of the other threats as well so build threats for instance salza mandates control build environment so the built environment needs to be kind of self contained so we can imagine a VM which gets spawned the the stuff gets built within that VM without any like outside influence so the build process cannot just fetch random stuff from github the build process cannot fetch random python modules all these modules need to be predefined available within the VM so that the result is is defined and ideally the build is reproducible that obviously can conflict with what developers some of the developers do today because a lot of these processes build processes are dynamic they rely on on github repositories during the build process they fetch the stuff from the internet well it's convenient but it can be risky so it's always like a trade-off between convenience and assurance and security and the other related topic is its provenance of pretty much all the data which are related to the build processes so meaning it should be locked what's being used for the build process how the build process is running there are the like what what arguments have been used and all that is available not only during the build process but also after the build process so that whoever is consuming the outcome of a build can independently verify later on how the build process went what was used and if there's nothing which is which is wrong or or insecure and obviously security measures for the build machine and the build VM are important as well yes please do you do um so that's a good question you can do security checks at any time appropriate or what fits you it's important to make sure that there are checks in place so for the build process itself the important aspect is that it's reproducible and it works with a well-known source code and data and packages and input right and if you check this beforehand when you are fine as well because when during the build process is if the environment is confined and limited then nothing can be introduced during the build process because you work with a known input and the output is deterministic right but you can obviously run build checks and security checks so the important thing to keep in mind is that this is kind of a repeated process so at the time of the build you what i've been describing right now is that you are making sure that the security aspects are met at the time of the build but the next day or next hour even there can be a security issue found in one of the libraries used in like third-party libraries in your own code and you need to react somehow and you just repeat the build and you either fix the bug yourself or you get a fixed third-party dependency or you remove a dependency and repeat the build and get a secure outcome again but this is really over and over and over repeated process absolutely absolutely yeah yes exactly that's how it's how it probably has to work nowadays i know it's like so that's related today we need to be able to react fast unfortunately the threats are coming on a like hourly daily basis and we need to be able to to react and respond you can limit what you can limit the exposure so you can be application contained in a like a VM in a container you can have a controlled environment so you can be relaxed a little bit but then obviously eventually you need to update the application or its dependencies somehow and there are measures how to update the application in place so like we do have stuff which is called live patching so you can update applications while running so that the performance and function is not affected but sometimes it can be used sometimes it can't so it really depends and thank you for your question regarding dependencies that's actually what i've already mentioned but a lot of these things which apply to source code and build threads apply to the dependency threads so you need to keep the provenance of the data there in case you get a like third party library you want to know what version it was what it contained what files it has and then how it was used for the build and be able to check and check that later when you are building the stuff or when you are verifying that the application was secure secure beforehand and obviously you want to check your dependencies i will probably be repeating that because it's what this by being annoying part of the process needs to be happening right so this is a picture of what we actually do as part of a building our operating systems less so we work with the community code and then take the code through this process up to the customer systems so we work with the community and we make sure that the community development process is sane so like generally communities are trustworthy so we upstream at least most lively ones they have like a regular review they have multiple maintainers and they are part of a maintainers group as well so we make sure that the upstream code is is safe but then we also need to make sure that throughout the process speak before it gets to the customer it's all verified and and secure so there is a code review before it gets injected into our build build service then there is a package selection meaning like how which components are selected for that specific package build then there is a build process and we are using the open build service tool for that which does exactly this confinement so each each build has its own VM it has its all its sources source packages and dependencies in it so it's reproducible it runs without any for part interaction without any external exposure so we know that whatever was in there is also getting into the into the output and there is nothing like what what can interfere with the outcome that gets tested and this testing can be also security testing but generally it's like functionality testing and then all this is signed and delivered and then customers can consume it so the packages are signed the repositories are signed and this way the customers the customer who is installing that specific package can be sure that during this build chain this build process nothing has interfered with that code and he's kind of safe to run it but as you said next day there can be a security issue somewhere so well we need to rebuild the package and we actually do it so some packages are rebuilt pretty often and each build is going through that process from the left to right and updated package can be available like in an hour and then obviously all the application developers need to grab that updated package and install it again all right yes please so let me answer a second question first so the signature is valid for for everything what we release so we actually we have a like signing key which we use to sign all the packages we have a signing key which we used to sign repositories and you can verify if you download the package from from suzer repository that you can easily verify that it's actually coming from us it's like usual usual signing mechanisms which are used so you just verify it the ways the same way you are used to verify stuff so that's actually pretty straightforward when it comes to the build processes within the company I mean it's it's tricky obviously because it's a trade-off between effort and security and like how much time you spend on on securing your environment and it really depends how much sensitive information you are processing how security there or who are risky environments you are running or you have inside your company so some companies are more security conscious or aware because they are running like processes customer data in some of the departments and in some some pieces some parties can be easier like if you have like POCs open source developers most probably do not need to be that that aware on the other hand you need to make sure that the overall infrastructure is not exposed to to attacks so it's fine to have open source developers grab stuff from github and developer applications as long as if this development process is not injected injecting the like malicious code into your production environment so this kind of separation can help the other aspect can be that you actually have a like team which takes care of the basic building blocks so we do have some customers like in in telcos which have dedicated team to the operating system and its components and they prepare repositories for the development teams and in some of these industries you are actually mandated even by law that you can't just grab random stuff and develop with it and so this team takes care of all these dependencies so they have like a operating system they have python python interpreter they have a set of I don't know 300 1000 python modules ready available to the development teams and these teams need to use only these modules to develop their application if they need another module they go to the team and the team goes to the vendor they talk to us and to be kind of provide these because this is really like contained environment and they cut efforts they are not even allowed to use like random unsigned or insecure things we need to have guarantees and support for for many years for these components so that's a good question somebody asked me the other day about s-bom as well I don't have full details what I can tell you is that we are exposing the complete source code with all the build flags to actually publicly we are we are only doing open source as part of our products we are exposing everything what we use to build stuff so if you install a package like from the latest version I pick like slash 15 sp4 if you install bash package you can go to sources.sus.com and look for for this specific package and specific build and you will see the source code for upstream you will see the spec file you will see the patches which are applied and the spec file will obviously contain all the details about the flags which have been used and the patches which have been applied so that's how you can you can actually build it yourself at the end using the source code and verify that what we are distributing is like the same depending the signature because you won't be able to sign it using the suzer internal key and one more question yes please so jeff do you happen to know what level is that all right so yeah so salsa is fairly new thing so google used their own specification to come up with salsa levels and we have been actually certifying against salsa just just recently so that's what we do and speaking of certifications there are more of them so salsa is a recent one and the one which can be used right away for software supply chain security there are more of those and I'm not going through the details because there are like different bodies different certifications and different different purposes and I like actually uh this table more because it illustrates what kind of certification are there and this is by no means exhaustive list there are many of those available but these are the main ones which we are facing when we are talking to our customers so common criteria is actually my favorite one because it's pretty complex it covers the complete environment so it's not about it's not only about the software itself not not only about the software components but also about the processes which are used to build the software it's about the processes which are within the company used to decide about new features about communication within the company it's about the it environment how the service are set up it's about the physical environment how the offices are used so that they check for the stuff that you can't enter the premises without being authorized and they're actually these offices are going to those offices checking the environment that you are really secure so that's pretty complex and there are multiple versions of common criteria certifications the other one which is mostly used in the US is FIPS it's about crypto algorithms and random generators it defines which crypto algorithms are secure enough and which can be used and STIC is about documenting how to make a system secure so these are sets of security guidelines which you should apply when you are setting up a system and configuring the workload there are a bunch of other certifications PCIDs for financial data so you need to figure out which ones apply some are mandated by law if you are processing financial information or customer data PCIDs this might be already something what you need to adhere to others are voluntary but you should keep looking for vendors who are certified against these certifications because it also improvements the overall supply chain security processes and environment common criteria as I as I explained is the most complete when it comes to the topic there are multiple levels against from one to I don't know seven the most the biggest level reachable for the regular operating system general operating system is level four because up to level four it's about the software and processes how it's built what components are used if you go to level five and above it requires requires formal specification of algorithms used in the software and that can't be really done for Linux nowadays so there are specific specialized operating systems which carries these higher levels of common criteria certifications we do have EL evaluation assurance level four and we actually have four plus plus means that there is a formal specification of the processes related to to remediation of a security flaws so if there is a security issue then it's defined and prescribed what the process is like who reacts how how they react what do they do so it's something what you actually ask for so this process is well defined in the in the security the certification criteria we have to have the documentation we have to have follow these processes to make sure whenever there is a security issue that this timeline is fault in every case and updated package updated component is released to the customers in a timely manner so that's I like this certification because it actually is something useful unlike some of the others it shows that the environment of a vendor is actually really secure according to the certification body and common criteria is used worldwide so there are some some countries which have authorities issuing the certification our countries do accept the certificates so it's pretty pretty useful certification and as I explained for some industries different certifications might be relevant like FIPS is relevant for the government agencies obviously specifically in the US they are mandated to to use FIPS certified processes and other industries can use these certifications as they see fit so really look for the vendors based on what your company actually does and this is a picture I showed before it's just to reiterate the fact that the certification should apply to the complete process so if I would be just certifying the delivery chain delivery part of a chain that's really not very useful because the rest needs to be secure as well so our certification is applicable or applies to the complete chain how we deliver software to our customers from the community code through all the development building testing signing steps and you can obviously do something similar with your software when you are developing you can also look into these certifications how we are how we are done and yeah that's it any over to Stacy to talk about what this any more questions right now can give you mine and you hear me all right so now that you have this great software implemented and secure and it's in your system how do you keep it secure well you need to patch it so I have a question for you how many of you have a defined patching schedule that you use on a regular basis and you know your systems are secure she gets a t-shirt you already have one all right so unpatched systems cause more than 60 percent of software secure cyber attacks and sometimes your cio's and your cso's are actually going against you because they don't want their systems going down and 56 percent of 56 percent of attacks are coming from vulnerabilities that are more than four years old that means that these systems have not been patched that's a problem so patch management is important because it fixes the vulnerabilities and it decreases cyber attacks it also is a proven a way that you can prove that you're compliant so for PCI for example you need to show that you're patching your systems on a regular basis and of course patching goes beyond software fixes and make sure that your systems are the best for your users so how are you managing your infrastructure some more numbers 33 per six million dollars is the annual cost of downtime due to cyber attacks that's a lot of money and you don't want to be responsible for it cyber attacks globally the price is going up 4.4 million dollars per cyber attack and the average annual hourly cost for cyber attacks $400,000 these are coming from a gardener report so susa offers a product called susa manager susa manager is the only it infrastructure management tool that manages and secures non-susa infrastructure so if you're running sluzz our product you're running red hat you're running ubuntu debion you can secure it from a single console with susa manager we scale with our hub architecture from 10 clients to up to a million clients and you can prove through reporting and dashboarding that you're keeping your entire mixed linux environment secure healthy and current and we allow scheduling and automation through salt and those ansible playbooks that you guys sitting around you can import those into susa manager and with our last release of susa manager these are the distributions that we support so everything from r sluzz 12 and 15 the r slay micro product and even the new alman rocky litics that is the said to us clones that are out there you can support it and we have customers that are using susa manager that are not using anything else susa we don't particularly like that but we do so just to go back a little bit susa manager pulls in the scap protocols the scap checklist and can validate your systems using the openscap technology through susa manager we pull in the cv checklist for common vulnerable vulnerabilities and we can check against your systems and provide those patches and then prometheus is a real-time monitoring solution that allows you to use that data real-time data to create grafana dashboards that are graphical that you can share with your cio's your directors your ciso's etc and i just want to go through a quick case study office depot is one of our customers they are a leading provider of business services they have merged with other companies i think the most recent merger was with office max and they were going through a lot of challenges because of all the mergers and acquisitions and the different infrastructures coming in they were seeing an increase in cyberattacks they also needed to minimize the cost of management and security they wanted their people to do more interesting things more business oriented things and they were just really wanting to get to a cloud first strategy so they brought susan they were they were able to save more than 40% in their it management costs with the combination of susa linux and susan manager they were able to automate their processes for patch management get themselves on a regular patch schedule and just basically really simplify their data management and their their engineers are now doing things that are innovative for the business and with that i want to thank you and ask for any questions so thank you guys we are out of time but we will be here around until friday we are in the sponsor showcase if you have any further questions happy to talk just reach out