 All right, you need more security in your application packaging which is kind of catchy title Let's get the boring part out of the way fast I'm a system administrator. I work at Red Hat as a support engineer I'm a little bit of a salty guy. So don't panic if I get too cynical or angry I'm also a member of CIS army. CIS army is the Argentinian systems community And I'm also one of the organizers for nerd Darla, which is kind of like our nerd conference down there And if you're wondering how it looks like Picture over a thousand nerds all together in the same place So before we actually begin the talk, I have a fair warning to give you As I said, I'm a system administrator. I'm not a developer The opinions on this talk are my own. They're based on my experience and my observations on the subject This talk has a fair bit of jokes all of which are made with good faith and I have to tell you and I have any magic solution for what I'm about to To show you I'm only pointing out the problems and showing you what we have right now and throwing my ideas So we can ignite a conversation serious conversation at this topic and most importantly always look on the bright side of life So first things first we just saw the term security and packaging together So let's get the boring definitions out of the way and define both So packaging is the process through which an application is delivered typically to an end user whereas security No, not that one That one this type of security in the broader sense of the term information security We are talking about being protected against the unauthorized use of information. This typically involves an attacker Attacking an application in order to gain something from it like for example stealing user data Now with a show of hands, how many of you have actually related both terms security and packaging like ever It doesn't count if you saw the title of the talk Okay Good So you're probably wondering what's the deal? Why am I mixing both terms since you probably never thought of mixing those and Why am I making a talk out of this when there's so much good out in there in the world with there's not The deal is that the current state of delivering software Regarding is with regard sorry to security implications is something like this so we are the little house in the prairie and we're about to be engulfed by a power plastic cloud and Where we want to go is this and some people think all is good with the world and we're already there But unfortunately, we're not and as you just saw we're there so In order to understand the point of all this we're gonna have to first look at the Different roles involving getting an application from the developer to the end user and that way we'll be able to understand what the problem is Also, note that some of the roles that I'm going to talk about might not be that evident at first so We first have the developer and that will be I think most of you guys The developer is mainly focused on coding and drinking coffee. I think He's the one who knows the most about the application the architecture etc. And also the correct way to install the application On the other side of the spectrum. We have the end user The end user just wants the application to work That is assuming we have a way of knowing what the user wants Whoever tells you that he's blatantly lying. We don't have any clue what the user wants The user of course knows next to nothing about the application. He's just a user He doesn't know how to install it. He's basically as helpless as this cute little baby cute, but helpless Now for the more obscure roles we have first the system administrator Whether you're running a hosted service for your application or you want to deliver the application to your users or just Keep the infrastructure up and running you're gonna need someone who has systems knowledge and it's gonna be this guy Maybe not this guy in particular, but someone who looks like that. So what does the system administrator want? As a system administrator, I can answer this we want a free weekend Free weekend no work and maybe a beer will be nice so for that to happen the application has to be In to put it in simple terms easily installable and it must not crash or it must survive Catastrophe an attack or whatever If it doesn't we have and a happy Cicely and Last but not least there's the security administrator or infosec for short I know I actually know some of these guys those big security conference in Argentina every year And all they want is to see the world burn. That's really all they want Actually, not all companies might have these guys in the case of some companies the information security role might be mixed in with other roles like networking and whatnot and To all those companies out there without a guy like this. I'm gonna tell you this you're doomed So to put things into perspective and better understand how do all the roles work together Let's look at it like this. Everything starts with a developer Deciding hey, let's release this. Let's make this available for users. He then goes over to the system administrator So the application can be deployed or built or wherever and eventually the application is delivered to the end user So the system administrator is basically our bridge here Then the infosec guys comes along and he seems to be lashing out at everybody But in fact what he's actually doing involves interacting with all three roles He gives security guidelines to the developers in regards to architecture and good coding practices he gives architectural guidelines as well as good operations practice to the sysadmins and Well, these two are the proactive ones So he wants to prevent bad stuff from happening, but there's also a reactive one In which he tries to keep the user at bay whenever the user decides to become a hacker and attack the application So in short the system administrator is the guy who will help you Package the application and delivery to your users Whereas the infosec administrator is the person making sure things run as smoothly as possible and Nobody dying in the process So these two guys Maybe for some of you it's the one of the first times you've heard about these guys Especially the infosec, but you really want to be friends with these guys it being friends will make your lives much much easier So now we're going to begin to see The problems that are inherent in this world and the first one being that there's a gap between the developer and the user Because the developer only wants to code and the user doesn't know how to install an application How how does it run and what not? So this gap could be divided in two axes the first axis is the ease of installation And this is basically essentially taken care of by the system administrator Whether it be keeping the hosted up and running or taking care to package the application So it can be installed by the users the system administrator is the guy doing this and the second axis is the Security implications. So this is overseen by the infosec guy. Basically the users want their application and Their data that they're putting into the application to be 100% secure, which we all know it will never be And the developer might not be so focused on the security side of things. So here's when the security administrator comes into the picture We're not gonna see much more about the system administrator The system administrator finally had a free weekend and is now enjoying his beer. So we're happy for him So basically at this point infosec guys comes in barges into the room kicks over everyone's desk and sets the office on fire So we have these two axes that I just show you and the caveat here is that both axes are in fact Related because the more easy an application is to install Typically the less security application is and vice versa The second problem we have and this is not inherently related to security but more related In an indirect way when we look at it in terms of packaging is the dependencies problem I'm sure everybody in this room at some point had to deal with dependencies So basically you have to ensure that You have all the dependencies every time you want to release your application But not only do you have to make sure that you have all the right dependencies you also have to make sure that you got the right versions of the dependencies and Eventually this problem will get very very very large and it's what's collectively known as dependency help So solving the dependencies problem is a thing and it's a thing that we must take care of In order to do that we need to choose a packaging method Now there are many ways to package an application and we're going to be looking at them right now So the question will be essentially which one is the right one which ones do we choose? So regarding the different approaches that we can take in packaging an application. This is a part that I've been dealing with for a Lot of years Let's go over one by one The first one we have is the typical get it from the internet and run it just in one line of bash or whatever So This is what comes to my mind whenever I find an application that I want to try out and how to install it something like this So I'm blindly running code from the internet in 2017. Come on You mean I shouldn't be looking at your code. I should just trust you and Finally, yeah, it's it's only one command It's all nice impact and all you had to do was build one big blob of binary shenanigans But what is the cost and I work in technical support and with the amount of security vulnerabilities? I see every day this obscure approach. I seriously cannot fathom how many things can go wrong with this one So let's forget about this for a second at least So next up we have a github or git or any kind of version control where you Offer the version control version to to your users. So show hands again everybody How many of you install software from github or git or any other? repose Okay, that's good now How many of you actually read the code you're installing? Okay, okay, that's good actually. I thought no hands were going to be raised at this point So yeah, we're getting there. I mean we need this right? We need a place to host our code when we hopefully if you're pursuing an open source approach We're gonna need it. Otherwise, nobody's gonna respect us At least everyone now gets to see the code. But as you saw not everyone does Ideally when we have this approach we can already start documenting the dependencies of our project So that's kind of solves one of the problems The person on the other end download in your project should know what they're getting and what should they need to have in order for the application to run properly Remember that not everyone reads the code. So this might not always be true Do you know that there's nothing and nobody actually enforcing this? There's there's no one forcing us to Put up a dependency document or anything of that sort You can basically upload anything you want to github because you're a strong independent human being But that's not enough. You need to be a responsible adult, too And last thing to note is that experience users are going to look for this kind of approach because like system administrators Or infos I guys want to look at the code want to know what the code has inside of it So yeah, we should always have this available, but it should not be the main way of packaging the application Now a rather new trend in the software wall in the last couple of years is to use Docker or background or whatever image format Is the new hit in town? We have to admit this is quite easy to use and easy to install and it's almost guaranteed to work It also solves our dependencies problem because we can just bundle them all inside the image That is in fact one of the main reasons for using this approach Forget dependencies, but from the user's perspective, do we actually know what's inside the container? I mean, I know we're talking about an open format We can disassemble it and look inside it, but we're not actually Encourage to do it so it has a tendency to the dog side and Another problem with this approach is that somebody could be enticed to start bundling libraries and dependencies into it So that it makes it makes it way easier for users to use and then at some point you have basically a big blob of Things and also when you have that many dependencies You have to rebuild the image not only when you update the application But also when the dependencies are updated for example in the case of a security vulnerability So basically this is akin to inviting the slender man into your home See what the dependency problem can take us We just wanted to make the have make the application available for users just a simple application And now we have this mythological fearless creature in our living room So the next approach is people or pi pi which I'm assuming many here are familiar with For sure we are solving the dependencies problem with people or pi pi and not only that we're doing it right We're gathering the dependencies from the same place And it's with solutions like this where you really need to start thinking about Community oversight in scientific circles. This would be akin to peer review You want somebody else to look at your application to find bugs and arrows and whatnot Now there is some community oversight When you're installing package from pay when you're uploading when you're during you upload after the upload Some not all and you're gonna see why in a minute There's also some digital signature support not in the way that we might know like GBG signatures It's rather via the tough framework the upper update upgrade framework Unfortunately the framework is outside the scope of this dog So I really encourage you to go ahead and take a read. It's really good And at least now someone is thinking about security But one mean thing that can happen and has happened with peep or pi pi Is what this guy and hope that I pronounce the name right Nikolai's Saha once did he did a study where in he submitted typos quality package names to pi pi So for example instead of submitting simple Jason He submitted Jason in the hopes that some user might install it and he watched how many people install those fake packages That's linked to the to the study and Blemie there were lots and lots and lots of installations of these fake packages And you know, what's the scariest part in all of this and the reason behind Nikolai study is that peep runs arbitrary code from the internet I'm not saying peep is not a solution I'm just saying we should keep that in mind whenever you download a package There's arbitrary code being run and if you're not downloading the correct package. It's something bad. It's bound to happen And so the last method that we're going to cover is distro specific package management This is saying that Linux distributions. It can be rpm Deb file the works and this is the part that requires the most work to set it up But the thing here is that not necessarily you guys the developers have to do this work Because the the Linux distros have a teams of people called package maintainers and the package maintainers What they what they do is basically take your application once your application has become a little bit famous and Somebody wants to use it and they'll package it for you. So they're kind of packages as a service And the approach is relatively easy for and users to install because most distributions right now have a graphical update manager in addition to the console-based interface and Most importantly, this is the approach that has the most potential in regards to security distros can sign drone packages when they're in their software channels and Another good thing is that once you make it into a community-based distro Software channel. It is the community that is tasked with improving the application and sending you bug reports All this for the special low price of free So it would take a lot of time to go over this whole process in detail But for those not familiar with it rest assured it is one of the most enriching things that can happen to you as a software developer So we've seen a fair share now with the different Packaging approaches in terms of security and what they can provide. So the question now is where do we go from here now? Remember we want to get from here To here and I'm not saying we're going to make it just yet Not tomorrow not the next week, but at least we have to start heading the right way So we first have to at least start trying to solve the gap problem. So how do we tackle this one? It's very simple. We just communicate. It is not a definitive solution, but it is the mandatory first step and one of the gap problems axis was ease of installation and a good way to Reduce the gap on this axis is to have a good communication channel between developers and system administrators And this is what the DevOps process is supposed to address. Although in my opinion, we still have a way to go But basically the developer side should inform the CIS admin side about their application architecture and help them understand the design process and The system administrators on the other hand should inform the developers about the best practices regarding Operations and if you do this right we can get to a better place in terms of packaging itself and eventually Make it better for the end users too Now for the security axis of the gap. The answer again is communication And this time between developers and infosec administrators an example of what can happen when Communications in this regard fails is the shell shop vulnerability. I don't know if you guys are familiar with it A couple of years ago. We had this big vulnerability in bash, which is the main interpreter in any Linux this run unique system So as the first patch was released people started scrutinized by us and the patches more and more and more and What happened is that on top of the original vulnerability? We've had not one not to but rather five different vulnerabilities all in the span of a few days And this happened in one of the most used pieces of software in the open source world So it can happen to anyone, but if you communicate Good and fast you can prevent this As for the dependencies problem a good starting point is to stop making it that easy for the users don't Expect the users to always be dumb give them some headway and let them you know Tinker a little bit and instead of choosing seemingly easy ways of delivering the application Prioritize the cleanest and the most open ones They are out there and a good example of this are package managers whether it be a language specific package managers like peep or always a specific package managers like young or apt or whatever and In fact, there are other alternatives that I've sprung up in the past year or two Like a flat pack or snap I don't have enough time in the talk to go over these ones But they're relatively new and they were considering because the ecosystem appears to be wanted to take that path Finally, if you really want an adventure, you can always go the distro way now as you can probably tell my now I am a little bit biased I do believe you have a good chunk of experience with the other methods So the aim of this talk was to point out all the the evidence and not so evident problems and Providing other way and before going to this way It is important for you to understand the difference between a community back to throw and an enterprise back to throw So in a community back to throw the work is done mainly by volunteers people who aren't being paid for that And they're just doing that because they love it Whereas in an enterprise back to throw, of course, there's a company behind it and there's tons of money behind it so Sorry once you've gained enough traction in a community based distro An enterprise teacher will pick you up and then you have more people looking at your code And I'm making this distinction because there's a circle between community and enterprise So companies often offer volunteers to community distros and then take the community improvements Pack into the products So we have befriend the community and you befriend them all but above all You should really befriend the package maintainers these are the people in charge of getting your software to the users and If you communicate with them early and often and if you let them know when a release is going to happen, maybe a Few days in advance, especially if we're talking about a security release that'll make things so much better And the idea is that you make this update cycle easy for them and I mean this make it easy for them I'm looking at you kid lap All right That's all so now we have time for some questions. I believe Thank you. Yes. Thank you for showing us the road to the fairytale World of unicorns. We really expected that We're really short on time. So I'm afraid we won't be able to have questions and answers Okay, if you have any questions, you can find me around the conference the next few days Just look for the guy in the black hat. That will be me So sorry it took more time that I that I intended. Thank you again