 This is the last session in the tower for the day. This is Weeding Out Security Bugs in Debian by Javier Fernandez Sanguino Pena Yes, I Either way I also need to announce that we're going to try and shorten this both by a little bit so that people have time to Get ready and catch the bus for the formal dinner that leaves at seven. Okay, right? I'll just hand over. Thank you Thank you all So Well first I'll present myself. The name is over there. She almost got it, right? What do I do for a living? Yes, so you can know my bias and what I can I do right and what can I do wrong? I work in ID security stuff So that's why I do a lot of side security security packages in Debian Okay, like Bastille or Snor or Nessus or whatever so I do penetration tests for large companies and What else and development of security gateways and stuff and the main goal of the presentation is okay So provide information for all devian developers Those that are here those that are way back home and those that here these talk afterwards on how to fix or really tie on to the Ways that devian fix the security box both in stable and stable and even testing Okay, so give you information you can probably think about more afterwards about how where where are we right now? Where are we going to and I'll go through all these different topics Starting so how is the security right now of a devian operating system? Okay taking a look at Based on the advisors with publish and what are the security packs? We usually have how are they handled? How many do we have per release? Okay? And how does it work? Where is that going to? Describe also the work of the different teams of the security team that works and handle security backs in stable The security testing team that handles security backs that go through unstable and testing for the next release And also the security audit team, which is probably the most recent of those and what works It really does. I'm a member of the security audit team So I will probably talk about some of the lessons we've learned when handling Security advisories and backs for developers that might be of use to you so we don't encounter the same problems again and again when handling security issues and I'll probably if we have time and that's probably the workshop of the most Worship side of the of the talk would be showing how some Insecure code and showing showing also some tools how to test that code That's some of the tools with the audit team uses, okay? And finally will probably and that's not for me to say we'll talk about how we can improve that in the long term And maybe some recommendations that we all have to do in order to improve the current status. So Well Yeah So what's the impact of the box that end up being security box in the operating system? So I guess all of you are aware of this But yes in order to repeat that and certain and focus the talk When we find a security bug in the well, we find or somebody else's find security back in the operating system It means that at that point in time Even though we might not know it yet all our users of the stable release are at risk. So they might get Cracked and all their servers might be compromised. It beat our remote Flow or a local flow that means that they at that point in time are a risk It means that both a degen developers on the security team Sometimes more the security team than the degen developers are really stressed out to provide a patch to our users and send that away So we can really fix the issue that it has been found When that is done, I mean sir mean we have a patch available Since we do back ports of the patches that really sometimes is maybe a three line patch But sometimes it's not and sometimes really difficult to to do because most upstream just Produce a new version with a security bug included and we have to all the security team has to remove the bug Has to remove all the non security fixes and just go for security fixes So when that is done and that is published to our security Update servers that means that at that point in time all those users that will be downloading things from security Dot devian dot org are going to stress all the double now. It's a mirror network now It's actually I think three of them right So and they're going to stress those and they're going to stress the bandwidth of those and that happened And that's the reason why we now have more than one security server, so we used to have one and now we have three and What happened? I think that started being changed one done security patch for X was published last year So the both the security mirror on bandwidth. Yes class because of downloading all the full X packages updated for with the with the security fix Even though we might produce a patch some users that are not running You know updates regularly and they are not Even they might not have for some whatever reason security dot degen dot org on their APT Sources list they might get compromised even when the patch is available. So that's that's a no-brainer to if you don't have the past It gets because compromised and in the end Well, and that actually happened and that we actually published and if you see some of the news Sometimes having published our related security time to fix on whether we do fix the stuff or not in the end I mean, there's a lot of bashing going on whether we do provide a good security support to our users running stable Or we do not because the data ends up affecting devian images as an operating system and nurse our workers project, okay? So as we will see the security backs tend to increase which is really so the resources all these resources that have been stressed out Get stressed out every time more and more Okay, so first comments of security backs and these are no-brainers But just also to for those that are not that much in the security arena Is that obviously all software has facts? Okay, we have to accept that as a fact of life. So we have to live with that And also Some of those bugs are going to be security backs in the end. Okay There will be very severity of security past is not the same to have a local Would compromise then to have a local, you know games compromise or whatever in the system Then have a remote buffer or flow in an essential package that is being installed. Maybe syslog or whatever So it's not the same. We're going to have different security box of this is a variety Okay, that's also very important when we go to the data face of the lies down lies and in statistics face Then we're going to go through But so it's not all the same and even so the kind of bugs that are going to appear in the distribution We'll vary by time. So we're not seeing right now But we're not publishing fixes for the same kind of box that we were fixing We were publishing fixes for three years ago That's sometimes because there's a new kind of bug that, you know, it's a buzzword and everybody's going looking for them Or maybe because the focus has he from some kind of box to another kind of box Or maybe because because we're really fixed most types of bugs of some other kind. Okay If you go, I don't know how many of you are aware of the severity funding. Well, he that company does Publishes will not actually publish this has a Product that it's a source code out in deep product. Okay, and they got paid Along with Simon take I think was this year. They got funding from the US government the Department of Homeland Security or whatever to do a source code audit on Free software projects. Okay, so they got funding to take all the source and audit that and publish the Vulnerabilities they found and give them back to the developers. So they actually audit with their tool that Samba Pasha Be as the open as H and some of the projects you have all the stuff about the website, but the the end of the One of the things that might be surprising is that they end up finding on average on mature Products and we're not shipping all my all our packages are not really mature products They got on average that those numbers of of security box He some some of those that they found were not really security backs that could be exploited But in the end that's the number of of bugs that it will eventually found and they got patched by by those projects in recently Okay, so that's 0.3 security box per 100,000 lines of code in those projects. Okay, so we're that's Probably the lower level we can see So that's a generic thinking what about We in dead end. Well, as you all know the size of the distribution keeps growing every year So we got more and more packages and that's not stabilized So we get going we keep growing and growing and growing and there's unlike other projects I may be open BSD that has a stable core and that's you know, doesn't very that much with time or Fedora Which they don't put that many new packages inside We keep growing and growing and growing and we're supposed to do security support for all the stuff that it's Eventually released. So if we release edge in December all the packages that are gonna be released with it should have security support And those are way more than the ones we released with Woody or Sarge and Sarge are way more than the ones We released with Woody. Okay, so I Gave a talk about Security three, I think it was the Defcon three right Norway. Well, yeah, Oslo It was a very fast talk. I didn't I had to take the plane so it wasn't my best talk It was not focused on fixing security bugs it was focused on security in general But some of the issues that I pointed already on this on the security side Especially the number of bucks we were getting every year. It keeps being the same. Okay So, yes, you have all I didn't mention this. It wasn't the first light all the data on all the Code I'm gonna use it. Maybe even tools not actually tools but code is that but Homer? Okay, so Homer Well Homer dot Mexico dot com dot org slash chair slash JFS. Okay, you have all the stuff there. So you can know that You also have the for the people that are outside here not on the Defcon people that they can go or slash whatever JFS Defcon security It's all the stuff that I'm gonna use actually. There's more stuff of people that they've been doing. Okay, so the data file there that's It has some of the data I've been using to use some of the information. I'm gonna show you here So let's see some lies So that's Total advisors pose for they can okay. That's that's the number if you see for potato we publish 197 they can security advisories and that number of Covering that number of vulnerabilities. So we not always Some advisories cover usually and that's when it's very typical with kernel vulnerabilities when there's a backlog of them an advisory gets published as covers more than one vulnerability. So That's the number of advisories and in parentheses. That's the number of vulnerabilities that were fixed on the advisories. Okay Those are probably lies so you can go and look them up yourself. You want to So if you look at the potato as compared to charge We already have that many DCH publish and we go way beyond those and we have That those numbers from you have them on the paper of millions of lines of code are based on the work of some guys on Spanish University It's actually The University of Ray Juan Carlos. So they do an analysis of software projects So they got that numbers of a million lines of code prayer Distribution prayer release we we've made so if you see there we We actually have a lot of of millions of lines of code and also a lot of Advisors being published. Okay, and that keeps going and growing actually that number the final number of mine of DCA's depends actually on how many years of maintenance we've done for that release. Okay, you probably see it clearly Here so that's the line You see here. That's the line for potato. So that's when we end up not providing any more advisories for that because we end up Finishing it wasn't a security support anymore So it remains stable that doesn't mean that it they didn't have any more bucks He just meant that we didn't fix it and them anymore And you see that's the line for woody, which is going in support You can correct me on this. I think it's June or July this year We're not going to provide a security advisories Well, that was June July this year or edge whatever came first So we'll probably be June July this year right So and that's the number of DCA's if you take an account search that it's been That's the count accumulated account of a vibratory's as time goes by you see more or less the same trend as for us for Woody which actually makes a lot of sense. We actually have probably as many backs as Woody and maybe even more We obviously some of these advisories that got fixed here were present both in charge and woody If you you have that on the data file There's an an open office a charter and you can look those numbers up yourself one of the interesting things is that The packages being affected with DCA's on a charge, you know, if you take them and count them aren't are Different than the ones that we they got fixed for woody actually most of the DCA's were packages being in section net and section web Which actually those numbers for the web packages are not as large In woody are not even in potato. That's that's one of the things I was talking previously that the focus of website web oriental properties that's cross-site description as called injection and File system file inclusion or whatever Are getting more attention here, so we have we even have more packages here than related to To web applications that we had in back in in potato or in woody We have lots lots of PHP applications that get security vulnerabilities published frequently, but We have actually have more DCA's published now for web applications that we had back in in woody Okay So those that's the first fancy graph and I think I have another one so I'm gonna talk about teams and then Try to show you some stuff That's so you get an over impression feel free to you have any questions up to this point Any comments here? No, I'm gonna go through This is essential because I want you guys to make sure that you know what the security teams do and so I'm gonna go through them Okay, so we have three security teams right now, which the one that's been running for longer That's this table security team, which is how to call the security team Which is the one handling security box, they are the ones that are Monitoring vendor slash sec that is an closed main list for security box among all They are actually linear distributions and other free software distributions. So includes VC BSDs Red Hat and Susie Mandrake, whatever We have the security testing team that's handle security boxing testing and they do that with public information So they don't use closed information for that I'll go that into a later Tomorrow till later and that is the security audit team whose goal is to find security box in the distribution actually both in stable and in unstable so How do these things work, okay? I'm gonna show you some more down nice before talking about teams For all actually the slides are over there also at the place I pointed to just in case you don't want to follow here the screen so more than lies of what security backs we usually get fixed in advisories Are these? Well as expected most of them are dealing with user input. So that's buffer road flows and Improper that a hundred whatever the that is But we also have design issues are going to more detail of what some maintainers do that are not very They don't think much of a security when they do the sign of their packages We have boundary condition access validation, but well the most of them are actually remotely exploitable so if you sum up Buffer will close with the most political you have all those DCA's that meant a remote local remote code exploits When they were published, okay, so those probably meant those with which have more severe impact to us So that's the data from 1998 to 2006 if you if you skip that and you go to that I guess for just one year You probably see the focus on some specific vulnerabilities and not another one But that's over. Okay, so And that's the fancy grab another one I made Because not all securities are Advisories are equal. We don't actually publish a risk information on the advisories Okay, so when we publish an advisory either use it doesn't actually know If he should apply that really fast or not or if he has the software installed so that's based on CVSS, which is common vulnerabilities scoring system Which is something that Johnny made by Cisco another on a feast and other guys Which actually had a scoring for our vulnerabilities. So that's That's what's the distribution of the our bridge are so if we're all thinking that the problem is we usually have in devian Are you know like local gain stuff that I mean for things that are not really that common in Deviance so then don't get that much is all that's not really that too So you here you see here the biggest of the pie and that's the actually the average Those are with scores seven that are actually remotely exploitable probabilities not necessarily Remote root, which would be you know the ten nine ten over there, but are actually really severe Okay, so that's actually the types of only see we usually have in devian Which means that we probably missing a lot a lot of other properties that are not getting fixed as much So that's the last fancy graph. So that would be probably the end of the data So before I go into discussing the teams, let's do some hands-on and So you can see what I'm talking about and actually that's that's a package that I made myself It's called hello insecure based on the hello package. That actually starts up as server them and Well, actually I wrote a lot of crappy code Well, actually I reuse some code and then make it crappy. Okay so If you go and take it take it, I'll show you guys and these are actually stuff that are is done by some Package maintenance that gets up needed to be fixed that as a security box. So I'll sit down and show you Okay, feel like a rock star with this here So what do we have there at the hell example I want you guys to look at so let's finish this first Go over here. Is that big enough? Yeah, okay, so Hello You all guys know the hello packages, which is an example of how to make a package So I made the hello insecure package, which is an example of how not to make a package and it's actually based I'm sorry, but I'm not used the US keyboard. Actually that's these laptops borrowed So it's actually the same stuff or minus some other things actually you guys have here these These are the differences between the original one and The one I made which hopefully won't see the light out there except for Google but So priority extra and let's not blame Santiago for this one. Okay Yep. Oh Sorry, maybe my can you hear me now? You hear me now? Yeah. Okay. Sorry So I say we're not gonna blame Santiago for that one. So I changed that but in any case this package provides an in a script So I actually provides a damn them for that Sorry for my means typing, okay And in a script, which is your average in a script. Okay, it also changes the post in post-inst Actually also removes the in a script. So it's nice there and here's what it does you sell a new Those in the scripts there Which is actually kind of average It adds it actually adds some new option to hello, which is hello minus s to run as a server I should tell you guys to be careful not install these packets on your systems unless you want to remote Compromise root compromise Did I say that first? No Hopefully not nobody installed it, right? It's something there's no internet. Oh crap Oh crap, I depended on that Well, good, you can you cannot even get to Homer. No You can Well, I cannot so you guys stop leeching Because I want to do some tests outside. Okay. I'll try them inside so Well, you fall from here Maybe didn't have it Um So it hasn't it adds a new argument to the main sources with the server and then let's see It adds all this crap. Well, it's not actually crap. It's based on Micro-init that Which is not mine. I borrowed it and they actually code to start a listening diamond port a hand 1025 Which is somewhere over there port number Maybe here It starts it starts up log even that you see it very nice luck and And it starts up Well, it's forks and it opens up the log and when you say something it will answer to you So we can start that up if you want to If you don't have it, we don't have internet that probably means that you I cannot get crap I wouldn't want this These laptop to get cracked into so I can probably launch it from here. I want it. I wanted to launch that at home So I couldn't get crack there, but I'll do it here. So you guys don't be too nasty Is that compound? Is that hello? Okay Okay, I'll probably have to distract that one. Wow. Oh crap. Sorry. I didn't know why I do that So we have hello that we can just start up and That means we can connect from here and Say hi That's a very nice Diamond to have right so actually we can do these two It's probably not that nice And we can probably do these two Think that buffers big enough Okay so These damn money if you saw that program that you see the children died there both children died with Six cementation fault is it here? Okay, that that package if you saw it it will actually install A diamond that has both the form on string overflow and a buffer will flow And it will study that with root privileges and then you're screwed if you start this guy okay, so that actually happens and One okay, you cannot blame the package maintainer for some of the stuff, but you can't blame him for some other stuff So let's take a look about Probably this thing so here the maintainer. So that's me Decided that hell I can No, yeah, I can just start up this guy Roots user so that means I don't have to fussy around with Whatever logs or it does he will work fine or whatever so that also means that I'm hoping about very big hole it actually starts if it's at the back then when it starts on all interfaces, so probably shouldn't do that either and You didn't see that probably but another and it's not that uncommon thing is to have stuff like that So when you saw the package it will create a default thing So it will not start a package if you just install it, but for whatever reason it will Change that to yes on one new installation And it actually will dump the changes to the temp file before moving them to the config file that actually is really Crap, but you see that on maintainer script So that means not only that if you saw did you get a root remote hole, but that you have a local root DOS because of following sim link attacks, okay And these are examples I probably you can see in some packages So those are depending on the maintainer some others are not but probably the maintainer shouldn't do this either which is enabling debug of The binary which actually is the one That makes this guy right to a lock Okay, so Even though this is crappy code Maybe the maintainers wouldn't have package it now There's some things that the maintainer could have done to avoid that being really crappy packaging So he could have have that package be running as a non-root user He could have written a proper posting script and he could probably have removed at the backcode from there Okay, at least change that to something useful and not TMP doing an F open which is also crappy, you know there So I Actually when when talking with some guys they say that even though the maintain packages They don't have to learn or see code or even understand that But they probably need that to in order to find security box. So if you guys want to those You want guys to spot some bugs, I don't know if anybody's how many people actually have Get to download these files from Homer Suggest Joey Two guys Nicholas Mario Okay Also, you cannot spot that many bugs if I go too fast But actually the bugs in the server code are here highlighted So if you guys want to look at them later, you have them nicely commented out comment it in actually So there's a few of them. Okay So here's an example of of insecure package that is some of It decision of the people write of the person writing the software itself and some is other decision of the maintainer So that's actually very common in David In some cases and I will actually show you some later on Okay So I promise to talk about the security teams on our wheel. So There Okay, any comments up to here. Yeah, I have a question from ISE. Yeah Benjamin asks Benjamin Seidenberg asks if will the security team? Will the security team do an audit of a package by request? Not the security team, but probably security audit team. Yes. I will see the teams later on Actually is one of the things I want to bring to you guys I mean, there's a few people out there and probably let's see if we can get more Into the team that can probably review stuff before it goes into either unstable or whatever Okay, so actually our goal is to fix that before getting to the release Joey there if anyone would like to download this locally. I have a copy on my laptop which is Dragon.local if you're using mdns or it has an IP address of dot one five two and it's in the Tilda Joey directory so in inside Temp so Dot one five two slash Tilda Joey slash TMP You can also FTP to Homer directly by number. It's a hundred dot one and that works as well So thank you Joey for that. I couldn't I didn't set that up I actually have a USB stick with all the stuff in it if anybody's interested we can probably pass that around if you want to Does anybody want it? Yeah, okay. Hope I don't have much confidential information there Actually don't use it that for company stuff, but So ask Haki if you want to just your hand up and I'll throw it again over there if you guys want it It's actually both What is it's in the USB stick and also what's up at people dot devian.org is actually all the talk source Gov all the talk the data these source go and many other stuff so you can play with so While you're copying that I'll go through the security team different security teams we have right now at devian Well, the main security team is the devian security team surprisingly and that is made up of It fluctuates is are usually four to six members Martin's rules is the one that it's leading it up now It actually has different kinds of members. He has the you know the members that can actually publish Security fixes on the build diamond network that you know actually gets those into the security Server and there's actually members that are calls the secretaries that are the ones that do the hard work and do the patches And you know follow all the stuff so that no one they they get called that But there's some full members that's how they call them and there's some secretaries for that So Steve came who is also in the security audit team is is actually a Secretary right now and there's some other guys right now that have been doing actually that this has changed because there was There wasn't enough people so some people got Included there recently so in order to be able to pick up the backlog of security fixes that need to fix in You okay So what do these guys do these guys are the main point of contact for see a devian for both bender sec Which is the private mailing list used for security disclosure, and it's I guess that's Noneful disclosure. Okay, so these guys get the heads up when Security bug is fine on open SSH, and it's not yet public Either because somebody found it doing an audit or either because somebody with the honey net or with the honey pot system got first Version of a zero-day code exploiting it. Okay, so this is usually discussed on vendor sec And they also are the primary content for cert Which are the guys that handle security issues, you know kind of those that affect all the whole of the internet So an open SSH back. We're probably had be handled by search. We usually which right now is actually called us Okay, so it's not the cert any longer, but They also tend Try to review and that's quite a cumbersome task They try to review a backtrack and all the security databases out there to see if there's a security Issue published there that it affects us so they if they find that then they contact the maintainer and try to get the security fix done, okay So they make the patches usually with help from the maintainer and that's not always true So if something happens something doesn't and they publish that publish to that to the build network that handles this security Build diamonds for all the all the architectures we support They on occasions and that is not that much often now They follow up on compromises to deviant systems in order to see if that compromise was because of a zero-day Vulnerability or it wasn't actually I think they don't do that much anymore because most of the compromises at least Those that are being discussed on deviant security mail is are actually people that have servers with default passwords So they end up being a crack because of that actually I have a Didn't mention these have a honey pot system out there We just run in search with no security updates and it's been running since I think it's last year May was it made? It's part of the honey pot system deployed by the honey net ions So if that's honey made honey net org and that system has not been Compromise at all for one year and it doesn't have any security patches But what you do see on on the outside on the wild wild internet? Yeah, we'll go there just a minute. I'll finish let you make the question What you actually see most of what I've we've been seeing on the honey net alliance It's actually ssh brute force scanners more than people trying Be zero-day or non-zero day for me this on your Linux system So they these guys go more for the windows systems, but still So what was the question there? Can you guys hang over the micro? I? Was interesting it's working out. I was interested in knowing about this honey pot server and we charge without security updates What kind of network services but it's running because that's the most important thing I think yeah, okay? So never service it's running. It's xim As an mta, so that has security bugs with advisories it has It actually has courier both emap server and pop 3 server It has a DNS server which has also security bugs needing fixing and I think that's about it Which are not very popular services, but probably all of them exploitable. Okay, so he's it's not a default install because Luckily well not luckily thanks to people doing the devian install work Thus the full devian install doesn't install as much Servers as it used to do back in days of potato. Okay, so it's not a default install It actually has accounts with maybe about I think it's around 300 different accounts that work Yeah, they've been generally to randomly with names of Spanish people actually the names are actually very funny So I have a random number random name generator that ends up with having 300 accounts with non-default Username password, so not put that easy, but it actually has four or five servers Services that could be remotely deployed and they happen to be not used at all. It actually replicates SMTP you know mail server setup that is very common in the internet, so it's mail server pop server emap server and DNS So That's the work of the security team. So This is the work of this devian security testing team these guys Some of them in the room follow up the issues that are public and in order to see what? Issues are really fixed or pending in the in the testing release. Okay, so they work with public information and that's Information from sources that declare a given version burnable or even the uploads of Pax maintainers that say they have fixed a security bug on some place. So they also look up bugs labeled tacked actually security on the BTS and see if they are fixed on different on Unstable and see if they are fixed on edge right now Okay, so the main goal of these team is to and they started working Correct me if I'm wrong, but I think they started working actually for Charge to make sure that the security box We wouldn't have to have published a lot of advisories as soon as charge was released because we had a backlog of security Bugs there so they actually help the security team during the freeze to make sure that all the backs security backs were fixed on On on charge. Okay. Well while they were being fixed on on seat They actually the information of them. I didn't put a reference there, but they others If you Google for that and or you've read the recent mails Debbie and the bell announced you will see that they have a Public page with all the bugs that affect Testing and affects it and there's actually a tool right now that is called dip is can That runs through that list and checks your local packages and tells you these and these and these are Burnable because you haven't hasn't been updated yet. Okay? So they review that they actually at some point in time if they need to be It's not that common, but because the main goal is to have those security backs going through seed into edge Right now, but they can and there's a build infrastructure for that right now They can provide devian testing security advisories Along with patches just for edge. Okay, so there's separate Security support for edge that is now gone official You probably see in the mail to devian develop announce What's devian develop announce or devian announce? Devian I heard devian woman, but I Know Okay, so That's your support started September last year and it's now official kind of so you can They're integrated with all the security mirror network. Okay? So you don't have to go to another update our archive to download the security updates for testing Yeah, there's a question over there Not really a question, but just some more information about the testing security team did You mentioned that the Sid and edge are tracked and by the testing security, but we also have Quite a bit of Sarge on their tract and one of the things that we end up doing most of the time is tracking Issues and we need more people to do that. So if you want to work Testing security, it's fairly easy to do and everybody's welcome. Yeah, actually, they have an SDM repo, right? Yeah, it has a database of the CDN names with security stuff And actually the only work that has to be done is add new CDN names that affect us, right and label them Maybe high-risk medium-risk or whatever so they can get tracked, right? Yeah, mostly it's trying to identify whether a package or whether a known security issue is in a Debian package and if not if so file a bug if not Ignore it. It's fairly easy work actually, but it's a quite a bit of menial work at times. So Yeah, and one of the things that really helps them track security boxes when maintainers fix a security bug Say so in the change log and actually put the CV a CV name on it so they can grab for it Actually, if you guys don't know what CV is you can go to CV dot meter dot arc and it's actually a Project sponsor funded by the US to provide names Distinct names to vulnerabilities. So you can it's actually they call themselves a dictionary and not a database So they actually give a name for that. We actually Not only for the testing for a stable for those advisors that have been published. There's a upon the website There's a page that lists all the CV CV names for all the devian advisors So you can go to a page and see if a CV name when has it been fixed so you can you can go to World a web dot devian dot arc slash Security and you'll see a link there to a cross reference table that actually shows you that force table at least They've been security advisors and there's other one of the one We probably have to update the information on the website to cover security at testing team now. It died. It is official Or more official, I would say So and now the security audit team it's actually made up of five or four people so you can guess that it's not enough people to audit the whole archive Okay, and it has even even Mom power issues and even more issues. So some members actually started audience auditing stuff back in 2003 so but the Group itself didn't merge until 2004 and start them at the main list and started working, you know together to do audit stuff So actually one of the things we do we've done is Each member of the team is focused on reviewing a certain type of vulnerability in the packages so it tries to find all the instances of that type of back so it's not like we take a Package we sometimes do an audit it fully but just look for some kind of bug in all the packages Okay, so like for some play focus at some point in time all on on Simlink tags on the temporary direct directories. Okay, so there were a lot of deviant and security advisors police related to that because of the other work Steve came focus a lot of on Many problems with said a gig games We actually could Have maybe have buffer all flows or about all the flows through environmental variables, so he audited a lot of them and Wolf I could not say the surname actually added a lot of former string attacks and and and some other buffer will flow kind of types on On many applications. So actually it's not it's not like we audit the full archive for all security box in a given package But we focus on a certain specific kind of vulnerability and we try to find that on all all the instances in the package that we find We actually end up finding more bugs which are not the ones we focus on So we might end up seeing crappy really crappy code and and ended up making some other bugs because of that so the As in end result of the last I don't know what that percentage is relates to but I think it's the last year Maybe since no maybe since it started of the devian security advisors that were published 81 were made up made up because it's devian security audit team found the bug informed the maintainer and devian stable Security team and that was got published as a DCA and actually there's a well That was those were bugs that were affecting the stable release Bugs that were not affecting the stable release got sent to you know the option maintainer and the maintainer and Got sent as bugs to the BTS and that's actually the count is a hundred twenty one or twenty two I don't I don't know if that number is correct. You all got you have all the listing of the bugs We found on the devian security advisors with that have been published because of that work up there, okay? We'd also have developed some tools To do some automatic source code review, which are very you know alpha status But kind of tried to use some other work, okay Any comments up to here? No Not an IRC either No, oh the answer for that. You didn't catch it. Okay. Can you repeat that on our scene yourself? Oh, yeah, okay, okay? Actually, I don't think I answered that actually, but yes now that we're talking about security teams The devian security audit team at times has been asked by maintenance to review some code And it has done so it we don't we didn't we haven't get many requests actually But we actually are very open to doing that if if somebody has to review either the design or or some code and see We can find some stuff on it before being published, okay? So Some tools we use that are actually in devian so there's packages for those and I want to present them So you guys can use that also, but you'll see and that's hands-on a Piece of work now that they are not as good as they probably could be if we got more work people working on them actually These tools are really not as good as the Karate tools So I'll mention them. We use rats and that's a rough auditing tool system Which is very really rough Okay, so if that's a C coded tool to review both pearl python C C plus plus and I think It means anything ghp. Okay. It actually works with a database that only it's only a listing of of Of actually library calls that are known to be insecure so it flags them and points them to you So it says don't use these don't use these because of these these these and it has a you know Kind of a risk metric associated with maybe using System call or get s call or open or f open call So it will try to highlight some place of the source code that are using Calls that are known to be wrong until you you maybe shouldn't be doing this as you might Think that ends up Generating a lot of false positives because maybe the guy that is using the system call is using it properly so You have to take a look at the code naturally to actually know if it's working or not Flow finder written by David wheeler is a Python tool that analysis. Yes He and C plus plus codes tries to go way beyond this maybe pattern matching on asteroids and tries to find the library calls that are known to be insecure and then Do an analysis and see if they are actually used properly or not. So like for example Formal string Problem in a c-slot call. It would not just look that and say, okay You're using c-slot here in this piece of code and it's known to be insecure But it will say you're using c-slot and you're not giving it the format parameters. So I know you're doing it wrong I know it for sure and Even it might look at the parameter and say, okay, you're not putting a limited size in the parameter So you're not doing it wrong completely But you should improve that to limit the size of the of the thing that gets sent to c-slot, okay? So it actually take a look at a little more in detail look of the code But it only covers those languages, which means that it doesn't cover all we have in the archive Then we we use p-scan which is just not general purpose only focus to find informer string overflows Then member of the audit team develop audit source, which is a pearl module that will actually run all the tools On the source code and we tell you the input of all of them Okay, it actually has a tool to color the code so you can show you in the code with a different color. Sorry What is the place that you have to look at for okay? Well, we all obviously use grep a lot For looking for source code. We also use a brute force search there that it's B Bfp tester that would actually try to look for buffer overflows on environmental variables and Actually on parameters, so try to find those kind of buffer flows in some programs by doing a lot of testing And some of the black box tools we might develop on To do some testing If you guys take a look at those tools, I think we have hands on now. Oh, sorry Okay Let's take a look at those tools with some really crappy code. I wrote a few days back Okay, so that's if you guys take take the multiple box Archive which is over there We'll see some really bad code or which is actually all there's not that density of bugs in packages Thankfully, but it actually has a lot. So let's take a look and let's use those tools and see what we get here So you get the feeling of what we do? You guys hear me, yeah, okay good, so I think I got this thing right here Close So in order to not make it easy I've removed the comments of the code So this is some code that will Use a log file. It actually the purpose of this code is to find try to find comment in the system So it's actually a replacement of which but Very very buggy. So it will actually use a log file. You see the location of the file there Hey, that's what it will try to run which is which It will initialize and running put into the log file who what which user use the tool and Then it will ask you for the command to find for using gds and It will execute something it will tell that to this log and it will tell you what you were finding for okay, so like So it's this code. I think I've provided in a compiled version All over the place, but if you guys want to compile it just run make and the make files You take care of that if you make it clean first So yes, so you take a look it will just do this. It's a really simple code, right? So If you run rats over this Which I should have I don't have a flow find later style here. So You guys can I have the output out here here. So we can see the output Here if you run rats, it will tell you these things. So we say, okay Code line number 23 you have a fixed local buffer. That might be an issue. You're using get em Don't trust that you using a spring dev. You maybe should use s and pretty as end print f using get s Why do you use that? Actually the compiler DCC will tell you not to use it s So if you check out the warnings you should see that are using a spring dev using syslog be careful with these I'm using system and be careful with that too Why should you be careful because maybe you can get stuff like this? Which is somebody trying to inject code with which is actually successful Okay, so a guy will just use the common to run LS That wouldn't be much of an issue if yes If this is a single common in the system, but it could be worse if it's a server Common and it could be even be worse if it's a set you did Said you it is that how you say it say you it said it Said you ID you ID. Thank you said you ID command in the system. Okay So that's an instant of a bug, but we have more bugs here if you guys want to check it out you can do things like this Okay, and We have a cementation folder because of the use of the environmental user variable. Okay, because there's a buffer over folder We can actually see that So you see there the guy trying to go over to AAA calls. Okay That's a typical buffer will flow Too many calls Okay, yes Actually, that's one of them But we can do worse also So what up? Oh, sorry That's just back coding. It's not exploitable. Okay, so what do we do this? Another cementation for why that actually is twice in the system in the code So that's because of the system of call that is the has a former swing overflow. Okay You guys see it. Is it here more or less? uh-huh But you'll guys will notice that if running rats Didn't bring up some other issues So like what about this? So what about oh, let's spin luck. I think it's that luck. Yes So you have luck there in a temporary directory Which is just Open it. Oh, no, please. This is a giveaway. It's no comments. No comments So that's actually because of this which is crappy and This which is also crappy So we'll get a pain to any file whatever that is. So that's vulnerable to a sim link attack So if I point a sim link to whatever thing, it will overwrite whatever it got there Even if that's a problem, even if it's a user none set you ID application because that means I can actually get The application to overwrite anything the user can write you so that might be, you know, pass profile You know his own data, whatever If you take a look at the rats output, you didn't say anything about temporary stuff So it means that one. So that That shows you that it they actually doesn't find all the bugs that you can have over there well Okay, if you guys We run floor finder you will get a different stuff which is this Which is more or less same But we'll see the difference now in a minute. Okay, so it's more or less it provides you, you know You're using syslog your system. You're using get amp using f open. That's not good. That's not good That's not good. It will actually provide you my metric based on the number of bucks you find You can actually these tools you can tell him tell them not to print stuff that might be, you know, a given risk metric so What bugs are actually there we can see just running this If we compare the files without comments with the file with comments, so that should show us All the seven security box it has over there At least the bugs I know I've run I've reading into it might have some more So how coded lock file location in a short temporary directory a buffer overflow here Seem like an attack here because he doesn't check the file Well, it doesn't check return value with season security but but still Get is with this you can get a stack overflow there You can get a buffer overflow here and you get a form string over folder and of course the common injection We saw first so that's probably a large density of bugs for what? 39 line screen file But well, I Took some time to actually fix so some of these are introduced So this is the guy that fixes those bugs or tries to fix those bugs Doing well actually didn't move over the log file stuff there, but you'll get the idea So this code actually tries to remove those bugs by not setting a Not setting a maximum Or size of the buffer trying to use fkds instead of kds trying to Check if the thing you're giving it to him come inject code on the system call And actually do a sys lock call that we're not overflow So So if we run this guy And we do the same things Probably would not work At least I tried to not make it work. Do I have to get em call here? I Didn't move that over Okay, so However, if you run rats over this guy You'll see one of the issues we have in the audit team Which is that even if those bugs are fixed you will eventually get all kinds of warnings about those bugs so we We went from I don't know how many were there in the original one It was 11 Question one two three four five six seven from seven to four warnings, which are really Did you account for the input having a single quote mark in it? How didn't see that in the code? Yeah, yeah, I use a regular expression for that, but you can try maybe I didn't fix it properly So you mean like this? Like this right well like it actually uses Go really fast here, but we don't have much time actually uses this regular expression to filter the output So if the input information doesn't feed that one, it will it will bail out. Okay. Yeah, that looks good Okay, I missed that. Sorry. Sorry. I skipped through that one Okay, so one of the issues here is that even though we know that's fixed you still get four calls out of 11 Okay, and actually the same happens for flow finder. I think I have the output here So you get three instead of how many of them were there originally on This guy on the box Three instead of ten, but you have to really have to look for okay One of you guys have the idea. So hopefully you have that already Okay, so Some less concerns of the audit team I'm probably going to go fast right now because we have the formal dinner at seven. So I have 15 minutes left Um Lessons learned I'll go into more detail on them, but it's actually you can look that by yourself You have some code there you can look at body It's actually some things I've referenced it either either DCA's or the back numbers that actually were related to that Well, some lessons is that interacting with developers? Maybe many are not really aware of what a security bug is So I got really scared when I was bringing temporary simulink attacks to guys and Through the BTS. I didn't really understood the attack and what it could happen with that one So that maybe means that security Well, actually package maintenance happened to know a little bit more about security coding in order to understand the Box on the issues with the box. Okay? Actually, there's a lot of stuff We found that we haven't fixed yet because the audit team tries to provide patches with their bugs Not just saying hey here you have bug here because we try to test if that bug is actually there if it works If it can be exploited and depending on that we take one action of another one Which means that there's a lot of backlog of probably bugs. We haven't investigated yet, but yes, the Yes, from my experience just from the temporary Simulink attacks possible simulink attacks are race conditions using shared directries. There's a lot of them There's a lot of them in our distribution Okay, one of the issues is that we have a lot also a lot of code to audit and unlike the open BSD team We don't have a common Repo of all the code so we have to go one by one downloading the packages and Reviewing that we even have to know if the guy making the packages use CDBS uses their patch or using whatever because we might be looking at code That is not actually the one that gets compiled and installed in the system So we have to actually not only download the code But you know even actually build the package to get the final code that would be used on the build Okay, so that's and that for one package is not a lot of work, but for 15,000 packages You can imagine that's a lot of work We've seen that the tools we have are useful, but still need some work There are a lot of false positives and fixin security backs takes a lot of time so Just some examples. I go through this slightly fast I just want to point to them because there might be interesting. I was involved in this They might be some other interesting DSAs, but I wasn't involved in them So these ones are ones. I think are interesting DCA 656 which was fixed. I think it was January last year It's actually a BDR Demon that is used to I I don't know what it's actually used that it to handle a video recorder on a devian system so you can Get a video stream from them. Maybe somebody's more familiar with that. I actually just found the bug and reported that The guys that did the package, I don't know if any of the maintainers here. No, yes What they what they did is install the diamond that would run as root, okay? Even though the install fire from the upstream say, please don't run this as root and what they did is well We just put an etsy the full file saying don't run this at startup Okay, which obviously if you install the package you want to install that because it only provided that diamond, okay? Um, so we found well, we found a bug Due to being able to actually overwrite any file on the file system Okay, because you could do a same link attack to an F open call and that meant that Through that DSA that that got fixed and it was a three lines fixed, but we couldn't fix on a DCA, okay? That it was running as root Which meant that if the guys that doing the package would not have taken the decision of run that guy as route That wouldn't be that much of an issue. Okay, because it wouldn't be up the DCA Maybe even a DCA will be published but not as you know, we didn't have to policy that fast because we will go from a Remote a route overriding any file to our mode local user overriding the use files It can get access to which is not a severe, okay? Um You guys have all the stuff. There are probably some of the deeps and stuff that actually went through the DCA on that file on DCA 656 tar This one is interesting because it's an example of a web application of PHP application Running with a database Actually, it's an I don't know if you guys know it. It's acid lab will ask it Which is a project that I started an academic university to provide a good interface to snort So you can actually see the alerts that snort sends with a base in a graphic way You could do analysis of the alerts that were sent and you know graphs and stats or whatever Actually the buck was found by somebody in a fork of acid, which is called base on and was published on this on the mailing list of sourceforge so that We found about it later than the buck was published when he was visiting base because fortunately I was both a co-maintainer of acid lab and a sponsor of the guy that was doing the base package Okay, so when we saw that and we saw the fix from upstream, which is really crappy because it was just Okay, they got published an SQL injection back and the upstream maintainer fixated it by Doing a removing how do you call that? Dot semi-colon on the input stream, okay, which is a really crappy way to fix an SQL injection back Okay, so you guys have there on the tar Is it a target zip file the upstream fix so you can take a look at it and you see how really crappy is that? Yes without knowing the occasion you can see that's not enough for an SQL injection back and the actual fix that got into DCA 993 which is actually a filter of all the data going in which fix both SQL injection and cross-excription box at least most of them in the application what didn't get in was something that we should have Earlier that is okay This is something that is used to you know a local administration to the buck is snort to check out his snort database Maybe shouldn't be open outside. I mean maybe this shouldn't be installed so that anybody can access it, okay? So those are the pieces that got into seed and that didn't get got in getting to charge So you got all the files over there? So lessons learned here is that option doesn't always know how to fix a security bug I mean they might try to but they might not get it right There's security bags that affect some packages to go all the way to other packages Which base was not unstable wasn't testing what wasn't even in testing was just unstable and Acid that was actually on unstable and testing and unstable Well, it's better to shake that is the web applications if you're not sure if they have been audited That's that's for sure and actually the fixes if you see the fixes on the deep actually the same fixes for the cross-exception attacks are The same as the ones that we introduced for the SQL injection attacks Okay, and also lesson learned here that it's mixed with the other DCA I talked about is that You cannot always Push a security fix that will be a hundred percent sorrow even because it changes too much of the code and And it's better that gets fixed beforehand because you're not gonna get a very good fix I mean the security team is gonna tell okay, we're just gonna fix this little but we're not gonna introduce new stuff New stuff here. We don't want to introduce Okay, there's here some more lessons and even more and if you're still bored you have another lesson there So I'm not gonna go through over them because we don't have more time So how can you help and it's really important So it's important that the package maintenance learn how to sport security box. It's not that difficult really it's tools help not a hundred percent, but some stuff is really evident But if you don't look for your own code the one you're writing either the one you're writing or the one you package there's there's no way you're gonna find a security box, okay, and so if If package maintenance also program the code they write properly They wouldn't be security box on the design issues. I talked about and some even the maintenance groups I show you about and there's people doing stuff that is not very secure and needs to get fixed later by DCA So it's important that package maintenance review the all applications and talk with upstream and track the security box there And try to help the security teams Track those backs too because it's not the same if they you see in the numbers of the security teams It's five guys six eyes four guys. It's not that many but we have more than a thousand developers We probably will get better if All the verb is will review the stuff they do package and maintain So how to prevent and minimize? So try to please please very pretty please don't include alpha beta stuff on and stable and if you do Market has not going to the release through edge Okay, we don't need alpha beta stuff that we have then to security support for two or three years I mean we don't need that really we don't And if you do for yourself, just you know leave it unstable, and it really don't have to just leave it experimental At least try not to get that into stable because that means that no verb powers You know, we'll keep with the low manpower. We'll have to deal with that Try to use a privilege users for damage and context I was trying to get that into the developers reference didn't make it through Andrew Biles, so I ended up putting that on security Deviant menu as a new chapter. So I'll try to Expand that so to provide tips for people writing stuff. So try to do it properly for devian. Okay, obviously a voice said Gid said you you ID sovereignty and talk about that But if you have security bugs in those are even worse and there's section of the policy that talks about that So please take time and read about it and try to provide a safe configuration for the stuff you package and maintain So if you don't are not sure that should be enabled, please don't There's a question here There is one problem with this as I Have so a lot of popular packages Not in the men in general especially web applications that are popping out everywhere that are made by people we which Spill out a lot of cold Fast and doesn't have any clue about security The problem is Which is the trade-off you have to do because because you want that suffering today man. You're so popular But you know that they could be many security bugs in there Okay, what is the trade-off the trade-off is please get that audit? There's a there's a team that will try to get that audited before it gets to edge if you ask for it So please if you don't you have if you're not sure that will introduce too many security bugs Don't just say okay We'll take about care about this later because the problem is not going to be you taking care of that It's going to be the security team taking care of that and they're already Very stressed out. So if you have an application that you're not very sure how if the design is good Either it's taking input properly. We have sco injection banks We help just try to bring it to the audit team and have that reviewed I mean if not, we probably end up having to you know, just say that application is up to these applications Are security supported and these 10,000 applications are not I mean what's best for our users? Okay, yeah But it doesn't matter if the security team isn't interested in maintaining it The question is whether or not the maintainer themselves is actually interesting in maintaining the package So if you're gonna upload a package to unstable You need to be in a position to take care of security updates for that package for the next couple years If you're not in a position or you don't think the security team is going to be in a position to do that And odds are they're not then you maybe shouldn't be uploading it So but if you are fine with doing that then it's fine to upload I mean if you're gonna do the work of the security team I mean I mentioned what the security team does in absence of the maintainer If you're gonna do that work you're gonna track bugs you're gonna fix them yourself then oh then do it But I mean you also notice that a security bug as I said before as soon as it gets announced You have time to fix a time to patch and a time to install the patch So that all that time taking account means that your users are vulnerable to that I mean they're gonna hold you accountable for that You better maybe ask them to use back parts or use testing or use whatever, okay? So the applications you maintain have to be reviewed So you should make sure that you review how option answers the box security box Some are simultaneously do not answer the security box or fix that In the wrong way and if he has a very bad security tracker and you can go to public databases and check that out Maybe you should not be packed in that If he has a lot so just to end up With a very bad mouth on my behalf Some conclusions and you have a few more slides, but yes, so you guys can take a look at them later Even though they're gonna be more security technology to use to protect our users And that means selling looks like Manix talk about a Monday. That means you see 4.1 that introduced attack protection Prevention so he prevents most buffer will flow attacks Maybe when packs access shield or S back or whatever gets introduced that means that those are gonna handle Some box, but not all of them. Okay, so you should Take that into don't think that it GC 4.1 or sailing is it's gonna be, you know, the magic one that is gonna Make us not worry about security box and not publish DCA so because they're there So we have to work in a group and that involves security teams and involves maintainers to weed out all the security box, okay? try to code there in Security way as I said and you can use those tools I've showed you in order to review your source the problem is those tools will support only some sort code So if you're writing on on maybe TTS TCL take a or Ruby on rails So I don't know what else maybe those tools don't support your code. So try to fix those so they do and You could do well I've passed all those lessons learned, but you could do well and review those And how that might have applied to you Because those past mistakes we learn about them might prevent future past mistakes, okay, so Running out of time now. I will actually out of time run So you have some more questions there? No, no more questions Not any questions. Okay, then thank you for coming