 Okay, so after a minor technical delay, we can start now. We'll be talking about Securities Support in Debian and tracking open security issues. Am I understandable or I hope so. I am Moritz Mülnoff. This is my kind, doesn't seem a bit louder. Yeah, okay. General overview about Securities Support in Debian. These numbers are for the year 2006. In 2006, we had 316 security updates issued by the Debian security team. In 2007, it's slightly less, but this is mostly due to the fact that we have had some advisories where more issues were combined into an advisory. For example, the recent PHP advisories fixed up to 12 vulnerabilities, while many of the issues fixed in 2006 affected only the packages distributed in a single update. Debian has a policy of backporting only security fixes. So to isolate the security fix and provide it along with an updated package, it's actually the only community distribution which does this consistently. Red Hat does the same, for example, for Red Hat Enterprise Linux. So does SUSE, for SUSE Linux Enterprise Server. Most other community-based distributions like Gantto normally have a policy of providing the latest upstream version and only backport patches in a few corner cases. We have had relatively few functional regressions in 2007. We've had two issues, seven advisories, again, because they broke behavior in one or the other way in 2006. In most cases, these were other minor issues. For example, the SMBFS regression only affected a fairly minor subset of the user brace, likewise for Mozilla, where, for example, the address book functionality of the Mozilla suit browser was broken. You can see that the share of web applications which had a regression in the updated package is relatively high compared to the others. Partly because some of, for example, syntax errors are a lot less easier to track. If you have an obvious syntax error in a compiled program, the compiler typically spots it. For example, in the case of Drupal, it was simply missing semicolon, which wasn't really spotted during the tests we made before we issued the update. Mine is certainly one of the biggest distributions. I took these numbers from a talk by Javier a few days ago. Edge has 282 million lines of code, 198 million of these are C and C++, so there's certainly some room for vulnerabilities. The current security members in alphabetical order are Dan Frazier, Joey Schulte, Noah Mayans, Steve Kemp and myself, plus Michael Stone, but he's mostly backup. You probably won't have seen any advisories from him during the last two years. We have, typically, if you send a email to either security at WNORC or team at security at WNORC, you might either get a reply in very short notice or it may quite also be the case that we're too busy to answer right now. As we get quite a lot of email, feel free to re-poke us regularly. We get quite a lot of mail, and we often do not have the time to check through old mail again. We only have time to do that when we have quite a large chunk of time available, like, for example, here during a DEB CONF. But other than that, feel free to pre-poke us regularly. We don't mind that, it's, yeah, it's just, we don't have much other time to do otherwise. We'll start with the typical kind of workflow security problem goes through. And afterwards, I'll raise some issues where that can improve, either on technical sites and either on procedural sites. We'll start with the discovery of a security problem. Security information reaches us through many different information sources. In many cases, it's one of the maintainers notified by upstream. I think this is about one-fifth of our cases. I haven't made exact statistics about that. This is just my gut feeling in most cases. We also monitor, of course, all security-relevant public mailing lists. So the most important ones are certainly full disclosure and backtrack. We also have, especially in comparison to other Linux distributions, a relatively high amount of security-relevant bug reports from users. As Debian has a technically very competent user base, many of these are able to dissect, for example, backtrace and spot security sensitive changes. Then there's a central coordination list called vendor stack, among which the security teams of many non-Linux distributions and some older legacy Unix vendors coordinate with each other. This amounts to, I guess, another-fifth of all the information we receive. Then there's search, which is only used in rare cases. They often only notice us about an issue if it's only known to the public or through other sources. So that's not really that important. Another important source of information, especially for the relatively unimportant of fringe packages, metro updates, Micah will talk about that later. There are basically two kinds of security information. On the one hand side, there's publicly known information, for example, vendor announcements. In some cases, there's also the situation that someone discovers the vulnerability and puts a certain embargo on it so that we do have the information available but cannot update this fixed package until a certain disclosure date is reached. It's not that important as it may seem from first hand. Most of these issues come through vendor stack. My gut feeling is that only up to 20% of all information which leads to a data security update is coming through one of the embargo channels. Once we've known about a security problem, typically the maintainer is notified. And if you have heard of a security problem, either from us or from a public source of information, be careful what to do with that information. Most of these vulnerability websites, for example, Secunia, do not put very much research in it. They find a new issue, open up an entry in the database, and then they typically only add advisories issued by vendors. So there's few research actually put into it. Be very careful with it, especially when trusting the affected versions. They often put affected version entries in it which do not match reality. Be careful with that, check the source for yourself. Also, do not trust security news sites or general news tickers. Most of them only copy the typical announcement information, paraphrase it a bit, and do typically not provide additional information. The only notable exception that I've put here down is Lino's weekly news. Their security updates are quite solid, and you can really work with that. Likewise, do not trust the meta descriptions. Micah will come to that as well. There's quite a lot of work involved in producing a security update. It is very recommendable to support us by producing a security update. You can prepare patches, which helps things a lot, because we do not need to work ourselves into a code base we don't know before. Especially if it's a more complex packages, you might be a lot more familiar with the source code. So preparing patches, sending to us for review helps things a lot. It's also very helpful to test patches for us, because in many setups, especially if they are a bit more complex, it's quite hard to do that. For example, imagine a complex Kerberos problem. Most of us do not have a Kerberos setup at home. Likewise for, for example, driver updates or complex software like voice over IP solutions. In many cases, we have to rely on the maintainer to test that for us. What you can also do is get in touch with upstream. In many cases, upstream should know best about the code base. They should be the ones who can provide information, and typically the maintainer has a better relation to upstream than we do, so their mails are probably applied to earlier. You might also be more familiar with the information flow inside the upstream project. Also, you can help us understand the vulnerability, especially if a software is more exotic or if it involves technology we don't have very precise information about. For example, Bluetooth, then it's quite helpful if you help us understand what the vulnerability is also quite about. We can research that information for ourselves, of course, but in general, the maintainer should have the main expertise in those issues, so it's quite helpful to provide this to us. You can also check which suits have been actually affected. For example, a certain piece of vulnerable code isn't built in the search version, for example, or maybe if other surrounding factors help mitigate an issue. Yeah, I think I already mentioned the second point. The more exotic an issue is, the more you're inclined to help. I've read many things about team maintenance, praising it as being the sole bullet to Debian maintenance at large. I cannot share this feeling for security problems. We often run into situations where unpleasant tasks like preparing a security update or doing this or maybe testing a security update to helping understanding it is shifted to another one. So, great, we're a team, so hopefully one of the other team members will do it. We've had, we have maintainer groups for up to 20 people inside Debian where no one ever bothers to update, for example, to follow up a security bug report. I guess, and I think I've heard that before from some block entries that some of the RIS team members have had similar experiences, so I suspect this is not limited to security problems at all. But I think there's also a solution which can be put down to mitigate that. For example, one could designate a kind of group contact for such more important issues inside group and other solution would be to more aggressively drop packages which are poorly maintained from the archive. So, if there's poor cooperation, for example, doing the lifetime of Edge to just drop the package for landing if things don't improve. The BTS has a security tag and has different severities, but these are not terribly important for rating the effect of a security problem. Most people are a bit confused by the descriptions which are put on the BTS help site. In generally, whether an issue has been filed with a grave severity or an RFC severity at all does not change things for us very much as we still try to understand these issues on our own and rate the severity differently. But the BTS is a very good place to store all the relevant information, so if you have opened a bug report about a special security problem, please CC it all the time so that all the information is kept on the central place in the archive so that we can go back. So, if there's a different issue which is still somehow related to the original issue, happens again, it's quite easier to find for us. And most other security updates don't close bugs. I'm not sure if this is a technical problem, but in general, please, if you as a maintainer have a security bug, please use the versioned BTS to close them with a proper suit. I think that would be better if we... Closes in the bug, in the change lock? Usually, so they're not closed technically or you just don't put the closes in? We've had issues where the closes inside the change lock didn't work for cyber security updates, and I haven't put that very much work into that. I'm not quite sure whether it's a technical problem or just in general, we just issue an update, so please use the versioned closes to mark them for the correct suit. Vulnerabilities are generally triaged by importance. While you may be the maintainer of this package, please understand that there are other security problems but also that there are other packages behind your package. We also have to look at the general picture. There may always be issues which are more important. In general, while the installed base is, of course, important, there are also... the much more important factor is to look at the ramification of a security problem. So, for example, where the case about a year ago where some signal handler related races were found in OpenSSH, while the OpenSSH Kerberos version theoretically had potential for code injection due to a double free, the OpenSSH plain version without Kerberos support wasn't affected, it was only a mere DOS. So, we put the OpenSSH Kerberos fact way before the OpenSSH update. Although OpenSSH Kerberos 5, I think it has only 100 users installed, while OpenSSH is, of course, present on nearly all other systems. There are often denial-of-service security problems. These are a bit hard to judge. Sometimes, especially if there's a company who is earning money on selling security information, always likes to label a regular bug as a security problem. So, if you have, for example, a denial-of-service bug, which leads, for example, to a crash, one should look very carefully whether this actually has security-relevant ramifications. We normally always fix these for high-profile server applications. So, obviously, if you can crash a patchy with a single HTTP request, this is obviously something which needs to be fixed. If there's a way to crash your browser with a specially malformed HTML page, you would need to trick the user into visiting and browsing it. This is much less of an issue. Actually, we don't fix that at all. So, if a user runs into such a website, which makes his browser crash, he will simply not visit this page again. There's nothing lost except maybe his browser history or something. So, these are borderline cases with 200 million lines of C and C++ code in the archive. You obviously cannot fix all simple crashes. C is simply not made for this. In many cases, you also probably need to design your setup a bit more carefully. For example, if you have an application which processes untrusted data, you should probably put in line a second filter or simply fix your setup so that the effects of a crash in one part of your setup does not have bad effects for the application at large. Many of these crashes are also fixed as generic bug fixes for your stable point release. I will come to that later. From time to time, you might notice security updates from other vendors where you wonder why Debian does not issue an update for that. Sometimes, security problems are just so important that we do not put out an update for that. Even if the update itself may be relatively harmless, it still has the downside that the admin needs to look at the update and need to spend time on installing the update. It takes away attention from other more serious issues. For example, if you see an Apache vulnerability every week, while five of these may be very, very minor, you simply take away attention from the really serious one which the admin may want to install immediately. Also, despite all testing and QA, you still have the inherent risk of regressions for each update. So putting out a security advisory for something which is not really a security problem does not deserve the users very well. For example, there are some... I searched out two examples. One went and fixed a denial of service vulnerability in libpng for the part of SysLinux which displays a boot up PNG logo. So if you have an attacker which is able to compromise the central boot directory where the PNG is placed, you have much worse problems than a DOS in the code which displays the boot up logo. Likewise, some vendors fix buffer overflows in GDB where running a binary in GDB may cause a buffer overflow in some ELF sections which is actually not that much of a security problem because if you feed someone a malformed binary, you have much more easier ways to execute code if you just execute the binary inside GDB. Concerning updates and updates which are not issued, we're now moving on to non-free. In general, there's no security support for non-free. While the security fact implies that security updates may be issued if it's provided to the maintainers, there haven't been very many cases of non-free DSAs in the last years. The main reason for that, I can only speak for myself in that regard, is that whenever, as long as there are open security problems and free software, I'm unwilling to spend my spare time on non-free software. And I have the feeling that most other security team members do the same, do feel the same. So currently, it's not... I don't think it's a very good solution that we ship a lot of non-free... and really risky non-free software in our archive right now. For example, the MedWiFi drivers, I think they allowed code execution by scanning for a certain access point or by certain beacon frames. Likewise, the NVIDIA driver, the non-free ones, which were affected by an integer overflow in the general code for the 3D driver, which was indirectly even exploitable if you just visit a malformed webpage and display a specially crafted image. Likewise for Flash, which has serious security problems all the time. So my recommendation would be that either a group of dedicated volunteers provide some kind of security support for non-free or that we either drop these at all. Providing security support for non-free has also massive handling problems. Red Hat includes the Adobe Acrobat Reader in the Red Hat Enterprise Linux product, and Adobe decided to raise the dependency on LibGTK from 2.2 to 4 during the lifetime of Red Hat Enterprise Linux 3. So they had to backport a complete version of GTK 3, which is only used for Acrobat Reader inside Red Hat Enterprise Linux 3. So this is not something we really want to achieve, I guess. Many of these non-free vulnerabilities are fixed in point updates eventually, but one should also point out that these point updates appear a lot less frequent in comparison to regular security updates. We also defer some minor issues into point updates. Once we've finished with the discovery step of the security problem, we now move on to actually building the update. There's a separate security build D network. Once the updated package is uploaded to the security host, all the other portables are handled in the security build D network. If you help us with preparing a package, please do not upload to Clacker directly, because if there's something broken with the package, we need to fix something. The process of revoking a build is quite difficult, so we normally would have to raise the version number by another version increment. So please don't do that. Please send us a depth div, we'll look at it, and if it's all right, we'll tell you to upload. Once the builds have been built in the security building network, they are signed by the security team members and uploaded to the security host. That's actually one of the reasons for certain drawbacks. If you notice a DSA, which lacks one or two architectures, this is typically due to problems with the build host. The problem is that most of these build D issues cannot be fixed by ourselves, and about two-thirds of all security build Ds are run by two persons, which are typically very busy, so it always takes a bit of time and prodding to get the build change route fixed, so this is a kind of inherent release, inherent problem to release all architectures in sync when a security update is released. But it does not happen very often, so it's not much of a problem, but a bit more redundancy would certainly be nice. Then we can have a look at the build times. We can see here, this is from an older build, I think it's a year ago. It was for KDE base. The build times for MCXA were about ten times slower than all the other architectures. ARM, as you may see here, is still bad. It would certainly be nice if we had a faster arm security build demon I'd asked before, and I think there were some offers and there are faster machines available, so this would allow us to actually speed up security updates a lot more if we had a faster arm admin. M68K also had other issues, despite the mere fact that the arc is slow. For example, you once issued an update for Firefox. For Firefox, you upload the package after six to eight hours, S390 and AMD64 have built, then you wait another day, and most other architectures except M68K and ARM are built, then you wait another day or two and ARM arrives, then you wait three days and four days, and then finally you receive a mail that the build failed on M68K because it ran out of disk space. Then you ask, build the admin, why is this the case? Then it says, oh, we only can use some very ancient SCSI parties that are no longer publicly available, so we need to dig them up with ebay. It really, really had some major issues and I think right now it's quite an improvement because it allowed us, for example, to do the security updates for Mozilla a lot faster than we used to do for the such times. It only had 10 users. If you look at Popcorn for the M68K users, there were no more than 10 users at any time when I looked at it, so it's not really that a lot of the potential Debian user base was kicked out here. Once an update is complete and advisories send out, you should always subscribe to Debian Security Announce while Debian Security Announce is broadcasted to many other multiplication factors like news websites, vulnerability databases. It's always recommendable to subscribe directly to the Announce list and you should not rely on the web advisory overview. This is mostly intended for historic data or maybe looking at which updates were issued at a central point of time. We have a fairly large target audience. We have 33,000 subscribers to a DSA alone plus a lot of multiplication. Backtrack alone has another 50k subscribers, so Debian Security Update reaches quite a lot of people. The web advisories are now integrated into WebWML like the rest of the Debian website. Hopefully this will be automated more right now because right now we need to convert the text advisories with a script and upload them to a CVS and it's quite annoying. That's also the reason why we sometimes forget to add them. So if you wonder why on the web overview a certain DSA is missing, that's because it's an additional manual step which can easily be forgotten. And it doesn't matter that much because the canonical source as said should be subscribing to the DSA mailing list. We're sometimes asked why we don't provide severities in our advisories. I don't think that makes a lot of sense because the application usage scenarios vary too much while some issues may be important, maybe more important in a different setup than others. You should judge that on your own. In general, we recommend to install all security updates we issue. We have a kind of implicit severity rating because more important issues are released faster or prepared faster, but other than that we do not put explicit severity taggings in the advisories. Often you may also have local effects which mitigate the effects of a security problem. For example, the recent Zamba vulnerabilities were not much of an issue for many people because it only affected untrusted users while this is typically not the case for a local LAN where Zamba typically operates. The same vulnerabilities would have been much more grave if they would have affected an application like for example Apache, which is actually meant to be exposed to the wider untrusted target audience. There were some proposed new features and apps. I don't think that's a good idea. You should always read advisories. In some cases it may be inevitable that we break existing behavior, so we need to be able to document that so that you can react locally depending on your setup. An example for that is the pseudo fix which changed the behavior of environment variables. It broke some setups for people who did not read the advisory text which actually explained how to fix your pseudo configuration. There are some kinds of questions directed to the security team which we often do not have the capacity to answer. If you have a question about secure programming or asking for an audit of your program there are more specialized lists for that. For example, Debian Audit, if you have general questions about setting up a secure environment using Debian, use Debian security at least Debian orc. We have a new feature in Duck which was implemented by Anthony Towns. I've put it into effect already a few months ago, but we now also have an overview page. It basically differentiates between issues which are embargoed which are still under embargo date and may not be disclosed. This is essentially the same handling as was used before. So until now the security queue isn't visible to the outside. We now have a separate queue which is called the unembargled queue where all publicly known issues are present and from which we also generate a status page which actually uses the same code like the new code. It's actually still says new and by hand packages somewhere but this will be fixed sometime. Right now it's mostly a preview and it will be moved to the proper security debut or costs eventually. Sometimes it's worth a case that an issue is embargoed initially and the embargo date is disclosed. For example, because someone has lifted the embargo for the proposed embargo date or because the Debian update is not yet ready we also have the possibility to unembargo an issue so to move it from the embargo to the unembargled queue. Right now, if you look at that URL you will find, for example, a Mozilla update for Sarge. This actually has been known for a couple of weeks now so it's present in the security queue. This feature is also used for if you're a maintainer and have worked with us to provide a security update then you can see which architectures have already built and which are available. We're having security support for testing since, I don't know, maybe one and a half years. I think the first step started. Security testing has certainly improved tracking and transparency of security problems in testing but it still has not managed to produce production quality security support for testing. Many issues are unfixed and unstable for too long and we also often have major transitions which, for example, the recent G-Lib C transitions which prevent security fixes from entering. There used to be a mechanism to provide security updated packages especially for testing and copy them to a different suit called testing security. This process is quite complicated so it wasn't used very thoroughly and there's also been quite a lack of manpower in that regard. So we met a few days ago and proposed some fixes which should be implemented. Right now, advisories are sent out for each issue. This will probably be replaced by a tool called DebsiCAN the Debian Security Analyzer which is already in the archive. DebsiCAN allows you to check your currently installed system base against open vulnerabilities. Micah will present it later as well. Then there are some technical improvements to allow the easier production of security updates and I think we also proposed that it would be interesting to relax the rules for security problems so that slopey maintainers may not have security problems unfixed in SID for too long. Right now, in comparison to several other Linux distributions Debian has less hardening features implemented in the stock distribution. We basically only provide the ones which are present upstream in either the kernel or in G-Lib C. Hardening features mitigate the effects of both unknown exploits and also which addresses the problem of vulnerabilities or zero-day security problems we don't know about yet and also mitigate the effects of security updates which have not been installed by the local admin. One of the problems, I've only looked briefly into that but that's the state of affairs as I know it is that some of these hardening features are not available across all our architectures and I think we should improve to that for Lenny. One should also note that security hardening features are not a silver bullet. Often we'll have the problem that a vulnerability which is suitable for code injection will simply be turned into a DOS vulnerability because in a service place, all the system simply terminates if it notices an attempt to allow code injection or to inject code. Fedora and Red Hat Enterprise Linux are fairly good into that regard. I think two interesting features which could be implemented for Lenny, for one's fortified source which is present in Fedora and Red Hat Enterprise Linux for I guess two years now and also the stack smashing protection from GCC which is upstream in GCC since I guess 4.1. It's based on pro-police by IBM and likewise as fortified source it's used in Fedora and Red Hat which has the nice benefit that many of the bugs unveiled by these hardening features are already fixed stream for one of the maintainers in Fedora and Red Hat Enterprise Linux. But one should also know that the security team will not have the resources to manage that all by own so this would need an interested group of volunteers who coordinate things and who are interested in coordinating with maintainers checking that the necessary features are applied and also look into eventual bugs that arise from these problems from these features. I think we also need properly maintained Erata many other Linux distributions have that so currently we only have Erata inside the installation manual we document known bugs with the installer but we do not properly maintain Erata for all the problems which are discovered during the lifetime of a stable release. This is not only useful for security bugs but also for general bugs affecting users as well which cannot be fixed in a stable point release. We could, for example, use this security-wise to document software which is so broken that you can no longer use it at all or maybe document cryptographic advances or document newly arrived weaknesses. This would need a maintainer I think there was a proposal to do this in a wiki but I think a wiki has quite some problems with that I think the best way would be to have a volunteer who can be CC'd so if there's a discussion for email that something needs to be added during the Erata and that this person can simply be CC'd or maybe a whole maintainer group maintaining this Erata but I think that's really an interesting point because it allows us to document all shortcomings we cannot fix. There's also some infrastructure improvements that would be quite desirable we sometimes have the problem that we need to search for specific source code file inside the whole archive. Right now packages.debian.org only allows you to search for the contents of binary packages it does not allow the same for source packages that would be really cool. Some people have also been working on whole-level source-level grabs or binary fingerprinting, Flory and Weimar once made a complete archive scan using Clemavie he searched for copies of libxpdf which was vulnerable and embedded in quite a lot of applications so having a general infrastructure for that would also help especially with more complicated security problems and I already mentioned that web advisers should be handled more automatically. There's already some existing code I guess in Duck for that so maybe this just needs to be improved. Another problem which is giving us headaches since sometime is the archive growth if you package an archive please think about that not every package is necessarily a benefit we are at about 15,000 packages right now and if the current growth rate persists like it is right now we'll run into much more severe problems especially be careful if you load image or software if you'll upload some network-related code or web applications be very careful if you upload pre-releases or test versions you know to be buggy especially if it's just a few months ahead to the next stable release we have quite some broken software in edge which was just released which was uploaded quite shortly prior to the release so that the problems in these applications were only noted after release. Some examples if you search for minimal with AppCache you will find hundreds of services which implement a well-known network service in a minimal way which is known to be secure and with a low memory footprint we typically if you want to package the 18th HDPD which is a minimal HDPD you should think whether it's actually a benefit it's not only because of the added overhead for maintaining all these packages but it's also quite difficult for an administrator to make an informed call which of these 17 HDPDs is actually good because if he doesn't know he would actually need to test all 17 so it's much, I think it's much more reasonable to limit the choice in some fields of application inside Debian. Of course we are the greatest distributions but sometimes we also need to limit ourselves because otherwise things get out of hand. Another problem with fringe packages is that they don't receive very much review they are typically written by one author no one else looks at them if it has only five users it's quite unlikely that one of these five users is doing a security review of these package likewise they will probably not be noted by security researchers so that's another reason to be careful be extra careful with network-related code this is actually related to the 17th minimal HDPD if you're mentioning. You may think that some software is just secure but even at the classical age at all had a security problem it was a temporary which would potentially allow overwriting other files of the user so in general every package may have a security problem eventually although it's of course more likely for web applications and network-related code. Another problem is that many maintainers only focus on their specific pet package they might need, they might have a short time interest in that package because it looks cool and their package lose interest after half a year and then it's still stuck in the archive and we need to support it for at least 30 months so if you select a package please think of that the resources to fix security problems are not endless each advisory takes at least two hours of work for more complicated advisories like for example the recent PHP updates it can cost up to 20 hours of work so be extra careful if you want to have a package in the archive if you want to have it properly maintained this also creates some work for others also make sure that a package is supportable for 30 months we have estimated release cycles of one and a half years once a release has been released it receives another year of old stable security support so it would have to go with at least two and a half years of security support if your application is so volatile that upstream doesn't care even after two months or that they only recommend to package the latest version it might not be a suitable candidate for a stable release there's a separate BOF on the supportability of 15,000 packages last time I checked in Panther it was Thursday at a quarter to ten in the lower BOF room so if you're interested in that topic you should probably attend that the positive side is that review of maintainers at large sometimes work sometimes works we once had a CGI script in the archive which allowed web-based password changes it was already at version 4 and was available since four years one would think that a central piece of code which allows password changes was quite... was coded with great efficiency and was carefully reviewed but once it was uploaded in the archive one user noted that the privileged username is taken from a forwarded environment variable filed an RC bug then other people looked at it and found several buffer overflows and this package never even made it into the testing distribution because an RC bug was filed immediately and I think it was removed three months after the initial RC bug was filed the funny side story of that is that later on upstream complaint that Debian was too demanding because they were insisting on having secure software which allows web-based password changes I've already mentioned web applications web applications are in comparison to traditional compiled code a lot more volatile less dependencies because they typically run inside a HTTP environment and are not necessarily coupled to system libraries they often have the problem that new attack schemes require rewrites of the whole code base to focus on new attack schemes it's not as simple as many traditional vulnerabilities where you just need to impose additional sanity checkings or boundary checks upstream is often less experienced so we often have upstreams who cannot really dissect the secure problem on their own and the upstream fix can be less trusted and it's also notable that many of these web applications are also packaged by maintainers who are less experienced notably often by people who are fresh from the NM queue so be careful when sponsoring a web application and make sure that you are able to fix up eventual problems which arise from the fact that such a package was uploaded there are about 10-20 web applications in the archive which have really disproportionate share of security problems Micah will note that again later as well and with web applications we always also have the problem that most of them do not release patches for security problems but just complete new upstream versions so we have to dissect the whole intradiff and check which security fix was done inside the whole release upstream coordination is much more difficult and some upstreams don't even know the concept of a patch if you look at the website of a very popular web-based bulletin board system they release patches which are described like then please enter text file such and such go to line 360 add these four lines then go to line 317 delete the next three lines and put the next four lines in so if you have an upstream which does not even know what a patch is then it's quite unlike that they are able to release quality patches for that so we definitely have a quality problem here another problem is that few other distributions package that many web applications several of these web applications are only present in Debian except from sometimes Gantu which only release the next new upstream version anyway so coordination with other vendors and patch sharing is also quite difficult I think the proper fix would be for about 10 to 20 of these packages which are really causing a lot of work to move them to volatile instead because that pretty much matches their users pattern even most of the maintainers maintaining such software often tell us that they're using the sit version anyway so it's also quite difficult to get the maintainers to properly act together in preparing a fixed update with many web applications there's also the phenomenon that many of these may very well be limited to an authenticated HTTP zone for example we have SQL ledger in the archive for which we decided along with the maintainer that the way SQL ledger is supported is that it's only supported if run inside an authenticated HTTP zone because it's a double based accounting solution this is not something you would typically put into the public internet for everyone able to success so I think for many of these web applications it might be a proper solution to just document that this package is only supported for a certain field of application and not at large for any application you'd ever possibly think of now I'll talk about three notable packages which are a bit different from the typical packages which cause security problems the first one is PHP you should note if you run an application written in PHP you should have a look at the readmedavian security file which describes the scope of the security support presented for PHP PHP does not have a sandbox mechanism for example as Java has one so security problems which allow code injection as security problems if they are only exploitable through arbitrarily executed PHP scripts we treat these only as security problems if they can be triggered through a web application Sean Finney the currently the most active and actually I guess the sole PHP maintainer is planning to integrate the sewer in hardening patch from Stefan Esser of the PHP Harden project into our mainline PHP distribution it currently has some problems due to architecture compatibility issues but I guess once these are ironed out this will be a proper solution which helps us to mitigate some of the effects we have right now inside PHP PHP often literally has a steady flow of security problems so we have quite some problems in the proper upstream security management process the good thing for Lenny is that we were able to remove PHP4 from the archive actually it is currently being removed but this will relieve us from maintaining both versions at once because according to my experience about every second we need to do a lot of work to remove PHP4 so that's quite an overhead because we need to back porch security fixes for all PHP security issues to the edge version of PHP5 the SARG version of PHP4 and the edge version of PHP4 which is quite an overhead Mozilla is another software group we are still doing isolated security fixes which were back ported for edge we decided to go along with the point releases released by Mozilla because the steady flow of security fixes is just too large another problem is that it's really hard to get people to test these packages with isolated security fixes the latest Mozilla 2.0.0.4 version for example fixed security problems this is not something you can realistically back port in a reasonable time frame with a sheer volunteer project shipping micro releases also has the benefit that we profit from all the QA done by the Mozilla group right now for example for the latest round of Mozilla updates we haven't heard of any regressions the only one was that we have extension which is the free version of Thunderbird had too strict dependency on some traybiff widget but really nothing serious so it actually works well for Lenny, Mike Homie and the other Mozilla maintainers are working on integrating Xulrunner which is kind of Mozilla library into all the other packages so that we need to fix all these security problems only in Xulrunner and do not need to issues all the updates for Thunderbird Mozilla and Firefox in together if you have a really security sensitive environment it's recommendable to run Mozilla or Ice Weasel without JavaScript support many of the memory corruption capabilities potentially leading to code injection are actually introduced by one or the other bug inside JavaScript and many of the web logic security problems can also be mitigated by disabling JavaScript entirely the kernel has a some people think that the Linux kernel is way too insecure while there may have been a phase a few years ago where quite a lot of data have been found this has significantly become better there were actually only very few local privilege escalation vulnerabilities found right now the last one was also fixed very quickly we were actually the first Linux distribution to provide a fix for that there was the case that someone posted a zero day exploit to full disclosure in the afternoon so this hit all the security teams of the commercial distributions badly because they're already in the weekend but it's actually the time where we have the most spare time available so this was quite a coincidence most of the issues fixed in the Linux kernel are local denial of service vulnerabilities many of these will not affect you if you have a really complex multi-user environment many of the vulnerabilities labeled as local denial of service should not be a security problem in most environments we still fixed them anyway of course because they're always the 10% of setups which could be affected by those there will be updated Xorg and Linux kernels with security support roughly I think it's not decided yet 9 to 12 months after the edge release so that people who need to run Debian Edge on a more current hardware have more recent driver updates available there's a buff on that on Thursday as well but I think it wasn't quite sure yet at 3 o'clock somewhere here in the Teve probably sometimes people raise the issue of enterprise support cycles are typically described as support cycles of 5 to 7 years this is technically feasible we certainly have the know-how to do that Woody had more than 4 years of security support because the start release was delayed quite long and we still had another year of stable security support so we were actually able to provide security support for the whole Woody release for 4 years right now for Edge with our current resources I don't think that's longer feasible it would certainly be possible for a selected subset by not providing security support for all the main servers in the archive but by only limiting it to for example XM and Postfix and to the base distribution and typical server systems because I guess that's typically the environment people may want to run in a 5 to 7 year time frame still this will not be able with a pure volunteer based security team at long so this would only be possible if some company who's interested in providing service level agreements for 5 to 7 years this company would need to invest manpower to actually make this happen I don't believe we have the resources to do that with a pure volunteer based systems right now the commitment alone for security support as we do right now is already quite high and it's already becoming too high in some corner cases so this could only be done in cooperation with an external entity I'll skip a few slides because Michael's part is also quite large I'll only mention some of the more interesting issues I'll provide some non-linux ports in preparation notably the HURT the K3VSD port and the Solaris based ports it will be pretty difficult to provide full security support for all these ports as well because it would certainly be possible to simply recompile all the updates issued for our normal Linux port and simply recompile them on the new ports but it will be fairly difficult to provide full security security support for all these because all these different kernels might induce completely different behavior and it might very well be that a vulnerability which is a non-issue under Linux may be exploitable on the Solaris kernel and likewise for the VSD kernel and of course the other way around as well so I'm not sure this is really realistic of course it would certainly be possible to simply recompile the regular security updates as done for Linux we have quite a problem with embedded code copies embedded code copies are copies which a certain upstream has included into the software package for example the LibAV codec and LibAV format library which has been embedded in quite some applications because the current version of for example Mplayer was in edge was not able to link against the system-wide copy provided by ffmpeg so this needs to be improved for edge as well web applications are also problematic we only have the recent case of a PHP package called PHP mail which is an interface to send mail out of PHP application although there is a system-wide copy available in the archive there were seven or eight other packages in the archive which include a local copy of this PHP mail instead of linking to the system-wide version so this would in the worst case require to issue eight more security updates and this really needs to be fixed actually it is already fixed by now because several bugs were filed but one of the problems is that many of these embedded code copies are only noticed once a security problem in the root package is found because many maintainers do not keep in mind that this may cause a problem I think there could be a better cooperation with people providing custom Debian distributions or derived distributions of course this cannot be a one-way relationship so we would actually need to have some positive side effects from these derivers so that we would be able to provide a better service for us or maybe free-release testing or review of patches I have also noticed that several derived distributions or live cities do not have security support at all it might be that the overhead for providing security support on your own is just too difficult and just to error-prome so if we provide a better service and do not have security support in place this will also benefit users at large then I'll close with some best practices we sometimes have problem with obscure patch systems if you insist on using a patch system please either use a standard one like Dpatch, CDBS or Quilt or if you still insist on building your own handcrafted patch system you will be able to build if one of the patches cannot be applied and please document properly how your package should be in a mood there have been one or two cases where we actually only noticed a few days after release that a certain patch could not be replaced for vulnerabilities where we do not have a reproducer problem so please be careful with obscure patch systems then we have a final help wanted session Michael will come to that come to some parts of the security tracker peer review and double checking for that will be very welcome the PHP and MySQL maintainers also need a dedicated volunteer who is willing to support maintenance and especially helping with security problems and you can also always follow up on open security bugs in the BTS most of them are always tagged with the tech security so if you have time available check open bugs, provide additional information and see whether you can reproduce them or which suits the effect now I will hand on to Michael to talk about the Debian security tracker and the way we track security relevant information inside Debian okay hi give a moment for people to leave I'm going to try to make this relatively quick because I know this is a long session so I'll start here I'm going to talk a little bit about how some of these security issues more it's mentioned much earlier in his talk about that maybe one DSA was issued on average a day in 2006 and as everybody here should know a DSA is only released for a package that's in the archive and is vulnerable to an affected vector that is a known security issue so how do we figure out whether or not a particular security issue is affecting a package in Debian more it's mentioned a number of different sources of that information but there's a lot of different publicly available sources of this information that are kind of haphazard and not very reliable in terms of making sure that all the issues are covered so let's see in 2006 we went through the CVE database we did this in 2005 as well and I'll talk a little bit more about what that is, the CVE database in a minute but we went through approximately 8,000 issues that were known and almost 6,000 of these issues weren't at all related to Debian so there were quite a few issues that were related to other organizations or were applications that weren't in Debian but were PHP or CMS related applications that haven't been packaged for Debian more it's mentioned the web applications and PHP being particularly problematic these numbers are kind of rough, they're based on the name of the program itself so only the programs that had the word PHP or CMS in their name are actually quite common for PHP applications were counted here so there was quite a few more Cisco we're not particularly interested in don't apply to Debian and Oracle as well but you can see that there are quite a few issues that aren't accounted for here but once we go through all of these and determine what actually is associated with the Debian package we can see that there are almost 1,500 issues that affect actual packages that are in the Debian archive and you can see this is about almost pi in how many per day that we have to deal with here so how do we do this I'm not sure why this is kind of pausing like this but so we have this security bug tracker in some ways it's like the BTS but it's going through some of the publicly released issues from Mitra and determining whether or not these are associated with particular Debian packages there's a web page here I don't know if I can click on it to show you but I'll try oh you don't have wifi okay well you can load this at home the page basically processes our data and we'll talk a little bit more about the data here in a moment and automatically generates reports for the different suites so you can look and see what issues currently are affecting stable, unstable testing and old stable as well as being able to drill down to the individual packages and see what issues are affecting the individual packages what versions of those packages that issue affects so this information is called from various data sources but primarily the issues are coming from the CVE database and then being cross-referenced with the DSAs that have been released by the security team and additional information that security team enters or is brought in from various mailing lists usually these issues end up getting a CVE assigned to them at some point and also issues that are discovered in Debian packages and reported to BTS in some sort or another so how does this work it's basically fairly simple there's some four fundamental guiding principles for how this tracker works and I'll go through these here really quick there this tracker is not people seem to think that the tracker is a way to keep track of what packages are being worked on by the security team to provide an update and who is working on that that's not what this tracker is it's primarily to keep track of vulnerabilities and weed out the vulnerabilities that are not ours and then identify those that are associated with particular packages or versions that those packages are affected so the simplicity here is that there's Subversion Repository on Alioth this is public you can check it out and it's essentially a plain text file that you can edit with your favorite editor and then commit back to Subversion and that's it it's very simple there's no web front end that requires authentication and ticketing and a prioritization that ends up driving you insane so this is essentially designed to be incredibly easy to do short amounts of work without having to deal with a frustrating overhead interface the data that is in this list comes from the Mitra database which provides CVEs formerly known as CAN entries these are numbers that are uniquely assigned to issues after they have been submitted to Mitra these come from a lot of different databases and from a community-wide effort to identify issues that are in various software packages like I mentioned earlier Microsoft macOS everything is included in this and they bring a lot of this stuff from mailing lists security mailing lists, bug tracks CUNIA websites and there are various members in the security community on this editorial board that weed out issues that are not relevant although there are a number of issues that are not relevant that make it through still so this is freely available from this website and it's run by Mitra which is a U.S. organization that runs computing research facilities for the Department of Defense the Federal Aviation Association the IRS and it is funded by the United States Department of Homeland Security they're the ones responsible for the technical or fear levels that you'll get at the airport currently we're at Orange are you afraid so CVE is basically which is kind of strange they're basically unique numbers as I mentioned before for known issues and when they're first put into this database they're considered candidates only and then after they have been confirmed they're turned into accepted entries although this is kind of a wishy-washy process there's a lot of the entries that maintain candidate status for pretty much forever although when something like PHP releases a new version and they mention in their change log that this fixes this particular CVE entry then it becomes accepted I don't know why that's coming in and weird orders but the CVE contains a brief description and references you probably can't read that but this is an example from the website for CVE 2007-2748 and I can't really read it here either but the description says that it's a substring count function in PHP 521 and earlier allowing context dependent attackers to obtain sensitive information via unspecified vectors and then beneath that are some references to change log entries in PHP some mailing lists archive entries and a couple websites where this is mentioned so the description doesn't have that much information in it so a lot of the times in order to determine what if this issue actually affects Debbie in any way we need to look at all these references and start tracking down in the source whether or not it's affected at all just because it says PHP 521 and we have PHP 5 in the archive we can't just automatically assume that this is affected by this particular issue because patches may have been back ported already or various reasons have been fixed so there are a lot of things in the CVE entries that are very difficult to kind of extrapolate the genuine information and determine whether or not this is a problem so there's specifically I think I say here how not to read these there are this last one mentioned PHP 521 and earlier so this doesn't necessarily mean that PHP 522 is not vulnerable so that has to be checked this one in particular they mentioned in the change log for 522 is fixed so it probably has been but it needs to be verified and the only way to verify that of course is to look at the source and look at the actual proof of concept or exploit itself to determine if it's there another common thing that happens is they will say that version 522 is not affected but that doesn't necessarily mean that the version prior or all the versions prior are also not affected it's not always very clear the versioning numbers what is affected and what is not and the CVE team doesn't necessarily always know or put that much research into that so we need to look into those in order to determine that and then the fun one here is when they say things like unspecified impact via unspecified vectors sometimes they'll say in unspecified package or unspecified program so these are particularly frustrating because it's not very clear what the issue is and sometimes those will go away upon review or sometimes there's information that may be embargoed or needs to be dug out a lot more before it's actually determined what the functional problem is so there are some other systems that are have been attempted to be created that are different than the CVE list that maybe are more machine parsable via an XML based language there's a couple examples here I won't go into too much detail but they haven't really caught on because they have primarily just been developed by some private security companies and weren't really worked out within the community and nobody really knows about them so they're who knows they might someday be something better the Mitra folks have this open vulnerability and assessment language and that's primarily used for determining system level vulnerabilities to issues so right now the Mitra database is pretty much the best thing that we have and the most centralized and available and parsable dataset that we can work with unfortunately there's a lot of improvements that could be made there so part of this process of tracking these issues is that a lot of this stuff is automatic we get these updates from Mitra via cron job just pulled into our Subversion Repository and committed and then these are of course sent out via our commit list and spit out onto the IRC channel Pound Debian Security that there's been an update and the data in this Subversion Repository is parsed regularly to generate this web page so all this stuff happens automatically and provides us with quite a bit of information already but there's quite a bit of it that is not automatic and that involves processing these individual CVEs that have been released every day some days there's none that have been released sometimes they dump 20 of them sometimes there are issues that come in from 2005 years later that for some reason they're not released and they suddenly release them but in general we need we go through and process these in order to determine how they affect Debian so the entries in this file each one of the CVE IDs have a to-do line that says check and we go through and look at it the particular CVE and try looking at the references to try to figure out if this affects Debian whether that information into the track or committed into the Subversion Repository and then the tracking information on the web gets updated so that it has something useful so an example here of a package that is not in Debian either not at all or maybe will be when it is packaged because someone has issued an ITP or an RFP this particular CVE was for Apple Safari so we changed the to-do line to a not for us and put in their Safari so then that issue is dealt with very easy to deal with because it's very clear that this is not something that will ever be in Debian an ITP we will note in the file as having an ITP label and put the bug number for the ITP so that it can be tracked and the tracker processes these and makes a list so we can keep track of all these ITPs over time some of these entries say reserved on them and these are possibly because they are embargoed issues that haven't been released yet are on vendor sec or a distribution has a block of CVEs that they've allocated one of these but they haven't provided of what the issue is so this will say reserved and later on it may have changed to something or some of these may get ended up getting rejected due to not enough information or the upstream disputing the issue not issues at all or mistakes in the tracker so these end up getting rejected so if we actually find a package that is in the archive that has been affected by one of these issues then we note the package name and the fixed version and a severity level and this one here is gallery this is an issue from 2005 that's quite a long time ago but this particular PHP issue in gallery was affected in the Debian package 1.5.1-2 and we associated a severity level with this particular problem as medium the severity levels are relatively arbitrary they're mostly used for triaging and determining the relative severity and how whether or not we're going to deal with this issue or not or maybe the security team needs to make an update on the stability they're not particularly detailed here let's see some of these issues are not fixed so we need to file a bug on these issues and we file a lot of bugs after going through these CVE entries and this one is a PHP bug and affects multiple versions of PHP so both PHP 4 and PHP 5 are noted here and the bugs and the severity are also noted the tracker will process this information and provide links to the particular bugs for this particular CVE in the PHP package so you can drill down and see all this information very easily and conveniently once we become aware that this issue will be fixed in PHP we'll note the Debian revision number that it will be fixed in because Sean will let us know because he's very good at that and you can as well that helps a lot these severity levels as I said are just used for triage and use your overview of all the packages we also have different severity levels that we file in the BTS when we discover an issue in the PHP package but as Mortz was mentioning not all these issues are release critical issues so they are put in there as these different four severity levels and if people are interested in how we determine what these severity levels are or whether a package meets a high criteria we also use distribution tags in the tracker to provide information about which distribution is affected by an issue because a number of issues are maybe only affected in SID for example or Sarge but the tracker data itself is only targeted at SID so with the distribution tag the tracker is able to process information in the applicability of an issue for the other distributions these are cross-referenced with a DSA list that we maintain as well and it will automatically change and add the information to the tracker page when it's been processed and the DSA information has been added this information is all manually added so sometimes people will complain that the DSA information is not updated that's just because it has to be added into the file sometimes this data can be slightly delayed here's an example of a Drupal issue that was fixed in 4.5.6.1-1 and you can see the CVE description has some pretty crazy versioning numbers but this one we ended up putting a distribution tag of Sarge on it because this particular issue although you can't see in this description if you look at the longer description and some of the reference URLs only can be triggered with PHP 5 and PHP 5 was not included in Sarge so this issue is also labeled in the tracker is not affected in Sarge more it's mentioned this but we also in the tracking system we don't support non-free it's just too much to track and the embedded code copies is also a problem in the tracker itself determining whether or not a vulnerability is applicable to a package is kind of difficult when there's a lot of different copies of things in there you have a question we can take questions here in the end too go ahead I have a question to the PHP 5 example you said PHP 5 is not in Sarge so the package gets not fixed or if I can go back here I'll show you that it's just noted that this package Drupal is not it's not affected in Sarge but it's also noted right above it that Drupal version 4.5.6.1 was uploaded into SID so this version fixed this issue in SID and then it propagated to testing but it's not fixed in Sarge so if anyone has .deb PHP 5 packages own fault we can't support people installing packages that aren't supported by Debian so if you install a back ported PHP 5 from .deb.org or whatever it's not something that we can track it's just too much information as it is these are particularly problematic I'd call them evil we have a list in our subversion repository of the packages that we know that embed code copies and what those code copies are we don't know all of them because we only end up finding them once we have actual vulnerability that has something like the libxpdf library embedded in it and then we start discovering that that's embedded in a number of different packages and we note this in our file so that next time there's a problem with libxpdf we make sure we check all these various packages this is really a problem and very annoying so it's something that we've maybe thinking about making a release criteria for Lenny is getting rid of embedded code copies there are a lot of issues where embedded code copies are legitimate there are many exceptions and these have to be kind of detailed before that makes any sense I believe we've submitted a bug to the policy documentation about this issue because this is a significant problem for the tracker as well as for maintaining the security packages and doing updates if you don't believe me try finding all of these in the archive and which packages they are it's not very fun and it's not very easy and once you do find them you have to go through every package to determine whether or not it's vulnerable to the issue if any one of these tiny mce javascript library or whatever is affected by vulnerability so all this data when we process it gets turned into this security bug tracker page automatically and this page is processed by this DEB secan software I'm not sure if that's how you pronounce that properly but this is a package that you can install in your system and it will query this tracker data and let you know what packages that are installed on your system are affected by what vulnerabilities that Debbie knows about this is particularly useful if you are wanting to deal with if you have a package on your system that you want to disable when there's a remote code execution or privilege escalation you'll get an email letting you know that we've determined that this issue actually exists it doesn't necessarily mean it's fixed yet but you can look in there and track that fix as it propagates through the archive it's a little noisy at the moment this program tells you all the vulnerabilities for like the Linux kernel if there's a vulnerability in the Linux image it's also in the Linux headers and all these other packages so it gets listed more than once something that we hope to improve over time so it's more useful other tools could also use this data like Nessus scanners or whatever to get more accurate data and reduce some of the false positives that they tend to have you guys as maintainers can use this information to keep updated keep us updated on the security information in your packages letting us know what versions issues were fixed in the release team can use this to see the state of the various suites and the security issues and you can also use this information to get security history of a package the kernel itself I mentioned just earlier is tracked outside of the system at the moment because in woody the number of kernels were so large it was really cumbersome and difficult to process this data in the tracker so we came up with a different method of tracking this it's in the kernel subversion repository and I don't have the internet here so I can't load it up but this page is... actually Micah that's moved we're now sharing an alias project it's called kernel-sec so it's also on the versions of the slides that are on my computer that weren't displaying on the screen have that corrected but yeah so don't try to load this one although if you go here I think there's a link to it essentially this is an effort that is coordinated somewhat with Ubuntu they also track their kernels in the system to a lesser degree but Moritz and Dan fill out each CVE and whether or not it's affecting different kernel versions that we have in the different suites whether it's in old stable, stable testing, unstable whether there's a patch that is available or has been applied to the kernel in the subversion repository with the kernel team or not or if this issue hasn't even been checked at all it's a good overview page to get a good idea of where things are at so here's some pretty statistics everybody likes statistics these are thanks to Stefan here in kind of in general here you can see that in 2005 and 2006 there are 15 to 22,000 issues that were in this database 7,000 more in 2006 there are a bunch of issues in here that don't have CVE entries that we end up adding or requesting at a later date the not for us entries which are the issues that I mentioned earlier that are Microsoft or Cisco or whatever that aren't Debian packages are quite large as well and then also here are the reserved and rejected items, the numbers these all add up to at the bottom how many issues actually affect Debian in these different years so you can see this is increased by a thousand or so in 2005, 2006 oh right, this is incorrectly the header is not correct down there, yeah the third column is actually the 2006 numbers so this is actually a mathematical operation just the way how I determined the data so the first column is the status end of 2005 the second column is the status end of 2006 and the third column is the difference so that's the number of issues in 2006 that we did in 2006 yes the number of issues are actually in the third column there so the number of issues that affect Debian 2006 that we determined were 1154 here total items were 7000 and not for us items were about 10,000 items that's also probably not readable but this was a this is a graph that a user on the mailing list submitted graphing over time the number of security holes in the various distributions the unstable one is kind of this reddish line the blue line is the testing distribution and the green line is stable there's this sharp drop here when etch was released there's something not quite right in the graph as this should be up here more but we didn't have time to crack this before the presentation so the last thing in the I mentioned we had these guiding principles involved in tracking these security issues and the last one the most important one and people tend to have the most problems with is this issue of transparency this has been something that people like to bring up on the mailing list as the security team lacking completely in any transparency whatsoever but this element of the security process is actually incredibly transparent so a lot of people get a little freaked out about the fact that all this information is available and and one of the responses to this concern is that all these issues are already the CVE issues are already publicly known and before they even become a CVE issue they have been published on various security websites approval concept, exploit codes have been published through bug tracker or whatever and there's only a very small amount of information available there's probably only about 1% that are vendor sec or embargoed issues that aren't actually in the tracker itself so all this information already exists so it's not particularly surprising that we're also making it available and one other thing is that if everybody on the internet already knows then Debian shares better know in order because it takes a while for this issue to percolate down to the actual maintainer to fix the issue and get the fix into the archive and then the users to actually install that fix so once these issues have been publicly acknowledged and put into a CVE database it takes a while before you're actually going to get a fix into your system so the sooner that the internet already knows about the sooner that these can be fixed in the archive and on your system and probably the most important thing here is this transparency about transparency is that this is really something that users and developers expect in Debian and what is really deserved as a project as a whole and it really improves the perception of the project to be as transparent as possible so there's some things that we're planning to do and we talked about a little bit earlier maybe one of these things we'll do is add some of this tracker security information to the package tracking system, the PTS so that when you go to look at a package you can see the security history of the package the data in the the tracker can always use integrity checks and improvement and this DebSecan software still is quite noisy as I mentioned so this could be improved so it's more useful to administers of systems so how can you help? Well, this is actually the tracking security issues is kind of a unique scenario in Debian where it's very easy to contribute in very minute ways and it's open on Aliot for people to join and participate, you don't actually have to be a Debian developer to participate in this aspect of the security work in Debian so it's processing individual CVEs can take anywhere from two seconds to determining that that it's a Safari issue or a Microsoft issue or much longer digging into that so you can really spend very little amount of time but have quite large impact on Debian itself so we encourage people to join us either in verifying your packages security data in the tracker or other information in the tracker there's a list of to-do items in the tracker if you're interested in participating you can look through this and this information or ask us to add you to the Aliot project and start tracking this information in more detail we're here and we're interested in showing people how to do this in more detail and there's some documentation as well in this version repository on more detail on how to do this properly so if people are interested we would like to show you more how to do this and get you involved because we have to do this every day if you don't want to if 10 people do one CVE every week that's significantly more than what we're doing already and of course fixing your security bugs responding to the security bugs that we file making sure that you include CVE numbers that we include in these bug reports in your change logs so we can watch these because we watch the changes reports on the mailing list and update the tracker with the information of the fixed version that's in the repository so knowing which version of the package is fixed we need to know that and also working with upstream to isolate affected versions it's not always very easy and very obvious so if you know this already because you have a relationship with upstream letting us know will help us a lot so that's kind of the end of how we're tracking things that kind of blew through that really fast because I thought we might leave a little bit of time open for questions we have 15 minutes so Abba's got a question back there actually at the very beginning Moritz said a lot of packages were basically unable to provide secure support because the team is not helpful upstream is difficult but if that's the case according to our current standards in a C-Bug so it really encouraged you to file such bugs so we can identify such packages and prevent them from going to the next stable version of Debian I sometimes try that this hasn't been very successful in many cases so I can try that again but often the short wish to have that package in the archive is for the maintainer more important than the general archive maintainability at all I think it's actually limited to 10 to 20 packages which are always the same candidates well if there could be some help from the lease team of course it doesn't help you there yeah I was playing through I started doing that how our 50k packages support will be of so I think it's a bit disrelated to the current talk okay the next one is to say there are some cases where we prefer to not to update to the stable system because it's just the issues are too minor basically and I've seen that such issues are being marked in the test and the security tracker as well but for most of our users they just never acknowledge somehow in Debian so how about sending out some we are not going to fix these bugs because they are too minor or something like that well yeah that's that's the idea behind the errata so we have someone who's maintaining properly maintained errata so that this person can be cc'd providing sending out advisories for such minor issues in my opinion a bit yeah this would be on the one hand side too much work and it would also take away attention from the really important issues that we could make it somebody wants per month or so but we can discuss it afterwards yeah it might be that it might be a good idea if someone maintains a errata for Debian sabreys that the delta between the last month is sent out via email every month actually we already have errata they are part of the release notes so we could try to make better use of them also for security yeah but right now they are mostly limited to installation bugs like s309 cannot be installed with a floppy disk we have two different things, we have an installation manual with installation errata, we have a release note with the release errata like for example for sarge for something if you're upgrading the sent mail we have some strange issues there and I think that's the edge release errata I think is the right place for such things yeah it might be but I think it still needs to be properly maintained so we need a volunteer basically definitely last thing I want to say is you said about the points at least they don't happen so often actually these days proposed updates is already usable for the normal people because package are reviewed before they are accepted into proposed updates so that might be even useful now I'm not sure how recommended it is to errata well if you recommend to put it in the app source if the stable release manager just takes it into account that every package uploaded to proposed updates may only be crafted very carefully so that they don't break things that's certainly an option but on the other side we also had a problem with on sent mail update where the version I think in stable updates had a slightly minor I don't remember the exact details but it actually led to the fact that some people who had already pulled in the version from proposed updates had a non-installable package so if this can be technically out sorted out technically I guess that's a fix as well yeah okay questions you mentioned briefly you had some collaboration with Ubuntu on kernel security issues I wonder if you'd had any collaborations with other distributions because a lot of the triage has got to be duplicated efforts albeit that most distributions have far fewer packages than Debian well most of these is handed for vendor sec because all the major security teams are present there especially with regard to the kernel because the kernel does not have a regular security management upstream they have a security management system they have a log contact list but for example they do not provide advisories to developers so vendor sec is actually the place where most of these kernel issues are analyzed and patches are discussed and the stages of patches is shared so that actually takes place on vendor sec some of the tracker information it would be nice to collaborate with other vendors but it does not affect versions because a lot of the stuff is very opaque we've talked at some points about maybe having a shared repository to keep patches for isolated security fixes and use that between different vendors but it's something that requires coordination with the different vendors and so far that hasn't manifested in any way it's definitely something that's useful with the standard tracker in particular we were talking Tuesday about maybe being able to get rid of our participation in that and going back and just using the standard tracker the reason we have the kernel trackers is because there's an outlandish number of different versions of the kernel in woody and then SARS we got down to two but there's still various architectures now that we're down to one or two packages that built for all the architectures we can probably go back and use the regular tracker but if we got to the point where we had other distributors fedora even commercial distributors it would be great to use a tracker and I'd be happy to maintain both copies of the data because the kernel tracker is a very simple way of looking at a set of bugs and also being able to see all the notes different vendors have made so if you know any other organizations that are interested in using this just contact me I'm happy to add them to Alioth Group I think once Lenny's release you probably can completely switch back to this tracker and get rid of the kernel tracker I think we have to ditch woody kernels before that it can be really good I ditched woody a long time ago exactly so if you start if deviant starts providing enterprise level support like over 5-7 years does that mean we would start supporting upgrades between older than one release ago? no okay I guess the practical implications for that are not possible because deviant is just too large for that so this would actually boil down to a scenario where it would only be limited to a certain subset of the whole archive anywhere because you can't do that for 50,000 packages Rated Enterprise Linux consists of significantly less packages than deviant for example if you compare Fedora with Rated Enterprise Linux there's also a huge gap in package size so if that ever comes to reality one would need to define an upgrade path from one of these enterprise release cycles to the next one but I guess that's feasible as well yes, my name is Peter Reynoldsen and I'm one of the person that was involved in the initiative to start the security testing team I must say that I'm really impressed how far you managed to get in these last few years and I would really like to thank you all in the team for doing a great job in bringing the attention of the project to security problems and making sure that we actually make a good job at fixing them so thank you very much you're mostly reacting on existing reports, security reports do you also have tools to proactively search for holes in the source code and something like the Coverti reports you might have seen that I've skipped some slides because we're already at 70 slides and I skipped a few but I'll put them online you can have a look at them I actually wrote my thesis about static source code vulnerability analysis from that I can tell that most tools suck there are a few commercial bonds which are quite successful Coverti pre-scan has a public setup which scans I think a couple hundred public important important free software projects but one needs also to say that from all these scans I think only a couple really severe security problems were found these tools mostly find correctness box or they find rather ancient security classes which are not really not really relevant anymore like it's pretty easy to detect format string vulnerabilities but none of these tools addresses integer overflows right now and they only they are only able to detect a very small subset of those so for example they found a local route exploit in X11 which due to a typo a wrong function was called so it was ended two conditions were ended and thus the function which made an authorization check always return true because the other one the other function was never called and I think that's the only really security problem that has been found by coverage in the last years it's mostly robustness and cleanness but all these tools are not really suitable and do not replace proper if you're interested in improving the security of one of your packages it's much more fruitful to look into fuzzing especially if you're a maintainer of some application which passes you could look into writing your own file for your media format or if it's a common media format there might already be a fuzzer available that's not to say that these scanners would not be useful I think that if someone wanted to or a company offered to do this on the entire archive I'm sure it would be useful if someone were to go through the issues that they uncover they're actually valid I think there would be quite a number of false positives involved and there is this Debian audit team that exists that might be more appropriate for that sort of work so I think it is information that could be useful but it's probably we'll boil down to a very small number of actual issues but I also have questions yeah so how many people right now are in the testing security team in the stable security team what would be the ideal number of people in these teams and why not merge them yeah in the testing security team there's maybe well you can see if you go to the alias page there's probably 20 people listed although maybe only four or five people are actually active tracking bugs Florian Weimer, Stefan myself Mortz and Sean is doing some stuff there are a few people that come and go at various times but Mortz had a slide with the number of people that were in the stable security team I think there was five or six depends on what they yeah it depends I guess on Mike and there have been discussions about merging the two into just having a security team although it makes sense to focus on stable or testing because each one is quite a lot of work in and of itself so that is definitely something that has been discussed in determining how to do that in a sense is kind of difficult specifically with relation to the embargoed vendor sec issues and whether or not everybody should have access to that information is kind of disputed specifically like if you're not a Debian developer and you can be added to the tracker ALIATH project then it starts to get a little questionable whether or not you should be having some of this information or whether or not it's allowable on the vendor sec list but this is part of the reason why AJ has made some of these changes to create the un embargoed queue so that public issues or CVE known issues can be done through that queue instead of through just an isolated queue that only people on the security stable security team can access and it's likely that Wilings tend to set with the security team another member in the foreseeable future on the other side this requires quite serious time commitment and also a certain degree of experience because if you break X11 or Apache or other demons during during upgrades you need to be quite diligent so in order to make this really happen one needs to be quite careful but I think we'll extend by one member in the foreseeable future because the aircraft is growing all the time so it's quite necessary to cope with that but I think we'll run out of time there should be an last question we can deal with it otherwise I would like to thank you for your attention