 Okay, hello everyone. Glad to welcome to this last session for today and this is actually a panel and I wonder if we have a lot of background noise Maybe we can start trying muting The other people who are not talking Actually a lot of background noise. Okay, so let's let's try to get started though So we are let me try to introduce the participants or more like participants will introduce themselves So we can make a small round to virtual round table So we can start When we order how the participants are listed here in the panel, so I will start from Alexander Alexander, please tell a couple of words about yourself Yes. Hello. Do you hear me? Yeah, do you hear me? Okay Yes, I'm Alexander Popov. I live in Moscow, Russia. I'm a Colonel developer since 2012 I started from Colonel development for embedded systems and three years ago I fulfilled my dream and focused on Linux kernel security. I Have a lot of fun with vulnerability discovery exploited exploitation of the vulnerabilities and kernel self-protection in positive technologies and Thank you very much for bringing me into this discussion. It is amazing that I can participate in LSS North America Thank you, Alexander So I guess Allison would be next. I am not nearly as cool Hi, I'm Allison after a decade of running software mostly in user land I have entered grad school and I am now a PhD student I'm advised by Dr. Woo song fang and my areas of interest include fuzzing automation and Security architecture so my OS experience these days is much more centered around Cherry BSD and cherry capability hardware as well as working with fuzzing tools and trying to integrate those into developer workloads. I I'm a little bit of the wild card on this panel. I'm afraid Clive discussion is good. Thank you. Allison. So I guess next would be a case Hi, yep, my name is case cook I'm employed by Google and I have been focusing on Upstream Linux kernel security and mitigation work. I sort of heard the cats that Make up the kernel self-protection project Yeah Thank you, and Mimi next Hi, I Worked for IBM TJ Watson the research center And I I work on the integrity subsystem here in the Linux kernel I've been doing that for over 10 years now. Thank you Mimi. So I guess I can go last So I'm Elena. I'm Intel and I have been doing various aspects of platform security and mostly Linux security I think for past like 12 years and I have been touching very various parts of Systems and recently also being kernel also have a background in crypto and recently I have kind of partly getting back to also to my crypto part of expertise though But let's now move to the panel itself So the title of the panel is everyone can see is that well, what's lacking in Linux security and what we should be doing about this And we just kick off this discussion. So we have asked our panel members To prepare some I don't know a couple of minutes or whatever we want to tell initially on the topic So they're gonna have an hour around around our virtual table where each panel member maybe brings like one aspect if you consider is important and Then at the same time the audience is encouraged to ask questions And you can do it using this Q&A interface and you can ask questions to particular panel members So you can just indicate where and the writing that will Okay, so to me me or and so on or you can just ask another question to a panel So all panel members can see the questions so we can They can decide to answer it later on this free discussion or maybe answer it during the time there Or if they're already talking and and let's see we would really love to make it like, you know live discussions about too much Kind of interaction or intervention from at least my side, but let's see how it goes So I'm never done it before visual panel. So we'll see So let's start again now around round table. So Alexander if you could maybe bring a point for things You're thinking on the matter first Yeah, thanks So 20 years ago Bruce now said that security is a process not a product it is true and it is pity because There is no easy solution for Linux security Challenge, but it is great because we have plenty of interesting work and it is enough for everyone I think Linux kernel security needs more people and organizations involved in it There are many sectors in this area and first I want to list security aspects in Linux kernel development process first Linux kernel page attestation Konstantin Rebitsyv the administrator of kernel.org works on that it is very important next kernel testing and especially continuous kernel fuzzing fuzzing is a very powerful method of vulnerability discovery and We all really appreciate the work by Dmitry Vyukov and osmosis quality team next static analysis and variant analysis they are becoming more and more popular as well and Integrating them into the kernel development life cycle could definitely improve the code quality, but That requires more resources and more people as well since these technologies Provide bug reports that should be handled by kernel developers and maintainers Next I think about Responsible disclosure procedure for kernel security issues. Personally, I think it works quite well. I used it several times and An important part of it is handled by Linux distros mailing list, which is organized by a solar designer alexander pislak next Applying the security fixes for the stable trees Stable kernel maintainers do enormous work, but they need more resources and regression tests As jar security people often highlight and I'm sure Brad will mention that again in his talk tomorrow As possible ideas from me. I see deeper Linux kernel vulnerability tracking and Security bug bounty in some form. I don't know. Maybe that could be valuable But that were the security related procedures of kernel development and now I want to look briefly at kernel security technologies There are a lot of bugs in the kernel code as we know and we can't magically fix them all So so we have the alternative ways First kernel has very good detection mechanisms sanitizers and various checks which are being constantly improved They give really excellent results in combination with continuous kernel fuzzing and There are also the kernel self-protection mechanisms that mitigate vulnerability classes or make exploitation methods harder and Their pioneers in this area are people from jar security and box team and case cook Leads the kernel self-protection project we where kernel developers work on such technologies for the mainline kernel And it is extremely important and that area is also very exciting I think because here in in this area you can combine the offensive and defensive security research You can create proof-of-concept exploits for kernel vulnerabilities develop the fixing patches do responsible disclosure invent kernel self-protection techniques that could help against your own exploits and Young horn from project zero does this kind of research, which is really inspiring for me personally and Usually this kernel self-protection features don't come for free. They bring performance penalty They bring various additional requirements for kernel developers and sometimes make the kernel debugging harder So I think it is very important to choose the kernel security features depending on the threat model of The information system which you build So I created the Linux kernel defense map that can help in this task This map shows their relationships between vulnerability classes exploitation techniques bug detection bug detection mechanisms and Kernel defense technologies So when you know when you develop the threat model of your information system, which is based on Linux kernel you can choose the kernel hardening features cut the kernel attack surface and Finally choose the proper security policy for user space and then force it using the LSM Linux security modules and I also Created the tool called K config hardened check that tool is used to check the Linux kernel K config against their hardening preferences and I'm very glad that some Linux distributions already use it In their kernel development and I even know that some big companies use quietly without credit This tool for commercial security audits, which is funny And now let me return to the beginning of this small talk. So Linux kernel security needs people and organizations and people Who really have this hacker spirit who really love this work? otherwise, if you work on things that you don't love you can't get the Your personal best results and organizations Linux kernel security needs organizations that really understand its importance and are Ready to invest real resources not rhetorically but in reality to Linux kernel security So that is my point of view. Thank you very much Thank you, Alexander So where you have brought many interesting points and I'm hoping we'll touch on them later But I would like to move to our next panel member to give an opportunity for her to also introduce points She can she considers important. So Allison you will be next there we go so Alexander is much more versed in the kernel what I'm going to push back against is that the definition of Linux security should not end at the user land boundary I think this is a myth one of several that I think that community has indulged in for some time and what's really lacking is Facing some of those and making some peace and agreeing about what our trade-off model is like there are different threat models out there What's the primary one? So I think the biggest fairy tale we have in open source in general But in Linux particular is the myth of the perfectly rational perfectly for user who's read all the security documentation Notice how all the features work has outlined a very detailed security model and threat model and can make all the appropriate decisions The vast majority of people just install their distro and move on with their day Red teams think this is great Features move between distros at disjoint pace moving stuff up spring tapes a while This gives them lots of windows that get stuff done But I think that's a thing that we tend to do as a community and as an open source in general likes to do this too Oh, that's dangerous We'll let the user make the decision about whether or not to enable that feature And the reality is most of them don't a lot of the security features sit in the way spin off by default And they're not really doing as much good The other thing I often see is that this sort of myth of Performance you can have a security feature as long as it doesn't impact performance under all arbitrary workloads ever We've already got what security features meet those bars We're not really going to advance if we keep holding to that I think at least different distributions need to decide our workload model is this We're going to test perf against that We're going to test our security features against those models and make those trade-offs Knowing that this is the workload. We are optimizing for we've got a lot of distribute distributions We don't have to be all things to all people, but we seem determined to do that It's kind of upsetting to all see security features almost always stop when it comes to going upstream Another thing that's maybe less interesting to this group would be I worry sometimes when I see users who are like a particularly Students who say oh, I don't have to worry about security. I use Linux now I'm gonna admit that when I was a young and feckless undergrad I totally enjoyed watching the windows people patch their software and going oh Updates that's nice. You have a virus. Hmm. Maybe you shouldn't use windows But now that isn't really the case like Linux is just as much a target You sometimes even more so and I think a lot of the user base still thinks that Linux by default protects them from security Evils and maybe that's a little bit of our fault. Maybe it's not It's a food for thought And with that I'm gonna hand it back to Elena Thank you, Allison so Very interesting like Alexander and Allison points were really like on on very different angles and very different sides of it So it's I think it's it's really nice to have so many different angles. So I guess next would be case Sure, I think a lot has already been covered I think Really looking at sort of the the history of where we see a lot of Flaws I think we need to start thinking more seriously about memory safe languages that don't have garbage collection and I think the An area that I got touched on here, but I think is really important is testing like we cannot Evaluate whether anything is fixed if we don't have a test we shouldn't accept Colonel code that doesn't also have a test This is sort of a fundamental problem with with how things are going in in the upstream colonel that's gonna require some change in in how the The social expectations and how things should should be organized in upstream Getting a push for actual testing I think is going to be really important because that gives us the parameters for avoiding bugs in the first place It lets us, you know bounce the fuzzer off of actual tests and things like that I think that that area is probably the Biggest spot that I'd like to see changes in the colonel community is actually writing the tests for the stuff that we work on Okay, so case it is a tool what I Mean are you done or oh? Yeah, that was it Yes, really hard we can figure out when someone's done talking in this in this medium It's not just you and I figure we can we can get into the individual I mean a lot is getting brought up. I figured we'd go through the intros first No, I just wanted to make make sure I'm not stepping over Okay, so the last one So I'm gonna be speaking about one Want my area of integrity basically and we're gonna I'll give you a short story to begin with My grandfather was a diamond broker. He didn't cut diamonds. He didn't even own the diamonds He basically sold them in today's terminology. He was the middle man Making the connection between the diamond owner and the buyer potential buyers He would receive the diamonds from the owner and show them to the potential buyers But every aspect of the transaction was based on a handshake There was no contract Just a handshake a person's reputation was built on that handshake Taking advantage of a deal obviously could happen Once twice a couple of times, but it was limited people learned who they could and couldn't trust unfortunately Handshaking in our time is kind of we're not Doing too much hand shaking and maybe it will become a metaphor in the future for trust but in But going forward We The In that in my grandfather's environment people knew each other Or they knew somebody who knew somebody who knew the other person and However with each link the degree of trust varied At the same time we We trust different people for different things I trust my family and friends to celebrate in good times and be supportive in bad times. I trust the dentist I trust my doctor, but I trust them all for different things Certification can provide some a level of trust, but it varies So and if we take this metaphor from how people behave To our world of computers We're living at the time where where we need to have trust We need to trust the manufacturers. We need to trust the suppliers We need to trust the firmware the OS the applications to do what is expected and nothing else And we know how well that is working We need the equivalent of a handshake to be identified To be able to identify the provenance of files on our systems before we can actually trust them So for years now, I've been asking everyone to sign and distribute file signatures with the file data I'm finally now hearing murmurings of it actually happening by a number of different companies Will this be the year that we actually finally see it? Will it be the year that the good values are published? so But file provenance knowing where the files are coming from is just the beginning Afterwards, even once we have all the file sign and we know what's going where they're coming from We need to be able to differentiate Be at different levels. We need to be just like we differentiate between My doctor and any doctor or my dentist and other dentists We need to be able to differentiate Who's doing the signing and and how we're going to do it? Is it? Is it going to be just one signature or is it multiple signatures? Is it at the package level? So that different systems can be signed can be installed on different systems and the and the Distro or whoever else is signing all the packages or is it going to be allowing the user to make that decision the user the customer So on and I hope we get to the point where the customer the end user will control what can run where But the packages themselves need to be signed Thank you. Thank you Mimi Okay, so I think we're done with our first initial round table Things so we have gotten actually a lot of for the a lot of questions for the interface I don't know how we want to make it just by going one by one and wherever wants from the panel because most of them are generic Questions to the panel to all panel members So I can just read a question and then whoever wants to comment on that can just just start commenting So they're not too many. We should not speak all over The first question was to the panel. So what do you see is the main reason for very large number of bugs in a return or release? And what should be done change to reduce the number of bugs introduced or maybe we should start from ourselves or but In a way with like the code is is how the code is written and that's it's it's one of the The big problem, but once the comment on this so this was generic question to the panel I think this gets into into testing like we can't we can't have any certainty that things are working without the tests Fuzzing has been a great way to effectively create tests blindly And and to find the boundaries of things that went untested, but when we have so much that is untested Fuzzing is pretty important, but it'd be better to have direct tests That's that I think is my my main answer to that Go ahead Alexander So I would like to add When I use fuzzing for searching vulnerabilities and think of it. I always see that it is missing the Actual check whether the kernel handled some input correctly or not. So when we fuzz the Kernel we check that it didn't crash. So there is no some memory corruption or Bad behavior, but we don't Check whether the kernel handled this particular system call correctly as we expected and Yes, I agree with case So this part is missing and maybe there are a lot of logical bugs in in this improper handling of The input from the kernel anyone else still wants to comment on this Well, throw in one last one Sure So the fuzzing so the research community really likes fuzzing the kernel because it's considered one of the hardest targets to hit It's a popular target for any sort of testing or Verification tool one because people who write the grants really like to hear about making half the world more secure But also because it's considered very difficult, right as kernels are one of the most complex and difficult things thoroughly test and thoroughly verify So the fact that we throw large numbers of lines of code in there Every release means that we're sort of exponentially increasing the number of possible paths and places there could be bugs Right. Maybe part of the answer is to make the releases smaller We might end up with fewer bugs more testing and more thorough testing is definitely part of it And there are a lot of places where that can be improved But it's a hard part. It's a hard nut to crack That is all but I guess this slowing down releases. So making like releases smaller and stuff is it It's like it's basically saying that we are going to raise the quality assurance and releases I guess goes back to testing. So always we can't stop the features from going in but there are a lot of features coming out That's also true And I do support every one point on testing because It's actually quite I remember back in time when what was I working on? I think it was the rev count work and I needed to modify rev counters to use this new interface in different parts of the kernel and Many subsystems when I was proposing the changes that would go back to me and say that oh you should test it before Obviously like you should test the change before you're proposing and It was actually for some places. It was very hard for me to find because I didn't want to write my own test It's from scratch. I didn't have any bandwidth to do this whole kinds of different subsystems I didn't know especially and for some places it was very hard to find tests around like I mean ready tests, which I would like Go and start and and in that light. I think this point about testing is It's really kind of answers in my opinion also the cemetery is the question to the panel that we had But maybe let's move to the next set of questions So I'm gonna write or read two next questions together because they're kind of continuing one point So the next question is a big problem study code analysis I think this is reflects and what Alexander was talking in the beginning when he was talking about what we need to do with study code analysis On the colonel so the question is the big problem study code analysis is a huge number of false positives that you generated Do you know any meta different of to reduce this and also do you think of the study code analysis? Which are built into JCC version 10 and plan Alexander do you want to comment on that or anyone else? Sometime ago, I used a different static analysis tools for for applying to Linux kernel and I see that The false positive rate depends on how deep the tool understands the specifics of the kernel and That depends So generic static analysis will not work for the kernel because it is too specific and but at the same time Writing the custom rules against some particular back button back patterns For situations bad situations in the kernel, which you as a developer of the kernel understand that could be Could work better and have less false positives and more over there are several tools which can be integrated into the kernel build process and it is some kind of hybrid method of combining the static analysis and binary analysis some sometimes it uses the The tracing output and you can get the missing information from other types of analysis and bring it into static analysis and find really interesting results. So I think I really like this This phrase let computers do their job. So Big kernel is really big and analyzing it Manually is impossible. So let's write some really really nice rules detecting the bugs and Give the computer that the chance to find this box. Thank you, Alexander Anyone else wants to comment in this? If not when we can move to the next question next question is officially to Alexander But I think it's actually interesting enough question with I think everyone could kind of answer that from their own point of view So the question is that Alexander are you mentioned that we need more people working on security? If you would have five additional people What you would you assign them to and how would it improve security? I think it's like really questions for everyone So what people think like where these five additional people could be used I guess case would assign them all to test Yeah So what are our opinions? I think case is a best person to answer this question He has a very big to-do list I think case will just I mean, I think that's the right place But I think I think actually You know hiring five people to do testing doesn't actually get the change that we want because we want a social change You want the person who's writing the future to write this test and to really understand how to write this So five people won't scale We need everybody to really understand that testing needs to be part of the development process Maybe those five people need to be hired to wander around and harass people into writing tests or to help them see the examples I'm not really sure But I don't know. It's it's a it needs to be a cultural shift And I think doing that by example is probably the most powerful way to do it. It's always thank you Anyone else has ideas for five additional persons Could I have one of them to run a bug bounty program like that can be a full-time job And I would like I Would like someone to help with the Scripts and interpreters to help the user space. We now have There was some work done on Python But Python isn't the only interpreter to take to pass down the may oh may exact and we need more people Helping on the interpreters Back to you Okay, okay, so looks like the fast people won't be very far, but um, I Don't know if I think any amount would be great to get additional people So I think one one person which I can think would Where one person's time could be spent into is that I remember it was to think last year linux security summit Europe but we had a talk by Month page maintainer and he was complaining There's no one who is actually properly maintaining Man pages and making sure that the description actually matches the what's happening Personal behavior and that he showed like could create many many security problems I think that would be one person you can also spend some circles list from my side So, okay, let's go try to read the next question Well, I have an idea about it. So In some conversation I heard a really nice idea So we can find the proof of concepts public proof of concepts exploits for linux kernel vulnerabilities and make cscolar fuzzer Find the bugs which Were exploited by by this proof of concept exploits. So I mean teaching the cscolar how to Hit the particular bug which were security was security relevant Currently cscolar during the fuzzing finds all the bugs and only little part of them are security relevant and maybe Making the fuzzing process focused on security Bugs would be really valuable So five people in this task and Dmitry Yukov would be happy Yeah, I guess we're gonna talk maybe tomorrow tomorrow We're gonna actually have the second session also a panel session at the end of the day And we will have actually Dmitry Yukov Participating in a panel session so we can actually he might be able to comment at that time. So So they really continue a discussion there. So going forward to actually I can see that there are a lot of questions coming Let's see if we're able to get down to the list In the time we have allocated. So next question is also Are any of you applying machine learning techniques to study? Just that the code analysis and fuzzing so basically Machine learning for study code analysis and fuzzing. So anyone wants to comment? That seems Taylor made the academic researcher So there's a lot of research in that space ongoing in the academic circles But ML is not it's very hot right now But it's not the hammer for every nail and so some of the things they are finding this ML is actually a bad fit for some of this But there is sort of active engagement. I don't know that any of it's really ready for prod But there is active work going on in that space Alex Yep, I wanted to say that yesterday was an excellent talk by Sasha Levin and he mentioned that he uses some neural network for For searching the patches which are relevant for stable and that helps to People who review patches for stable and Filters the patches. I guess that's a good pointer. I think all the talks are recorded. So People I want to see the talk which has passed I mean both relating security summit of OSS so you can go and watch your talk So like the talk Alexander recommended just now So does anyone wants to comment still on machine learning? What should we move? going once Okay So the next questions actually set of questions about the testing so very basically asking what testing challenges are they seeing in security space and What should they really improve and change in the testing? So we've already talked about it. I don't know if case for example you want to add anything there or Sorry, I keep dropping out. I Think I think an interesting piece on the testing is It would be nice if we can sort of get Subsystem maintainers to you know, we're not gonna get it There's no such thing as really top-down in Linux criminal development So I'm pretty sure if we ask Linus nicely if we can no longer accept code unless there is a test for it He's going to say no So usually what we need to do is get subsystem maintainers to say hey look You need to have a test for this before I'll accept that code and as that As as other subsystem maintainers see that that actually improves things In those subsystems that becomes something that more and more people want to see I think that's probably the only way to actually make change in the kernel ecosystem itself Yeah, so basically start from maintainers and I think when it comes to maintainers We have very different policies. So some are much stricter about testing, but some are you can say more relaxed. So So, yeah, this very is greatly they don't have any kind of uniform to manage understanding or even alignment about the matter Anyone else still wants to comment? Go go ahead Thank you so one of the major changes that i'm having is with That with the tpm with hardware basically we're how do you How can we get it the tpm changes? And we need to be able to do testing with all different packages and all different releases But that's software, but how do we test with different versions of hardware? And how can we automate this testing in the It would be nice to be able to do this, you know the same way that we do package testing with just spin up another Vm or another container or something it would be nice if we could somehow tie this into actual hardware I think that's actually a big big point In regards to we don't have even like Emulator we don't have anything which would be reliable reliable for this matter. So It's a hard problem It's a hard problem. Yeah, I agree right now I have a number of laptops on my kitchen island to help with Actually, I had a problem with testing against tpm last week just so that I also run into a case that I don't actually have and the setup I've got I've got to have very weird setup for a different reason, but when I realized it Okay, I've got this very weird setup where I'm able to test what I need to test But I also need to have tpm working and it actually wasn't working for that case I mean it was partly emulated environment, but yeah, it's I see what I mean Okay, so should we move then next to next question So the next question is actually for Mimi because it relates to integrity So Mimi, what do you want to see to improve in file provenance? pgp encryption blockchain or another similar immutable ledger or open question My favorite question. I want I want them I want the Files to be signed by the person who's who creates the files the user packages. I want um, I want the signature to be included in the In the rpms in Debian in whatever package maintainer you have We already have methods for verifying those signatures We just need their inclusion for so that the file data and the file metadata come together And that it's not something that you do post install But that they're delivered And packaged together Thank you I hope so I'm not so This is I guess this is the downside of ritual panel is that even if like you suppose now answer a question If there is a follow-up question, it's going to show up somewhere at the end of this many pages And and there won't be a way to actually map the follow-up to this So let's see if we get to follow-up questions So but if I start to jump I'm just to operate with like if I'm jumping between the pages We will miss some questions. So I'm trying to take them consequently So, yeah, so really last question on this side is It's again, I'm carrying on testing. So there is a comment from person here saying that I'm related in new current developer And I was actually somewhat shocked at the lack of testing patches I understand that there is a lot of testing which happens on the maintainer side Is it something we should be more public? So basically saying that Should the maintainer share some wisdom with us and kind of explain how they're testing with patch itself And we have a couple of maintainers and this one also Maybe we can found comment in that There are a lot of testing There's a lot there's a lot of different architectures Um for testing the kernel, um, you have LTP you have Um, you have the XFS tests you have, um You have the The kernel self tests you have Um, the night knew I was just someone just recently told me about k-unit tests But it would be really nice to have a set of security tests So that we could see how they can how they will work together And that's one thing that's, um missing And we need more of the tests. So if you're new and you're looking to help write tests, um There are many pack your many, um packages that are already exist that need help Um, and I can point you in the right direction Thank you Mimi Does anyone else case you do some testing? Certainly yourself Yeah, not suggesting Well, I mean in the security so for security mitigations, I've tended to where possible Write tests for this the lkdtm Module tests for all these things that crash or kernel or kill threads and do other stuff My intent has always been to test Any feature that gets added does any mitigation feature that gets added. So From that perspective, I think that there is a fair bit of coverage on that And it tries to test classes of problems That should get caught by by various mitigations. Um, so lkdtm does have a bunch of those tests already I think we need I think we need to be a lot bigger than that. Thank you case Does anyone and yep Yeah, I would like to I I think that, uh Cisbot reproducers which cause crushes in the kernel are really A really brilliant resource and it would be nice if they if those reproducers somehow Can be integrated in the kernel continuous integration Some kernel continuous integration from framework because Cisbot has a lot of information about about the kernel actually it has the version which crushed it has the reproducer Which crushed the kernel it even Does bisection? It finds the patch From which the kernel Started to behave bad So it has a lot of information and this information could be integrated into, um, for example the process of Preparing stable trees or something like that. So it is it is clearly the regression tests which with against the bugs which were fixed and maybe running them continuously for Let's say every kernel release Could could find Really nice bugs and Solve a lot of headache. I think Thank you, Alexander Anyone else wants the comment? Or should we I don't know actually how much time do we have is it like still 14 minutes? Do we have one for lower for panel? I think it goes to 35 Okay, yeah Okay, so let's continue There are more questions on the other pages Yeah, yeah, I'm saying I'm not down even to the first page. So So we have to speed up or but the questions are visible So people can and actually the question person who asks your question is also visible So if anyone wants to if we don't get to answer all the questions If you want to answer to people directly after I think that will be great So so the next question we're going to take about your regression box So there is a question regression box. I have disappointment. How does this still happen to kernel development? Do tests get dropped to what's going on? So it's actually an interesting question because we have high regression box and we have had bugs which I think showed up After a year isn't showing what it's like. It's was the regression introduced a couple of years ago So anyone wants to comment on this appointment case Sorry, I keep dropping Yeah, I mean we're just on a testing Yeah, I think here the point maybe if I try to kind of speculate in the point that the Person who was asking questions trying to bring is that do the case tests get dropped? So is it really like do we only get these cases when we don't have tests? Or is it like Is it possible that you had a test and when we caused your regression which was Somehow, I mean when it gets unnoticed because regression can be not just functional. Well, um I mean, I haven't seen a lot of cases where we regress a major test. Um, I You know, we've rewritten entire the entire exec path for example, and we only have very minimal tests for the exec path We just sort of ad hoc Examine it and make sure it's okay. And then we sort of kick it out and hope no one Notices any breakage. That's sort of been the the style that testing happened with which is here's the code now Someone else gets to test it But it would be nicer to have that but a kernel ci.org is is sort of becoming the dashboard for our things building our things running And getting more tests in the kernel self test tree. I think we'll get us closer to being able to notice when things break because You know, you'll get a patch You know a series gets thrown out on the list and it says it's broken So, okay. Well, I won't pull that Oh, that's it Anyone else wants to comment? if not when we can go to the next question so um Actually, we have a lot of questions in the testing. So and we have discussed testing already quite quite extensively so maybe we can take a question on the Something else Does anyone from panel see an interesting question? I mean not out of the order if they want to take Or is the question like on memory leaks that memory leaks makes our code vulnerable any suggestion tool to catch memory leaks in large code bases Does anyone wants to comment in the memory leaks? there was a kmem leak tool Config option of the kernel, but I'm not sure whether it is alive right now case. Do you know? I haven't seen anything from out lately And I guess memory leaks is considered like compared to some other vulnerabilities Probably not as high as when people throw theirs. It's not like a first. It is not like use of the service So it's not probably the biggest problem which people trying to to fix first so There is a big problem with memory leaks um When it is when you can trigger it trigger it remotely it becomes a remote denial of service, which is A very interesting bug. So there were some cases when Memory leak war was reached in network subsystem was reached remotely and That was That had quite high CVSS score So how do people think so? Is there any questions you particularly want to bring or should we just keep going for the questions? I just fear that if there are some topics you want to Discuss here. So we want to get to that down to the list because I have very little time to Left So does anyone wants to bring some still one of the questions from the endless long list of questions So if you're not going to reach There's one that just popped up that might be Interesting This might also be throwing a stink bomb in the room. Sorry Sean Thomas writes who is looking out for bad actors within the community? And I think there was a talk yesterday discussing uh Because wolf gangs how many bad actors you might need in a community of open source to destabilize it Or to a road trust or to make it non-functional. So Tying back to his presentation yesterday. I think that's an interesting question And one that isn't about testing And uh, I would like to add that open source Has this supply chain problem so, uh, usually software uses some open source tools libraries and so on which can be Out of date or sometimes even big door. So it is very important to know which components Do your software contain so The patch attestation Which I mentioned earlier It is very important If if we speak about the secure development of some open source project and also some binary signing or If you use The software which loads Some other components is important as well. And uh, as for patch attestation. I recommend checking the blog of Konstantin Rebbetsov He is the administrative administrator of kernel the torque and he is working hard on improving the development process of Linux kernel community. Thank you, Alexander Does anyone wants to comment in this too? To remove the next the next question which looks like we still might have time um Let's see. There's really many and most of them are actually about testing There's one which is a bit of a user space sites We have been covering a lot on the kernel. So there is a question Is there is a lots of user space applications that interact with the web? For example chromium and male demons Do those user space applications need any crown security features that kernel lacks at the moment? So hopefully because I think we Recently our focus of people working on Linux security coming to us switched greatly to the kernel So have we forgotten user space at all? Or what what do people think so what what does user space need what we forgot to give? Well, do we actually even think about what various new kernel features or syscalls and whatnot what the impact on user space is? I mean, I don't see a lot of people talking about hey, I want this thing in the kernel for these reasons I don't see a lot of discussion about why Why or how it impacts user land? Which is why I think that we have this myth about Linux ending at the user land boundary and what happens beyond that is somebody else's problem not so Yeah, I guess I was Second word by saying what I think there's a problem and then and I have faced it actually myself maybe some months ago when I was discussing this Kind of memory production interface So I think some kernel people they they think from the kernel point of view and when you talk about the interface addition or something like that They think how they see how it's going to be implemented in a kernel They see like what it takes the complexity and things and kind of they see that part But when how it's going to be used from user space and would it actually that interface makes sense from user space? That's not always in a picture. So what's the and use case? and and and and like because that can affect also the interface design and Decision if are we going with this interface that way much more than just like internal kernel So that that's I would fully second to say that I think the lack of theme to the end use case Which is actually going to be used by the user space like how is that interface for example is going to be used Is it actually making sense to be this way from the user point of view? Because you can design very nice interface from the kernel point of view Which would be clean and implemented behind correctly and so on and so forth But user space won't use it because it would be just I don't know too difficult to clumsy Not clear too much thinking and we have examples. I think like when I analyzed the Libraries was it I think I analyzed some of the common crypto libraries and things I was trying to analyze How many of them are using for example features like mlog or mad advice? um Features which try to limit like the memory. So for example, they allocate buffers Like we have open ssl and other crypto libraries which allocate buffers for the keys and so on So do they actually use this? This with kernel interfaces to to block this memory and so on and I have I have found almost like no usages so I found a couple of usages like I think open ssl was using it for the Where secure heap functionalities of you enable secure heap when it uses interfaces always no And it's an example of interface which currently exists in a kernel and from kernel point of view It's working. So I mean you can probably even test it I don't know if you're a test, but I'm sure I've tested it it works But uh, it's not being used much and the question is why and maybe because again user space wasn't consulted Yeah, if I were to describe a security feature Oh Sorry, it would be it making it hard for people to misuse them And more guardrails essentially But anyway, sorry Alexander Yeah, I would like to add about user space security The main thing I think I think the main Point about Sorry just to stop but we have I think less than one minute left So if you could just make a quick point because I'll have to close the panel. Yeah Yeah, so uh, the main point about user space security The main thing the kernel should do is isolation between user space processes So when the isolation breaks or not enough, uh, it is really bad So the main thing that kernel should do is to isolate user space processes properly Thank you, Alexander. So I'm afraid we're out of time. So thank you everyone for our attendee to panel the participants and of course the panel members so we can clap to the panel members virtually and I guess we will be seeing all of you tomorrow in our second day of linux security summit And at the end of the day we'll have another panel on the same topic So you can keep your questions also prepared Thank you everyone Thank you very much. Thank you