 Welcome to the Q&A. We have 30 minutes of questions and we have a lot of them so I won't go into too much of an introduction but I'll still introduce Guillaume who was doing the workshop on the fleet and OS Query. So Guillaume is the head of security at Fleet Device Management, the company behind the open-source fleet management platform for managing and using the OS Query. While he prefers working in startups he's been working in security forever in organizations of all types and prefers working at the bright side of things and things that work instead of repeating 30 year old best practices that never have. Alright so the first question and most of voted is I think mostly for Caspian. What is the impact on mental health of accumulating incident response and what can be done to better handle it? That's a great question. So it kind of depends on what the level of incident response people are working at and how often they're dealing with it but in a lot of cases your first incident is going to be stressful. If you're dealing with a lot of incidents it's going to be stressful and sometimes you're going to see things that you absolutely do not want to see, hear things you don't want to hear and experiencing things you know that you won't want to be there for. So I think the mental health impacts can be kind of high and it's important both for employers and people running security teams and managers to understand that the folks that they're putting on the line to do this actually do you know experience that and to have that awareness and have that support in place when it's possible. Alright thanks. Any tips for capturing momentum after an incident to get to get better practices in place so like input post incident? Oh my god are these all for me? I feel really special right now. Yeah a couple one post incident basically take what you've learned from the incident it's gonna make a huge difference if you can say okay you know let's not waste this incident let's learn something and let's build from there and in terms of capturing momentum from the teams like it also kind of goes back to a mental health thing right you've basically got knocked down you have to get up again oh wait that wasn't yeah I was gonna go on with this song but it's not a great situation and a lot of times you know you can use the incident as a rallying point for people but the most I think the most important point is to not point fingers you know to not turn this into a situation we're like yeah so-and-so is responsible for this so they should you know they should get fired or demoted or whatever that's not something you should ever do in these situations. Yeah I think we this the blue teams in general they we have a reputation of pointing fingers that maybe we shouldn't be doing or maybe we don't do it but I think people external to the the security people they they see security as pointing fingers and it's bad for the the business or the industry as a whole. Yeah and I think so I've worked with Guillaume in the past in a couple of in a really fun role at a former employer and I've heard a lot of his stories of kind of like losing it and getting angry but being positive and I think that's actually kind of important in these situations like be angry at the incident not at the individual. I think to add on to some of what you're saying about you know not blaming a particular person or etc like Google has a culture on SREs of blameless postmortems after an incident happens yeah and I think as much as possible that that also applies to security like something bad happens document what you should have done instead there's something that's critical that you absolutely have to go fix and it's not someone's fault that I don't know the alert system failed or whatever right. There's nothing more there's nothing I'll get a proof of concept built faster than some terrible all hands-on deck like security something happening right I wouldn't point to specific things but over the past few years like you could imagine those moments where it's like everybody stop what you're doing we're working on this one thing like those are often times where you end up getting to proof of concept the things that you wanted to all along and you know suddenly it's running on every machine and you're like okay maybe we could do more of that and that's a much easier place when you've kind of proven that out out of panic and then going from the the proof of concept to actually implementing correctly is another topic but it's really complicated as well. I have a question for either Eric Ormea so you talked about the security keys and web app and so it's a fairly new technology but the technology is there it works and I think that the adoption is quite slow and my question is I think I think people a lot of people even security people are not aware of the benefits so the biggest one being it's basically unfishable so how do we speed up this process of getting people to know it exists it's very good you should you should get to that at some point. I would say that I have a much easier job because I work in a corporate environment where you can sort of just say like hey we're going to use these like you know we start enumerating all the places where we don't use them we have exception processes for that but if you look at like what the Fido Alliance is trying to do they're trying to get the whole world to get on to that and you've seen things like their white papers of trying to have even like soft Fido tokens that you can move around between devices and back up to your like cloud right so I think that these are kind of two different topics right like a corporate entity is going to have all of their weird little things that you're going to have to know that okay this doesn't use web often or it wasn't used to security key to get the entire world to use security is key is something I'm dramatically under prepared to speak about yeah. I'll add a couple things one is that your SSO probably has a control that says something like require security keys so yeah you just like prep your your employees to be ready for that and send them all security keys and all that kind of stuff explain that you can put multiple credentials on the same key like that they don't that they can have lots of keys they all work like all that kind of stuff so you need to spend the money up front on those keys and flip that switch but that's absolutely something at an enterprise can do the other thing I will say is I think some of it's going to be forced on us in a good way so GitHub announced a couple months ago that they're going to force all developers to have 2FA not security keys but just 2FA baby steps I think in 2024 or 2023 I got an email it's time it's like really really time like I'm shocked this hasn't happened before but a lot more people are going to just start using second factors and security keys and adoption will continue and about a get up I think I think they said that I think 15% of the users have 2FA which I thought was really low because their developers they they're closer to security than most people and they should be aware of the benefits of it but I don't know the 15% but the last public number I saw was 11% of projects that have over a hundred developers have developers who use security keys or sorry use 2FA because I don't think GitHub tracks which ones which that you're from that they publish yeah all right question for Yuri can you successfully distinguish obfuscated JavaScripts from minified ones because I think minified JavaScript is pretty common in in the web as a whole because it reduces payload size and there are lots of benefits so how does your your model do with that yeah it's a great question so let me first explain a few words for those who are not sure what are the differences so unification means that you do not intend to hide the code but rather compact it so usually the purpose of minification to make the code run faster and and then it just changes the names of variables mostly and like compresses the code to be on a single line so as you one of the examples I showed was indeed me minified so in our case first of all the minification of itself we didn't see an interesting cause to distinguish between the cases so from our point of view minification is just clear text and and this is why we did so so I did try to create a model that do a multi-class classification like three classes minified clear text and obfuscated and I didn't see that these model performs better but but again the point of view was to distinguish ultimately between obfuscated and clear text and this is why so in this case it didn't work better than that but in general to sum up it's possible to to you know spot minification by machine learning very similarly to what I described before in my in my presentation and if you really want to then just you know just use a minifier create the data the data set as you like and just run a similar model and I guess it will work thanks question for I guess anybody that work in incident response or in a blue team given an incident have you ever considered letting the threat actor operate in the hopes of capturing more tools and tradecraft is there someone who has a story to share about that can you talk about the story I don't have a lot of stories to share about that that I can talk about but I that is definitely a thing people do yes I I mean I currently work at a place that mostly handles ransomware incidents so tools and tradecraft we have a really big Intel team for that we've got you know we're basically that's what we do but yes I have seen that done the game that you're playing with that is a dangerous one because you are basically saying let's let this roll out further and you're doing that hopefully with legal oversight because there are a lot of questions that are going to be asked after the fact if you basically let any threat actor wander around in your network and start messing with things yeah and it's a big it's a big gamble because I guess one one of the good reasons you would want to do that is if you if you suspect the threat actor has many footholds you want to identify them before before notifying the threat actor that you're on to them so it's a it's a really tricky questions and I don't think you can have a general answer it really gonna depend incident on incident no I I mean really this comes down to a visibility piece again and this is one of the things that I think a lot of blue teams struggle with is kind of being able to see as much as they can on the network and you know there's also the controls which I know you guys were talking about because I got to see that so there are elements of this where we're only just now getting to the point where we have the technology to maybe see this as well as you want to but there's a lot of resistance to putting that in in places all right question for Guillaume at what size of company does something like OSQ query and fleet I guess start to make sense I would say if you're if you're remote and you have computers that are owned by the company as opposed to BYOD like 50 people 30 people 50 people probably starts to make sense I think as soon as you start having enough systems that you're wondering what's happening on them right so if you if you're like a dentist office and you've got three computers and it probably doesn't make sense but you don't need to have a thousand definitely under a hundred makes sense and is there would there be an upper bound like if we have 60,000 machines is it too much or should we go another solution no I mean the thing with with OSQ query is there's like 300 different types 300 different tables with data that you can get from some of them can be extremely generate a lot of data for example there's a table that will tell you about every single connection that's established if you run that on a Linux server that's a load balancer and and like you're you're on the internet and you get a billion connections every like it's probably not the right tool but I've seen a lot of deployments with you know 100 200 300,000 servers and it scales really well it's more about where do you put the data after but it lets you be very precise about what you're collecting you don't have to grab everything and then decide what you're going to do with the data later right you can decide okay we have a new use case we need this table we're going to collect that every 15 minutes every hour every week and then start using it so it's not the same type of you know let's grab everything and figure out what we want to do with it later which can be very expensive I guess you know whoever sells a license based on how much you can index or sells hard drives loves that model but we don't think that's the right model in most cases especially if it's a big environment thanks Maya or Eric I guess what level of logging do you think it should be in place for each level of zero trust so I guess this references risk-based access I mean I don't know exactly where but it but like I think the a lot of what we were trying to explain in the talk is that this is so complicated and like logging when you talk about logging is not one thing right like device logging endpoint you know with like stuff like or you know and yes like OS query which is kind of logging but kind of not visibly these hard yeah and yeah so I don't know if I have a great answer of like what you should log I mean experience wise there's also this kind of weird interaction between like you need different kinds of logs now right because you're not just gonna have all of your things on the corporate infrastructure and you can do that right where you can you know say oh yeah here's all the IPs that came in and out so I would say that logging is just like another one of these wonderful challenges that you'll experience and is not really a good answer I mean log everything I mean is it a question is there an answer to response where you you had too many logs like absolutely and the thing is like I will always say log everything like it is we have too many logs sometimes that takes us a little bit of time there are really good data engineering tools are really good databases for this stuff so if you've got the right tooling in place it's not a problem I have been in places where you know I keep making this joke about grep like it I've done it it sucks and it takes a long time but I too many logs operationally as a problem too many logs in terms of an investigation never is there's there's there's no such thing as too much evidence grip is the best tool ever I also say to like logging like on traffic on the network is maybe a little less interesting than like I have an application proxy and it knows exactly who is connecting and what device they're coming from and so you know I don't think it's a matter of like more logging or less logging it's like sometimes you just get better quality logs out of these kind of solutions too I will add on to that I feel like a lot of people have built these like proxies or load balancer proxies whatever they happen to be and they're like wow now that all my traffic goes through here I'm gonna do all kinds of other stuff to it too and you're like that's not that wasn't the point like you can solve that problem in other ways it's harder but that's not you solve the problem in the same way because you really had someone to put the logging I mean if what you what you really need at the end of the day is going to be some sort of audit log of like this user accesses application at this time and did these things and there's lots of ways to do that it happens to be easy with a proxy but there's lots of ways to do that for Yuri have you found that a lot of non-malicious JavaScript is obfuscated to for interdictural property reasons so do you have like numbers or something you said it's a valid use case but how how widespread is it do you know you mean the the intellectual property thing so I think in one of the papers the www 19 they really did a deep analysis of of the distributions of JavaScripts in terms of what happened there for our data in imperva and as I mentioned it wasn't really feasible to tag tag it manually so we use some some heuristics just just to test the model but it's not it's not something that we can you know accomplish in any way if we have just an example like almost half half of hundred thousand scripts as a test set I have no way to take security research I mean we have many but I can't really ask them to tag to tag it and to say which one are by chance you know intellectual property so obfuscation type yeah you would have to reverse them and then steal their intellectual property or something like that about machine learning in security we have a comment here it seems that everyone wants to use machine learning for stuff like ID system security is it worth it or can can it be just a waste instead of boring things like good logs so I'm gonna rephrase it where do you think machine learning fits into security as a whole as an industry as a as a part of doing business yeah it's a good question so so generally if you ask data scientist what is machine learning good for something he will tell that most probably yeah it is it's very good so so I think there are many applications for machine learning today as you know the algorithms become more more sophisticated and more more libraries you can just download and easily use and then you don't really need to understand the inner working of the models so but indeed there are cases where you don't you cannot use machine learning let's maybe talk about this negative a negative example so when when can't we use so there are some cases that we don't have enough data and there are cases where the data is too dirty so you need to make you know to it always starts with the data when we look on the data we can see if it has to be cleaned if we have enough of it if we have a very unbalanced data set so these are maybe like the corner cases where if you just taken out of the box algorithm and apply it on on on such data and you maybe come up with something strange so there is no you cannot really replace this pipeline of of gathering the data and a good amount of it if you want to solve like anomaly detection and you have 10 examples then you know you don't have to use machine learning so that's I think my take on this as a rule of thumb I'll add on to that because I think the we talked a little bit like risk-based access I think people like oh yeah you can just decide whether or not someone can have access to this application and just use spring to some ML on top and like it's exactly what you're saying that like anomaly detection is a great example that's actually kind of bad for machine learning right like if machine learning is good at a couple things one of the things that's really good at this sorting things into two categories A and B that works well if A and B both have lots of data if you only have 10 examples of attrition attempts to your system like you just can't train a model on that right it's not possible and so something like risk-based access you don't want to put that in front of a user who's gonna like get even more upset if they can't access their email or do their job or whatever it happens to be like sometimes you can do some of that stuff after the fact to like look at logs but even then like you don't have enough data to train a model if there's just nothing to go on yeah I think this thinking comes from the fact that a lot of security people have very little experience in machine learning and they they they either go to the side of ML can't do anything or ML can solve everything and it's a we need to talk more to data scientists about about this stuff so so so we can know that ML can do this but it can't do that and just trying to maintain even getting your signals to fire in a big system like I talked about the the quality of you know stuff you get from an operating system even just ensuring you're like like I do a lot of development where you have like nice hermetic tests and you can say like you know does this run if I run it against other system like a big corporate infrastructure you're not really gonna have and test very often and you know signals rot pretty quickly so even just getting signals to fire that's like before you even can take that structured data and do machine learning on it that's already a hard enough problem right but but I still think that we don't need to discourage people from from doing machine learning I think that it's on the country so I encourage everyone to you know download some some libraries to be proficient with pandas and scikit learn and stuff like that and you can see many many examples when you can do really cool things pretty easily and I think that will help you to understand what machine learning is good at and what's not really and it's much better than you know just like using buzzwords as Maya said just sprinkling them that's a great advice even though it's spooky I don't want to pip install scikit learn just you just have to wait a little bit right yeah it's kind of a big package all right probably last question maybe another one after that I need to ask as I saw on the on the Twitter two weeks ago interview questions for a suck analyst role and the interview questions were like you see on the windows even like even ID 46 24 what do you do what what does this indicate or you see error code windows 11701 what is it and I thought this was really you didn't know what these things mean oh come on I I've been I've been working two years in a second I would Google these questions I actually if someone asked me this in an interview for this role I would probably walk away from the interview so so I have actually asked that in interviews but it's usually what's your favorite event code not you know you see this and then what do you do because it's not fun to put people on the spot and I think I didn't mention it was for a junior analyst role so so yeah so my question would be for for blue team roles what would be or analysis role or IR roles stuff that requires analysis and deep technical knowledge sometimes what would be your your your good interview questions I love that you're all looking at me now 46 24 really honestly it's yeah I'm gonna stall and give you all some time so a couple things that are bad with that interview question is it has specific knowledge that someone has to already know which is bullshit it doesn't show that the employer is like willing to train you on the job or like help you learn anything again which is bullshit and like the only like we literally went to trivia I don't know two weeks ago in one of the trivia categories this is EFF trivia so like take that with the great assault one of the categories was like CLI commands for deprecated systems and like I think our team got every question but that was embarrassing and I contributed nothing to that absolutely nothing to that anyways so I stalled now you have some interview questions to me it's more about the process that someone's going to follow to get more information right so okay so what would you do if you saw that like even if you told me well I'm going to Google the event codes obviously that's what someone's gonna do that's that's a good answer but then maybe I'll tell you what the event code means where are you gonna find more information about that and then I feel like even questions around networking sometimes can be useful just to know if someone has like a decent understanding of how networks work and it doesn't have to be like oh calculate the subnet of whatever you've got three seconds that doesn't really matter here's an IP packet what did the check some yeah exactly but it's more about just like the basics and then you're you're curious and your process for getting down to finding it is good like troubleshooting is the same thing right like if you just want to troubleshoot something you need to eliminate theories until you get to what's the thing that's broken and when you're investigating it's it's pretty similar so to me I'd rather hire someone with less experience that seems to know how to look for things and come up with theories and look at that then someone who can recite Windows event IDs by heart so yeah I agree completely with this because typically if I'm doing this if I'm running the IR team or the sock I'm not gonna be looking for somebody who is gonna be able to calculate packets like this is what Google is for right you know the skill testing questions are nice process is more interesting so don't you know don't be able to answer things by heart but understand how you're working I also say as when we do a lot of interviews for my employer like we this this sounds like advice that you be giving to like somebody who's newer but like we've read your resume who we know about what level you're supposed to be we've assigned someone to you who we like knows a little bit about that domain so they're like there to have a conversation with you so I think the biggest advice for that is just like if you put something on your resume please be ready to talk about it because weirdly that's like how we determine how we're gonna have a conversation with you to figure out all these things of like you know okay here's what you know but like you know are you interested in this or you know are you clever if like we give you these two pieces of information and you know something about this domain like you know can you put that together and yeah in one of my first interview for an internship when I was in the university I someone asked me about a personal project I put on my resume and I was like Pikachu face surprised and I was glad someone I was really talk about it but I didn't expect someone to to to ask questions about about all the aspects of my resume but it makes sense once you're you're you're used to interviews so this is all the time that we have so thanks everyone for the great community session and we'll see you tomorrow