 Welcome back. If you're just joining us here in the AppSec Village, I hope you are ready for a fantastic talk. If you're still hanging around from the last talk, glad to have you back. This is going to be awesome. I wanted to take a few minutes to thank everybody who helped to put on DEF CON this year. I know this has been very challenging for people. It's challenging for us at home. It's doubly challenging to bring a great conference experience to DEF CON. We really hope that comes through. Also take a minute to thank the AppSec Village. Some people have put in amazingly long hours to make this stuff happen. We want to thank them as part of that. Additionally, if you want to get a shirt, head over to the AppSec Village website. You can purchase your shirt there for the second incarnation of the AppSec Village. For our next talk, we've got Paula Barr and Stanislaus Mulvaux talking to us about turning the offensive security mindset into a developer's toolbox. Hi, hello everyone. Thanks for joining our presentation, which is going to be today about our journey into turning offset mindset to developer's toolset. This presentation, we're going to be two of them doing it. It's going to be in what we can say three parts. The first one is going to be about a tool that we created inside our company, which is called Chop Chop, and we will come to what was the context, what was the need for it, and that we're going to explain what happened. The second one is how did we want to broaden our audience into having more and more people using it, and especially developers. The last part is going to be the demo that we're going to do with Stanislaus, who is here just next to me, and who's going to do it. So three parts and trying to explain our journey into how we tried at least to achieve it, and some of the mistakes, drawbacks, and things that we learned on the way. So hopefully, we hope that you're going to learn something about it. All right, let's start. This graph, we actually took it from CVE details, and it just shows the number of vulnerabilities and numbers of CVEs that we got starting from a couple of years back to basically now. So we took this graph last year, and we can realize that like between 2016 to 2017, the number of CVEs is just like doubled. And this is something that we can see. Every day and every week, we get like new vulnerabilities coming in. We get Oracle Bolton, which is just like full and hundreds of vulnerabilities are inside. We also get like Bolton of Microsoft, which are just like showing up a couple of hundreds of vulnerabilities or let's say dozens of vulnerabilities each month. And this is something that's growing, obviously. And we're hearing a lot about this. And in the last couple of days, last couple of weeks, we got a number of vulnerabilities during the lockdown, which was just like massive and especially critical vulnerabilities. So this is something that we can feel, and this is like getting really, really big. And when we're taking those numbers, we realize that just by divining it by 52, the number of weeks that we have per year, we realize we get something like 319 new vulnerabilities. And this is the number of new vulnerabilities that we get per week. And this is huge, huge for a team to just like realize that we do have to process all this information. So bear in mind that we are not affected by all those vulnerabilities at the time, but it really depends on your context, on our context, on your customers context, and so on. But this is something, some information that needs to be digested by your teams to just realize if you're affected or not, and if you're vulnerable to it or not. And this is a huge thing to actually like realize and to process, especially depending on like the size of your team. And usually when we get the new vulnerability, like the MS-17010, or let's say the new vulnerability for like the DNS remote code execution on Microsoft, which happened a couple of weeks ago, we usually get the management because they hear a lot about this on Twitter, on specific magazines, on specific websites, and they actually hear about like this, there's like a really big buzz around those vulnerabilities. And usually they're coming to the security team and just asking, all right, are we affected by this vulnerability or this other one that we heard about? And this is something that we can understand it, obviously. And the problem with this is we get to the point where the security team is overwhelmed by the number of new findings that happen each day. And you get like a backlog, which is full of things that you need to process. And it turns out that the management is coming and coming and coming again to just like ask if we are vulnerable to this, to just answer to the management, to the executive, and so on. So something to bear in mind, this is part of the context that we are in and why we actually had like this old tool ID. And we started like getting this. So the context was based on this. So the problematics we had was something circa 2016 when we started having like this kind of philosophy and mindset. And the idea was we had thousands of servers and domains that had to be scanned daily. We wanted to scan them to basically for statistics reason, KPI and having some like fresh data because obviously as soon as you do a penetration test or any kind of testing, we can consider the test outdated couple of hours or let's say a couple of days afterwards. So we wanted to have like fresh data all the time and daily fresh data. Second of all is we did not have direct access to the servers in some specific situations. And this was based because the servers were materialized because it was managed by other entities and a couple of other reasons that we had. So we didn't have like SSH access on all the machines all the time. So this is why we needed to have like another way into assessing the security of the applications that were running. Exposed on the internet inside as well in the network and really we are like a really big context with this. And we can see that the situation is quite pretty similar to what we can call black box penetration testing which means that we don't have like administration access to the server and that we need to like find other ways to qualify and to have an ID if we are vulnerable to this specific value. The tech landscape that we had was also unclear. In 2016-2017 there were a lot of vulnerabilities especially with WebDAF if you can remember it was like quite hot hot topic that time but also with struts because there were a couple of critical vulnerabilities which happened in a couple of days. So these were the things where okay there's huge vulnerability with struts are we vulnerable to it? Yes, no. And can we quantify the number of servers which might be potentially vulnerable to this? And the last point but not least is interesting in speaking of which was like this lean security and is the idea of ability to perform checks every time so again and again. So basically this kind of like loop this lean loop that we can have where we would be able to like spot security regression. And this remembers like a cool anecdote if we can say this way is one company quite big company having a bug bounty program and after speaking with them they actually like told us that they paid like some really good bounty for a specific researcher who found like remote code execution on some of the production service. They fixed it but unfortunately couple of weeks later they redeployed the same vulnerable component and another bounty researcher found it and they had to pay twice for the same vulnerability that they had that they had already patched and that they already paid one researcher for it and this is what I'm going to call security regression. When we taking the tech and like the developer landscape unit testing is actually one of the foundation of any kind of like new development and I think that like this specific mindset and security regression should definitely be something that we could put on our top priority just to make sure that we get all those regressions as soon as possible. We have everything automated right now. We do have like a lot of pipelines for our developers. Everything is going faster and faster and faster and I think that we can improve our way into this and this is some of the reason why we created our tools and we wanted to tackle this issue. Security regression was one of the big part. So this was our context. The needs that we have, the needs from the field and this was again from like 2016, we had to assess new vulnerabilities, new risks coming in and we had to do this quickly. So we have 2017, we had struts, we had RCE, so remote code execution on JUPO, we had a webdav, we had like lots, a lot of things happening obviously and we had to process this as faster as we could. So this was one thing. The other one was we had like a small team working on this. What I'm speaking of small teams was three to four people. All of them were all security oriented which is really important for the rest of the journey because this started introducing some bias which means that we were security people working with security tools that we wrote and we were more or less like in our circle and in our comfort zone which means that when we started and wanted to deploy it to other population and other kind of people, developers, DevOps, it turns out that like we had some interesting gap I would say and those bias that we introduced them, that we introduced we're going to try to give you the mistakes and how we tackle these issues. But never mind, we will come back to this later on. So we needed a tool which would be fast, so something where we could just say okay, here is the new vulnerability, run it across all our perimeter and see if we get hit. The second one was having something reliable and when we say reliable, it's mostly interesting for the deterministic return which means that if we would run the tool twice we would make sure that if it hits, it would hit again. So we would have like there would be like nothing like random in the processing. So we would know exactly how the rules would be triggered and if we would run it twice, we should have the same output, obviously. And the last point but not least would be no false positives. So it's interesting because we need to find like a threshold, right? We need to have like a tool super fast but on the other hand we don't want any false positives which is really hard to combine all together, right? So it's two criteria which unfortunately don't really work together but we think that we did quite a great job and this is something that we managed to do. And considering that a hit, chop chop hit would be a real hit which means that if we get something in the output, in the console output, we would have to open an incident, call someone or start doing like an immediate action, right? And not having like a lot and lot and lot information to process. So really trying to like do some kind of like sniper thing and just making sure that we get to what's really valuable. So those were our needs and yeah, trust me, we had a lot of them. So if we can take an image, an analogy, it would be like more or less like going to the jungle, right? And it would be like something where we just have like a machete and like start just like chopping trees to try to like create a path in like the jungle so that we could reach our destination point. But this was like more or less the case. As you saw beforehand, it was more or less like an approach of like black box testing where the tech landscape was a bit unclear. We didn't exactly know what we were doing and what we wanted to do is like creating hypothesis and trying to see if it would be valuable for our context and making sure that the tool that we release and that we get would be something also valuable for other organization so that they could customize it for their own environments. This was like mostly like the whole soul of the application. So yeah, this analogy, really the analogy of jungle and having something where we would have to like start chopping trees and making sure that we're not in danger. So we were saying chop, chop, chop, chop and what does it mean? It's actually a funny name and it's something coming from Cantonese which means hurry and suggest to do something like without delay and right now. So we thought the name was interesting and exactly what was like the spirit of the tool so this is why we called it chop, chop. We wanted to have like a tool where if an emergency alert was coming in we would be able to like respond right now and having results in no time. So it was the whole soul and this is why we kept the name chop, chop and release the tool with this name. The tool initially back in 2017 was written with Python. We started with Python 2.7 then migrated to Python 3. We got everything working and following the upcoming month we started getting plugins that the security team mostly wrote and in the end we came up with 80 plugins so 80 things processing which were done going from trying to find if there was like a .git folder exposed on the internet or let's say in the web route accessible in the web route to also like checking if the RDP port was also open checking if the port was here and was just responding. Couple of things like this and we started creating plugins which were specific to our environment. Those ones were not released this gives you the idea of like the flexibility of the tool. So taking a domain name or an IP and afterwards doing some processing and any kind of processing basically. So as I said it was mostly doing things with HTTP and this was like the whole soul of the application first but then afterwards people started integrating like new protocols and new modules to start interacting with other services. So we had everything running the ability to run scans daily with what we called deterministic websites which was one of the things we were really eager to have so daily scans basically every day and allowed us to answer two questions. The first one would be to answer to the management are we affected? Yes, no. And then afterwards being able to quantify the number of servers or number of IPs which were actually affected with this specific vulnerability. Is it going to be one? Is it going to be 100? Is it going to be dozens, hundreds, thousands? Is it going to be all the servers at once? So it's actually quantifying the results and we were able to do this. So we would count every console output every line we would get and we would have like an idea of how many servers would be affected. And this allowed us and this saved us like a lot of time. As I said, Chop Chop initially was written in Python and this is an example of a plugin that we wrote and this one is to try and check and ensure that the Git repository, that Git folder is not exposed in the web route. So what it's going to do is quite simple. It's going to test on port 80 and on port 443 in HTTP and HTTPS and it's going to check one specific path which is going to be .git slash config and it's going to check for one specific word to appear in the HTTP response. Based on this we're going to have a severity which ranks from 0 to 10. 10 is going to be critical, 0 is going to be informational quite similar to what we can find with the CVSS code. Something similar. Again, this thing, we will talk about this later on but we changed it in the new version of Chop Chop for particular reasons and because we thought that having this specific range would be way too far and way too lot information to process and based on the audience we wanted to reach this is something that was a bit useless. But we will come back to this. We had a description ensuring that the Git repositories is not accessible from the web route and one fix what we can call remediation. It was really important for us because afterwards, after using it and using it we started getting people and non-security people using the results of Chop Chop and it turns out that some of them were lost by just having an Excel file or anything like this just an output, Chop Chop output just didn't know exactly what to do with this so we started implementing remediation to give some insights and some inputs so that people could work on their own by just giving them a bunch of pieces so that they could start working on the remediation and mitigation. This is the kind of plugins that we had written in Python, giving you some information so you just had to inherit from the plugin class that we had and afterwards you would just have to override a couple of things and you would start to go. Here are some things that we managed to find on production systems. For example, here we managed to get repository access for SVN with some files wc.db which allowed us to fetch anything from the SVN repository and afterwards getting the source code so this was quite problematic. We also had things like htpassword which was not interpreted by the application server for some reasons misconfiguration, things like this but this is something that we managed to show and the idea was having a hypothesis and trying them across like a large scope, depending on the scope you are that you have but this was the idea having a hypothesis taking things maybe from the bug bounty we were speaking about bug bounty beforehand so just checking things that bug hunters are trying on large scopes and you will see in the reference that there is a nice talk from James Kettle cracking the lens which is like some research that he did in 2017 it's something quite similar to what we tried as well and we got inspired so this is why it's going to be in the references and if you haven't seen like the article feel free to reach out and just check it out it's absolutely awesome research even if it's like 2017, absolutely awesome and just gold so feel free to get it afterwards and you will get it in the slides so this was some things that we were trying alright, maybe it's not going to be interpreted what do we lose? just like maybe writing a couple of lines of code and seeing if it's going to match and if it's going to trick somewhere fortunately for us it tricked and we managed to like remediate it in a couple of hours but also like internal information disclosure so here we had like the service status page which gave us some information nothing super critical but still it could give like URI so basically like endpoints and maybe some kind of like internal endpoints but was also giving some ideas about like the IP address range that we were using internally so those kind of things that like we didn't want attackers to have on their hands alright so and we wanted to try this across like all our servers again and finding them so this is the kind of things that we managed to find last but not least here is for example with the wildcard so basically some cross domain .xml files which are way too permissive ok way too permissive allowing anyone to just like craft some payloads and being able to like start discussing with our website might be afterwards like stealing some cookies, stealing some information and we wanted to just like trick it on all our perimeter this is something that we managed to find one rule doing it all so yeah it worked pretty well and we were really happy with this so now what we had was hypothesis we had the tool written by a security team used by security people and it was working we were happy everything was good but then afterwards we wanted to try the extra mile and the extra mile was actually a couple of extra miles and by saying this is ok we had our tool internal tool and we wanted to make it available to everyone what we mean by everyone is mostly developers so that as soon as having a tool that they could integrate into the pipeline and in the CICD pipeline so this was our goal and basically the polar star that we were trying to target and it turns out that like the first issue that we found was actually based on communication developers don't necessarily get security things it's one of the golden rule which means that we're coming with our needs we're coming with our ideas with our beliefs developers come with theirs and it turns out that we need to find a way on how to get a line we found this interesting thread on Twitter so we're not shaming anyone or doing any kind of finger pointing but we think that like it's an interesting thing so we got someone on Twitter basically showing and giving some idea about like past experience and started using acronym IR, especially if you're working in security this is something which stands for incident response and this is perfectly understandable but it turns out that like in the comments we got some people where IR didn't mean anything so people started looking up and found that like it was an acronym maybe for infrared cameras and it's something that if this person doesn't know the acronym there's some issue with like the communication and we need to find a way into like achieving and finding a bridge between them and the last picture is just showing on the right that everyone is right everyone is right based on like what they're looking up but it's just that we need to get those two people aligned to make sure that they can move forward and move together in the same direction security is not against developers developers are not against security I think both of them can work together but just like trying to like get the same beliefs at least on some very specific topics so yeah interesting things about like the communication channel and speaking the same language and when it comes to security it comes with a lot of details SSTI XSS SQLI but also when we're speaking about CDEs so we get like thousands of CDEs as we saw earlier but also like MSBelten MS08 067 but we also have like name of vulnerabilities Heartbleed, ethanol blue blue born and this is like a huge referential that we do have like in security absolutely crazy security people are actually like losing track of all this right it's so much information we have like so much information to process so yeah we have like those like naming named vulnerabilities they have websites, they have videos they have almost like choreography and things but also like all the names of vulnerabilities and classes, vulnerability classes that we have so much information to digest it's already hard for security people and people working in this for day long right and we've been doing the careers in this but what about developers how can developers keep up with this information when we just like giving them a report some of them might just be I'm just completely lost I'm just completely lost, I have no idea what this means I can see this presentation but I might like look like a fool if I don't if I just ask the question what is this acronym for so it's some information that we need to to take care of and just making sure that we get the interesting bits to developers and that's actually another question and we're not getting to the debate maybe afterwards in the Discord but should developers keep up with all this information should they and I don't know but I don't have the information so far I don't know, I have my opinion but I'm happy to discuss and to chat about this afterwards in the Discord with you all so a lot of information of process of vocabulary that we need to take care of the second part was if we get a chop chop hit and the application is flagged because of some reasons, maybe some plugins got in and so on, now what now what is going to do the developer I don't know, it's just this there's a URL, it's giving some information but what is it going to do about this what is the risk and what is the kind of remediation it can put in place to mitigate this but it needs to understand the risk and what can happen what are the bad things that can happen with this specific thing and in his environment and in his context so security vocabulary is one thing, remediation is another one and bringing remediation in the tool and giving this capability is actually like bringing assistance to the developers which means that we are going one step forward to them towards them and what's going to happen is developers are going to come to us as well afterwards and it's going to be a win-win discussion and everyone is going to be aligned again just having secure application running production and everyone is going to be happy with this so bring anything valuable as soon as you have a hit and that you have something that's actually triggering in your environment so it could be a link to whatever kind of explanation it could be like a research document it could be anything on Twitter it could be just a thread or it could be also a pointer to internet resources do you use the OWASP ASVS for example so maybe that could be a link to the specific section and just referencing it so that developers could start getting information developers are super curious as security peeps so I think it's a good opportunity to have this discussion and this conversation with them just bringing assistance and helping them into achieving better security within their application by giving them as much input as we can another part is contextualization and contextualization is also key again I'm not going to go into the debate and I think that this is something like a question that you might have to ask yourself is should we consider the same criticity if we get chop-chop-hits for one machine which is exposed on the internet and one machine which is completely air-gapped in a segregated environment where only two people are reaching the application and this specific server it might be some kind of trade-off that you need to find maybe finding some balance because if you start answering yes, this is exactly the same criticity the same exploitability you might just start giving red flags to developers because everything is going to be critical and based on this developers have a lot to deal with production errors they have a lot of development they have deadlines they have a lot of work that they have to do and if we start getting security tasks on top of it they're going to start getting overwhelmed and having too much information afterwards they're not going to be able to process any of it so we're going to trade-off about what we try to achieve with them and what's the image that we want them to perceive from us so balance and trade-off are now part of the game last but not least is about collaboration and we truly believe that collaboration will empower your organization and by saying this is like finding the best tools to do it so for example, a lot of people used chop chop in June we had a lot of people asking us what's the difference with Nikto and we are users of Nikto, we find it absolutely an amazing tool and it works greatly but it works greatly especially for security oriented people because it's going to give you usually a lot of output but afterwards you're going to start to have to dig in you might have between one line on the other something which is going to be informational another one which is going to be critical but this is going to depend on your experience on what you think about it and you're going to have to do those choices on your own are the developers going to be able to do it? it might not be possible because they might not know anything about security so this is why we want to help them by doing some kind of triage in all those findings and providing them good inputs so finding the best tools is extremely important and some of them might be tailor made so something definitely to bear in mind second of all is making sure that the learning curve that you're going to have if you're looking for people to collaborate which means writing new rules helping maybe doing a pull request on your project and doing this kind of collaboration make sure that the learning curve that people are going to have to start digging in the code base that's not going to be something too crazy usually when we speak to developers they find the security quite mystic it's like some kind of black magic happening behind the scenes they're using some weird commands and so on and this is why if we start putting on the shelf a new application that we want people to start playing with that they start collaborating with us we need to make sure that this is mostly friendly to new people coming in security and writing tools and writing rules and so on so something to bear in mind and again this depends on the context where you are in which means if your developers are using YAML in this case go for it if your users stick with XML for some reasons because in your context this is what they use stick with it and go for it it's really going to depend and I think that the last point discernment and clear thinking for your choices within your organization are key this is one of the most important things that we learn is just checking the context and make sure that the security adapts perfectly to this context within your organization because if you're trying to get a bigger audience and broaden the usage of your tools you need to actually get them in because you need to look like them so that they can start getting it without any issue and last point but not least change the posture from no you can't to yeah let's see how to make it for example if some people want to use the tool because they also want to do some monitoring and things yeah why not they should be able to do this if they start asking questions about new technology new frameworks that they want to put in well obviously there's going to be a balance into is it something that's good for my company, for my organization but you need to have this discussion and not having just like really close answers so that's we think like some of the keys that we got and that we managed to see following this development and trying to like broaden the usage of our tools so this is why we created Go Chop Chop and what I'm going to do is give my seat to Stanislas and he's going to present you the tool and hopefully make an awesome testing session and demo with you cheers hello everyone and thank you for watching this talk I'm Stanislas Moldeau I'm working currently as an security intern at Michelin so I present to you how we created Go Chop Chop which is the second version of Chop Chop which Paul presented you so Go Chop Chop is a command line for dynamic application security testing and it's written in Golang its purpose is to scan several endpoints and expose services files, folders through the reboot and we created the tool so it can be fully configurable by developers even by non-security people and I'll present you how it works and how easy it is to modify and add a role for example so here we are testing our own web application which is exposed on port 80 so for the purpose of the demo we exposed a git config on the web application so we can see here we did a curl on a web app and it returns a status code of 200 and a core repository for motivation and so on so now we're gonna try the tool on the web app and so here alright so we launched GoChopChop on our URL fuba.com and it flagged the git config vulnerability and it gives you a severity level which is high in our case a name for the plugin and a remediation we can be used later to reduce the vulnerability so now we're gonna see how to configure GoChopChop to make new role and how you can configure it alright so the rules are written in YAML and it's very easy to create a new one or modify so here for our example we have the role git config so it will scan the http response if we give it a name so git exposed a status code of 200 which will scan the http response to see if the http response gives a 200 status code a match which is a list of strings the tool needs to find in the http response so in our case we saw it was core there is other things like no match no match is a list of strings which needs to not be in the http response and another one is headers for example cotton type application disson and you have to give it a remediation so here we say do not deploy.kit folder on production servers a quick description so the developers understand what is going on and what it has a severity level of high so here verifies that the git repository is accessible from the site and severity level of high so you can easily modify it for example high to low level or for example if the developers do not really know what to do we can give foobar so if we launch it now it will say I don't understand foobar you need to give a severity level of informational low, medium and high so here we can modify it again to for example low and if you relaunch it it will take into account the modification and the severity level goes to low so now we are going to do a new role for the sake of the demo we created a quick html quick html page so yeah we see here we created a demo which says defcon rocks so I want the tool to flag it and to gives us an output on this so here I am going to scan the main endpoint of the web application so what we check is we give a name to the role of defcon over here we need a scope of 200 and we need to match the string defcon so if the defcon is found in the html response it will flag and a revocation and description and the severity so we can test it and here you see the tool flagged it on the main endpoint the severity level and the reverberation alright that's pretty much it I will Paul will finish this presentation and thanks for listening welcome back thank you Stanislas for the demo so what we learn from this from going from a offensive mindset to creating a tool used by developers and what we learn was things that finding the same language same vocabulary and speaking of which speaking the same with the same concept is extremely important so going to always make sure that you get that they get your points and that you get theirs as well which means that if you have some beliefs if you have some things and reasons that you think that the application should be secured this way just make sure that you also understand what are their problematics in terms of why maybe it can be something based on the organizations but this is something that has to be done definitely that has to be done and having you to have a big conversational yeah in two channels basically third point is bringing assistance to like better collaboration so by as Stanislas showed you before by giving some remediation this could be like giving links giving like information maybe yeah to the ASVS OWASP to bunch of resources and trying to like give the more information to the people so that they could secure it like in a timely manner this is something that bringing this will actually help them and helping them will bring like definitely more joy and they're going to be able to help you as well if you're looking for something again don't underestimate collaboration to broaden your audience based on your context if the developers that you have that you're trying to target are using XML and you're coming with something in YAML this is not going to work the opposite as well because everything which might be shiny or anything like this might not be the real solution and you're not going to bring anything to them so try to bring something super valuable this is extremely important a last point but not least that we have on our checklist is the contextualization in trading of the trade-off having this balance as I showed you before Stanislas during the presentation you are able to change for example the severity of one specific finding if your organization has a specific context for this specific trigger this specific hit especially with the kit export I would not recommend putting it in low but this is something that we can do and that's programatically everyone can do this even non-developers people you could definitely imagine a project owner or anyone just like you have this file and you can actually start tweaking a couple of parameters so this is something that we can do and even though you can tweak the parameters if anyone is just going wrong you can see that there are a couple of checks that Stanislas developed to make sure they like to give some insights about ok it failed but why does it fail and ok so it's got an enumeration and you can only choose from those specific values but anyway so those are I would say the lesson learned from having an offensive tool and putting it into developers hands we do have a couple of references and obviously we are just on the shoulder of giants and I think that like those specific links are quite valuable especially with the what we try to build and I think that those talks and those blog posts are actually similar to the mindset that we have so first of all and we talked about it with cracking the lens from Jen's kettle and I think it's interesting in terms of the scope he has taking everything from like the bug bounty platforms and having hypothesis and running it and seeing if it triggers anything or not the second one is the keynote from the stick, stick is a French conference in France and we had the privilege to have Alex UNESCO in 2019 speaking about security with developers where he started dealing with rust and bunch of things like this where he was actually like launching the debate about like should developers keep up with the compilation flags to avoid having any kind of memory corruption errors et cetera and it's an interesting discussion I think that might have like some English subtitles because the talk is in French but definitely something that I would consider to watch afterwards the last two is actually from Harald Mir and two great conference talks the first one learning the wrong lessons from offense and the second one from besides Doa which is make more stuff and this was oriented to security people so those four references are absolutely awesome and we recommend you to watch them it's worth the watch definitely so thanks a lot, thanks a lot for your time thanks a lot for watching feel free to check out our github if you want to follow the project and start collaborating with this and if you do have like one advice embrace Aquinas, have fun and yeah collaborate with your organization to make it more secure cheers, bye