 Hi, Sarah, how are you? Hi. So you were able to just screen share. I thought I might have to bestow that upon you, but that's great. If you can see Dr. Rebek's slides, then yes, I am sharing. Great. It's actually a PDF. I didn't get actual slides, so I'll just scroll through the PDF. Sounds good. So then I'll hand off the meeting facilitation to you. Does that sound good? Or I can make the intro announcements. Just there. Thanks everybody. If you just want to do some quick housekeeping and, you know, post-cubecon, welcome back kind of stuff, whatever you want. Well, everybody should feel free to feel obliged to add yourself to the meeting notes. If you're not in front of a computer, just say so, and somebody can add you. I'm going to open up the chat, and I just put the link in the chat. And then we have, we're trying out a new format where this is going to be a presentation. We're going to use the whole meeting time, except for some of this bookkeeping upfront for the presentation. And then if you have announcements, just put them in the document or shout them out in the chat and then maybe people can, who are taking notes, can help. And then we can use the meeting for this time with Dr. Reback. And then we can just post the announcements on Slack at the end, or just take a few minutes to shout them out if we end early. So with that, I will hand it over. Does it, I don't know if it shows on the participants. Does it tell you when you've got a dial-in user or how does that work? Because I don't see her all in the end. Well, it is just now 10 a.m. Okay. Okay. So she's not here yet. I don't see any dial-ins. So yes, I also want to welcome new folks from who might have heard about us at KubeCon or got reinvigorated. And thanks also to Brendan Lum for stepping in on the intro because Dan Shaw couldn't make it to KubeCon in San Diego. So we have, I don't think the chat is persistent for new folks. So since it's just 10 a.m., I will post the meeting notes to the chat again. So our process is to add yourself to the attendance. I'll also dig up the new members link, which has instructions about how to get engaged with the sick. And then when Dr. Robert, is it, maybe we have a phone call. Hi. Dr. Rybak, are you on? I will unmute the phone. The phone person is unmuted. You can hear me now, right? Yes, you can. Fantastic. Great. Excellent. So first of all, I just want to apologize for some background noise. There was a bit of a, actually a scheduling mix up on my side and I'm actually on my way home from a customer in the train. So I found a quiet place in either case to sit, but there's no other people around. I apologize. I'm not usually quite the technique. But then again, to be quite honest, actually I actually am quite the technique usually. So we're going to be talking about open and transparent pen testing and also the particular workflow of my company. And oddly enough, I would say that my being a flight chaos monkey in terms of scheduling and working a whole lot from the road is actually really quite par for the course. And I thought about the rescheduling, but I think actually this gives quite a representative look actually of how my life actually is because we're a fully remote company. We've got the staff that are all over the world. We've got folks throughout Western Europe. We've got folks in Australia, in India, in South America, also in South Africa. So we really are a distributed company in the true sense of the word. So actually to be quite honest, we do not have an office, which is kind of, I guess, modern for how a lot of companies work. And this actually does lead into the whole discussion of chat-ups. Because of course, GitHub is, I would say they invented chat-ups. You all actually know what chat-ups is. Have you heard of this topic before? I confess that I have not. And also, Dr. Rebek, as you go through the slides, just give me a prompt or a next please and I'll advance the slides for everyone. Yep, no problem at all. So chat-ups essentially is, it's a term that was coined by GitHub. So like radically open security, we are a fully distributed company. GitHub also is a rather distributed company. I believe they might have an office in the Bay Area, but the majority of their staff is located all over the world, quite a few overseas, really all over the place. And what they've done is, I would say their primary office, and I say office in quotes, is basically chat-ups. Many of us, of course, are familiar with chat-like environments, I mean things like Slack, for example. How many of you use Slack or something similar? We need to be able to raise hands or something. I think a bunch of us use Slack, I do. Okay. It's quite common these days, I mean for development teams. And yeah, it's a really great way that if you have distributed teams, it doesn't necessarily have to be a distributed team, but it's a great way of combining both synchronous as well as asynchronous communication. I see somebody said IRC, indeed. We actually started using IRC, but we actually switched over to RocketChat because it was easier for customers to deal with. A lot of customers were actually a bit intimidated by internet-related chat. However, if you guys are familiar with IRC, you probably will know about old-school chat bots, right? The kind of bots that takes that slash-hand command that the Christian just typed in and that interprets it. These bots essentially have been ported for new chat media like RocketChat, Slack, MatterMos, Campfire, a number of these different chat platforms. What GitHub did is they built, first of all, an open-source chat platform. I would say, oh, sorry, what happened to hip chat? Yeah. Good question. Not a clue. But so yeah, basically they built an open-source chat bot that's called Hubot and you can find it in the, well, in GitHub, in GitHub's GitHub. And what it does is it takes, I would say, their operations and it kind of functions like a command and control center for their DevOps and for their system administration and generally for their, I would say, their business operations. So, for example, their system administrator team would use this chat bot for doing things like, say, deploying a new server. So they could say something along the lines of, Hubot, deploy a new server. And then Hubot would then say, okay, thank you very much. I just deployed a server. Here's the information. Or you could issue it another command, something along the lines of, you know, Hubot, please display uptime statistics, you know, for these particular servers. And then Hubot would come back and it would say, you know, okay, well here's the uptime of these particular servers and here's some pretty graphs. And the really nice thing about chat ops is it enables human written conversation to basically be able to be interspersed with text. Oh, by the way, Robert, I forgot to say you can advance the slide. By the way. So, actually, next one after this, sorry. Okay. So basically, the idea then is that, you know, you are really turning your chat room into the command and control server of your, well, control, sorry, command and control center of your actual operations. So people in a distributed team can basically do some work and then other people can comment on it. And all of this is in line. So it's a really great way of actually coordinating distributed teams. So I saw this presentation that GitHub gave at the DevOps days in Amsterdam. And for me, it was like, you know, the heavens opened. It was like this light bulb went on and I was like, oh my God, this would be so amazing for penetration testing. So I ran home and we had put basically installed RocketChat and, yeah, basically immediately installed QBOT and away we went. By the way, Robert, I'm noticing that you actually have an older version of the deck. I think that Stephen actually sent you the wrong version of the deck. I'm sorry. So it might actually be best if we just leave it on the slide and I just continue with telling the story. I think we actually don't need slides too much anyhow. So anyway, so I will continue on with the story. So then basically the, so yeah, so at that point I figured, you know, this would be really excellent for penetration testing. So the first thing that I did, you know, we installed QBOT, of course, the first thing that my penetration testers did is try to hack it, of course. You know, but then after that, you know, I really discovered that there's a whole lot of different tools that you can basically install and be able to run from this chat command line. So for example, one thing that we do is we have like a penetration test in company in the universe. We have developed our own penetration test report generation system. You know, we automate all these board, all the boring bits away. We have open sourced our system for doing this and we also made it an OWASP project. So if you actually look up OWASP pen text, then you can actually find this pen test reporting system that I'm talking about. And of course it's open source. Everyone is free to use it, download it, install it, benefit from it. And what it does is it is XML based and it uses XSLT style sheets. So basically anyone can, you know, replace the logo and the house style with their own logo and house style and they're free to use it themselves. Now the way that it works is that typically you would have a, well actually could you go to the next slide? I think some of what I want to tell is probably still on here. So Robert, yeah, exactly. So you can go back up again, sorry, this slide, yes. So you know, this is, again, this is a very old set of slides, so this has changed quite a bit since then. But you could see originally with the raw spot help menu, it came installed with a list of default functions that came with the chat bot. But of course the majority of that we uninstall, I mean obviously if you see something like raw spot shell command as a hacker, that's not the kind of thing you're going to get really happy with. But we basically tossed out the majority of that stuff and instead retooled it with the things that we needed, sorry, I'm on the train. So if you can then go down to the next slide, excellent. So you know, this is basically an illustration of this XML-10-test reporting system that I was talking about, this OAuth 10-text. So what it is, is it starts with, for example, if we want to write a quotation. So if we are making a quote, what we do is we start out with this thing called quick scope. And what you can do is you fill in an A4's worth of text and then what you can do is type a command into the chat bot that basically says raw spot and we call it raw spot because we're radically open security and that's basically the name of our chat bot. So we say raw spot, quick scope and then the name of, for example, a test repository. At that point, what the chat bot does is it takes that A4's worth of XML and then it expands it into 13 pages of XML and that's the 13 pages of XML basically contain all of the boiler plate, so of course all that boring stuff that we automated away. And then at that point, if you want to make like little fiddly changes to the boiler plate, that's the moment of which you can do it. The next thing that you can do is you can then type in another command into the chat bot and then basically that would say raw spot, builds and then the name of the PDF. Actually, can you go down to the next slide, Robert? So here's an example of the expanded XML, okay, go down to the next slide. Yeah, and after you hit compile, this is basically what's coming out. And as you can see, this pentest report looks pretty much identical to every other pentest report in the universe basically. It contains all the necessary things, indexing, classification, threat levels, short description, technical description, and also remediation advice. Can you go down again to the next slide? Yeah, so again, I apologize, this is really super bad. It's super ancient, you've replaced all this stuff since then, we don't each have a command. Anyway, I apologize that you have an ancient version of this back, but in either case you can I think still get the idea. What we do have these days is a command called raw spot build, and then you put in the name of the repo. Now what happens is that raw spot at this point, it actually, you can see it's doing, the first thing it does is doing a git clone. So it actually takes the git repository in which the XML has been pushed. It clones it, and then on the server locally on that side, it then runs the XML toolchain to essentially compile everything. And then what it does is it spits out a clickable link that then at that point, anybody who is in that chat room can actually click on it, and then it opens up basically that PDF. So that's actually really super handy, because anyone who is in that channel can automatically see what you're doing. In the newer versions of that guy, I apologize that you have the wrong version, but also another thing that it does is it automatically password protects the deck. It would be much better if I could share my screen actually, and see if I can. The only thing is I'm on this browser-only version of this. Yeah, I'm not sure if it needs the download, the plug-in, would be able to present what's coming over. Yeah, sorry. I wish I could get you a new version of the deck right now, but it's probably not time. Yeah, but it's really not real. We can post it on the meeting notes after the call. Yeah, that would be better, because yeah, I apologize that my project team sent me a version. Anyway, but the point is that, yeah, exactly, actually there's even better example reports that you can look at than the one that you just posted, Chase. If you go to our website, so basically RadicallyOpenSecurity.com, if you go up to the top, then you're going to see basically a portfolio section, and this is actually a far more recent version. What you can see is quite a few different, like for example, click on the XFAT XML parser report. This is the most recent version of what comes out. You can actually see that this is what comes out of it. Of course, ignore the content for a moment, because this is not relevant. This is just a job we did for open tech phones, or maybe for Mozilla Foundation. Oh, okay, never mind. I see we did it for months. Scroll down a bit more. Oh, could you scroll down a bit more in the PDF? Robert? Oh, hello? Are you not seeing my screen, or no? Oh, sorry. Can you guys still hear me? Yeah, we can hear you. Yeah. We're looking at some live XFAT. Okay. Ah, okay. So basically, what you can see is basically, again, it looks more or less like every head house report in the universe, including automatically indexed findings. It automatically generates all the summary tapes, it automatically generates also statistics and whatnot. So anyway, but this is enough. I think this is maybe composed to this week.mozilla.org into the Zoom group chat, then people can look at it later at their leisure. Yeah, it's in the chat, thanks. Sorry? It's posted in the chat. Okay, cool. I was just saying it's in the chat. Excellent. So let's go back to the PowerPoint presentation. Are you seeing the slides again? Yeah, Robert, could you go back to the PowerPoint presentation? Hmm, okay. I think it is. But is everyone else seeing the pen testing slide? Pen testing chat ops too? Yes. Okay. There might be some lag on your screen, Dr. Reebok. Yeah. Okay. For some reason, I'm still on the PDF, but okay. Yeah, I think we're all seeing the slides again. Okay. I can't see the slides for some reason. I'm seeing the slides, so I think it might be a lag on the train internet. Yeah. Yeah. It's possible. All right. Well, anyway, so going back to the slides, I can't actually see them right now. But let me see if I can maybe refresh this. Give me a minute. It would be kind of handy if I can see the slides. You know what? I'm going to leave the meeting and then log in again. See if that helps. Hold on a minute. Sorry. I'm called in on my phone, so that's going to rejoin the meeting again. Sorry. Rejoining the meeting again. All right. Now I can see the slides again. Excellent. So, basically, at this point, you can see on the slide that this is the Git lab repository, just sort of on the back end of that, well, basically up at the test report. Most of the time, you don't really need to use it, but it just gives you an idea of what the structure is like. So, can you go on to the next slide? Okay. So, obviously, you can use a chatbot for something along the lines of test reporting. That's great, because that's something that any test team needs to do. But then that really begs the question, what else can you do? So, here's another example. At one point in time, we did an assignment where we needed to do some passive scanning. So, we built a tool. We also open sourced it. And this takes the output of showdom and census, formerly scams.io. And it correlates this information. And this is then something that we can launch from the chatbot. So, as you can see, you know, this is starting to get away from reporting and to get a bit more towards actually that are functional. And again, it's all sort of starting to launch via that chatbot. So, could you please move on to the next slide? Great. So, this is something else that the chatbot makes possible. So, we've got this concept that we developed called red-blue pen testing. Now, what that means, of course, I'm sure you all are familiar already with things like red team exercises or blue team exercises. We take a group of developers, participants, and DevOps folks. And we basically take about a dozen of them. We split them into two teams. And we actually gamify their pen test. So, they actually compete against each other to see who is best at hacking their own stuff. So, it's a really unique experience, you know, for these engineers because, you know, they are used to being in the role, of course, of developer. And it really takes them out of their normal role. It puts them in the shoes of the hacker. And because they're competing, you know, it puts them in a totally different mindset. Now, each of the teams is guided by one of our professional pen testers that rapidly opens security. And the number one comment that we tend to get from these developers once the exercise is over is, and I quote, I will never look as code the same way again. And that's why we do it, you know? So, you can see right here how we're actually making use of chat ops to build the scoreboard. So, in this particular case, I type in a command, good job blue. The bot at that point then says incremented blue, 24 points. It prints out a motivational image. And then, you know, again, this is just to keep people motivated, you know, to be hacking, to put them into that place of gamifying it. And, you know, this is the kind of thing that you can do with chat ops that is very, very difficult to be able to do with a physical room. Okay? Can you go to the next slide, please? Actually, it turns out that there's much, much, much more that you can do with a chat bot. You can run, you know, tools for, you know, things like scanning and explication. So, port scanning, web vulnerability scanning like W3AF, SQL injection tools like SQLMAP, forcing passwords with Hydra or other related tools. Of course, the word list and everything go on the server, but it can give you a single unified interface that you can use to run all these tools. It's super handy. Also, in terms of reconnaissance, you can do things like who is Google, the passive scanning tool, you know, let me duck that, go that for you, you know, whatever. And then you can basically do a bunch of those kinds of more standard recon using tools like that. And again, all of this can be invoked from the chatbots. Similarly, there's other exploitation things that we can also do via the chatbots. So for example, hash-cracking, you know, this is a very common thing that pentest teams do in the course of an engagement. But, you know, what we can do is we have actually implemented rainbow tables on our server. So what that means is I can take my device and it can not necessarily even just be my laptop, it could be my cell phone, it could be, you know, even a smartwatch, if I can get it through the couple layers of authentication to get into our chat environment. Basically, any web-enabled device is capable of interfacing with this chat-op system, which means that I can be sitting on a bus, you know, somewhere in the middle of Lord knows where in the Netherlands, like I am right now, actually. And, you know, I can then run rainbow tables against your hashes, you know, for my cell phone. You know, and this is power, you know, this is the ability to be able to work from anywhere, you know, and really to be able to bring together a distributed team. Now, sorry, one, two, three. Is that the train safety lecture? Sorry about this. So anyway, but, well, sorry. I guess some of this chaos kind of makes my point. Sorry, I'm changing trains right now because I have to go back to Amsterdam. But the point is that we really can work from anywhere, you know, and when I say from anywhere, I do really mean from anywhere. So the idea then is that we developed also this concept that is called peak over our shoulder. Okay. And of course, the, you know, the title of this talk was really discussing openness and transparency and pentesting. So radically open security, you know, we are really, you know, the company was started because of some discontent that I had about the pentest industry as it is. So I used to actually work in the cyber crime team at ING Bank. And the, you know, we had certain incidents. And then during these incidents, what happened is that some of the larger security consultancy firms were called. They came in and then they kind of had this attitude of, you know, we're the experts, you know, we're leads. Security is hard. Stand back. We're going to solve everything for you. You know, and then we're going to give you the report and then the big invoice and, you know, everything like that. You know, at that time, I also said to them, you know, well, okay, great. You know, my name is Melanie and I work for the cyber crime team of the bank. So if you guys are so lead. Actually, can you go back to the previous slide? Sorry. I'm not on this one yet. So thanks. So, so basically, you know, if you guys are so lead, that means there's probably a whole lot that I can learn from you. So, you know, what I would actually like to do is watch what you're doing, you know, and well, to make a long story short, you know, these consultancy companies, the really large corporates really did not like that. So, you know, they did things like throwing away bash history logs, working in screen and like quote unquote forgetting to turn on logging, even though I reminded them like, you know, 10 times to do it. And then ultimately, you know, I literally stood next to them and looked over their shoulder, because to be quite honest, that was the only way that they couldn't get rid of me. So, you know, I mean, slightly annoying. But anyway, you know, I was a bit upset about this because, you know, for me, security is a process. Security is a mindset. You know, security is not a set of band-aids that one applies, you know, to, you know, fix whatever errors that you happen to have this time. I mean, you actually need to convey that hacker mindset. Otherwise, the developers and other folks are going to continue making the same errors, you know, next time. So, basically, sorry, just give me a second. I'm going to wait till after the announcement. Okay, I think they're done. So, yeah, so basically, you know, and I just didn't really like how commercial these companies were acting. You know, I mean, the culmination of this was I actually bought a box of analysis tools from one of these companies. And I, you know, after we purchased the box, I wanted to log in. And I asked them, you know, hey, could I perhaps have the password so I can log in to this box? And they were like, yeah, but you know, there's like proprietary tools on here, so we can't give you the password. At which point I was just like, well, you know, WTF did we just buy? You know, so it was sort of this arrogance, you know, and lack of willingness to educate. You know, that actually was one of the reasons why I made radical, I started radically open security in the first place. I mean, I guess now is probably a good time to talk a little bit about my company's business model. I don't want to talk about it too much because it's a long story and that's not that this presentation this evening is about. But some background information that you should know is that radically open security is a not-for-profit computer security consultancy company. This is weird. Needless to say, it's extremely weird. You know, and it's also confusing, right? What do we mean by not-for-profit? Well, it turns out that, you know, I wanted radically open security when I first started it. I wanted it to be different, you know, but I didn't know how to do that. So I went to my friend Michiel Lainers from the NLNet Foundation and I said to him, you know, I want to make this different, but I don't know how. And Michiel said, well, you know, there's actually this very interesting tax construction from the Dutch church. So, you know, basically what it is, it's essentially, as he said, it's sort of a kind of tax designation. And it's called, in Dutch, a fiscal forms there from the instilling in English, a fiscal fundraising institution, an SFI. Now, what this means is that sometimes a kind of feels like we're in the Netherlands, right? For all of us. Anyway, so let's continue on. So basically, actually, sorry, what was the last thing I was saying before the announcement? Okay, right, the SFI. So basically, we, so sometimes the church wants to do a commercial spin-off. So, you know, they'll like make some counties or something and then they're going to raise some money and that money goes back to the church with the tax benefits. Now, what happens is that, there's an example of this in the Netherlands. And it is this language institute that's called Regina Taley. It's otherwise known as the Nans of Bucht, and this is a pretty well-known language institute. And basically, the Nans started the language institute. It's a really great independently operating language institute of really high quality. In fact, our queen in the Netherlands learned some of her Dutch there. She's originally from Argentina. And what is this? I missed the intro. This is Melanie. Who? I'm not sure how to answer that question. Sorry. I'll add in the chat. Sorry? I'll add in info in the chat. Oh, okay, cool. All right. So basically, so that's the kind of the way it works. These Nans started the language institute. The language institute earns money and then that money goes back to the Nans again. That's the way this FFI construction works. So I said, you know, that's really interesting. I'm going to take this FFI construction and I'm going to make my commercial spin-off here to consultancy company and I'm going to make my so-called church the NLNet foundation. Okay? So NLNet is a Dutch charitable foundation that it actually was originally one of the first ISPs of the Netherlands. And it was acquired by what's currently our national telecom company and basically the whole thing got turned into a foundation. And for the last 20 years they've been giving that money from that acquisition away. So they fund projects like the EFF, GNU, Tor, Chitzi, DNSX, WireGuard, Academic Research. I mean basically anything for, you know, digital rights, open source and anything for a better internet. So basically what I did is I ensured that 90% or more of my profits legally had to go to NLNet. So basically the profits for my company would go in full to the community from which my hackers originally came from. So, you know, of course, which is weird, I think, you know, the last 10% of course is basically our cashflow buffer and that is what enables me to actually run a business with it. So we are a social enterprise but it actually goes further than that in the sense that we actually give basically all of our profit away to charity. And of course there's very few social enterprises that do, that really goes that far in trying to make sure that that money goes back to social interest. So, yeah, so basically that is sort of the weird story about my business model. But then if you want to hear more by the way, I gave a talk in June at TEDx Berlin and the talk is called Post-Growth Entrepreneurship. If you Google it, you can find it. And if you all want to know more about sort of the story of my company and how we're nonprofits, I would recommend that you listen to that TED talk. I don't want to say too much more about it right now because again I want to focus on security to see a thing and I don't want to tell a business model story to you but at least I think it's useful to understand this much to help you understand the context of my company better. So anyway, but we have core values of openness, transparency, and open source. So basically all of the tools, the software, the architectures, everything that we create, also documentation, legal documents, trainings, everything that we create, we optimize for releasing it directly into the open source. So if you look at our GitHub repository for radically open security, it's super active. I mean we also have a GitLab.com repository as well which also has some other pieces because we use GitLab CI for some things. But basically, yeah, in terms of openness and transparency, that's when we came up with this whole idea of the peak over our shoulder workflow for penetration testing. So the way that it works is, you know, I already told the story about how I was looking over the shoulder of that consultancy company when I was working at the bank. Well the idea is we invite customers to join us in our chat rooms so they can actually overhear every single conversation that our pen testers are having. And so they can also see via the chatbots that every single action, you know, basically our operational action that my hackers are taking. You know, because there's the chatbot and we've got it hooked up to the GitLab repository. So every time one of our pen testers makes a comment, for example, on a GitLab issue for that repository, the chatbot then says, you know, this pen tester just made this comment at this timestamp click here for more information. And then if you click on it, as long as you've got access to that repository, it takes you there and you can actually read the comments that that pen tester just made. Same thing with pushing scan results into the repository. Again, it's just using Slack, you know, basically the Slack API. And, you know, so what it does is it provides this constant feed of, you know, blow by blow exactly what our hackers are doing. You know, and this provides, you know, unprecedented transparency into the pen testing process. You know, and it sort of takes that black box of security consultancy and it actually explodes it inside out and it makes it so transparent that it's almost annoying. You know, I mean, that's basically our objective. But it turns out that, you know, it is a really popular formula and it's actually gotten us a lot of customers and our customers are super happy with it. You know, we've got a very high retention rate of customers. You know, and we, you know, it turns out actually that customers like being educated. You know, I mean, they like being treated with respect. They like, you know, if you optimize for knowledge transfer. You know, I mean, it's kind of nice if, you know, almost every single pen test that you commission that it's almost like a training exercise at the same time that you're actually receiving a pen test. And it turns out that it's actually value for money. That's really hard for the corporates to compete with, you know, because, you know, we've automated it as such that the openness and the transparency is actually built into our infrastructure and facilitated by the chat room and this chat box. So we can basically, in the course of doing our normal work, you know, in the normal way, it's designed, you know, to bring that transparency to the customer. We don't even have to do anything extra to include them. And of course, you know, because of this, you can imagine how attractive this is. So going back to the slides, you can see at the bottom of the exploitation bullet point, it says spear fishing. So here's an example of one thing that we built that makes use of this peak over our shoulder. So we do fishing exercises for lots of different kinds of customers. And one thing that we can do is we have built a fishing suite. This is also, by the way, open source. This is available again in GitHub. And what it does is it's a set of scripts. Now the first script can be invoked from the chat box. It can take, you can basically fill in a URL or push a newsletter basically into a GitLab repository. And then what happens is it can actually scrape that for you and then add instrumented links. Now the instrumented links basically go to our web server. And of course, the web server is connected via the Slack API with our chat. So the moment that a subject, you know, basically that's receiving this email, or not the subject, I'm sorry, one of the receivers clicks on a link in this email, the Slack API sends a notification to RocketChat, and prints out a message saying, you know, this email recipient address just clicked on this pretext name at this timestamp. So you can literally watch while your own staff is getting fished in real time. You know, this is great, you know. And again, none of this would actually be possible if we did have chat ops. Similarly, you know, if you use forms like for example sometimes we'll have fake Office 365 forms or AD logins or these kinds of things. And, you know, again, people can fill in the forms and then in real time the chat bot can say, you know, these are the credentials that were just entered into this form, you know, from this email address. I mean, this is actually really a lot of fun, you know, so the security officers can sit there and watch people playing with the form, even the fishing form that you just sent them. I mean, at one particular occasion, I can remember that we were fishing a hosting provider. And at a certain point one of the technical members of staff noticed that something in the domain name wasn't quite right, right? It's a fishing test after all. So at a certain point this guy started playing with the form. So we could see him putting in things like, you know, username, I don't know, something, you know, password, something else. Username, I don't know, woohoo, password, nice tri-assles, you know. Username, I don't know, some sort of thing. And then password, SQL injection. You know, no, the SQL injection didn't work, but you know, he definitely got points for trying. You know, of course that begs the question of, what's this guy in a sandbox environment while he was playing with our form? But nonetheless, you know, at a certain point we kind of raked from hunch. And the security officers were like, yeah, man, thanks. That was a lot of fun. Yeah, that was quite entertaining. So, yeah, and again, these are the kinds of experiences that I think really only are possible with this kind of form of peek over our shoulder, you know, with actually including the user in the chat room. But I think there's actually much more that you can do. Anyway, can you go to the next slide, please? But there's actually much, much more that you can do via the chat ops. You can also use it for all of the other kinds of operations that you would have for just running a business. You know, radically open security, just like any IT shop, you know, we need project management. And like some companies that we use CanVan, basically to manage our workflow, you know, we've got a lot of different projects going on, so we need some way of being able to make sure that the work is flowing through the system. And we also use sometimes the open source project management software CanVan which is basically an online CanVan board. We've actually made it so that we can dump the content of the CanVan board from a command to the chat bot. And we can even do other things. This is kind of fun. I don't know if any of you are familiar with the Ship It Squirrel. Have any of you heard of the Ship It Squirrel? Yes. Okay. So somebody knows about the Ship It Squirrel. So the Ship It Squirrel is basically this really silly geek thing. I can't really call it anything else. And if you type in Ship It, then the chat bot spits out the squirrel. Yes. Unfortunately, this doesn't work in Zoom. But yeah, you basically get a squirrel. And every time you get like a different kind of squirrel. Anyway, it's really silly. But you know, when you have geeks in your company, we kind of like celebrating shipping things. So, you know, and silly little memes with squirrels and them are as good as any celebration. So, but the funny thing is, we actually took that Ship It command, and we actually coupled it on taking the status of that project, basically based upon the chat room it was in, and then moving that project status automatically to done. So it was actually a really great way of getting lazy pentesters to update the Kenboard, you know, who typically, of course, don't want to be bothered with such things. But again, it's the kind of streamlining that you can do with, you know, with the chat bots. Similarly, we've also got a system called Git Notes that we developed. And it's a bit like a support desk service, so think ZenDesk, you know, that kind of thing. We've got a number of magic email addresses. And if we are performing a particular assignment, and we're corresponding with a customer's business unit, you know, we've got some customers, of course, with whom we get a very large number of pentests. And we, of course, have to correspond with our different business units. And then the security officers want to know, they want to be looked in on this communication with the various business units. So as long as we can CC or BCC this magic email address in our correspondence, then what happens is the mail server also again uses the Slack API to put hooks into our rocket chat so the chat bot can then spit a message into the chat room saying that this message from this sender with this subject line was just sent to this timestamp. Click here for more information. And if you click it, it takes you to GitLab where a copy of that correspondence was automatically pushed into the repository in the notes directory. You know, and this basically enables everyone who is in that chat room and who also has access to the corresponding directory to literally with zero extra work on our part except for using these magic email addresses to be looped in, you know, to all the correspondence going on in both sides for that project. It was super handy, you know, super efficient. And again, it's chat ops, you know, that's making this possible. Same thing also with our tracking. Of course, if there's anything that hackers hate, it's administrative overhead. So things like keeping track of billable hours is the kind of thing that pen testers hate. However, we've made it really painless using this chat bot so you can say essentially raw spot charge the number of hours and then a short description of what just happened. At that point, the chat bot comes back to you and it says thank you, pen tester name. You just charged and hours. There's now M hours remaining in your pen test. You have now completed L% of the total pen test and then it spits out a progress bar basically showing you how far along you are. Of course, this is now super easy for the pen tester because all he had to do was, he or she had to do was type in this one command to the chat bot. But beyond that, this is also great for the customers because remember, the customers are in our chat room. And when you're doing consultancy, of course, you have a time box and that means that time is money. So if the customers can actually see in real time exactly what the time is being spent on, then they know exactly where that money is going. And customers love this. Similarly, if we, for example, are approaching the end of the pen test and let's say that we're at the 50% point and let's say we made a scoping error, it can happen. Scoping is more of an art than a science and occasionally it's easy to underestimate things or perhaps the situation is a bit more complex than we anticipated. So basically, in this particular case, we can say to the customer, dear customer, it looks like we underestimated the amount of work that this is going to be. We're now at the halfway point. We need to adjust the scope and dear customer, can you please tell us what your priorities are so we can make sure in the second half that we can actually focus on the things that you care about and that we can decide together which bits are going to be relegated as future work. So this gives customers control and this is something that, of course, the customers really, really like. It gives them knobs that they can turn. It gives them choices they can make and ultimately, again, it's nice because it really enables us to be able to do expectation management with the customers and even when things go wrong, the great thing about this process is to see things going wrong from a million miles away and they can also be part of solving it. So this also tends to make sort of very happy customers. If anything, just for the reason that they feel like they have more control and they can actually steer our pen test to a certain degree. Obviously, they're not steering us completely. That also wouldn't necessarily be the intention. We're pen testers and also we've got ideas about the things that are useful to be done but all the same, of course, half of pen testing and half of security as it is is just context and of course, the customer has the context. We don't. So it makes us able to really operate far more effectively. By the way, can you also hear me? I just lost Zoom for some reason. We can hear you. At least I can. Cool. I'm just going to give you one second to see if I can join this meeting again. I can't reach the Zoom server for some reason. Oh well. Anyway, so for some reason I can't find the Zoom server anymore. Oh well. Sorry. But in either case, this is really excellent for expectation management. All of you have companies and it's worth, I believe, asking the question, how can I use this openness and transparency in my own company? How can I take my own activity to also a security company or perhaps you're a hosting provider or maybe you're a development company? I mean, stop and think. How can including my customers into the process and really bringing them along and optimizing for maximum education, how can this make them happier? And what you're going to find, again, is this openness and transparency, this has market value. And it really leads to, I think, a win situation. I mean, people do ask sometimes isn't the customer annoying in the chat room? Don't they ask too many questions? Things like that. My answer is no, absolutely not. These customers are not annoying. And furthermore, not only are they not annoying, but they're actually super helpful because a customer is like an oracle. It's like having an oracle in your channel. And then you can ask questions like could you please reboot this server it's not working? Or what does this function do? Or how does this code path work and why? Or are you using a strong password here so we can know whether or not it's worth our effort to attempt to brute force it? And that's the thing. I mean, when you're working within a time box, of course anything that you can do to improve the feedback loop really makes for a better experience for the customer. So, yeah. So ultimately... Sorry. Let this announcement finish. And actually we can't really hear the announcer. I think the audio is pretty good. Oh, okay. Okay, the announcement isn't bothering you. Okay. Well anyway, but the long and the short of it is and I think we're also almost out of time because this was supposed to go till 8pm, right? Yeah, we're supposed to end at the hour. Though I think folks can stick around if they have questions if you're available. Okay, cool. Then basically just let me make a few more points and I'm going to wrap this up. So essentially what I'm describing means, you know, this is all great, you know, for openness and transparency, but of course what it also means is that your infrastructure now is production. And when I say production, in the full sense of the word, if your infrastructure goes offline, guess what? Your whole company goes offline. So it does mean that you do need to, you know, treat it as such. And you know, in that sense, I mean, radically open security is also a DevOps company. It has to be. Because again, you know, we can't afford things like outages. So we most recently have been busy with quite a large DevOps chattering. We would basically be busy refactoring everything, put things into Docker containers, make sure that people can launch, you know, their own instances of all this stuff, just quite simply. Also for isolation, containerization, also for security reasons. You know, obviously access control is also a point because you don't want to, you know, for example, customers executing commands to do things like read out your project management keyboard, obviously. So we use the role-based access control for this. And we also have some accessory kinds of channels like error logging, debug logging, you know, things where basically development staff can actually get access to certain error logs and server logs, but without actually getting a password and a logging account on the server. So, anyway, I think that's probably enough that I'm going to say about the functionality, but the last thing that I think is really worth mentioning is the fact that this whole concept has been really very successful. So, you know, we've won a lot of prizes for our workflow, you know, and on top of that, you know, in basically five and a half years time, we have grown, you know, radically open security has grown to a company with 40 staff members close to probably, gosh, by now probably close to almost 100 customers. And we also have won multiple awards, okay? You know, we are at this point in time a preferred supplier for Google. And also, you know, for Mozilla and the Moss Project, Open Tech Fund in the United States, but also in the Netherlands, banks, insurance companies, hosting providers, health codes, you know, but also, SMEs, universities, and also for the nonprofit sector, we do work at cost price for NGOs, nonprofits, and civil society. So, we've got a huge variety of customers. And the other thing also is, again, we've won quite a few prizes. The Dutch Chamber of Commerce called us one of the 50th most innovative SMEs in the Netherlands. Christine also called me one of the, well basically the most innovative IT leader of the Netherlands. And also, the European Commission called me one of the nine most innovative women in the European Union. Sixth, you know, which is kind of interesting, given the fact that I'm actually not even European, I'm American, but okay, good. But the point is that, you know, between the open and transparent testing workflow that we've tested, or that we've developed, and also the business model behind it, in dividend-ending out to essentially our profits to charity, which is basically the implementation of basically of Mohammed Eunice's vision of type 2 social business, you know, we're really kind of onto something, you know, and you know, you know, this has led to some much larger questions, including, you know, thinking about the business in general, thinking about Silicon Valley, because of course, you know, we all work for tech companies, and we know what influence the whole capital scale exit model of you know, building tech businesses, what kind of impact that has. So, yeah, I mean this has led for me to this really huge journey, you know, this journey is still continuing, which has just gotten weird in the recent years, including my actually developing a brand new course for the business school at the Free University of Amsterdam, and of course, you know, as a former assistant professor of computer science, I never thought that I was going to be developing business courses, you know, but it's actually worth 6 European credit points in 2020, you know, so all this stuff is going on. I mean it all started with pentesting, but in the end, you know, it's about social business, it's about openness and transparency, and I think that there's a lot of lessons that anyone can take from this. So anyway, I think that this is enough in terms of presenting, and I was thinking maybe now might be a good time to open this up for questions. By the way, I did close my laptop, so if somebody types in a question, if somebody could read that to me, that would be really helpful. Yes, happy to do that, and thank you for presenting, and especially at the end of a long day and on the trains of answers and whatnot. Sorry, this is a bit more hectic than I'm used to, but I apologize for the slight inconvenience, but then again this is actually so representative of how life actually is when you have a distributed company and just, you know, all life is crazy, if you know what I mean. So Sarah, from a meeting perspective, can we keep the line open, or does it terminate? Yeah, we can just keep it open, so we understand if people have to take off, we'll capture the questions and answers in the notes. But yeah, let's open the floor for questions. Hey, it's Mark Underwood if I could jump in, and great presentation. I apologize for asking who the heck you are in the middle of the presentation. So you know, the refactoring process caught my eye because there's organizational refactoring in order to support DevSecOps in general, and I'm wondering where you see this process in light of what we've got to do with the CI CD pipeline building anyway. Like, I'm inclined to see red teaming and maybe this gamification process as really part of the CI CD pipeline and not a separate process like it used to be. Yeah, yeah, absolutely. Yeah, we also do some DevSecOps work for our customers and definitely, you know, thinking in terms of DevOps, you know, it's not just that we're trying to patch things, but it's that we're trying to create tests for things so we can test for that condition and fix the problem forever. So, yeah, I mean, but, you know, but the same way also with our own infrastructure, I mean, again, this stuff is production and also for OPSEC reasons. You know, I mean, we need to make sure that we're not getting regressions. We need to make sure that things are not spaghetti, which of course at the beginning they were. I mean, you know, when we first developed it, but the last like year and a half we've been basically all of our development has been been paying off technical debt, you know, but it can't be any other way, you know, because, I mean, ultimately, we too are a software company and you have to put that time in, you know, just to make sure that everything remains manageable because also security relies on it. And just to close the thought and yield the floor to everybody else here, you know, from the cloud native point of view, we are trying to understand what the tools that come to us should have for hooks to support processes like the ones you describe in this, you know, and so some of it is test frameworks and, you know, access to logs and transparency about the logging process and management of access. So, you know, I think there's a lot to be learned from this, but maybe the biggest one is just following the standards that are in OWASP and other places. Would you say that? Yeah, I mean, there's definitely a lot of best practices. And I do believe personally that the DevOps community gets a lot of things right. And I think also that the security is well served by learning from their lessons. Thanks so much. Sure. Any other questions? Well, I guess to follow up on Mark's question, do you have, so we have a lot of folks in the room who develop software tools as well as people who are security experts who do similar to what you're describing. For the tool vendors, are there things that you wish that open source and their vendors who have open source projects, most of us, or contribute to them? So for what would you wish that more tools that you use did right, or more open source projects did that would facilitate chat ops and the work that you're doing? Well, definitely any kind of integration with the Slack API you know, because I mean, of course mostly we've been setting this up ourselves. And essentially you can really use the Slack API to hook up to pretty much anything. I mean, the chat bots can essentially run shell scripts from there. You can invoke almost everything. There are scores which have GUIs. And of course GUIs are a bit more problematic in trying to hook that up with chat ops. You know, if you're using something like say, Burp Sweep which is one of the very few tools, not open source tools that I sort of allow my hackers to use. So I didn't hear that. Sorry, I didn't hear the word. Oh, I said Burp Sweep. Burp Pro. Just because, you know, I mean, that is so in the standard. I don't want to depang my hackers by saying they can't use that because it's not open source. The other exception, the commercial tool that it's okay to use with my company is Burp Pro. Again, same reason for reversing. It's the industry standard. I don't want to take that away from them. Except for that though, we basically use open source for everything else. For the most part, I mean, I think, you know, it does pretty much what we need. Of course, I think one of the more important things also for us is if we do find security problems that they're responsive. Rocket chat, for example, we have found a number of zero days in Rocket chat and we reported it to them and I have to say they've been super excellent about fixing them. So, you know, in fact, they were so super excellent about it that at a certain point they actually decided to hire us as a security vendor because we were doing so much pounding on Rocket chat. They figured they might as well at least pay us for it. So, but yeah, but basically that helps, of course, because you know, as this is production, you know, it's also a tax surface but, you know, just like with any open source, we need to make sure that we can examine it look for vulnerability which inevitably we're going to find but when we do, we need to make sure we can report them and make sure that those things are fixed pretty quickly. So, I would say that that's also pretty important for any open source tool we're going to use. So, one just like detail. So, you said integrate with the Slack API. Does that mean that the other like IRC tools integrate with the Slack API? Like, so you're using Rocket chat. Like, I've made a Slack bot but I haven't and I've made an IRC bot but I haven't really looked into if I were had an open source project and I wanted to integrate with chat ops is there one API that's supported by multiple vendors? Um, I mean other than yeah, I mean other than explicitly supporting a Slack API in some way, shape, or form not so much. I mean, I think if you just can allow for your tools to be able to be run on the command line then we can work with it and that's what I'm saying about anything with a GUI like, like IDA or like, well maybe not IDA but things like Burp Suite you know, I mean it's, you can't really invoke that on the command line just because it's, you know, it's very much point and click based for the most part. So, those are things that are difficult to integrate into chat ops because you can basically invoke most things, you know, with just using command line scripting and as long as you can write a script to invoke a tool then it's something you can easily integrate into your chat bot. It's just anything that involves a GUI you know, where you can't quite as easily script it that those kinds of things are more difficult to integrate with chat ops. Great, thanks. Sure. Any more questions? Well, I wanted to give everyone else, I could probably keep you here another hour with questions. I don't know if they're monopolized either your time or the floor so I'll let anyone else do questions but if not I have a few. Okay. Anyone? Quick question, the use of rocket chat versus Slack. Yes. The use of rocket chat versus Slack. So basically I don't use Slack and why do I not use Slack? I use the cloud and because data governance matters to me. You know, I mean it's just like they say there is no cloud there's just other people's computers. I mean I'm dealing with people's penetration testing data. I can't put my customers pen test data into some other third party companies cloud. I just can't do it. My ethical responsibility dictates that I can't do that. You know and beyond that I believe if I'm not mistaken that Slack was just acquired. I forget if it was Facebook or someone else but I know one of those companies I believe just acquired them. I mean that's precisely the reason why I've never used Slack. I mean on top of that also of course you know, rocket chat is free. I mean both free isn't freedom but also free isn't beer and that's also kind of nice. You know for the same reason why I think it's just about the Elk stack and not Splunk and things like that. Okay so just to clarify and pull everything together we've been talking about Slack API we've been talking about rocket chat are we saying and the question earlier about compatibility I think was more about compatibility between the chat tool and the bot than about the bot and the world so I guess the question is you know are we saying are we inferring from that? Are you implying that rocket chat supports compatible API with Slack? Yeah, no it does. I mean rocket chat explicitly supports the Slack API. I just wanted to you know we're navigating all around that I just wanted to be sure that was on the table in so many words. Yep. Yeah that's a feature of rocket chat. So a question is coming in do you do security audits for CNCF and what resources do you recommend to project maintainers to instill a security mindset with their projects? I guess that's too separate. Okay, yeah I've never done tests as such for CNCF but it would be open-minded to doing so. I mean if anybody wants to test I mean email afterwards I mean please. That being said what signs of things would I recommend that folks use for instilling security awareness? You know I think that it starts with the developers and I think you know really making sure that I think security doesn't stay in the ivory tower of the CISO department I mean it's all well and good if we're training our security officers but at the same time I mean I think you need to also have this knowledge in the dev teams, in the business also to really make a really meaningful difference. I also think that it's useful to think about for the limited security budget you have really just to think about how you're spending it because you can spend it on black boxes you know I mean there's a million things like monitoring black boxes that are out there but at the end of the day I just don't think that is really much value to you partly because vendor lock-in if you ever leave a vendor and it's closed software of course you're going to lose all the aggregate knowledge and usage of that tool which makes actually very little sense you know I also think that you know anyway I think it's better to always use open source if anything just so you can actually look at the software see what it's doing understand it, audit it make sure it's not doing anything unexpected also in terms of data governance another problem with some of these security black boxes is Lord knows where it's actually sending your data and what that third party is doing with it and I think also that you know open source you know we don't need black boxes we need white boxes and for me personally I'm really personally of the belief that anything you can do to sort of increase your usage of open source and the amount of open source in your organization that's only going to benefit you security-wise you know I mean surprisingly enough at one point in time I spoke with somebody from the FBI you know not the most open of organizations but they actually said that the FBI makes heavy use of open source and you would ask the question well why is that quite simply I mean the answer is so much predictable it's because if they can't look in the source code they don't trust it and as the FBI they want to be able to audit whatever software that they're using for all of their top secret stuff that they're doing and again if you think about it it makes sense all of radically open security runs on open source that's probably the exception of VerbSweetNidaPro but the entire rest of the company that's what we use and I think it really is usable for anything that you need and yeah I think we can run a really top notch penetration testing company using that I think if open source is good enough for us it's probably also good enough for you guys and it also then because it's also free isn't beer it enables us to take our limited security budget and stretch it further and really to be able to use it on meaningful things rather than just large licensing fees which I think are pretty unnecessary it's far better I think to actually invest that into training your staff because half of security is really internal context and you really need folks that understand the business installing and figuring any kind of monitoring boxes but also being able to perform security audits I mean you need half the battle with security is always context I mean the devil's in the details so I think again get that money that budget away from the licensing fees instead put that into making your own organization more knowledgeable and I think that that already will help to get you a whole lot further do we lose you it's possible that we lost connectivity hello okay maybe we didn't lose you my question if you have time for another question as we from kind of the CNF security perspective is we're trying to create a security assessment process for open one of the conflicts that we've seen or I've seen is that you know as an open source project is largely a volunteer even if their company is sponsoring their time or directly payment there's this kind of this value for money you mentioned I'm not getting paid to write this code my user space wants to have features and yes security is kind of this thing that everybody expects how do you advise our type of organization is trying to foster security with these open source projects like how can we pitch that value for time that a security audit a security assessment is really worth the extra effort right well look I mean you get what you pay for and if you send your budget to open source maintainers then they're going to continue working on the project you know and if for whatever reason they would drop it I mean somebody else could fork it and then you could pay them to continue it you know I mean I think in the end that is provides probably better continuity and you know a better chance of not losing access to whatever it is that you've invested in if you stop paying so I mean of course open source isn't perfect of course security problems I mean just like any software does it's not perfect but all the same you know if you need something tweaked in your software you know something improved something customized I think you're going to have a far greater chance of actually getting what you want if you pay an open source maintainer to do that work for you versus you know if you're going to give that money to say I don't know Microsoft or you know one of those big corporates and then expecting that they somehow are going to customize their system for you that probably is most likely not going to happen so so yeah there's another angle to that question which is like so we have a number of open source projects that are focused on security right and of course they have security expert maintainers who are very thoughtful about the details and we might quibble about certain details but it's in the details right and then there are other projects where their purpose is not security and sometimes a maintainer of the project who might be getting paid from a specific company that cares about that project is like well yeah security is not our concern and so one of the things we're working on is how to positively evangelize that you know without you know like there's the hammer kind of things where we can you know put your things as requirements are being part of the CNCF but we prefer to make you know this kind of awareness that you're talking about to make people feel like security is their concern right and so I think some folks are struggling like some of the projects are more are you know in the they're further away on the spectrum of like you know they think of themselves as a library rather than realizing that somebody just follows their docs and puts it on the open web and then they've just created a vulnerability right right well yeah I mean obviously the one project is more conscious than the other the best you can probably do is you know try and help them with this you know provide them with information if you have any zero days responsibly disclose them you know if the situation is really bad that bad I mean I guess you could always work it and you're maybe paid somebody else to try and create some you know other version of it perhaps that's a bit more security conscious but not that that's nice or anything but I'm not yeah I'm not entirely sure what to answer to that in my experience I've never had that problem but I mean I am not I don't know which situation you have in mind so it's the kind of thing that if you wanted to brainstorm about it I mean I could talk to you you know talk with you about it afterwards but I think without knowing more of the specifics so I'm not sure in a generic sense how to broadly fix that yeah I mean it's the kind of thing that you know like our approach is just to like engage with the project and that sometimes and we're hoping that adding some extra volunteer help right from yeah our community might mitigate some of that yeah I mean in my experience the open source project I've dealt with have been quite receptive but I can imagine that there's probably some projects that aren't are there any other questions I don't I don't think so I think Chase asked about from a CNCF perspective so maybe this is a question of the group you know security should be part of the graduation gradient the graduation process I think that cuts to the question of should if there's a group like the CNCF is kind of overseeing the open source projects at some point should they just be hard and use the hammer and say security has to be part otherwise you can't be affiliated so I think that I'd like just to answer that from what I know of the CNCF process what they did before our group became the CNCF special interest group on security is they offered an audit as a benefit to projects that were in what's called incubation before they quote graduate right and so all projects get an audit for graduation and I think that what we've observed is like that we're doing with our security assessments is a big there's a big gap around what does a project think it's risk profile is and where its boundaries are right is what their customers might realize are their boundaries and maybe some of us in the security world might disagree with like hey actually you are security boundary should extend a little further right and so we have I think an opportunity to recommend ways that the CNCF can create like you know if projects are already graduated then we work with that community to to work through it like and so one of the like the example I'll just you know mention it so that everybody knows like what we might be like the one that I've heard about is that there's a talk by Justin Cormack in from Kubecon Barcelona I think where went through all of the audits that had done so far and found something unexpected I thought was unexpected in the Prometheus audit where the Prometheus team said basically security is not our concern that is a exercise for the customer we are a interface on your like we collect audit logs and display them and the customer is responsible for securing this and then it's actually quite challenging to do so and you know I think that personally if I'd been involved in that early on and it's sort of before my time I would have said no that I don't think that's an appropriate stance but that's a debate right so that's a discussion and so I think that that's where we're like where do we put in those kind of questions the Q&A right like at what point in the process do you're like wait a second right and so we've just designed these security assessments that are maybe a little heavy weight for that because they require a lot of effort on the project to document what is their risk profile what is their threat model you know right is their project supposed to do anyhow and you know and so that's kind of one of the things that we're kind of working through we've got like you know 40 CNCF projects at different stages and we have the opportunity to kind of recommend you know checks along the way and we want to add in a way that ideally is like educational and friendly sure yeah it's really a question of do you use carrots or do you use sticks I mean I'm typically more liver and carrots than sticks but that being said I believe some amounts of integrating you know security assessment you know can be helpful but I think you need to present it as a as an opportunity you know and not as a kind of punishment so right now we're participating with the NGI project together with an element and there's a whole bunch of open source projects and they're getting European Union funding and part of the requirements for that funding is that they get a mini sort of a basic scan you know a sanity check from radically open security it's a very small tiny scope so I mean it's very limited in how much we can actually help them in this sort of like little mini quick scan but nonetheless you know we're presenting it to them not as this inconvenience you know that oh and you have to do this and you have to do that but more just like hey it's a really great opportunity to get some help early in the architectural process and you get it free while isn't this great you know I mean it's possible I think that it's nice how Anil net has you know has presented it to the project and I mean maybe some of them might view it as a hassle all the same you know if you can get them a little bit excited you know perhaps they can also grasp it as an opportunity opportunity to increase their security awareness which to be fair I mean that is what it is yeah and I think that's a good idea like you know like just talking through it we could like some of the challenges like if a CNCF end user which is like one of the companies that's deploying cloud stuff uses this project with it end up being secure and so you could imagine if the CNCF is interested in paying for this like to create a like a test right like okay we'll have a team that's interested in deploying the software deploy it and then we'll you know right after they deploy it will do a chat ops like you know pen tech like some kind of exercise with y'all and then see like hey did it come out secure what are ways that we could improve that like you know things like that that might be something you know if at some point you're interested in brainstorming about this further I would be happy to do so but preferably at a moment where I'm not standing outside of transportation cool so yeah maybe we should let you go and are there any final questions Robert oh like I said we could do a whole nother Q&A session so I'll let everyone have the rest of their name back just a big thank you yeah this is fabulous yeah wonderful thank you so much for tolerating my chaos but yeah I guess it's par for the course so really truly appreciate it thank you so much no worries and again if any of you need anything feel free to just shoot me a mail Melanie at RadicallyOpenSecurity.com I'm always happy to brainstorm with you guys about whatever you need right thank you very much okay well thank you so much have a good evening alright bye Melanie I'm just gonna stay on a little later and make sure I've captured things from the chat alright alright thanks everybody alright bye from a meta perspective just from a format do you think what do you think this was good do we maybe wanna have shorter presentations and more time for Q&A or this worked out okay what do you think the people who still here have any feedback who gonna say no to this I think this is all good stuff but this is Mark I think the questions about what we need for CNCF probably should frame it for the people so they know you know what our kind of needs are I don't know if it's requirements exactly but because I think she probably could have tailored this to address some of that well I don't think that that was the motivation for the talk I don't know Robert you invited her but from my perspective when I saw the talk it was a pitch description it was more like how can we all use transparency how can we all use these open source tools and chat ops for in you know our work should we recommend these things to you know like I think it was a awareness of this methodology and the open source tools and in fact the invite to her and just to give you kind of the the meta detail behind how it all came just for everyone else who wants to do this in the future I mean it was a blind call out for me I had read about the Mozilla zero day stuff that her company had been doing was impressed but mostly was fascinated by this open transparent definition of doing the pen testing because I thought it was very appropriate to this discussion security assessment should be versus a code audit you know trail of bids that kind of thing so I just I reached out to her said like here's what we're doing we're doing security assessments we've seen say yeah it'd be great to hear about this open transparent model which is the more commercial closed model and I left it fairly blank slate for her I didn't want to be dictatorial about here's what we want you to present so the chat ops and the methodology part was that was what you wanted to present and I found it actually is going through the call I've got like two pages of questions and concepts that I'd love to kind of bring into the fold at some point around how this might inspire us to do things both strategically in terms of what we might want to do or not want to do for security and transaction people might want to do in various CNCF projects but as a blank slate exercise that generated a bunch of brainstorming ideas I think it was great but I do appreciate the fact that you know others might see more value in a more targeted like here's the problem here's a scenario what's the resolution kind of frame a specific to CNCF yeah it's be hard to make a complaint about this presentation agree agree I was highly valuable so you know I don't know if I was clear in my questioning about the DevSecOps thing but we're in the middle of wrestling with this very problem and you know while you know I'm on the IEEE standards group that's working on a standard for transparency for autonomous systems that's a hard problem all by itself trying to be transparent but support commercial software which you know open source is great we're all in that space but if you're a business you need somebody that you can pay to take care of things and so we are in these hybrid models where we've got lots of open source lots of commercial stuff and the standards around how to do DevSecOps are I think one of the challenges and rolling CNCF products into at least commercial enterprises that are regulated like the one where I work is a big part of the challenge that's why I had the question about standards and how to embed the normally adversarial red teaming process which is really pretty unhelpful you know you find problems but you don't generally fix anything beyond the scope of that and that adversarial process which he's trying to get away from is a great approach but it leaves open a bunch of questions which is probably why you got a couple of pages of notes yeah I mean I'd see some examples of where you talk about the transparency in like an automated DevOps environment I see this happening in the AI space so and someone on the chat was going back and forth about healthcare and I do some work in that area as well but this has come up explicitly there was a talk at Stanford a couple weeks ago about AI and healthcare and they were very explicit the practitioners and clinicians were saying look there's AI tools out there by commercial vendors they claim to do XYZ but because we can't see what's going on there's no way we're going to deploy that to patients because of concerns around patient safety and bias and they gave great clinical examples about how bias and the AI tools can lead you very quickly to bad outcomes and so I think the same kind of rationale applies to other safety critical things I'm not going to use a commercial tool that's not transparent or a commercial service is not transparent if it's in service to mission critical or safety critical applications because we can't have a larger group understand and review what's going on it might work the first 10 times it might work the first thousand times but the next time it fails then there's some serious consequences yeah this is a longer conversation and maybe worth having in this venue sometime but the piece of this that I'm trying to find solved in the CNCF ecosystem is some automated testing that has transparency built into it and so adversarial red team or blue team testing that's embedded in the pipeline that uses CNCF tooling and frameworks is part of the solution you know and so that's you know that's where my note taking tends to go in these meetings and we have some projects that try to do that like for the software supply chain you know that are partly automated and partly not and it's interesting that she's trying to automate more of this with all the XML streaming here but I think this is a what it's part of the SDLC really that we have to find a way to put security into it and what are the standard interfaces going to be yeah I like the notion of having kind of an interface for adversarial webhooks if you will for lack of a better concept so pipeline, dub-obs, AI anything that's going to be a fully automated or autonomous process has some call-outs to apply this kind of testing or even just transparency dumps for lack of a better word yeah well put oh well like said we could probably go on for hours alright thanks everybody thanks Robert for initiating this this is great oh my pleasure thanks sir be well thanks sir