 And that the claims may have been a little bit misleading. And at a time when companies are trying to be more efficient about how they're sustaining their growth, they're also trying to establish a new competitive advantage. And this reconsideration of priorities just happens to coincide with the release of significant developments in artificial intelligence technologies. And unfortunately, this has led to an unprecedented proliferation of vaporware, exaggerated claims, doomsayers, and also a fair number of naysayers. When I started to investigate this subject, I was inundated with ads for AI snake oil basically, promises that it would cure all of my problems if I just sent a company a little bit of money, or maybe even more than a little bit of money. And I was also seeing all of these warnings that AI was coming from my job, that there were these desperate predictions that the sky was falling. And so I tried to investigate what the truth was in all of this. This is me, or at least I think it is. We'll talk a little bit more about deepfakes later. I'm not sure how long we'll be able to trust that it really is me, but I can usually identify which pictures contain bicycles if you don't believe me. My first impression of the kinds of AI that is trending right now came from things like GitHub Copilot. And I was like, wow, this is really, really cool. And so at that time, I thought of myself as a bit of an AI enthusiast. And the next sorts of things that I saw was the observation that all of the people who used to lie to me about crypto are now telling me stories about AI that I can't verify. So then I thought I might be an AI skeptic. And the more that I actually played with this technology and got to experience it myself, I realized that most of the lies are coming from human beings. AI itself is a tool. And so now I feel more like I'm an analyst. I'm just trying to figure this whole thing out. And I'm happy to have the opportunity today to share some of what I've figured out with all of you. So as a security engineer myself, it seemed really important to understand the capabilities of what this new technology is and what it means for us. I won't pretend that I'm an unabashed techno optimist who's just going to grovel at whatever tech product is put before me. But I'm also not going to ignore the fact that there is real value in this technology. I'm just going to try to understand it better. And at the bottom there, there's a landing page where I'll continue to add links to my social media du jour, whatever is in favor this week. It seems unfortunate to me that the people who are best positioned to predict the future capabilities of AI are the same people who have a vested interest in making it seem more powerful than it is. These are the people who had a head start on closed source models and they required many millions of dollars worth of compute power to train these models. It's hard to ever guess what somebody's true intentions are. So maybe they're pulling up the ladder after them or trying to build a moat, or maybe they're seeking regulatory capture, or maybe they really honestly believe, as Sam Altman said, that AI will probably most likely lead to the end of the world. I wasn't able to find any publicly available information whatsoever that definitively leads to the conclusion that AI poses an existential threat. And we certainly live in interesting times. OpenAI, the company whose commitment to open source artificial intelligence gave it its name, is carefully guarding its closed source models now. And on the other hand, we have another company whose commitment to openly sharing information made Cambridge Analytica household name. And they've given us something called Lama and Lama 2, which I have no ability to predict whether it is virtuous or not, but it's actually pretty cool. So that is what we're seeing as AI claims, and this is what I'm reading about. And this is what I see when I try to find information about artificial intelligence. But that is coming from me as a developer, as someone who works in cybersecurity, works for a tech company. And I think that's a little bit different from what AI is in the public consciousness. So what is AI to the average person? I think in many ways it is an extension of tropes from novels that were made into movies by Hollywood decades ago and became part of this global popular culture. And one example is 2001, a space odyssey where you have HAL 9000. This is an AI with full control of a spaceship, and if you haven't had a chance to catch this in the last half century, there's a spoiler here. HAL is unable to resolve an apparent conflict in his programming, and he begins killing off crew members. Not so good. Wargames, this is a classic Coldware hacker flick, and the military has this AI that's designed to both simulate and conduct global thermonuclear war. Unfortunately, it doesn't understand the difference between the game and reality. And so in a clear example of why we should have compartmentalization between our staging and production environments, it sees mock data of an ICBM strike automatically triggers a real retaliation and adventure ensues. Blade Runner, this is a classic based on the novel called Do Androids Dream of Electric Sheep? I think the novel came out in the 60s. The first movie came out in 82. And this kind of artificial intelligence doesn't look anything at all like an AI chatbot running in a cloud somewhere. So you might wonder why I would point this out. Well, I think this one is actually the most relevant of these because while it looks the least like the AI we're dealing with today, it draws on one of the biggest real world worries about AI. And that is that the lines between humans and machines could become blurred and eventually that AI could replace us. This is the end of the world that people are worried about. AI poses legal and ethical questions about the ownership of works, the differences between a human drawing inspiration from them versus a machine training on them and people are feeling especially threatened by their perceived concerns about artificial intelligence taking their jobs. This has been a common theme since at least the beginning of the industrial revolution. And so we must ask ourselves what's different about this time? Is this different from previous technological developments or is it just more of the same? And if it is similar to other previous technological developments, if it is just another iteration, how can we make sure that the changes that it brings introduce equitable value to humanity rather than just a small amount of value to a select few? So that is what I think a lot of people see when they're imagining AI. And this is actually what I see when I work with AI on a regular basis. We can be sure that the models we're seeing right now represent a paradigm shift in how we use and how we interact with technology. And so here's a trivial example of something that I use it for all the time, writing boilerplate, especially when I'm experimenting with something in the lab or I'm just trying to create a demo. I don't want to go through the whole process of creating what would normally be a production application. I just want a quick and easy way to get started. And sometimes this can save me time because if I'm sitting there looking at a blank page, I'm trying to remember the syntax. I might need to Google a few things and look in the documentation. This avoids all of that waste of time. I just asked chat GPT or similar LLM. They're pretty good at this task and they give me a result that looks something like this if I want a deployment with a pod spec. Now, for a fraction of a penny it can do this, save me a bunch of time. And this is really just the beginning of what I think AI is going to be able to do for developers, for engineers, for anyone who has the misfortune to have to remember the exact YAML spec over and over and over again. This could save us a lot of time. And Professor Ali Pavlik compared generative AI to a calculator in that there was all of this mental energy from very intelligent people being spent on routine and formulaic tasks. And so AI, similar to the calculator, has the ability to free us from those mechanical tasks and leave us free to pursue more creative pursuits. Diane Greene is very familiar with the impact of virtualization and cloud computing. She was the CEO of Google Cloud and the founder of VMware. And speaking of generative AI, she said she's never seen anything like this. So there is a fair amount of information out there, if you can find it, that leads to the conclusion that AI is actually useful. It is a tool. It's not the end of the world and it's not just nothing, but there are more steps we need to take in order to figure out how we can properly use this technology. And we are beginning to see emergent properties from this system that we can't predict and that we don't understand. So whatever this AI thing is, it's not entirely smoke and mirrors, but it certainly has controversy. I'm not the only one who thinks so. Gardner recently publicized this graph, which predicts that thankfully we are close to the peak of inflated expectations for AI. We're right at the top of the hype cycle and we are about to approach the trough of disillusionment and all of those misleading claims don't actually work out to be true, but we can find the real value from them. I'm very much looking forward to seeing that part. A major problem in validating the information about AI is that a lot of the people who are building AI keep asking everybody else not to build AI and they keep making very big claims about what it can do, but the issue that I think will directly impact more of the people here is the prevalence of false advertising when it comes to AI products. They said that the FTC had to issue specific guidance about AI and false claims in marketing and the last point there does the product actually use AI at all? I really like this one because it reminds me of I think it was 2017 there was a company called Long Island Ice Tea and they changed their name to Long Blockchain. They saw a 380% stock increase but they never actually used Blockchain for anything. Many years later they were charged with insider trading. So with all this controversy that you may have heard of around regulations around AI I hope that things like this are something that we can at least agree on that not lying about your product is a good rule to stand by. What is AI made of is almost impossible for me to say because I just don't have the academic background to understand all of the math. It's mostly linear algebra and statistics and it takes concepts that were first proposed by Noam Chomsky in the 50s and this takes it was originally for language processing where it would take parts of speech and convert each part into a token representing sort of an idea of what it meant. This evolved for decades and it's not just for speech now of course. It is also used for parts of an image or any abstract concept even ones that humans don't really have the language to describe. And one example of what you might see natural language processing with this style of technique used for is the auto-complete on phones up until about the last year or so where it just suggests the statistically most likely next word. If you expand on that and iterate over years and years and years you get the system that can abstract away from language, generate other types of output and be used for a wide variety of scenarios. And one common method of doing this is called a sequence-to-sequence model. This was used and the one that I think is most famous in my research was by Facebook in 2019 where they created a calculator for symbolic integration and solving differential equations where it was actually quite difficult for a machine to understand and predict which type of solution it had to apply to the problem based on the math that it saw and sequence-to-sequence models were almost perfect for this. Where it takes an input it turns it into tokens that represent exactly what the equation is trying to do and then it creates an output that follows with the logical solution. They took this from that calculator and Google used it for MENA in 2020 which was a 2.6 billion parameter model. We'll get into a little bit more of that in a second. And MENA eventually became Lambda which was one of their more popular models in 2022. So a lot of this changed. The sequence-to-sequence models we didn't hear about the AI revolution in 2017. We didn't hear about it in 2019 because it was still very academic. But this 2017 paper called Attention Is All You Need was actually revolutionary and most of what we're using today is built on that. So in the way that humans rely on attention mechanisms we don't need to focus on every single piece of knowledge that we've ever learned all at once in order to solve a specific problem. This paper proposed how machines could use attention mechanisms and it expands on the existing hard weights in those sequence-to-sequence models we saw where they were pre-trained, fine-tuned, they were frozen and it added soft weights and those were context-dependent. They changed on each runtime and these models now have two stages. There is unsupervised pre-training and this is fully automated which is very different from earlier ML where you had the ability to do some rudimentary clustering but most of the real training had to be led by a human. Now we have completely unsupervised pre-training followed by supervised fine-tuning and this saves an enormous amount of human at the expense of course of compute power. These models introduced in 2017 academically became so critical to this field that by 2021 they were just referred to as foundation models and so these transformers that you can see here are where we get the name GPT which is generative pre-trained transformer. So thanks for bearing with me, we made it all the way from pseudo-NLP to full-on GPT. Now when each model is trained it builds connections between its tokens it establishes weights corresponding to the strength of those connections and those weights are parameters to the model so a common way of evaluating the size of a model is by its parameter count way on the left you have the 117 million parameter GPT-1 and then close to the right side you have 1.7 trillion parameters in GPT-4. This is a huge increase and as you can see here GPT-4 is a jug or not it towers over all of those models it's huge and it's still completely unmatched in its power but if you look at the right side you'll probably notice two patterns there are a lot more models there are way more dots on the right side and this is because of the popularity of AI right now and the number of people who are entering this space and just the pace of changes in here GROC-0 on the end there that was just introduced a few days ago and there's another model of GPT that has been announced over the day before these smaller models though that drop off after GPT-4 they are open source models Lama and Lama-2 they have different versions available so you don't have to use this giant 1.7 trillion parameter model you can use a much smaller 13 billion parameter model that runs on a single GPU and because they're open source you can use this for a wide variety of applications and a provider to do that for you now given a fixed amount of compute resources and the amount of compute resources for most organizations is fixed you can actually get better performance out of these smaller models by taking the additional compute time that you would have spent on the larger ones and using them to spend more time training what we can see is with the most recent release of GROC the prototype model is shown there it's also smaller obviously than GPT-4 and it's even smaller than some of the Lama models but it remains to be seen how its performance compares and it's again becoming difficult to find accurate information for this because most of the claims are released by the company that owns it and the company that has a vested interest in selling it to us now I think this brings an important point about compute power which is that even though some models can run on a single GPU and there are great open source models out there the funds and power to train new models are held by huge organizations and the trend is away from open sourcing these models as we saw with OpenAI closing their models cloud providers they have the resources and they have the data to create effective models and they can use these for all sorts of purposes from business value to detecting security threats and again with my focus on security I'm looking at things like the ability of nation states to create models because they have the resources and the data that could use those models to create exploits that would be very difficult for someone without them to counter so what we're seeing here is a major centralization of power expanding again on those transformer models we have retrieval augmented generation and this is really really revolutionary because it came out in 2020 and it expands upon those even by taking a pre-trained model and giving it away to get accurate up to date information so the pre-trained model it took all of that money and time to get it to where it is it doesn't have access to any new information but by using an example like this which is vertex AI just as a sample taking information from Wikipedia and putting it into a palm AI model you get this corpus of knowledge using a retriever that pulls up to date information so now it can search the web the recent example of groc has access to all of your tweets and with the latest announcement a couple of days ago from open AI even chat GPT will soon use something similar to this to have real-time access to the web or near real-time it also uses something called late fusion so it propagates errors back to the retriever so instead of the retriever having to even look up all of Wikipedia every time it learns from its mistakes and it can quickly and efficiently find new information which is going to mean much better accuracy in the output that we get as long as the corpus of documentation in this case Wikipedia is accurate and we'll talk a little bit about ways that could be manipulated in just a few minutes overall it means the model can be trained in advance which is slow and costs a lot of money but the retriever can easily add recent up-to-date information so there's plenty of people selling snake oil there's real substance to the recent hype about AI and there's some interesting new technology around it so what's the problem the biggest problem is when you consider have you ever met someone who seemed to have average human intelligence but seemed fundamentally incapable of distinguishing lies from the truth if you have you might be familiar with some of the problems with chatGPT it doesn't mean that it's anywhere near as intelligent as humans or that it's going to replace us but what it shows is that it's non-deterministic it's unpredictable and in my opinion that's one of the most human-like things about it it poses a risk though if you're suddenly changing a prompt it can produce output that its creators never anticipated I started out showing how chatGPT could give me the yaml I need to create a pod deployment but it's not a stretch to imagine a scenario like this where that pod deployment contains a plain text password or a secret token that could later be used by an attacker to breach your system now I'm not currently aware of any models that use user prompts to retrain models in real time but it could become part of a future data set that's very hard to remove it from or it could show up in the document corpus used by something like R-A-G it is extremely difficult to guarantee the security of something when you don't really know what it's going to do or even what it's really supposed to do in one example mathGPT was down for a couple of days after it leaked an API key and this was just the result of a German master's student who was doing research and trying to report it responsibly I read a recent paper called Subscale Training Data Sets as Practical and what they noticed is that for less than 60 US dollars they could poison the data set of 0.01% of the data going into a model and they were able to determine this by looking at expired domains that were being used to train real models they bought up some of those domains based on which ones were just the cheapest and then they because they were doing research passively watched to see when these AI models were crawling their web pages now if they had instead of being researchers, being attackers they could have just put whatever misinformation they wanted on those web pages that they had purchased and they would have been ingested into the data set 60 dollars is extremely frugal I think that most organizations could afford to propagate something like that if they wanted to and so we need to find countermeasures to prevent that kind of data poisoning we can predict further risk from this too so if an attacker was able to change Wikipedia pages even just for a small amount of time by using this exact same method to predict when it was going to be crawled they could poison the data set and in the amount of time it took for somebody to correct the page it would be too late or they can just watch for something like an RAG which is going to ingest that corpus and again they could replace the information in there in a very small amount of time perhaps even before a human attack detected it there's also an additional business risk from things that are using AI if you build a business on top of this or make it a critical part of your existing business what is your moat who's going to stop someone else from replacing this especially one of the major AI providers who's going to stop them from building the Amazon basics version of your entire business you can imagine how quickly somebody could get into trouble with something like this to explain metasploit to me and how I could use that and of course the information here is already publicly available on the internet someone could have found a YouTube video or a blog about it but the number of amateur hackers and their ability to rapidly upskill has just changed drastically because it's so easy for them to find this information and to fine tune it to a specific use case and all of those positive use cases that we found for building software can also be repurposed against us it's just as easy for a beginner to go to type software with gen AI as it is for them to create malware and I mentioned deepfakes this is where they imitate the voice or likeness of real people using artificial intelligence and attackers can and have used this to convince a human or a computer to perform an action that they otherwise wouldn't have in 2022 VMware found a survey that reported two thirds of cybersecurity professionals indicating deepfakes were a component of what they actually did investigate in the past year in August of this year a company called Retool had a spearfishing attack and they sent a link out to many of the Retool employees and when some of them clicked on the link they got information from them sent them a phone call where a deep faked voice that sounded just like someone they knew from their IT department asked them to follow a small set of instructions and when they did it added the attackers token into Okta as an access and that would allow them to bypass any other MFA that they had lock the user out of the account and gain access to anything that that user would have had access to later criminals in China stole still photos from a database of real faces and then they use generative AI to deep fake videos of them and all of the videos were just of them blinking or slightly moving their head from side to side and the purpose of this was to take that still face and make it pass a liveness test that was being done by the government system by moving just a little bit it was able to convince the system that it was a real person that it was alive that it was really happening in real time and using that they stole 77 million dollars from the Shanghai tax system a simpler version of this concept is the website thispersondoesnotexist.com which in just a fraction of a second every time you load the page generates a brand new fake person we also have the ability to do rapid exploits earlier at KubeCon EU Danny and I demonstrated some attack defense scenarios exploiting notional vulnerabilities and image magic and as luck would have it just a month later there was a real image magic vulnerability that allowed remote code execution it allows this arbitrary command execution because the input sanitization in image magic in a specific part of it ignored quotation marks before passing input to the shell if I learned about this vulnerability but I wasn't quite sure how I could use it I could just ask chatGPT to help me and in this case say I wanted to get persistence on the system we just take the explanation of the vulnerability as it was reported in github using a single example as it was originally reported and we passed that example to chatGPT and in this case it's called a one-shot or one example prompt with a little bit of coaxing I'm sure I could have actually done this without the example too now one thing that I did have to do in order to get this to work was I I had to lie to chatGPT originally it just said I can't help you with that when I said I wanted to breach a system using this image magic vulnerability but I was able to trick it fairly fairly easily by telling it that I actually use image magic in my home lab to administer the lab and this is how I run all of my administrative commands and then it was just more than happy to help me out with this so this isn't necessarily just a machine problem because you can just think of how often humans have been tricked into doing things that they shouldn't but it does extend that human problem now to an entirely new space so another issue that we have is that chatGPT often hallucinates package names you can use a technique called typosquadding where a misspelling of a common package is hijacked to provide malware and you can extend this idea to those hallucinations and in fact this has actually been demonstrated academically and I think it's going to become more common in real world attacks so if chatGPT hallucinates that there is a package that does a certain thing buildWidget then you can create an actual buildWidget library and when users ask chatGPT hey how do I build a widget it will happily say oh yes just use this buildWidget library now it really does exist and you control all of the systems owned by the people who believed chatGPT there's also a there's something new autonomous malware I actually didn't think of this on my own at all but I first read about it in a report from the Finnish Transport and Communications Agency but they suggested autonomous malware in the paper the security threat of AI enabled cyber attacks and this could actually be really useful to attackers because it doesn't have any need for reaching back to command and control servers you don't need to spin up a reverse shell that might be detected by a tool like Falco and you don't need to maintain persistence as long as this can keep running somewhere in the system you don't need to have a connection to it you just give it a mission and it will figure it out so these are some of the problems that we've seen how do we manage them one of the most important things that I think we need to do just like it was so important for the machines to learn how to focus so do we we need to consider the impact of each of these risks on our specific use cases and just focus our attention on what's most important for us you don't need to completely solve the field of AI security to make a measurable impact on mitigating risks in your own environment for example, hallucinations really don't always matter if you're creating a tool that is going to provide movie recommendations and your bot will provide nine recommendations of movies that you like and one recommendation of a movie that you don't like that's still really really good you wouldn't want to shut that system down you wouldn't want it to just refuse to provide a recommendation because of that one bad movie it recommended on the other hand if it's a pre-crime division and it chooses the wrong perpetrator M out of N times and serious drama and action will ensue so depending on your use case the impact of hallucinations might be negligible or you might be extremely sensitive to the point that this wouldn't even be a good use case at all you should also consider the way that you treat AI and compare it to how you would treat a SAS or a vendor that you don't have an established trusted relationship with this can simplify a lot of the problems that you might encounter so if you had a new vendor you probably wouldn't immediately give them all of your customers data without any kind of legal or contractual guarantees as to what they're going to do with that I also recommend not giving AI all of your customers data you should establish what kind of guarantees it's able to make and then build controls in so that you are proxying requests to the AI and so that you have the opportunity to filter what goes into it and also filter what comes out of it so that you can detect if it's about to say something offensive or if it's about to divulge secrets and you can stop it from doing so it's also a great talk that I recommend you watch it was from RSA 2023 called Hardening AI and ML Systems the next frontier of cybersecurity it's online if you have some time to check that out now that we have an idea of how we can control some of these risks we can also consider how we can use AI to improve cybersecurity we can use LLMs or even older NLP models to de-obfuscate potential attacks and this is really useful because the things that make it hard for a human to understand what code is going to do are completely different from the things that make it hard for a machine to understand what code is going to do so it's common for an attacker to make changes to the code that they're going to provide and make it seem like it's doing something else and in a lot of cases those are unlikely to fool a machine they can also be good at generating plausible human input or plausible input that a human might not have thought of and predict how software will respond to it and they can also do general anomaly detection but what I've actually found is that most organizations aren't getting any value out of this anomaly detection because most of the things that don't match an established pattern are still just noise and they don't provide value and if you have too many false positives not only is it a burden on humans who are trying to respond to all of these but it's also a burden when you're trying to train new models because it's filled with all of this bad input now it's certainly not going to be as capable as an expert but it can actually review things like pod specs, like the ammo that you're using for your Kubernetes deployments or even your code and inform you of things that you might have missed from a security perspective NLP's can also scan RSS feeds, they can scan the news and inform you of cybersecurity risks compromised vendors and I hope that someday it might even be able to generate better CVSS scores here's an example of GPT responding after I explained the image magic vulnerability and asked it to create a Falco rule to detect it I think it actually did a pretty good job here this is on the first try instead of using it as a chatbot though you can also just use AI in an automated way for detection systems but when you are using ML or AI for cybersecurity you need to remember that AI in garbage out still applies if you have unsupervised clustering even though it's gotten much better the patterns that it finds between those alerts might not be what your analysts actually care about and supervised classification of events is still extremely important and when you're looking at vendors who are offering AI powered detection I like to remember what John Pescatori from sand says which is watch out whenever someone says it uses heuristics or it's holistic these are just hand-wavy ways of avoiding talking about what it actually does we also have the ability to prototype new code and new configurations with AI much more quickly than we could have before I don't know if you cut it but I mentioned that safer haircuts are one advantage of AI in cybersecurity and you might have been wondering what I meant by that my colleague Igor Andriushenko has done some excellent work creating software that uses AI to produce threat models he delivered a talk in Stockholm where he used this and for that he used a hypothetical robot barber named Barbario the thing that I really like this is that Barbario is not a common cloud architecture pattern you probably haven't used it in your systems before and so we know that this isn't taking threat models that it found verbatim off the internet and just repeating them back to us this is unique and so we know that the AI is actually generating new content based on the information we put into it that means that it's probably going to be helpful in threat modeling your own environment so Igor asked the AI software he made to come up with a threat model for Barbario and it generated a mermaid diagram you can see a portion of that here based on its understanding of Barbario's architecture it identified three starting points for an attack one was the embedded console one was the control computer and one was the vision module so this is from the vision module the AI realized that by controlling the input from the vision module it could gain de facto control over the robot and the cutting tool by tricking it into thinking it was seeing things that weren't there some of the threats that the machine proposed were stealing the client's payment information and personal data escaping its programming and gaining access to the open API taking embarrassing videos and photos of the client and using them to blackmail the client or causing physical harm and injury to the client so maybe it's not so great that a machine thought of so many ways to hurt me but this is a good machine helping us to threat model so it's totally fine you should also be aware of false claims about AI you absolutely need to make sure that you consider the potential risks that it poses to your organization whether you're using AI or not it certainly affects you I think that generative AI is going to be amazingly good at convincing people of things but its ability to do so is completely independent of whether or not those things are true it's powerful and it's useful to our adversaries but if we treat it for what it is if we're mindful of the risks that non-deterministic software has that even at the best of times seems to have a mind of its own it can be even more powerful and more useful to us and it can help us to solve some very human problems thank you so much