 I'm a professor at University of California Irvine in computer science and I'd like to thank my co-authors over there, Yuki Sawa and Ram Bhakta, they're both students of mine, and then Chris Hadnaghi who supplied me with the samples and talked to me about social engineering attacks in general. So the idea, okay so before I go into the slides, basically the whole point of this is to detect social engineering attacks while they're happening. Now what the system actually does is it assumes that the text is already transcribed, right? So it takes typed text. So if you're using a chat application or something like that, then you already have it typed. If you don't, if it's over the phone or something like that, then you would have to have speech to text in order to do this, to do what we're doing, right? So we assume we have text and we analyze the text that the potential attacker is saying. So we assume it's the dialogue between two people. We analyze the text, we process, we basically parse each sentence as it comes along, and we try to detect if it's a suspicious sentence, if it's asking you something you shouldn't be asked. And if it does, then we alert the user. That's basically the whole thing, okay? So if there's feedback, I'll try to fix it, but that's the idea. So the first slides, you know, now that I look at them, this crowd is probably a little, people know about social engineering here, so maybe some of these first slides are not that important. Everybody's social engineering, oh, there are lots of definitions, psychological manipulation to get somebody to do something they shouldn't do, da, da, da. You know, it's not new, it's just con people, whatever, convincing you to get something out of you. And there are a lot, you know, with technology, you can do it through different vectors. You can do it through email, or through phone, or through text, or whatever you want to do it through. They are popular. There are lots of social engineering attacks. And you know, I think this crowd is speaking to the choir, but here's one data point. This is some Verizon study that said that in 2013, 29% of all data breaches involve social engineering attacks somehow. You know? That's probably an underestimate to me, but whatever. That's one data point, but you all probably believe this part already. So I don't have to really stress this. Types of attacks, so these are just some types. They're sort of simple attacks, phishing attacks, baiting. There are lots of different reasons to do attacks. And if you were here during the course of the day, you saw some attacks, right? Those are the best examples. You know, you're trying to get information out of people that they don't want to give up or try to get them to do something they don't want to do. So I don't know, I'm trying to define this term dialogue-based social engineering attacks. I'm trying to differentiate the attacks that we're looking at, we're really focusing on, from, say, phishing attacks, right? So you know, there are some attacks, like generic phishing attacks, which are very broad, right? They're not targeted at any particular person. They're very generic. Oh, I'm a Nigerian prince and I have $10 billion, you know? And they're not targeted at anybody in space, in particular, just you massively, like, they won't catch many people, right? But if you send it out to many, many people, you still get raked in some suckers, right? So there are attacks like that. And there are defenses, there are tools, actually, that people are proposed to detect attacks like that. I'm not really looking for attacks like that. I'm looking at more dialogue, like, basically, just like the stuff that you saw, if you were here, you saw earlier today, where somebody is targeting you, right, or your organization, and they call you up, or they come up to you and talk to you, or they chat with you, and they're trying to fool you, right? And they probably have done some research on you, right? And so, it's a more targeted attack, it's a one-on-one. So I'm calling these, and, you know, they're, I'm talking about pretexting and elicitation, they're very, they're subtle techniques that are used, and I'm not an expert on this, you know, Chris is, and other people are, but there are a lot of subtle techniques that are used to make people feel comfortable and make them want to give up information. So, but those are the types of attacks that I'm trying to look at. So I'm talking about dialogue, somebody's calling you or talking to you, and talking to a particular person, not just sending out a mass fishing email, something like that. So, okay, so what, so this is just pictures of how you might use the thing that we're making up. So, say you got an email client, or a texting application, something like that, it sends it to R, that yellow thing, that yellow box is our social engineering detection system, right? The text, as it comes through, gets sent to our system, and it tells you, oh, this is an attack, and it sends that data back to your application. Like, if it was a texting application, maybe it, it would put something, print something on the screen for you, something like that. Or it could be like some audio video device, so it could be a cell phone or something like that, or, you know, some eyewear, that, you know, Google Glass style thing. If you had speech recognition, you had some speech to text software, you could do the same thing, okay? But the idea is that the tool is supposed to be, here, reading the sentences as they come, checking them, see if they're suspicious, and if they're suspicious, you send it alert back, in some form. So, you, okay, so one thing, one thing is because it's a dialogue-based attack, I don't want to rely on, sort of, vector, I want to make it generic. So it doesn't, there are certain vector-specific cues that a lot of other detection techniques use. So for instance, like, if you're detecting phishing attacks, you might look at, say you're looking at some phishing website, you want to say if it's a phishing website, you might look at the images on the website. Like, there are a lot of papers that have been proposed on this type of thing. But that would work with a phishing website, right? But I want to make it just more generic. I can apply it to any type of vector. So I'd like to home in on the communication. See, most of the previous techniques that look for phishing and things like this, they don't look at the actual meaning of what's being said, okay? They look for other things that are suspicious about the phishing email, something like that. But they're not trying to take the text and interpret its meaning at all. And so that's what we're trying to do. We're trying to do some sort of semantic analysis. In first, something about the meaning of the text to see if it's actually a suspicious, like, a bad thing. So, yeah, so hopefully superficial analysis is sufficient. By that, I mean, trying to understand meaning is a complicated thing, right? So hopefully I don't have to go too deep. And so I'll show you what I do and how often it works, you know? All right, common features of social engineering attacks. So what I tried to do is figure out what was detectable about these attacks. These attacks, they're varied, okay? And there are a lot of strange, I mean, several books that Chris has written about social engineering attacks and all the different ways to do them. And so that seemed intimidating. How could you possibly detect all those things? And I really can't. But I can say, I think this is true, that any kind of social engineering attack, in the end, you've got to do one of two things. This is my claim, okay? You've got to ask somebody for some information that they don't have authority to have, or you've got to ask them to do something that they don't have authority to do. This is my theory, right? So, if I claim. So I think, look, if every social engineering attack has to do one of those two things, at some point in the conversation, they've got to say, okay, so what's your social security numbers in some other way, right? Or, okay, go ahead and shut off that route or change your password or whatever, right? At some point, I can't detect all the elicitation and the pretext and all that, but maybe I can detect that sentence when they come down to a point where they ask the question or they command you to do something they shouldn't command you to do, maybe I can detect just that. So that's what I'm trying to do. So, basically what I'm trying to do is, when, as the sentences get parsed, you want to see if the sentence is question or command, because I'm assuming it's either a question, ask you for information, or a command telling you to do something you shouldn't do. So it's got to be a question or command. So I look at the sentences, try to detect if they're a question or command. Then if it's a question or command, I try to detect the topic of the sentence, okay? Basically important words in the sentence that would indicate the topic. And then I take that topic and I match it to a blacklist of bad topics that I have. And if it's in the blacklist, and I say, okay, that's something you shouldn't be asking about, alarm. So that's the idea. By the way, if there are questions, go ahead and ask. Like I don't mind, you know, you don't have to, but you can. Yeah, go ahead. Maybe you already can do this, but you can compare it to how it works in some blacklist of words. Okay, so you, okay, let me just repeat it, right? You're saying, do I have a comparison of how it will work with a blacklist of phrases? Yeah, like, how would you perform better for just some sort of word list or something like that? Uh-huh, yeah, okay, so, ah. Okay, I sort of have that. So what's being asked is, how would this perform better than just having, like doing a simple thing where you just have a list of bad words and you just, you know, if the bad words come up, then you say it's an attack. Okay, so I have, hmm, I don't have a direct comparison of that. I published a paper on that, on a simple thing like that earlier in the year, actually. And I, the reason I did this was actually because of the complaints about that work. Because people said, look, you can use, like I have specific examples of this. I have a slide with it. But you can use the same words and not in a malicious way. You know, so that, so in general, people said, look, that could happen. This technique's not good enough. They published it anyway, but they said that was the argument. Now, I don't have a direct comparison, but my other technique, I'll talk about it when I get to it. Yeah. Now, okay, that's all right. So is the list of more phrases than words? This is looking at sentences, sentences, yep. Yeah. And parsing the sentences. Yeah. Go ahead. That's an excellent, painful example. Okay, so let me zip, no, let me, okay, let me, no, no, that's all right. That's all right. That's all right. That's good. Let me renounce, we state the question, right? Just so I can hear it. So he said, look, say I got some statement, like click on this link. How do I differentiate that from being like a legal click on this leak versus an illegal click on this link? Right? Okay, so, okay, so I have a good, let me go straight to the end, right? There are, in the end test cases, right? There are three social engineering conversations that we analyzed, right? That were given to us by Chris. And there were 10 sentences that were malicious, okay? Where they either asked a question like this or they made a command, you know, a bad command. And of those 10, we detected six. Now, the other four were just like that. In fact, click on this link wasn't exactly one, it was like click okay. And there was another one like that, okay? All four were like that. So, okay, so now I'm doing my future work, right? So the reason why, with that, right? Like click on this link. What link are they talking about, right? There's no context. So the problem with what I'm doing right now is this one sentence at a time. So if you can get the information out of a sentence, it'll work. But if you say click on this link, that doesn't have enough information to know what link you're talking about, right? What you have to do is remember what was talked about before in the conversation. Let me just say this real quick, right? You've got to remember that. And so I have a plan for that. I'll say it at the end. But basically at this moment, no, it doesn't, it has to be self-contained like all in one sentence right now. But I think I can fix it. You had a question? Yeah, so what you have now has been a surprise. Why not just do the math as you go? I'm sorry? Say it again? No, so you have Bayesian priors. I mean, you have something. Why don't you, I'm sorry. Bayesian priors? Bayesian priors. Okay, first, first let me say I don't know what Bayesian priors are. So to be honest, I don't do machine learning, right? And this isn't a machine learning approach. I will learn that. I'm trying to learn that. But I don't actually know that, so I'll just be frank. But also it's not a learning approach. Me, I see, I think the problem that, like what you were just saying is that, you know, click on this link, right? The information is just not present in the sentence. I have to use the previous information in previous sentences. And I'm not sure of what you're saying which would do that for me. Okay, I have to talk to you about that. I don't really know. I mean, I had another thought of how I would do that. Not with machine learning. Machine learning's fine. Everybody loves machine learning these days, but I don't know it, so. So I'm not doing it right now. You know, one day, I'm seriously learning machine learning now because people keep saying, oh, you gotta do machine learning, you know? Natural language processing, everybody does machine learning. All right, anyway. Oh, question, question, go ahead. Oh, question. Yup. Okay, let me repeat it just to make sure everybody heard. So you're saying that, again, context, right? Specifically, you're talking about the identity of the speaker, right? So it depends, so if your mom asks you your health, that's cool. If a stranger asks you your health, that maybe is not, right? So how do you differentiate, right? So first let me say I don't, right? But, so right now, I just assume everybody is suspicious. I trust no one, okay? That's basically because of a limitation of my approach. I can't authenticate people. I can't figure out people's identities. That's a whole other field, you know what I mean? There are people who do research in that. How do you find, maybe you can identify somebody based on their voice or something like that. But I have no way to identify somebody. So I'm assuming that everybody is untrusted, which is a weakness. And at some point, this work should be joined with some type of an authentication mechanism. Then if you're in some places, some workplaces, they authenticate. Like if you're working for the U.S. government, maybe you can enforce authentication. Every time you call somebody, you can identify. But in a casual working environment, maybe you can't. So I can't do that. And yeah, basically to consider that type of context, I would need the identity and I can't figure that out. So I would have to work with somebody else on that. Yeah. You guys are immediately hitting all the sore points. That's what I find good, though. Yeah, so just as a side note, I beg for money. I apply to the government for money to do this type of research, right? And just like I don't know, a month and a half ago I got a rejection on this. And these are the comments that they made, so. So I'm not surprised, that's all right, though. But I swear, I think I can fix these things in the future. The identification thing, I really, that's really a big problem. I don't know how to do it. I have to work with somebody else completely. But the context, the local context, that I can do. Question? I sort of just having a straight path to be able to start having it work. Yeah, yeah. So that lets the user know like, I'm starting to get threatening or questionable and the marriages are coming at me and I need to be more alert. Yeah. And I'll have to like, maybe just by like, if you're calling your doctor, you may be able to do that. So it's almost on your doctor desk when you verify you're being hurt. That's perfectly normal. That's what I'm gonna check you in. But then he says to be flagged, he's like, oh, I know what that is. So you can let him know I've been going. So let me repeat it. So basically you're saying, look, does it have to be a binary decision? This is bad, this is good. Maybe it could be, this is sounding a little more suspicious. My suspicion goes up, it goes up, it goes up. And finally there's a threshold. Yeah, that could be done in a pretty straightforward manner. I didn't do that, but that could be done. Go ahead. The purpose of this software that we're creating is it to terminate a call while it's in progress or it's a lot of the human element because if it's a lot of human element then we need to find a solution, right? Because sometimes you can get a call from your credit card and they say, we think your credit card has been used in, I don't know, somewhere in Africa or in Asia. So give us your credit card number or they ask you for some information like that. But if your screen pops up saying, you care for, you might then make the decision and I'm gonna find it. Okay, so the question is what's the purpose? Is the purpose to say, if there's a phone call and I detect it, it's suspicious. Is the purpose to terminate that phone call or to just alert the user and let the user make the decision? It's to alert the user and let the user make the decision. So now they might be idiots and continue and give up the information, but yeah, it is just to. You don't need the detection, right? You don't need the detection? I disagree, because if you don't, see the thing is if people essentially fall asleep, if you heard those talks today, right? They're not thinking, like their minds are somehow, are they like, look, I'm bored, I don't wanna be here, I don't wanna talk to this person, let me just answer these questions and be done. They're not aware enough, but if you put an alarm in front of them, it's like warning, attack, maybe they'll wake up out of it and cut it off themselves. You know what I meant was then you don't need the positive to be identified. Oh yeah, oh that's true. Yeah, but you don't wanna have false positives either. See, if false positives bother people, so you would like to have the identity. Okay, so no, that's good. Okay, let's keep going. Okay, so natural language processing. So it's a broad thing, there are a lot of parts to it. And I'm not really a natural language processing expert, I mean I know enough, I've learned it over the last few years to be able to do what I'm doing now, but basically the idea is you wanna understand human language, now that's very broad, right? What does it even mean to understand? So I've heard it said that you understand something if you can use the information to solve some problem, right? So I'm gonna say that I need to solve this problem of detecting if a sentence is malicious or not, and that's all I need to understand about the sentence. I don't need to understand everything about the sentence, I just need to be able to make a decision malicious and not malicious. So that's the limit of the understanding that we're trying to get, that the machine's trying to get. So understand that whatever the sentence is, it's a question or a command, and understand that the topic of that question and command is a forbidden one on a black list or it's not. Okay, so this is just sort of an outline of national language processing, and I'm not gonna go into depth in any of these, I'm gonna zero in on what part we actually are looking at, right? But natural language processing could be, research is done in different levels of abstraction. So for instance, phonology, we're talking about the sounds, the sounds that people hear, we're not touching that at all, right? I assume you've got speech to text, we're not even looking at the sound, the audio, we're looking at the actual text. Morphology is individual words and the meanings of words. That's not crucial here. More what we're interested in is syntax and semantics. So semantics is really what we want. The debitational meaning of a sentence, what the sentence means. But the meaning of a sentence is related to the syntax. So the meaning of a sentence is basically a combination of the meaning of the individual words and then the syntax, the order in which they're placed in the sentence. So sentence meaning equals words plus structure, right? Meaning syntax. So you gotta know what the words are, but you also gotta know they're, and this goes to what you were saying, right? It's not just sufficient to say, oh, it's got these words. If you put them in a different order, they mean something different, right? So for the structure, that's why we need to analyze the syntax. So we analyze the syntax to try to extract some parts about the semantic, the meaning of the sentence. Now, pragmatics, just so you know, is more high level understanding. It's more like understanding the meaning behind what's said, right? Which is beyond what we're doing, what I'm doing. It's more like, say somebody says, oh, you should try this toothbrush, okay? Now if they say that, and you guys are shopping at Walmart for toothbrushes, then that's fine, right? Then they mean literally what they say. But if you just ate some onions and they're like, you should try this toothbrush, then they're saying your breastings, right? That's pragmatics, right? That is beyond what, you know, and I see humor is based on pragmatics too, you know? But that's beyond the scope of this work. So we're really focusing on the literal meaning of the sentence and we're analyzing the syntax of the sentence to pull some of that out. Okay, so when I say syntactic analysis, so we're analyzing syntax. This is the arrangement of words and phrases, words and groups. So if you remember way back in elementary school, you learned grammar, you know? You may or may not remember this, but you know, and I hated that, but now I've had to go back and learn a bunch of it again, right? For this purpose. But I don't have to know it all because basically there are parts that deal with most of it for me, but I have to learn the basics. But the idea is, you know, words have parts of speech and they're grouped into phrases, chunks, you know? So you have noun phrases, verb phrases, prepositional phrases and all this. And this describes the syntax of the sentence, the structure of a sentence. So, oh, and here's my little example, here's one example where, you know, so Pete killed the job versus the job killed Pete, right? Those two, the same set of words, I reorganize them, they mean two very different things, you know? So I'm just trying to say, look, you can't just look at the words, you have to look at the structure of the sentence as well. So we have to analyze the structure of the sentence and for that, we do parsing. So with a context-free grammar. So you may not know what a context-free grammar is, some people do. If you know about compilers, you know, compiler theories, this stuff comes, actually this came straight from compilers, or maybe vice versa. Anyway, what you do is, so we got the sentence, the man took the book, okay? It's in red and it's parsed. So on the right hand side, you see that parse tree. And the parse tree shows how the words are grouped into phrases. It labels each word with its part of speech and then it shows how those words are grouped into phrases and to combine to make a total sentence at the top. And the way that parse tree is made automatically with an off-the-shelf tool, right? I did not build that tool. That's like a standard thing. Making parsers to create those parse trees, that's like a standard known thing, okay? And you can just get them for free. Stand for parsers is what I use. So you give it though, you give it a grammar, BNF, this is sort of a common form for a context free grammar. And this grammar basically on the left is just the rules of English, okay? Now this is like a tiny subset of the grammar for English, but so that you can see the top one says S, S is, each one of these letters, these symbols that they're called tags. And this is a standard tag set, the pen tag set, there's a standard tag set. So S is for a sentence, okay? So the first rule says a sentence can be a noun phrase and then a verb phrase, followed by verb phrase, okay? And then the next one says a noun phrase can be a determiner followed by a noun, right? And so on, right? And then, you know, a determiner is the and the noun could be man or book and verb could be took and so on. So given a grammar like this, you can create a parse tree and there's plenty of tools to do it. And the parse tree, you can see how looking at the tree, you can see how the words are grouped into phrases, okay? So this type of information we need to analyze the structure. We wanna see what the phrase structure is so we can infer something about the meaning. Because basically, in order to recognize if something is a question or something is a command, you gotta know about the structure of the sentence. So the first thing we do when we get the sentences, we hand it off to a parser and the parser gives us this parse tree that you see on the right. Actually gives several parse trees that are possible and we take the best ones. Okay, so then once you got the parse tree for the sentence, then we look for patterns in the parse tree that match what we're looking for. So we're looking specifically for, first we're looking for questions and commands, right? And so, and there, you can see them as patterns inside the parse tree. So if you see a particular subtree, then you say, oh, that's a question. And if you see another subtree, oh, that's a command. Now how did I find that, figure that out? I went to many grammar books, like thick old English grammar books, and they have many rules. Oh, you know, here are all the hundred ways to write questions, right? You can write a question like this and put this first and that first. You can write a question like that that there's a long list, okay? And you can look at them, but looking at that, I can say, oh, based on that, I can, that, actually, I didn't do this. My student Yuki did this, okay? I take too much credit. I did part of it, but he really did it. But he looked for patterns in the sentences and said, oh yeah, that matches the rule in this part of the book, okay? So questions can be formed like this and like that. That's this pattern in the tree. So for instance here, this is a question. Can I eat, okay? Now, this is a question because it, so can is a modal verb, okay? And I is a pronoun. And if you invert them, if you have them in that order that you see them there, the modal verb and then the pronoun, that's a question. That's the type of question. Now, if you reverse it, if you have pronoun and modal, so if you say I can eat, then that's a statement. But if you switch them, then it's a question, okay? So that's the type of pattern that we can see in the parse tree. We can look at, so we see in the parse tree where it says above can, it says M-D, but like modal, like ah-ha, go ahead, question. Yeah, yeah, okay. So what he's saying is the problem with that is inflection. So inflection meaning the way you're speaking, the tone of speech, right? So another way to do questions is to put da-da-da-da, raise the pitch at the end. That's a question, that's true, right? You know, and you put that pitch, I don't deal with that. Okay, that's phenology and I don't deal with that mostly because I don't know enough about digital signal processing to handle that. There are people who do research in that, but it was more complexity than I really wanted to deal with. But you're right, you can get a lot of information from the sound and actually to tell you the truth, the next thing, the thing I have a student working on right now is telling where sentences end and the next one begins, right? You can tell that a lot of that from the sound and we need that, right? Because we have to parse sentence by sentence. So I need to know, oh, this sentence ended and I need to look at the sound. So I have a student looking at it right now. There are tools that do some of that. Question, oh, sorry, go ahead. Yeah. Okay, he's talking about the verb. Go down to the train. Yeah. But you deal with differences. The different senses of the verb? I'll say no. Basically, because in the end what I'm gonna do is I'm gonna take the verb and the direct object and look them up in my blacklist. And I guess, yeah, I don't. I guess if there was a sense difference like that, like, you know, take, yeah, take train, I mean, it's hard to imagine. But I mean, I know what you're saying. Yeah, I know, I know, but I'm trying to think of a social engineering context. Yeah. Yeah. Yeah, I'll say, I mean, I don't deal with that now. You're saying the fact that verbs can be used in many senses based on, and you can tell what sense of being used based on the direct object, right? You can say, oh, I'm taking a train, I could be taking, or if I'm taking a ball, I can't be riding the ball to the street, into the, you know, store or something. Yeah, I don't do that, I don't do that. But yeah, it's possible, but yeah, I don't do it. It could be useful, I can't, yeah. Okay, okay, so there are patterns in the tree and that you can recognize, and so that's basically what we do. So there's a pattern, it's called T reg, T reg X, you know reg X, right? Regular expression matching. Well, there's a tree version of that called T reg X, then my student writes patterns in this pattern master and it looks for patterns in the tree. And so we can see, so based on, you know, on the parse tree, you can find sub trees. Basically we're looking for sub trees as a parse tree and when we see a particular sub tweet, we can say that's a question or it's a command and know that we should examine it more closely. Okay, and then we have topic blacklist. So like I say, we're gonna get the question, get the command, try to figure out what the topic is, the verb and direct object, and then we're gonna look it up in a blacklist. So somebody has to manually make a blacklist of these action resource pairs, action being the verb, resource being the direct object, right? And somebody just has to make that up ahead of time. And so we have an example one that we use, but somebody has to give you that. And you know, you can have, you know, click link or send money, stuff like that, you can put in your list and we just look through the list. Or you could use a wildcard, you know, if anybody asks about the firewall at all, then you'll hit it, something like that. Okay, making a topic blacklist. Yeah, you gotta do it manually. Oh, question, I'm sorry, go ahead. Yeah, I'll just take this one, right? I'm sorry, I didn't hear the question. I know, like I'm showing there, I put an asterisk for the action. So I just put a, at the bottom, you know, a star and then firewall. So you could leave it alone. But then it would catch anything, anytime somebody's asking you about a firewall, it would warn you, which might lead to false positives. But you don't have to have, it's strictly have an action. You pretty much have to have a resource. Or put it like this, you don't have to have a resource either, but it's hard for me to imagine a case where you wouldn't have a resource, but yeah. So yeah, you tend to get false positives more if you leave it more open. Okay, so yeah, you have to make this and typically when you make the blacklist, I would imagine you'd take into consideration whatever resources you have to protect, you know? So if you're at a bank, then you wanna have bank specific stuff. Say you got some organization, you know you have network equipment, then you would say manipulate network equipment. You wouldn't want somebody to tell you to go reboot the thing or change the password on this network equipment. So you put stuff specific to whatever you're trying to protect, I would imagine. Okay, and then detecting malicious statements. So you detect the question of command with a pattern. I'm just showing a little, what I'm highlighting there is the subtree that tells me, okay, this is a command. So like in this instance, so there are several ways to make commands and imperative statements, right? Like this sentence is reset the router. That's a command. So how do I know that's a command? Because you got the verb and there's no subject in front of it, okay? So I didn't say Joe resets the router. I say reset the router. There's no subject in front, right? So then it's a command, right? And that's a form of imperative sentence, right? So that pattern, where if I see this subtree that I've highlighted where there's a verb phrase with a verb in the front and then a noun phrase after it and there's nothing before the verb phrase, no noun phrase before it, then I say, okay, that's how I detect a command or that's one pattern for a command. There are a few more. Okay, so based on these rules of English grammar, we make these patterns that are subtrees and each one of these patterns, you have to be able to find the action and the resource, right? The verb and the noun. So each one of these patterns would have associated with it based on rules to find the verb and the noun. So like this guy, that pattern that I've highlighted, see under the VB, right? The verb under that, that is the action. And then under the NP over there, the noun that's under that, if you look further under the NN, the noun, that's the resource. And so once I find that pattern, I can say, okay, here's my action, here's my resource and pull out what I'm calling the topic of the question or command, command in this case. And then I basically take that and look it up in my blacklist. And if I hit it, then I say it's an attack. So this is the structure of the system. Just say, oh, go ahead, question. Ah, yes. So somebody's speaking incorrect English? Yes and no, you know, go ahead. Yoda? Yeah, Yoda would mess it up. Yoda in his damn force. I would say, they would, well, I don't know how your English is, but you know, I would say, look, it depends on the parser because the Stanford, so first, my first stuff I would say, yes, it would tend to fail, but the Stanford parser is pretty darn clever. Meaning they use machine learning to train that thing. It knows a lot of broken English, you know, and it'll parse correctly in spite of the fact that the English is broken. So it may work anyway, but that wouldn't be because of me and because the Stanford parser is smart. But generally, yeah, it would tend to fail if English isn't correct. And which, by the way, is a weakness, right? Because say you're looking at texts or something, people write garbage all the time. Yeah, yeah, I don't know, but yeah, I don't know what, yeah. But then, yeah, that's right. But you know, I could also argue that, you know, when you're having a person-to-person conversation, usually, if you're trying to social engineer somebody, you're usually not using broken English. But I don't know if that's true, maybe I'm wrong. Yeah, that's, yeah, yeah, generally you want to be clear, right? Actually, in fact, that brings up something, another complaint from my rejected proposal, which I'll bring up with you guys to what you think. So one complaint about this was that they said, look, you know, this is gonna be an arms race, right? Because as soon as you get this system working, then attack or social engineers will say, oh, well I'm gonna say something in a way that this doesn't detect, but all of you will go out there and figure, you know, you'll see what I'm doing and do something to mess it up, right? And so it'll be an arms race and then we'll go back and forth like malware, right? So I would say that it's not as bad as malware. Malware can change forever, you know, you can detect it then they twist it, but with human speech, you can't change it so far. I mean, humans only understand so much. When you start saying, how many ways can you ask a question? If you garble it too much, the target won't understand what you're saying. So I think there's like a limit, like you can only go this far, right? And then you can't go beyond that. So I don't think an arms race would happen. I mean, eventually you'd cover the space of what humans can understand and then you'd be done, you know? Question? I can trick your system to do false positives all the time and they're gonna turn your system on. Yeah, yeah, so yeah, if you can trick my system to do false positives all the time, nobody will use my system. That's true, that's true. And look, in the state that it's in, like I say, you know, it missed like four out of 10, right? In the current state that it's in, I wouldn't use it. But all I'm saying is this is a start in the right direction is what I'm hoping, you know? Yeah, I'm not saying this is the end at all. Yeah, that's true. And you could go for a long time trying to minimize the false positives, adding more and more context, to be more sophisticated about it. Yeah, I would imagine that it would take a long time before it was tuned well enough to be really useful. Question? Wait, I'm sorry, I'm having a hard time hearing you, but I think you said if the attacker is feeding the target information or the other way around. Yeah, but I could detect that question, you know what I'm saying? I'd be like, oh, somebody, what? I mean, if the other questions have the action and so for instance, manipulate network equipment, right? What do you say, like, give me an example of what you're saying that has no action. There's a verb in there, right? Running, running's a verb. You know what I'm saying? You can't have a sense without a verb, I think. I'm sorry? If something exists. Yeah. I think you're learning what you want. You want to start confusing your target. You're, look, what you're talking about, talking like you're without it. You're talking to your target for your life. I wouldn't hang out with you. You know, I'll tell you something, something to go, something that, something about what you're saying is, so the verb is, is is quite a pain, right? Because is is very, it's ubiquitous and generic, like it's a useless, you know, it's a sort of useless, is alone and it doesn't tell you anything about the action, right? So in those cases, you have to use other information to sense. So for instance, if you say where is something, then the is doesn't tell you what they're asking for, but the where does. So, and I haven't done that yet, but that's what I would have to do. I would have to use other words. But I can, I can detect yes or no questions and see what resource they're asking about. You know what I'm saying? I can, I can detect yes or no questions. And I, and if they're asking, okay, okay. I mean, look, yeah, I don't think I totally disagree with you on this, but anyway, I'll keep going. Okay, any questions? Okay, all right, so here's the system and I already said this, but I'll say it real quick. So we parse a sentence. So you get a sentence, parse it, you get a parse tree, you detect if it's a question of command by finding patterns in the tree, then you extract a topic by basically from those patterns, you find the verb and the direct object, the action and the resource, and then you basically compare that. You look it up in a black list and if it matches, then you, then you hit. And that's it. Oh, and for this, we use plenty of off-the-shelf stuff. So parsing the sentence, we don't do that. We use the Stanford parser because it's a really good parser. You can just download it and use it. And it has a, it's pretty, pretty thorough. And we use TRJX to detect the patterns. Also part, actually it comes with the Stanford parser download. It just finds regular expressions, subtrees in the tree. And we detect the patterns using that. Okay, and yeah, okay, so just some related work. I'll be kind of quick with this. Other people who have done similar related things. And these are kind of old, so I don't know if they're useful. Yeah, maybe I'll be quick. Fishing attacks, so like with fishing attacks, fishing emails you want to detect them. A lot of these fishing attack approaches, what they do is they look for, they don't look at the meaning of the text, okay? They look for other things about the email that looks suspicious, like the URL, URLs and stuff like this that are in the email. But they don't look at the text and try to extract the meaning of the text. So that might work on an email, but not like in a text, not in a dialogue. Oh, go ahead. Yeah, yeah. Yeah, okay, so let me just say what you said. You said, look, there's been 30 years of research in fishing detection and still it happens all the time, right? So what does that say? It says they suck roughly. No, not that, let me not say that. You know, that's gonna be me. It says it could be improved. Okay, it says it could be improved. That's all it says. I know, I shouldn't say that because probably people who did it are right in the room, you know? So I'm sorry, that was a joke. Don't take it seriously, question? Okay, so first let me say not enough, right? What we did was we took, yeah, well, we'll get to the slides. You know what, let me go straight to them. Yeah, people are asking. Yeah, okay, sorry. Yeah, that's it. Okay, so the examples. So we got three examples. So I'm Chris Hadnaggy. We can't reveal a whole example, you know, because people don't like it when you reveal, when you expose their attack against them. So, you know, I just put little bits of it in here, but we got to look at them. There are three attacks and three dialogues, and there's a number of sentences in each dialogue and the number of violations. So violations is the number of sentences in there that are malicious, that are either asking a question that they shouldn't ask or asking you to do something you shouldn't do. So, across all of these, there are 10 violations, okay? 10 sentences that are malicious that we should be able to detect. And we detect, oh, and here's the topic blacklist list that we used. This one, now we also use synonyms and hyperonyms. We didn't put them all in here, but synonyms work too. You have to pay attention to synonyms. But I didn't put them in here. And then what do we get? We detected six of the 10. So this is what we did. We did it on these three dialogues and we detected six of the 10 with zero false positives. So that's not nearly enough. And actually, that comes to my biggest complaint about the work is that I haven't validated enough, right? So for that, I need lots of social engineering attacks that I can scan through and I cannot find them, okay? Now I hear them, I heard them all day today, right? But I can't record that. So it's a real problem, right? I mean, I can run it on just data that I know doesn't have social engineering attacks in it. But I wanna run it against attacks so I can see if I can catch the malicious sentences and I just don't have any. So if anybody has any, let me know. Yeah, so I did it on as many as I could. Yeah, okay. And then let's go back real quick to what I was saying. Okay, so some of the patterns, benefit-or-approach, yeah, this is related work. This is the thing that we did where we just looked at the words, okay? And we didn't look at the order. And we did it, actually what he did was, well, my student, that student, this is Rom, one of the same students, what he validated on, he looked around for just, you know, scrape the internet, finding people who posted attacks. So basically, people like got harassed by somebody trying to lure some money out of them so they decided to engage in a conversation and record it and then post it, okay? So he found three things like that. And the thing is those aren't exactly good because the person, the attacker, the target knows they're a target, right? So they are not responding in a natural way, you know? But anyway, that's what he validated on, you know? This is previous work, though. And actually it worked pretty well. It worked pretty well. It was mostly tech, like Facebook chats, two Facebook chats and one email conversation, you know? And that was sort of broke, like no sentences, like people didn't put periods, no capitalization, you didn't know where sentences started and ended. So it was harder to analyze. But what he did was such a simple analysis. He just looked line after line, he looked for, actually he did the same topic, blacklist thing, except he took the action and the resource and looked for them on the same line. So if you find a line with the action and the resource, he said, okay, that's an attack. And it worked like 80% of the time. But it wouldn't work in dialogues, you know? Dialogues are more complicated. Generic text, you know, it works all right. But so, yeah, and now I have my, here's an example. A specific example of where I use these two words, reset and router, the first sentence is a command you shouldn't do. The second sentence is just a comment, right? And it has the same two words, but they mean different things. And so we could differentiate between those two. I would say, oh, the first one's a command and the second one is not, right? And I wouldn't detect the second one. I wouldn't, and also wouldn't, I wouldn't associate the verb in the direct object. I'd be able to tell that the router is not the direct object of the verb reset. So I wouldn't associate them together. So I would differentiate those two. Now then you might ask, well, how often does that type of thing happen? I don't know, you know. But I think there will be cases where, you know, you can't just look at the words if you have to be able to look at the structure of the sentence. Okay, so command detection, I already gave you one of the patterns, you know, go home, stop right there, you put the verb in the front, you don't have a subject in front of it. So you can detect that pattern. This is reset the router, same thing, right? There's a verb, and there's a noun phrase after it, but there's none before. Then there are these softer imperatives. So often direct commands, you don't necessarily want to command somebody to do something and, you know, it wakes them up. So you might say in a nice, oh, please do this, you know, please reset the router, right? Or if, you know, or actually you might say it like this one, you should reset, you could do this, right? You could, you should, da-da-da-da, right? So this one we're detecting, I'm calling it a soft imperative. That's actually not the term, the grammar term, like in English grammar there's a word for this type of a sentence which I don't remember. But basically that should, could, that's a modal verb. So if I see a you, so basically a subtree of this tree, I see the you, and then I see some modal, and then I see a verb, then I see a noun phrase after that, then that pattern. So that subtree of there, I can say, okay, that's one of these softer imperatives where you're sort of suggesting, you could do this, you should do this, you know? You might, but still it's a command, it's just sort of a softer way of saying it. Question detection, okay, we already did this one where you invert the modal and the subject, and actually the parser catches this immediately, that S-I-N-V tag, that means the subject's inverted, this church for S subject invert. So if you see the subject, the S-I-N-V tag, you know it's a question, and it's with an inverted subject, the modal order. Also S-Q is a similar tag. So that's actually a simple pattern, you just see S-Q or S-I-N-V, and you know it's a question. Open questions, so these are these, who, what, where, when, why's, and actually the parser catches all of these, it has tags for every one, if it sees who, what, where, when, and why, and how, all being used as a question, it detects them and puts it in the appropriate tag. So for instance here, you see in there that WHNP tag over the what, right? That tells me, oh, this is a what question, right? And so there, the parser will actually catch that. If it sees the who, what, where, when, why, how word in there and the sentences in the question structure, it'll put that tag in there. So all I have to do is see the tag, and I know it's that type of, it's an open question. Go ahead. If you could start with something such as your password please. Your password please. That would not, I would not catch that. Yeah, yeah, that's true. Nobody has said that in my six examples, but yeah, I would catch that. Okay, and then topic extraction. So each one of these patterns, they're associated with an action and a noun phrase, and you can pull them out. And detection algorithm, I already said this, is just restating it more formally, except to say that parse and best, so the parser, it doesn't just come with one parse tree, it comes with a bunch of parses that are legal, and we just pick the best one and use that. Or the end best, and we look through, actually yeah, 10, we're using the top 10, and we're looking to see if any of them are attacks. And then, you know, you extract the topics and you look it up in the black list, just like I was saying. And then here are the results that I already told you about. Yeah, results summary, so yeah. 60% recall is because it has the false positives, sorry, no false positives, because it has, because it misses four out of the 10 senses. So some examples, I can't show much of these dialogues, but let me just give an example. So there's a true positive up there. If you could email me the cert, that would be great. You know? So yeah, that's positively an attack. So, and you can see you could, so that's one of my imperatives, right? You could email me, email the verb, cert is the resource, and we can catch that. False negatives, there were four of them, and they were all, they see these ones that needed context. Click run, that was one of them. Yeah, click okay, that was another one. So you need context. You could use linguistic context, which is a context in that conversation earlier in the same conversation, okay? So for instance, you say click run, what are you referring to? Oh, well two sentences ago, I told them to start this application, so I must be talking about that application, right? So you can keep track of noun phrases that are used in previous sentences, and then, you know, associate, you say click run. Well, which one of these noun phrases, let's say you talked about a tree, you talked about a web app, and you talked about this, oh, web app. That's the only thing where I could click a run on, you know, and you'd do that. Now I'm not doing that, but that's what I need to do. Also, situational context might be sufficient, meaning when you say click okay, or click run, I mean, even without knowing this, even without seeing the attack, we know that some, there's some program running, and it says okay in front of them, and there's a button that says okay or run, you can click it, right? So clearly they must be talking about something running on the machine. So you can know stuff even without knowing the actual linguistic context. You just have the situational context, meaning the context that we all know just because we have life experiences, right? So we might be able to use some of that. Room for improvement, I've already said this. So look, using context. Yeah, this is the main thing, but other things too. And I think, oh, and authentication, so I mentioned this too, the fact that everybody is untrusted as far as I'm concerned, and you know, if your mother asks you something, or a stranger asks you something, it makes no difference to this tool. So in order to do, I can't do authentication, but I would, you know, it could be, if somebody else could do authentication, then you could join this with that, right? Go ahead. I don't care who's the problem. So if you don't know the attacker, you can know the attacker. Mm-hmm. Well, if I'm a forever, whatever I am, and I'm going to work survivor, I'm obviously going into the wrong direction. Yeah, I'm going, I'm doing it. I'm obviously going into the wrong direction. I'm doing something beyond the scope of what I see. Ah-ha-ha-ha. Yeah, so you, I see. So you say, so I can look at, you have to see, okay, so what, let me just repeat what you said, right? You're saying, yeah, let's say I'm a clerk, I'm the target, I'm a clerk, and you know, I'm being said, oh, I'm being told to go reset the router. And the clerk is not supposed to be resetting the router, but, and I know who the clerk is, I can know who the clerk is, so I can look at what they're authorized to do. That's true, but there could also be a situation where the target is actually authorized to do exactly what they're doing. You know what I mean? And then you would- Then you're an S.O.L. Yeah, yeah. So, but yeah, you're right. Yeah, if they're being asked to do something they shouldn't do in the first place, then you should be able to catch that if you authenticate the target. And that's it. Any more questions? Thank you.