 Okay, let's take a speaker and let's invite the speaker and then thank you. You can go ahead. Hi My name is Venkat. I am a professor at the University of Illinois at Chicago Today I'm going to be talking about I'm a system security researcher I'm going to be talking primarily about the work that I have done In collaboration with some of the folks in this room about About an approach for detecting advanced and persistent threats as part of DARPA's transparent computing program and I will primarily summarize our work from system security viewpoint and then Finally share my thoughts about how where we are going with this and how AI research can help so Just to sort of Give some background on the problem Targeted cyber attacks on organizations are getting more sophisticated stealthy and serious Generally the goal of the adversity is to steal data Operations or destroy infrastructure and it's carried out over a much longer time frame sometimes weeks sometimes months And they remain undetected most of the time and it's often customized to the target rather than using some sort of commodity tools or malware and Thousands of incidents every year affecting large to you know small to large organizations We heard about the Equifax, you know Married chains was attacked and The cost could exceed you know 100 billion for the US economy so I Just a quick summary of the state of art in this space, you know, the APT is leverage different attack methods span multiple applications and hosts and Enterprises today collect a wide variety of audit information like security information and event management and intrusion detection systems application firewalls Identity and access management tools All this is very useful, but it's hard to correlate information across these tools and therefore When an attack happens We can't really produce the big picture right and often this is done by cyber analysts involving a lot of manual effort to piece together all the information that these tools have and And so our goals are to automate this process analysts shouldn't have to manually search for interesting events and We want to you know, and then typically the information collected to assist this sort of scenario reconstruction It consists of millions to billions of records So how do you want to organize these information in such a way that? that that we can we can produce the big picture and We also want to produce sort of a Compact summary of the causal chains that summarize the attack Enable you know and we want to do this in real time So that an analyst can quickly assert and whether there is an intrusion understand the root cause and determine the impact and And also our goals are also to assist with cyber threat hunting So when threat intelligence becomes available, we want to know whether a particular threat occurred So the main The main vehicle or tool that we use in organizing all this information from systems or provenance graphs Very simply put they are directed acyclic graphs whose vertices represent system entities like processes sockets files and agents users and groups and Edges represent some sort of dependency relationships like you know information flow dependencies between say subjects and objects and The the nice thing about provenance graph is that they they sort of leverage the full historical context of activity in a system And the reason about the relationships between different events and objects, right? and Primarily the provenance graphs in security have been used for forensic purposes, you know, there has been some influential work back in the back in the early 2000 and around that in using provenance graphs for for forensics and But there are quite a few limitations We are looking at problem a problem where the number of events that we have to process scales from hundreds of millions to billions per day right and To operate on these very large graphs might take hours or days to complete it's not timely if we want to do real-time threat detection and forensic analysis and There are you know because the information collected is quite coarse at the level of audit records There is also the problem of dependency explosion where the graphs might include Many irrelevant, you know dependencies to the final scenario so as such, you know provenance graph have not found sort of widespread adoption in you know threat detection forensic analysis and But but they are quite promising and so we set out to actually use this provenance graph framework to To actually do real-time threat detection and forensic analysis. So So there are three main results may primarily from the DARPA transparent computing program and now we are part of this program called Chase So the first is a paper that appeared in use Nick security For you know for a system that we call sleuth and second is a system called Holmes that we That we published in Oakland this year and third is another system called pyro that is going to appear in CCS this year so this talk is sort of a summary of these three results what we have done with These systems and also to show how AI Can help advance the work that is laid by these by these systems, right? so So sleuth was the first paper that we did that we did as part of this project. So basically the Architecture of sleuth looks like this. So we take audit logs from Windows Mac Linux and there is also some implementation for Android and we organize this as sort of a dependence graph in main memory and then we define In our policies that trigger alerts on this graph and once You know, there are these alerts we We so the the graph that is laid in memory is tagged it means that it has information flow labels and that allow us to Identify the root cause of the attack and also do some impact analysis So once these alerts that are defined by policies are raised We are able to traverse these graphs both forwards in time and backwards in time to identify the impact and the root cause So So what are tags? So for instance, you know subjects and objects are assigned Trustworthiness and confidentiality tags, right? So there are different confidentiality levels You know, you can think of this more like, you know, the security labels work that was done in the 70s on You know multi-layer security and so on So but here we have, you know carefully curated these tags so that, you know, they work well for this problem so So we distinguish The notion of code trustworthiness versus data trustworthiness in order to control the propagation of labels Processes have two tags the code trustworthiness tag and the data trustworthiness tag and As you can see the separation aids detection and speed the analysis by Focusing on a fewer set of root causes and then we define confidentiality tags like secret sensitive private and public Now we define policies based on these tags that are then propagated You know on the dependence graph so tag assignment and Propagation are Customized using policies. So for instance, you know, the tag initialization rule might look like this. It might say that An object with a certain name You know is assigned a benign tag if it is from an internal IP address or If it is from an external IP address, it's assigned the unknown tag, which means that it may be untrustworthy, right? and Now when tags are initialized, they are propagated in main memory from input to output in the graph and There are, you know, of course custom policies that override these, you know, standard rules So so now we can talk about how we can detect attacks using Using Providence tags so the approach is to focus on You know the motive to ask whether it is possible for an attacker to advance some high-level objective like, you know, run, you know, some untrusted code replace some, you know, important files in the system or do some data exfiltration and Means can the attacker control, you know, an action. It is the process performing an act that is untrustworthy, right? So the nice thing is that it is sort of doesn't require any labels once we define these very general policies There are probably six or seven of these policies in our system Then the reasoning happens nearly, you know, automatically, there is no need for training and Assuming that the province data is not compromised this sort of resists any adversarial manipulation so So these are the main four policies in the system untrusted execution Suspicious modification data leak and untrusted execution preparation as you can see they can be defined, you know, fairly easily using the tags that we have set up so So here is a small example where it shows the use of these so here is Firefox that is actually you know downloading This file, you know Bob CRT 1.0 that is then subsequently written to slash lib that gets linked to every executable That is produced by a user Alice, right? Now the tags allow us to capture these events as they happen in the audit records so for instance if when the modification to the bash RC file happens which Then which then subsequently links this file to CRT 1.0 this ready this produces a subsequent suspicious modification, you know Modification policy this raises an alert and then there is there is the untrusted execution preparation alert when the loader actually links this file so as you can see all these alerts are raised and We can traverse the graph, you know provenance graph, you know backwards to understand the root cause of this attack so so Now I just briefly talk about the kind of adversarial engagements DARPA did under this program. There was a red team that was tasked with Producing APT style attacks on a private network and data this these attacks range anywhere between a few days to a couple of weeks and The data was made available to performers like us and we had to detect and report these attacks in real time So this sort of gives some This was only the first and second engagement that this system. I'm reporting the results on subsequent I will talk about some subsequent engagements later but these were relatively Short engagements the longest Running attack spanned 79 hours attacks were only carried out during the business hours of a particular day and And these were sort of the alarms that were raised When in that data so as you can see in 39 million events we could see only five untrusted execution and one suspicious modification alarm so fairly, you know The reduction in the amount of data cyber analyst needs to look at is very significant and We can then do the forensic analysis starting from these alarms and So one common analysis that we do is the backwards analysis whose goal is to identify the entry point of an attack so every The entry point is a source which is in the provenance graph of vertex with in degree zero and The starting points are the vertices that are suspect marked by attack detectors and The problem is to find source vertices from which a suspect vertex is reachable The the complications are that there may be multiple sources multiple suspect nodes so So what we do is to define weights on this graph in such a way that it prefers shorter paths over longer ones and Prefer edges that correspond to a flow of untrusted code over untrusted data and these preferences are encoded using some custom edge weights to Dijkstra shortest path algorithm and Similarly, we can also do for forwards analysis for goal of forwards analysis to see what the attacker did from a given alert point Right, so it is sort of an impact analysis backwards analysis identifies the root cause, right? so Again, if we do a straightforward forwards dependence analysis, this might yield a graph with hundreds of thousands of nodes So again, we have to be careful here And so we define cost matrix where we follow the edge with untrusted code untrusted code trustworthiness labels Where it is given a lower cost and to some extent we also follow the untrusted data labels, right? so once you do that then the kind of Scenario graphs that you produce Doing both the forwards analysis and backwards analysis kind of summarizes the whole attack. So here is the the final scenario graph produced from you know several millions of original records So as you can see the graph here the circles are all the process names the Rectangles are all files and diamonds or sockets and the edges are directional they denote the flow of information and The the numbers on the edges denote time stamps So you can see that you know firefox receives some data from a socket and then it forks this executable called Mozilla nightly which goes on to perform some intelligence gathering in the system like You know running a few system commands communicating with an external IP address and finally All this information is exfiltrated to another IP address in this in this edge 40 so all this information is Is is summarized, you know in a compact way in this graph, you know Here is another graph that was done on a set of attacks on a free BSD system So here is an is a graph for For attacks done on a Linux system. So in general, you know, this is How our system reconstructs the whole attack campaign you can see the number of entry points for For Linux, for instance, this was a data set that had about 40 million records the number of entry points programs executed key files involved and Most of them were sort of correctly identified except for one instance where Due to some label propagation. There was some error, but otherwise So the the important thing was that none of the attack information was made available before the engagement, you know concluded so when we Did the actual exercise with DARPA We identified all this before any sort of ground truth became available and then We this is the table that compares the ground truth information with the actual detection results of our system and We have done fairly well So I think that sort of summarizes what we did with sleuth and Now I'm going to just switch gears and talk about a second system that we built on top of sleuth That we call homes where homes We have identified ways by which we can improve the detection accuracy by adding support for custom rules, right, so this appeared in Oakland earlier this year and Homes makes use of the inside that APT behaviors Often confirmed to certain kill chain, right? This is there are different kill chains out there in the literature But this is the sort of prototypical one that was published by Mandy and you know several decade back so So we you know when we started this project, you know we did an analysis of about 300 APT white papers and most if not all APT is confirmed to this kill chain, right? In particular the high-level steps of a campaign are you know causally connected and Our inside was that we could use this connectedness between the high-level steps as a basis for APT detection so here is for instance a detailed provenance graph for a From a system where the activity in green represents the set of benign activities and the and the activities in the red box denote the the attacker activities and As you can see there is sort of the engine x server that reads from an external socket and then makes a piece of memory executable representing sort of an initial compromise phase and then there is a CNC communication with the external world that denotes some sort of a foothold establishment stage and Then privilege escalation is performed so that you know the user gains access to super user privileges There is engine x then performs a number of commands that constitute some kind of internal reconnaissance and then cleanup is done so that the tracks of the attacker are completely removed and Then finally engine xx will trades all the data sensitive data read from the system So as you can see The various phases of the attack as summarized in the high-level steps below They are all sort of connected in this picture and whether we can use this connectedness as a basis for detection So the key challenge here is how to fuse these point alerts representing various high-level events into a strong detection signal and if we manage to do that we may be able to summarize the high-level Activities of the attacker in the form of these steps and that may be even more useful to a cyber analyst Than the kind of graphs that I showed earlier that were produced with slow, right? So the main approach is again to use Information flow as a basis for alert correlation and campaign detection So how so there is the set of audit logs that represent information in very low-level events like system calls And then there are these APT stages like initial compromise established foretold Internal recon and so on so there is a big semantic gap between these ideas and the actual information present in the audit logs So we bridge this gap by way of you know tactics techniques and procedures that is from the the attack framework from Mitra and We so once we describe, you know the various tactic techniques and Procedures that allow us to kind of seamlessly connect the information in audit logs to the high-level attack steps So so again we use the front end of the sleuth system so there is the provenance graph laid out in main memory that allows us to do fast graph analytics and then we describe the TTPs you know in that as You know as matching rules on events described in the provenance graph and once there are matches to TTPs They are the connectedness between the various high-level events is summarized in what we call the high-level scenario graph and We have a way of generating a signal from each of these high-level scenarios and once that signal crosses a certain threshold We declare that particular campaign is ongoing So that is the high-level idea behind homes, so here is Particular TTP example, so the initial compromise state Stage is on you know involving a certain process P Involves you know one of these you know TTPs So for instance there could be an untrusted read in our TTP where the the read set of events constitute the event family and This could be different for different operating systems and this is a relatively low We associate a severity level with each TTP and then perform Checks on what the prerequisites for matching this TTP are in this case We say that the socket needs to be from an IP address. That's untrusted So likewise that is a second TTP that describes when certain piece of memory is made executable and The established foothold involves you know some shell execution So once we have you know sort of develop the system where we this describes About you know 45 TTPs that capture a variety of common tactics used by attackers on these operating systems and now So so one thing you may have noticed in the rules is that there are certain prerequisites for each TTP So they include information that may be Derivable from you know you want arguments like file names and permissions, right and They allow us to connect two different activities of a campaign and help reduce false alerts as part of a campaign So an important problem to address here is the dependence explosion problem spurious dependencies can result in dependence explosion and so we address this by asking a key question of And and defining a notion of influence we ask what is the influence that an attacker had in affecting any particular dependency and and so we develop a notion for Any flow F or a dependency F we develop the notion of ancestral cover where we say You know a set of all process that influence a certain set of dependency at certain dependency F is the ancestral cover for F and We want this minimal ancestor cover To be the number of minimum distinct compromises an attacker needs to affect in order to create this dependency So the example is given here so bash You know forks engine X which then writes to engine X error dot log Then cat reads this file, you know and and you know SSH forks cat This is really a false dependency because you know engine X error dot log is a log file That is affecting the dependency whereas look at this Firefox forks bash and writes to recon dot sh Which is then read by bash. This is something that an attacker who has compromised Firefox can easily control as opposed to Controlling this dependency of you know cat from engine X error dot log This may be very well be a benign activity in the system But this is something that the attacker controls because attacker can actually fold bash and affect this dependency so so We in order to calculate, you know these notion of influence we define Condition called path factor which is computed incrementally as we build the high-level signature graph and and the value of this path threshold could be set based on The threat levels and organization phases. So for instance by default we can say an attacker needs to compromise utmost to Independent, you know exploits in order to affect certain dependencies in an enterprise system And that sort of works well, but more of this could be derived from empirical evidence data so and Then we add a bunch of noise reduction techniques to in order to generate a signal a strong signal So the idea is to exclude from benign, you know matches to tactics techniques and procedures to to reduce noise in the overall system and so So what we actually get in the end is this high-level graph that describes the various steps of an APT and each of these internal nodes is actually a TTP and you can see the Edges of this high-level graph or edges between the dependencies between the various entities here So in effect we are able to produce this very high-level graph that summarizes the activities of a particular attack campaign and We are able to compute a signal from it that says how How confident we are that this actually represents a campaign and that's where the the severity levels of each of those TTPs actually matter This allows us to convert any given threat into a numeric score, which we can then compare against a certain threshold so So so so given any threat we we have a weighting function that allows us to compute an overall threat score and and then we use a standard Severity score mapping that is published by NIST, but it could be any others based on Empirical evidence again based on data from actual attacks so so we using You know the DARPA data sets we measure the optimal threshold value you know and Also, we deployed the system in a live detection setting where you know again in an engagement where we didn't know whether a red team is actually performing attacks and Here are some Statistics on the provenance graph size since our provenance graph is mostly from sleuth as you can see it's the the set of Number of Events so for the topic size that is dreamed in in in MB You can see how the the graph size grows and this is the trace duration in terms of percentages and These are the published records So here is the high-level scenario graph of Windows system so it sort of here These are all There is a threat score that is computed directly from the scenario graph based on the severity levels and the various labels here indicate you know the various objects that are there and the events that were performed are labeled along the edges and using the overall score that we compute and we compare that against a certain threshold and and Say that it's a so overall our thresholds were somewhere less than a thousand so and this is This is well about the threshold so we know that this is an attack But at any point the system keeps computing these high-level Scenario graphs over And and and keeps comparing them against at the threshold and when When the graphs when the campaign meets that threshold that's when we alert the user so So you I mean this is an interesting graph that shows the effect of The path factor and the learning we do some learning of the benign activity and this box plot tells us What is the effect of doing these two so for instance? This the green denotes the benign activity and the purple denotes the attack activity. So You know with without the learning and the path factor As you can see there is some overlap in between the benign and attack activity But once we throw those things in then there is a clear separation between the benign activity and the attack activity so That's so this also gives the highest benign score in the data set so you can see everything is the benign threshold The benign score for all the high-level scenario graphs is less than a thousand And for the various campaigns you can see that this is Really high so this is one This is one campaign where the score came fairly close to some of these but otherwise Everything is well above the threshold the threshold was somewhere like So so we did some experiments to actually identify the optimal threshold value, but I will spare the details So homes allows us to visualize the high-level APT in real time We mitigate the dependence explosion problem in the use of provenance graph by defining this notion of minimal ancestor cover and We showed how effective these techniques are in against actual attack campaigns I'm going to briefly talk about how we use provenance graphs for threat hunting in a system called pyro and this is work That is going to appear in two months in CCS so So often the question threat hunting is about Asking anytime there is a published threat you want to ask have I been a subject of this threat as my enterprise been a target of And you know the state of art is to look for indicators of compromise like hash values File process names IP addresses domain names do sort of some sort of you know manual or semi-automatic searches and It's sort of limited against the use of you know legitimate looking names It or updated or repurposed attacks It doesn't work and any kind of polymorphic malware this clearly doesn't work and our Thesis is that the relationships between artifacts and IOC's contain essential clues on the behavior of the attacks and And it's often tied to attackers goals, which are often difficult to change So what we want to do is to look for you know this behavior in the systems activity So when you think of these large provenance graphs that contain millions of or billions of nodes We want to look for a smaller Subgraph that represents a certain threat. So for instance here is a simple threat that talks about This process actually communicating with a certain external IP address doing things to a registry And interacting with the browser and you know also interacting with a certain executable Now this is sort of a threat behavior that is present in a certain malware We want to know whether this was actually manifested in the system and we want to do this in a matching efficiently and So we are Going to model this as a graph search problem. So The provenance graph we call GP It's a kind of laudit logs modeled as a graph. We've used that in sleuth and homes, right and The attack graph or attack behavior graph is modeled as a query graph, which is much smaller And both are labeled typed undirected and we want to do sort of a look for a match You know and this sort of match is not an exact match. It's not an exact sub graph match But a match on flows that is we do want to allow for some approximate level of match and So this is how we set up the problem So here are the indicators of compromise expressed They are expressed as a query graph and then the system behavior is captured from audit records as a provenance graph and we describe the problem of graph alignment where we look for an embedding of this graph inside the larger graph and Again, we compute an alignment score based on what we think are candidate alignments and then we can When when that crosses the threshold we raise an alert that particular threat has been found So I'm not going to talk in a lot of detail about This work just give some very high-level thoughts and then move on to the discussion part of the the talk so here is a Query graph example from for this thread. So This is the executable that is actually an executable that writes to a file C and then X will trades information to E which is an external IP address and D is you know some kind of a registry So the query graph can contain general terms like regular expressions like star so that we can match Even if there is a variant of a polymorphic malware that's performing these actions the exact names don't have to match and and it summarizes the actual piece of malware now if you look at this is the provenance graph and this is the query graph and We can identify that there are two possible candidate alignments here So one alignment is So the numbers on the edges are just for illustration You can see that this is one possible set of alignments firefox along with these and here is another possible set of alignments Here where firefox Interacts with you know seven the edges labeled Seven eight and nine so that's here So the question is which is the better alignment and it turns out that this is the better alignment It describes the threat better, but then how do we come up with these alignments, right? So again here the key inside is to define You know notion of an influence How can an attacker affect a certain dependency in the provenance graph and And and we do since the flows are not Exact flows that are described in the query graph. There could be multiple intermediate nodes in the provenance graph We want to make sure that this is sort of robust against, you know, attacker actions such as forking multiple intermediate processes so we define this notion of influence score and then Using this notion we define a metric called the goodness score that actually captures how good a particular alignment is and And this is the particular metric that we define for the goodness score that describes an embedding of the query graph inside the inside the provenance graph and We raise an alarm when the goodness score bypasses a certain threshold value So So we did several experiments one involving the transparent computing data and also with real-world data incidents with publicly available language descriptions and We set the threshold value to be three, which means that we expect the attacker to have what most three independent, you know compromises in the system and Here are some alignment scores for the, you know, TC data set so So, you know, most of these the we were able to identify the alignment in you know one round So we start by identifying some candidate alignments to the initial seed nodes and then from there we search using the goodness metric on what are some additional candidate alignments for most of these we were able to identify them in one iteration In some cases it took two iterations here are here is an evaluation on certain, you know malware reports and The thing is that we did the comparison of pyro with Commonly available cyber threat hunting tools like red line low key and Splunk The important thing to note is that those tools only use, you know, certain, you know key indicators of compromise So for instance file names and hash values is used by red line Low key uses, you know, again the same thing Splunk uses process names and pyro uses sort of a more of a give behavioral graph matching so, so we were able to take all these isolated IOC's and piece them into Query graph representation and look for a thread and come up with a way by which we can do this, you know automatically and And and conclude that the threat happened at the very high level of confidence instead of relying on sort of human intuition and manual verification So that Summarizes our work on pyro So I am just going to talk about some general thoughts on what I Think are interesting. So we have worked with very large graph. Some of these graphs are, you know, billions of you know nodes and edges How can we How can the analytics take advantage of, you know, advances in learning and one very simple thing that we can do which is like really low-hanging fruit is to Automate construction of these query graphs that I talked about. So for instance Here is a natural language description that talks about email clients downloading office monkeys And video dot zip, office monkeys Self extracts to office monkeys dot exe Monkeys drops monkeys dot exe and player dot exe under a temp So as you can see this sort of natural language description of a certain thread Lends itself to a query graph representation But you know, perhaps one needs to build A good, you know NLP based technique in order to automate this process if this is there then every cyber threat intelligence report and Blog could then be, you know converted into a mechanizable representation that we can search very fast It's fairly low-hanging fruit. There is much work there But this is something that if it exists then Enterprises would be able to search for threads in real time with the help of, you know The main memory provenance graph, the query procedure of pyro and the Mechanization of converting natural language descriptions to query graphs Now we described how we can use a tag-based reasoning or information flow based reasoning and also misused rules like TDPs in Thread detection But it's likely that, you know more sophisticated stealthy attacks that take advantages of well-known limitations of these approaches In order to stay under the hood, right? So anomaly detection, you know, which has been studied well offers a careful offers an interesting midpoint in this space, right? How can we How can we integrate Anomaly detection, you know along with the reasoning offered by these frameworks, you know That involve information flow Analysis and you know TDPs, right? So what is the role for, you know, learning based, you know approaches in particular? How do we combine, you know, tag-based, you know detection with anomaly detection so that we can identify Novel threats that try to stay under the radar of, you know, information flow dependencies, right? So that's an open question right and Often real world deployment of systems, you know, often we get training. So we are talking about billions of Event records of any sort of data that we need for training, right? We cannot guarantee that at any point that these Training data are free of, you know, malicious attack, attacks are omnipresent everywhere, right? So learning based techniques will Acquire these attacks and, you know, conclude them as part of normal behavior, right? And it may be possible. So today I talked about You know poisoning, right? It may be possible for an attacker to slowly shift And you know in in ways that we cannot detect slowly shift the threshold of you know these Detection techniques. Now the question is how to select features and learning methods so that they are They are robust to these adversarial actions. I think The use of tags and the use of, you know, reasoning about information flow might offer some interesting ways of thinking about adversarial robustness in these situations With that I so I would like to thank DARPA, you know for funding this research under the GCE and CHS programs NSF and collaborators from Stony Brook Mohammad Huzain, Junao Wang, Scott Stoller and Sehkar and also my students and postdocs Saadeh Milajerdi and Burano S. Hathay and Rigal Chumimo. Thank you and any questions. I'm happy to take any questions. Any questions for the speakers? Just a couple of connections to make Really where it brings together this consolidation of external threat intelligence Internal threat analytics And connecting it to the external threat intelligence And there are of course challenges. One is of course the volume of velocity But once you can get the sub graph of what that is you're coming up, it's a city that captures the There's a very interesting question Who makes certain choices I think the main chapter to do With practitioners that today are sitting in front of a science team So what's the procedure? There's also a paper in black hat Where they're also trying to go down this path of threat intelligence Actually my student Saadeh he interned at Microsoft with the same group Any more questions to the speaker? If you say your trusted application has vulnerability that somebody exploits Would that show up as an attack? It depends whether the information again if you read from an untrusted So typically a vulnerability application becomes vulnerable after reading some untrusted data So there are tags that are associated with the untrusted data as it comes from let's say an external network Then when Firefox reads it and then it performs an action say it Sponsor shell which goes on to do other things So the tag propagation will capture some of this But if the action happens in such a way where all the operations happen in main memory Then the kind of signal that we will get is weak Because the audit records don't tell you exactly what happens at the level of individual memory operations They only tell you at the level of system calls So Firefox may be tagged as untrustworthy data process But that signal again is fraught with some limitations So it really depends on the particulars of the threat But the tag based reasoning does allow a way for you to reason about that particular situation And this is where the distinction between code and data trustworthiness I guess everyone is waiting for the lunchtime Thank you for the great talk and this is thank you for the great work in the threat hunting and the threat detection Thank you very much