 Hello, Mike Test. Thank you. Let's get started. Apologies for delay. Welcome to the session and I'm Binu Ramakrishnan from NVIDIA and I'm going to talk about Building Trust at the Edge, Lessons Learned. The work I'm presenting is broad in nature and there are many people who contributed to this effort so I would like to take the opportunity to acknowledge their contribution and grateful to be here because of the teamwork. Here is the agenda and outline what we are going to cover in today's session. We'll start with a brief introduction to Edge Computing and we'll get into the third modeling, attack models and motivations and we introduce the building plots which is used to orchestrate security at the Edge and we end the session calling out some of the challenges which we encountered as part of the journey. How many of you are security practitioners here? Okay, all right. Edge Computing, what is Edge Computing? Edge Computing brings compute and storage closer to the sources of data and it enables applications that require ultra low latency, high scalability and high throughput. In our case, our application is provided AI inference at the Edge so it enables applications like AI inference to process data in real time, conserving network bandwidth and saving the response time. So if you look into the overall architecture, the one we are describing from Edge Computing perspective, we have a cloud layer which is used to manage or a fleet of Edge devices which are deployed at the Edge and the primary reason that Edge nodes are deployed closer to sensors is it will allow the Edge nodes to do real-time processing. Examples of Edge Computing includes retail, shopping malls, airport, hospitals. The Edge Computing combined with AI expected to transform industries from retail to healthcare, manufacturing to smart cities and many of the Edge systems are mission critical and industry largely underestimates the critical need to provide highest level of security for the applications running in this Edge. Before we get dive deep, let me try to categorize the Edge Computing models to broadly into two. One is the Edge Appliance and the other one is Edge Platform and what we are covering today is Edge Platform and Edge Appliance provides end-to-end experience. It is mostly purpose-built and it is capable of doing predefined set of functions and the data collected from the sensors are processed by typically trusted applications. Whereas in the case of Edge Platform, it is a generic platform that allow or support wide range of applications such as manufacturing, retail and healthcare. It is relatively powerful machines, for example, can run applications such as AI in France. The fundamental difference here is the data is processed by the customer deployed applications rather than the trusted application directly from the provider itself. You can think the appliance more like a vertical built-up versus Edge Platform is more like a horizontal built-up. Here is a reference Edge node stack. We have hardware, firmware and on top of that we have Linux Kubernetes and all the applications which are running as a port in this Kubernetes cluster. I understand that each Edge stack differ. You may not be using Kubernetes. You might be using free BSD instead of Linux. In our case, we support both x86 and ARM. However, this talk is more focused on x86. We are still working on the ARM side of things. But overall, many of the things what I'm going to discuss will be applicable to any Edge systems. Here is a variation here. Here you have a system that is BYOD. It's basically bringing your own device. You buy the device, you hook it into a system and use it. As you can see here, it creates unique security challenges. For instance, who is responsible for keeping the firmware up to date? In the case of x86, it's the UEFI lockdown and so on and so forth. So it creates some challenges from a provider perspective. I'll get into those soon. Here are some assumptions and scope. The assumption here is we have a cloud managed Edge platform service. Essentially, a cloud-wide service that manage fleets of Edge nodes which are deployed in the customer location on the remote sites. The scope here is the Edge node deployed in the customer premises. We are approaching this from a platform or service provider perspective. What is not in scope is we are not going to talk about cloud security. That could be something we can. That will be a topic for some other time. Likewise, we are not really talking about the security of the IoT sensors. Edge security, what level of security is desired? Where do we draw the fine line? It largely depends upon your business model and your risk appetite. Like I mentioned, you can get into an appliance mode where it's probably relatively easy to secure, but you will provide a fixed set of functions, whereas in the case of a platform, you are essentially providing a general platform which have more security consequences. The other factor include whether it's a purpose-built hardware or you are entertaining, bring your own kind of device, so all those things can impact the security threat landscape. To secure Edge system, we first need to understand the threat landscape. Who are the threat actors and what are their motivation? How do they attack and what is the impact? The threat actors include nation-state actors. Those are the group either funded or supported by government agencies around the world. Organized group are, they provide, in short, cyber crime as a service or command and control as a service. The other group are very minor but still discontent of employees and activists. Activists, essentially they champion for certain costs and indulge in these kind of activities sometimes. Their motivations are steel data and intellectual property. For nation-state, stealing is primarily to achieve some security, strategic objectives versus organized group primarily do that to make some money. ransomware, they lock you out and demand ransom. Steel computer resources, once the machine is compromised, let's say, sometimes these computers are powerful enough and they will make it part of their command and control. They often use for crypto mining and things like that. Denial of service, that is another aspect here. For example, activists can really disrupt it. They don't really gain out of it but it can really create chaos in the system. Attack models, how do they attack? There are physical access because this is the edge system and since it is connected, over the network attacks are also possible. Vulnerabilities. Vulnerabilities are everywhere. Any non-trivial piece of software have defects and some of them have security consequences. So, attackers always look for opportunity to exploit these vulnerabilities in the system. Hardware and software supply chain, the attackers trying to do some backdoor either in hardware or software and trying to gain access to these edge systems. What is the impact? Depending upon the industry, for example, if it is a healthcare, there could be risk to life and disclosure PII and other sensitive or confidential data. Financial loss, litigation, losing market to a competitor, impacting a brand and it can disrupt services causing chaos in your system. Imagine Friday afternoon in LACT, all the traffic lights go off at the same time and you can imagine what can happen. Things like that. Here are some additional considerations here. One is the physical accessibility. This will lead to unauthorized physical access devices. The other scenarios include it is susceptible to side-channel attacks such as timing attacks, power consumption analysis, cold boot attacks, things like that. And since they have the physical proximity, it can also mount firmware. It can basically modify firmware and exploit any firmware whenever it is. The other factor is heterogeneous hardware. It is mostly the bring-your-own-device kind of scenario where there are so many vendors here and that significantly increases the attacks of us. Likewise, there is no real standardization in the pre-boot environment, especially in the firmware world. Different vendors support different kind of tools that is also something to consider. The third one again is related to the bring-your-own-device shared responsibility of the edge nodes. As a provider, we need to trust the customer and the customer, you are expecting customers to lock their UFI, lock their firmware, pass the system in time. That is another thing to consider. To build security, we need to identify the security building blocks. A security building block is a unit of technology, process, or a combination of both. In this side, I am introducing the building blocks, which is used to orchestrate security. I just call it out. One is the hardware root of trust. The trusted computing base, firmware security is very important here. Identity, authentication, authorization, secure edge, enrollment, encryption and data protection, remote attestation, runtime and boot time integrity, threat detection and failure reporting, supply chain security, vulnerability management and over-the-air update and defense in depth. We will cover this topic in next few minutes. Root of trust and trusted computing base. The diagram in the right shows how the chain of trust is established from boot to the runtime. And there are two complementary technologies here. One is the UFI secure boot and the other one is the TPM or trusted platform module and the measured boot. Here you can see the secure boot use the UFI as the root of trust and measured would use the TPM as the root of trust. What is secure boot? Secure boot ensure that the node boot only using the software trusted by the provider. And if any of the software that is part of the boot chain is compromised then the signature match will fail and the system will not boot. So it is more of a hard fail. Whereas in the case of measured boot, we have in the TPM we have something called platform configuration registers which basically is used to store a cryptography measurement or a hash of each of the software layer in the in the boot chain. Our integrity measurement architecture extend the measured boot to enforce file runtime file integrity. The TCB it's not from a strict sense but mainly from a broad sense. This is a set of this is a stack we need to defend against. We need to make sure that the integrity of the stack is protected. TPM is a separate topic but this is a brief 30 second intro to TPM in case if you are not familiar with the TPM. A TPM is a co-processor on a motherboard which is capable of doing or generating secure random numbers. It can generate keys and hashes and you can also use to store cryptographic keys. And it can also do cryptographic operations such as RSA and AES operations. And one important thing at least more relevant for us is it provides platform configuration registers essentially a set of registers around 25 or so 24 I think that is used basically to store some hashes which measured boot use. It is used for performing attestation. And one thing to note here is this device is passive and it is a very it is a pretty slow device actually. Secure boot and measured boot I already covered so I would skip here but the main thing here is in case of secure boot is more like a hard failure if the signature of any of the component in the boot chain fails then system will fail to boot whereas in the case of measured boot it will compute the hash and update or append the hash value in the TPM PCR registers and it is passive in nature but it requires an external service to attest these measurements. Edge node enrollment. You have a device now how do you securely introduce this device to the service provider. Imagine you as a customer you bought a device. What is the process there? There are multiple ways to do that. I'm just describing one way and there are a lot of variations possible in simple terms the way it works is you have an account with the service provider for example the same way you have an account with AWS GCP. So you have that account now you when you're ready to register a device what you have to do is you go to the the web console whatever the provider provide and say I want to register it so one method is they provide some kind of a token one time short short leave kind of token say yeah here's the thing you just go you take our software and deploy it in your edge node and you also pass this one-time token and internally it install the software as part of the initialize the TPM generate the TPM keys and it register the TPM key with the edge through bypassing both OTP and the keys associated with the TPM. In the end what happened is the edge system knows that okay oh this is the edge node I know that and this is how I can authenticate with that edge system and the TPM protects cryptography keys and operation and prevent attackers stealing keys and and key cloning essentially. When we talk about edge node it is not just one application there will be multiple applications running in edge node likewise if you talk about the cloud there are multiple applications running there as well so how do they communicate with each other and we need to have a strong authentication in place to communicate with these systems and how do we do that? Let's talk about node identity so you'll be one you'll be thinking yeah we have a TPM based identity so we can use that to authenticate with cloud services. Yes and no it works in some case but it is cannot be used there are some issues here it is easy for the edge agent to connect to the edge node manager because they already have the TPM's public key but that is not the case with many other services in the cloud where the edge wanted to communicate that means you cannot really communicate beyond or to any other service other than the service that has the TPM's public key. Imagine you are a provider which is managing hundreds of nodes or thousands of nodes managing these keys is also a little difficult. I mean we need to spread across or share these keys with many services so to remedy the issue we introduced a cloud authentication service or token service what it does is it actually do the authentication with the TPM and it issues a OIDC compliant JWT and any service and that is a short load of ephemeral and any service that is compliant with OIDC it can make it work. Again you don't have to use JWT you can also use ephemeral certificates and things like that. Here's one example here you can see the edge agent request TPM to sign you can see that edge agent is making a request to the TPM to sign sign the node token and it make a request to the OIDC service token service and the token service authenticate and oh this is coming from I know this edge and I know it is a legitimate request and it will issue a short-lived OIDC token and all the subsequent connection from say edge agent to edge management service they can use this OIDC and through OIDC discovery this service can authenticate the token and allow this edge node to access the services alternatively step one and two can be replaced with mutual TLS with the TPM backed key using pgcs 11 interface. Let's talk about remote attestation it is a complicated concept but let me try to explain the array is a mechanism to you array is a mechanism used by the service provider to verify that a system an edge system connecting to itself is is booted with trusted software. Let me discuss about one scenario here here the edge imagine the system the edge system booted up and the first thing they do is you try to connect to the service providers the cloud service essentially and the cloud service say yeah I know you are edge Bob I know you through the TPM key but I do not know whether you are booted with the trusted software let's do this I give a nouns you just you just go and get a PCR code so the edge system take the nouns and contact the local TPM and say edge generate a PCR code for me and what is a PCR code PCR code is nothing but a PCR measurement signed by the TPM attestation key and and and the PCR code and an event log sent back to the edge manager platform configuration registers yeah it's a TPM construct it's basically provide set of registers where you can only extend the value you cannot really modify it I I deliberately not going into those details because those all are kind of another topic so but I hope you understand the larger context so where did I stop so so the PCR the value upon receiving the PCR value we just compare with a reference value and see whether that matches if there is a mismatch you saying something okay if there's a mismatch then what happened is it trying to pass the event log and see where the mismatch occurred this diagram is not really good but please bear with me so how do we get the reference state that's important thing you can you can get it by different means one way here is remember I showed you the note enrollment process where the system is actually booted first time so as part of the process it's basically get the measurement at that stage and consider that as a reference quote it is more like a trust on first use kind of model we following but but there are other variations possible if you know your hardware well then you can also know what will be the value at that stage as a as a runtime extension to attest so we can we can actually extend it through IMA it'll support the runtime integrity check IMA what it provides is the file integrity check like you can basically hash the file and any of the files modifies and then we can enforce or at least we can detect that okay this file has changed in runtime can you be loud we use tls the the pcr values are signed by tpm at private key so we cannot manipulate that okay let's let's we have a q and a so we can discuss and if I can also sync up with you offline yeah so data security data protection is critical and whatever we data we are storing in the edge it has to be encrypted this will protect when the drive is stolen repurposed or reused data in transit we need to make sure that whatever data transmitted over network is protected tls is a recommended protofolder tls is very critical not just for securing the data but it is it is critical to protect against man the middle attacks it's very important secure deletion of data if the drive or nvme is retired or repurposed then we need to make sure that the data is cleaned up network punching holes into customer firewall to allow inbound connections to edge services is easier said than done the the problem here is the customer network they have their own it policies and and some organizations are pretty big and they have separate organization to manage those things and if you want to open let's say some firewall acl it may take ages to do that or it may not even happen at all the recommendation here in this kind of scenario is since generally firewall generally allow outbound connections particularly like ports like 443 so always initiate connection from the edge device that way you can really reduce the dependencies and it's also good practice not to run any services or listeners listeners specifically network-based listeners on the edge nodes just reduce the attacks of the area runtime integrity and threat detection for edge platform threat detection is an important function and such systems are required to detect and alert anomalies that occur in the system during runtime here are a few rule set examples reversal we can have a rule set basically to identify a reversal and reverse dose reverse dose is someone getting to the edge node and trying to mount a denial of service in the local network the another thing is you have a part which is a imagine it's a web server it's only if you see you can have a rule set to detect any kind of bash exit so that's that's i'm just giving some examples if you know the certain patterns you can update your rule set to detect certain CVEs for example like lock 4j lock shell kind of vulnerabilities again detect brute force SSH crypto mining we can we can track the outbound request and see where it is going and depending upon the IPs we can determine if it is a crypto mining happening or not then we can also track based on stratum protocols usage which crypto miners generally use we can also tag certain files and see if anyone touched that and we can get notification that will help to prevent data exfiltration supply chain software developed in recent times heavily rely on open source components besides the security of the software we built open source security of the open source also have direct impact on on the edge the fact is that any non-trivial software contain vulnerabilities and some of them has security consequences we have both hardware and software security risk when you talk about hardware we have what is that uh introduction and seeding introduction is essentially you're modifying uh in transit from manufacturing to the end user or the consumer whereas seeding is you're putting some backdoor as part of the manufacturing process itself and some the software side of things there are many ways to compromise third party packages or third party or open source libraries like compromising GitHub accounts updating making changes things like that so what are things we can do here you have to source these packages uh get to be careful and if possible yeah if there is an option not to use then don't use it every is now open source is not free it bring a lot of liabilities to your to your stack and you scan as a perspective you scan for vulnerabilities looking for CVEs and malware etc secure distribution of whatever artifacts we build that's important uh over TLS recommended published checksums um there are some new advancements here on image signing uh six store i don't know you're familiar with that but they offer uh tools uh for signing and verifying software uh they it is good uh for container signing at this moment but uh but there are a lot of other artifacts as well you need to consider build supply chain provenance or some kind of attestation of the build process uh adjusting the s-bomb and things like that uh in toto and salsa those are the two options in this space uh again those things are in these early early stages uh vulnerability management and over the air updates uh vulnerability management is a continuous function to identify and patch uh vulnerable packages in a timely manner and OTA or over the air update is a vehicle to uh to update or patch these edge systems and OTA happens uh in this uh following condition not necessarily a full list of things but in general like uh remote attestation failures whenever we do remote attestation remember like there is a pcr quote comparison if any such failure happens the uh it you get alerted and they may determine a there is some bad happen we need to remaze the system sometime we require uh physical presence of sometimes you can do over the air uh likewise we identify some vulnerability in some of the packages as part of our stack then we will do a OTA uh regular upgrades generally go through the OTA and other uh recensing routes some kind of security attacks or something we identified as part of our threat detection system sometime we use OTA uh to uh update note that the OTA may change the uh some of the pcr measurements so uh we need to make sure that we are updating the reference state accordingly in a uh in a reliable manner it's a little challenging but it's possible so all the things which I discussed the the building blocks those are the big big things but that is not the end but there are a lot of small but important steps we need to do to make the system secure uh I'm listing out a few but there are many more um from authentication side we generally use uh recommend uh use of ephemeral tokens and certificates um we should have a our back for customers uh remote console for device administrators if you want to manage the edge device from the cloud isolation uh we have a host and application level isolation um we can enforce uh that containers can run in an unprolaged mode um and between applications we separate using Kubernetes namespaces operating system we we have to harden the operating system it's very important we use a syscat as a reference uh in addition to that we also do other things uh for example we strip down the OIS to make it as small as possible uh that way it'll reduce attack surface and also reduce the the need to pass the system because some of the unused packages may have vulnerabilities and we end up patching the system we can avoid that uh read only file system for system partitions uh and encrypted uh data partitions uh some examples here and hardening of the Kubernetes cluster uh enabling our our back policies uh and provide admission controllers to enforce certain policies the policies include uh you cannot you can run only as a particular unprolaged user you can only open certain ports only certain inbound outbound are allowed uh you can also apply up on our policies sitcom things like that um uh you can also use uh the the uh the six store which I shared previously uh it also provides some uh can also use admission controller to basically uh verify the docker uh container images signed container images um we have dedicated namespace for customer workloads so things you can follow uh so if you are listening this probably be the most uh uh interesting uh slide I would say uh this is uh basically uh show uh all the things uh which we discussed and uh you just show how all those things are mapped and connected uh if I describe it we have we just convert it into three phases here one is post deployment uh what all steps we uh we have to do and then another is during the deployment phase then the third one is the pre-deployment a post deployment where some of the runtime controls kicks in uh in the in the pre-deployment you're talking about third party software and what all things we do there we talk about uh applications native applications which we build what all things we do there talk about Kubernetes what all things we do there and operating system very important component what all things we do there and there are many things uh some nuances uh I excluded that is for clarity but uh so all these software are built and put into some artifact factory and we can potentially provide uh some integrity like uh you can you can publish uh check some hash uh you can sign those artifacts those are some other things you can do once uh the system uh and and and you can see here the user basically pulled uh the software from the from the providers artifact repository and uh you basically install that in the edge and at that stage you will do a pre-boot protection like ufa lockdown and secure node enrollment those are some of the activities part of the deployment and when you talk about uh the the runtime we have the runtime protection we have runtime detection runtime detection include uh tpm attestation uh falco ebpf logs uh then secure boot and runtime prevention include immutable file systems uh k uh Kubernetes namespace isolation admission controllers uh and the trust we depend upon uh the tpm and uh and the ufi uh tpm backed identity uh tls ca bundle uh yep uh so i hope this will be helpful for you to visualize all the things which we discussed challenges uh building trust on disparate uh disparate pieces of hardware and software provided by variety of vendors is a uphill task uh here are some of my observation on issues uh came across uh spot of our journey uh secure boot uh it is cool but remember it is hard failure if something bad uh happened you just stop uh booting up uh you that is that may not be the thing you want because you have a cloud service which is monitoring or supporting ed device and if the ed device is not even coming up you you don't even get an opportunity to see what went wrong from that perspective it will be better to have measured boot than secure boot both are better but secure boot other other problems include uh we have you have custom kernel modules or unsigned you have to use some unsigned third party modules so how do you manage that uh you need to update uh the ufi database that require physical presence uh so those things uh can create more trouble uh and uh stability of the tpm pcr bank uh we come across uh the values are not consistent between uh hardware providers like sometimes we're just changing uh even the for example the bios pcr bank zero uh not supposed to change but sometimes we just reboot just change uh those things like that uh and lack of standardization in firmware uh reboot environment is another thing we have to deal with there are some uh redfish api all those things are coming up but uh i think it has to evolve i think uh supply chain attestation is it in early stages and i believe it's not ready for a wide adoption yet but that is something to watch out uh artifact signing package check some work it's easy simple um container signing is doable now uh but signing and verification of other artifacts uh currently it's not trivial uh so with that i is a teamwork and uh that's it yeah it's arm based it is arm based no no no the arm uh we are currently working on uh um a solution to work on jet sun we are we are considering a uh firmware tpm uh but still early you know maybe let's talk offline yeah yeah uh yeah sure i need to talk about uh yeah so if you are using secure boot you have to rely on ufi to trust but if you are not using secure boot you don't have to rely so secure boot it is a hard fail uh hard failure right you you have a boot chain now you are starting your application and imagine one of the component in your boot chain is compromised or uh something happened and the signature uh there's a signature mismatch at that time the system won't boot yeah yeah yeah let me conclude this uh yeah yeah i'm just yeah yeah yeah yeah yeah uh i i didn't mean to make it that way but it ended up that way it ended up that way yeah when i started chatting out but i still if you pay attention it's uh it's not very complicated uh yeah when you when you see first time maybe you get overwhelmed but if you just look through that probably it'll be not very hard uh yeah which which one this one uh you're talking about this this one or something else uh what i'm uh no what what we are presenting here is uh some other experience uh on building an edge system an edge platform and some of the patterns we can use across oh you're talking about the use case okay i think i described as part of the edge computing introduction uh one use case here is imagine uh one simple use case i can tell you like the air inference uh you have an edge system that can process live traffic feeds looking for traffic violations that's one example and you can use this edge system in the factory floor as a robot and it can do some AI inferencing the robot uh that's another use case and there are so many use case related to edge AI inferencing these are like two just uh two examples uh the customer at least in our case get a uh an approved kind of device from uh oems which uh which provide is approved device and they bring it in and they plug it in yeah uh yeah this is more of a generic platform that means we can fit into different kind of application different industries and uh we have verticals built on top of that for various verticals so i just only covered the the platform uh yeah we are on time uh let me uh any more questions before i can wrap uh okay uh yeah go ahead oh i didn't i i deliberately excluded that uh because that is another another uh bmc i i didn't really add it here but yeah if bmc is another thing we need to really uh protect that what we uh that's where the network side of things become very important because bmc security is primarily the network security if network is not secure then you are kind of compromised um all right i'll be around today and tomorrow so if you have any questions if you want to have a conversation feel free um thank you and have a good rest of the day cool so i'm gonna go ahead and get started in just a second here actually i think i'm gonna use the restroom since i had a ton of caffeine as i finished this presentation on the car right here so since i don't want to piss myself in front of you guys i'll be back in just a second all right so i'm gonna go ahead and get started uh my name's alex and today i'm gonna be telling you guys the story of how i accidentally made a hardware business at 18 so actually just last week i was at the hope conference in new york which is hackers on planet earth and i gave basically the same presentation with just about the same amount of preparation as well which was starting the presentation a day was supposed to give the talk a day before i was supposed to give the talk working on it all the way up until the last five minutes as things happened for me so my name is alex lind i'm an open source hardware developer and also a cyber security content creator so i'm usually um making education focused content on the hack five youtube show if you guys aren't familiar with hack five there are cyber security vendor that creates products for pen testing red teaming and that sort of thing i mostly create beginner focused content where i create walkthroughs tutorials that kind of thing but more recently i've been focusing on hardware and physical attack exploits and i'm going to be showing you guys one of the tools that i developed today that i've been showing off in some of my more recent demonstrations so when i'm not creating content um well i also create more maker oriented content on my personal blog where i have walkthroughs for different hardware stuff that i make i create various projects like the one i'm going to show you guys today but my primary interest is in low-cost microcontrollers in ways that you can use them um for low-cost hacking like wi-fi hacking signals intelligence and that sort of thing so just an overview of what this talk is going to be about i'm going to be telling you guys my back story sorry i'm a little out of breath so i'm going to be giving you guys my back story i'm telling you how i got to the point where i am today i'm going to be telling you a little bit about the business that i started completely by accident i'm going to talk to you guys about the ideation process of the product that we're now selling which is called the nugget it's a cat-shaped circuit board and i'm going to tell you guys about some of the mistakes and key things that i learned in the development process of this product so hopefully if any of you guys are product developers or makers um you won't make the same mistakes that i made or that we made um along this process so what is the nugget the nugget is the project that we've been working on over the course of the past year it's a cat-shaped circuit board that's designed to teach beginners about cybersecurity and hacking so we currently have two splits of this product one of them is focused on physical attack vectors like usb hacking keystroke injection and that kind of thing and it has a little gamified interface that walks you through the process of running some of these attacks so it's really great and beginner friendly and we also have another version of the nugget designed for wi-fi hacking so this one's more oriented towards things like network defense running wi-fi attacks and it's a great little walkthrough for beginners who are just getting started in these areas in addition to that it also comes in a pretty easy to put together kit that you can solder together yourself so this has been a centerpiece of some different soldering slash hacking workshops that i've been doing over the course of the past year and it also makes for a great beginner programming tool and it supports some languages like python and arduino and stuff like that so i'm basically just going to be telling you guys about how this product was developed and also some areas where we screwed up so first i'm going to start with a little background about myself so i first started getting into hacking and programming i would say sometime around five years ago in middle school when we were issued ti-84 calculators in math class and if there was anything i wanted to be doing in math class it sure as hell wasn't math so i taught myself how to program on this calculator actually with a built-in interpreted language called basic so i taught myself how to create like bitmaps and like games and stuff like that and i started selling it to other kids at school but once i got kind of bored of that um i looked into some more hardware stuff like how i could actually hack the calculator like overclock it and add wi-fi so i could create like a mini ioc channel to communicate with some of my friends in class and some crazy stuff like that but eventually when i got bored of that um i would say sometime around the summer between middle school and high school um i learned about the raspberry pi which was a low-cost micro controller or micro computer for $35 and that really piqued my interest because growing up i was kind of poor i didn't have a computer at this point but seeing that there was something so accessible and so cheap that i could learn hacking on um this is really interesting to me so i picked up a raspberry pi i started learning about linux and sure enough i discovered that there were all sorts of interesting things there were all these different distributions that you could use for specific applications like kali linux for hacking and all this cool stuff still catching my breath so i eventually taught myself how um to do stuff on the raspberry pi and one of the very first things that i taught myself how to do was actually hack wi-fi so as i said i grew up kind of poor we didn't have any internet growing up so when i had to do schoolwork i would basically have to skate over to starbucks use their wi-fi and then go back home um and then just sit in silence so i eventually taught myself how to hack wi-fi so i could have that um from home and be able to do my schoolwork and that kind of thing um and i eventually started teaching myself um how to do some other stuff like programming and that kind of thing but i found it kind of frustrating that there was just a lack of general documentation on the internet for some of the stuff that i was trying to teach myself so eventually i created a youtube channel where i started documenting various tools that i was learning i started documenting walkthroughs of how to set up various linux tools that kind of thing for other beginners that were out there and here you can also see one of the first um raspberry pi based projects that i worked on called the audit pi but um getting my start in wi-fi hacking really peaked my interest just about that general subject um with signals intelligence and that kind of thing so i started trying to teach myself a little bit more about that and that's where we get to the very first signals intelligence project that i worked on so the first um yep okay so the first project um the first independent project that i worked on that um actually had some contribution to the project that i'm going to be showing off today was called the creep detector so this actually used the little handheld device that i made here called the audit pi which was focused around a raspberry pi in order to basically what i wanted to do with this was showcase a way that an everyday person might find signals intelligence or wi-fi hacking or that kind of thing useful so the project that i ended up coming up with was what if i could create a device that allows you to determine if maybe a creep or a pervert is stalking you so that we did this was we used a signals intelligence technique called war driving which is basically where you can drive or walk around and map the physical location of wi-fi devices around you we used that technique to determine if basically the presence of a wi-fi device like a cell phone or laptop or something like that was spotted at multiple unique coordinates along the route that you are traveling so this is one of the very first projects i came up with um on the raspberry pi it was a combination of um some different linux tools and also also python and the demonstration ended up working so that was really cool and rewarding and this is the project that i worked on with my friend kody who i'm now running um this business with today so while i found um working with the raspberry pi very interesting i wanted to get into things that were a little bit easier to get started with and also more low cost so this is where my interest for microcontrollers developed microcontrollers were really cool um they were really cheap these are things like um arduino nanos the expressive line of chips um at tiny's at megas that kind of thing i thought they were really cool and i saw some research out there on how these could be used for more um hacking oriented stuff besides just like developing so i thought this was really cool and i wanted to find ways to integrate this into the signals intelligence research that i was doing i didn't really know where to start there wasn't a lot of great beginner documentation on using um some of these microcontroller platforms for hacking so i didn't really get into that until um i would say somewhere around like my sophomore or junior year i got this crazy ass commission from this guy who wanted me to create a device for him for tracking down a cell phone that he lost in the forest so i had no experience with creating anything of that sort but i was poor i wanted money this sounded cool so i said great i'll take three hundred dollars and uh give me maybe a month or two and i'll have something for you so i started looking into how it would be feasible to do this and i built off my previous research with the creep detector project um to also use signals intelligence in order to track down his lost cell phone so the way that this worked is i built this platform around a microcontroller that we'll look at in a second called the esp8266 it's about you can find them for like one dollar eighty on ollie express maybe five dollars on amazon but basically there is a library that allows you to do promiscuous wi-fi sniffing and you can look for a particular type of wi-fi packet called a probe request so even if you turn off your phone not if it's completely shut down but if it's just like turned off and left somewhere your phone or other wi-fi device is still going to be emitting a type of packet called a probe request where it's basically looking for networks that it's seen before and constantly emitting this out into the air so i thought hmm what if i could program this microcontroller to look for that specific packet have it hone in on the signal strength of this device and then use an antenna to be able to track down the physical location well it turns out that crazy ass idea worked he was able to not only find his phone but he was able to find multiple cell phones that people lost in a local river so he lived near like a resort or something like that and he would go rafting quite often but it turns out um lots of people brought their phones there and like waterproof bags and they ended up like dropping them um also at this point i was still poor i didn't have a cell phone so i was like hey do you mind sending me one maybe so that's how i got my very first phone um from this random guy on the internet that i made this crazy device for esp8266 i have a picture of it in an upcoming slide if you want to take a look but yeah all this is relevant because this is how i got my background in signals intelligence and also microcontrollers but i was really interested more in the low cost aspect of being able to do these kinds of attacks so anyways i was doing all of this um sometimes towards the end of high school i was getting like occasional commissions and that kind of thing but i found myself at a point where um in my senior year i discovered that i was one of my only friends who didn't apply to college so at the very last minute i was like scrambling to put together these applications and then i realized i don't really want to do any of this i don't understand why i'm applying to college this isn't really what i'm looking to do right now i'm more focused on like um it just didn't really speak to me but also i didn't really have an option to you know like stay home or something like that because my home life sucked so i tried to start a personal brand called lind labs where which is my last name where i tried to advertise basically the services that i would commission which was creating custom hardware software web apps and stuff like that that wasn't enough to sustain me or be able to move out from my crappy home situation um but as it happened the friend that i worked on that previous project with the creep detector he was creating cyber security videos for a company called hack five which is where i now create videos for and he also ran a live stream on a different channel called security forward but he basically said hey we basically are in a struggle right now to produce content so if you want to come out to montana for just two weeks that's where they were they moved out there during the pandemic just because it was like kind of hectic here he said hey if you want to come out to montana for two weeks create a bunch of content for us maybe you can stack up a couple thousand dollars move back and figure out what the hell to do with your life so i was like oh great that sounds cool i'll make a little bit of money but i still don't know what i'm doing in terms of college or if i'm going to move out or what i think i lied to my parents and said that i got into uc santa cruise or something and that after i was going to come back that i would go there spoiler alert um those two weeks ended up becoming a whole year that i ended up staying there and then also um just another side thing my friend was previously working on a little project with a friend that didn't end up working out so well so he had a ton of extra um just like random hardware components on hand so he was like hey uh while you're here if you also just want to help me come up with a design so i can get rid of all this extra hardware and maybe sell it on the internet or something that'd be cool so we ended up starting out as a fun little project also became the high intensity startup that we're now running today and dedicating most of our waking active hours towards so it's quite grueling so this was the initial thing that we came up with so my friend had um he had a bunch of leftover screens he had some buttons and then he also had a wi-fi microcontroller that we're going to look at um in upcoming slides so what we ended up coming up with was this design which you can see is sort of akin to the probe hunter that i showed in a previous slide so i said hey why not just slap this all on a board and then we'll sell it as maybe a soldering kit we can teach classes or something like that put it on the internet i don't know it was just kind of a random idea we just wanted to get rid of this hardware we didn't really have a focus for this it wasn't really a product that we wanted to bring to market so just as a joke we decided to slap some random graphics on there we decided to call it the hack cat um we put a little cat boy on the back we have a little cat emblem so that's where this all started out so now this brings me to my next point which is the first lesson that we learned pretty hard and this is why all in one hardware platform suck or more specifically why hardware multi tools suck so we didn't know this going into developing this product because we weren't even looking to design something that we wanted to bring to market we just wanted to basically dump all of our components into this design sell it all on the internet and then be done so the first reason um in a second i'll get more into the actual storyline of how this product was developed also let me check the time so the first reason why i think this sucks and this is also a pretty common mistake i think that a lot of makers and product developers end up making is initially it seems like a great idea to create like an all in one multi tool to try to appeal to basically everybody this is a tool that does everything who wouldn't want this i think this sucks for a variety of reasons but the first one is you're the only one who actually truly understands how to use this product um like as you can see here i think i found this design on tindi it's supposed to be like a wi-fi multi tool with like a screen and temperature sensor and like a bunch of other random crap but who knows what the hell this is it's kind of crazy so this brings me to the first case study which is my own personal project the probe hunter so let's analyze why this kind of sucked so initially it started out as a single purpose design for tracking down wi-fi signals that was really cool but once people on the internet after i started publishing this design um started gaining some interest in this i tried to make it do too many things i started slapping on an sd card reader so maybe if you wanted to save wi-fi packets to it you could do that a gps module so if you wanted to do war driving it could do that and then eventually it just became this general purpose wi-fi reconnaissance multi tool iot embedded project that does everything and no one was attracted to that it was a piece of crap nobody bought it that being said um while i do think multi tools suck it's important to know who you're developing the product for so if you are creating a design and it's for yourself as a product developer or a maker or something like that i do think there is some value to adding extensibility and modularity so for example one of the other products i worked on this was also a pretty crazy ass idea i seem to attract that sort of attention it seems but some guy who worked at a gas station wanted me to create a mesh network a cryptocurrency mesh network that detects when truckers pull up to the gas station and then automatically incur a charge through bitcoin wallet so that way they don't have to physically get out of the truck so it was a pretty crazy idea um but i was like hey if you're gonna give me money i'll do it he ended up sending me two thousand dollars i was like that sounds sick so um i wish i had a demonstration of what these actually did but i thought it was so crazy i never bothered to document it but these were some of the nodes that i ended up developing um and the underlying circuit board actually reuses my probe hunter design so in this case um a multi tool is kind of useful if it's just um behind the scenes and it's just for you as the developer so in my case since i knew what i wanted to do with it and since the customer just wanted a polished product it never had to see what was under the hood that was an okay application for it if i was trying to sell this as a general purpose multi tool not so much so this was the microcontroller i was talking about earlier which serendipitously turned out to be a great design choice so this is the d1 mini it's a particular form factor of the esp8266 so you can see up here um there's this little wi-fi chip on this board that's the esp8266 it's a really cheap low-cost wi-fi microcontroller that you can use to do things like develop embedded products that do wi-fi stuff but what's also really cool about this is it has some great applications for wi-fi security you can do things like wi-fi attacks such as the authentication so you can kick people off their wi-fi networks you can also use it for gathering reconnaissance to see when your i don't know your pervert neighbor is home by based off the presence of their cell phone but when i was developing the probe hunter platform i chose to go with this particular form factor since now that i really wanted extensibility in mind and i thought i was going to create a multi tool for everybody turns out they had a bunch of these different modules that you could just stack directly on top so that way if you wanted to add a temperature sensor or another sd card reader or maybe a battery module it was really easy to do that and i thought this was something that consumers really wanted so i ended up um designing around this platform and that's also the chip that we happen to be using in our current design today so that worked out one example um is this d1 mini war driving rig that i created so if you guys aren't familiar with war driving i think i explained it kind of briefly but basically what it is is it's a technique where you combine wi-fi and gps so you can drive around and map the physical location of um wi-fi devices and pin them down to a gps coordinate so this little creation here is basically just the d1 mini with the sd card module stacked on top um in order to log wi-fi data hooked up to a gps but in this particular case um i forgot where i was going to go with the slide but in this particular case we were able to use this for some cool demonstrations um such as war flying i ended up actually making a video on the hack 5 show if any of you guys are interested where we wanted to see if we could physically pinpoint the location of one of our friends in like a five mile radius by just flying a drone over for like 20 seconds around a huge park and we were able to actually use this to triangulate like their exact location but anyways on to the back onto the point of um why hardware multi-tools suck let's take a look at a case study from the company that i'm now sort of working for which is hack 5 so hack 5 creates very specific products that have well hack 5 creates um pentesting products with a very specific purpose and they generally have a single use case so i think this is great because um with a clear cut purpose it's kind of hard to get wrong it's kind of hard to get the idea wrong of what these products are supposed to do and although some of them might have alternate uses um that isn't the main selling point so it's kind of hard to confuse beginners i'll get more into that in a second but for example um these are some of the devices that hack 5 cells and each of them has a very clear cut purpose depending on what kind of engagement you're getting in for example if you want to hack devices over usb you'll probably go with a usb rubber ducky which is a keystroke injection tool if you wanted to do man in the middle attacks you have the plunder bug you can plug this into your phone to read packets plug into ethernet and you can sniff people's traffic um if you see an open ethernet port in the wall and you wonder what it's connected to you have the shark jack you can just plug it in it runs nmap or some stuff like that and then is able to give you that data back if you want to get into wi-fi hacking they also have a product for that called the wi-fi pineapple so it's all very clear cut and it's sort of a i would call it like a portfolio business model where each of these has a very um specific purpose and they're not trying to appeal to everyone so it's really easy as a beginner or even as an advanced user to determine which of these products you want to pick up and it's also not convoluted and crazy like my other ideas so this brings me to my second point which is beginners want to feel smart so how many of you guys have seen this product or heard of the flipper zero cool so actually for some of you guys that don't know what that is i'll explain it in a second but this is actually an example of a multi tool that i think fell a little flat unfortunately so the flipper zero is basically a it's a multi tool for sniffing a variety of different wireless protocols it can also do stuff like physical hacking so it has like these little breakout sensors that you can hook up to custom modules you can do sub gigahertz sniffing you can do stuff like open up garage doors you can maybe clone someone's badge that they tag in to work with you can open up tesla charging ports you can do all sorts of cool stuff it's meant to be a general purpose multi tool for all sorts of hacking um actually i'm gonna go more into this point but basically this thing was marketed as a multi tool for everyone and i think where this design actually fell a little flat was the fact well one thing that they did get right actually was that it's supposed to be a friendly intuitive design that's meant to attract beginners you can see it's a gamified interface that has like a cute little dolphin that guides you through like how to hack different things it has like a little d-pad that's in a gameboy-esque style um but they really um like ramped up the advertising on this one and when they actually delivered it to the audience that they were trying to sell to i feel like it fell a little flat in terms of the actual documentation for um beginner use cases some feedback that i've been getting um on this device i we do a weekly live stream where we just basically ask people about like um any hacking questions or what tools i think is cool or not one of the biggest pieces of feedback that we've been getting on this particular device from people that bought into this hype was that they pick up this product it looks cool and all but they don't really know how to use it there's not beginner documentation on the actual hacks that they can do with it and it's more of the advanced audience that actually knows how to use this tool which is really unfortunate um but this brings me to the point that you really have to know the target skill level that you're trying to appeal to if you're gonna create a device if you're trying to sell to beginners you don't want them to feel let down when they buy this product i feel like a lot of people bought this expecting that it would be easy to figure out like a really cool hacking project to do with project to do with but then they were kind of let down when they didn't really know how to get started and result only they put it on the shelves and felt kind of stupid of course it's not the case for everybody um some feedback that we got from some advanced users who have this product is that they felt like it was packing too many things in one product and they typically advanced users don't really want that because it's kind of limiting it's kind of limiting to have like an all in one um thing that isn't particularly good at everything if that makes sense so that was another key mistake that we made we didn't know any of this going into designing our product because as far as we were concerned we just wanted to get all this crap out the door sell these extra components and that was that so here you can see the first design that we worked on that was called the hack hat that's basically the board we just slapped all these components on to um but the thing was we didn't really have a big idea we didn't know who we were selling to we didn't know who the audience was we didn't know our target skill level or any of that but what we decided the best thing to do was was basically widen the pool and make this a thing for everybody this is a tool for everybody you want to learn how to solder you can do that it has a microcontroller that supports arduino and circuit python so you can use existing community stuff that's out there in order to interact with that board and that was kind of a lazy approach um just leaching off existing resources but I'll talk about more of that in a second so in addition to that in order to widen the pool that we were trying to get rid of all this hardware to we also came up with this crazy paperweight of an idea which was called the long cat so it was basically a longer form factor and it had a battery on the bottom so that way you could now carry around your portable paperweight but then as a joke I came up with the idea for the nugget so since we were sticking to a cat theme just because my friend really liked cats I said hey what if we came up with a conference badge kind of thing where it's actually shaped like a cat and um there's some existing community projects out there that we'll talk about in a second but one of them was called the wifi deauthor so this was actually the project that my friend was initially working on that ended up flopping but the software was still out there so I said hey why don't we run this on this nugget board and we'll see if any people are interested in this design sure enough this one ended up becoming a hit in the other ones not so much this is also the initial oh this is also the uh the author project as it is today so to clarify we don't develop this product but the deauthor is basically a software tool for wifi attacks that runs on that microcontroller I was talking about the esp8266 so it serendipitously worked on our platform but you can see that developer that things didn't end up working out with is now creating some of these other crazy designs like the deauthor watch and also that thing in the corner yeah so this was the initial design this is where it all started so we drew that pretty crazy whiteboard drawing that is pretty akin to the actual design we have today and then we have this crazy ugly board design that I literally pulled out of my ass in like five minutes and then for a rejected design for obvious reasons we have the wifi penis which we also called by some other pretty nasty names that I won't repeat right now but if you want to ask me at the end of the talk what that is maybe I can tell you guys so at this point it was more of a design experiment to see which of these three designs would sell better um so we had the wifi nugget the hack at the long cat but the wifi nugget is the one that ended up sticking at some point so the first month into developing this product fortunately we made the decision that developing all three of these platforms at the same time was a terrible idea so we decided to focus specifically on the nugget instead still we had no clear idea what this product was actually for it was just sort of a beginner kit you could solder it together it sort of ran a wifi hacking tool that's kind of cool but it wasn't our project um it was a general purpose beginner hardware developer platform it had like breakout pins on the back that you could connect sensors to if you wanted to and it was a beginner programming tool so it was an all-in-one it was your one stop shop for doing whatever the hell you want provided you actually knew how to use the tool so that was the first big fallacy of this design before we actually gave up um designing the other boards you can see we went through a little evolution process here i wish i had bigger pictures of these but we ended up designing some other crappy prototypes of the hack cat and the long cat that didn't exactly pan out oh yeah and then down here is the evolution process of the nugget which the circuit board hasn't changed much but the enclosure has a little bit it's become more of like a gameboy type thing okay so this brings me on to the next key thing that we learned which was that if you're trying to sell a product you really need a big unifying idea so this wasn't something that we realized for a while but basically the point that i'm trying to drive here is um obviously creating a multi-tool sucks because if you can't really explain in a single line maybe one or two sentences what this product does and if you barely understand what it does yourself you're going to have a hard time explaining or selling this to someone or explaining to them why the hell they would actually want to use this tool um this slide is poorly made but i'll elucidate in a second so one common mistake um that i think a lot of designers get wrong and actually where we went wrong is that a lot a lot of them essentially try to do what i call um is just reinventing the arduino so what i think arduino got right with their platform is their target audience is developers if you're trying to sell a product your target audience isn't developers you're creating something that's supposed to have a single purpose use for the person that you're trying to sell to and you should really know who that um who these target people are so i think arduino got that right and that their target audience is um basically developers and people that are um trying to develop products but i think if you're doing what's basically recreating your own kind of multi-tool you're basically acting as a shitty proxy for devices that already exist out there um there are some exceptions to that for example adafruit i'll talk about this more at the end but um what i think adafruit got right is they identified the need for um well basically what they did was they also created their own versions of these multi-tool hardware platforms but they repackaged it in different form factors and that's where they have a one-up over something like arduino you can see they have i think this is called like the lily pad or something it's supposed to be extra beginner friendly instead of breadboard wires you can clip on alligator clips you can plug in a battery so it's portable they have the trinket which is like a really tiny version that you can use if you're not doing a whole lot of hardware stuff then you have the feather which is also battery powered it has like a ton of expansion modules kind of like the d1 mini um but i think adafruit is one exception to this case um that got this right so um i'm actually going to go ahead and skip through some of these since i'm coming short on time but somewhere around the second or third month of our development process we decided to do something bold so we weren't we didn't exactly have a product that we could just market and sell as like a finished thing but we did something kind of crazy where we decided to just advertise it on hack 5 through the videos that we were creating and see if anybody wanted to buy this so we were kind of working backwards we were just experimenting to see what kind of people wanted to buy this design what they were most interested in so the very first video that i did was how to use the nugget in order to detect wi-fi attacks damn i wish i had a video but basically the demonstration that i did was i had one nugget that was set up to do wi-fi sniffing and another one that was running deauthentication attacks to kick clients off a wi-fi network and i basically showed an interesting proof of concept that allowed beginners to visualize what the actual process of running one of these attacks looked like so that really hit people liked that demonstration and they also liked that it was a really cute form factor so we actually ended up selling out of our entire stock within i would say the first two or three weeks of this video going out so that's when i decided that i wanted to focus more on these beginner wi-fi hacking demos since people really seem to connect with that and since most of our target audience that was watching these videos were beginners at this point though we still wanted to try to appeal to everybody so this was still a general purpose multi-tool for hacking for programming and hardware expansion so my friend kody decided to start experimenting with micropython which is a programming language that runs on the nugget so this is where things got a little bit confusing but after selling out of our initial stock we decided to see how long we could just run this thing out let's produce another round of these boards order more components see how long we could ride this project out and continue selling it to people still really with no clear idea of what our product actually did except that people thought it was kind of cute which was actually kind of disappointing because lots of people would just buy our product thinking hey it's cat-shaped that's kind of cool but then they got kind of pissed when they realized it didn't actually like do a whole lot at this point so we basically were just experimenting with this design creating random videos every so often I was creating Wi-Fi hacking videos my friend was doing programming videos and then I think sometime around October or so of that year we flew to Los Angeles and we hosted our very first workshop featuring the nugget just randomly so this was quite a mess since we still wanted to appeal to everybody we advertised this workshop as the general workshop to learn programming with Python soldering and also Wi-Fi hacking so I was in charge of teaching soldering and then right after I taught them how to hack Wi-Fi and then right after my friend taught them how to program with MicroPython so this was all a complete mess people were really confused about this product and it was kind of honestly embarrassing you can see though so the workshop that we did in October we decided to go with we decided to create like Halloween edition nuggets this is something that ended up actually sticking and this is something I wish we committed to earlier on but there's a point I'm going to get to later on in this presentation if I don't run out of time which is telling a story through your product so one of one of the ways to do that is by essentially creating like a character or an icon that people can associate with in our case the cat theme really hit with people and I really wish we leaned into gamifying that more towards the beginning but that wasn't something we realized until the very end but yeah these are some really cute designs that we advertised and ended up selling a bunch of these products so anyways about four months into this project we were getting some pretty consistent sales people were buying our product via our web store which is Redia which we were advertising through our youtube show so we were getting some pretty consistent sales also some pretty consistently confused customers but I was creating more wi-fi hacking focused tutorials so I was showing people how you could do stuff like detect when a particular phone or device enters a room with the wi-fi nugget and I was also showing how you could do some other basic networking stuff like create a like how you could legally practice hacking by turning your nugget into a vulnerable access point so that was all really cool and people really like this idea my friend Cody was still focusing more on the programming aspect my friend Cody was still focusing more on the my friend Cody was focusing more on the damn it it's like I'm going through puberty again my friend Cody was focusing more on the programming aspect of things and at the time it just so happened that the developer of the D1 mini platform that I talked about earlier came out with this new design called the S2 mini so this is where things became really confusing so my friend Cody started experimenting more with this S2 mini since it was based around a hot new microcontroller that could do a bunch more cool things it had more pins it was dual core it was faster and it also had a really hot feature called native USB which could basically be used to run USB attacks and some of that other cool stuff which I'll talk about more in a second but most importantly it supported a language called circuit python which in my opinion is better than micro python it's easier to get started with so my friend started focusing more on the programming aspect while I started focusing more on wi-fi hacking so this is where we created sort of a dichotomy with this product because now we were really confused about what our product actually did we did something kind of stupid where instead of actually developing something new with this new hardware thing we decided to just go ahead and start also selling this on our store alongside the old microcontroller so now we had a split of products where we were selling a wi-fi nugget that was called the D1 nugget with the old microcontroller and the new one which was called the S2 so this is all really confusing because as an average person coming to our store and seeing these two very arbitrary nebulous things how the hell do you know which one to pick and more importantly more people were interested in the wi-fi hacking aspect which the S2 didn't support so we got a lot of confusion where people ended up buying the wrong product they got pissed at us we got a lot of customer support emails and that was a real big nightmare so around the fourth or fifth month this is when we really decided that we needed to figure out what the hell to do with our product so we had some pretty good sales at this point I think we were selling somewhere in the hundreds which was pretty good because we only put out maybe four or five videos and basically the return of investment on that was pretty good for how little effort we were putting into marketing these things so at this point the prospect of actually working with the cybersecurity vendor hack five came up where they said hey maybe if you want to start selling this through our store and you come up with a cool use case for it then maybe we could collaborate on something problem is people were already confused enough about what the hell this product was supposed to do so we would have a hard time selling this to the hack five customer base so we really needed so we at that point we realized that we really needed to come up with our own big idea and something that we could actually sell this product as but since we were also confused about what the hell our actual product did we decided to really double down on the idea that the nugget was truly a multi-tool for everybody and that we were the only ones that got this idea right it supports two different hardware platforms you can program on it you can do wifi hacking basically everything terrible idea though so what we ended up coming up with was that we needed to come up with our own customized software so what was initially lazy about what we tried to do and also is a mistake that I think I've seen I'm not going to call out any specific projects even though I have a few in mind but one of the biggest mistakes that we made was we essentially designed this product and we said well since this is a multi-tool we're just going to dump this into hands of people and we're going to let them figure out what to do with this product terrible mistake and that's also a mistake that I see lots of other makers sort of kind of doing I'll talk more about that at the end actually but basically we were just relying on the fact that there were other community projects out there like the de-author like circuit python and Arduino and we said hey we're going to give this to the users they're going to figure out how to use it themselves that didn't end up handing out so well so we decided to create our own customized software for it called the invader so this one was kind of a spin-off of the de-author project not entirely but we wanted to pack more features into this and make it into a wifi hacking suite so you could use this thing to learn about network defense you could detect if there's a threat on your system by creating a honeypot with this you could also use it to detect the authentication attacks disassociation beacon floods that kind of thing it was supposed to be a general tool for learning about wifi hacking and basically everything you can sort of see where this was going this also became a mess quickly because my friend just kept suggesting like more features to out of the product without me ever actually finishing any of the like initial suggested ones so this also sort of started to become a general purpose like multi tool for hacking and that was kind of a really big mess I'm actually gonna skip this because we're low on time but this was a whoa shut up so that was supposed to be a demonstration of the nugget invader but I'll save that for maybe another talk so at this point we were still selling the d1 nugget the s2 nugget people were really confused about these very abstract naming conventions but we decided hey we finally have a direction for the d1 nugget which is the one that supports Wi-Fi attacks it's going to run this product called the invader or this software thing called the invader and that's what we're going to start selling it with as for the s2 nugget it was kind of lame we decided to sell it with circuit python on our store and inevitably people just bought this product and they were like what the hell it doesn't really do anything but yeah that's where we were at at this point so I think this brings me to my last point which is about picking a niche and then I'll wrap up real quick but sort of in the ideation process or I guess like the creation process of this design since we initially started out as I said just wanting to get rid of all these components and we didn't really view ourselves as actually developing a product we worked backwards to learn a lot of these pain points that we now realize today and one of the very first one of the most important things that I think any product designer should take into consideration when creating something besides knowing your target skill level which I talked about earlier you should also know really who it is you're selling to specifically that's not something we really knew we were trying to appeal to programmers and hackers and hardware developers and everybody but we eventually narrowed that down by just putting our product out there and figuring out who was buying it and then eventually honing in on that so anyways sometime around the fifth month we were in a rush to produce the D1 Nuggets we were going to sell it with the Invader project on Hack 5 since that seemed really promising I would say we were somewhere around like 600 or 7 modules that we sold on our personal store with really poor advertisement but our partner Darren who runs Hack 5 promised that we would make numbers way above that if we started selling with him so we started creating we started batch processing all of these Nuggets in like two days I managed to assemble I think like 150 to 200 by hand and then I spent the course of a month really going ham on developing this Invader project and making it all specced out so we could sell it as a finalized thing on the Hack 5 store and in the meantime while my friend Cody was waiting for me to finish developing that he started working on more circuit python proof of concepts so this is where shit really hit the fan so oh also this is so I don't know if any of you guys noticed but we were developing like through whole modules but we eventually pivoted towards like an SMD design somewhere in the middle of this it's a bunch of nuance I'll actually skip that okay but anyways somewhere around the fifth or sixth month when we were set to launch this product on Hack 5 my friend Cody discovered that the S2 Nugget that he'd been experimenting with could run something really novel it could do USB attacks which was really cool so if you guys aren't familiar with HID attacks which is human interface device attacks this is essentially where a device like the Nugget emulates a keyboard and when it's plugged into a computer the computer is going to inherently trust this since it thinks it's just a regular keyboard that's been plugged in however it can be pre-programmed to run like a bunch of keystrokes super fast and type them out into a victim computer such as this demonstration that I'm doing here so we found out the S2 Nugget had that capability my friend was like hey this seems a lot more clear cut what if we focus on developing the USB attack capability of the S2 platform and sell this to Hack 5 instead it should be a simple proof of concept it's more in line with their tool since they also have USB attack products and it's a lot less convoluted than this dumb invader product and project that we started going started working on that's also another point that I want to bring up kind of disconnected is if you're developing a multi-tool everything is a feature so it's hard to focus in on which things need to be improved or added upon since there's just too many there's too much shit to focus on so anyways we decided to make this crazy pivot at the last moment and I spent another month or so just going hardcore focusing on developing this attack platform instead which we ended up calling the rubber nugget and then I ended up also focusing or completely switching over to this new platform so if you guys are confused about the name the rubber nugget is basically a port manto of the USB rubber ducky and our nugget project so that's the hack 5 product that does keystroke injection it looks like a normal USB but when you plug it into someone's computer starts typing out a bunch of crap and it hacks them so at this point we were kind of having a branding slash identity crisis people were really confused about our product they didn't know where to buy them since we were selling on our store it was called the nugget invader we had the rubber nugget we had all sorts of crazy crap going on so we really needed to basically figure that all out so we ended up going with the name rubber nugget so after I finished developing the USB attack software we decided to do the soft launch on hack 5 in over the course of two weeks we hand assembled over 300 of these devices another key thing is you can see we also damn it oh here we go you can see we also made a little design improvement where we put an enclosed faceplate over it and like a little button so it felt more comfortable than the exposed circuit boards that you might have seen in the previous slides but yeah anyways this was really crazy this was a hard pivot for us and of course lots of our personal customers were a little let down that we stopped developing the invader they were still confused as hell because the D1 and the S2 were floating out there they still didn't understand that naming convention but then all of a sudden out of nowhere this new character appears on the hack 5 store called the rubber nugget turns out we sold out all 300 of these modules on the very first day people thought this was really cool and we ended up turning it into something that was a little bit of a more complete and polished product at this point we had our own packaging that I designed we had our own little insert cards with a QR code on the back I think I had my own sticker that I included but basically this was a USB attack suite for beginners and basically what this offered over some of the other hack 5 tools was it was a cute gamified version of some of the existing hacking tools that made it really easy for some of the more beginner audience in their cybersecurity customer base to get started with this kind of thing because it's a lot less intimidating than like the rubber ducky that I showed you earlier okay so I'm gonna go ahead and wrap up by telling you guys basically where we are today I did not update any of these slides cool but basically some key takeaways that I think are really important having cohesive branding I feel like I wonder if I will slide on that having cohesive branding and being able to tell a story with your product allows the customer to really connect to it I think what I'm trying to say with that one is if you yourself don't understand the product or you don't understand who you're selling to then customers or people that come to this product they're not gonna really understand if it's for them they're not gonna know if it's right for them and they won't want to pick up this thing I also think having more on that idea of telling a story with your product I think it's also important to have I guess I would call it like well like one thing that we leaned heavily into and that we're starting to focus on more is more of like the graphics and the branding and that kind of thing and we're actually allocating like more of a budget to graphic design but one thing that people really connected with with our product is the fact that of course it was cat themed it was really gamified and that kind of thing I think that aspect is particularly useful like when you look at some other companies out there like maybe GitHub for example with their OctoCat I think that's really cute people connect with that that kind of thing that's something that I think is really important to be leaned into besides that I think also the community aspect of this is really important if you are designing a general multi-tool for everyone and trying to leech off existing things that are out there kind of like we did at the beginning which was creating a platform saying you know you can use Arduino circuit python all these beginner all these community stuff that's already out there I think that's a terrible idea and it's going to be hard to build a community around your product that will want to reciprocate and also develop stuff for you so also another key thing about this design is all of it is open source so anyone that wants to create any of our projects or inspector hardware it's completely open source and that's also been a really big thing for us I think so I'm sort of going off on a tangent here except I've prepared these last slides but the last thing is I think also generally a thing that I personally learned from this is well actually I would say that working with and targeting this towards like more of an open source community and like makers and hackers and that kind of thing they've been generally more forgiving about our product and also have helped us along the development process with like this experimentation and that kind of thing and also people have helped contribute to the actual like software and everything like that so really giving people something that they can believe in and understand and giving it like a clear focus I think is really important so stuff that we learned you should choose a niche in the end we learned that that was our beginner cybersecurity audience and not programmers, makers, hackers, developers oh another thing that we learned is identifying who your competition is I think that's one of the first steps if you're actually trying to create a product that you're bringing to market identify who your competition is how you can achieve basically feature parity basically how you can get to the steps where they are identify who's above them and then eventually work up towards that so we worked backwards from there after all of these months we're finally identifying who our target competition is and we're sort of working on the next features to build from there tell a story I just sort of talked about that one yeah that's basically it and now where we are today we're still developing hardware we're still trying to sort out this sort out this branding shit show and we're currently about to hand assemble a whole batch of 500 modules for DEF CON so actually we were just we just did our first experiment without sourcing our assembly to China for the first time previous to this we were just hiring random makers in the community but we made a huge screw-up that I think is a really important takeaway where one of the modules particularly the S2 module we decided to go with a cheaper vendor on AliExpress without actually testing the design but it turns out they forgot to connect a single capacitor to ground which caused our design to take an arbitrary amount of time anywhere from like 30 seconds to a minute to start up but we have 500 of these modules that I now have to fully fix by hand over the course of the next two weeks so if you guys are also designing hardware just a random takeaway make sure you test every little component before you actually slap that shit on a board anyways that was my talk if you guys have any questions anything you would like to hear elucidated let me know thank you guys so much for coming this was a very last minute talk I also want to give a shout out to scale for letting me submit my talk actually last night and then approving it for the schedule today but yeah it was really cool to be able to share this story with you guys thank you guys for coming if you want to learn more you can follow me on twitter at alex linda see what i'm up to you can also follow our project on twitter at hack hat tech and if you guys want to pick up a nugget I have some up here at the end so thank you also I'm glad I didn't piss myself yeah it's actually crappy hello is everyone able to hear me all right I think we'll go ahead and get started unless anybody knows anybody who's waiting for the waiting to come into the room got some stragglers coming in welcome so welcome welcome my name is Nathaniel McCullum and I am the CTO and co-founder of profion I'm also the co-founder of the nrx project which you're going to hear about today and the title of this talk is confidential computing why it has to be open source now what are we actually talking about here when we're talking about confidential computing well there are three different types of runtime isolation and you're very familiar with two of them at least the first is the standard virtualization model and basically we want to with virtualization we want to have isolation because it's really important but what is it so there's basically three properties of isolation the first one is confidentiality this means that people can't look in and see what your workload is doing the second one is integrity which means that people can't tamper with your workload while it's running and the third one is availability which means that your workload actually gets the resources that you are expecting to get now generally the the last one availability is easily observed so if your service is down or if it's you know if it's not getting the CPU cycles you expect it to it's performing poorly then you should be able to observe that pretty easily the other two however are much more difficult to observe and there are three different modes in which we can talk about isolation so type one is workload from workload isolation so this is where I think of its typical cloud system where you have a host and you have multiple workloads running on the same host and you want to make sure that one of the workloads can't reach into one of the other workloads and fiddle fiddle around with stuff and again we're trying to get those three properties confidentiality integrity and availability so the the workload one workload should not be able to for example hog all the CPU resources now we can do this pretty well today right we have C groups this is this is a solved problem but confidentiality and integrity are a little bit trickier and we're going to see why that's because the second type of workload isolation type two is host from workload isolation and this is basically we want to make sure that the workload cannot attack the host itself and this should be pretty straightforward you don't want workloads that are running on a system to be able to attack the host because why well then they'll be able to attack the other workloads so it's really important to notice that with type two isolation the entire of our security model is predicated on type two right so even though we want to type one we actually have to have type two in order to be able to get it now we can do this pretty well today and web assembly for example was designed is a new technology with using new new primitives to be able to provide type one and type two isolation so you've probably heard a lot about web assembly outside the browser and how it's going to hit the hit the cloud space and that's because it provides a lot of great tools for being able to do these two types of isolation in a way that is very reliable very lightweight and so forth however we have a third type of isolation and that third type of isolation is workload from host isolation that is we what if the host becomes malicious and begins to attack the workloads well there's really nothing that handles this today at least not in our traditional cloud paradigms so VMs don't handle this a host can definitely reach in and tamper with whatever it wants to in a virtual machine and containers don't handle this for exactly the same reasons type three isolation is very important for many types of workloads if you have sensitive data for example this could be medical data just to give one example could be financial data it could be regulatory data where you have to comply with laws for example the GDPR however there's also sensitive applications as well that is where the application that you want to run may contain algorithms that are themselves a trade secret and there's number of places that do this lots of AIML for example comes to mind and so we want to be able to protect both sensitive data and sensitive applications but how do we do this protecting from the host now the host could be vulnerable for a variety of reasons it could be because the hosting provider that you that you are running on are themselves malicious right but that's probably not the most common case more common case would be something like a system administrator who goes rogue and decides that they want to cause as much damage as they can on the way out the door or it could be just simply that there is a kernel compromise there's a number of kernel compromises for example that give you ring zero access from outside of the server and so these are really big problems so the host can actually be malicious either based upon the fact that there's an entity on the other side that's malicious or just simply due to a compromise and privacy enhancing technologies are top of mind for these types of situations where you want sensitive data and sensitive applications these are things like homomorphic encryption for example but we'd like to introduce confidential computing now hopefully many of you have heard about confidential computing confidential computing is the protection of data in use by performing computation in a hardware based trusted execution environment and this definition was crafted by the confidential computing consortium the confidential computing consortium is a linux foundation project and it is based on open source software so like other linux foundation projects you can basically start an open source project there and collaborate and this is precisely what the confidential computing consortium was set up to do the ccc has broad industry adoption there are players like intel amd arm red hat microsoft facebook excenture ant and so forth the list is quite long in fact that's not even the full list not even by a long shot so everybody wants to play in this space why? well you see confidential computing uses trusted execution environments and this is based on cpu hardware so it allows you to actually craft a workload that runs in a way that is protected and encrypted by the hardware itself so they encrypt workloads and they predict integrity and confidentiality and this is true even if for example you have a tap on the the bus to the memory right you you literally cannot observe and cannot tamper tamper with the application even if you have direct access to the hardware so what's actually happening without a tee memory pages are are unencrypted so you have a typical workload you've got some memory running and everything is running in an unencrypted way this is what we normally do every day but with the tee the memory pages of that workload are actually encrypted and they're encrypted by the cpu itself so there's new hardware in the latest generations of chips this would be ice lake xeon from intel and the Milan generation from amd and they actually have integration with the memory controller and there's built-in circuitry for doing the the the encryption and so basically when when it comes time to set up one of these execution environments basically you tell the firmware I would like one of these environments please and it generates a new cryptographic key that's used for all of the memory associated with that workload and then all of the memory is encrypted transparently using that key the only way that you can get memory into the initial state is to ask the firmware to put it in which the firmware does a cryptographic measurement of and so now we know exactly what the contents of that trusted execution environment are and then finally you can begin execution by asking the firmware to essentially jump into that tee for you and it's really important that each tee instance has a separate cryptographic key right so even if you were able to tamper one of those keys it doesn't necessarily impact all of the other workloads that are running on the system and it's also important to note to this that this is happening as a hardware primitive so this is not something root has access to if you have a kernel compromise and you are on ring zero you do not have any access to this memory so the memory pages are only ever unencrypted within the cpu so as soon as you do a load or a store instruction the cpu is telling the memory controller transparently to do that encryption back and forth once the memory comes into the cpu and is stored in the registers it will only be allowed to use in those registers when you're executing in the context of the tee and then it clears the state of the cpu when it swaps out of the tee for other applications and there are two broad categories of tees so we basically have implementations from all of the major hardware vendors on the left hand side we have process based tees and the way that process based tees work is that you have a normal linux user space process and when you set up the tee it is a sub region of that memory so you don't actually get a different set of page tables to manage it's all just the normal user application but a certain region of that memory is encrypted there's two primary examples of this technology the first one is intel sgx this is one of the older ones trust zone from arm came earlier but we do not believe it meets the definition of a tee which is why it's not on here but intel sgx is about eight years old but really just saw limited adoption in different places until they brought it now it's now available on every ice lake xeon server as of this year the other one other major one in the process based tees is risk five sanctum so for those of you who are interested in the risk five community there is a design for a sanctum which provides process based tees there's also another one which is actually not on the slide because it's not been released yet which is arm is releasing you see realms on the right hand side they are also going to be releasing a process based te on arm in the coming months on the right hand side we have virtual machine based tees and the way that these work is different than the process based tees so in this case you bring up a virtual machine and then everything that's in the virtual machine is encrypted so every time you cross across the the virtual machine boundary and are out you're doing you're crossing into the te at that point and there are several examples of this technology as well amd sev intel tdx tdx has been announced but it is not yet available in silicon but they are working on patches for the linux kernel you can go google that for more information sev has had three generations now the first one came out on the naples series processors they improved it again in in the realm generation but really where it starts to get really fundamentally interesting is in the latest generation the malon generation because that's when they finally added integrity support for memory pages as well so that's the first one that now fully meets the criteria of the confidential computing consortium a couple other worth mentioning here I've already talked about realms pef is from power so IBM is working on this as well but let's back up here and take a look at the larger picture what are we actually trying to accomplish we're going to talk about what nrx is in a moment but I want to first start off by just giving our nrx foundational principles there's 10 principles that we have in the nrx project the first one is that we want a minimal trusted computing base so in other words there should be as little code as possible in order to get an application up and running and this is just simply for attack surface the smaller the trusted computing base is the harder it is to attack second is the minimum number of trust relationships so who do you have to actually trust in order to get all of these pieces up and working and we want to be able to minimize those as few as possible number three deployment time portability it's really important to that once we have tees that we'd be able to do what I call functional equivalents of workloads and that is when you're doing cryptographic measurements now of these tees to be able to guarantee what's actually inside them you need to be able to determine is this workload the same as another workload well if you've compiled the native application for example on every single tee they're all going to have wildly different measurements and so now you'll have to build a huge management framework on top to track which measurements are which applications and it's very very unwieldy and we want we want something that's simpler than that so think about for example a blockchain use case where you wanted to be able to execute a job and you wanted to be able to certify that this job ran in a tee through an attestation report and then you wanted to put the results of that attestation report on a blockchain for everyone for time immemorial to be able to validate well we want this to be simple and self-contained and so we need some kind of deployment time portability in order to be able to determine that this is in fact this workload no matter whether it ran on Intel or AMD or ARM etc there's another factor of deployment time portability as well which is that vulnerabilities happen and they happen in hardware and in fact every single tee I've mentioned has been compromised now it's not it's not we shouldn't be fatalistic about this the situation is in fact rapidly improving attackers are really having to put a lot of work in order to be able to to actually penetrate these but what happens if you have a hardware of vulnerability today and your application is running and it has all of the stuff that keeps your business running inside of that tee well you've now essentially lost all of that security that you've had because you've consolidated your trust onto a particular hardware vendor so what we want is we want the ability to deploy a workload to another host to another piece of hardware without having to recertify without having to do any sort of you know additional testing it should be just an instant switch if you're running on one piece of hardware to running on a different piece of hardware when it becomes vulnerable principle number four is the network stack outside the TCB and this comes from years of experience looking at kernel vulnerabilities and in fact all of the worst kernel vulnerabilities were all in the network stack and so what we want is we want for the network stack to actually be outside of the tee and the reason for this is because if there's a vulnerability in the network stack that's going to be really really bad and then we don't want that to be inside the tee we want it to be outside number five security at rest and transit and in use that that's pretty straightforward we want to enforce that we always have encryption on everything whether it's on the disk whether it's on the network or whether it's being used it's an entire envelope and once data and code enters the system it never leaves unencrypted number six auditability pretty straightforward that goes with the minimum minimal trust of computing base as well we want it to be small so it can be audited we want it to be seven open source so that it can be audited right many eyes make it much easier to find bugs eight we have an emphasis on open standards on memory safety number nine and we have a commitment to know backdoors and we make that as a public commitment to everyone who might use the project so what is nRx nRx takes web assembly it allows you to take an application written in whatever language you want to compile it to web assembly and you can deploy it using nRx onto any of these hardware platforms so why web assembly well importantly it gives us a bunch of these properties that we really care about so it gives us that minimal trust in computing base it has a minimum number of trust relationships we have deployment time portability we have auditability because we can actually see what is actually happening we know that web assembly is well defined and we also have open source and open standards the same thing with rust nRx is written in rust and rust gives us auditability because it's it's pretty easy to look through rust code to make sure it's doing the right thing it's much harder with c and c++ for example it's also open source all the code we do is open source and as well as our compiler right so we were open source all the way down and rust also provides this memory safety which is a really in critical aspect of the system because memory safety if you if you don't have memory safety then it's going to be a source of bugs for you and we don't want those kind of security bugs inside the TEE so why open source well with open source again we get this minimal trust in computing base we get to use the best libraries that are out there that are well reviewed and well understood and we don't have to write everything from the ground up we also can make sure that what we're doing is small and tight and anyone can audit that which is the next point it's also we obviously want to be open source we want to have open standards and no backdoors so if you look at our traditional virtualization stack let's let's talk about who you actually trust so in just if you want to deploy an application in the cloud these are basically all the different parts you have to deploy you have a CPU and a management engine usually the cloud takes care of that and then you have a bios in EFI usually the hardware manufacturer working in conjunction with the cloud you have firmware for that same thing but notice that they're different colors and that's because all of these are coming these components are coming from different places and this is a lot of trust relationships that can go wrong if you're deploying a common operating system usually you'll have like your bootloader kernel and hypervisor as well as user space and and any middleware you know from that sort of same provider but if you're deploying an application just on bare metal you still have to trust all that stuff if you're doing virtual machine you need to trust a whole bunch more stuff because inside the virtual machine we're going to reproduce all of the stuff that's on the outside including the bootloader and the kernel the user space middleware and multiple applications and this doesn't really change if for example you're doing a trusted execution environment using virtual machines and so when you when you actually have a TEE that's VM based what what actually happens here is that we exclude all of the stuff that is outside of the virtual machine because now the virtual machine does not have an ability to tamper or sorry the host does not have the ability to tamper with that virtual machine and so now we've just clawed back a huge amount of simplicity because we don't actually have to worry about vulnerabilities in all of that part of the stack but we still have a lot of stuff that's inside the VM as well things like the bios bootloader kernel user space in middleware etc one of the things that's really important by the way and it's one of the reasons why nRx has chosen to deploy web assembly rather than a traditional guest virtual machine with a full operating system is because the only way to maintain compatibility and this is how you're seeing everyone implementing TEE VMs in the cloud is that the host is providing the bios and so the cloud provider provides the bios that sits inside of the TEE and then you do everything else but I thought the whole point was to exclude the host from that relationship altogether right and so we have a in order to maintain this sort of compatibility with existing stuff we have to we end up compromising the integrity of the system so what we actually really want is we want to minimize that and reduce all of those layers and just have a single TEE runtime where we can run our application inside and so fundamentally what we end up here with is three components that you need to trust and this goes back to that question of how can we have the minimum number of trust relationships so the first thing that you have to trust in a TEE system that's well designed is you trust the CPU manufacturer well I hate to break it to you if you don't trust your CPU you probably shouldn't be doing computing you know there's yeah that's all I'm going to say about that there's really no way you can have any guarantee about the output of any sufficiently large program if you really don't trust your CPU manufacturer so besides the CPU manufacturer who else do you have to trust well you need to be able to trust that TEE runtime so the bit that's sitting below your application inside of the TEE and this is precisely what NRX provides NRX provides a common open source TEE runtime that is something that can be audited tested because it's open source and it gives you that minimal or relationship so CPU NRX and your own application now I hope you can trust your own application maybe you maybe there's reasons not to trust your own application but I suspect that most of us have a pretty good trust in the code that we're actually writing but there's actually some more that you need to trust because what good is actually bringing up a TEE inside of a cloud if you can't have any proof that it's actually a TEE right I can if you tell me you know please launch my program and let's bring it up in the cloud and I'm the cloud cloud provider and I say oh yeah I definitely set this up it's like super secure best it's the best of securities what kind of security do you have oh it's all of them all of the all of the ones that you like yeah all the securities yeah you've got them all right well this is a bit like asking a kid to grade their own homework so what we actually need to do is we actually need to be able to get some some sort of non-forgeable message out of the application to prove that what is actually running inside of the application is in fact tamper resistant and this is where an attestation service comes into play now again the way that the cloud providers today are building attestation services besides the fact that they bring up a virtual machine with their own bios on the inside of it inside of the TEE they're also then running the attestation service in their own cloud and they are managing in the case of I won't say any names here but they're actually managing the list of the CPUs that are that are good within their their own cloud so this is this is kind of a problem and so you need to be able to trust your attestation service and the way that attestation works is basically if you remember earlier in the talk the firmware brings up the environment and the only way to get pages in it is through the firmware well this is where all the we get a cryptographic measurement of everything that's put inside of that that TEE and the attestation is a cryptographically signed message from the firmware out indicating the state of that particular application and the attestation service is something that's running either in your own infrastructure or hopefully inside a TEE but not in a way that you can tamper with it and the attestation service actually checks the signature on that message to confirm that everything is correct now because this is the modern internet and we do things in a modern way you're probably also going to have an application registry that you're deploying applications from and this definitely also needs to be in your in your trust model although there is there's ways we can minimize that trust by using for example append-only logs and signatures there'll be something interesting some interesting news in the coming months from the bytecode alliance which is the web web assembly consortium regarding this but I will not steal their thunder so this is essentially the components you need to trust right besides the cpu and your own application there are there are three things that you need to trust the te run time the attestation service that validates that the application is actually running in a way that's tamper resistant and the application repository that's going to deploy your actual application into it this is what needs to be open source notice the three pillars are all there right if you can't have confidence about these three important pillars the application repository the attestation service and the te run time then you cannot have confidence in the system because all three of these are required in order to have an efficient system at work so we're going to we're going to give a little demo here now I will ask that I'm going to invite all of you to come to try.nrx.dev just not now hold I know you want to do it hold on yeah if we are going to be giving a demo and I just don't want to swamp it's with traffic while we're while we're doing this demo so I would like for someone to tell me which platform they would like to demonstrate we Intel I heard an Intel oh we've got both of them let's put it to a vote okay who wants to see Intel first okay who wants to see AMD first oh that's really close you know what it's really it's really close close because that's exactly the point you need to be able to move your workloads from one to the other without any friction whatsoever so I'm going to I'm going to do the smart thing and I'm going to click the start now button which randomly picks one because counting y'all's hands is just way too much work okay so we're going we're going to log in again because we are allowing people to run random workloads here and I'm going to walk you through an application this is a very simple Echo server now the thing that's nifty about this is I'm going to toot my own horn a little bit we had a profion one of our engineers Richard Zach just landed in the last few weeks web assembly support into Tokyo and Rust and so this is you're seeing the the newest stuff hot off the presses but this is exactly the experience we want we wanted to just work out of the box with your existing language without having to do anything special no extra tools no modifying your application we want you to be able to use the existing tool frameworks that you they trust so we're using Tokyo which is an asynchronous framework for rust and really the only bit of code here that's specific to running in an arx is that line and the reason for this is because we're going to be pulling in from outside a pre-opened TCP listening socket so it's already it's already opened that's the only thing that's WebAssembly specific everything else here is just standard Tokyo and so what we're going to do is we're going to set the socket to non-blocking we're going to create the final listener and then we're just going to start our loop and we're going to listen for incoming connections when a connection comes in we're going to spawn a background task now this is this is async this is not threaded yet but this is asynchronous so we're going to spawn a an asynchronous background task to handle each incoming connection and we're going to come down here we're going to read bytes off of that socket we're going to handle when the socket disconnects and then we're going to write the output from that to standard output and back on the socket right pretty straightforward application and so let me show you how to compile it for web assembly normally and if you're doing rust you would do a cargo build if you want to do it for web assembly you add target equals wasm32 wazzy yes I can embiggin it all day long my mom taught me how to do this so yeah so normally you would run just a cargo build but we could run a cargo build we're just specifying a different cpu target web assembly is just a virtual cpu for those that don't know and web assembly is deployed in browsers all browsers today for many years so it's a it's a well-recognized mature technology there is a wasm64 in development so the standard is basically done but there has not been a huge amount of demand for it interestingly enough the there are some well so there's not there hasn't been a huge demand in relation to the to the amount of effort that it's going to take to implement it that's the way I should put it because we get a bunch of sandboxing stuff for free when you can have 32-bit pointers on a 64-bit system and so there's a whole bunch more work to implement it but people are working on this yeah it'll it'll be available so yeah so wasm32 wazzy and we have an application now that that's it so if we look in directory here you see this tokyo-echo.wazm and that's our application that's all we've done so we want this to be an exactly out-of-the-box experience and we have to have a little bit more and the reason for this is that when you're running an application in NRX you can't take things like environment and arguments from the host platform because the host is not trusted in this model right so you have to be able to grab those from somewhere else so we have a configuration file called nrx.toml and I'm just going to paste this in here and this is pretty straightforward we're just defining that we want standard in standard out and standard error as well as we want to open that additional listening socket now if you remember back here I said that this line of code right here is the only thing that's WebAssembly specific well this is because we are pre-opening this file descriptor number three and this is right here where we've created that file descriptor number three so that's the only thing that's really specific to this environment and so now we're going to pick our wasm file and we're going to click deploy I'm sorry oh the file the file is pretty small it's well that's the debug version let me do the release version you're taking me off the path in my demo this is where all the dragons lie so I'm just adding the release flag and this will just build the application with a bunch of additional optimizations including optimizations for size so yeah yeah yeah so it's two it's 2.1 meg but that's everything right think of it this way the entire nRx runtime plus the application is still smaller than the bootloader and linux in booting a VM right so just put that in perspective okay so we have our application running here over on the left unless it hasn't timed out yet and uh we're going to all right it's connected and so I'll say and there it is yep so now that's interesting but let's let's run that same workload on a on a different technology so we did intel first now we're doing AMD notice there's no changes whatsoever to the application and if you were deploying in this in kubernetes right it's really just like a matter of picking a different a different target that you want for for your application so you can be up and running in a vulnerability in a matter of minutes so we're going to do the same thing on s and p I'm connected again and this ah whatever you guys can see it now you can see that this is actually a distinct system that's running now I talked before about making sure that we could have an envelope where every time that data and code entered the system it stayed in the system encrypted and never left unencrypted so one of the problems that we have I don't know anybody saw the article recently that there's something like some crazy number of open security jobs in the world right now and a big part of this is because although we've learned how to do devops at scale for example with github and your cloud provider right we still haven't learned to do security at scale yet and so everything is like picking and matching these like different tools that you have for issuing certificates and getting renewals and making sure that disk is encrypted and you have to sort of glue it all together in a big ruby Goldberg machine and it's very time consuming and it's very error prone and if you're asking a junior guy to do it you have no way in hell that it's ever gonna be secure so what we want is we want to be able to we have people telling us all the time we should shift our security left how many of you have heard shift security left right okay the problem is shifting security left doesn't actually work and it just blames the developer for not being able to be perfect every single time is anybody perfect every single time in this room I'm not so I don't think that is and blaming the developer doesn't really help us we actually need to shift our security to the right we need to be able to scale our security the same way we've learned to scale our developer experience and the same way we've been able to scale cloud deployments and so let's see what it looks like to do an attestation and we'll do it on S&P again since it's right here and with our attestation we're going to we're going to enable the steward steward is the attestation service and we're going to change the protocol of our port from TCP to TLS and we are going to run the same exact application again so what's actually happening here now is that the the workload has gone and done an attestation as part of the startup procedure so it went out to that remote server gave evidence through its through its attestation report to the server that this is a tamper proof environment the server has validated everything cryptographically to make sure that all of the measurements are known and so forth and only once that's done does it issue a certificate for a private key which was generated or for a certificate on a public key for which the corresponding private key was generated inside of the TEE and so we can now do openSSL and connect to this and you can see that we got exactly the same thing now notice what I didn't do I didn't install a bunch of other additional tools and I didn't have to worry about whether or not my key is going to be compromised or whether my disk was encrypted because everything was encrypted always and so there was nothing to mess up right all I had to do was change the configuration and we were able to successfully shift the security to the right so let's do another demo this one is going to be a video demo just because I want to show a pretty complex attack and that requires me to have root access on the server so let me go back to here and so we wanted to show that you can actually do something that's a little bit larger than just an Echo server so this is actually a multiplayer wordal clone that we wrote and basically instead of trying to you know guess the word of the day you're actually trying to guess the words that the other players have put into crypto and so as you type the words in they show up so it's typing in the word world and there it is so forth I'm going to skip ahead here and so notice purple we got a match that second player had successfully guessed the first player's word and now this is important that the unlike wordal today wordal is generated for those of you know the word is sent to the browser and everything is done service or client side and so if you want to know what the word of the day is and you want to cheat you can just do that trivially right with any browser debugging but in this case all of the words that are being guessed by the other players are now held on the server inside of an anarchs keep and because they are inside of this keep everything is protected from the host and so we want to see what it would look like if we wanted to cheat in this game so we're going to put on our attacker hat and we're going to run the crypto application using web assembly using wasm time wasm time is not an arch now please don't fret I'm not picking on wasm time wasm time is fantastic in fact it's our web assembly implementation as well inside of the keep so this is not about about dising wasm time at all but it is however about pointing out how easy it is with root access to be able to find secrets in an application so we are we've launched it in wasm time and we're now just going to grep the memory for words so we found the process and we're off to the races so in a moment what we're going to see here is we're going to see the words that the other up there it is young is one of the other guesses that other players have made and so all it takes is one root level vulnerability to have this level of access to all of your virtual machines all of your containers all of your applications running on a host in a traditional manner that's a problem and we need to be able to do something better so we're going to show the same exact thing running again by the way we would get if you were running anarchs in debug mode you could scan the memory like this too right there's when you turn off debug mode there's there's no protections so we're going to do the same thing again here we're going to attack crypto but this time we're going to do it using anarchs and so we're actually doing an upload here we're uploading the application to drawbridge and then we're going to drawbridge is the application repository and we're going to deploy it right out of the application repository that's that line anarchs deploy it's actually way simpler than this today this is an old video oh I missed the good stuff so we're going to do the same thing we found the found the anarchs application and we're going to do exactly the same memory scan just for the different process it it means that I don't remember it's a kernel it's a Linux kernel thing and I just don't remember exactly what case causes that and that's it we didn't find any words so just simply by changing which host we were using whether we were using wasm time or whether we were using anarchs on supported hardware we were able to defend the application from attack now this is actually a call to action I somehow lost my slide share there we go we want to see your elite skills so if you go to anarchs.dev slash crypto we will basically give you everything you need you'll have the anarchs application have crypto and we want you to hack it and there will be cake and by cake we mean hardware and cash so please come give it your best attack and the important thing here is that profion is about to launch enterprise support for this and we want to make sure that it's ready to go so please please come and do that we will be announcing the winners of the first round at black hat which is coming up here very shortly and so yeah you have a little bit of time to get your submission and basically if you can run crypto in anarchs and you can successfully attack it from the host side then yeah you win last let's do one more demo so we actually didn't make this demo and I think it's important to show this demo because I want to show that other people are making stuff too and it can work in anarchs out of the box so this demo was actually put together by microsoft this was given at kubecon so we gave talks at kubecon for there was a web assembly day at the beginning of kubecon and basically everybody showed up and showed off all their web assembly stuff and so microsoft basically touted what they've been working on for web assembly support in dot net so they gave this this application and it's a little greenhouse monitor application that it's entirely written in C sharp using I think blazer is their web framework and yeah this is so let me pause the video here for a second actually get back yeah so this is the actual source code for the demo and we we forked it just so we would have a copy of it and we but we also set up a builder so that we could build all of this in github actions and the reason we did this is so that this is completely reproducible you can go and look at exactly how this was built and we're going to run this on try.anarchs.dev now this is a little bit older version of the demo system but we're going to we're going to do exactly the same thing as we did before which is we're going to choose that wasm file and we're going to copy and paste our config and now it's running it is C sharp so it's going to take a second to come up yep there we go and there it is so that's that's C sharp running in web assembly on top of an arcs we did not write this demo so I think it's important to show we're not the only magical people that can do this you can do this too so let me just give imparting a few a few last words this is essentially what the narchs project is about we have three open source projects the main one is called narchs that's the narchs binary and that is what we use to create a keep a keep is in castle terminology we follow castle idioms and so a keep in castle terminology is the the strongest most central secured place inside of a castle and so that's the the narchs binary creates keeps we have two additional services steward and drawbridge steward is the bits that does all of the attestations for you all integrated into the system and then drawbridge is an application repository for being able to upload your application and then deploying it with just a single slug kind of docker style right like you're normally used to doing with docker or podman and so basically what this allows us to do is we can unify these two different types of tees and we can run the same application on both by using web assembly web assembly has a standardized interface or it's rather standardizing it's in the process called wazzy and wazzy is starting off with things like arguments and environment variables and sockets but there's actually other really interesting wazzy APIs coming out like wazzy crypto which is a full fledged cryptography API there's a wazzy nn which is for doing all of your AIML stuff for neural networks as well as many others there's a bunch of people working on different APIs and these are all standardized right when you're when you're running on top of nRx you're not developing to some custom platform we want to give you a standard out of the box experience with standardized protocols and APIs and so this is the bit that actually runs inside of the TEE the web assembly engine the the wazzy API and then your application will oftentimes have a language binding for example if you're writing in C you'll link against wazzy lib C rather than GNU lib C or some other lib C and this is what gives you the ability to run CRC plus plus on web assembly and all of that ends up in the keep and again the the total package here is smaller than just your bootloader and your kernel on on a Linux system and this is really actually important for density reasons as well because if you're running VMs today we can share lots of pages you know if you've got 100 different VMs running on a system and they all have the same kernel you don't have to de-duplicate those pages in fact that's the way firecracker works and you guys know firecracker the minimal minimal VM engine so they they basically map the Linux kernel on the host side and then into the guest and then those pages are all de-duplicated across the host which is how they get density but in an encrypted environment you can't do that you can't share pages across multiple systems because each page is encrypted with a separate key and so there's no sharing at all we've got to be able to claw back that density by reducing the size of computation and the end result is basically you just have this this te and all it is is the minimum amount of stuff that you need to write and run your application and everything else is up to you as the application developer so please come check us out our website is anarchs.dev you can try anarchs.try.narchs.dev and which is what you saw today check out our blog and github as well please star the project on github it's actually really important to us so we'd like to know that you're out there and that you're interested please please start and let us know and we're a friendly group of people so come chat with us on chat.anarchs.dev and we would love to hear from you last but not least I did mention that we are going to be releasing enterprise support for this that will be tbd but it's in the coming months and we are calling this profian assure so you basically we will have you can run it yourself we'll provide support for it we can run it using our services and we will support you and give you resources there yeah and that's that's basically where we are well questions can I answer for you right now the limit is we got four gigs on was a 32 was a 32 so but it's designed for microservices right and this is this is precisely what we're what we're trying to build is lots of smaller services rather than big monoliths yes yeah the nrx runtime itself just stick it in a container and make sure it has access to the device nodes the biggest pain point will be that not all the kernel drivers are upstream yet so the good news about intel sgx the the last drivers that we need are currently staged for merge for the next release so that will be coming very shortly and amd s ebs and p is still a ways away yet but you know if you're looking at running this in production come talk with us we'll provide you a kernel we'll give you a stable experience that's what we're here for that is a great question say it nice and loud no they're not no they're not it's actually a misnomer so what actually happened was Intel modeled sgx after trust zone initially and it was incredibly small you could the initial limits were like 196 megs at the most and and many systems had BIOS configurations that were significantly lower than that so but what they did was they realized over time that the real market for this is in the cloud and they were trying to do it first in mobile and then we're trying to do it on laptops and it just didn't get traction there and so what they're doing now is they're pivoting it to the cloud so what they have done is they removed it unfortunately from the newer generations of the mobile devices so it's not in your laptop anymore which has led many people to think that sgx is going away which is not true sgx is continuing not not only is it continuing they brought it in in the latest generation of the server chips and also what about their new technology tdx isn't that going to replace sgx it's actually not the virtual machine technology is actually built on top of sgx so both will be here for a while yep yeah that's exactly the case and basically I won't pick on intel I like intel but I don't think the rollout of sgx was managed very well and so you got a few sparse systems that that had access to it for example there was some seventh gen celerons which where they were target and adam which they were targeting towards the mobile space and then they had it again in the 10th generation in like the the i7 1065 and the i5 1035 CPUs very good CPUs by the way but they yeah they they brought it back for those but then when they did the overhaul for the 11th generation they dumped it so it's no longer there on the client side but it is there on the latest server chips in fact the latest server chips are the only server chips you can get it on so you have to you have to have ice like xeon yes yeah exactly so there's definitely a huge market for it in in edge space as well as in the cloud one of the things that we are seeing is that the place where people are most interested in it today is they have data and applications that they can't put in the cloud because they can't disclose them or they're regulated in some way and so they can't so for if you're in the banking industry at all right FRTB is coming for you whether you like it or not yeah oh great great question so SEV the second generation not not the Milan generation was available in the AMD epic embedded line and I can't talk about future AMD products but there's a lot of demand for it so that would give it to you essentially in a mobile form factor ARM is also planning on ARM is not a manufacturer right so they're they're releasing the IP to manufacturers and you will see this in a variety of of form factors as well we do absolutely intend to support ARM realms as soon as it's available in fact we support ARM today you can actually run NRX without encryption on a Raspberry Pi yes no and in fact the earlier generations were vulnerable to specter attacks so side channels still remain a problem and CPU vendors need to fix those problems the the good thing I would say is that everyone is paying attention to those problems these days and they they are getting worked on but you should expect to see some more side channel vulnerabilities and yeah there there are a good example of like a tax that are practical today on on SEV there's a plundervolt is another one is one of the ones where you can basically trick the CPU into acting in different ways on different keys if you undervolt the CPU so that there will be support for or defense against that in the next generation of chips but you can see that like these attacks are getting harder so you actually not only have to have physical access to the machine now you actually have to be able to revolt it yeah the in fact some of them are software controllable yes glad you liked it yes so there's a reason why this domain says snp.equinix.try.nrx.dev because this is running on Equinix today if you also go back to the main page you'll see other platforms coming soon that does include a large three-letter cloud so wait and see yes with what is pkvm I don't know yeah I correct I believe it I believe that that's one of the dependencies for cca realms which is coming out which has already been announced so yeah basically you need that support first then you need the memory controller encryption support in actual silicon to be able to swap the keys out for the memory pages and once you have those two things then you know basically you have an api interface to the firmware at that point great yeah the cca realm stuff is not yet out there are you know there's an emulator available but the the firmware interfaces aren't finalized yet well I think we're at time thanks everybody for coming hello oh yay the good news is I will not be singing today all right mic check again all right so it sounds like we're good I think we're officially at 430 or so thumbs up 15 seconds we'll start I am going to do the podium thing because I've got notes and Carlos my friend has shamed me that I did not prepare sufficiently to remember my notes so I would like to occasionally glance at them and say Mary this is just like old times Mary used to be one of my students she always walked in late with a starbucks this is awesome I'm having a flashback all right so good afternoon everyone thank you so much for being here we realize that you have a choice at scale conference and either there's a really good one after here or you want to get a C for it I don't know so my name is John Ticklick I'm hoping my voice is going to last as you can see the title is on my it's about end of support operating systems this paper was originally created as part of my sans masters program I'm glutton for continued education as long as I work at a place that will pay for it so sans technology institute is the accredited university portion of sands I'm sure many of you have heard of sands so one class to go I should be done in a few weeks but that's how this paper came to be and coincidentally this paper was supposed to be finished when scale was supposed to be held where March right March and then this got delayed I was late on my paper so it all worked out all right so just a little about me first I figured I throw up my weird Al and William Shatner pictures just to let you know I am one of you right we all embrace the same things I heard a funny quote yesterday you know do you have 30 years of experience or 30 years of doing the same thing so I thought that was kind of cool because I do have 30 years of varied experience over the years was in the Navy Reserves for 28 years that's actually how I became a got into cyber security I met one day a young man who was a PhD working on something that px said was called Tor and I didn't know what he was talking about but I'm like now I could say hey I met the guy who created Tor and as I mentioned in graduate school one more course I have been since 2013 an adjunct faculty member at both coastline college who has a booth if you haven't stopped by there yet and Long Beach City College and I recently accepted a full-time teaching position at Long Beach City so starting this fall I am a full-time professor part-time Boeing excited about all that so what are we going to talk about generally open source operating systems and what happens when they reach end of life and what I'm thinking about what this might mean to you is if you're working someplace where operating systems open source are being used because the management is like oh it's free yeah we like that but understanding that you've got to pay migration costs eventually you've got to move on and then if you don't what are the impacts what are the potential risks that you're putting on the network and then if they really have to stay with an older operating system educating them on what options are available for continuing to extend your maintenance and then again if we got some old stuff on the network like a vax machine in the back of someone's car for those who came over here earlier you know what else can we wrap it around my friend Carlos and I I don't know if you were there back in the early days of where we worked where there was a system that's like no one touch it we don't know what it does but we need it right you know so we just had a wrap stuff around it to to protect it quick show of hands how many people are running Linux at their home been easier to say who's not running Linux at home okay yeah so I'm in good company right you know if you go to a different group of people and they're like Linux what's that what are you talking about yeah we are at scale excellent point that would be uh it would be funny if people didn't do it so what are the risks of unsupported operating systems so I I can't say I claim the quote or created a quote but I'd like it every day is a zero day right there's always something new and our typical hygiene for a vulnerability is oh get the latest patch we can't do that with end-of-life operating systems because generally there are no patches also if we have application software running on there and we want to update that either for um vulnerabilities or functionality typically we start getting hindered by that right you know the vendor will say it's not supported on that older operating system you start having a little bit of a Frankenstein system that you have to manage so what are all these costs that we're trying to balance you know migrating to a new operating system and if you're running dozens or hundreds of systems that's a real cost paying for extended maintenance it's relatively cheap per unit but then again how many units you have to scale that out to I mentioned finding other controls there are a lot of notable examples of windows going end-of-life please don't boo his throw stuff at me because I said the W word but it is an operating system that we almost learn to live with right it's still 70 plus percent of the market interesting I was teaching a cyber camp for high school students last week and I went and brought up to metric real quick how many desktop systems or windows and I remember bringing up bringing up that chart when I first started teaching and windows still had 90 plus percent and now it's in the 70 so you know it shifted but it's still extensive out there so I was in a navy I mentioned that the navy's it 21 which was windows XP they maybe love to put 21 in front of you know in program names it 21 was using windows XP and they were nowhere near ready to migrate when the time came so they were paying Microsoft hundreds of thousands of dollars and for those of you who trust ATMs that are willing to use them they are a lot of them are windows based and they typically get caught up when you know the next windows iteration comes along in fact I was on a phone call just this week where they said the navy issued a warn order because we just can't say a notice and about windows 10 going end-of-life and having to move on to windows 11 I'm like okay when will the insanity stop so why did I pick Ubuntu it was a great case study for me one I was using it at work I don't talk really about what I'm doing at work because I didn't want to go through the whole process of getting that approved but it's what inspired it I think whether or not okay let's do that so the whole room raised a hand when I said who's running Linux at home how many people are running Ubuntu at home okay so we got a mix there we could have a religious war over Ubuntu distros here but it's still widely used and there are two versions of Ubuntu under what is called ESM extended security maintenance 1604 is the one that most recently went end-of-support that's what I use at my work and then 1404 a few years earlier in 2019 went end-of-support so we've got two great examples to look at and see how these systems are aging both with their ESM support and systems that don't have that support these slides will be you know I will put these slides up on the scale site so you'll eventually have them for the links now interestingly I spoke with a person from canonical on what I was trying to accomplish and he had written a paper more of a sales point of view on 1404 and said here's all the stuff we fixed under 1404 I am taking a different approach of here's what's broken in 1404 1604 because I'm not trying to sell you anything so how did I go about this I tried to find open findings in both 1604 and 1404 that were that had ESM extended security maintenance installed and receiving those updates and one that basically is frozen in time only has patches up until the software went end-of-support I used census I started off using show-down because that was like always my go-to for internet of things and one of my students helped me out and pointed me to census which honestly I had never used before and gave me much better results in identifying systems canonical the makers of Ubuntu were of little to no help because I should have thought that out before I said that on recording in telling me how many systems were out there using ESM they pointed me to a graph that just basically didn't help at all they just didn't really want to tell me how many people were buying and how many systems they thought were out there so basically I was just going out and trying to find 1604 systems a canonical rep did help me come up with a keyword search and it's in my paper that this is based on how to identify Ubuntu it is not 100% accurate like you can't just go touch an Ubuntu system on the internet says hey I'm 1404 you have to look for certain versions of packages and then from that discern whether or not it had ESM on it so it's a little bit dicey I could not perform vulnerability assessments on those systems out in the internet because it didn't belong to me and we all know we don't do that so right now remember it doesn't scan a system doesn't belong to you you can scan your friend you can scan your friend's nose but you can't scan it so the three flavors I did baseline system you go to the canonical website you pull down Ubuntu 1404 1604 don't apply any updates you just install it that's it then I took those systems cloned them and did an update and downloaded all available updates to when the software reached end of support and then it became frozen in time and then canonical allows you to license up to three systems for free with ESM so I installed ESM on a 1604 1404 and have kept those current so I did scans back in February and towards the end of this presentation I'll show you scans that I did this morning I like to say I did them this morning so you have most current data let's stick with that story also in my test environment just using virtual box to manage the VMs the vulnerability management tools I use so there we've got two VM acronyms Nessus Essentials which is free and will allow you to scan up to 16 hosts and that's free pretty much forever and then a little bit apples and oranges the insight VM again the VM stands for vulnerability management is a little upscale of a product I only got a 30-day license so it's more comparable to Nessus Professional but I don't think at the time that Nessus was giving me a Nessus Professional license for any length of time so I had to use a little bit slightly different tools and you'll see that you get different types of results from these or at least data to analyze and then I did the vulnerability assessments on each of the VMs that I described in a previous slide so between on the 1604 the three flavors no updates no ESM and with ESM which one's going to have the most vulnerabilities the first one okay yeah this wasn't a trick question I'm just trying to get audience participation here so starting from the left dark red is critical high medium low and then the blue are info and those aren't really vulnerabilities that's just details on the scan and the system so as you can see the baseline system is a hot mess with 21 critical but you say you know just we don't really need to look at the each number but just this is like a good management chart right if any of you have ever needed to communicate to management they like colors you know keep it simple and honestly right if you're not a technical person we could joke about it but you know if you're not a technical person and someone is saying hey look it's really bad and it's got a CVE number 9.0 and they're like what are you talking about but it's like red bad oh good I understand so we could see that as we get to the 16.04 ESM again these are the results they originally did back in February that the ESM system wasn't bad right you're always going to have a gap between your plugins being the most current and the the vendor being able to provide the update so that's not surprising at all also if you're okay another show of hands how many people have used some type of vulnerability assessment tool in their work have had to do this oh look at this preaching to the choir so you know you can't trusty things right out of the box right you can't just turn this report over go oh it's all red we need money and all that right you've got to analyze these things none of us none of us would ever do that right we would not use fear uncertainty in doubt to sell our products so Nessus uses a scale that they call the VPR the vulnerability priority rating it's basically you know various math equations of how to determine the badness of a particular vulnerability so I picked one of the criticals in here and happened to be a Samba finding we're going to go dig into that in a little bit but first 1404 findings so I've already quizzed you on between the baseline no ESM ESM between the 1604 and 1404 which system do you think it's going to have the most vulnerabilities okay so we're all over the map here nine yeah windows 10 so surprisingly and we'll have a chart that will put these together surprisingly the 1404 had less findings that follows the same trend that the baseline is worse than the no ESM is worse than the ESM but it didn't have as many findings and I do have a little bit of a theory on that if you're a Monty Python fan the theory does not involve a Brontosaurus but yeah I got some less on that one and so we'll we'll talk about that a little bit but again 1404 same general trend between the three and Elle can her theory on the Brontosaurus for those of you who know what the heck I'm talking about so Ness's essentials is very basic you know like I showed you on the previous slide right you know these are just screenshots of essentially what it gave me so rapid seven because I was using a higher scale product gives you really nice visualization right so this would be the nice kind of thing that you would want to potentially use in management communication and there's two particular ones I want to talk to and I'm going to show you closer on the next slide this one so like the management go well that looks bad well that's saying the percent of systems that are running obsolete operating systems so interestingly Ness's was able to determine that two of the systems had ESM and we're getting security updates so it called it out so that's good right so you don't have to deal with that finding and explain to an auditor it's like no no we're getting updates rapid seven was not doing that it was just saying they're all obsolete because it you know not the biggest problem but an example of of of slight differences between the products the other one I took some offense at personally and it's exploitable assets by skill level so okay so this is the for management look all obsolete bad then we've got the one on you know percent of exploitable by novice spoil alert I was not able to perform any of these exploits so apparently whatever's below novice knucklehead is is is where I am so I who knows what goes behind these ratings but I will let you know that I've had some of my pretty smart students help me out with this you know can you follow can you find something that is plate like we have found data that says here are the exploits you know there's information on that out there I don't have enough savvy to actually work any of the exploits yet I'm going to continue working on it because I would like to really be able to show because as you think about the things that we've shown so far and even though we took a little dig at management right I guess I too late to ask if there's any managers in here but well probably not because everyone raised their hand that they were using linux but the okay so I apologize if I if I offended any linux using managers there so the what was I going to do what was I going to say John oh yeah so the exploits you could see how we were building a story that you would want to demonstrate your management of we're finding these systems and if you're in anything that has compliance requirements HIPAA PCI you know I can't I'm sure we all could shout some other ones out you know the use of current operating systems is going to be like the first entry and if you're saying well look we're still on something older but we have extended security maintenance at least you get a little off of their radar depending on what their skills are but yes I've yes sir oh okay so yeah I get your point right the the expert cannot fit the expert cannot figure out the novice ones right yeah so it's a management chart yeah um and I didn't create it this this came from the tool right so um yes your point is well taken I know what what was the the Charlie Sheen movie in short okay Carlos you're counting how many movie references I make um the Charlie Sheen movie where no it wasn't Charlie Sheen it was um M. Night Shyamalan where the aliens come down to the farm signs right they came all the way across the galaxy gunner planet and they couldn't work a doorknob it's like how does this work I don't know okay Mary please keep me honest on time because I'm going to just go into my stand-up routine pretty soon so here's hopefully a good table for you so this is the vulnerability findings from my first round back in February top table is 1604 bottom table is 1404 and then going left to right baseline update to the time it went into support no ESM and then updates with the ESM so this gives you another visualization of you know again there was one critical and 1604 two critical and 1404 so it gives you an idea of where they were but again the surprise being that like the baseline was appeared to be better on 1404 my theory going back to Ms. Ann Elk for you Monty Python fans is that you know how like antivirus definitions have to change right what's going on out there in the world plugins change as well that go into these tools I suspect that maybe there aren't as many plugins in Nessus for 1404 and one of my supporting facts of that theorem is 1404 Nessus does not detect it has ESM right it flags as an obsolete operating system only detects the 1604 ESM so just a hunch but again another reason to try multiple tools depending on what operating systems you're using and seeing how your results vary all right so I had promised and I'd like to keep my promises so this is going back to that critical finding and again remember that for every finding you have you're going to have to spend some time analyzing so just to give you an example so if you've got one critical finding well maybe it's going to take you a half hour an hour to go dig through CVEs and packages and find out if you have you know eight or 21 and they're like we can't update it we've got to make it secure now you've got some you've got some substantial work to start documenting what is really there right because we can't just say oh it's got 21 it's bad right we you know if we can start finding false positives and get them out of the way if we could find mitigations you know or find out the things got web packages installed and it doesn't need to be a web server what can we remove you know that that term that we I always tell my students Mary knows this you know words to use to sound smart when you're at like the scale conference like so we need to reduce the attack surface right you know though all those cool little terms we like to throw around you know that starts coming to play here so here we are we're just going to look at our one critical and we will drill down into some of these packages but what it comes down to is that the the this one finding had three CVEs and and I I have to define CVE once a week to my manager and I'm not even making that up and he's like let me write that down and I'm like well write it down the same place as last time we won't have this discussion again so and I was saying the wrong thing for the longest time I mean if you've ever okay how many people have been to Defcon good number of you right in a drunk hacker jeopardy which I really want to do but I don't drink and I I know the guy who runs it so I'm going to ask him one year if I could have a designated drinker you're like I get it wrong my friend drinks but they'll ask things like what is CVE is there anyone does everyone here think they could tell me what CVE stands for you'll drink for me okay I've got a I've got a new friend I notice you've got a drink in your hand you're leading against the wall so you're off to a good start so CVE and I used to think it was common or I used to like think it was computer vulnerabilities exploits right that sounds like a good mean it's common vulnerabilities and exposures it's like the worst acronym for what it is actually documented but CVE is tracking all the vulnerabilities are out there so there are actually three CVEs associated with this one finding so now wait there's more so we have to go dig into that and if you go look into that it will say okay this is the package version that needs to be on there so just to give that a little more do you I feel like a optometrist better now worse now better now okay other eye so we've got these three CVEs and I found out that one of them says okay or not one of them but all of them are resolved if you have the right package in there now I don't know why Nessus didn't determine that package was in there one of the things and in my in my day job I have this has been my life right going and especially before Nessus started detecting ESM packages and it wasn't even Nessus' fault so it turns out that when canonical provides a list of updated packages they were not giving the names that included ESM out to the world in the oval format that all the vulnerability management tools use to import make their plug-in so it wasn't even Nessus' fault it was turns out was canonical sometimes it wasn't finding tools with package names with ESM in it so I had to do a lot of manual reviews and this is an example the package is actually there it's a false positive and you could have the warm fuzzies that in fact that did it's funny like so I actually had an anxiety dream about this presentation the other night good news for you I am wearing all my clothes and right thumbs up car all is everything good and then I actually had it because this guy just walked out and reminded me it's like people just like totally walked out of my conference and like said this is like a total waste of time and I'm like oh look so far so good thanks for staying all right so that was my results from February time check please oh seriously oh my god so much people need to shut me out all right so we really are close to the end so five months later this morning I fired up everything updated the plug-in said let's look at the scans today so one more round of of challenge the audience how many additional security updates do you think there are in 1604 shout out numbers 15 okay for those of you no numbers okay let's just go to the surprise there were 63 security updates in 1404 and 124 in in 1604 oh important thing I neglected to mention earlier extended security maintenance only provides updates for critical and highs if they feel generous they might do a medium but mediums and lows are not going to be addressed for the most part so it's only highs and criticals so you could see that there's substantial number of updates since that time so I did my scans again and created another graph so now in this case this is only 1404 this is 1404 in February this is 1404 in July so you could see the general degradation if they're not receiving updates and then even a little bit worse receiving ESM updates because it's not their priority to do ESM updates they even say that that's like back burner of of you know supporting the current operating systems and then next slide is the same thing for 16 and 16 is still nice and shiny for you firefly fans right so nice and shiny so ESM's looking good the non ESM system is slowly becoming a dumpster fire all right so quick analysis ESM addresses your criticals and highs and is effective at doing that even with the ESM updates 1404 is not aging well and then as I show you've got to you've got to review the results you can't just trust right you because we found out for that one ESM system with a false positive but I do think that you could use these features or use a presentation such as this if you have to convince management that there is a reason for paying for this for this maintenance and then I don't know what the difference is between my final analysis in summary but like I said you could use this type of process to do this cost-benefit analysis with your with your management and then do this at regular intervals right so I did it five months later and I saw a definite trend in both 1604 and 1404 and then what I would like to do next not right now with all of you but what I like to do next with this is do a more thorough analysis with with rapid seven since I was getting rapid seven was finding more 1404 findings than necessary so I'd like to dig into that a little more there's my contact information I also teach at Long Beach City College so you can always find me there so thank you so much for time I do want to do a shameless plug for two people who are here Adam and Marie my friends children and they were giving a presentation 1230 tomorrow quit plug what is the topic customizing VM for use in cyber defense competitions right so if you got any kids who are doing like cyber patriot national cyber league anything like that go see them and make them really nervous and stare at them thank you so much for your time I will hang out here if anyone's got any questions or we want to talk about old technology and I hope enjoy the rest of the conference and stay safe you went there you ran away old chicken whoa sorry sorry let me see if that works that's okay right you can hear okay cool kill y'all because I know it's gonna kill y'all and I I pop my peas too so okay I think oh yeah I was supposed to say person in the AV room I'm very very sorry please pan back because the chances of me standing still are probably like negative 100 so I mean I can tell you now I'm not going to so please just well I got told that they have to control it and like try to follow them and no it's gonna be fun guys it's gonna be fun or y'all I'm gonna go with y'all you have to be gender neutral y'all y'alls and y'alls the proper way to conjugate all that I don't have my notes up anymore I use my notes don't judge me yes for those that came in we know it doesn't quite fill the thing but you should have been here trying to get us to figure out how to put that on the screen so we're just gonna pretend just cut the left side off in your brains now that's all they're gonna focus on I don't have to worry about you judging my talk I think we're about time right I've killed six minutes in 43 seconds according to this alright last person okay so thank you all for coming to my talk APT's Transition to Linux I know that's like the general way you're supposed to greet people like thank you for coming to my talk but I really mean it first of all the fact that you showed up so I'm not talking to an empty room that means a lot but two years ago just with everything that happened I had to leave the Linux world and it was a very hard decision for me to make but at some point it's feed your kids or do what you love and you're gonna go with feed your kids hopefully and so being here and getting to present in front of all of you is like it's a homecoming to me and so I'm so excited but a lot of you probably don't know me except for my friends who are kind enough to come to the front row so my name is El Marquez and I'm a senior technical trainer and as well as a researcher over at a company called Grim I'm also nervous to just go with it I normally that's all I would say and I would walk away from this slide but I think what I do is really important for you to understand the research that you're gonna be seeing today and that's because I get to take my trainer hat off all sales and just put it away and then I get to go to a dark room at my house and hit the white papers and just find a rabbit hole that interests me and chase that rabbit till it comes to an end and dig into another one and as long as I'm passionate about what I'm learning they don't care just come up with something so my entire job is to go do this tie it up in a nice little bow come over here and present to you to give you actionable steps that you can take into your company and do tomorrow even if you haven't had the time to do the research on that I want to give the biggest thank yous to two individuals Nicole Fishman they've worked with the IDF for over six years the Israeli Defense Force as well as working on getting new Linux malware variants and new undetected malware just as a whole specifically her you know thing that she loves is Linux so Miko Miko is an interesting thing I as you can tell already I'm really nervous because this is my research and it's one of the first times that I've gotten to present it also along the way I was spitballing of people you know you do peer reviews you talk to people and I had a lot of people tell me I was going down the wrong rabbit hole that I was wrong that I wasn't going to find the findings that I wanted and I was terrified because I already committed to doing this talk and so finally Miko is the F secure head researcher he's famous in his own right go Google him he's pretty you know big up there I didn't know him but I knew he was good so I just messaged him and much to everybody freaking out with Miko I need your help this is the research that I've done could you please review it and he took the time and he talked to me he gave me some of his research so the reason I take the time to say all this is thank you because I'm not standing out here by myself right now I'm standing on the shoulders of giants and I really hope they listen to it okay not going to get emotional okay so today what I'm going to do is I'm going to address an elephant in the room and I truly hope that that elephant isn't in here seriously I really do but it hasn't happened yet because the talk that I think you're expecting to get is not the talk that you're going to get and you may disagree with me and everybody says El don't say this in a presentation but I am not a standard presenter if you disagree with me that's fine that is the beauty of research that you can establish your own beliefs and take into your company what it is that you need so even if you don't agree with my premise please stay and listen to the research itself because it's going to help you I'm going to do this in a form as a story and this is where I start talking I'm going to cover talking walking this is where I'm going to take you on a trip to the past because I firmly believe that if you don't under your span your past even in technology there is absolutely no way you can defend in the future then we're going to go to the present because if you don't understand what's going on right now you will not build a secure a secure path to the future on that I'm going to I'm going to drop some truth bombs I'm going to be not completely honest with you all because if I'm holding back my punches I'm not helping you at all so at some points I'm going to call out the flaws in Linux and you know what get over it we all know that it's not secure by default and if that's shocking to you please talk to me after this talk because we have some serious conversations we need to have with that being said we're going to go to poll the audience please please please be interactive where this presentation is going to be really boring so who here believes that a nation state is what an apt is well all right first time I haven't had anything okay who here believes that organized crime should be brought into the definition of an apt organized crime and a nation state don't just not raise your hand thank you and who here believes that a hacktivist should actually brought that into it so nation state organized crime and hacktivist so I had like 90 percent of you not know what it is but that's okay we're going to be working together all right so I understand why people believe that it's a nation state and actually that was the original definition it was brought on by militaries referring to other militaries as the nation state type of attack I worked at a company before where 99.1 percent and I'm making up that statistic but that's what it felt like came from the Israeli army and I had a few I'm gonna go with heated discussions on this topic and they were firm in their belief that I was 100 percent wrong that nation states were the only thing that could truly be an apt so then I went and talked to Miko about this and he said actually 98 percent of the malware that's analyzed by fsecure and they were one of the biggest researchers when it comes to malware is actually attributed to organized crime and more and more we're seeing hacktivists be successful in their attacks because they're able to bring in advanced tactics that we normally didn't think that they had but this is where I differ and this is where I go to the talk that probably all of you didn't know you wanted to attend and that's what I disagree with every single one of those definitions of what an apt is I think what those groups are are their cyber crime unicorns and I stole this from Miko I'm gonna admit it but I think it was adorable so we're going with it and what it is is these groups are people who have been able to receive funding in order to attack in an advanced way I mean we've got nation states of course they've got funding they take it from their citizens and whether their citizens know or not it goes to fund military operations those military operations are jobs they have a nine to five probably longer in which they develop malware in which they get a way to be able to attack a system in a way that we haven't expected before but we also have organized crime and yes organized crime sometimes gets their funding from nation states but other times they can start small I do a little campaign get a few hundred dollars they are businesses they reinvest their capital into their business they attack again get successful reinvent their you know tactics and they just keep doing that to the point where they can hire their own attackers their own malware developers and continue to include their code base to get stronger and stronger in a way that you know what we don't reinvest in our companies as well as they do and that brings me to what I believe the definition of an APT is it's exactly what it's meant to be APT stands for advanced persistent threats and APT is any attack regardless of who has launched it that is persistent and by that I don't mean they keep going after you I mean it could be that but it's also that they've established persistence in your system as well as a viable threat and the reason that I say viable threat is too many times we say oh cozy bear or fuzzy bear or whatever bear Russian APT APT that you want to talk about they're the threat why? why are they a threat? they can exist all they want to if they're just sitting on the sidelines going we've made enough money we're good it's not till they launch that attack it's not till they take actual action that they become an APT and I think unless we begin to understand that and understand that the threat itself is the unauthorized code or the malware that occurs we cannot even begin to defend against true attacks so in order to show that I want to go to a journey in the past and this is where I start just kind of being honest y'all most of us and I'm Texan so y'all most of us are not Linux enthusiast we're Linux zealots we see Linux as this different beast we're not like windows we're different we're open source people can look at our code we're a community we're strong together cool great attacks are still working good job on the community we have to be realistic to what we're doing as a community we need to accept that Linux is vulnerable and that these attacks are going to continue to happen how can I prove that well I'm going to take us back to 1971 with the advent of pushing a button with the advent of the creeper worm now the creeper worm is actually just it's known throughout the community to be the first piece of malware to ever attack a system and I'm going to read this to you so I make sure I get it right it specifically targeted the DEC PDP-10 mainframe computer running 10x operating system using ARPnet and spread through floppy disks there you go there's all the English you needed to get there the reason that I took the time to read that is think about how specialized that worm was why was it that specialized because there was a community that believed that that inframe was secure by default so somebody created a piece of malware not to be malicious but to prove to them that it wasn't so walk with me a little bit further up into history to 1986 with Brain.a Brain.a is recognized as the first virus that attacked an IBM PC the first PC that we had and the thing is it was a proof of concept virus people were like oh I've got a PC these things are so secure and that really was a concept that we had so two brothers Ahmad Farooq Al-Aviv and Bastille Farooq Al-Aviv wrote a proof of concept virus to prove to people that PC was definitely not secure now how can I give that attribution I wasn't giving the attribution because I spoke to another researcher frankly I didn't have to do much research oops all I had to do is click the right slide all I had to do was look at the code itself they gave themselves attribution this wasn't meant to be malicious all they wanted to prove is a security flaws funny story though okay so 25 years after Miko found this piece of malware he goes I really want to meet the people who wrote this like wouldn't that be cool if I did so I believe Infosec is six degrees of separation and really that's how I found Miko right you know somebody who knows somebody and so he put it out there and two days later the brothers contact him back and say it would be great to meet you here's our address that's the address they gave him they still live there he could have shown up so what do you do with that right you jump on a plane and you go to Pakistan so I'd introduce you to the two brothers who wrote the first piece of malware that infected a PC so at this point I've walked you through and you understand that mainframes were vulnerable obviously right we're like duh history has taught us that PCs are vulnerable and I think a lot of us in this room have made that judgment already so why in the world did we ever have the belief that Linux was secure by default and if you're sitting going I never believed that you know bull those people in front of us those people who came before us that was a real opinion that they had if you never believed that it's because you were never told it and you were in this industry after that mindset so I'm just going to call you all on that and me too I was told that to begin with and I truly believed it so that brings us to 1986 in stag now I'm going to probably say it wrong stag stag stag whatever it was if anybody knows please tell me but it is believed to be the first Linux virus to ever actually be out there to ever be found and I don't understand why we didn't see it coming like we saw it in the mainframe we saw it in windows like duh we should have been prepared for this our defenses should have already been ready because guess what it was just like the other two it was a worm now here are some strings that we're able to pull out of the code and the one that I underlined it is my favorite because it did a better job of system administration than we do it went hmm are you up to date oh no you're an older version of me let me update you and if it wasn't infected it went ahead and infected it and it was actually you know what was I going to say it was an actually interesting story okay so on February 5th the creator of stag came out and he was like y'all I think I messed up on a mailing list he's like this thing has gotten out of hand it has gone further than I ever thought it was going to my bad that night McAfee came out and he was y'all can't believe what I just found I just discovered a brand new piece of variant called stag so I mean I guess if you read a mailing list and you find something new technically you discovered it I always find that and I always find John McAfee's stories interesting so no funny there those who know him next takes me to 2001 so 2001 we had a brand new piece of malware called by 2001 Linux had become a big target a fast growing target for virus developers because we still didn't have the security in place that we needed to ramen ramen targeted specifically red hat systems running web servers now why do I running as a web server why do I bring that up because historically we should have already known as we walk through this that malware was going to be written in such a way that it came to a specific target it was a fun piece of malware didn't do too much it went and defaced the site and said hackers love noodles and let me tell you as a hacker we do love noodles this was so one of the notable and I have to say this just because of Texas two of the notable victims were NASA and Houston and Texas A&M right down the road so you know just saying I'm not an Aggie just in case anyone wondering where that came from all right so it was one of the first pieces of friendly malware I love friendly malware I love malware it's a wrong thing I hate what malware does but I love malware itself like I'm just going to own that but what it did is once it went in and it defaced the website and let me come on is it really that bad you could bring your website it's not that bad of a virus but then it would go and it would patch the way that the exploit that allowed it to come in it's a nice little virus but so now I've gone through let's see we've gone through the mainframe we've gone through the PC we've gone through Linux what's next where am I going to go cross-platform malware we've walked this road how the hell and I'm going to go with that sorry people if you have to blur that out how the hell were we not ready for this how did we not think that this is where we were going and cross-platform malware comes from the fact that we were all lazy straight up if you're a Linux person you're lazy you will spend two hours automating something that you could have fixed in two seconds that's just the way we are we don't say lazy we say clever ingenuity whatever we're lazy so we're developers if you're a developer in this room I'm sorry but my friend Corey says something he says there are front end developers and there are back end developers and then the rest of us who are stack overflow developers I love him well why are we going to take the time to write a new code base when it's right there just pick it up shift it put it in it's going to work why am I going to take the time to tailor Linux malware tailor what macOS malware tailor windows malware when you know just take the code put it on each one of them and it works and it makes sense so bad bunny was a ransomware a ransomware worm there you go and like I said it attacked all variants here's the part that I find interesting and the part that honestly makes me a little mad I'm an emotional presenter and that's that bad bunny was discovered in 2017 and was attributed by a malware bites to likely have been created by the authors of petia and non petia if you don't know about these two pieces of malware write them freaking down right now and as soon as you walk out of that room go and read about them these were critical pieces of ransomware that brought countries down to their knees we watched this happen we watched the devastation happen to it we watched this ransomware tear through entire civilizations why the hell weren't we ready for it when it came to Linux did we think that we were that special because petia targeted critical infrastructure for countries guess what bad bunny did bad bunny targeted nation states targeting their critical infrastructure having the exact same results as petia and not petia yet Linux was special and we weren't ready told you guys we're going to drop some truth bombs today that brings me down to IP storm we're getting closer to modern day we're getting closer to the present I promise IP storm though it says on there 2020 let's build a story was actually found in 2019 as a windows piece of malware targeting the inter let me get this here the interplanetary file system as a peer to peer network I always it's the tongue twister for me and so it was named IP storm by the researchers by 2020 my friend Nicole she's my bestie now and have a guy I'll mentor I call her Abby actually found a brand new variant of IP storm so one year later they found a brand new variant of that exact same malware using a lot of the same code base running on Linux a month later they found it on macOS because it was a windows virus how would we live that's windows that I mean that doesn't matter in the Linux world right they we all know they have viruses y'all they lift and shifted that virus right over to us that should terrify us the fact that we knew the code was out there that we knew what it did we knew its signature we knew it as an anomaly and it still worked on our systems and one of the things that I got so when I worked at Inezer which is a company that they're at I got so tired of saying this it was to the point where I said it so much at a conference that I just rolled my eyes so when IP storm was found it had 0% detection on virus total and so has maybe 99% of the of the Linux malware that I've talked about it's not working our security practices are not working put it out there we need to accept that it's not working and we need to start from that point when we're building our defenses maybe meaning that we should pay attention to the history now I've actually built in a water break for myself because as y'all can tell I talk a lot and I talk really fast so I wanted to put this up there and maybe I won't I can kind of see it I can share my slides later on because it shows how many of the shared features happened between two variants y'all give me a minute please I'm shaking all right so I will share my slides so you can see this but when I say they lift and shifted code this was written in GoLang which is a cross-platform piece of uh Pillar of the Crabless Mower yeah a scripting language so literally the code was right there it's nothing new it's nothing we shouldn't have seen so that brings me to my present and I've talked a lot about Windows and everything else and hopefully y'all understand what it is that I'm saying when it comes to an APT because at a certain point maybe these attacks aren't even that advanced they're advanced because we allow them to be because we don't put the right uh right security in place so let's go into why I decided to talk about this to Linux because I do think it's important and I think it's something that we don't think about though I'll cover some of the basis that we do why Linux did you all know that China has its own version of Linux how about the Turkish government that the Russian government uses Linux that should be really big in the news right now because you're going to see that in the cyber attacks that are happening did you know in 20 in 2001 the White House moved over to Linux did you know that the Austrian capital uses Linux that in 2004 the Venezuela government moved over to Linux why Linux if you believe that a nation state is an APT that's why Linux because an APT is going to be used from a nation state against another nation state and how many of you think that the White House sat down with everybody and offered proper Linux security training before transitioning into it we'll go with probably not so they're already going to know where our defenses are going to be weak and they're going to go after them why Linux the cloud and yeah everybody's probably rolling their eyes okay oh you can skip this slide we all know hey what are Linux servers but what is the cloud made out of Linux servers we all know that joke I think this is a huge issue that we're not paying attention to and this is going to be one of the primary reasons that APTs are going to continue to grow and be successful because how are we transitioning to the cloud as quickly as humanly possible that's where the money is we don't need to worry about security in the cloud y'all AWS is going to do that for us we can just I mean after all we're responsible for what's in the cloud they're responsible for the cloud itself I call that and I stole this from somebody but I don't remember who the cloud a responsibility model because it's a joke we tell developers hey go use serverless technology use lambda use azure functions and security's just built in and that they're doing it both of those have officially been breached and go look up the article called we hacked azure functions by Inezer you know the advanced technology that we used to hack that netcat one command so yeah the cloud is secure by default I can only imagine what not friendly researchers could have done with that it's terrifying to me and we're transitioning without training one of the a lot of things here make me mad I've spent time speaking to hundreds and hundreds of developers and hundreds and hundreds of Linux and security individuals what makes me mad is their jobs are going away because they're being outsourced to the cloud more and more security positions are being left not filled because the cloud provider can do that for them how much training do you think the developers have received in Linux security we're just told hey make sure that you make more secure code that's literally the training that they get so now we have to provide them with training with Linux by the way they also need to be trained in the cloud it doesn't matter that the cloud provider is running Linux because half the time you're pushing buttons on there according to Gardner and Sands 70% of vulnerabilities occur on the application level so developers in the room might want to get on that you know write some more secure code I know that's what you all are told and 80% of attacks are successful in the cloud because of misconfigurations why Linux because it's easy to breach now let's go over to why Linux smart cars we all have smart cars right so grim has a division that is dedicated to smart cars and y'all I don't want to say their jobs are easy like I'm sure they work really hard but how easily they complete a task shocks me I want to go work in that division because I can have results it's not like writing training where it takes me a few months give me an hour I'll have something on a smart car it's just crazy so much so that the federal bureau of investigation came out and said hey smart car builders could you do us a favor and I really wish this didn't read this way but could you do us a favor could you build your cars with cybersecurity in mind the fact that that had to be asked by the federal government just freaks me out one of the things that's there has to be absolutely no in-person contact with the car in fact a researcher in Germany recently was able to breach 25 different Teslas around the world without ever even seeing them that should scare you because do you know how many cars that you might not even think of they're not Teslas but your car has some smart functions who's using you know the in-tune audio system or any of that none of that is secure right I've got it too I wanted to figure out how to unplug it because I can promise you it's not going through any type of secure tunneling I've used the app neither is that app developer sorry I just met a new friend who's an app developer he told me horror stories so why Linux smartphones and once again this might be something like yeah okay L Android can be hit by viruses we know that just move on okay if you guys are going to accept that that's great Android is inherently really insecure but what do we tell hey mom dad grandma why don't you use this iPhone iPhones are more secure right it's going to cringe they're secured by default Apple does a great job of vetting what's in there yeah secure by default should be scaring you at this point but let me introduce you to Pegasus Pegasus was created by the NSO group I trip up when I talk about the NSO group because there are certain parties in organizations who have deemed them to be a threat group they themselves do not believe they are a threat group they believe that they are a malware as a service company providing a service to help being able to promote diplomacy and yeah diplomacy throughout the world they want to ensure that governments can ensure the safety of their people do your own research I'm here as a researcher not someone to cast votes on what's occurring so please do your own research make up your own mind but they've been able to develop Pegasus which can go into an iPhone and break its encryption and make everything clear text this isn't an oh it's a proof of concept this is occurring and has happened so much so that the federal bureau of and that's actually just need to start saying the FBI hey acronyms but the FBI has actually gone in and purchased a license for Pegasus you don't purchase Pegasus you purchase a license and if the NSO believes that you're doing something bad they'll revoke that license from you so once again you decide but they actually it is believed when it came out that they were going to use this for domestic surveillance is that good or bad well you know for domestic surveillance the Mexican government recently used it to capture a chapel one of the world's biggest drug lords so if you don't know much about Pegasus please go and research it it's really interesting the reason that I bring this up even though it's on iPhones is why are we still in the mindset right now that something is secure like iPhones are not secure by default just because we don't know what's out there doesn't mean it's not out there we didn't know about Pegasus for a while and why am I bringing it up because what part of us doesn't think this is going to lift and shift the style of attack how strong this attack is it's going to lift and shift to move over to Linux yeah I'm using Signal it's all encrypted right I'm using Telegram it's all encrypted until the next variant of Maurer comes out and proves you wrong so why Linux I'm being so negative today the cloud we've covered it the cars we've clouded it smartphones we've covered it did you know that Chromebooks are actually outselling MacBooks at this point SpaceX yay we're sending Linux to space we're already out there miss people didn't know Mars rover by the way you know it seems itself happy birthday poor little robot it was running Linux IoT devices I almost left this off the slide because I have an eight hour workshop on the flaws and vulnerabilities inside of IoT devices I could give a three hour talk without talking at this pacing and not stopping about how horrendous security isn't it but then you would all wonder like why didn't you include IoT devices did you not know about that if you don't know the inherent risks of IoT devices I recommend a book that Miko wrote called if it's smart it's vulnerable or just go in and type in IoT security and be prepared to take every smart device out of your house I wish I was kidding if you want any advice on how to move that to its own VLAN not related to this talk please talk to me it needs to be segregated especially if you have kids okay off my high horse there so why Linux I can actually bring this down to one sentence and I'm stealing it from Billy Sutton who was a very well-renowned well-known bank robber so when he was asked why rob banks because that's where the money is why rob Linux because that's where the money is now some of you did vote that organized crime could be a form of an APT so let's talk about team TNT now TMTNT was officially announced as a threat group by MITRE recently FYI I called them an APT long before that you know just putting it out there that's my street cred team TNT is a threat group that's primarily targeted cloud and Linux environments specifically containerized environments a bit of transparency here we're Twitter friends I always say we're friends but I have to specify we're Twitter friends and the way that this came about is they are very open on Twitter you have a question you've done research they'll tell you and a few researchers attributed research to them that wasn't it and they straight up called them out like you guys are wrong you need to do your jobs better so they recently announced that they were going to go under like they're it was too close Interpol was too close they're done they're going and I said wait wait wait before you guys leave do me a favor I don't want to mis-intribute you could you read my research and tell me you know if it's good worst they could have said it well actually worst they could have done is a lot of stuff but the worst in my mind at that time was they could have said no but they read my research and they came back to me and they posed this interesting question to me and I want to know what you all would say with the mindset of a researcher they asked me do you believe that we're criminals and I said well I don't know how to answer that I don't know what you mean they said the only thing that we've done is gone into an environment that was configured to accept connections that was configured and expecting to concept connections from the outside world what were they saying the only thing that they'd attacked was misconfigured Docker containers is it a crime to go replace they're already accepting people to come in now I'm not here to talk about what they did after to judge I'm here as a researcher and I understand what they're saying and I'm here to present to you what they talk to me about we're such good friends y'all that they believe in my it's okay to be new campaign I've got stickers too if anybody wants it hopefully they don't mean it's okay to be a new crime group but you know whatever I just thought I was special and they retweeted me so let's talk about team TNT now they've been active since 2019 attacking redis servers I call them a cloud and linux native attacker I think that's awesome they started they were born in the cloud they were raised in the cloud they've attacked in the cloud I will go back and forth saying what they do and what they did because they said they were going under and then they did some more things so whether they're retired or not we're just gonna have to keep watching to see interesting story and just the second on what they came back to do so I should have sorry just ignore doci that was something else I was gonna go off on a tangent but decided not to but they use actively tried and trusted compiled binaries what do I mean by that I mean we should know already what they're doing they go out there and they find open source malware that's available on github and they steal some of it and they put it into their code and they make it work stuff that's been out there what Mariah was what 2016 2006 that it was just completely just put on github that code still works and is able to bypass our detection on linux by the way like we need to pay closer attention and how do I know because you'll go and I'm gonna skip one for a second here team TNT is an open source contributor they're an open source attacker they take the scripts they bring it and they contribute it back to the community do it we tell how great open sources so they tout open source as well they also take windows-based tools such as mini cats stuff that we mini cats was one of the things that I when I got into security I learned to use it what like three weeks in it's not that powerful of a tool except for the fact that it is the reason I say it's not that powerful tool is because we should be prepared for it we should know how it works so what happens they relabel it mini pie make it work on linux and suddenly it's a great tool again y'all this is getting a little bit ridiculous if you can go with me here now where I was gonna go is I told you we talked a few times and I've told you I have favorite kinds of malware my favorite favorite malware in the whole wide world that holds a piece of my heart is lemon duck team TNT recently came back attacking containers with lemon duck now I didn't tell them this so I really don't think I had any place to play in it but when I first saw it I went oh hell can I talk to our legal department real quick but now I had no part in it but the reason that lemon duck is my favorite piece of malware is y'all he is the best system administrator you could ever have he is so much better than 90% of assist admins that I know because he goes to an environment he cleans house he goes is there any other crypto mining software on here you're out I might clean you up I'm gonna get you out of here and then how did they get in what did they exploit all of that's gonna be saved how are your resources going ah you know what we can fix this a little better make things go a little better are we updated let's update this stuff too he does such a good job and all he wants out of that is to be able to do some crypto mining look he wants to get paid so do the rest of us hopefully y'all know I'm being sarcastic but he is a great little piece of malware I actually have heard stories of systems that were actually attacked by malware such as lemon duck that it did such a great job that they decided to allow the attackers to stay in the environment there's a dark net diaries I forget the name of a windmill form that was attacked but the windmills got such great ability to be able to pass power after they were attacked that they just went all right your hired is our sis admins and let it stay so you do with that what you're gonna do now lemon duck is continuously being updated it's been around for a while since 2019 the reason that it's one of my favorites is that every single time I swear that I see a new variant it has new abilities to it it's still working 2019 all they've done is add new abilities but guess what they add a new ability they go past our anomaly based detection they add a new ability they've changed some code they bypass our signature based detection like I said zero percent detection on virus total now as a part of it was originally a bit it was originally a cryptocurrency miner and then it went to a botnet and now when I wrote it down it now steals and removes security controls it has a stronger C2 it's a command and control server it has learned my little duck has learned how to go and protect other systems you know make sure that all the network is safe good little duck and it ultimately has been human friendly it goes and it drops tools that other people can come in and make sure the system is secure it's insane but that's actually what's happening they will go in and fix your resources so they can get better crypto mining going on lemon duck currently works on windows linux and has recently been seeing transferring to edge things such as wi-fi routers I've been droning on and on so now let's go to the future what is the future why I'm excited to announce to you that the future is now lemon cat I could do not so as mower is developed like I said why would we reinvent the wheel people are taking code out of lemon duck and they've attributed now they've changed lemon duck as the variable and named it lemon cat and that's how they're doing it to bring in new mower why is this effective I just like literally go and make sure that your systems are protected maybe the best way to see if your system is hacked or sorry breached is if it's running better than it ever has before and I wish I was joking so start looking for things like would you ever have thought to monitor for that like it sounds ridiculous but that's what attackers are doing that is the future they want to hide they want to establish persistence in a way that you might not even mind things that you're not going to be actually looking for so lemon cat is our new nice kitty now I had a lot of different things that I could talk about and I really wanted to boil it down to what I thought would really be impactful what do I see the future being ransomware without a doubt how do I know that because I pay attention to what's happening in the windows world I don't have a question about the fact that this is our that this is our future now I never assume that anyone knows everything because the Lord knows that I don't so I'm going to go over different types of ransomware really quickly first of all there's crypto ransomware this is what a lot of us on the Linux side are familiar with where they go in and they lock down a system and they say we're not going to give you your information back unless you pay us this amount of money we'll give you the keys you can get your information back lockers are what most people are attributed know about so you know in the movie where the good guy goes and he sits down and he opens his computer and there's this huge skull and crossbones with you know all this pay us this amount of money you'll never see your dad again that's what lockerware is doxware is one of the things that scares me the most my partner and I have both been targets of doxing to the point where I had to move and he has more security cameras than I had ever cared to admit doxware is terrifying and that's where they either find something about you they lock down your information they lock down your PC and they say if you don't pay we are going to leak your personal information this could be anything where you live where your children go to school what your pick call your parents step and start giving them information about your online activities even videos that they've managed to get it's a very scary real thing so I don't think that our future is going to just be Linux servers in the cloud Linux boxes and the devices that you're using right now are going to be targets of Linux ransomware we have leakware which is very similar to doxware except what is the number one thing to somebody please tell me the number one thing that we tell companies to do to help prevent successful ransomware attacks somebody's gotta know come on train users what else backups right definitely backups backups backups did you say three so I said normal one offline and you say off vacation the off location sorry it was like what it's on vacation so it doesn't get updated okay guess what I work in the security field I do I know exactly what they're going to give you should I ever decide to be a little more friendly with my lemon duck guess what I'm going to know that you do first of all I'm going to go after your backups which is why they say you know one that's offline but second of all before I do my ransomware I'm going to expiltrate your information and it's gotten to the point where we are like okay yeah they're going to release my information whatever I mean how many times says my password and my social security number been out there I'll just change my password it'll be fine I hate that that's where we've gotten to I mean how many of you have been hit by the target breach about four times at this point but what they're doing is imagine that you're a business and you are living on proprietary information you have a brand new product that's going to come out that's going to make you millions if that is leaked to your computer to competitor you're done you might as well have paid the ransom because there is no money coming into you it is terrifying now we have a ransomware as a service I'm actually going to come back to ransomware as a service because I'm going to geek out in it in just a minute but what I would say with all of this now that I've explained all of it how the hell were we surprised that this was going to happen because let me take you back to 1989 with the first piece of ransomware wasn't that exciting like it's been around for a while and so the first piece of ransomware actually went to 1989 it was called the AIDS Trojan it was created by a biologist named Joseph Pop and what he did he took 20,000 floppy disks infected them with the AIDS virus I'm sorry the what the funny part is getting to a moment so I just know so he infected it with this virus and handed it out at the World Health Organizations AIDS Conference and with the disk and I'm going to make sure to read this the disk had a little label that was called AIDS information and introductory diskette but it came with the pamphlet and on that pamphlet it said that this would adversely affect other applications on your system it also said that you will owe compensation and possible damages to the PC Cyborg corporation and that your computer would stop functioning as normal the AIDS virus spread even though having all of this information in the disk I think he was looking to make a hacktivist point here to my point that hacktivist organizations can be successful now like I said I'll make this available I know it might be hard to see but what it said is it was they have good customer service y'all I mean all you needed to do was $189 sent to this PO box in Panama and he would help you get your files back and look if you had more than to why would device infected he wasn't going to make you pay $189 for every single one of them just send a one-time payment of $378 and we've got you covered that brings me to my predictions and when I'm going to get to geek out over ransomware as a service my prediction is that threat groups and ransomware as a service groups are going to become more user friendly so let me give you a true story except I'm going to use myself as the example let's say that I'm an elderly person and I use my computer to stay up to date with my grandchildren I have so many amazing pictures of my grandkids on here then I let my grandkids use the computer and it gets locked down by ransomware what am I going to do I don't understand how these things work there's a 1-800 number here I can call so I'm going to call the way to 100 number and the person and I'm like oh my god my grandchildren's pictures follow calm down ma'am it's okay what you need to do in order to get your files back is follow up how this but what is bitcoin don't worry we're going to walk you through this and they walk you through the way to be able to purchase bitcoin to send it to them and if you say okay I've sent you the money but I don't know what to do now it's okay ma'am I'm going to stay on the line and I'm going to help you go through and unlock all the files that you need this is a true story this actually happens and why is it such a big deal because if you already know that this threat group will make good on giving you back your information are you not more likely to pay that ransom one of the coolest parts yes we're going to get out together on bad things they're also user friendly when it comes to other attackers okay so imagine that you're a baby criminal organization and you don't know how to get started but you know you want to be a bad guy and you're going like okay how do I get started I can make some money out of this but I don't have the development power to do it I'm going to go higher ransomware as a service well I still don't really know how to do this it's okay they have a GUI so you go in you buy you can log in and you look and you're like oh what kind of ransomware do I want okay I have the information that I want for my attacker so I'm going to choose this one put in my information the ransomware group says okay you know what and I'm making up numbers here because it just depends on the organization okay we're going to help you through it you're going to provide the information you're the one taking on the responsibility for doing the attack all we're doing is providing a tool that is for research so you go in you do your attack you're going to get 80% we'll get 20% or maybe the other way around they didn't do anything to provide you with service so you're going to pay them some money and what do you do you're a good business you're going to reinvest you're going to come back I don't know if you can get it on Google I don't remember how I found it but go and try to find some of the documentation on this stuff their documentation on how to use ransomware as a service is better than 99.999% of open source documentation than I have seen that's awesome all right so we're going to continue to see criminal unicorns continue to grow reinvesting in money we're going to continue to see code reuse why not I think I've explained that through the whole thing we're going to increase to see we're going to see an increase in cross platform malware because we're going to continue see growth in scripting languages like golang we're going to continue to see attacks on managed service providers now I have another talk on that but I wanted to add this in here because imagine if I got control of AWS imagine if I just got control of one hypervisor on AWS imagine if I got control of a managed security provider imagine what havoc I can read guess what is happening and it's being effective and it has happened I stood on the stage at GERCON I like showing this and I said imagine if the NSO group or I'm sorry imagine if the group that went in and did and did log for J and did all these other things that I'm blanking on right now because I know I'm running out of time went and actually went for AWS as a whole you know what happened a month later that same attack group went and hit AWS now luckily they I do believe they as far as I know they were able to block it they have absolutely no responsibility to tell you if it was if sex is successive successful but anyways and it's happening in other countries go google it find out a lot of information it'll terrify and not let you sleep at night so we're also going to see malware developers shift their attack styles that was one of the things that I keep saying we work in security there are good hackers like myself that will go in there and we will do bad things on your server to prove to you that it is possible and how to defend it there are those of us not me that do this as a day job and now know what the hell's wrong with your system and we'll go home and know how to attack other people's other people's devices especially if they're running the same type of applications we have to continue to stay not only one step ahead but stay up to date what's happening on this exact moment so I can't leave you without leaving you an advice now that I've told you all this bad stuff learn from the past if you've learned nothing else here nothing is ever new again it's all a continuous repetition don't plug in that USB because it might be infected guess what we used to say don't plug in that floppy disk because it used to be infected it's the same damn thing assume breach I don't care if you're using a Linux desktop a Linux server whatever it is what I want you to do when you're setting up your security controls is I want you to assume the attackers already in your system too many times have I seen somebody clean up their system and start ignoring things because they've pushed the attacker out and all the attacker did was establish persistence and is sitting there quietly waiting for you to go back to normal every single step that you use even on your desktop and I don't have time to go into it Google the virus evil gnome if you don't think that we can completely own your laptop with a single piece of malware and watch you get into all these other servers assume that we're already in there think like an attacker how do you do that it's actually quite simple what is your company's business objective I'm a security person I'm a sysadmin I don't care I do my job you should care because what if your company is about to release this brand new feature brand new exciting feature back in the day pretend we're back in time and they're going to go on Oprah because I don't know who replaced Oprah and we're just going to get inundated with traffic this is going to be awesome guess what the attacker did the attacker watched Oprah the attacker knows that this is going to happen we know your code is not going to be secure it's the first deployment of it you're probably rushed to make that deadline anyways you're already expecting a ton of traffic it's going to be easy for me to hide let's say that your objective is to have proprietary information you're making the new AI guess what I'm going to go after guess what I'm going to do I'm probably going to distract you on something else because I know that I'm hunting down that proprietary information if you know what your company's trying to do to make money then you're going to know where the attackers are going to go after now there was a whole keynote that was about this so you know what go and watch Eva's talk and it's know what it's on your system from applications to code because if you don't know what's running on your system it's going to be 100% impossible for you to be able to defend it I went over by six minutes I think but with that thank you all very much remember it's okay to be new and I'm happy to answer any questions here or out in the hall