 Welcome everybody, welcome Philip. I hope you enjoy the talk. After the talk, if you have any questions, that's when everyone is kind of available to answer anything you have Monero related or Coinbase related or Cryptocurrency related I guess as well and Blockchain related, right? They're all kind of related in some way. Okay so enjoy the talk and hopefully talk to you soon. Okay look at that, computer worked. I'm going to apologize in advance for my voice. It's not keeping up super well this weekend but with the amplification hopefully it'll go okay along with some water as well. Good morning. Got a good group of hecklers here today. That's going to be a good time. Thanks for coming out at 10 o'clock in the morning. I'm sure some of you just stayed awake until 10 o'clock in the morning so more power to you. You can go to bed soon. As I said my name's Philip. For the last two and a half years I've led security at Coinbase. Normally this is the point where I ask the question who's heard of Coinbase but I'm pretty sure that's an obvious question here. How many of you guys are customers? Nice. That's a half-ish. Really cool. Thank you. So I'll skip the sort of what is Coinbase but the other way I described what we do is we are on the world's largest CTF with the highest stakes sort of outcome. So we store somewhere on the order of $15 billion of cryptocurrency which is not something that anyone wants to lose. Before we get into all that stuff I'll just want to talk about the elephant in the room for a second. I'm the first talk of the day. Half of you are asleep. Half of you are drunk from last night. But so I appreciate you came out and I think what I want to sort of give you back for your attention here is a bit of insight into what it takes to protect a modern cryptocurrency exchange. We'll get into why that's hard in my point of view. As well as some sort of viewpoints into where I think we as an industry need to get better. And need to get better in order to push crypto, actually push crypto to the masses. Not just a very small board room or a very small room in the middle of DEF CON. So I've been doing this security thing for a while. I try to pick where I work with a very simple lens. This is where I find the most interesting attackers. And that's led me to a bunch of different places over my career. But I will say cryptocurrency is, I think, takes the cake so far in terms of interesting challenges. And really in terms of doing things that no one's ever done before in this context, which has been pretty cool. I would be remiss if I didn't say Coinbase is hiring. Coinbase is always hiring for security specifically, but really across the board. If what I'm talking about is interesting to you, you want to learn more, come grab me, happy to chat. So bottom line, what are we going to talk about today? Number one, we're going to talk about a little bit the industry at large. Give some context on why cryptocurrency is actually a hard, or being in exchange is actually a pretty hard problem. We're going to talk about a bit about how Coinbase looks at the problem. I'm not going to go through our Coinbase's entire security program because we would be here a way longer than an hour. But I am going to hit out a few sort of interest areas that I think are interesting and how we execute security in this environment. And then we're going to talk a little bit about the industry as a whole and where I think we can improve and do better and actually push ourselves out there and become the kind of industry that my mom is okay investing in. It's really my bar when I think about this stuff is I think about my mom. If she could interact with whatever we're building in the same way that she would interact with Citibank or Wells Fargo or anything else that has to do with money, then like right direction. So that's a big number. That number is the total losses in cryptocurrency 2011 to 2018 across the industry. It's compiled by a non-profit called Crypto Aware. They're actually a really cool little non-profit. They're focused on sort of user security and advocacy in cryptocurrency. And actually my opinion is they're underreporting that number. That number doesn't generally include the sort of retail scams, social media stuff, tech support scams. So that number is probably actually significantly larger than we can track in terms of exchange compromises and major scams. I'll give you a slightly scarier number. So of that almost three billion, it's been lost 2011-2018. 1.7 was lost a year to date in 2018, right? That is the wrong slope for that curve, right? We want that curve, that slope the other direction. We want that going down. We want less losses over time. And so that begs the question and this is really what sort of starts our approach to a security program of Coinbase. Why is this hard? What is so hard about protecting crypto? I'll say another way we break down this industry is in terms of causes of loss. So this is a graph from the blockchain graveyard. If you guys are not familiar with it, it's an outstanding resource maintained by a guy named Brian McGee and Magoo, who he basically crawls whatever public information there is out there about a breach to give an organization, tries to find a root cause or tries to sort the wheat from the chaff and puts it on his GitHub repo, right? And then over time he charged that. He's tracking breaches from 2011 to 2018. Really great summary of each one. If you're interested in that time, I highly recommend if you're in the industry, you go read it because it is a list of lessons learned in our industry. Or put another way, a list of things to avoid in our industry that's really, really important. Blockchain graveyard, just Google blockchain graveyard is the first result. Kind of a unique term. So this is total he's tracking in their 59 breaches since 2011. That's an average, by the way, to save you guys the math of eight breaches a year in this space. Incredibly high number, higher than basically any other industry I can really think of. Besides maybe the payment card industry, but that's such a huge industry that the numbers don't really quite compare. So we ask the question again, why is this so hard? This starts to give us some answers. Give us some insight into how are we losing, right? And the interesting thing to me is by and large we are not losing because of esoteric cryptocurrency vulnerabilities as an industry. We're losing because of bread and butter, server volums, app volums, not enough customer off. We're losing because of scams. We're losing because of security problems that we as an industry have been working on for decades at this point. What does it say down here? Protocol cold, what is that? Four breaches of the 49 that he can actually point at a protocol level bone or something really deep in the cryptocurrency world. So do we go back again to sort of why is this so hard? If server breaches are the cause, why haven't we solved this problem yet? Anyone tell me what movie that's from? Yes, outstanding. Give that man a cookie. Which die hard? There you go. So those are bearer bonds. Those are $100,000 bearer bonds. And what I think about the security model analog for cryptocurrency, what I think of is digital bearer bonds. A lot of people say cash. I don't like the cash analogy because if you've ever tried to move $10 million of cash, it's actually not really easy. It's very heavy and bulky and not easy to move around. Bearer bonds are super easy. That stack right there is, if it was real, which obviously it's not, but is easily hundreds of millions of dollars. Yes, sir. Yeah, give him something. That was awesome. I expected no one to get it. Good job. I may throw more of these out there now. She had more screenshots. Fail, sorry. So why is it like a bearer bond? I'll quote you from Wikipedia. The bear bond differs from the more common types of investment securities and that is unregistered. The records are kept to the owner. The transactions or ownership, however, physically holds the paper, owns the bond. Does that sound familiar to anybody? That sounds a lot like cryptocurrency to me in terms of the threat model for theft. So what we're trying to do is protect an asset that's globally valuable. Bitcoin's a Bitcoin no matter where you are in the world. It's digitally transferable, which is actually fairly unique and is irrevocable. If we set up to design an asset that people would want to steal, I'm not quite sure we could have made a better one. I don't know what we'd add to that to make it more attractive to attackers. So that's why it's hard. We're trying to do and protect a new asset, a new thing. I'm going to leave the sound at the end, if you don't mind. What a slide. So like I said, we're protecting really what is a new class of asset that has fundamentally different risks from previous assets. In a really interesting way, we're doing that in the context of an online service. It's not like this thing is in a vault somewhere necessarily. Most of us in the industry who are building these systems are building them in a way that users are interacting with them on a website, on an exchange, on a wallet, on whatever. And because we're protecting a brand new asset class, we as an industry are learning as we go to some extent here. What works, what doesn't work, and how it works, especially around the areas of protecting these assets that don't quite fit the mold of anything else in the world. So we're inheriting the threat model as an organization in this space of part maybe social media company, part bank, and part something else that's like not yet defined. So we're trying to fit solutions and controls into a framework that is new and sort of we're naturally having a teething pains doing that as an industry. So no wonder this is such a hard place to exist. So great, it's hard. Shock, I'm sure all of you are shocked to find out that cryptocurrency, defending cryptocurrency is hard. So what does it look like for us to defend this stuff? So like I said at the beginning, I'm not going to go over Coinbase's entire security program. We'd be here for a week. But I think there's some stuff to talk about that is interesting and unique and how we approach this problem. And the first thing, and actually I'll talk through a few of these interesting things, we actually open sourced a fair bit of this stuff already. And the stuff that's not open source, most of it is moving that direction over the next six or nine months. I'll highlight sort of what bits are open source, what bits are coming, and what bits you can learn more about in other talks. The other thing, one of our sort of foundational ideas here is that trust should be created through transparency. Not through blind faith. So if I'm asking you, hey, trust me with your money, I should be backing that up with A, and here's why. Here's what we're going to do to protect it and make it safe and keep it safe. So we spend a lot of time talking at conferences and events about sort of the tools, the techniques, about what and why and how and where we do it. And our attempts to continue and do it even more. So the first and most important thing to think about when you think about Coinbase's security program is the people. So today Coinbase is, call it 500 people, security at Coinbase is 30 people. That's 6% of the company is focused on security, which is an insane ratio for most organizations, most industries. And I think especially when we're talking about an asset like cryptocurrency where there is a ton of innovation and we build a lot of our own tools, we're really forward-looking, you have to start with the people because they're the ones that are going to innovate, that are going to find the new ways of thinking about securing this stuff, that are going to actually be the ones solving the problems. I can't look at a vendor for this. There are no vendors that look at, they're probably some, but there are no vendors that say, you know what, I'm going to protect your cryptocurrency and you put it here and we're going to make it safe and make it easy. So it goes back to the people and I'll say, I'll just say once again, I'll sit out there, we're hiring. Just saying. Some people over there you can talk to if that's interesting. So this is a pretty picture of what we're not actually going to talk about, but it's very pretty. It's a high-level architecture of Coinbase, very, very high-level. So high-level that's not actually useful, but it's pretty, which is why I have it. So if we go back to that blockchain graveyard slide for a second, number one leading cause of these tracked breaches or these tracked sort of losses is server breach. So when we think about our security program, we walk through why is this hard. One of the answers is, you know what, attackers are walking through the front door. So let's talk a little bit about how we close and lock the front door in this kind of environment. We got Coinbase and Cryptocurrency in general, I think we're actually super lucky because we don't have legacy to deal with for the most part. We get to build the stuff from the ground up and we get to build it. I see this as lucky, others will disagree. We get to build it under constant pressure from attackers. My philosophy, this is one of the reasons that when I look for places to go, I look for places that are great attackers, I see you like an attacker. You never innovate as well as you do when you have a clear and present threat or danger to innovate against. It motivates you, it focuses you, it helps you do your best. So we get to build this ground up new technology under pressure, under focused attack. What better place could we go to build something? Other people look at this and say, we're stressful, and it is, but it's also amazing. So what have we built here? So Coinbase is a fully containerized, every single service in Coinbase is deployed in a container. Virtualized, we're in AWS. Continually and immutably deployed service. Let me break that down for a little bit. First of all, this is all based on a custom orchestrator we built internally called Codeflow, which are open sourcing piece by piece. We open sourced the actual deployer itself called Odin, I don't know, three months ago. It's a thing that takes a description of a deployment, it's a JSON file, and actually makes it happen in AWS. And it's like the rest of Coinbase, service-oriented architecture, so we're open sourcing piece by piece of Codeflow quite honestly as the development teams are happy with it and want to actually get it out in the world. Codeflow handles code from PR to deploy and prod. It handles the entire path and manages everything from consensus requirements on code submission, which we'll talk about more in depth, to security scanning, to CI CD, to build, to consensus on, or to secrets management, to deployment. It's all in sort of one long CI CD path. What that means is that we can make a lot of this stuff transparent to our developers, build a really overall very, very safe and secure CI CD pipeline, which I'll get into a little bit later. So let's dive into the rest of that. So containerized, right? Like I said, every single service in Coinbase is in a container. Some of you were probably shuttering when I said containers because you have some sort of container fear. Containers can be the best thing in the world, the worst thing in the world when it comes to software management and deployment. It can be the worst thing in the world when you back into it, as you discover six months after the fact that your dev team is using containers. It can be the best thing in the world if you walk into it eyes wide open and say, you know what, we're going to be able to use containers. But you know what, we're going to make it easy for you guys to use containers. We're going to manage a bunch of it before you. So at Coinbase, the infrastructure team manages a base container that all containers in Coinbase must descend from. We're not pulling stuff off of Docker Hub. It just won't work in our environment. Developers, development teams then base their services from that set of base containers so that we control the underlying layers. We control the patching. We control what tools are installed, whole nine yards. The developers are defining service-specific Dockerfiles. As those of you who have some container exposure know, you can do a lot of damage in a Dockerfile. As if you, you know, the most obvious and consistent example is, you know, a beautiful base layer that then has like a curl open SSL point, you know, 0.9 directly on the file systems library. That's not great. And it's very, very difficult to detect that in a lot of cases. So what do we do? We actually run our Dockerfiles through a linter and say, hey, development team, you can't actually do that. You can't use run and wget in the same line. We're not going to let you do that. We're not going to let you shoot yourself in the foot. But instead, here, go through this paved road, this other way of doing this. We'll get you the packages you need that are up to date that we can track and update and make better. And that not only takes load off of our developers, but it means that we control our environment in a way that's really hard to do outside of that kind of setup. We control every single package, every deployment, every line of code, every version. We know where it is, when it rolled out, where it rolled out, how it rolled out, right? That's awesome from my perspective. So you might ask, okay, containerized and virtualized, like what? What? Right? Why do you both? What's going on? So there's, there's, sounds like a train is right on the outside here. So there's two reasons for that. One is we, when we think through sort of the threat model on cryptocurrencies, one of the things you quickly come up with, with this again, this digitally transferable, non-revocable, will be valuable currency, is that this is one of the relatively few, in my opinion, industries where dropping and burning an ODA actually might make sense, right? Most of the time, probably economically doesn't work out in terms of like risk of loss versus risk of gain. Here it might. So one of the foundational things we, we looked at is we should always, this is, this is sort of a common flag, right? We should always layer our security. So how we do this is we, when we, this is also one of the reasons we will code flow, because nothing else can do this. When we deploy, right, we get containers on, on verts that are mutually trusting, right? Where if you popped one, you're probably going to get to those anyway, because there are credentials sitting in, you know, sitting on that one or otherwise it's highly likely you're going to be able to pivot. We then ensure that containers that don't have a mutual trust relationship never exist on the same virtual machine, right? So that means in order to hop from a front-end's web service to a back-end payment service, right? You're not going to do that through a side channel on the verter through popping the, you know, an ODA prevesque in the Linux kernel. You're actually going to have to do the hard work of moving through my environment and pivoting where I can see you, right? Not trying to do it in memory on a, on a Linux system. The other, the other way I think about, and we think about defense a lot is, you know, there's the common defenders have to always be right, attackers have to, you know, be right once, which is, which is true to, to, as far as it goes. The other way of thinking that about this, this sort of setup is that attackers have to play on my playground. They have to come to me and exist in my environment. So then my mission then is to make my environment as inhospitable as possible to anyone who's going to come and try to take my crypto, right? And if I, if I do my job well, I can, I can make that actually quite frustrating. So, that was the wrong thing. So, at the end, let's talk about that, that continuously deployed a meetable bit, right? So, we deploy on average, there's, we've published a blog post about this, so you can actually look at the data, something like 20 times a day. We deploy a lot. Every time we deploy, we are rebuilding that service from the ground up with no overlap. New verts, new containers, new security group, new ASG, whole nine yards. There's not even any network connectivity between the two services. If we're deploying 20 times a day with that kind of, of rebuild sort of structure built in, that means on average, I think our average lifetime is something like 1.5 hours for a service running in prod. That's a fairly frustrating environment for an attacker to live in. And really what it does, and this is why I particularly like it, is it makes the attacker re-exploit every single time, right? Exploits are inherently unstable, especially remote exploits you're trying to land on some random AWS virtual machine in the cloud. You're not maybe not sure what actual operating system is underneath there. You're going to crash stuff if you're re-exploiting that frequency. And when you crash stuff, even if the rest of your exploit was completely stealth, I'm going to know because it's going to crash, right? And we track that. So then I can come back and figure out what crashed, why it was going on, right? And respond. This goes back to, as an attacker has to live in my playground, that means I get to set the rules. Let's talk about AppSec for a second. We do a bunch. This is one of the focuses in our program. Again, walking back to this fun graph. Right? So right after server breach, ignoring unknown, because it's unknown as unknown, we have application vulnerability, right? So we spend a lot of effort on making sure that the systems and services we deploy are safe, are defensible, are tested, are documented, are threat modeled. And then fundamentally we understand what we are running, not just so that we can defend it, but so that we can act to prevent anything from happening in the organization. So one of the things, I'll walk down this overall thing. The first one I'll start with is Sallus. So this is on track open source. I would guess before the end of the year. So Sallus is our stack analysis framework. So what we did early on was looking around, we had the realization, of course, that hey, you know what, if setting aside the server breach thing, attackers would come in through the app, by and large. So how do we make sure that as we're shipping, and we're shipping very fast 20 times a day, as we're shipping these updates, how do we make sure it's safe? So one of the answers to that is through automated analysis of code before it hits prod, in a way that gives engineers immediate feedback as to why, if assuming we flagged something, why we flagged it, what's wrong with it, what they can do to fix it, and hopefully not do it again, and in a way that aggregates those stats over our entire base of developers, so that we can look for hotspots, we can look for issues, we can look for engagement. So that's Salis. Now, one method we could have done here is let's build a static analyzer. But that doesn't make sense to me, because there are a lot of great stack analyzers out there already. What's missing is the ability to point them at what they're best at, and normalize their results in a way that we can then use downstream to say, hey, the problem exists here, this is what the problem is. So that's what Salis is. Salis is essentially a framework for interacting with static analysis tools that can pick and route based on the language it detects for a given project, pick the analyzers that are relevant for that language in that project, take the results back in whatever format we get from the random analysis framework we chose to use for that language or that problem, put it into a common format across the board, and then use that to interact with the original pull request and say, hey, line 15, this tripped this rule from this analyzer, hey, developer, please resolve this before you're allowed to deploy. And then Salis will say, and by the way, you can't merge this change until this is resolved. No humans, no humans in the loop, no human interaction, but our development teams get security scanning, they get instant feedback. I've gotten it in a code review. And we protect Coinbase itself from whatever that change may have been. It's a really, really cool little tool. I'll be really happy when we get it out in the world. This is the second thing we focused on, again, was this idea of sort of the CICD pipeline, right, and if we're integrating security into an organization that's moving this fast as Coinbase, we have to be as close to zero touches as we can possibly be. So this mirrors a little bit of what I talked about for the infrastructure side of things, about that. We want to integrate with development practices seamlessly. We want to be there as part of the CICD pipeline. We want developers to find it easy to write security unit tests. Part of following up on any incident is how to make sure this doesn't happen again. So we want to be there to help them say, you know, let's put a security unit test in this CIC pipeline. We want that deployment pipeline to be easy and straightforward and have developers not have to worry about managing their systems, because we can manage that for them. Consensus requirements on code merge, this is actually, I think, a pretty nifty thing that we do. So to push any code change out of this, so you want to push a change to Coinbase, you're going to have to do, you yourself as your submitter are going to have to go get, call it, three other people to say, you know what, this is a good idea. Those three people can't have added code to the actual PR, question has to be, two, three is totally independent people. As you do that in line with the pull request, it's two-factor authed. So you get that nice push to your phone that says, hey, you just said that we should deploy from service X, commit hash Y, was that changed? You mean to do that? And assuming the answer is yes, then great, everything merges, it gets deployed. Assuming the answer is no, that kicks off a sort of a minus one process where we can actually say, you know what, you actually need to do this, this is super terrible, you should require four or five or six reviewers depending on how many minuses the pull request is getting. This lets us ensure that no one individual is another sort of core concept we shoot for, no one individual, no single person can do a thing that impacts Coinbase or the PII, the fiat or the cryptocurrency that we have stored. It should always require a conspiracy, because conspiracies are fragile and they're scary, and they're really high risk for the conspirator. We want to, again, the attacker in this case has to play on our playground, we want to make that playground rule as hard as possible for that attacker. Concierge AppSec program, so this is one of the reasons the AppSec team is probably my biggest team at Coinbase is because we want not to be the guys that sort of parachute at the end and say, no, everything's terrible, go home, this is crap, you can't deploy it. That's not productive for anybody. Instead we want our AppSec team to be in the stand-ups of these development teams, to be wedded at the hip, to have all the same incentives to get where they're trying to do so that as that team is writing code, is making changes, that AppSec team member knows exactly what's going on across the board and can render sort of that independent third party, oh hey, that's actually pretty risky, let's engage and help out and that's making this change. Again, this is the app level, for an exchange especially, is one of the most worrying things that we deal with, we want to make sure that we're providing that expertise and we're providing it in a way that developers want to use it. They want to get engaged. Another thing I'll highlight here that I think is quite cool is that we're getting much, much more into recently is blockchain monitoring, right? So if you imagine a world where there's you don't have to imagine it, it's today, what are there? 1800 or so assets, somewhere in that vein, in that vicinity. 1800 or so crypto assets in the world today. As we think about looking at assets to add, you guys saw we just added ETC. We've talked very publicly about the desire to add more. One of the things we want to be sure of is as we add those assets that we're proactively looking at those blockchains to say, is this asset still safe? Do we know what's going on with this asset? Is there a 51% attack happening? Is there a contract indication of a function in a token that should never be invoked so that we can take immediate protective action on our systems so that we're protecting the extent we can with whatever the vulnerability is our assets or our customer's assets more appropriately. This is also an area that we're pushing pretty hard in and I hope we'll open source some tools in as we get that stuff built and rolled out. Like I said at the end of the day what trust should be based on transparency. You guys shouldn't have to wonder how in protecting your crypto. You should know. Let's talk about the detection response. Because at the end of the day you can't win them. You need to have the ability to detect when things go wrong and you need to have the ability to respond safely. So that goes with the heart of this first one. This is my favorite name for our project, Dexter. He's our friendly forensic assistant. A few Dexter fans in the audience I can see outstanding. The rest of you can google it later and you'll laugh when you do it. So what's Dexter why don't we build it? Go back to what I said earlier no one person should have the ability to impact or steal Coinbase, crypto, PII, fiat, etc. Incident response, that's one of those areas where the response is frequently we'll just throw all that shit out the window and we'll just go ahead and respond as an individual. That to us is extremely dangerous and worrying. We don't want that. But at the same time we want to be able to respond to incidents quickly and very agile. How do we square that circle? What Dexter does is two core things. Number one provides a consensus based approach to executing forensic commands. So something happens some instance is doing a weird thing in production and maybe it's an instance that's actually dealing with crypto. I don't want an incident responder hopping on there. But the incident responder can spin up Dexter, start an investigation and say this is weird, give me a process listing give me basic live response processes LSOF maybe some stuff from PROC depending on what the actual problem was and that investigation spins up and then it sits there and waits for a plus one from another incident response engineer. Depending on the commands you're executing via Dexter, the number of required reviewers changes. So maybe process listing LSOF not a big deal, it's just a plus one. Maybe he then says you know what I actually need a memory dump that's going on here and I need to really get deep into this. That's going to require a lot more consensus. Maybe it requires a plus two, plus three maybe it requires sign off from me, or maybe it requires sign off from legal. But it lets us very flexibly define who and how can execute sensitive commands even in a fast moving, highly critical incident response situation. We never want to be back to that place where one person can do a thing. It's also built to be sort of, there's a bunch of other cool things about it around in our ADB environment it's highly segmented. So hosts normally can't talk to each other, there's no place on our network where you can go talk to everybody to do incident response. So it's S3 based so we're pushing into a queue that the hosts are monitoring, they're pulling down and saying is this for me, is this for me, yes no maybe. It's all backed by DBG signatures so we're actually basing this stuff in crypto not on like a software counter that says oh yeah you plus one of me enough I'm good to go. This is actually, so the guy who wrote this is open sourcing it at Derbycon this year. Just got his talk accepted. I don't know, it's two weeks, two, three weeks ago. If you're at Derbycon you should really go to the talk because it'll be awesome. So the second thing I'll talk about here is again another derivation of that container based stuff. So I do another talk you can go find from ShakaCon I think was the last place I did it that's incident response and dockerized in containerized environments where I talk through sort of what's unique and special about containerization when it comes to incident response and detection it's a really cool and interesting environment but the thing I'll highlight here is that because we've dockerized everything everything's in a docked container everything is running isolated that means we can very easily pull specific behavioral information about a given service right? So the reason that this stuff doesn't work well across the board normally is because there are just so many signals to deal with, right? There's systemable signals, there's if you try to do this on laptops, God help you because that's even worse but even on server environment right it's really really hard. In our environment and say how is this docker container behaving? How is this single process acting? And is it acting different than its peers? And then based on that we can do things like using core basic tools, auditD, eBPF look at that and say how is this acting at a system level different than its peers? Is this behaving in a way that's indicative of an attacker being on this environment and if so, we go back to Dexter and we can actually respond very very flexibly to that, to that area, figure out what's going on and sort of figure out what we need to do next. The last thing I'll talk about that we do that's cool and special is we log everything. When I say everything, I mean everything. We do this for a couple of reasons. We want to go back to that deployment cycle. We're deploying 20 times a day, average lifetime is like one and a half hours. What happens if there was an incident and we discover eight hours later? The container is gone. What are we going to do with it? The answer to that is we log and most importantly enrich everything immediately. When that container is gone, I can still walk back and pull all the logs that were issued by that container. Tagged with that container's name and version number tagged with the processes, the resolved process names whole nine yards. Most of the stuff I would get from a live response I can regenerate from logs that we maintain of these instances and I can get it quickly and get it searchably. The most important thing, I think the most important and different thing we do here is metadata, metadata, just all of the metadata. Anything that might possibly be useful in the future will tack on the log. Up time, we tack on the logs. As we roll into this any future investigation there's never any to the extent we can, any ambiguity. There's never any question that we have the data we need to make a determination as to what happened. So we've got about I don't know seven or so minutes left before I'll open up for questions. Before we do that, I want to talk a little bit about what we can do better as a community. I think there's three specific points I want to drive home. One is today there's not enough there's not enough intelligence sharing in the community. There just isn't. We're fragmented ideologically or fragmented commercially and we do not talk to each other enough even though we share the same opponents. The same attacker coming after me is coming after Jim and I is coming after anybody else here who's running an exchange or a currency. We share opponents. Why wouldn't we share threat intelligence about those opponents to make us all stronger? One concrete thing here I'll flag really quick is we recently working with some other exchanges and some other traditional financial folks started an FSISAC working group. If you don't know what FSISAC is, it's Financial Services Information Sharing Analysis Center. It's been sort of a cornerstone of what traditional finance, how traditional finance is. It's been around for decades. This is sort of a non-partisan not owned by any one company. A way to interact and share with peer organizations. We started this working group and we're reaching out now to anyone and everybody who was involved in the crypto space to say come join help us establish this sort of basic level of security trust and cooperation so that we can all get better. Number two and this is actually also related to FSISAC we're not really good at standards in the community. There are a few out there the CCSS is one that's been out for a while and gone some traction but when you go back and you look at sort of how traditional finance evolved over time I'll give an analogy that I'll see if you guys like it too. If you were in the 1800s 1850s right, walking around looking for a bank to deposit your money. You walked into town, walked out of the bank this is nice, this is a place, this is great one of the questions you're probably going to ask is tell me about your vault. How are you going to secure my money? Because bank robbery was much more common there are a lot more risks everything was more fragmented and as an informed consumer you needed to ask that. Today let me ask them to just pull the room into a bank and said before I deposit my money I need to see your vault. Anyone ever ask that? Now, you know why? Because that industry grew up, it built standards it built and through those standards it built trust with people. It's a more complicated story, there's insurance involved there's regulation, I'm simplifying it but through the use of standards we can build trust with customers, trust with regulators so that the question shifts from where it is today for crypto, I'm going to deposit my crypto tell me about your security to where it needs to be, tell me about your services tell me about how you deliver better service than whoever else tell me about why I should put my money here because I'm going to get these other benefits. So the ask is either under the auspices of the FSIS like working group or via an organization like the CCSS or whatever we as an industry need to invest in standards we need to invest in not just creating standards but coming together around standards to driving ourselves as a community holding ourselves to that standard so that we can be worthy of that trust. Last and this is obviously self interested point really not enough folks in the security community are looking at crypto and saying like you know what that's a great place to do security we see some of it it's starting to change certainly over the past year I've seen a change but I don't think we as a community are doing enough to talk to the rest of the security community not just about how cool crypto is because it is really cool but about how what an interesting problem set this is right how interesting is it to secure a thing that's never been secured before to write the book on protecting an asset in this adversarial environment right people should be knocking down the doors of cryptocurrency companies to say wow that's a great challenge like why wouldn't I want to engage there and I think they're not because we as a community again we're not doing the outreach to the rest of the community about the interesting security implications of this space we're talking about how cool crypto is this is also a picture to say if you want to go on base come talk to me and with that what are your questions let's all let you pick who gets first we are making that comparison because anyone who has access to a private key can immediately gain access to the funds so the question is how does that differ from just a password to a bank account because there are other controls there right so a bank would have the ability to do fraud detection on transactions the bank would have the ability to do geodetermination you're in the US and there's a login from Ukraine like maybe we should do something about that there are other steps you can take in the middle there as opposed to possession of a private key possession of a bear bond but it's like a debit card for example anyone who extracts the money can just run away with it sure, but depending on how they do it and where they do it if you have a debit card that debit card is going to do a bank account and there are controls on the interaction between that debit card and that bank account the debit card doesn't necessarily mean I get all the money in the bank account because there is a control environment between the two there are daily limits there's the ability of the bank to say I'm going to freeze this debit card the core of the comparison I think is once you have a bear bond once you have a private key there is nothing anyone else can do to limit or protect from the loss of that value thank you hi, I have a couple of questions about custody to have a compliant custody solution, number one is it necessary to provision identity for the account so that you know exactly who they are secondly, for custody is centralization essential to secure custody and thirdly, when it comes to the regulatory regimes that I guess create the most work for you, I mean is it going to be the OCC or is it going to be the NYDFS like where does most of your as far as complying with regulations who's the toughest regulator the first question is really a question about sort of BSA and AMO practices and in general yes we have to, for US consumers we have to walk through BSA requirements we have to do KYC on customers before we're dealing with their assets for the second question on what was it, centralization on security I don't necessarily it's a hard question to answer on the abstract, because security can be defined relatively so certainly an individual can invest in a level of security that's good for their needs and their threat model without ever having to put assets anywhere right and a lot of folks do, especially in the crypto land I think the other side of that coin is people choose to use a centralized service because we can invest in centralized security or they don't want to spend the time and effort to build what to them as an acceptable sort of security arrangement for their crypto so for example we're a centralized service and then the third one about regulators really I see this space as the regulation is evolving and working with regulators from a company like Coinbase who has always been a regulation forward what we always want to do is work with them and educate them around how this space should be regulated in our opinions I think calling out anyone regulator as hard besides being unwise just generally also doesn't make sense to me because all the regulators I've worked with have come to the table with an open mind about crypto and wanting not just to make us go through a check the box of screws on us but actually wanting to go back and forth about a new asset and how the old framework should apply to this new kind of asset I've just read that the NYDFS particularly the cyber security rule and the new AML transaction monitoring and filtering rule that's becoming I think to the extent that it impacts everyone else obviously it impacts us because we're operating like a national institution you talked about the mist you talked about the mist model security and detection like you mentioned containerized versus what segment of networks the difference is in the layer granularity containerized workloads the security boundary is at the process the workload layer as opposed to network segmentation where you're segmenting at the server or at the host group of hosts layer the level of visibility is different as well as the actions you can take are different to me I personally prefer the segmentation on both but if I had to pick one I would pick host because you get a much richer dataset out of a host based detection then you get out of a network based detection we're just pretty much all just yes yes so user trade data is actually one of the most important things that we look at we think the last thing we want to do is expose user trading activities both in terms of individual privacy extremely seriously in that sense as well as like folks that are active traders with specific strategies they're executing the last thing we want to do is be in a position where we're leaking data on those strategies in a way that can be taken advantage of by others so thinking about thinking about that environment a little bit control of any sense of the like that is really predicated on making sure you know who is accessing it for what reason and when and why and having the ability to flag deviations from any sort of normal patterns there which is at a high level the approach we take very restricted access lots of monitoring on interactions with those kinds of data sources and sort of active oversight by the security team on that kind of activity high level sorry I'll just start calling on people now since we have no longer mics you and then the gentleman in front of you so we do use dedicated instances like the workloads security or the workloads sensitivity merits it the second piece there and I've had discussions with folks about this in both directions but because we move so rapidly in the environment it's sort of a side effect of our deployment strategy it's actually fairly hard to end up on a host with one of our systems reliably and the setup for that attack is pretty significant the gentleman in the white shirt and then black someone talked about custody and regulation earlier and one thing I wanted to ask you is that many centralized exchanges are in the process or have launched 0x like exchanges like on chain custody what's the house you yeah so we acquired a company called what was it three years months months ago that's operating as a 0x based relay so yeah and from the regulatory standpoint do you have any remark on that or not really I'll say that it's coming back outside of the US first yeah John I hear and is there a way to enhance the security overall by sharing some of those codes yeah so that's an interesting question he actually offered a multi-sig wallet for a long time had an extremely low uptake rate from customers so low that it was when we did the evaluation on should we keep this feature or not the code we could simplify by taking it out was a better trade off than keeping a feature that almost nobody used right so I think if the customer demand is there sure but we just don't see it thanks for coming and giving the talk what do you think what needs to be done for Coinbase to be able to accept Monero and other similar strong privacy coins that are out there fair enough I actually am surprised that question took that long look I think we've seen some really promising moves from regulators and from the privacy coin space Gemini in particular has made great progress with Zcash and getting regulators comfortable with privacy coins I think for us the primary thing that we want is to make sure the customers on our platforms are getting access to the assets that they want and that they want to use primarily in terms of A in terms of our digital asset framework that we've published and said here's how we look at digital assets but also in terms of what's actually going to be most useful for our customers and what do they want the most and then we figure out how can we do that yes just a follow-up comment on that the reason that the die-hard bonds were valuable was that they're fungible also I had a question on your pretty slide which makes it look like the hot wallets you guys named that cluster Knox is that actually true? it seems a bit obvious is that a comment on our poor naming schema? or we like simple names short names Dexter, Knox in the corner there one of the things you talked about was the various things that you've built internally and are now in the process of open sourcing so sort of in the bill versus buy open source world it seems like the period immediately after open sourcing an internal product is sort of the most dangerous period you get to get people fixing things externally but the code's out there now so what are you doing to defend yourselves from the potential zero days in your own software scenario your own internal vulnerability scanning for those types of systems? that's one of those very broad questions we could be here a week to answer but we don't treat internal software any different than external software in terms of our overall appsec program so code flow is going through the same scanning tools the same appsec process the same testing the same everything as a public basing service that's part of our overall Coinbase services probably the second part of your question is interesting is the risk tradeoff between exposing source code and not exposing source code I think it's a really interesting and sort of nuanced tradeoff to make between the sort of principle that trust should be transparent and risks we have around releasing code that might have issues in it in particular open source components like Odin which we did several months ago we spend a lot of time and effort looking at that before the release we tend to release small pieces of code as opposed to like here's a 100,000 line behemoth we want to release here's a 5000 line tool because we can we can we can't put that much more effectively in terms of appsec it can connect to S3 and then the endpoints pull S3 for information so it's not, it can't connect to anything, it could just talk to an S3 bucket I think we might have time for one more question there's a bit more time if you have time yourself I think schedule for 11 o'clock I might just stay up here for an hour if you want to do a couple more that's okay, there's supposed to be a lot but if you don't that's fine too I'll do a couple more and then get out of here you mentioned intelligence sharing and FSISAC as a working group I'm curious about how valuable that data is for you as like probably your tech stack is completely different from what's far go sure and that's why I think this is there's two pieces, one is the financial actor sharing and one is sharing within crypto so two answers to that question one is yes, my tech stack is totally different attacker behavior is probably not that different how they, the kinds of things that they target, how they move internally how they act, things like that I don't really care about IPs and hashes that's good but it's not what I really want what I really want are attacker behaviors because that then feeds into my roadmap right, attackers want to do this how can I make that hard, frustrating annoying prone to failure there's nothing else I'm going to call it