 We have Ryan we have with us Tom we have Aaron and there's one more already stuck in traffic You may have experienced this as well this week And I'd like to give the word to Tom to kick off the first talk about the issues since DevCon 5 Hi, I'm Ryan Lackey. I've been a cypher punk since the early 1990s Started the world's first offshore data haven company haven't go on Sealand in the North Sea Did defense contracting building satellite and cellular networks in conflict zones for a while and Then I started as CISO at a bunch of crypto projects I'm currently CISO a chief security officer at Abertas insurance a crypto asset insurance company where I see a bunch of Interesting risk events happen. Thanks Ryan. We have an hour Wow, I mean there have been a lot of security incidents, so I guess we have plenty to talk about but I just realized we have a full hour Hey guys, I'm Lane very excited to be here I Have been kind of steeped in hacker culture and hacker ethos since a pretty young age Kind of high school 14 15 years old specifically kind of penetration testing white hat hacking ethical hacking this kind of stuff I So kind of fast forwarding to the present I was an Ethereum core developer with the Ethereum Foundation for a couple of years And for the past three years have been working on a project which is a new layer one blockchain called Space Mesh Based on a novel consensus mechanism So I currently lead R&D at that project and just generally kind of follow the security ecosystem very closely Contributes to you know kind of security Response here and there and just think deeply about it in the context of my work at Space Mesh. Nice to meet you all My name is Arun I've been a full-stack developer at bug bounty hunter for a few years and just generally like a very active DeFi DJed I was previously lead developer at a web 3 security Like related startup called anti.finance and I'm currently head of research at Marana Ventures and before crypto I Was a scientist in system biology where I did research in high throughput protein sequencing and Tom Howard who's currently stuck in traffic as the founder of DeFi payments while it's Monsendo Crypto options exchange power trade. He's an unreformed DeFi degenerate angel investor And he's now got a newly formed seed fund network zero He's working on building tools and infrastructure to support the network state concept that Bellagie Developed as sort of a successor to the nation state So there's a whole bunch of security incidents that have happened in the last three years The world was very different in 2019, and I'm sure we've all we can all remember How things have changed? I think there's some some broad categories that we're going to discuss Of things that have happened broken down by category and then hopefully for 10 or 20 minutes toward the end our views of how things are going to get better because the current state is terrifying like I I wonder how people who have Very large fractions of their net worth in these in projects locked on the on-chain sleep soundly every night I certainly don't and it's probably something that That that is not sustainable and its current level of vulnerability over time So I think to start with one of the things I've noticed over the past like at least 12 years in the overall crypto industry is a lot of the vulnerabilities that we're seeing in crypto are actually old vulnerabilities that were Either core internet infrastructure or fundamental to human nature. They're not things that are really novel and new to any of these protocols. They're a whole class of things. They're like Insider threats personnel issues key management leaky abstractions Bugs versioning and things like that so there's a whole range of things that we've seen that that have really affected people and I think some of the incidents that we can talk about Clearly fall into that. I mean, there's a bunch that I can think of but if either of you have any that you can think of They come off the top of your head Specific incidents and then the kind of vulnerability it was I Think it could be interesting If you want to go this direction to kind of do sort of a taxonomy because there are so many different buckets I mean there's at least kind of three or four different buckets and you touched upon a couple of them right now Some of them are social engineering related. Some of them are bugs in the code another bucket is Multisig where kind of like your your vanilla bridge exploit where you know one or more kind of keys were compromised From where I sit the most interesting of these is kind of low-level bugs in Sort of low-level libraries that are that that live there that lurk there for years And and they're not discovered until the whole world blows up So the most obvious and salient example of this is the the Binance hack that just happened a few days ago Right that was like a very low-level library and it's kind of in retrospect Incredible that it was there for years and that code is used not only in the Binance ecosystem But also in the cosmos ecosystem and cosmos kind of got off lucky luck Very luckily in this case because the vulnerability wasn't present there in the same fashion I don't know we could dig into one of these or we could do taxonomy in whatever direction you think is the best way to go That's my favorite category of vulnerability Yeah, actually just to like jump on that point before we move on Yeah, it's really important to realize that this code is very complicated and it's very new Actually Vitalik gave a great talk a couple of days ago talking about bringing roll-ups online So I believe it was the the optimism of fraud proof I think it was a get-none actually it's about 34,000 lines of new code any and any small bug in that code could lead to Eventually billion dollars being lost. So how do we bring these like very complex pieces of software online? There's a curing bill of dollars in kind of a safe and sustainable way and Yeah, definitely looking forward to exploring that question Jerry. Yeah, I'm happy you mentioned this. So like I'll try to share a little bit about what the world looks like from the perspective of a core developer Right who worked on Ethereum previously. You know, this is relevant in the context of my work on space mesh And I think just any core developer we have this notion of a TCB TCB means trusted compute base and What it means is that there is some amount of code Which is 100% critical to the functioning of the network and any bugs or vulnerabilities or exploits in this Inside the trusted compute base are kind of apocalyptic right they're kind of like like stop the world scenarios like like many of the exploits we're talking about here and This is not new to crypto It's not new to web 3 right this concept has existed forever It's present in operating systems and things like this But as a core developer touching upon what you said a moment ago about kind of the diff You know looking at the changes that optimism has had to make to existing code like go Ethereum or you know Any anyone building Let's say modifying or tweaking or adding on to existing code like this is this is what keeps us up at night as core developers We want those changes to be as small as possible those diffs ideally to be on the order of like tens of lines rather than hundreds of thousands of lines Because every one of those represents a very real possibility of the introduction of a bug right another Principle of computer science is that there's no such thing as bug-free code doesn't matter how good of an engineer you are It doesn't matter how good your software engineering practices are there's some number and you can reduce it right you can reduce the Incidents of bugs by following best practices which again have existed for decades things like code reviews pair Programming there's various techniques, but it can only go so low I think Microsoft or someone did research on this a decade ago and found that the most senior most experienced programmers are still Introducing one bug for every thousand lines of code period one bug for every thousand lines of code Go Ethereum is probably on the order of a hundred thousand lines of code right and let's say Some of those bugs have been found, but there's still bugs there for sure So anyway, it's just as a as a comment as a core developer like this is literally what keeps us up at night We want to minimize the number of changes and that's very very difficult when we want to add features and bells and whistles You know add new op codes to our VM What's another good example? A Lot of blockchains are using web assembly as a VM now This is actually what I worked on in Ethereum this thing called E wasm, which was a candidate to replace EVM after after the merge Web assembly is a whole complex ecosystem and if we start putting web assembly based smart contracts on chain then all of a sudden the compilers and Interpreters for web assembly are now inside the trusted compute base So you've added a hundred thousand lines of code or something to your trusted compute base So one I'm jumping ahead here, but one potential mitigation strategy is to like Really minimize the number of changes the number of things you add minimize the size of your trusted compute base On that front I mean the concept of a TCB is something that I read about in the rainbow books that were the The NSA is security guides back in like that. They were written in the 80s So this is a very well understood concept in theoretical computer security Unfortunately, it's kind of theoretical because a lot of Systems there's there's two classes of errors here one. There's bugs in your TCB To there's not identifying what really is the scope of your TCB correctly There's a lot of the craziest things I've seen like the Yuga Yuga hack where a moderators Account recovery credentials for a discord server were essentially Used to cause like 200 like a crazy like multi multi billion or multi million dollar loss Was like 200 million dollars or something like that So, uh, yes There was I mean These things happen, you know in in in all companies in all industries But it just just your point about credentials being stolen. This is what happened with that massive I think it was uber right about about a year ago or something Someone got credentials of an administrator by a social engineering and then got inside the slack and got inside, you know Yeah, so we have to be careful of this stuff too Yeah, this is actually happening in a crypto quite a bit Now mostly with with discord and twitter. Um, actually is that xbt? A couple weeks ago I had like a real interest twitter So I was talking about someone of a access to an admin twitter panel And basically that was what allowed them to take over all these prominent twitter accounts Yeah, I think one of the main issues is we're using essentially consumer tools like discord telegram twitter things like that for large financial applications and The people supporting the applications that we're using are not building them for the use case. They're being used for They were built for gamers talking to each other about Progress in a game and things like that So expecting them to provide the level of security needed for a trillion dollar ecosystem for security It just doesn't make sense and it's unfair to them to to really blame them in any way and we need better tools We need better tools It's interesting. I think I agree very strongly with what Ryan just said The situation has gotten a little bit better pretty much Everything has two-factor authentication now and if you ask any CISO any security expert in the world What's the one thing you can and should and must do immediately to increase security? It's not only enable but force two-factor authentication on everyone So it's a little better than it was like five ten years ago Like when the discord didn't previously have two-factor authentication now they do and I think it's I think it may even be on by default But I guess I'm kind of I'm curious. I don't know the answer to this Do we think that those tools that we use to coordinate communicate etc? Need to be custom built or do we think that we can adapt existing tools to those purposes? Yeah, I'm a big fan of not reinventing the wheel if we could take from the Let's say 50 plus years of security research in web 2 and adapt it to web 3 Yeah, there's no brainer. We definitely should do that Yeah, I mean I think most of web 3 security is actually web 2 security. So these lessons have already been learned by enterprise users I would go a bit further than just saying two-factor. I think we should be using single sign-on hardware keys that are not fishable like the whole concept of Getting hacked because one of your emails clicks on an email and goes to the wrong URL Or is message something unsolicited on telegram instead of signal or telegram instead of discord where they routinely come with Someone go to a url. That's a fishing url Click on something and then boom you're out 500 million dollars is crazy Like that's the vulnerability of the system not of the individual that clicks on a link and the the idea of mandatory fishing training for users Is woefully inadequate for protecting your systems if it's possible to have a button that you push Right next to another button that you push every single day and you push the wrong button And it destroys the world like the problem is not the person pushing the buttons The problem is the person who designed the buttons and put them there so You really need to have a single sign on so you can revoke credentials for an employee if an employee here Like it's very common to have a laptop lost in a bar or have a Cell phone stolen which might be unlocked and things like that you want to be able to immediately revoke everybody's admin credentials So I think essentially if you have administrative credentials for any project which includes public publishing to social media Doing any sort of action on treasury Anything like that And you don't have the ability to revoke those credentials remotely on loss in a single Like in a like five minutes after the event is reported You are making a big mistake and if it's possible for a user to go to the wrong url by accident or by fishing and Lose their keys or use their keys incorrectly. You're also making a big mistake. So single sign on and a hardware enforced Key credentialing are the way to go but luckily the the broader security industry has largely accepted that and we now have things like past keys on ios and We have like the the phyto 2 ecosystem and everything else So the key management in the traditional enterprise security world is getting a lot better We also have great open source sso tools as well as commercial tools so so that world is getting better, but the problem one of the biggest problems with security is the The attacker can find any vulnerability that a defender needs to be protected against everything And you get a lot of cases where I mean everyone knows as a team Like you have a certain number of hours in a week for your people And you want to spend your resources on building the product because if your company fails If your project fails, no one uses it because you don't have any features. That's that's also very bad So If you have one hour to spend you probably want to spend that on working on your product and not locking down security So there's a lot of cases where the the floor of security is not a uniformly high enough floor There's a vulnerability and things like that. So I think making tools better making things secure by default is really the solution there So one thing I want to touch on with adapting web 2 security practices to web 3 is that one thing we have in web 3 The around web 2 is the systems need to be uh sensitive persistent And that's like a very important factor and that's one reason why Metamask doesn't have 2fa. I mean part of the reason is also because of the way eoas work and you could have multi-stakes with 2fa But if you want like a non-custodial wallet built on top of the ethereum platform, there is no real way to do 2fa on eo a you have to have a spark contract based wallet Which I think people are kind of moving towards in the future But like one example would be for instance the mango market hack last night People always say after these d5 hacks. Well, why don't you have let's say like some kind of circuit breaker or something That says okay, if you're trying to withdraw $50 million from the spark contract, maybe you have a human being involved in that process and One reason why we might not want to do that and I say with nastress because in the end if any spark contract Which has the ability for an admin deposit is not decentralized So most spark contracts and most deeper protocols don't really have a true decentralization as of yet And I think that's probably a good thing at this stage, but I guess like Assuming that that that wasn't the case and the smart contract was truly decentralized You can put a human being in that loop because that introduces a point of attack for regulation and a point of attack for For other kinds of compromises like you mentioned So how do we do this in a way which is which is programmatic and sensitive persistent? So I think there are going to need to be need to be some new design patterns in in d5 And you know there's things you could do like for instance you could For instance, maybe even help like an open zeppelin Kind of like smart contract library that that kind of like meters and says okay Like don't let you you shouldn't be able to withdraw more than like one million dollars worth of usdc every like hour or so And I think that's fairly reasonable Although maybe for example cases it might not be and it's also might be difficult to do it in a generalizable way Given that if you have for instance Oracle attacks the smart contract might not necessarily know what the value of the assets being withdrawn are So you need to be careful about about that code Yeah, I agree with that stuff. I mean we're we're kind of jumping ahead a little bit to like mitigations and solutions Um, I'll just add one more comment on that and then Ryan you can steer us in one direction or another but uh So you know the title of this talk is notable security and since since devcon 5 So obviously we're talking past three years a lot has changed in the past three years You know three years ago I would say best practices along the lines that you're describing were just emerging They didn't really exist We were still at that point struggling with basic things like re-entrancy And I think now we have a much better. What's that? We still are. Yeah, we start. This is true. This will be a thing forever Um, but you know, we have libraries that you can include now from people like open zeppelin that protect against common classes of attacks Like re-entrancy. That's just one example Um, you know basic things like not having ever a single Point of failure which is to say like a single key pair or something right you and and as as as he pointed out You know, uh externally owned accounts was just to say just simple basic key pair based accounts And ethereum are a single point of failure. You lose the key your hose. There's there's there's no getting it back So having multi signatures Having cool off periods or delays right both for governance as well as anytime you're moving funds Right having tiers so that you can kind of move a certain amount of funds instantly But something over whatever that amount is, you know requires a delay of two days three days Um, I don't know if people are familiar with this But the reason the dow hack the dow of 2016 wasn't worse than it was the reason that the uh The white hat hackers were able to recover the majority of the eth was because there was a delay actually multiple delays Baked into the protocol right so it could have been far far far worse Yeah, in general you have the the frequency of an attack the potential value at risk the amount that is Actually stolen which might be limited by all these other countermeasures And then any sort of recovery procedures that are possible And we've definitely seen a lot of progress in the last three years on both chain analysis and uh a lot of other You oh cool cool great, okay, uh, so we've seen a lot of um Positive changes in the overall ecosystem where we have uh Now some chain analysis tools. We have a huge ecosystem a security audit and infrastructure existing in the world we have probably Hundreds of audit firms both coming from the web2 world and dedicated to web3 smart contract audits So a lot of these vulnerabilities are our bugs in code and um, I'd much rather find a bug in code in An audit that I've just paid a couple hundred thousand dollars for rather than in the wild We also have more of a culture of bug bounties and reporting So if there is a bug found I can legally make a couple million dollars as a bug bounty and be a good person And also build my own reputation do it publicly everything else rather than just exploiting it and Probably taking 20 30 million dollars plus in dirty money and being a horrible person and Having to hide it and everything else and no one learning from it So there's a lot of a lot of positive changes that have happened The counter argument to all of this is just the whole ecosystem has continued to grow So if a vulnerability affects one percent of the ecosystem or vulnerabilities affect one percent of the ecosystem The total dollar amount and the total number of users affected just goes up as as growth goes Oh Yeah, just a couple quick thoughts to add to what ryan said the tooling that ryan was alluding to has gotten really powerful And and a lot of it is open source, right? So like folks like trail of bits Have multiple libraries multiple tools that you can use to do Inspection and analysis at various layers of the stack and I would actually say I wonder if you guys agree with this Simply running these open source tools today will probably get you more than you would have gotten if you paid six or seven figures for an audit Three years ago. Uh, so there's really powerful stuff. I mean it's it's Depends on your your threat model. Like it may not be sufficient. You should still do audits Copy at mTOR, but there are very powerful tools out there Yes, so there is quite nice. Um, one thing I want to say about the existing Models that I do think they're they're fairly efficient. Like I don't I don't think that uh Aside from a few auditing firms that have very stellar petitions like say trail of bits or or did dobb Most times when I see an audit, I'm not necessarily thinking like oh, this code is safe because we've seen many times that audits are are faulty like they're human-driven human code people can miss bugs either way and another thing about bug bounties is actually that uh The system is is just not really working very well. Um as a bug bender, I can tell you Ironically, it's often you make more money if you if you hack the smart contract and then hold the funds for ransom And then keep 10 percent versus just reporting the bug which is just insane but So that's an argument for possibly increasing the amount like there's a whole calculus of if you offer people too much for Um reporting bugs, then you end up with uh bugs get reported through that channel and possibly introduced and you have I used to do security contracting in Iraq and Afghanistan and there was a concept of like paying people if uh us forces caused a um Injury to someone or our death And there was a crazy kind like I would obviously say you should pay as large amount of money as possible But um if you make that amount too much there was an argument that people would intentionally cause innocence just to get the Pounty, which is terrible. But um you have the same risk with insiders like one of the whole categories that crypto companies do not address um Really at the level that traditional finance and everything else addresses is insider threat You can have a team if a team is a few people that have started a project together and have worked together for a long time That's one level of trust But if you've got a team that's hundreds of people you're hiring people using like linkedin and whatever else Somebody joins you don't really know uh what people's motivations are they could be um Incompetent they could be nation-state affiliated. They could be Criminal affiliated everything else they may introduce bugs intentionally and then Exploit them they may discover something in the course of their work and then pass it off to a friend externally who exploits it There's all sorts of stuff like that traditional finance solves this by having separation of duties So no single person would ever be able to um have a vulnerability that would ideally that would be able to exploit things But that isn't really compatible with an ecosystem where stuff is being built as you're as you're going Like you you have a trade-off basically you can build new stuff or you can Prioritize for extreme safety and resilience against this kind of insider threat And uh, I don't think it would be good for the ecosystem if we became so paranoid that we didn't allow innovation um, so yeah Right our final speaker has arrived for your panel Yeah, come on come on down Welcome welcome Sorry about that learning about bogata traffic this morning getting d-dost on the road Um cool. Yeah. I'm tom. Uh, I founded a d-fi product called mo sendo a few years ago focus on d-fi payments Uh, also co-founded a crypto options exchange Which is a centralized exchange that dealt a lot with custodial risk and then Have been an investor in the space for a while and just started his e-fund called network zero so I've actually been following the mango markets hack Even on the right over here. There's some interesting developments It'll be interesting to get to at some point And so one other point just before we move on was the The interesting argument about decentralization the binance hack that happened a couple days ago They Were able to mitigate their loss because they were able to just stop their chain Which is crazy from a centralization perspective, but did save them 900 million dollars. So that's the counter argument against Proper decentralization Also, the hacker bridged some of that money into us dc, which immediately got frozen Which is a centralized stable coin. So yes So, yeah So actually for that binance hack specifically of I would argue that was certainly change sovereignty is an important factor And so this is an important factor of mitigating loss for sure And that's something that for instance protocols which have strong bridges to ethereum don't have because if that theorem bridge is hacked Basically, they have no ability to roll back changes on ethereum to recover that money But with the binance hack in particular the hacker had about three hours to wreak havoc and he didn't So that was almost like an issue of hacker incompetence I think that binance could have lost quite a bit more if the hacker was a bit more competent in that situation Yeah, one of the things that I think tom's a particular expert in is a lot of the d5 vulnerabilities which in a weird way are some combination of software bugs and economic vulnerabilities and Figuring out exactly what's what in that space is difficult The the main value of of having a taxonomy is because you can then identify trends and underlying risks and assess them So I don't know if it's worth trying to figure out which of these assets We which of these risks are more economic versus more Software bugs and how to mitigate those but if you want to talk about some of the d5 issues that you've seen over the last Three years Yeah, have we gone over flash loan attacks yet? Okay Not yet. Oh excellent So I find that very interesting because it's like this novel attack that no one imagined This and this has not not possible in any other ecosystem, right? It's not. Yeah It's uh, and it's new since the last dev con right uh flash loans only came into existence in late 2019 early 2020 And then you had the first uh flash loan attack happened in each other 2020 in february I remember the bzx team being quite excited about their product and then that day they got flash loan attacks and I like to not assume a ton of prior knowledge. It might be worth explaining what a flash loan is and what a flash loan attack is Just as a starting point. Okay. Yeah, so flash loans Flash loans are interesting and actually You probably only hear about them in negative context, but they're actually a net positive for the ecosystem They're actually very useful So flash loan basically allows you to borrow A nearly unlimited sum of money without having any money yourself But you have to do it within a single transaction So at the end of the transaction say there's multiple lines that make up a transaction You have to repay that loan at the end of it So people use it a lot for arbitrage bots or Keeper bots liquidation bots these keep financial markets in order because Coder who has the skills to run these things but not the capital to Can use flash loans to say borrow a million dollars to arbitrage a position across markets into place So I like to say that ethereum's superpower is composability Permissionless composability right the fact that you can build a smart contract. It calls into a smart contract It calls into a smart contract kind of ad nauseam You actually quite quite a few layers deep and yeah flash loans are just a form of leverage, right? So they allow you to Pay a very small amount of interest I mean Probably amounts to thousands or tens of thousands of apy on an annualized basis But because it's for the span of a single block time a tiny amount of interest and just Just just just jack up to a very high level a very high amount of leverage The amount of funds you're moving around and using in that single transaction and that can flow down some of the most sophisticated Flash loans they touch dozens and dozens and dozens of different contracts. They go to you know, 20 30 layers deep It's kind of fascinating to unpick them as a developer very I just want to Distinct make a distinction between composability and and Adamicity because you can have composability over multiple transactions But the atomicity of a transaction is is what is the key distinction here with flash loans Arun you wanted to say something Yeah, yeah, just to kind of like add a bit to that point What's about flash loans for protocol perspective is basically riskless I could lend out any money for my my protocol and get repaid back by the transaction It gets rolled back. It's not repaid. Basically. There's no down to the protocol and it makes the market more efficient but Yeah, I think the issue is it provides like asymmetric ability for Four hackers to kind of wreak havoc. It's definitely like a very powerful tool I remember at one point last year dye was considering Having the ability for people to mint arbitrary amounts of dye in a transaction And then repay it at the end of transaction and you could imagine just Like what people could be capable of they could just like mint billion dye and then repaid another transaction That would be kind of insane at least with the current flash loans if the Guess it's aren't in the smart contract. You can't use them So there's only it's a hundred million dollars of us DC the smart contract I could only use a hundred million dollars to see but if I could literally Create like a trillion dollars of dye out of thin air and then repay it. That's a lot more dangerous Yeah, yeah, one of the the issues is you could certainly do this with most of these exploits would be possible without the flash Lens you'd be able to if you had a large pool of assets Exploit something that requires a large pool of assets to exploit it But the set of people who have a large pool of assets lying around is much smaller And it's very hard to move a hundred million dollars on chain anonymously and then The whole chain of moving stuff to and from and everything else So this opens up to potentially any attacker in the world can can exploit one of these Vulnerabilities and not simply the people who have very large pools of capital lying around So the the key to these flash loan attacks is that you basically have one smart contract system Which is doing imperfect accounting on some other smart contract system and When they're coding it they think oh, nobody could ever possibly have that amount of money to manipulate that imperfection They make assumptions, but those assumptions are not checked. Basically, right? So After flash loans happen, I kind of view it as like the immune system of defy Like if coders are taking shortcuts and they're they're taking these assumptions and being imprecise in their calculations A flash loan attacker will eventually find it and exploit it So it actually encourages people to be more responsible with their coding practices Yeah, we were talking earlier about how much more mature things have gotten in the past two years since debcon four basic Or sorry, she's got five. Sorry. This is debcon six Um, but yeah, I think it's like a fascinating category of of attacks because Uh, they're not exactly bugs like code was kind of working as intended But like yeah, they were able to get like insane amounts of leverage The other way that the flash loan attacks are commonly used is for oracle manipulation So, uh, the most common attack is Say you've got some protocol that uses some asset as a collateral And say that collateral Actually has very illiquid markets and maybe there it's only one uniswap market So they can use the flash loan to just wind up that market to pump up the value of that collateral deposit that collateral into say some sort of lending market and then withdraw Um assets against the value of that collateral and then they unwind that because they have to repay their flash loan and so that Collateral becomes worthless again So that's been another common vector of attack, but that's been There's ways to mitigate that With instead of using a you know, just a uniswap pool as an oracle There's some proper oracle practices to fix that but that's been quite interesting as well I I just want to point out these this this class of attack If you want to call it an attack is really interesting and really fascinating But I think it's very important to make a distinction between This and other forms of security incidents because I am inclined to actually put these sorts of attacks And I'm again, I'm using scare quotes Into the mev bucket, right? It's as you said a moment ago It's code working as it was intended and it's this is not new I have a background in high frequency trading right We just call this arbitrage right it's people finding mispricings or Um, yeah, it's exactly like you said when there's some sort of a mispricing or an assumption That's baked in at the interface it tends to be at the interface of two different kind of systems or something And and taking advantage of that. Um, I'm not sure I'd call that a security incident like an actual Bug in core code or something so it's important to make that distinction And by the way, we should talk more about mev because mev is also really interesting This is why I've been using the word attack instead of hack Okay, cool. Yeah, uh, I just want to make a point that Like more generally speaking because smart contracts are immutable Um, when you write a smart contract code, it's it's very unreasonable to imagine like I have to write this code so it's a cure for the next hundred years against any future innovation that could possibly happen in DeFi Like like all these smart contracts written before flash loans lots of came along You look at smart contracts. Uh, so how do we get around this? Well, it's like really difficult, right? Because uh As we've seen protocol upgrades, um, sure the devs can be responsible and kind of upgraded after protocol in response to like New developments, but there's still like money and assets in the old protocol that can't be withdrawn And a huge class of attacks comes from People just like not being up to date on what's happening and not withdrawing their money from from stale protocol So I'll be one very classic example was when when the whole terra luna blow up happened There were so many mispricings everywhere because everyone had some assumptions about the price of luna On many different chains because everywhere and then people just reading havoc I remember venus mark has got here pretty bad by that one Yeah, I think we see a large number of things where In the traditional security world you you hear about like the zero day versus the the well-known exploit that's been out there Things like the profanity key generator That yes, you if you were the first person like one you should have reviewed it two if uh, you were the first person to Fall prey to that Yeah, that sucks But everyone else who suffered from that and hundreds of millions of dollars lost after the well publicized exploit existed and clear Way to mitigate it existed That is a failure to stay up to date on on threat intelligence and act quickly Which is a problem of not knowing all the components of your ecosystem like your asset inventory and Having a procedure in place for handling this kind of crisis I don't know if we want to get into that more because the luna Not even exploit the luna threat was known for four years publicly And people chose to Ignore that and some people weren't informed on that but I don't know if I don't know if it's worth exploring that but I mean a lot of very prominent folks among them like investors security researchers Had were very vocal from You know months or even years prior About about this class of of of economic attack right because we're talking about economic attack now And in fact, they were warning signs right because there was um There was uh, I don't remember the details, but there was an attack six months prior or something against Um one of the other coins or assets or something in the notary ecosystem And the price kind of it kind of de pegged and then went back to there were like warning signs ahead of time as well Yeah, what I what I found interesting because I've talked to a lot of people about this incident because it affected a lot of people Um So can we can we recap the incident? Yeah, sure So it is a notable security. It is a notable security incident. So um You know, it basically uh, this terror luna Design protocol was basically it had a mechanism to mince a a so-called algorithmic stable coin Where you burn the the sovereign asset? luna in order to Mince UST which is supposed to be pegged to a dollar And the idea there is that you'll always be able to redeem it for a dollar's worth of luna Which then gets minted Um if this sounds like circular logic, it's because it is it is So Many including myself have been vocal publicly about the death's viral inevitability of this design And I don't think anyone expected it to get as big as it did But what I find quite interesting is that um Not only did I would say people who are less informed in the space Who it is not their job to maybe know better, you know, okay, it's understandable that they didn't understand that what happened Uh And they were kind of sold a risk-free thing that wasn't risk-free Um, there was very professional traders And other people in the industry who also thought that this would Not result in a death spiral. They thought it would work and they were in it It's like it's like looking for a perpetual motion machine, you know, it kind of keeps moving until it stops Yeah, so this was I think another this was actually I think more example of like the the hubris of success of the bull market Where everyone was like, oh, we've like solved Like we're soft money financial problems. We've solved the perpetual motion machine. It's going to work It's working and it got just like really big Um, and it was it was kind of just destined to not Work but um, so so something else that's new since the last dev con which and this is actually the the aspect of the terra luna situation That I find the most interesting this goes back to composability, which we talked about before It's the intertwined nature, right? It's that we have now assets built on top of assets built on top of assets And so when one failure occurs, right three four years ago if something broke It didn't cause this cascade effect throughout the ecosystem and actually in the case of terra luna It was across multiple ecosystems, right? I mean bitcoin began crashing and there's a whole reason for that They had this terra luna. Sorry terra foundation guard Treasury that held bitcoin they had to start selling bitcoin to defend it but this um, uh, what's the word for it this? Uh contagion yeah the contagion that spreads throughout like this is a new class of economic risk that didn't exist before I think we need to understand it better because it will happen again But I think contagion has existed in in tried to fight for quite a while people are are well aware of of contagion and But you have something like like central banks to kind of like perhaps contain it and and Limit it which arguably is is not good for the long chromatomic Viability of tried to fight markets and perhaps it's better With d5 for these to kind of play out and just kind of like it's almost like a forest fire Just let the the fire burn everything and then regrow everything versus just kind to keep these like sick trees alive forever Like we have it in web 2 So one thing that I want to kind of touch on with the whole terra luna incident is we got started touched on It's the idea of like the illusion of like security, uh, which was actually very dangerous We had the LFG reserve. We had people like jump and delphi and many prominent people kind of say hey We know about that spiral. We have a handle. We have six billion dollars of vtc We're going to buy everything and it's going to be fine And uh, I think many people looked at it and said okay. Well jump has a lot of really good quantitative researchers I assume they did that kind of modeling. I assume they they Rend numbers and we're like, okay, this is probably going to work when it turns out No one knew what they were doing So how exactly do we do we like mitigate this particular? Class of like issues. I think it's very dangerous when you have a situation where people Know an issue can occur and then people also think other people who are very smart also Know this quicker and we see all type of spark contracts like When alameda jumps into a spark contract I think people tend to assume that because they're in that spark contract They've audited it and they they know what's going on. They put a million dollars into it They probably know what's happening. But in reality, there's like several examples Well, alameda is a theology no spark contract for yield and there's been critical bugs in that spark contract Which is just kind of insane And it just kind of goes to show you to like yeah You really cannot rely on like reputation authority in this space to make your decision making and that makes things Very difficult I think for the average person working in d5 because people who are not experts in the systems Rely on experts and those experts have been showing time at a comment again to Not be doing proper due diligence on their own There's a range of attacks that have happened where the system as designed was a good design And even as deployed initially was a good deployment There was uh forget which bridge it was that had nine separate validators that they collapsed Uh ronan ronan. Yes. Sorry ronan. Yes. Yes. Uh, they collapsed it down to smaller than the the blast radius number and well they so there was four Four validators controlled by the foundation and five were independent And then there was basically it was a usability problem. So Withdrawals were happening too slowly. So one of the by the way the for run by the foundation were like all in the same AWS machine. So they were basically one and then They convinced a fifth one to give them like Temporary permission to like sign for them. So basically five of nine were running on one machine which got compromised Yeah, so that was like, okay, we started with good practices and then like We The I think the term is the normal normalization of deviance. It's a concept from like aerospace engineering So one of the interesting things about crypto is and a lot of people I talk with come up with the same thing Is if you're deploying an immutable smart contract and there's reasons why you want to be immutable protection from forced upgrades Malicious insider or everything else, but if you're deploying an immutable system You're really in the same safety regime as aerospace nuclear engineering everything else Not something similar to normal software engineering and you need to have procedures in place for that kind of Ecosystem where traditionally you have Way more design work than you have implementation Lots of review you have review of your review process feedback on that and it's very different than normal software engineering And it's not really how stuff is built in the broader ecosystem Super briefly as a developer to speak to this This is really hard, right? Most of us when I say us, I mean core developers also application developers We don't have this background or this training These practices have begun to spread a little bit through the industry and we do now have folks Contributing to projects like Ethereum who have backgrounds and things like aerospace But we need way better education around this stuff. I really strongly agree with that To that point And to an earlier point about tradfi In the tradfi world There's something called ratings agencies. This doesn't exist in crypto and in tradfi You pay professionals to evaluate the risk and give it a rating of a financial asset Not even a security not even security wise. There's also security ratings and certifications as well But so we're missing that in crypto There's no professionals that are actually actively monitoring Even the economic risks of these very systems The thing that defi has that tradfi doesn't have is real-time transparency and auditability So if you look to the 2008 housing crisis That was basically, you know A series of Packaged and repackaged and repackaged debt instruments that Eight layers deep nobody could see through to what the underlying was and they just trusted that the ratings agency That gave it an a-plus that everything was a-plus turned out there was a bunch of You know f through a Debt within that package Which caused people to basically buy risk that they didn't know they were buying With defi we have that real-time transparency, but what we we're lacking right now is Basically professionals that evaluate that risk and give ratings on it or give metrics on it so i'm seeing The beginnings of that happening i'm seeing a couple of projects that are focused on becoming this risk-oracle risk engine So i'm excited for that in the future because you know, we will combine this transparency and defi with The the various ratings mechanisms or risk evaluation mechanisms that are coming out massive massive massive opportunity Seriously anyone out there who's interested in this stuff who has a background in finance who's inclined This stuff is sorely needed and there's huge Lack of the stuff in the market today. It's a huge opportunity like work on it talk to us We'll we'll help you make sure you're talking to the right people Yeah, actually funnily enough Like yeah, yeah, so so anti which was the project i was recently dev on it It's essentially trying to do this in a in a decentralized way Um, I think that one of the issues with having a kind of like centralized risk auditors Is the fact that they are felt when i'm saying decentralized risk auditors are perhaps better It's just the entire system as a whole. I think Is fallible to some level of human error and any kind of appeal to authority. I think is Kind of dangerous. I mean just because we have moody's this p-rate doesn't mean that tried fi is is immune to like contagion stuff like that like I mean It it it definitely isn't like like these these companies I guess like The issue the thing you haven't tried fight of in defi and so i'm kind of rambling at here is you have this kind of like fail Safe mechanism the central bank was able to kind of rescue everyone in case everyone gets a bit out of control Uh, and I am not sure if like moody's makes things like that much safer It's probably better than than having nothing at all But it definitely is not like ideal solution and probably is is not going to work that well for For defi where we have no safety net. We probably need something a bit more robust So the big problems with rating agencies is that they don't have any skin in the game So what i'm really bullish on is insurance Uh, and other types of systems where they're underwriting the risk rating that they're giving um So there's lots of different ways you can do that whether it's with a regular insurance company, uh, which is whatever toss does or uh various defi mechanism designs This is something that i've researched very deeply and, uh, seriously if anybody wants to work on this Come talk to me because, uh Yeah, they're i'm aware of some very interesting things there Yeah, we do underwriting for a lot of problems for technical risk Projects for technical risk and things like that and the problem of how much can you trust a software audit or a software security review auditors actually hate the term audit when applied to their their work product, uh, but, um How do you trust different firms and honestly? Every firm i've talked to can do great work the the thing from an insurance and a reliance perspective is You really need to look at the The worst work that they'll pass and make sure that that is high quality because if you're going to just blanket trust the security review performed by a firm you really care about their internal quality control process and Knowing that if somebody it's a new system where they're not already expert in it or where perhaps they don't have They're they're very busy. One of the biggest problems in the security review industry is it takes three to 18 months to get queued up for a security review at a lot of these firms and as a result There's a huge pressure to ship And then get your review done while it's in production or get your review scheduled before you're finished building it In which case you haven't reviewed the system as it's deployed So one of the critical things that we absolutely as an industry need to improve is better tooling but also more security professionals in the industry so that we can do more of this stuff in parallel because every project out there needs to have this work done and it's a contented resource and You you absolutely want to be one of the ones that gets a security review and not one of the ones It's waiting while you get popped I don't know if we want to touch on something that we've kind of uncovered which is that technical auditors are basically Expected to take on this role of doing an economic audit as well And they're actually not the best people to do that, right? They've they've categorized the known economic attacks, but For instance, the mango market attack, which just happened yesterday was a blatant financial oracle manipulation attack They manipulated if you're not aware mango got drained for about a hundred million the attacker increased the price of the mango asset by Manipulating the price of mango across various markets and then borrowed better assets than mango against that unrealized position and like the A technical auditor could have looked through the mango code and the mango team is super smart And and you know everyone involved like as you know knows what they're doing but like this was a This is a very You know weird edge case that somebody with a financial risk perspective Would have been the only one that would have caught that just to be clear There are folks who specialize in the financial side of things. So shout out to gauntlet. Shout out to block science, right? They have tools They're experts in this but those the only two I know of whereas there's dozens of auditors who are like you're describing technical auditors Who I think would not have this sophistication to catch this category of bugs. So again Yeah, yeah Yeah, so so one thing touching on the economic audit I do know some auditing firms that now do this kind of auditing separately and actually there are Like for instance, kampan avay will will retain people to do ongoing economic audits And I think it's very necessary to be able to adjust the parameters of your protocol Uh, it's subcategorized way, of course to would to adjust for the changing Kind of parameters of like what's actually in that protocol because otherwise in a situation like for the mango markets Um, if you're not aware, this is a very capital attempted attack It took I think around five or ten million dollars of the usual capital to do that attack And the thing is like I think most people looking at mango markets are like Oh, this protocol has ten million dollars in it when it's audited its oracle attack is going to be very expensive pull up It's not worth it. Once that's a hundred million dollars in it Doesn't this oracle attack come very viable So you actually need like a real-time update of these parameters over time to maintain the safety of the protocol And that's something that's kind of missing except with like kind of like a tier one or s tier protocols in deep eye right now I'd say another interesting thing that's happened over the last three years is the rise of the Consumer nft space which is bringing a lot of new people into the space that that really are they're they're from an art background They're not people who are software engineers in a lot of cases who think about systems failures If even if they don't think about it from a security perspective, they think about software bugs So there's a lot of people who really don't have the The background or the the experience to to know what to be afraid of just sort of like naturally And we haven't built as an industry the right tools to make it safe for people to do things without understanding All the details of how it works like there's a lot of people who drive cars who don't know that much about how Airbags work and seatbelts and everything else. They just know if you do it. You're probably okay within certain parameters We are nowhere near that level of Professionalism and reliability in the in the space and we'll see all these People that are that are new to crypto because of they've come in through the nft world And they fall prey to the same issues that that happen to early regular crypto users Ten years ago and are largely mitigated by community knowledge and tooling there So it's kind of scary that there's the the high end of these d5 vulnerabilities Which are new classes of attacks, but some of the most basic things are happening to a new class of users right now as well So I think a notable security incident along these lines is like owners of board apes Revealing their private keys to social engineers on their discord and losing their apes You know, I think you're right Ryan I think like it's incumbent upon us as designers and builders in this space to like build better tools and and Install some of those safety rails or airbags like you're talking about But I just want to highlight that as someone who thinks deeply about this and has been working on this for years There's a tension here, right because the easiest way to install safety rails is centralization Right is to say, yeah, you know for my grandma like maybe she's not going to really effectively manage her private keys Maybe she's not really going to understand a ledger or a multistake So maybe the best thing for her to do is keep her crypto on coinbase And and outsource custody to someone else and that in some respects is antithetical to like the whole reason We're here. So there's no easy solution to this, but I just want to highlight that tension And that's where I approach this as a protocol engineer and as a developer Yeah, I think kind of a good balance between this is that the base protocol should always stay true to like the tenants of crypto It should be decentralized such a persistent permissionless all this stuff So Bill can build one of that on top of that and I think that It's kind of on us to educate people on the right way to onboard themselves Like grandma should not be probably creating a metamask wallet. I think that's generally something that is is Difficult to secure I think for the vast majority of people and in the current situation as we see with the entity space that Like there's no way we can onboard a billion people into crypto with the current state like people Like it's just like it's just going to create like a feeding frenzy for the piranhas that are constantly looking For ways to steal people's money. So we need to figure out these more robust wallet systems for these people And I think that's probably a better compromise We onboard people through like let's say account abstraction wallets or or some kind of like non custodial wallet solution with the hypervisor protocol layer which manages signing and policy networks And then we keep kind of like the more bare metal like quote-unquote riskier but more permissionless wallets for people who need them But one of the things that scares me is While these these losses might be relatively low a small dollar values relative to the larger defy particle hacks and things like that The number of victims is very high and politically if you have a lot of victims that individually like a loss event of any kind and It makes it very easy for regulators to point it at like a grandmother who's lost something or otherwise Unsophisticated user who fell prey to some some system or possibly was scammed and then pushed for regulation So we have the centralization risk as a as a the potential to use centralization as a counter measure to a lot of This stuff is a risk of centralization. But also Acquiring a very very bad Public relations nightmare as well as regulatory nightmare is also an existential threat to the overall industry Cool. Do we want to open up for q&a? How are you? Excellent. Well, you talked a little bit about account instruction So I want to know if you could Talk a little bit about that because I think that it's Like the solution to have like easier custodial systems without depending on I mean self custodial systems without depending on centralized entities So what do you think about that and how we are advancing? And when do you think we will be able to have that for the general public? So I I think there's like two classes of account instruction So just to define the term very generally kind of abstraction is Essentially, um, when you make no distinction between smart contracts and individual wallets on Ethereum right now Individual wallets are fundamentally different than smart contracts because individual wallets are our public private key pair smart contracts are Are derived very differently and I don't necessarily have time to go into that right now But in a kind of abstraction, they're basically the same smart contracts individual wallets are smart contracts And we have the ability to do that right now with things like like multisig which obviously are are somewhat valuable And in the future we may eventually end up doing doing away with the distinction entirely So people with roll-ups were trying to do that, but eventually ended up deciding not to because You know making these kind of protocol level changes just made it very difficult for a dream developer to kind of compose and build on them Um, but nice thing about kind of abstraction is design space is is very wide people could do things like multisig They can do things that are much more complicated like have like off-chain oracles and you know Involve human beings in the process and do basically kind of the sky is the limit in terms of what people are kind of building here One interesting project that's kind of Actually never mind up Yes, all right, that's not quite it. Yeah, so we built this for space mesh. Okay. We built account abstraction I think we May be the first layer one. Maybe the second near has something similar It was really really really really hard to your point. It had very deep profound implications and ramifications on the protocol But it's a very powerful primitive and it means that The really simple way to think about this is every account by default as a multisig and there's always multiple keys associated with it Again, this is also the case in near so shout out to near for for kind of like pioneering some of this work It's a very important primitive and I think it will go a long way towards Addressing some of the kind of classes of attacks we discussed here So I would like to ask which security incidents in your opinion was The most impactful not in terms of value lost but in terms of sending ripples across the ecosystem I'm okay. The first one that came to my mind This actually touches upon a point that ryan made a moment ago Which is that it's not necessarily as you said as well about the total amount of funds But it has more to do with the the number of people impacted and the sophistication of those people There was a hack in a salana ecosystem wallet about four or five months ago Does anyone remember the name of the wallet slope wallet? Thank you And what happened here was a bug in the wallet that exposed the private keys because of some telemetry kind of tooling Very common. I think it was called century or something. I think we actually use it in in space mesh as well And the reason that I find this one so impactful is because this was the actual total number of funds was very small And I think it was eight or 10 million dollars worth or something So it was like maybe even less than that But it was thousands of users who were impacted and each of them had a very small amount of money in their wallets And they woke up one day and their funds were gone That I don't think has ever happened before that particular class of attack and everyone was panicking and no one knew Yeah, it was a very scary attack Even scarier about that is no one knew which wallet it was because a lot of people were reusing Seed phrases across multiple wallets. So they thought it was a salana base level vulnerability or something They wasn't even localized to a wallet. It was very scary watching that in real time Just psa because apparently it needs to be said Do not log private keys into your logger And and also use hardware wallets. Okay. This this class of attack is not possible with a hardware wallet that too I think probably the most impactful On the space has been the combination of the ust collapse which led to the three arrows collapse Which led to an entire collapse of the crypto credit markets And has really raised a lot of red flags with regulators like we've seen a lot of Regulatory pushes because of that combination of ust and three arrows And that has impacted a lot of You know non-crypto native folks who are sold a risk-free usd interest rate So it did happen to be a large dollar value, but it's having heavy political ramifications For the final remark Plus one for ust Plus one for salana Thank you. This was really I learned a lot. I hope you guys did too Thank you very much that everyone else on the panel