 Hello, my name is Michael Llewellyn, and this is a talk on Ethereum hacks and how to stop them with opens up with this is a re recording of my talk for dev con 29. And I hope all of you watching this virtually will enjoy it just as much as the people that did in person. So without further ado, I'll go ahead and get started. So, the agenda that we're going to go through is going to be about a 60 minute talk probably a little less since we won't have q amp a. Since this is a virtual talk, the introduction will be going over my story and blockchain how I got here. A little bit about open zeppelin which is the company I work for and a lot of tools will cover will be open zeppelin based. Some of the Ethereum security case studies that we're going to go over are going to talk about Ethereum hacks that we've seen in the space. Some of them old some of them new reflect on the what these incidents made possible what we learned what the community ended up coming to agreement on and then explain how we now can prevent these incidents using tools with open zeppelin. So without further ado, let's get into it. So first of all, my story and blockchain started out in mining. I was actually in school when I first learned about Bitcoin. So I started mining on school computers made a little extra money, got very interested in it at that point because I suddenly learned how to print money with the computer so naturally I got very curious went down the rabbit hole of crypto economics and technology and everything I was just fascinating about the space. And once I started getting into it I started to actually go to local meetups here in Dallas, Texas, and began consulting professionally as soon as I started to meet people that wanted to learn more or needed help setting up minors or doing security on their wallets that was the sort of thing that I ended up providing as a service. So we, once that happened, you know I did some traveling you can see me here at the Bitcoin Center in New York City before I got the beard, and then basically in the blockchain space ever since. So over time, I started to learn more about other applications for blockchain beyond just the money in Bitcoin. I still very much love Bitcoin but I also saw the need for things like smart contracts and Ethereum ended up coming out around the same time that I was studying for my computer science degree. So as I was learning computer science also started to learn how to program smart contracts on Ethereum, and then started to do that professionally as well as a developer so I did some startup work, worked on some ICOs as well as some startups doing supply chain here in Dallas. And then eventually started to also do some corporate consulting so I worked at Thompson Reuters doing tax accounting. Did some supply chain at Toyota using an Ethereum blockchain and then also did some consulting at Verizon on some high culture fabric instances so got got around a lot did a lot of work in the blockchain space both startup and corporate. So during that whole time I was using opens up on contracts so this is a library will talk about more detail but as I learned more about it. In the last year I actually ended up applying to work at opens up one. After doing a lot of consulting I wanted to work at a single company and opens up and luckily accepted so I joined the team this year as a project manager for their security services team. And you can see a picture of us there at a retreat in Mexico recently and we all get to meet in person for the first time, or at least I did. And then so in addition to opens up one I've also been working in the space as a lecture at UT Dallas teaching a blockchain course and then I've also been doing some public policy work with Texas Blockchain Council trying to basically help the Texas government make blockchain technology easier to use and you know clear up regulatory technologies and things like that we've passed some laws and in the recent months, got some good traction going and hopefully that will continue. But we're going to talk about security so without further ado, let's talk about open zeppelin and the work we've been doing. You can see we've worked with a lot of companies in the space we've done around for years, working coin based compound of a continue have a lot of relationship with these customers either using our products or auditing services. I think we're very well known in the space and this is going to be a talk that gets more detail on like, you know, how our products and services are really reflected in the systems. The hacks that have occurred in the ways that we've addressed them. So without further ado, let's get to the actual tech the hacks the reason that people come to deaf con to learn more about what are these vulnerabilities and how can you protect against them. So the Dow hack is the first big outs the first real instance where we learned that security on smart contracts is not going to be easy. And in fact it can be quite destructive if not done well. So, many people know about the Dow but for those who might be new to the opinion space. The Dow was an early set of smart contracts built for crown funding a theory and development projects. For those who might not know even about Ethereum smart contracts. These are essentially pieces of code that reside on the Ethereum blockchain and Bitcoin, you know, people move money around using private keys you have a public key that kind of corresponds to your wall address and you have a private key that you alone control that lets you send money back and forth. So, these contracts actually let you do that with code so instead of there being a private key that controls funds. There's actually a bit of code that determines how those funds are going to be handled so you still have a public key you have a public address that corresponds to a smart contract, but the code will actually determine how funds that are received are handled and then potentially send elsewhere. So, you can use these smart contracts to do things like escrow, or even manage a large amount of money for a kind of decentralized organization so the Dow stands for decentralized autonomous organization and its purpose was to be kind of a venture capital suite of smart contracts that helped fund development projects in a way that was kind of like crowdfunding with Indiegogo but with decentralized nature to it. So, this attracted a lot of investment in the acquiring millions, you know, millions of dollars of Ethereum was deposited in these smart contracts, and that ultimately led to it being quite a large money pot. And so on July 17, 2016, the Dow suffered a hack that resulted in the loss of 11.5 million a theory, either, or $50 million. And the attacker was able to achieve this by exploiting multiple vulnerabilities, including a re-enter and see bug that let them recursively call the smart contract and pull out funds repeatedly before the balance was actually updated to reflect the initial withdrawal. So this basically let them lead the contract dry. And they were ultimately able they were you know the funds were locked up for a little bit. And during that time, this caused quite a stink because this was a significant amount of Ethereum. And at that time, so there was actually a decision by the community and the Ethereum Foundation team to effectively fork the protocol and use that fork to return the funds to the user and essentially reverse the hack. Unfortunately, this was very contentious and a lot of people in the Ethereum community were not happy about this because it kind of went goes against the core tenants of the blockchain to be censorship resistant for transactions to the final. And this split led to a fork of Ethereum classic, which was a version of Ethereum where the hacks were not reversed. It stayed the course and that community continues to this day as a separate spot off. And so that did that did cause a lot of issues that the hack was reversed on the main Ethereum chain or the chain that most people recognize as Ethereum today. So to give you an example of what this really looks like what is any re-intrancy bug. Let's look at this example so you can see this contract as a kind of kind of an example of what happened it's a simplified version of the Dow, where you have this withdrawal function. And if you look at it line by line notice that the first line is just going to check the balances of the message sender which is the person calling that function so it's basically saying, you know, is the amount that they're trying to withdraw. It's lower than or equal to the balance they have in the smart contract. And what it will do is if that condition is correct, it will then call that smart contract through the fallback function, which is important, and send that value to the contract. And so it's important about this fallback function is you're effectively giving control to whoever's calling the smart contract and if it's another smart contract and that smart contract can actually execute code against the original victim contract. So as an example of that, you know, let's say that we called it with this attacker contract and then the attacker contracts fallback function is then triggered when the funds are received. And because you notice that the balances at the bottom of the victim contract is decremented so the amount is withdrawn from their balance after the amount is sent. And if you're able to essentially call the withdrawal function before it returns to decrement that balance then you can actually call the withdrawal function again. So it goes to essentially, you know, the attacker contract recalls withdraw function. Once again it goes through and checks the balance and the balance hasn't been updated so it thinks it still has an amount to withdraw. So the funds are sent to the attacker contract the attacker contract and again calls the victim contract and this essentially repeats until the funds are our blood dry. So this is a effectively how the Dow was was hacked with this range and see both. So what were some takeaways from this. And this was this was monumental this was a massive hack on the Ethereum space where, you know, prior to this I think smart contracts were kind of seen in a rose colored light of. Well it's on a blockchain therefore it must be secure but what we did learn is that ultimately our contract developers need to write that code securely. You know logic bugs are going to be immutable. If you do actually deploy that onto a blockchain and you also need to account for malicious calls from external contracts and accounts so even though, you know if we look at the code for the Dow on its own it seems secure it seems to be accounting for the balance correctly. We're not accounting for the fact that we can see and other external calls can potentially do things that are not expected. So ultimately there needs to be a lot more considered with a smart contract rather than just the code itself. We also learned that when you update state and balances you should do it before you send funds or perform external calls because if those external calls do something you're not expecting then the balance. You know they might be able to do more things that the balance should be updated prior to. So, you know that was another big thing we learned to make sure to update your balances prior to sending funds. We also learned that forking a protocol in this case is is possible to reverse an attack you know that is an option, but it's, it's not a great option it can heavily damage the community, and the platform's reputation for censorship persistence so it's something that really should be a last resort and the Ethereum community has not done a hard fork like this ever since there had definitely been calls for it when other hacks have occurred. But this is really kind of a one in done thing we haven't seen any other community hard forks to address a hack. You know, anytime after this and I think that's probably for the best because it does come with a lot of risks that that become the more common occurrence, which would potentially reduce the censorship resistance of that protocol. Finally, but of course, we need better security standards and more secure celebrity implementations to prevent these exports so a lot of this code that we better that is being written, ultimately needs to have a higher level security standard that needs to go through checks and tests and most importantly, we need to use implementations that are already battled hard and threatened to try to write our own code. So, how can we do this well, we can use the opens up and contracts library. This is a battle tested open source library smart contracts on Ethereum, or for Ethereum and other EBM blockchains that is available on GitHub, as well as a website and many other places and as an MPM package so really what this is is just a way to say hey if you're going to create an ERC 20 token. For example, or 721 token, you're going to want to do that with the set standard, instead of trying to write that implementation yourself and open yourself up to make a mistake so there are hundreds of thousands of smart contracts out there. Well, I don't know about hundreds of thousands there's a lot of smart contracts out there. The vast majority of the projects for example, use the standard of code and use a smart contract implementation, primarily because it's so battle hardened that there's an expectation in addition to that you get additional functionality you can make your, your tokens possible you can create snapshots, and a lot of other extensions that are available, kind of as a as an add on that are very simple to use. You can also add in proxies for upgrade ability and cloning that makes, you know, upgrading your smart contracts possible so if there is an issue that comes up with a smart contract or you want to get change. You know smart contracts are immutable by default there's no built in functionality to solidity or theory to upgrade your contracts but we do have logic that makes it possible. We also have the ability to provide access control who owns the contract what roles can people play in minting tokens or burning tokens or pausing tokens there's you know you can create a whole access control paradigm using a lot of our contracts. Governance which is kind of extension of access control people can have time locks on certain functionality you can have potentially the ability to create governance protocols compound governor, like the compound protocol something that we're implementing right now. We have utilities, you know basic stuff photography math create to which lets you do deterministic generation of smart contract addresses data structures things like that, and upgrade plugins which just help you do upgrades directly to the smart contract to hard hat or truffle. And then finally security module, which is the thing I'm getting to for how we can address things like a reentrancy by using some some modules were created specifically to ensure that you cannot be exploited by the same sort of bug that exploded the Dow. So to give you an example of this for preventing renters exploits we actually offer two options depending on what you're doing. The first and the most simple is a reentrancy guard it's essentially a function modifier that prevents reentrancy so all you need to do is put that modifier on your function. And whenever your your function is called by an external contract it prevents that contract from being able to call any other functions inside of your, your smart contract so essentially it just makes reentrancy impossible. The other option that you have in the case of payments, because in the case of payments, for example, if your contract, if you're calling a contract that is a rather you're sending funds to a contract that needs to update any token balances maybe that, you know, balance relies on your smart contract and there are cases in which they have to call your smart contract in order to handle that transfer. You have a model called full payments, and some modules to go with it so essentially it's a reverse pattern of a push payment, which is essentially what happens when you make a transfer to an external contract directly from your contract. In this case rather what it does is it says if you have a balanced draw, it takes those funds and sets it aside in an escrow, and then that escrow can be withdrawn by that contract any separate transaction so essentially, because it's in a separate transaction there is no possibility for reentrancy in that case. And that's that's the advantage there so that's that's a separate model that just provides better security against things like reentrancy. So both of these are options with the contracts library and there's a lot more details where that came from. You can check out the blog from from Francisco that talks more about we can see after Istanbul and the current modules that we have. So that was the Dow pretty pretty famous but let's get into an even messier hack, one that really I think broke the assumptions that there are coders out there that can't make mistakes. The parody multi sick hack that proves that no one is perfect. So the parody multi sick hack occurred with parodies multi signature contract library so parody is a well known company in the blockchain space. At the time they had a parody node that was essentially an Ethereum node, or for syncing with the blockchain and the node came with the idea of a basically contract library that was multi signatures so if there were funds that that node controlled, you would create a multi signature library that allowed multiple keys to have control of those funds. And this was all deployed as a smart contract on the Ethereum blockchain. And what this did is essentially let users create their own mythological contract. But that contract would be dependent on a core library or its logic. So, for example, you can see these two contracts below owner 123 owner 456 both have different balances both control different funds but the logic that they would use for some withdrawals and many other things would all depend on this one library contract that they essentially are dependent on. And the reason for this is that you can save a lot of gas costs, you know you don't have to deploy the same logic over and over again. You can also let you know that team wants to do updates to library contract it would automatically apply to the contracts that are dependent on it. So you have a lot of advantages with this, but there are also some vulnerabilities. So an attacker was actually able to find a vulnerability on July 19 2017 and stole over 150,000 of their ether tokens or $30 million from free high profile multi-sig contracts that were storing some funds from token sales. And so this was very significant hack the attacker was able to do this by obtaining exclusive ownership rights to each multi-sig by calling an initializer function inside the library using delegate call. So we'll get into that in a little in a second in terms of the details of that but basically because of the dependency in the library contract and was an exploit available. So there was actually kind of a follow up to this. It wasn't a hack but rather more of a kind of an accident more like where they deployed a new library contract. But there was a bug in that where anyone could call the self destruct function that will destroy the library contract. And that actually caused $150 million worth of funds to be locked up on November 6 when that that function was called because all of the multi-sig contracts out there to dependent on the library contract for its logic no longer had the logic to do with draws the funds were frozen. So this caused an additional loss on top of the initial hack. So very damaging and I'd recommend reading the post mortem from parody below to want more details on that. But to give you an example, you know what that hack looked like and this is actually the code from the hack summarized. You basically have this library contract and it contains an initialized wallet function that lets you initialize your wallet with the owners and requirements and time walks and things like that. And the idea is that this should only be called once it should only be called after the contract has been initially created and then no more. And that's the way it was it was set up. But if you look in the contract victim, and you see this fallback function there, you can actually see that there's a way potentially to call it again. So if you go through you can see that initially has the logic to handle a deposit and then otherwise if there's no deposit or no value attached to that contract or that payment. You see that essentially it says if there's any data contained in the function call, then the delegate call function will be called on the wallet library and it'll pass that data onward so basically says if there's a function that I don't know about. You know the contract victim doesn't know about it'll pass that on the library contract is they hey do you know this is a function that you can handle etc. And the attacker could basically use this to call the initialized wallet function. So he basically, you know gave it the attack his own address and then called the multi contract victim and it delegated that call to the library contract and allowed him to set his own, or him or her allowed to set their own themselves as the owner of that contract and that's essentially how this has occurred. So it seems very simple but it was missed because of visibility. It wasn't clear that the initial wallet function was visible to the contract victim, or rather the dependent contract, even though it really shouldn't have been. So, what did we learn from this. So first of all we learned that anyone can make a mistake because parody was founded by Gavin Wood, who's a former CTO of Ethereum, and the creator of solidity and the EVF. Of all people should be aware you would think of all the pitfalls of using Ethereum smart contracts wasn't always clear exactly how involved he was but he definitely this was his company his team. He probably looked at the code at some point. So even the most reputable that all of us can ultimately make mistakes in these in this regard so it was, you know, definitely a great wake up call to say that you know no one is perfect and we ultimately need to be able to have processes and standards in place to make sure that you know any developer can fail when you to catch that before something like this can occur. The other thing we realized is that functions defaulting public visibility are a serious issue like contribute to exploit such as this one. And so there's actually a change to solidity and version 0.5 or down it's required that you specify the visibility for all functions so it won't compile unless you say if it's public or private so that at least it's clear. Because you notice if I go back to this there was no, you know, function, there's no visibility modifier added to say what the visibility should have been wallet so it defaulted the public and then that made the attack possible. So the other thing we learned is that complexity can definitely add to security risks. So using libraries and delegate call and increase that because there are definitely more scenarios that can occur. So it was trying to save gas but that ultimately led to a vulnerability being possible so it might be better to simplify your code rather than try to make it more complex just to save gas or optimize things. Optimizations are still important, you should still do them as much as possible but you should always know that there's a potential trade off to security if you're adding complexity to it. Another important thing is that no bug bounty exists at the time to incentivize reporting of critical issues. It's not clear if this would have prevented the hack if the hacker would have preferred to be paid by bug bounty rather than do the hack. But, but we know that this has been successful in, you know, current times to catch a lot of bugs in production. So ultimately what we learned is that a bug bounty is is essential to catch it by this, you know that was an option that might have prevented it we're not sure but it's one that should have at least been available at the time. And then finally, another thing we learned is that audits are important and not just important but but the need to be very clear and eventually need to be recurring. There was an audit on the multi-stake wallet at one point, but changes had since been reintroduced before the attack so it was very unclear which commit had been audited. So it's really clear that we have recurring audits on code if it's changing over time. And that there's clear get logs to ensure that we know what was audited in with. So naturally with open Zeppelin, we do security audits and we definitely encourage all of our clients to work with us on having a clean get log to make it clear what was audited what was patched what PR what commit. So it's very clear to the community and to future audits, you know what what should be considered potentially secure and what has not yet been checked. And we do these audits with many teams and many companies out in the world and we know that there's also a lot of other audit from out there and we would encourage everyone to get audits if not with us then at least with someone. Another thing that we learned, or rather one thing that we are learning at open Zeppelin is that there are limitations to audits, not because we're not good at our job, we feel very good at our job but because we're ultimately there's only so many of us and there's only so limitations in the entire ecosystem so having a quality assurance process in place is still critical because ultimately the messier your code is the more bugs that there are contained in it. When we initially go through, especially if they're simple, obvious bugs, the more time that takes away from us to do our job and so we highly encourage all of our clients, and anyone in the industry anyone deploying smart contracts to go through a rigorous QA process within your process and build that in as a requirement for any code that goes to production or even goes into an audit. You should be doing this regardless of whether you get out so you should still get audits, but you should absolutely have a QA process that catches this many bugs before the audit even occurs. So maintaining a disciplined get workflow with transparent log, having cold requests that are approved and merged with clear understanding of what was changed and when. That's very important. Having that's not sec comments to all contract code and keeping it clean so we want to make sure that we know, you know what that code is doing having an understanding of the arguments and the return values that makes it far easier to read. We want that you need to be documenting the architecture and all the protocols that's another thing that helps with an audit as well as any community members trying to understand how to use the smart contracts correctly. Sometimes there can be bugs that occur, not because the smart contract is has above but because there are users that under misunderstand how to use that smart contract and the cause of the loss of funds, or other issues to occur so documenting everything is crucial. Also running an exhaustive test suite is important if you make any changes you're able to test against that. It also makes it clear to both community members and developers and auditors what your code should be doing or what you expect it to do. So having that one person code coverage or at the very least close to it, another very important step that we want to, you know, we would encourage a lot of projects and protocols out there to follow and adhere to also using code libraries you know opens up when is obviously one of them but there's also a lot of other, you know contracts out there that people commonly use such as compound governor Bravo, or the genosis. And we think it's important to try to import those as dependencies through MPM, or something similar rather than copy pasting the code just because copy pasting the code doesn't make it clear what version you used sometimes that can be bugs introduced. So when you're auditing code you have to take a closer look at that versus if you're pulling it from an external source that we already know and trust then that always makes it easier to audit cut down on time a bit and it also just is best practice. And then finally just having your own internal security reduce you know take time exclusively just for digging into the code and making sure it's secure, trying to break it yourself. So I think just having regular times where that's the only thing you're doing, or rather you know you finished a big implementation. Try to dig in and see if there's anything you missed that we're really cut down the box we've seen a lot of teams that will catch bugs prior to an audit that we start with them when they're they're giving more time to do that. And then run bug bounties regularly like definitely have that available for the white hat hacker out there to make money on pointing out full abilities your code rather than opening the possibility for someone to say well, I'm only going to get profit. I'm only going to get paid for finding this by hacking it. You never want that to be the only option they have so definitely keep a bug bounty program out there. So with that being said, let's stick into the last hack we'll talk about today. This is a more recent one the urn why divulge exploit. This is your example of flash loans and how they are coming for everyone's collateral a for not careful. So a little quick primer on flash loans. These are a new type of uncollateralized defy lending protocols where liquidity is borrowed instantly and we paid in the same transaction block so this is as opposed to other lending protocols out there were typically need collateral. So, for example, if I wanted to, you know, borrow some, some funds from maker Dow I would need to put up some ether or other funds as collateral and then I can borrow against it usually it's it's actually over collateralized. In this case, they will you can do with the flashlight because it happens in the same transaction block you actually get guarantees as a lender that someone can pay you back. So for example, if a trader wants to flash borrow an amount just to make a trade, you can borrow that amount make that trade to some sort of a defy protocol or or or decks and then you know take some profit from that, and then return those funds, plus whatever you do to that protocol, all within the same transaction block so the idea is that if anything happens along the way, or anything could happen along the way that would make it impossible for the trader to repay their loan in full. Then the transaction never happens or gets reverted and essentially the lender has like a guarantee that they won't lose that money. So this is a really awesome tool for traders and people that want to use more financial products out it also means that people have a very powerful tool set in there. A very powerful tool to potentially make profitable hacks by conducting economic attacks so this was first introduced by Ave in early 2020. So this is still a fairly recent development, but it's definitely gained a lot of traction recently there's other protocols that now support it. We now see a lot of exploits using flash loans so this effectively what they'll do is they'll be able to borrow a lot of money and use that to manipulate on chain pricing by you know unbalancing a pool or a dex and kind of messing with the pricing mechanism. By just having so much collateral that it that it kind of overwhelms protocol kind of like just a whale attack except anyone can be a whale with flash loan. But what these can do is they can you know take out under collateralized loans as a result, they can trigger liquidations and then also increase the profitability of other exploits so they find an exploit that lets them, you know, but you know, basically mess with the balance of the smart contract or something like that, if they can unbalance pricing they can potentially increase the profitability of that. So let's look at this specific exploit with your finance and urine is a lending aggregator what they essentially do is they optimize token deposits with pools, including the be one wide I vault, which utilizes another protocol curve finance for strategies to so to give you an idea of this you have the urine wide I vault. Essentially, it's looking at the curve free pool which has three different stable coins in it. And it uses that to understand the pricing that it uses in its strategy specifically in how it calculates deposits and withdrawals. So what the hacker was able to do is call the loss of 11 million dollars to the vault, while making a profit of 2.8 million by doing several steps that mess with the price and allowed them to make essentially a loss deposit some withdrawals. So, essentially what they did is they just made those deposits and withdrawals and an incredible exchange rate and after 38 minutes, the urine team was able to mitigate the attack by disabling deposits and saved about 24 million other 35 million die. It could have been a lot worse that they weren't able to do this and saw it immediately. So basically what the attacker was able to do, they were able to receive a flash loan from the lending protocol imbalance three pool using that flash loan so essentially causing a imbalance that caused the price to be wrong. And then they made deposits into the urine wide I vault, and then withdrawal and essentially, you know, got it added unfavorable unfavorable price. And then they did a better breakdown of this is a diagram built by Steven from our team. Essentially, you can see the attacker depositing funds so they essentially deposit to the stable coins withdraw another stable coin tether that causes an imbalance and they deposit die into the victim vault. And because it's using the exchange rate from three pool. It does a deposit and causes a loss and then the attacker is able to do the deposit back in a free pool with the US DT they had originally taken out withdraw the die from the vault at another bad exchange rate and then withdraw US DT in the last election so essentially through the whole and then and then they do an exchange on on m curve to withdraw us DC and in the die and in the end they they've ultimately at a very large profit and the victim vault has lost out. So you know this shows me and they were just able to essentially keep repeating this I think they repeated about five times to to make a considerable amount of money it could have kept going if the vault had not been frozen. So what are some takeaways from this. The first is that flash loans have very much changed the game by making economic attacks far more accessible so prior to this. There were economic attacks that occurred but they were kind of only available to Wales in terms of actually like changing the pricing on exchanges or the changes just had very low liquidity. So, you know, anyone can potentially balance a little bit of money but now flash loans make it possible to do it for anyone that can just do it within a single transaction block so economic vulnerabilities and your protocol like the ways that you're, you know, automated trading strategies could be broken by another trader are now almost as important as logic bugs because of the fact that someone could cause nearly as much damage to your protocols that they were able to break it directly and withdraw funds in an unauthorized way. So, definitely an issue that that auditors and security teams everywhere need to be aware of. You also have dependencies on decentralized exchanges like curve and other on chain pricing, but basically opening these these protocols up to attack with flash loans and other ways so we have these dependencies now in the defy space, kind of like Legos where everyone depends on each other for pricing or trading strategies and so you have to really know the implications of using third party protocols especially for on chain pricing and so you need to consider using things like off chain oracles or or multiple aggregated oracle pricing to make sure that you're not dependent on just one protocol that could potentially be manipulated so definitely something to consider. Also something with flash loans is that they're technically only viable for an attack if they are completed one transaction block so while this isn't great because it definitely could cut back on the user experience by by increasing the time it takes to do certain transactions or certain functionality. I consider multi block flows for for sensitive functionality that you don't feel is well protected against a flash loan or something like it. So not a great option but something to at least consider. And then finally the attack was really only mitigated by a quick response from the ear and team, when someone noticed that there was a complex transaction pattern occurring and ultimately that early, you know that that early notice was able to save millions of die. So what we're learning is that fast threat detection and response is critical for protecting user funds. We no longer can treat smart contracts as something where you know if you secure the code prior to deploying and then you deploy it then you should be good. You know I don't think that was anyone specific attitude yet but I think this has shown that they're they really can't be any more complacency about production code even if it's a smart contract you need to be watching that clearly and and directly to make sure that you know if there's an issue you can respond quickly. So one thing that you can use to respond quickly is open set when defender so this is a new product that has come out for operations automation and security infrastructure. It has both the UI and SDK provides teams a lot of options to secure their smart contracts and respond to two issues. So when a few of the features we have one is admin, which is an interface of contracts administration can bring your own keys if you want and use it to basically do a specific resolutions to upgrades to deployments anything that requires potentially a multi sig or just you know you want to go through a clear process with your team to ensure that it's deployed correctly. This is a great tool to do it. Another thing we have is relayer, which is a hosted key solution to ensure they're kept secure as well as doing reliable transaction delivery so it's just a way to ensure that when you send a transaction. You're able to sign it securely and get it processed without worrying about specific gas issues, or you know it not going through you know the relayer essentially handles that for you so especially if it's an important transaction, like pausing a contract that's under attack. You want to get it there as soon as possible and relayer helps do that. The next thing we had is auto tasks, these are, this is essentially serverless code for automated tasks and off chain logic so these can be handling like specific things like you know, updating your contract every day. For specific things you want to do off chain or even responding to things directly that are happening such as a security incident. And then we also have sentinels, which are monitoring tool for monitoring transactions events and then triggering notifications and auto tasks for certain things that occur. And then finally we have advisor which is essentially collection of our blockchain security best practices for you to look through and consider when you're building your projects. So, let's take a closer look at urine and think about how we could have actually mitigated this even more. So, from the urine exploit post mortem, we saw that the opportunities to automate detection and mitigation steps for an even fast response, you know, were possible so you know this is from the post mortem you can see that the complex transaction pattern is what first peak their interest and cause them to jump into action and cause the contract. So we were able to respond within 11 minutes within 11 minutes of noticing the issue occurring, which was probably about little little over 30 minutes after the attack had started to occur. Then they were able to respond and so what we want to see here is that you know the your team did a great job responding but we would want to do is provide them a tool that lets them do this even better. So there's a solution that Steven Lander came up with in a defender workshop recently that we did that you can see below. But to summarize here, what we can do is we can take the factors that they notice me attack, you know that the it was a large complex transaction which meant there was a lot of gas being used. And there were also multiple functions being called all in the same transaction there was an earned function withdraw function and deposit function these are all things that might cause you to worry, you know someone's doing deposit and withdrawal then they might be doing that specifically because they're causing a trade imbalance, for example. And we can also see there was a loss of fun we see that the balance is going down after each transaction and that there are probably a lot of funds being lost. We compare this from, you know, this box previous box to see that this is not normal occurrence. So ultimately these are all factors that you can use to take into account on how to prevent this sort of attack. So looking at kind of a, you know, a solution that we can put together using defender and the parameters we just discussed. We can, we can basically take a step through each one of them. So, like the admin tool, you can use as an interface for contracts administration and use that to deploy. So this is a safe contract that lets you have a two out of three multi-sig arrangement, and then use that to control your vault contract, you can do things I've upgraded or posit directly from this admin. So that's a great way to ensure that your team is able to securely respond to things with with a clean interface or or however method you prefer whether you're using arc, you know keys broaching for you, or keys that you have yourself will work well either way. If you have that all contract deployed, you want to be able to keep an eye on it so you can use sentinels to monitor the function calls and events. There's anything fishy going on or if there's any specific events you want to know about you can get notifications through emails like in discord telegram or even data dog. And if we want to let's say specifically look for high gas usage, we can trigger Sentinel to respond to that notify it and we can also have Sentinel invoke an auto task. So what auto test can do is there are other triggered by schedule or web hook or of course of Sentinel, and that can execute custom code such as a way to analyze the transactions occurring to determine. Hey, are any of these factors matching potential attack factors so it can determine if there's a loss in the contract that that's causing you know the balances to drop significantly. And then it can several send a web hook to the team for an alert notification if the loss exceeds the threshold and that team can then respond manually by going to the admin panel and then you know pausing that contract. Another way they could do it optionally is they can actually send the mitigation transaction through the relayer. So basically instead of even having to wait for someone to, you know, get online and press the button and pause the contract you can actually let the system do it itself, where the relayer will send a transaction to the vault contract and pause it directly. So you have these two options for ways to respond to attacks. Ultimately what we want to do with offenders make it easier for people to have clear security operations of let them attack. I'm sorry, respond to attacks quickly and effectively, because time is on the essence and this is a system we think would definitely improve that and I know that your team is now using offenders so we're hoping that definitely improves them as well. So, in summary, for this full talk. What we've gone over our three major hacks and what we've learned. So with the Dow hack, we showed, you know, contracts can be broken from an external attack using a renterancy. Ultimately smart contracts are new paradigm, there's going to be a lot we have to learn security so using simple modular code that's been battle tested and built with the theory of security in mind is crucial opens up in contracts is definitely standard we believe in but there are other contracts that also are good to use in certain scenarios that are battle hard so better to use these simple systems that already exist rather than build it from scratch yourself. The parody multi sec hack show that any skill developer can make mistakes. So we really need audits and bug bounties and an internal QA process to ensure that we keep smart contracts secure so there should never be a dependency on one person being smart enough to avoid a mistake. So we should have multiple steps in place throughout the process of building code and securing it prior to employing it to make sure that we're not going to miss anything and even if we miss something we have bug bounties to ensure that if someone does find a bug, they have a chance to report it and make money rather than hack and make money and the team is in a far worse position so you know all these things we think are essential for any project that's handling millions of dollars. And finally the year in why they evolved exploit, along with similar exploits are showing the growing prevalence of economic attacks between flash loans and we're just seeing a very, you know, the attack surface of scoring there's more complexity involved b5 protocols than we're used to in the past so, ultimately, you need to have operational security as well as kind of a post deployment security plan. So, you know, to respond to instance quickly are it's going to be critical to mitigate the unseen attack factors so there's always going to be something you potentially miss, and having a way to respond to the things you miss and mitigate the damage if not prevented entirely is really crucial and every team that's the points in our contract should be keeping that in mind and using tools to protect against it defender is one option we highly recommend but if you want to go with building that code yourself or whatever you think is best. We would encourage you to at least have something. We just see defenders the best tool because we obviously have everything ready to go right out of the box. So ultimately, the attack services growing for smart contracts and you really need to be keeping an eye on how this complexity changes if you're going to be making deployments in the space. We have attacks frequently right after dev con there was a $600 million attack with poly network. That's like kind of a cross chain protocol so I mean cross chain interactions are a whole thing we didn't even get into in this talk that's becoming growing area of potential attack. So ultimately developers really need good security practice, both before and after deployment to mitigate these sorts of issues. We'll probably be seeing more hacks in the future and the best thing that every team can do is to ensure that they're covering all the bases possible. Learn from these past hacks or the very least those aren't something you have to worry about and for the hacks you know can't foresee being able to respond to them quickly is going to be the make the biggest difference. So, last but not least just consider what we have it opens up when we have contracts library that most people now use. But we also have an audit team that works closely with a lot of the best teams in space to ensure that we have, you know, they have secure code. We catch bugs release reports allow the community to read through them and understand them. And then we also have a defender product that lets teams manage their smart contracts securely and respond to threats quickly. So, that's it for this talk won't have any, you know, questions here since this is virtual but I do encourage you to check out opens up one after this. You know we have I have links here for contracts or audits our defender tool. We also have a jobs posting we have plenty we're hiring security engineers and open source developers to work on these sorts of tooling and audit these sorts of projects. We also have a forum where you can ask security best practices either about our tools or about general smart contract issues that you might have and also read our documentation for more details on what we support. For example defender supports not just Ethereum but finance smart chain arbitram moon beam and many other medium based blockchain out there. And then finally, if you have any questions or you just want to reach out and available at that email linkedin Twitter or whatever you prefer. So, thank you for listening to this talk, and I hope you have a good day.