 So, hello, and welcome to Building Augur Lessons Learned. In this talk, I'd like to tell you a little bit about myself, and go quickly into what Augur is, and how it might architecturally differ from dApps you might be more used to. I'd also like to talk about what went wrong along the way. During the build and deployment process, we had issues with test net deployment. We had issues with our integration with 0x, and we had issues with the Ethereum JSON RPC interface. And finally, I'd like to talk about some of the new exciting things we're going to be releasing in Augur v2. My name is Scott Bigelow, and I got involved with Ethereum in early 2017, when I wanted to understand what a smart contract was a little better. I had some previous experience with Bitcoin back in 2013, but I just didn't get it. And I felt like the best way to understand was to pick up a tutorial and, you know, write the standard ERC-20 contract. But to do so, you needed to get on a test network. And the network that I chose at the time was Coven. This is a parody-only proof-of-authority network. Proof-of-authority means that there's a small set of validators that are eligible to mine this network. So if you want to come by any Coven-Eath, you need to go to a faucet. And one of the popular faucets at the time was this Gitter channel where you can come and type in your Coven-Eath address. And somebody will come by maybe in four hours, maybe in 36, and will send you some Coven-Eath. Now, I come from a background of software development. So my first thought was, well, we should automate this. And so I wrote this Coven Gitter bot that's monitored for these things and delivered Eath to the people who requested them in real time. And this was really my first interface with the Ethereum ecosystem. And I thought it was great. And I hadn't even gotten the smart contracts yet. This bot has been running for just over a year now and has fauceted almost 30,000 times. So if you've gotten Coven-Eath in this Gitter channel, you got it from me. But this really led me to wanting to get more involved in the Ethereum ecosystem. And I started work on Augur later that year, in 2017. Augur is a Ethereum-based decentralized prediction market protocol. This is a system where any user can come and create a market, which is really just a question that has some set of outcomes and an expiration date, some future date where this is going to be resolved. Before that resolution, users can stake Eath on what they believe that correct outcome to be. And after resolution, the market creator is responsible to come back and inform the blockchain what the real world outcome was. But of course, there's no reason to trust this market creator to give you the correct answer. And that's where Augur really comes in. Augur has created a system of economic incentives such that rep holders, the holders of the Augur rep token, are incentivized to come and dispute answers that are incorrect. And this dispute process can last a long time. This dispute process can go back and forth and back and forth. And if the system ever notices that there is a major problem, the system ends up forking. And every holder of rep needs to make a decision. Which side of this fork do I want to be on? And we believe that holders of rep token are going to be incentivized to want to be on the true fork. The fork that represents the true outcome. To accomplish this, took quite a bit of code. Augur is nearly 5,000 lines of solidity code and the base deploy is 41 contracts. And architecturally, it's a little different too. If you've used the Maker interface before, this is another depth, which is a series of smart contracts and a UI, a static UI, that can make dynamic calls into Ethereum and run an ETH call. This is a simple function call, it's just a read call to ask questions. I might ask things like, hey, how much MKR does this account have? How much dye does this account have? Now these questions come from the current state of the Ethereum network. All this data is stored in the storage of the contract space. And there's a little bit of logs here. They can tell you some things that happened in history, but mostly it is powered by asking questions about right now. So Augur does this a little differently. The UI does not communicate directly with the Ethereum node. The UI has an intermediary, which processes logs to reconstitute the state of Augur. And because Augur is a decentralized system, this isn't a server sitting in a rack, this is your box. And if you've ever seen this window before, that's why Augur is presented as an executable. Because this service that powers the UI is critical for its functionality. And so why do we do this? Why do we take the Ethereum database and turn it into another database? So we get a few features here. We get relational data. The Augur data model is fairly complex. We have lots of different things that relate to each other. And if each one of these needed to be some ETH call that led to some other ETH call, this could be dozens and dozens of queries just to look up a single page. By putting this into an index relational database, we can serve that with a simple single joined query. We also get data locality. No matter what your latency is to that Ethereum node, you are serving this data local. It's all ready for you. And users are the ones who are contributing the titles for these markets and outcomes. And whenever users are entering text, you're gonna need a good way to find that text, something that's not exactly accurate. So we have string matching and search patterns that help with this discovery. But there's another reason. I mentioned that there were 41 base contracts. But that's just where it starts. When you create a new market, that is a new contract. Plus you're gonna have two to eight outcome token contracts. You provided an initial report, that is a contract. When you provide a dispute, that is a new contract. Has a week passed, that is a contract. It's called a fee window. And so this chart from before, where we talk about the 41 contracts that make up Augur, is dwarfed by the number of contracts that have been deployed since its launch, which sits at over 5,000. Augur is also a fairly old project. We're currently sitting at block 6.6 million. The release took place around 5.9 by involvement 4.4. But if you wanna go back to that very first Augur contract deployment, the first time that somebody said, I would like to make something like Augur and deploy it to the network, need to go before block zero. This system was developed well before the Ethereum mainnet block zero was launched. On a test network that I think at the time was called POC nine. But this is actually before Solidity. Solidity 0.1.0 was released here. And so these contracts were written in Serpent. A different smart contract language that compiles down to the EVM code. So when the Reptoken launched around block 2.4 million, this token was also written in Serpent. And work got underway to develop the Augur contracts in this other language, Serpent. Unfortunately, around block 4.1 million, we asked Zeppelin to take a look and then how's this doing, how's Serpent going? We had them both look at our smart contracts and the Serpent compiler. And what they discovered was that the Serpent compiler had a critical vulnerability in it. It wasn't an issue with the smart contract code, but even the Reptoken itself that had been deployed was vulnerable to EVM code that came out of the compiler. And so the Reptoken had to get redeployed at this point. It was rewritten as a Solidity contract. And all of the Augur contracts that were already ready for audit had to get rewritten into Solidity because it was clear that the market had really gone this way and this research really justified that effort. So your smart contract is no more secure than your compiler, which is why we were proud to release yesterday a joint effort between Augur and the Ethereum Foundation for a Solidity compiler audit. And they found some great stuff. They had over 12, they had 12 issues that they considered to be high or critical. And these issues are, some of them have been addressed, some of them will be addressed, but luckily none of these created EVM code that left our contracts vulnerable, but we feel a lot better about the platform that we're leaving behind for other developers to build upon. One of the issues we ran to was a test net gas limit issue. So I mentioned we have 41 contracts and those take 50 million gas to deploy. But one of these contracts, the largest of them, takes 5.8 million, which isn't a problem for Mainnet. Mainnet has a block gas limit of 8 million. Robston, however, which was one of our main test networks for development at the time, had a block gas limit of 4.7 million. It could not contain that one transaction to deploy the largest contract, which was critical for our system. Here's a chart of the block gas limits over the last two years. You can see for large periods of time, Blue, Mainnet could host large contracts, but there was no test network where you could actually deploy these and try them out and share them with the world. So while Robston and Mainnet run the same code, there's a critical difference. Robston isn't used as much and miners look to how full a block is to slightly increase or slightly decrease the block gas limit for the next block. It's up to them, they don't have to use this as their criteria, but it's what they generally do. So what's a project to do when they want to ship to Robston? Let's just fill up the blocks. Let's get these miners to think that these blocks are really full, so they'll start voting this limit up. And so that's what we did. We wrote this smart contract whose only intention was to waste all gas that came into the transaction. Now, knowing what we know now, we could have done this a little better. If you assert false, you can actually hit an EVM opcode that doesn't waste all this time on these full nodes, but still effectively wastes gas and gives you this block. This block that is full of failing transactions, but it's full. And I started in test nets and I really wanted to help people with the faucet that I built. And so this kind of weighed pretty heavily on me. I was wondering like, why are we destroying the usability of this for days at a time for people who want to use Robston? Now I'm just kidding, somebody kept mining empty blocks and so they kept voting that gas limit down. So we had to do it ourselves. We spun up our own miners with a target gas limit that was already set to higher. So even though these blocks were mining were mostly empty, we kept voting that gas limit up. We became more than half of the network and we're able to get that gas limit up. But it really got us involved with how are we developing our software? Is test net really the right place to be doing all of this work? Here is a repository where we have created Geph and Parity in various incarnations. So this is connecting to Robston, having a local instant seal, having a local mining thing that mines blocks about every second. And this is a public repository where anybody can come and create these Docker files and images to have a consistent environment to redeploy to. So you have this local Geph that you can spin up on your local system, that's great. And then you can put contracts into it. And if anything goes wrong, you just tear it down, bring a new one up and redeploy. But for us, this redeployment actually takes a long time. It takes about 20 to 30 minutes to do an auger deploy with all the seed data that's important for our development environments. And so we took it one step further. We wrapped after the contract deployment into a new Docker image. And this became the base with which all our entire development environments work. All our developers use these images now and we version them. So whenever we have a new set of contracts to test out, we say, hey, everybody dash 12 is up and all developers pull down dash 12. And it's exactly the same for everybody. The same transaction hashes, the same addresses, the same block timestamps. So we know that we're all building on something that's the same as if it was up on Robson on something public, but it's local and it has one second block times and it has a block gas limit of eight million. We get complete control of how this thing behaves. There are new tools to do some of the same things. This is Ganache. This wasn't released by the time that we were working on this, but I think we would look into something along these lines. This is an excellent local test RPC. But no matter what you're using, invest in your Ethereum development environment. Our velocity went way up once we got this right. So we had an issue with Xerox after deployment. So I mentioned there's a 5.8 million gas contract. This contains the market functionality. If every time a user created a market, they had to deploy 5.8 million gas to be really expensive. So we deploy a delegator. A delegator is a contract that has its own contract state, has its own address, but defers all of its functionality somewhere else. The only problem, when you use this contract, you incur about a 2000 gas cost whenever you bridge this delegation. That's just the base cost of a call like that. But imagine that this isn't the market. This is the Reptoken contract. And imagine the caller isn't a person who's willing to spend 2000 extra gas. Imagine this is Xerox and the Xerox contract. And also imagine that the Xerox contract created a limit, a hard-coded limit inside of their contract that said a balance of lookup should never take more than this much gas. And they're right. That's a lot of gas to do a simple storage lookup and return that data, but not when you add in delegation. And we blew that gas limit for that one lookup. So congratulations, you've deployed. Also, you're delisted. So we had some ideas about how to solve this. We could've wrapped the Reptoken in a non-delegated contract in the same way that you can wrap Ether into an ERC-20 contract. There's no reason you can't wrap an ERC-20 into another ERC-20 that isn't delegated and is compatible. But that is a horrible user experience for the user. Luckily, we talked to Xerox about this issue. They were able to execute on their governance and they upgraded their Xerox Exchange to a V2 that has a lot of new features, but lifts this gas limit so that the Reptoken is again compatible with the Xerox Exchange. So we have these interfaces. We have ERC-20. And these things are great and it's great to follow these interfaces, but it's no substitute for testing your integrations. There's all sorts of crazy things that these contracts do that are on top of the ERC-20 interface. And if you really care about integrating with somebody, especially if you're a little different, test that integration out. So here's how different the Reptoken is. If you look on Etherscan, the Reptoken is not verified. Not because it isn't verifiable, but because Etherscan does not support verification of a contract that was deployed by another contract. And that's what the Reptoken is. And even if it was verified, this is what it looks like. It's just a bunch of assembly delegating that call to another contract. So it doesn't really instill much confidence anyway. So let's talk about JSON-RPC. So one of the nice things about deploying your software is that you get a chance to get some real user feedback. Now luckily, we received more actionable feedback than this. We got some GitHub issues. We had failed to fetch parent block. Stuff is missing. Balances are gone. I can't find anything. Things are shutting down. Sinking is a major issue. So what went wrong here? I mean, so from that prior list of Docker image we built for Gath and Parity, we took testing very seriously and we tested all of these things. We tested Infira. We tested local and remote and POA. We tested the whole thing. But in the end, this is what happened. We had an issue that was specific to Infira and specific to Mainnet. So what happened here? Well, let's talk about EastGate logs. Our system works by, when we're synchronizing the logs with the blockchain, we say, please give me the log. Please give me new blocks. It's like, great, I just found a new block, block 100. Cool. Give me the logs for this block. Here's an array of logs. Now Infira does something different on Mainnet. In fact, they do something that I would consider to be just a little bit crazy. Infira serves over 100,000 transactions per second. That's the thing that they do is different. That's the thing that I think is a little bit wild here. And to accomplish this, to accomplish this feat, they need more than one server to do it. They probably need a huge quantity of servers to do it. In fact, if I had to guess how many servers they have, this isn't a public number, I would guess they probably have like, well, let's say two. And in order to have more than one server, you're gonna need a proxy. And so let's walk through this conversation again. Hey, please give me a new block. Here's block 101. Great, can I get logs for block 101? So this system doesn't necessarily have the same perspective with the Ethereum network as the one that delivered you block 101. And so the system, this is just a standard Ethereum node, thinks to itself, well, what the hell is block 101? But the problem is that it doesn't say that. It doesn't say what the hell is block 101. It says empty array. So I don't think this is in Fiora's fault. This is JSON RPC, kind of not necessarily lying, but not really telling the truth either. So what solutions do we have for this? Well, we could ask for all logs and just throw everything away and then filter out the auger logs ourselves, but know that the presence of logs means that we talk to a node that had that block. But we get empty blocks all the time. Okay, but maybe we could use like the logs bloom filter and if it's empty, we know, but it's still a lot of logs. So how about we get auger logs and crypto kitties? Because we know that crypto kitties has lots of events all the time. So we're gonna get like a totally adorable cat in our JSON response, but we're gonna discard it, but use that as our indication that we talked to a proper node. And we seriously consider this, probably more than I'm willing to admit. But the problem is, is I don't want to wake up one day and write this comment as a response to a data quality issue. So a solution that would actually work, well what if every time you got a blog that was empty, it was like an empty response, you remembered that and then the next time you heard about the next block, you made a range request for 101 to 102 and then if you saw 102, you knew that you actually did hit 101. It's just, it's getting crazy, right? So what we did was, there was this EIP out there that had been out there for over a year that said, how about we query blocks by block hash? This is a more unique identifier. And more importantly, how about when it's not there, you let the person know, you don't kind of fuzz over the fact that you've never heard of that block, you deliver that to the client so they can make the right choice on their side. And if anybody heard Jack Peterson's talk last year about bounties, I think this is one of the critical success stories of bounties. Here's an EIP that had been sitting there for 15 months. After Augur experienced these issues, that EIP, we'd be able to use bounties and engaging the network to get it merged in two weeks and implement it in geth and parity in six. That is an excellent response, right? And that's a killer ecosystem. And so, but in the meantime, we had to say, okay, you got one node, you're good to go. If you have two or more nodes, whether you're in fear of it, whether you're anybody, we can't do that until this EIP gets merged. But then we got other issues. We had parity. There was a parity specific issue when people started launching their new nodes and I didn't understand this because remember, like I come from COVID, like parity is my jam. I know this works with parity. So what happened here? How does parity getting these exact same issues? A single node. Let's talk about ETH get logs. When you synchronize with the Ethereum blockchain, it goes something like this. You start at zero, you go to 6.6 million, you're good. The parity develops something called a warp sync where you first sync, so you pick a point in the middle, pretty close to head, and you sync that part first. Then you say, hey, I'm good to go. And later you backfill from zero to that warp sync start point. But you think you're synchronized when you're right here. And if you issue a get logs for this, you get your logs back. If you issue a get logs back here, you get empty array again. And if you issue a get logs that spans this point, you get half of the logs you're expecting. And you can imagine the kind of havoc this wrecks on a system that was deployed right here. So we made sure we documented this in our readme. We tried to detect the behavior automatically. But really the solution was, we've been working with parity, and this is actually scheduled for a future release to deliver the correct information, to volunteer more information. And we've seen some similar issues on the geth JsonRPC interface. It's all about eith call. When you call a function that reverts via that eith call, you get 0x. Eith call. When you call a function on a missing contract, either because it will never exist, or because you're talking to a node that is not at the block height that has that contract deployment, you get 0x. Let's talk about eith call. When you talk to a server that is overloaded for a contract function that will succeed, you get 0x. And so we're left wondering, should I retry in 100 milliseconds, or should I never talk to you again, because this is an unrecoverable error? So what's the lesson here? I think we don't quite know what Ethereum's JsonRPC is currently for. Is this a web API, or is this a database query layer? Because these have different success metrics. This needs to be 99.99% available, but it's okay to have a little bit of caching in there. It's okay to have a little bit of fuzzy answers. This is serving a web UI. But this needs to be 100% accurate. And you need to, if you can't be confident in the answer, deliver reliable errors to the user. So I'm proposing, and I believe it is currently a web API. And to support that, here's the documentation for the JsonRPC interface. This is over 100 pages of documentation. The word error appears zero times. And I think it is not a database query layer, because if you were to pretend that the issues we've experienced appeared in the MySQL bug tracker, these would be big issues. If querying for new data returns success with zero rows, or querying for old data returns success with zero rows, this would be a big deal. But I don't really fault the JsonRPC, or parity, or geth for these implementations, because what we've built really works. We've built big things, and they work great. I haven't had a problem using the Maker interface, but if I ever did, I don't know, I would just refresh, and everything's fine. If I was in that one in a thousand case, right, I got a bad get log. But when you're synchronizing the database off of Ethereum in this way, when you lose data, you start building with missing data. Or your synchronization has paused because you don't know when or if you should be retrying. You know, and Auger's not the only system that is trying to treat the Ethereum JsonRPC like a database. The Ethereum ETL project, where Google is running daily exports of the entire Ethereum blockchain and putting it inside a big query. But this is a daily export, and there's lots of value to be had there in research and reporting and charting. You can do some really cool stuff. But this is a different product than an ETL process that tracks head closely and can serve as the API for an application that is more complex. So I believe that ETL is gonna be the future to building advanced applications like this. And so why am I up here talking about some of these specific issues? Why don't you just go file some issues? Because I think that what we're talking about here is cultural. And I think if this, we need to decide if this is the kind of thing that we wanna care about and we want to request out of these APIs and work towards achieving. So I think we need to care more about reliable data access. And there's lots of projects underway that are leading this charge. In fact, you know, the folks that in fear we've been talking to are pushing ethql. This is a new way of talking to the Ethereum blockchain. And currently it is resting on the JSON RPC interface, but the intention is to put it inside of Ethereum. And so this is our chance to really get it right and enable ourselves to build these big applications. So what's next? What's coming up in Augur V2? So currently all markets in Augur are denominated in Ether. Ether is the only token that we accept for trading on these order books. But we have decided not to add dye as the token that will be used here. We have decided to replace dye as the only token that will be traded on the Augur platform. Because we've realized that not everyone who finds use in Ethereum necessarily wants to also be long Ether. Especially not for six months or a year as long as some of these markets can take to resolve. We're also working to improve the user experience here. Downloading an application and having it sync locally as some window running on your system, like that's not really an optimal experience. That's not really the way that the web is supposed to work. And so we're working to take the same architecture and to run it inside of the browser and use some modern browser features to accomplish a lot of the same things that we need, but do it in a way that is transparent to the user and doesn't require this extra UX burden. Because I think this is gonna be a powerful pattern for larger and larger groups to implement. Because if we wanna take this chart and change the scale of it again, I wanna make sure that whatever project does that has a great platform to build on. An audited ecosystem, development environments that can support large arrays of engineering teams, integrations with other projects that are well tested and not just relying on simple interfaces. And to build on a reliable data platform. Thank you.