 My name is Ben, I'm the CTO at BitGo and so BitGo is like an exchange API that many exchanges like BitStamp, Corbett and Upbit use for their wallets. So we, you know, we do scalability and security. Yeah, my name is Jake Craig. I also work at Coinbase, Senior Software Engineer on the crypto payments engineering team. So we're the team that sort of runs all the blockchain integrations at Coinbase. If you do sends or receives hot wallet cold storage, all that kind of stuff goes through our systems. Yeah, been working there for a bit on Ethereum. Cool, and I've probably introduced myself as well. So I'm Eric Scrivener and I'm also on the crypto payments team with Jake at Coinbase. And he did a pretty good job of summarizing what we do there, so I'll leave it at that. And all right, so we'll start with some softball questions here. All right, so this one is one of my favorite ones. I think this should be asked more at Ethereum and DevCon events in general. So if you were an EVM opcode, which EVM opcode would you be and why? Well, I'm on stage in 2018, so I think I have to be the PC opcode. Fair enough, all right. What about you? This is interesting because I actually looked up the opcodes recently. There's quite a lot of them. Yeah, I'm interested in what he's going to say, but I can't have a guess. Anyway, my favorite opcode is call. All right, all right, I can see that. Oh yeah, and the why? Why? Yeah, what do you enjoy about call? Because I think it's what makes Ethereum so interactive and expandable. All right, fair enough. And Jake, which one are you going to be? So I've got two. I think the first one might be what you're guessing. There is an opcode called Coinbase, which is a very convenient one. But that's not going to be my choice. My choice is going to be XOR, which it's just because, you know, all the new cryptography stuff in Coinbase, in the community coming out and stuff, it's just XOR with some powers, basically all that at some level. And it's just a really exciting thing to play with. And it's this very simple thing, but it's just so powerful. All right, no one picks self-destruct. Okay, that's good. That's yours. Okay, that's mine. Okay, so let's kick things off. Let's move on to the medium ones here. Okay, so I really like this first one. So what is your biggest challenge integrating with Ethereum at this moment? Let's start with you. Okay, one huge challenge is a nonce management in terms of you can have rapidly fluctuating gas prices across the network. And if one transaction gets stuck, then it slows up all the others and you either have to resend the first one or just wait it out. That's a huge challenge. Yeah, I think we see that as well. It's a pretty common issue when you have very large wallets. So with, you know, with large wallets in Ethereum, as many of you know, the accounts have a nonce. And whenever you create a new transaction, it has to have a nonce, which is monotonically increasing. And the way that it was designed that way is because it's a lot easier to track for double spend protection and all that. And so say, you know, you're running an exchange or something like that, and you are going to be sending, you know, 10 transactions per five minutes. And this can be, you know, 100 transactions, you know, over 50 minutes. And if transaction number four, you put in a wrong gas price, then by transaction 50, all your transactions can't clear because four isn't clear. So I think that's a big challenge there. You know, do you want to talk about some solutions? Yeah, like you said, we've all sort of had this issue and it is one that tends to come up at scale as you're doing just a lot of them back to back, right? If you're an individual wallet, you know, you send one transaction, if it takes a while, maybe that's okay. Or you can really easily sort of replace that. It's just sort of replaced by fee mechanism. And it's an interesting one because like it sort of depends how you structure your hot wallet as well. There are different ways to do it. Some exchanges, you know, tend to use like a sort of single or a few addresses for everything. So they have a lot of transactions going through that address. Other ones use, you know, contracts and stuff or, you know, lots of them like we like Coinbase in our wallet, we just basically have just, you know, hundreds of millions of addresses. And so we're always have it all over the place. And so when we hit the situation, it can be fairly limited in scope, but it can affect like a subset of customers. And there's not great solutions for it yet. Other than replacing it, which is a solution, but it's a more sort of complicated one to manage. And especially when you've got a lot of these going on. And so this thing where sort of the gas prices can spike and sort of delay everything is really challenging. This example that I've seen in our systems before is transaction two minutes apart, same gas price. One took two minutes to confirm and the other took 13 hours. And that's not a really great way to build, you know, systems and to build a business on Ethereum. And so it doesn't just affect us. It could just affect a business doing this who doesn't have, you know, as sophisticated infrastructure to handle. And one of the things that we've sort of talked about and seen come up, but a few times is, you know, this idea that comes from sort of Bitcoin of this child pays for parent style of transaction, where it's a similar mechanism of sort of recovering sort of pending ones that are like chained together in some ways. Like in this case, it's chained by like the nonce. It's different in Bitcoin. But the theory is sort of you put a new transaction out like on top of all these with a high fee. And then that sort of can subsidize the rest of the lower fees and get it back to closer to what the real network fee is. So miners will pick it up and they'll pick up all of them at once. So you just send this like transaction with this really high fee and then all of a sudden, you know, you're maybe, you know, 50, whatever transactions are all, all can get confirmed and you can sort of recover that and have sort of a better experience for your customers. So that's, yeah, that's, that's an interesting one. Yeah. So let's talk a bit about that. So, so firstly, you know, RBF like replaced by fee, it works today because it's, you know, you can do it yourself. But one of the larger, I think, confusions with that, you know, when you change the fee, you also change what you're signing. And therefore the TX hash gets changed. So if you're displaying a TX hash, TX ID to the customer of the person receiving the funds, then it will be different. With CPFP, it's, you know, well known in Bitcoin land. The, the common challenge is, you know, the idea is you have 10 transactions not confirmed. So you put another one transaction 11 and it's got an outsized fee. And the outsized fee helps to tell the miner that, hey, you know, if you want to get this outsized fee, you, you, you've got to mine all these other 10 first. The, the, the challenge for the miner then is how are they going to figure out that, that this string of 10 transactions combined with the 11 is really worthwhile for them to mine. Because if they were to do this infinitely, like after 1000 and 1000 and one transactions, then they can run out of memory, just waiting for you to come up with another big, you know, Yeah, that was the thing that was brought up. It came up in Ethereum maybe a few months ago now. When there was this exchange that was sort of incentivizing lots of volume on the network. So it was just kind of backing up the network. I think this is another sort of big challenge with Ethereum is that individual token sales, individual, just new, you know, games or whatever can like significantly impact the network very, very quickly and very fast. They start the token sale, the fees skyrocket. Now everybody in the network is paying more. Any app you built is just now all of a sudden paying more. And then now you also, as you know, anybody sending transactions get them stuck because the estimate can only update once it has all the new data. So you're kind of at this, you can't ever predict it necessarily. You can have some sort of models and like generally it works and it updates fairly quickly, but there is this sort of delay. And that was a thing brought up when it was started to be talked about in Ethereum a little bit. I believe on the parody repo there's an issue that has some sort of proposed model, but they're specifically talking about sort of like DOS vectors and stuff with this, like how long you track it, what's this buffer and is that configurable and how does that actually work? So to summarize here, it sounds like volatility of fees is the biggest challenge here and there's no good solution at this point. Is that the deal? There's a solution. It's just a challenging one. I think because like Ben said here, the transaction has changing and stuff, it's a more complicated thing to like re-match things up. If you have sort of two pending transactions with the network and one of them will confirm and generally it would be the higher fee one, but I think in theory the lower one can also confirm. So now you have to have these two sort of states and like, all right, which transaction is real and then, you know, one of them will confirm and so it's just a more complicated problem to implement. It does become a potential challenge too, because it's even like child pays for or parent pays for child and it's could potentially, okay, so now we have and you kind of touched on this then where maybe miners run out of run out of memory space when they're waiting for all these pending transactions maybe, but if hypothetically that did exist, if it was implemented poorly, it could lead to this new design pattern where everyone's putting, you know, one GUA or whatever it be as all of their transaction prices and then if it doesn't clear within some X time period, then they put out a transaction to kind of clear those out. So we have this now almost waves of transactions going through just because people want to pay lower their costs, right? So it's a tricky, tricky situation. All right. Okay. Well, it's, it's, so all three of you seem to have consensus on this. Yeah. Well, this is one of, one of the things, right? I think another one that it's not just us, but we definitely experience this is just running notes, right? Running notes, keeping them up to date on the size of the backups and stuff. In particular, the sort of like archives, state notes that store like the full history. I think Offrey of Interpairty just released a website about like that constantly tracks the size. And so if you run like Parity, you can get it to like 200 something gigabytes. But you don't have this sort of full history one. And that one tends to be, I think around these days, 1.8 terabytes or something. And so just the size is the thing of just transferring that over the internet when you need to like spin up a new one just takes a lot of time. I think ours are ones when we spend them up for like three, four hours right now to sort of do that. We've got things to get them faster, but a fresh sink is like, I think the last one I did about a month, a month and a half, something like that, which kind of flows nicely into the next question, which is, do you think that integrating with Ethereum is getting harder at that low level that we're talking about here? And if so, do you think that we're at some sort of critical juncture with respect to that? I think for, for instance, I mean the blockchain size keeps increasing every day. So for that purpose, it is getting harder in terms of re-sinking nodes or syncing a new node. If one crashes or something like that, you lose the database. In a lot of ways though, there's a lot of tooling that makes it a lot easier to integrate things. So things like truffle or ganache are becoming really, really easy to use and there's a great user experience around dev workflow now for, even just for local development, but also for production. And it's, it's this interesting kind of duality. So some parts are getting harder, in my opinion, some parts are getting easier potentially. I think to go to production with Ethereum is getting a little bit harder. I really like the developer on-ramp. So, you know, the developer tooling and all the, you know, building or building tools are getting better and better at welcoming new, new, new developers. And that's why we have so many developers at this conference. I think that so far as running a node, when I last fired up, you know, a node for myself, you know, we have lots of nodes in production, but sometimes you want to hack yourself. I think guess, if you, you know, running in the fast, fast mode is actually not too bad. And I think it's right now, it's not a critical juncture because at this point, the block processing time is still less than the time it takes to propagate and at the time that a block, you know, gets found. And that, you know, we can talk a bit about more about that. Like that was different earlier this year. But I think that right now it's quick, where you can fast sync all the way to the head of the chain. And then you can just, you know, your blocks are getting processed quick enough with, you know, modern hardware. It's not totally a problem. And of course, you know, you really want to be running a proper archival node in production and running several of them. So that's a different challenge. But to get onboarded, I think it's actually still pretty easy. I agree that like onboarding definitely doesn't, and they're having a lot, there's a lot of really cool stuff there to like get up to speed and like start being able to do things. But like you said, sort of at this sort of like scale thing, or when you're running like businesses that are worth a lot of money and a lot of money is moving through, you also want to like have sort of the full history yourself and always have a copy of that. And so you, this kind of initial sync time, while like, I think for a lot of these cases, this is not particularly important or like syncing sort of every block into sort of your own database that you have sort of your own full history that you can analyze outside of, you know, just the sort of latest state. Like one of the things we have is our data team wants, you know, full copy of the blockchain in, you know, SQL that they can sort of analyze and do things with. And right now, so we sort of manually build tools that do that kind of stuff. It just takes a while, right? Just because of the pure size. Do we know how big that is? How much, how much your data that is? It depends on how you want to index it. Right. So I think that this project I saw part of the grant team showcase, it was called vulcanized DB or something like that. It was pretty cool. You tell them what you want to index and they, they will run it through and put it into a Postgres. I think it's a great idea and it's nice. It's lowering the bar so that not everyone has to build an indexer. You know, you guys probably build an indexer and we have an indexer. So everyone's building one, but the more it can be open stuff. All right. I've got another easier one for you. Well, maybe it's not easy actually. It could be easy all the way to hard. So, so parody or gap or a laugh if you like that one. I'm partial to parody. I found that it's been more reliable and faster sink has been nicer and we can get into the whole parody trace parsing issue, which I'm sure we will. Yeah, yeah, but we can definitely hop into that. Yeah, that's definitely what a damaged parody had early on. Yeah. Well, that's what I've heard is with parody. What we're sort of ending at here is this idea if you're on ether scan, you see contracts sending ether around they call them sort of internal transactions and sort of detecting those and seeing like where money moves is a complicated thing. I actually gave a talk on this yesterday. That will be recording of and stuff about sort of the ins and outs of how that works and stuff. But from my understanding, I wasn't around when this just happened, but parody was one of the first ones to give an API that made that more like accessible and easy to sort of figure out what happened. And guess now I believe since 1.8 it was does have an API for this too, but it's not a standardized API. It's actually a custom one for each one. And so there's no standard way to process these. And that's I think something that would be really interesting to see in the future. Yeah, it's interesting because like there are lots of transactions that happen that appear nowhere in the blockchain. And so I think that's one of the interesting differences between Ethereum and a lot of other ones. But Ben, what about you? Yeah. So the first question, I think both parody and get okay, random both depending on what the network situation is sometimes some of them are challenges processing certain types of transactions and blocks and some of them, you know, then the other one goes faster than. So I think running both is the way to go. All right, fair enough. With, I think that this is a huge topic, internal transactions. For us, it's one of the largest pain points. So at BitGo, we use multi-sig wallets. These are wallet contracts, smart contracts. And for us, when we send out from the smart contract, we want the receiver to be able to see the funds easily. On more modern exchanges, they do record the deposit because they can look at these internal transactions. And so a little bit about internal transactions. So in Ethereum, you have transactions and transactions are what I think are the first, you can call it like the first hop in which an account sends something to an address where the address is either a contract or another account. And the challenge is after that, if you send it to a contract and tell the contract to do something, then internally the contract is chip making changes to state and doing internal messages, some of which may be sending funds like Ethereum to other addresses. Now, the challenge with that is that currently in the Web3 API, there is a get transaction which does the first one. It shows you all transactions that come from accounts to smart contracts. But usually what a multi-sig wallet is, is that you would send a transaction from one of the signers to the multi-sig wallet and then a multi-sig wallet would then send out the funds from the smart contract. And that is a smart contract transaction or internal transaction. So there's no RPC or API to easily get a hold of those transactions. As a result, some of the exchanges, they don't recognize deposits automatically for such smart contracts. And this, I think, will be further exaggerated because now we're moving towards, what was that, the unified ID wallets? Universal login wallets, which I'm a big believer of from a UX standpoint. But remember, they're also smart contracts. So we need to solve this problem as an industry and make more people actually take in deposits from smart contracts, because that's what Ethereum is about, right? It's a smart contract. Yeah, and it's a scary thing to process these too. You can have some pretty dangerous sort of bugs that come up from doing this the wrong way. And so there's plenty of exchanges and stuff who don't want to do this yet because it is this sort of scary thing to get wrong, whereas transactions and stuff are fairly straightforward to kind of tell what's going on. These aren't. So it's a big risk. And especially as we move to more and more sort of wallets doing things, these are going to have to be supported. And we need to make that easier. That kind of flows nicely, actually, into the next one that I kind of wanted to throw out there. Sorry, we're going to go, okay. It's probably going to the same place. Yeah. So, all right, so I guess the next question would be wallet contract or like scan individual blocks and transactions and invoke transactions. Which one do you prefer? Yeah, so that is exactly why I was going with that. There's pros and cons, right? So either you're generating a new address for every deposit you want to take as an exchange, and you have to monitor these interventions, actions, very, very carefully. And there's then you have to sweep them into your hot or cold wallets. And it's this big process or even as a contract side of things, if you build things as a contract, I think Bittrick's does this and maybe some other ones too. Things like sweeping gets easier. You can monitor events in your smart contract. So internal transactions aren't a problem anymore. But now you have increased costs for each individual customer who needs to a deposit address, but may never deposit any ether in there. You have to handle tokens as a completely separate case and it definitely complicates things. And there is that trade off. One of the things I don't particularly like about wallet contracts, especially if it's like a single one, is that it puts all your transaction history in one place. Privacy is a really important thing and we want to have. And so contracts, especially this, and because of the costs, like you're going to be incentivized as any sort of business to do sort of one, right? Unless there's some extra cost or something for a new address. And if you have one, now all of your history to that exchange is in one place. And so anybody wants to see how much ether you had and how much you sent there, they can tell that and maybe how much you have in that exchange. Maybe that makes them more of a target. If they follow a pattern, like things like Bittrick's contracts, like you can detect them easily. And so you can easily see how much Bittrick's is receiving through that pattern. Or it's like the way that we do it, which has some other downsizing costs is like we generate new addresses just all the time. Almost any time you get an address from us, it's a brand new one that's never been seen before. We send transactions. It's coming from a new address. It's never been seen before. Clearly that has some things of just having a lot of addresses that's its own problem. And I think we might get it up later too about, you know, sort of batching transactions and stuff and doing a lot of sends. You have some interesting challenges there, internal transactions. But I think like, I don't like this idea of just like linking everything together, all of your activity in one place, especially if you're going to like send from there too. So I think in terms of having one address per client, I mean, per customer of the exchange, pretty much many exchanges are used to this concept from, you know, a variety of other digital currencies. And so I think on that matter, what we're doing is whether or not you're a contract or address, you got to have one address, you know, one address per, per the online, you know, customer deposit address. And so what happens in the end is that when you try and do that with contracts, you're probably going to be using some kind of forwarder contract in which what it is is a really small shim that forwards the money from this unique deposit address for each customer into some, some main wallet. And, you know, sometimes the challenge with that is, with us at least, we wanted to be a multi-sig platform. And so we wanted, you know, all our wallets and contracts to be, you know, not controlled by a single key. And that was one of the requirements that we set out to fulfill. And that's why we did it that way. I think that, I mean, one advantage of that for us that we, I guess, haven't talked about is that with contracts, you're, you know, instead of looking at the internal transactions by yourself, by looking at transaction receipts, you can look at the events because you can get contracts to fire events. And actually you can do it within the 21,000 guests with it. You can get one event or something like that, barely enough. And then you can, you can code your indexes to pick up those, those events and use them for, you know, deposits. So, so there's one advantage there. The disadvantage is, you know, pay, you have to pay the gas to, to deploy the contract, which, which can be very costly. And well, I mean, I think there are some EIPs that are out there that are not too far away from being landed that will allow us to do, you know, basically pay to script hash. In other words, you can pay to an address, which is a contract address without actually it having been deployed yet. And then someone else can then deploy it after they see that, oh, okay, someone actually deposited that. And this is really useful in the case of an exchange that has 100,000 users and only 50,000 may use F. So you only need to set up the address after they, they deposit. You do bring up an interesting thing there. I think about like the gas limit stuff, right? So this is the thing that different exchanges have to make different decisions on customer sort of sending Ethereum somewhere. And if it might go to a smart contract, if it might not, and the trade off of like, what sort of UI do you present there? What sort of gas do you charge your users? Because, you know, you standard Ethereum transactions of flat 21,000, you can sort of guarantee that. Or at least in the contracts, there's no guarantees necessarily. There's things that you can estimate it and it's probably right, but it might change too. There's some ways it could. And like, that's an interesting thing in exchange to have to do, like us on like on our sort of brokerage of just coinbase.com. You don't see any gas stuff when you send. And it's a very like intentional decision to keep it simple because it's a very confusing UX sort of design pattern as I'm sure people have talked about in the design tracks here. And it's this interesting trade off of like how to handle that. And as like a business, how do you set that? And how do you sort of expose that to users? Yeah, it's an interesting question for exchanges and I mean, coinbase kind of both, but both exchanges and wallets really same UX problem really. It's the same UX problem that Ethereum and really all the cryptocurrency industry is kind of going through right now. Yeah, it's just, it's probably not just Ethereum. But we can't really just sweep it under the rug. I mean, the main issue I think really is that customers are expecting to pay, you know, they're used to paying fees, which are percentage or flat fee. And in Ethereum and other chains, you're really paying for compute or space. And not only that, it's an auction of block space or computers. Interesting. All right. So I think Jake, you'd mentioned batching briefly down there. Do you guys want to maybe chat about that a bit? What do you think about it? Is it something that Ethereum needs? Is there some sort of alternative thing that maybe would also be interesting to have? Yeah, I think I only first brought up, we were sort of talking yesterday and I sort of brought this up and I hadn't thought too much about it deeply. But like I mentioned, we have a lot of addresses and so we have all sorts of customer sending and all sorts of amounts. So, you know, we'll end up with a lot of addresses with very small amounts of Ether. I will need to, so let's say we have a set of addresses that have, you know, one ETH in each and we need to do, you know, a 10 Ethereum cent. Well, we need to sort of combine those into some other address. And then after that confirms on chain, all, you know, 10 transactions that might take, we need to, so we need to first send 10, wait for them all to confirm. Then once they confirm, we can then send out this 10 Ether. So, we call this like consolidation internally. And this is sort of a complicated thing to manage. And it's an interesting thing because like Bitcoin and sort of these UTXO chains, they have a different way of approaching this that doesn't have this problem. And so once you, one of the reasons they do that is so you can have sort of a new address for every sort of change amount. And so you can basically always be using new things. Whereas like on Ethereum, it's really hard and it's all about this privacy stuff again, where you wouldn't want to do that because if you wanted to do a send, you'd have to do all these like intermediate sends to finally be able to do it. And so I think it'd be interesting to consider some sort of like batching strategy or like multi transaction thing where you could say, you know, from all these addresses, I don't include all the signatures for them. Do all these transactions at once into some like destination. And that's sort of one atomic send there because also you probably don't want to do, because you could also, I guess another way instead of consolidating and then doing a 10th send, you could do like 10 individual 1e cents that destination. But if it's like a payment service, they probably aren't going to handle that well. If they're expecting 10th and they get like 10 1e cents, they're not going to know where it's supposed to go to, right? Pretty bad UX if a user goes to send one transaction and they receive 10 on the other end. Yeah, that's probably going to confuse them. I think with the theorem, actually it's a bit, it's pretty cool in different sense that in Bitcoin, you know that you have these unspends and you can tell whether each unspent is economical to spend or not. In a theorem, you can almost say like, okay, if you've got like some dust, you can just wait because remember, you're not spending per unspent, you're spending from the account. So, you know, if someone sent you a lot of, you know, one, one ways, you could wait for, I don't know, Brazilian of them before you actually took it out. Usually I think, you know, consolidation is a good idea to just do it like when the account, when the network is less loaded. So, you know, when nobody's holding any ICOs, well right now not many people holding ICOs, but on weekends or something like that, when the gas price is pretty low and keep it low so that you don't actually impact the network because you very well could. One of the interesting things I want to ask, Bobo, do you hear, you know, what have you seen with regards to batching outgoing transactions? So, you know, with other coins and without getting too much into that, a common pattern is you wait every five minutes or 10 minutes and you batch out a send, you know, of whoever else is trying to withdraw during that time. With a theorem, this has largely not been done yet, even though it is very possible even today without making any changes whatsoever to consensus at all. You can just use a smart contract and do a multi-send. For me and like some of the conversations we've had about this sort of thing on Ethereum and Bitcoin, really any chain is privacy again because like you now link all your sort of customers together in some ways and we want to generally prefer that, like we like to sort of default to privacy and try and maintain that as long as we can and so when sort of batching comes up in that, that's always the thing that like I mentioned earlier about like wallet contracts, like putting everything in one place and maybe just Ethereum is the wrong chain for the types of things that I'm talking about and I think there's ways we can get there. But for me, I haven't seen it and you can definitely pull it off like you said with smart contracts and stuff. Yeah, I think that's kind of the Ethereum, for lack of a better word, purist answer to like the batching question is, well why wouldn't you just use a smart contract for that and that's kind of Ethereum's answer for everything which isn't a bad thing but maybe there's trade-offs that you don't want to make there in your server side architecture that we spoke about earlier. So you're saying if you have a wallet contract, I guess it becomes very convenient to do these kinds of multi-sends but if you don't, then there's no like alternative mechanism, right? Yeah, you could theoretically design a function call in your contract that would multi-send to a list of addresses that you pass in. And it's all technically the same Ethereum address but then that just happened all in the internal transactions but then like you said they're all linked together and... Well you also need a balance on that account too that's like sort of public and probably associated with you in some way. So like it depends how you structure your hot wallet but like you don't want to probably have all your money in one place. Maybe but yeah. Yeah, right. So essentially lots of I mean lots of exchanges end up with a hot wallet somewhere and I think for shapeshift it doesn't matter as much because you do like one transaction at a time, right? So you'd like to send that but maybe one of the reasons why people don't use this as much in Ethereum is like I think Ethereum people are really used to seeing a transaction where they have a source and a destination and if you look at that like on EtherScan you know it's quite clear it's always got a source and a destination so it's very one-to-one in how people like their mental model how they think about a transaction. You know in other chains you always have multiple outputs. Yeah it's also I think a little less necessary in Ethereum because like we talked about the non-stuff earlier in Ethereum you can do these sort of serial but like back-to-back sends like on from a single account right if you just increment the knots you can continue to like send from the same account. So you can you can queue up all of them at once without any sort of delay or is like on Bitcoin and that style of chain you have to sort of if you have a big sort of input of you have to sort of wait for that to do its thing and then do another one or there's other ways you can do it but it's it's a lot trickier whereas like Ethereum just like just keeps sending from the same account and it'll let you do it and like it should generally work and it hasn't been too much of a problem we've seen where like any sort of need need for batching yet either because there's some like gas savings potentially. Yeah I think I think that that like the gas savings I think that with Ethereum maybe we aren't as mature yet with doing gas optimization you know with Bitcoin for example because things are so expensive you want it to save every little bit you could so you're paying the same amount for signatures on the inputs and if you can send a tenant once instead of one at a time you do that right but with Ethereum I think it's still a little bit earlier with the gas savings now it's mostly like how do we price the way you know get gas price rather than an actual gas spent and really for all of these conversations chains once like an actual Casper sharding some scaling solution gets implemented they're all just completely different because we have a whole new set of problems and yeah I can't even imagine that would be like with a with a sharded network it'd be much more complicated. I think about I think we might switch gears here a little bit so what we got here I guess I would love to talk about privacy with you guys so how are you thinking about it what do you think the status of privacy and fungibility is in general are we there yet what are some of the interesting prospects and how do you trade that off with scaling well in terms of Ethereum we're definitely not there yet for privacy okay definitely not there yet but it's like snarks exist now but it's it's really not easy to use unless you have a PhD and even then it's it's difficult it's it's definitely tricky and really they are their best implementation should be in some sort of plasma-like implementation or in a shard of some kind so it's almost a catch 22 in that which which comes first the scaling of the privacy yeah so I think it's not there not by any means I mean you can see how easy it is to track you know I think on the monitor scan they have labeled like shape shift one there's a there's an address there which shows you know where you do your transactions through and that's that's the most public thing I've ever seen having said that I mean I don't feel like you can really trust the so-called you know pseudonymousness of bitcoin you can always almost think about it as those are those transactions will be reviewed when you think about privacy it's not just about like privacy today and okay maybe it's a little bit hard to track today but the blockchain is forever right so so you going back going forward like five years later all of that data is going to be there so you're going to have to to live with whatever you're putting on on that today so I think that secure you know security is really important and privacy is one of you know one of those those important things for security because it makes it just that much harder to secure something if you have a very big target amongst other reasons so I I just don't see this as something that is being worked on enough yet from a theorem I think that we have scalability challenges and I think they're going to go after those first because that is what drives adoption I mean whether you like it or not they want to drive adoption yeah I find I'm really excited for sort of the prospects of like the future of privacy but yeah I don't think ethereum is doing maybe enough to to be concerned about it or to like there's definitely teams in the space working on sort of off-chain sort of style solutions which I think could be really interesting but just the general sort of blockchain community as a whole there's a lot of really interesting stuff going on right you know there's all the Zcash and such doing all the snarks and then there's you know Starks now and you know bullet proofs and all sorts of things happening but I think you brought up a really good point about like this sort of like foreverness this sort of infinite sort of data thing and this has already happened with Monero where you know they had some sort of bug where you could in theory sort of I think in some ways you could I don't know the details but you could effectively like sort of de-anonymize some of their transaction history because of some sort of you know vulnerability there and that's sort of already happened and so in a lot of those this new photography it's also new and you know most of the photography we're using is like fairly like well established tested and stuff and so we do these you know security audits but a lot of it's also still backed by this this there's a word for it but this idea of like it hasn't been broken yet and a lot of people have tried we can't prove it's totally secure but we think it's pretty good and so we'll keep using it until then and a lot of the newer stuff I imagine would have like similar properties except it doesn't have like it's not battle tested in that way and so while there's like really smart people doing it and developing this stuff I think that's a really like big sort of risk in general to it but I think it's definitely a worthy goal that we should we should go for as you know businesses and anybody wanting to build on this you know I don't want my whole history to be for everybody to read anytime I send somebody something from any of my you know Ethereum wallets they can potentially see my whole transaction history and I could split it up in all sorts of addresses and try and do that but then it's it's like that problem described earlier even on like a just individual level not even the exchange level of well that's really complicated to manage it's also expensive just fee wise cool all right so so another easier question here but what are you guys most excited about with respect to Ethereum it's a good question and I would like as a something we would like to see in there or as a yes only you would like to see some project that you think is interesting or some you know future prospect that you're really looking forward to I definitely think that for me it it probably goes back to the whole nonce thing that we kind of started with it's that's a one huge pain point and something some solution to that even at the protocol level would be very nice it's also something that Jake mentioned yesterday when we were chatting was it was there's it's very interesting concept of because we have it's more of a solution to the internal transactions parsing problems that we've had where maybe we could implement events at the protocol level like contract type events at the protocol level and I hope I didn't take the one you were going to say just now but no no I think yeah yeah it's a really interesting idea like this late like because I think it's a good way because like events are this later known entity and people know how to deal with them because of your c20 and 721 and they're gonna continue to be used and people have tools and stuff around processing them in really interesting ways and so I think that would be a really nice way that you could you know extend you could bring these sort of internal transactions out of this sort of mysterious world that you need to parse to get them and I think it's that's one way yeah I think it's a great it's a great idea I mean when you look at it what what people what a lot of people actually want is they want to see when the theorem moves and there are a lot of ways that smart contracts can send that today which don't get shown up with get transaction API so I think events are one of those ways if you can buy those events whenever whenever any eth moves that's that's a big way that that that would make it easier for integrators to see these these transfers and credit deposits another way you know could be give them what they really want which is a get transactions API which actually gives them the transactions of all f and you know that's easier said than done but you know I I don't even know if it's that hard right like we we've all sort of experienced this parsing of these traces like it's a doable thing I mean it'd be really easy to like write a tool that does this like it's it's not so much that it's like difficult it's just that it's just kind of there's all these nuances that don't seem to be really documented and so like various node implementations could absolutely implement these APIs especially ones that have sort of custom ones but I think what would be ideal is like a standard sort of RPC API that that wouldn't require like protocol levels because I think it's going to be a lot harder to sort of sell but you could absolutely I just have any protocol level changes at all we wouldn't need any consensus changes it would be a new kind of RPC API extension to web 3 or something like that and I think people you know from from someone that that for us because we care so much about platforms and having developers use platforms you know if they have to go for a talk like yours you know great talk but if they have to go for a talk in order to learn how to do it that's probably too hot you you want to just give it to them right away just give them the the API and it should do what they want while we're on the topic of things that I'd love to see nodes implement is the I believe it already exists in the EVM as far as I know I think it came with metropolis is the returning data with a revert opcode being able to have actual error codes I can send back in my errors and not have to do extremely tedious solidity debugging for hours to find out my typo so there was I think there was a talk about that I have I didn't see at this time I'm sure there was yeah trying to standardize standardize the return codes yeah okay yeah I know they implemented in metropolis the opcode to actually return just arbitrary data like you can pass a string to revert but I haven't heard anything for standardization yeah I think there's someone working on it I think it's really important I mean it's like kind of like you have HTTP status codes you should have returned all codes yeah yeah one other thing I think it is interesting that would be be cool to see it's it doesn't exist but it's it's like some more sort of infrastructure or sort of tooling around like running nodes and running nodes and sort of like a high availability setup right and a lot of the nodes of today sort of expect you to run them in this sort of single configuration of just you have one thing running on one box and you expect it to never go down and never break and that's just not reality right these every piece of software will break at some point it will go down and so if you have a lot of sort of transactions stuff going through also you can't just use one like it just wouldn't work at the sort of scale you may see and so you need more than one and then you get into all sorts of issues where you make a request to one node through a load balancer and then you make another one and you get a different result back because it hit a different node that's the different part of the blockchain so you have to sort of deal with this in like interesting ways whether or not you sort of take data from multiple nodes put it on your own database and sort of handle that inconsistency there or I mean just run one node I suppose is an option there's a lot of choices here but there's not much information on that and I think other blockchains have other like proposed ways of running them and more of this like high availability setup for like if you want to build like you know reliable sort of infrastructure around these to make sure that you know nodes aren't your problem and you can just kind of focus on on the chain I think we need some better tooling there yeah I think that sounds like a great grant project actually I mean think about it right like because we have like I think Ethereum is really cool we have like the most clients I think of any blockchain probably most different implementations it could almost be like connect to the web 3 of gas and parity and be like a load balancer across both and and only accept things that that both of them agree on or come up with yeah we have an interesting setup that kind of does that but not about the agreement thing where we actually sort of chain them together we chain together different node implementations so that if there is some sort of consensus breaking rule in one of them it'll sort of stop there and not sort of get to our the rest of our systems because like these things happen and this is a way to sort of mitigate them and so yeah trying to sort of detect consensus and detects like 51 percent attacks and things like that there's probably a lot of interesting tools that can be produced out of there that I think a lot of exchanges into companies building are are doing on their own but there's no sort of shared stuff out there that's actually quite interesting but also risky so when you look at like consensus breaks sometimes what people do is they attack one of the implementations and cause it to you know maybe see a block as invalid when it should be valid and you know the safest way in my opinion is to connect to both of them and then see if there's a substantial difference in the two chain heads by you know a significant amount of work right if you chain both of them is actually good in that you well kind of good in that you get the most valid one in that both of them will see it as valid but that could be something out there that has more hash power that you don't see as valid so right yeah it's sort of it's sort of like it gives up uh it's a liveness right in the event of a chain split it's what it gives up liveness since instead of like just keep running on and like hoping you're on the right side of the chain it says we're just going to stop and we're going to figure out what's going on here and we're going to investigate it and then attempt to make the right decision there and catch up or whatever we need to do but um that's sort of that's a I think a great point though that is the sort of trade-off it makes to do something like that we're going to stop the world and until we figure out what's going on yeah everything okay the world weird all right I think I'll do the I'll do this last question then I would love to do some some audience questions if we have them in the last 10 minutes there so um we'll do an easy one which is uh so what's so what's your favorite thing about working with the theorem versus the other cryptocurrencies you work with morning um obvious answer smart contracts right there's stuff there's so much more that you can do so many more interesting creative solutions to a lot of the problems that aren't specific to Ethereum necessarily um things like it does offer its own share of problems but and and really just the community like there's a huge community around Ethereum and building this out that does make it never a boring day right never a boring day it's it's always interesting there's always something new happening it moves so fast that you you have to almost struggle to keep up with it sometimes that's a good problem to have yeah I think that the community in Ethereum is this for me one of the strongest probably the strongest one um it has I think you know what the most developers um which is a great indication for me as an engineer of where it's going to go um I think that in general you want to build when you build something you want to see like other people are using it so there is adoption as well and you you feel the most you feel more welcome building something on Ethereum um it's like you know hey I built this and and everyone likes to try and see what kind of depth you built and I really you know I really enjoy that yeah I think that's a big part uh I think yeah similar answers for me is yeah I mean that what people can do is smart contracts like I don't really do much work with them myself but to see what people are able to produce especially like all the recent sort of financial instruments and stuff things like like die existing and you know the ability to be able to create some sort of thing that has a stable value without anything actually behind it is this really interesting concept or all the people are doing with you know the the relays and all the the variance of you know loans and all that is somehow happening in this world and it's kind of crazy that that happens and you can just very easily go on to some you know some wallet app and instantly you know take out a loan or something and and that kind of works um and then the community is yeah also really I think generally healthy community and doesn't have as much of the sort of tribalism you might see across all different blockchains um there's you know some of it of course as there is anywhere but generally I find that the community is very sort of open and willing to work with other blockchains and stuff and try and push sort of everything forward I think like one great example this was the um the talk yesterday on verifiable delay functions um that they're talking about during in Serenity is this the it's this thing that's not specific to Ethereum like they're actually bringing in other chains and want to work on this together in collaboration and come up with this just better cryptography and better system to you know have this you know new concept to work for for everyone and not just be this like singular thing for Ethereum like it's you know our our baby and we'll just you know keep it here and nobody else can have it they're just like let's let's share that with the world and let's you know make it all better because it's we're all it's so early days and all this stuff there's a lot to figure out and I don't know who the winner is going to be and maybe we'll all be winners maybe none will but we want to you know all just work together and try and be healthy and get there all right okay so we'll open up to questions if anyone has any all right I'll just hop down here with this microphone and to you so you guys are on the front lines I think they'll have a really good perspective on this I'm curious what which projects you guys think belong on chain kind of where you draw the line for that and you know as assuming that we improve efficiency and scalability and stuff do you guys think that will change in so how yeah I'll talk first yes I I have I'm conflicted there sorry I I like on chain just because it's sort of the singular place to integrate with you know the title this is integrating with Ethereum and it's as we've talked about it's already sufficiently complicated and the more things we add on top of that especially layer two things and various protocols it becomes even more complicated so the sort of onboarding cost gets a lot more expensive and it's not as easy unless we build tools that can somehow do this but if you know now some new protocol comes out that you know all of the exchanges all the wallets need to integrate and you know you scale that up you know it's a multiple different ones that's going to be fairly difficult and so keeping it on chain you know has some nice guarantees there just simplifies it overall but clearly that has sort of you know scaling and limits and stuff and there's a lot that can be done off chain and and maybe you know publish there as just some proof and I think there's also a lot of value in that if we can have you know really good sort of second-layer solutions to you to pull it off yeah so I would give it like I think there's a trick answer to that because you said on chain I think like side chain right now is the best because you know with side chains the checkpointed into the main main net you kind of get a lot of the advantages there of being on chain in that it's really easy to integrate because people can see your theorem transactions and you've got an EVM on the side chain so you can run quite a bit there but it's not going to scale eventually still I mean a single that could still take up you know all the gas on its own side chain in the end so but for now I think where we are at that's that's that's the best we have and it's going to get us there until 2.0 yeah I kind of see both sides of it like for simplicity doing it all on just the main and theorem chain is definitely the easiest way to keep it simple and for scalability purposes it's definitely nicer to do it on a side chain of some kind there's I think some kind of balance there um I would at the very least I think anything that should be anything that's considerably expensive I mean you can like crypto key anything that clogs the network or sufficiently expensive or has sufficient network should be in some side chain or shard or something off the main chain of course um anything more simplistic that maybe either doesn't have traffic or like I'm not sure if it makes sense for like new developers coming in after like 2.0 drops and stuff like that should jump right into oh yeah we're going to develop our thing right into a side chain this is my first my first app is on plasma and it it increases the barrier to entry for Ethereum which is one of the things that makes Ethereum so great in my opinion so there's a balance there depending on each individual app all right so I think you your moderator asked a great question earlier about what was your favorite feature of Ethereum but I'd like to ask the opposite what is the biggest pain point that you as scaling Ethereum developers deal with I know that Morgan from shapeshift you mentioned earlier that dealing with the lack of return codes was a pain point is that the biggest pain point what's the worst part about dealing with Ethereum at scale well so I think there's really two sides of that is there's the the solidity side and the smart contract and what's a pain point of dealing with those and that that whole ecosystem and then the server side architectures and integrating addresses or do we choose a wallet contract um personally having doing both sides yeah error codes is a huge pain point for me um it's it's a huge time sink trying to debug issues there when you have essentially two maybe three error codes that all basically say the same thing it's definitely a pain point I think it's the internal transactions challenge where other people are not viewing them easily for me um because that you don't have much control of yourself uh you can yeah no matter how much time you spend with yourself you're not gonna get someone else to accept the transaction which they can't see yeah I think there's a few um I think like that's a huge one that's like prevents people from sort of getting on board and getting there I think there's like there's things we can do that will make that make that easier and these are things that like eventually once you get them kind of working correctly well they do become not so much of a pain point um but they're hard to get right the first time um things that we see like you know regularly I'm going to say two things one thing we see regularly um not much recently everything's kind of died down but all the gas price stuff like earlier this year all the time gas prices spiking things get stuck and and that's a pretty big pain point and we don't currently have systems um on ethereum that sort of automatically recovers that and we talked about some ideas earlier that could help with that um the other one um that's more of a rarity but also similarly for like building these sort of high volume systems if you have to start tracking these nonces and stuff yourself and if you get one out of order you're in a really weird spot because if you're doing like multiple sends for customers there and you like skip one now you have so if you have like four or five six and you don't do five for some reason you've got this hole and so the other one's just going to be pending forever until you fill that in and like what if you like I don't know retry something else and then you send another transaction you can't cancel this six now like it's out there and so you have this weird like like we can't introduce like canceling transactions that would be like not really possible but um that sort of thing of like calculating these nonces like you can't because you're not just using a single node you have to sort of track that yourself and make the right decision there and so that's a challenge to like do right and then when you get it wrong um specifically going too high going too low the node will reject usually um it just depends where like your nodes sink that but um going too high creates this thing where it's like oh yeah now what and you don't know how long it's gonna last it's in the mempool we had a thing once where you know we filled in the gap and it went out we didn't expect it to and it's like oh well I mean it was okay but it's not a fun thing all right any other questions out here all right we got one way at the back we're gonna run I know standardization sorry better standardization for like your server architecture or or smart contracts themselves they're closed um where do you guys feel the governance process is for that are you guys talking one to one to each other or is that on EIPs or um it's kind of like how do you have those conversations together in the industry to to get stuff done yeah that's a good question one thing real quick on that like that's one of the reasons why we wanted to put together this panel um is to get people talking to people talking public I think there's a lot of sort of secrecy a lot of things are shrouded and they shouldn't be and there's no real reason for them to be um but I think we all we all approach things different ways like if we were to talk about how we all run our nodes or how we all do not stuff I imagine we'd all sort of do it differently and there's things to learn from each other there like we've already learned things talking about you know the contracts the different tradeoffs there and the way they do things in the way we do things and I think the the big thing is yeah at least get people talking I don't necessarily have an idea I think the best place to do that there's definitely you know some you know just every now and then you talk with somebody or reach out but I think it would be really cool to see more sort of open spaces to discuss the things and especially with sort of you know uh like big participants of the network being able to contribute to those conversations maybe I think it would depend on the company because I think also companies like to do a lot of sort of protecting of their information too and that becomes sort of challenging um but I'm curious it really is like all open conversation and it is kind of segmented that conversation right now whether it's Gitter or Stack Exchange or conferences or GitHub issues there's tons of places where a lot of the same conversations are happening so I don't really know of any centralized place where this is all happening as of right now um but it is all about really conversation and just seeing what other people are doing this is a very new space and we're all kind of just figuring it out and a lot of things you almost have to learn from either experience or listening to somebody that did get the experience and made those mistakes already do you think like it is interesting because it is the thing that I don't think are people talk about enough is sort of the types of problems we were talking about today I think a lot of the people working on their protocols and stuff tend to be very low level and closer there and have you know specific ideas around how people are going to run nodes and stuff together um but I think that you know at sufficiently large volumes you have to do it differently and I don't know of a space where people really talk about that yet so I think definitely ether research would be an interesting place to to sort of discuss that or ether magicians or any of the sort of forms like that all right we got we got one quick comment here so um just another vote in favor of just some sort of central group group chat I don't care the venue but when I'm having issues with like all these like holes and nonces and sending transactions and like all that stuff just if we can find some central group chat where I can ask you guys a question that'd be really nice so thanks so there actually are a few um but at the same time when they get too large then people start leaving and start tuning out so it's a it's a challenge um what level of noise you want to be at and how much uh how publicated yeah I think it would be good for us to do blog posts or something about this so we've only got 30 seconds I know the panels out here but to end on a an interesting note it sounds like there's a lot of challenges from a developer point of view I know that's what you engineers love um but like why are you here what's so exciting about you to be in this space in five seconds none of the challenges are insurmountable and we're seeing the one to scale and the one to get better here and I I think the community is great and that's what I mean yeah I mean sort of the the privacy and sort of ownership of sort of your own funds and you know all the sort of developments there like fascinating to me all this sort of talks today about all the privacy and all the cryptography and stuff is it's interesting I think can just lead to a lot of you know good in the world and people sort of you know all right all right we gotta we gotta kind of wrap it up here if you morgan if you want to five seconds real quick brand new preta set of problems to solve that haven't been solved before and we can solve them together