 Hi everyone, my name is Reta Trinkler. I'm the chairman and co-founder together with Mona Lisa of Mellonport and the Morrison advisor to the Web3 Foundation. So great, so let's start with formal verification. So formal verification is the act of proving or disproving an algorithm with respect to an intended underlying algorithm. So this algorithm is specified in a formal specification. So I'd like to ask all of you what makes a good specification? What makes a good specification? That's a good question. So I think theoretically the answer is that the specification accurately captures the intent of the developer who wrote it. Making sure that's the case is obviously very difficult, so I think a lot of that depends on sort of your tooling and the feedback that it provides when you're writing the specification, but in general I'd say a good specification accurately reflects the intent of the developer and also encodes sort of all of the assumptions that the smart contract makes very explicitly and very clearly so that if someone were to update the smart contract later or something they would all be very nicely in the same place. I just want to add a little bit more. I think a good specification should be easy to understand and should be easy for other to read and build test case later on. And it's great if it follows some standard as well. I think in general any specification is better than what most people do, which is no specification. So that's probably a good place to start. Ideally you'd write a specification and then argue over that for a few days at least and then write your tests and then begin coding. But a lot of people kind of do it maybe in the reverse order or something like that, or maybe they never even write a specification, so. If there is an answer, capture it, but maybe a good specification should put a contract in a bigger picture, what kind of game people can play on it, what would be the best strategy for rational players and so on. Yeah, if a specification fits into this crypt economic or economic, game theory, reasoning that something nice. But yeah, this depends on the application. Thanks. So what is your take on a specification kind of specified, abstractly with formal methods of mathematics versus a specification implemented, for example, in Haskell? So one is kind of like a reference implementation in Haskell and the other is just kind of abstract, purely theoretical. Well, I think that a reference implementation in Haskell is a perfectly fine specification, honestly. I think a lot of people when they think of a specification think of kind of just an English language document or something like that, but at least in our group we focus a lot on executable specifications. So I think a Haskell implementation, that's a whole hell of a lot better than no specification once again and at least you can test it, right? You can run tests against your specification even before you go to implementing the final thing. So that's already a huge advantage over an English language specification, for example, but certainly English language specifications are easier to write. So maybe start with that and then do the Haskell or something like that. Yeah, I totally agree with everything that was just said and I sort of want to add that you can have good or bad specifications in any format. So absolutely while it's nice to have a specification be testable and clear and readable, you can have really, really good Haskell language specifications or you can have pretty bad Haskell specifications. And I've even seen English language specifications that are not entirely bad. So sort of goes more to the expertise of whoever's implementing it, I think. Great. So like, could you maybe share a bit about what is like new and unique about formally verifying smart contracts blockchains kind of compared to, you know, safety, critical hardware verification and what are the corresponding opportunities of it? And what is new and unique about formally verifying smart contracts or blockchains compared to like, for example, verifying hardware and what are the opportunities of it? Well, I guess smart contract has a new sort of like romantic, right? So we have the concept of like transaction at the concept of gas and solar token and ether, which, you know, doesn't exist in, you know, existing system like, you know, critical hardware or software system. So when you like formally verify it, I think you need to define new property that you want to verify that only exists in, you know, smart contract and also Ethereum. There's definitely some really nice things about smart contracts. They are bounded in terms of their execution time. They're generally very small, simple programs, at least compared to something you'd see in a safety critical system, a car or an airplane or something like that. And they directly handle large quantities of money. So actually, I'm really excited. I think that I said this last year, and I still strongly believe it, that as a space, we have an opportunity to set standards for how software is developed even beyond blockchains and smart contracts. Because I think the users of smart contracts and blockchains have a much stronger understanding and demand for this kind of technology than the users of a car or something like that. So Ethereum virtual machine has no non-determined behaviors, no non-defined behaviors, because that means consensus folks. So this possibility of different clients following different folks is a huge problem. But it's really a blessing for us on top of the virtual machine, because that means the specification is really always deterministic. That's a huge improvement of proving anything over Intel chips or ARM chips. Yeah, you're right. Pretty much agreeing with what people said. And the non-determinism makes it easy. The bounded gas makes it kind of nice, although sometimes it's hard to reason about exactly how much gas a contract will use. And basically, you know, what's making the most driving force behind this is the amount of money being at stake, essentially, as opposed to just normal programs in other settings. It may move over, because it goes really, it goes to make it really hard for me to understand the question. Yeah, it's kind of interesting how people care a little bit more about money than their lives, right? You have someone lose their 50,000 endowed tokens and they're clamoring for formal verification. But then Toyota has an issue in their ECUs that kills 20 people and Toyota customers could not care less about the software in their car. So I just think that's kind of an interesting observation on humanity. It is kind of interesting, yeah. Great, so could you maybe talk a bit about the limitations and challenges of formal verification and maybe also introduce the project that you're involved with and how your project kind of tries to make things easier or better? Well, so my background, interactive theorem proving is not user friendly. So take class of computer science students of 30 people, maybe a couple of them are willing to use interactive theorem proofers. Maybe only one of them can finish verifying a program of like 20, 30 lines in the months or so. So it's kind of challenging. Actually verifying software using these interactive theorem proofers is so challenging that I don't see much, I don't see many successful applications. But for the reasons described before, I believe in a theorem it might still make sense. I guess it's really hard to formally prove existing smart contract, especially the complicated one. Because first of all you need to represent them in some mathematical model that you can easily prove some property later on. But it's really hard to convert or transfer or interpret the program into the new language or the new model because of the compatibility issues. And also sometimes you cannot even explore all the possible outcomes or properties of the program. And most of the formal verification tool requires strong background and human involved. And most of the project and companies do not have these people. So really we need some sort of user-friendly developer tool so that you can just run the tool with some existing smart contract and it will produce other helpful output for the developers. Yeah, so I work on sort of a number of projects in the correctness space. One of them is I'm advising this sort of heavier weight formal verification project called KVM. I'll let Everett describe that since he's actually the lead. But to speak to the broader point I think there's like a very wide range of correctness techniques that we have here both formal, informal, and semi-formal. And another one of the projects I'm on, HYDRA, is sort of a lighter weight approach to this where you abandon full formal guarantees for sort of low cost and easy startup. Which I think is a very valuable thing to a lot of companies who aren't going to be able to, as Lloyd said, hire the right people or have the right expertise. At the end of the day, formal verification is still super expensive. It's still hard to do and it still doesn't give you perfect guarantees. So it's understandable why for a lot of people, especially now that the tools are not totally mature, the business case sort of isn't there. So I think we need to develop a range of techniques and ask ourselves what are really our high assurance contracts here to use our full-fledged techniques and how can we get the other ones to a better level of security with like the lower hanging fruit techniques? Yeah, I think I agree pretty much with what Phil's saying here. And actually it makes me really happy that I see so many people just emphasizing the importance of testing and of writing specifications because just those two steps alone force you to first clarify in your head what is it I'm trying to do, which so many people don't even do. They just sit down and start kind of plonking away at code. And then second to kind of concretize that by writing some tests and then say like is what I did actually implementing these tests. So the tests kind of serve as a semi-specification if you will or a concretization of the specification. Regarding the KVM project that Phil mentioned, I think it helps with the entire, there's the entire range of verification that you can go through, right? You can start at testing and you can move up to some sort of runtime verification or runtime model checking up to model checking, which is where you explore the whole state space of a program and then up to theorem proving where you use more advanced techniques to prove that a program is behaving correctly and K gives you tools for all of those things actually. So I think KVM can be a useful tool in that space. It doesn't have quite a nice user interface unfortunately, so it suffers from that, but I think it can help a lot in all those areas. Thanks. So it seems like a common theme seems to be it's like too complicated. So how can we make tools and or how can we develop approaches that are more accessible to the end user? I think in the end it's not reasonable to expect the end users or users to read logical formulas or maybe K configurations that are more accessible, but it's hard. So at the end what we want is a tool that can synthesize an adversary based on the unproven goals where this scenario can happen and it can play it on web browser. Maybe not an animation, but like it should look like somebody attacking your contract. That would make the problem much more easier to understand. So I guess usability is always a hard problem, right? And that's also one of the motivations that why we build O&T, the smart contract analyzer based on symbolic execution. So the idea here is that users do not need to understand how O&T works. They only need to paste the smart contract into O&T or into the web browser and they click analyze and we can just show all the possible vulnerabilities that O&T can figure out. Yeah. I personally think demand sort of breeds progress in this direction and we're sitting in a room of some extremely smart and talented people and we've seen some projects that really push the envelope in a number of spaces. And while the tools are hard to use and there certainly could be improvements made there, nothing here is sort of magic or anything like that. So I think sufficiently motivated developers who are incentivized by the market to develop these tools is really what we need to push the envelope here. Another thing that I think is really helpful and I'd like to especially congratulate Melonport for taking this approach, is to take these things out of academia and into industry and use them on real contracts with real developers who are working on the ground in Solidity. I think just that step could do a lot for usability and academics are not really the greatest at writing super usable systems sometimes. So more broad participation would definitely help. Yeah, third thing that you just said he seconds it. But yeah, definitely the approach that Melonport is taking seems to be the correct one, just make the tools usable and make them have check for some specific properties automatically. Certainly all of the verification tools can be instantiated to do those sorts of things but it takes effort and takes developer time and it takes people who know the tools to do that instantiation. And that's not gonna happen overnight. So these lighter weight methods like Phil's Hydra stuff that's really gonna be the stop gap until we can have better tooling for all these high assurance tools. Thanks very much. So the last question from me and then opening up to the audience is essentially a shout out from you guys. So what would you like to see kind of from different ecosystem players? So from the users, the developers, the investors, the project leads and the researchers to make formal verification mainstream. In the current ecosystem, I think it's still the user's responsibility to make sure that they understand the contract they are interacting with. Some people are like proposing markets for audits and so on, but these are not working yet. So actually I have to repeat Martin, I mean it's hard to say this but you have to read the byte code, learn the EVM in order to make sure that you are interacting with a reasonable thing. It's hard to say this but it's something like that currently. And then yes, something like audit market with the reputation system or something like that very gradually it may improve the situation but yeah, now due diligence is required for all parties I believe. So I see there are sort of being two pushes that need to happen. One of them is bottom up. The users need to really start demanding that if there are techniques available to ensure the correctness of their contracts, they're either used or a good justification is provided for why they're not used. We certainly have a few tools right now, we've seen presentations today about quite a few of them. So users really need to start being more selective about where they put their money and while sure it's tempting to sort of buy into every single ICO and flip it for 2x or whatever, that's not the kind of space that sort of incentivizes and fosters rigorous proper software development. And so I think users and investors need to become more selective and developers need to sort of enable them to do that by really going through the processes in a way that's serious and in a way that they put in a concerted effort to get the maximum assurance out of their contracts. Thanks. Yeah, kind of echoing what you each said a little bit. No amount of money is a substitute for just understanding, right? So at some point you just have to read and understand the bytecode and understand how EVM works and understand how Solidity compiles to EVM or whatever tool it is, whatever high level programming language you're using, how it compiles to EVM and what the implications for security are in that setting. And the people doing security audits, they understand this really well and they, like they were saying earlier, they just read the code and read it again and read it again and read it again. There's no substitute for just putting time and effort into high assurance software. Well, I guess, you know, in order to make formal verification popular, I think we need to adopt this first, right? So I mean, I hope that we can have some sort of like ERC that, you know, that allows, you know, people to put, you know, maybe Oriented tag, Hydra tag or, you know, K-EVM tag into every single smart contract that has been, you know, deployed in the Ethereum blockchain. Thanks. So anyone from the audience would like to ask some questions? Yes. Maybe it's better if people with questions should come to the front. That's probably a good idea for the next person. So from an artist standpoint, how would you like to see people who provide smart contract audits either contributing to your projects or using your projects? I assume it's sort of maybe one or the other, but maybe there's both for us right now. Does that make sense? Yeah. I do do some smart contract audits and I would say ideally the role of these kinds of tools in the long run would be to sort of put most of that business right now out of business to an extent, like at least that's my personal goal with it. In all my audits, I always make recommendations to the team about which tools are state of the art and what they should be running on their contracts. Certainly, auditors can run these tools on the contracts as well, but at the end of the day, my personal opinion is that it's the responsibility of the team and the team to sort of seek help when they're not able to do something like this, especially because most teams would not be willing to sort of pay the hourly for an auditor to run, securify on their code or whatever when they could do the same thing themselves. So I think the role of auditors to answer your question is to sort of inform the team about what's available and what they think could be useful for the contract and then it's absolutely the team's responsibility to follow up on that and put the time in. Yeah, often I think the hardest part is just the spin up time, right? So you have some new tool that you're trying to bring into your tool chain and then maybe it takes a week or something to get it fully integrated and get it to be seamless, but once you have that spin up time taken care of, then you can kind of begin to make incremental progress. So what I would like to see is people taking the case semantics and making extensions of it. So we have already several extensions. One is a small gas analysis tool that's not very featureful or very complete or sound. It just kind of gives a rough estimate, but it'd be nice to kind of tighten the screws on that. And then there's another tool called EVM Prime, which is just a small extension of the EVM semantics to a higher level language. And if people wanted to hack on that, that'd be really cool. I'm answering specifically for KVM because that's the project I'm working on. But there's all sorts of other analysis tools you can write directly in K itself as an extension of language. And the hard part is the spin up time on K, of course. But there's no avoiding that with a project like that. Yeah. Just a button just for one second. It would be really nice also if these auditors and these teams sort of complained a lot more about the tools. One, one, one. Yeah. So for most of the research focused around either tokens or crowdsets, which are relatively smaller size. And also it's quite similar. But as more companies start delivering the product, which they promise to do based on the funding, the size is going to be a lot bigger and the product will be quite diverse. What would be the challenge when it comes to verifying these actual product rather than talking in the crowd cells? Please quick answer because we need to work. Sorry, the question was what are the challenges for these teams that are using these tools mostly? It's better without the mic, actually, I think. Yeah, so yeah, that's a great question. So the question was about most of the verification right now is tokens or crowd sales because that's sort of what everyone's doing and it's all the same. But what about when we get more complicated contracts? I personally think people are always going to push the envelope and do things that are more complicated than we actually can verify. And I think that's really cool to sort of feedback in. But the more complicated, the more experimental your contract is, the more time you're going to have to take and the more you're going to have to consult experts who really know how to write new specifications and not just reuse those same old building blocks in the old ones. I think auditing or finding bugs in the contracts is a bad game to play because you have to beat the best attackers in the world. So going forward, if you try more and more complicated stuff, you will need much stronger usable tools. Otherwise, the game is not fair. You have to find all holes. And a very good attacker should find only one hole, critical vulnerability. So it's a very, very skewed game. So maybe a good tooling or maybe a good game like Hydra is necessary. But I think we are kind of ready to take the challenge. Great. Thanks very much. That concludes the panel. I'd like to thank all the panelists. It's a huge honor to share the stage with some of the best researchers in the field. Thanks very much.