 This is a workshop we'll be talking about using agent-based models for analyzing DeFi protocol and the examples we'll be using is through the Vega Market Simulator. We do a quick introduction and then I'm going to jump to a slide which will show you where to go on GitHub so that you can hack along. I really assume it's a workshop, right? So I'm expecting everyone to have a computer in front of them and start hacking. But if not, then, hey, you know, you can just stay along for the chit-chat. So my name is David. I've been with Vega for the last four years. I mainly look after the kind of financial risk and economic design, game theory analysis. And together with me is Tom, who's coming from traditional finance. He's a quant with Vega, working on the research team. And one of the speakers, Mark, isn't here, but he's done a lot of the RL stuff. So he's the main brain behind that. So I'm going to jump to, I think, here. So yeah, we'll go back to introducing what Vega is in a minute. But given I don't know how fast the Wi-Fi is going to be in here, just running through a few first set-up steps that we can get going with while we're presenting. For anyone who hasn't noticed, there is Wi-Fi specifically for the workshops called DevCon Workshop. And the password is buildit22. So give that a go if the Wi-Fi is being slow. So this repo at the top here, Vega Market Sim, should be open. Over there, clone that. That's your first step. You're also going to need go19, I think it's necessary. Python, ideally 3.10, 3.9 might work. And poetry, just because it's probably the better package manager. There's a requirements.txt if you just need that as well. We can go through actually setting it up in a bit. Then if you want to be able to view our sort of front-end console, you'll also need some UI framework. Yarn and NVM for node management. And if you want to try some RL at the end, then PyTorch is a good starting point to download. So yeah, broadly, if everyone can make a note of that URL before we switch back, and then we will start on actually presenting. Everyone good? We'll have an actual break for setting everything up in a bit. Yeah, so we're back to the introduction, but it's sort of a workshop. And if you need help, the person who isn't speaking can actually come and help you with whatever it is you're trying to do within this framework. And if you're not having luck, then we can see if one of us will have better luck. And there are not so many of us, and we've got, I think, two hours. So we've got plenty of time to actually get everyone going and hacking. So in terms of what we'll cover, we'll quickly introduce Vega protocol. Then I will try to make sense for agent-based simulations in the context of analyzing DeFi protocols. And then, you know, we'll introduce what you already started setting up, which is the Vega market sim, which is relying on something called Vega Null Chain. We'll explain what that is. And then once we have that, we'll sort of explain how to use the market sim, how it's set up, you know, what environment is there, what agents are there, and what you can do. And at the end, we're going to show you how you can, you know, train reinforcement learning agents on top of this and why you might possibly want to do that. So that's the plan. So we start with what is Vega protocol. So Vega protocol is a layer one blockchain for derivatives trading. It's set up from the ground up for trading margin products. And the price discovery is either through limit order books or through auctions. Where do the auctions come in, typically, when you're setting up a new market? You don't know what the price is, so initially, it's running in an auction. The auction determines what the opening price should be, and then if everything goes well, it's running as a limit order book. If there are big price moves, according to the risk model, it might switch to a protective auction to make sure that the big price move is really, you know, what's happening economically. It's not someone with a fat finger, you know, forgetting a zero on an order or adding a zero on an order and causing mayhem. Anyone can create a market. What it means is you submit a market proposal to the blockchain and then it goes through the governance and it's voted on. So that's what's meant by permissionless here. And what kind of markets, at the moment, it supports cash-settled futures. There is road map for supporting any kind of derivative pay-off language which will be probably wasm-based, but that's just on the road map for now. It's cash-settled futures. And so basically, if you can find an oracle for the market, then you can propose a market and it will run. It's got a bespoke liquidity provision mechanism because it's not based on now ever-so-popular constant product markets, but it's based on a limit order book, so you have to do something slightly different to bond liquidity to a limit order book, so it does that. And at the moment, the assets that have value that you would use, a settlement asset on Vega, they're bridged from Ethereum, though in the future there will be other bridges. But Ethereum bridge is there. You can, on Vega, before proposing a market, if you wanted to settle in some other asset that's not there yet, you can propose the asset. The asset will then be bridged from Ethereum, so you can imagine that there'll be USDC, Feather, Ethereum, whatever people want to do. If you're designing something like a trading platform, some of the goals that you might have in the design really sort of run at cross-purposes between a general-purpose blockchain and what you would ideally like for a trading platform. So the things that I picked out are a general-purpose blockchain. You need something like a gas mechanism for people to pay for the cost that whatever transaction the chain is processing for them. The cost has to be covered by the gas. And if you're trading, this is really suboptimal in the sense that there are transactions that you want to include and you don't want to penalize them by charging for them. If someone wants to post a limit order, that's good. They're providing liquidity. You really don't want to discourage them. If someone cancels an order, that's good. They're providing price information. You probably don't want to discourage that either. So that's in products, they're effectively promises. If someone doesn't look like they're going to keep their promise, you want to close them out immediately. Ideally, you don't want it farmed out to some bots that are running somewhere because then you're opening yourself up to the TV and a whole host of other problems. So you can have atomic closeouts. Again, for margin, you want to do potentially quite maths heavy risk computations. And you don't want to be paying for gas for those on a general-purpose blockchain. You can try to do fair ordering or as fair as your definition of fairness allows fair ordering of orders. You have one specific application in mind, so you can optimise for latency, for instance, rather than optimising for data throughput. So you're not fighting at cross-purposes with other applications that would be running on top of a general-purpose blockchain, which would have other design goals in mind. One thing that I didn't put on the slide, but that's also a very good reason, is that sharding is actually much easier in the context of trading because something happening on this market can be completely separate apart from talking together through a treasury shard. So you have a very simple use case in the sense of what needs to know about what, you can design sharding relatively easily. Okay, so that was Vega. Now, sort of more general case for doing... Just trying to get rid of the mouse. More general case for doing agent-based simulations in the context of DeFi. So, you know, we've all seen various DeFi protocols, and we've seen, you know, when things go well, it's great, and then when things don't go well, it's called a hack or an exploit. But quite often it's not an exploit of a code bug, though, that can happen. It's simply an exploit of the mechanism design, you know. You go, you want to design a protocol, you start, you think, what incentives should I create so that I've got the desired behaviors? And you do it in isolation. You really try to think it through, right? You take the smartest people you can find. There are, you know, dozens, hundreds of incredibly smart people in DeFi. They do their best. They do what makes sense. They think it through as much as they can. But it's difficult, it's really, really difficult through all the consequences, especially if you have one incentive mechanism here, another here, design may be in isolation, and now they're interacting. Right? Yeah? Oh, Vendi is a name of Claus's Fairness pre-protocol for transaction ordering. He can tell you all about that. He's there. You can just go and show him, no? Yeah. Yeah, so you put two incentives mechanisms together if they were designed in isolation. It's not, you know, completely clear that the combined effect is actually what you desired. And then, of course, DeFi is all about composability. We love that. But once you introduce composability, you're introducing another list-proof stuff. You set up a hypothesis which is testable. You will never prove that it's true, but you can set it up so that you run the experiments that should disprove it. If you tried your best and you've not disproved it, then you let the hypothesis stand till proven otherwise. And if you're wondering what this is, most of you have probably seen Game of Life. It's the idea that, you know, you can take a bunch of very, very simple rules and they can lead to very, very complex behavior and sort of agent-based models are one thing that will allow you to analyze this complex behavior. So, you know, definition of what an agent-based model is from Wikipedia and everyone's favorite agent. And quite often you want a lot of agents or a lot of similar agents, so agent Smith is good. I've used this picture actually twice. When you talk about Sibyl attacks, I like to use that picture, but here it's not about Sibyl attacks. Here they're good agents who are stressing your protocol. So in terms of, you know, what type of agents you would find in agent-based modeling, the simplest ones are what I call zero intelligence, which is a bit insulting, but what it means is that they're not learning and they're not optimizing. You just say, this is the state of the world and if this is the state, then they're going to do this and if the state is something different, they're going to do that. So these are the actions that have hard-coded actions and that's what they do. They're the simplest to set up, they're the simplest to understand and, of course, if you hard-code the optimal actions, then they're perfectly good. If you don't, then they're only as useful as the actions that you've hard-coded. And so the next level, you can think, okay, you know, each agent, I'm not going to hard-code the actions a priori, but I'm going to tell the agent exactly what the environment, what the rules of the environment that they're in are exactly what the rewards are for interacting with this environment according to the rules and I'm going to optimize. In other words, I'm going to solve for the optimal actions and that's something that comes under the name either Markov Decision Processes if you're doing discrete time steps or control theory, stochastic control theory. It's a reasonably well-established field and there are a bunch of algorithms. I could spend hours talking about algorithms that they have for solving problems. There are a few problems that you can actually solve by hand. One of the most popular is the Linear Quadratic Regulator, which, you know, together with the Kalman filter was one of the great successes of signs of the 60s, you know, landed on the moon and all that. So, you know, good stuff. And then sort of the fancy new stuff is the reinforcement learning where you actually don't have to tell the agents what their environment is. They just repeatedly interact with it, observe the outcomes of their actions, observe the rewards and update their behaviors. So, these are the type of agents you might want to run, you know, on top of your environment to see what is happening and of course what is appropriate and what you need will very much depend on what you're trying to simulate, what you're trying to test, but basically this is your zoo. The typical setup for these things, sort of you understand, you have the agent, agent takes action, it changes the environment in the sense that it moves it from the current state to the next state, so sort of new state comes out, a reward comes out, and the agent takes in the reward and the state and again chooses the next action and you repeat. And of course you can have, you know, these setups that you repeat sort of forever until infinity, maybe with some discounting maybe you assume that there is some steady state or there will be a terminal time when the game stops or whatever problem finishes and you evaluate how everything went, whether it went well or badly. And then one thing I wanted to point out here is because it will be relevant if you're looking at any of the code that we're providing, is that the agent, as it's learning, it's sort of, it's in a state, it takes an action, collects the reward, sees the new state and writes on the new action. And if you record this, that's sort of all the data that you need to train the agent sort of offline, separate from the environment. If you collect these SARSA sequences, then you can train the agent separate from your environment. Okay, so that's not so important now. It will be more useful information later. Right. And I think I'll hand over to Tom now who will walk you through the vega market sim. It's a workshop. So if you have questions, I think the gentleman there already said, yeah, you're setting a perfect example. Keep asking questions. Oh, do we need a microphone? Hello. Do states-based approaches have any role to play in parameter optimization? Like a control theory-based approach for optimizing an effect. Yeah, yeah. So you would set up your environment which in the case which we will show would be the whole market, how the trades happen, how the orders are matched, and then you run that whole thing through the simulation, see what happens and then you might want to do the optimization on top of that. Right, but that's an agent-based approach. You could use analytical solutions. Let's call it an agent-based approach, a numerical approach, but a control theory approach. Yes, if you have an environment which is sort of sufficiently simple that you can really write down all the state transition equations, then you can go and solve it through either pen and paper if you're supremely lucky or through something like policy iteration, value iteration. You could use something like Matlab or what that's actually numerical, but you could use software, obviously, too. Yes, and there is software out there, yeah. Does this work for creating bots? Trading bots. Yeah, I mean like the whole framework you have here. Yeah, so we're... I mean Tom will talk through that so I'll jump a bit, but basically, you know, the environment on its own, if it's just the market, you know, it's an empty market. It's out of the opening auction if no one proposes anything. So you automatically need bots, so we will talk a bit about the bots that are provided as part of the environment, and then, of course, you can write your own bots that can do, you know, well, whatever they want to do. We've got some basic market-making bots. We've got some basic, you know, price-taking bots. We've got some basic momentum traders. But Tom will talk more about that, so I'll hand over to him. Just a quick question. So in this type of systems, do you take the code from what you are testing? For instance, for the Vega protocol, just quickly check that it's all go code, right? Yes. Okay, so do you take this code and run it inside your simulation, or do you create a model of that? Okay. And do you think that that's scalable if you want to... if you want to simulate thousands of millions of agents, because perhaps you need that. Or do you think it should be more, in some cases, you need a model of what you are actually testing? So we're running the whole Vega stack, again, Tom will show that, but you're absolutely right, and in some situations, especially if you want to do reinforcement learning, having a simplified model on which you can train and actually go on to the full stack, which will be inevitably slower than you would like, because things are never as fast as you want would be useful, but we don't have that. At the moment, we're learning on the whole thing. That's why it's important that we use an algorithm which collects the Sarsa sequences and so then can learn offline, but we don't have model. There are reinforcement learning techniques which sort of build up model from observations and you can do a lot of clever stuff, but I'm far from being an expert on that. Thank you. So one thing that I didn't say is one reason why DeFi is so good for agent-based modeling is you can try and people have been talking about agent-based modeling in the real economy and economists would love to do that, but they don't have the environment on which to execute it. And in DeFi, we do have that, because things are typically open source. I can go and take the smart contract and run things through the smart contract. I don't have to create the environment. I just say, okay, I care about how the smart contract interacts with this one. I take whatever ganache hard head, I put it there, I set it up as my environment and I pump thousands and thousands of simulations through that. I can't do it in the real economy, but here we can, so it's awesome. Okay, more questions. Can you talk a little bit about the reward states more in terms of kind of what the stopping criteria might be and I guess in the context of the derivatives market, are you really looking at things like protocol solvency or profitability of certain agents that are performing trades or kind of what are the reward metrics? Because I'm assuming they're not like fun but more financial kind of outcomes. So for us, it's very problem specific. So for example, if we want to test what some of the network parameters should be, we set up an environment and then we look at things like if I've got a rationally behaving market maker and the parameter is X, what is their return on capital and I'm thinking well, maybe this is letting them earn too much money, it doesn't make sense. But in terms of the reinforcement learning guys, you would typically look at financial rewards and then you can either run it thinking of it, maybe you're thinking they're trading perpetual so it's just forever or you're thinking that they're trading a settlement market and so the reward comes at settlement, but it's very much something that you have to design for the specific purpose that you have in mind, this is sort of more the toolbox and I'm going to wrap up the questions, hand over to Tom and then you can ask afterwards because I have a feel that with the questions we're sort of jumping ahead a bit so I want to hand over. Thank you. Discussing the market simulator, we'll get to doing some ourselves in a minute but first I think it's useful to go through and cover slightly more technically some of the questions, how we run these simulations and how we get interesting results out. So the Vega market simulator, broadly the way that Vega itself is set up, we run with a tendermint consensus layer for those who don't know, tendermint is a cosmos based consensus, the problem with that is that it's generally tied to our block time which is about a second so if you run the whole thing you can watch the market going second by second and you can interact with it but you can't run as the question in the back said thousands of scenarios, run it through a year, do all these things so what we do, we strip that out we run what we call a null chain entirely on one PC and this just broadly accepts any transactions it's sent it still checks, they're valid it still runs through the whole Vega core logic but just adds into a block, allows you to control time forwarding so you can it'll just sit and do nothing until you tell it to run forwards or fill up a block and so you can inspect at any point in time for however long you want what's going on. Then on top of this once we've got that running we have in Python API layer which lets you express your trading actions, your agent's behaviors in much more market primitives without having to worry about the fact that you're running on a blockchain. Occasionally it still creeps in because that's the way the world but you can broadly talk about an agent trading on a market without having to worry about what block it's in whether it's been confirmed and this kind of thing. Once we've got that we then build out scenarios and our tests so I'll cover possibly next slide exactly what a scenario is but we can with a range of composable agents composable setups build out scenarios to investigate and test either certain parameters test that the market doesn't explode, test the core itself doesn't explode or if you just wanted to build an agent you can try and have a realistic market and throw yours in there. How this all works is your Python layer manages we've got two different kinds of interfaces at the moment a Vega Service Null which spins up this as I mentioned a service entirely on your machine or what we've recently created which is Vega Service Network and that allows you to connect to a running external network and run the same agents that you've just been running locally but on the real network that's going on so that can be useful in some cases too it allows us for example on our test net we can run agents and simulations that other real people can interact with and see how that's working we'll demo all of that in a bit so yeah back to setting things up and viewing I'll go through the viewing in a second but yeah I was going to break here anyway so we're back here I don't know how many people have managed to get to the end of this but I think we'll probably take a 10 minute break to catch up anyone who hasn't we can wander around and see whether everyone will manage and then 10, 15 minutes we can come back and continue with doing something more interesting with it. If you think you've got it set up try and run these scripts at the bottom we can also tell you something more interesting to run if you're stuck I think I had a question yes so I can then more in a more real way basically test my ideas it's actually it's something we're pretty much currently working on is being able to better snapshot an existing running chain and be able to totally restore from that there's the capability you can run your simulation and save that and re-run the same thing so you can have an example that you run and then run your things after we don't currently have a way to pull from an existing network okay thank you very much that people should be using in their protocol and I tend to tell them that they can more easily solve for tokenomics problems by including a market mechanism to optimize and let the market do those kinds of things example Uniswap v3 solves the hard problem of how to set the curve by allowing concentrated liquidity so my question is do you guys have an intuition for which kind of simulation which kind of systems require simulation and which kind don't good question I guess I'd go back to as David was saying earlier it depends on how well you can if you can solve it mathematically that's probably always going to be optimal but when it reaches a certain point of complexity yeah I don't know how you would do it without either an agent based simulation or just the casting modeling or that kind of thing is there an upper limit to the kind of complexity that you can model in this because I'd imagine like in this example of using a market mechanism that adds a lot of additional complexity would that be an issue in the Vega simulation in terms of complexity of the simulation yeah I mean it multiplies the number of agents in the system by orders of magnitude if you have to use a market to source lots more participants I guess is there an upper limit to the kind of complex system that the Vega market simulation can handle well practically I guess there is enough limit the system itself we're hoping will be able to scale to many thousands of people so that should also scale to the number of agents so I guess it's a goal of both the Vega protocol itself and the market sim thankfully have similar things because ultimately what we want to do is build a realistic market as long as we can run a market on Vega protocol itself we should be able to do it in the simulation cool thank you guys going back to the points that we were making I absolutely agree that if you if you're designing tokenomics or protocol novel then either you can prove that it works or you can use this agent-based simulation to sort of not prove that it works but show that it doesn't and go back to the drawing board in terms of training there are two questions I can have the environment I can have some agents that the agents take their actions quickly and get on with it but of course the moment you have more than one optimizing and learning agent then it's the whole game theory and they're probably it's also very difficult to set up or very difficult you have to think carefully about how you set it up because is agent A also trying to learn the optimal response of agent B so that it can anticipate etc and then they can end up chasing themselves in circles and all sorts of weird and wonderful things can happen it's game theory and RL comes in and it's very problem specific but you can end up with situations that they don't learn you can end up in situations then they take a lot longer to learn because there is more than one learning can I have a show of hands of the people who are actually trying to set it up on their computer okay okay okay so we're sort of now is the time to take the break and we're gonna have a look around okay so I think a few people have it running at least we're gonna I'm gonna crack on and then we'll have another break in a minute for doing the next bit so we can keep going around for anyone who's stuck on this bit but wanna give more interesting things to do on top of that so a quick aside once the the null chain is up and running the marxim is up and running you see anyone who got it working would have seen something like on the right here is it's a printed output log of some stuff that's going on inside marxim exposes various APIs for this but often you don't want to just have to manually call python APIs for every single thing that you might want to experience so we've got a couple of useful portals on your data you can if you've set up the UI components as well change a flag this run with console equals true in generally wherever you happen to be running your Vega null chain that will bring up in a web browser the nice console that you see at the front there that is the same console that people use on Vega protocol itself so that has theoretically pretty much everything you need you can even trade manually through that as well if you want to log in as your agents that you've set up will probably demo it a bit in a bit you also can it's actually always launched if you check the logs there will be this GraphQL port and if you go there and you know GraphQL you can query the entire system with fairly hopefully clear GraphQL queries on docs.vega.xyz there's a link in a bit the entire structure of your GraphQL queries that you want as well so for anyone who has got it running for a more interesting scenario that's something you can try out running I'll leave it up actually let me skip to the next thing this is what I'm going to talk through with this up so anyone can type it up so the components that Vega itself runs on a validator node that we're also going to be running locally that also spins up there are a few different ones we've got Vega which is the core process that's the go blockchain client that does all of your transaction processing and outputs your states that is obviously it's a point in time proceeding thing so we also want the data node which is listens to external events output by the core node itself and saves those into a Postgres database and allows you to query all of that historic data and that's generally what you'll be using when you're querying any data from Vega so additionally there's a couple of extra things I mentioned previously that isn't necessary but it's a very useful front end there's also the Vega wallet the wallet is basically science transactions and ensures that any sort of custom signing we have some proof of work to stop spam attacks and stuff and that handles all that it's optional here because for the null chain what we actually do is skip all of our validation so that it will blindly accept if you send a transaction and say it's from whoever it believes it's from whoever so you don't necessarily need that what it will spin up is if you spin up a console the two interact and lock together so we use a full wallet for that so once we've got our market sim setup how do we actually run our scenarios they're set up as ideally plug and play components the overall thing we set up is we call a scenario that consists of an environment which is your background environment and your Vega setup so that will include things like how many steps you want to run your simulation for how your Vega is set up and what kind of logging you want that kind of background detail we then plug in the various agents some of whom will be setting prices they could be acting randomly and each of them will have something they do you'll see when you build one yourself in a bit something they do at startup often they'll force it themselves in tokens that kind of thing and then they've just got a function and each step call that function and it does whatever it wants to do it pulls in data from the market it trades they do whatever so scenario environment you'll see that when we go in a minute to the agents themselves so what's an agent it's fairly simple ultimately it's an initialised step where they generally force it sometimes they will do things so we'll have an agent that sets up all the assets because you start with a blank blockchain you don't have any assets any markets that kind of thing so there's you may see if you're looking through a market manager who does all this setup for you then they'll have a function that's called every scenario step however many steps you do generally will sort of randomly shuffle them so that you're not always getting the first agent called first in any step and then a finalised method that mostly you can ignore but often you'll want one to sort of settle the market and things so that's how you want to set up your agent there's one more thing I mentioned earlier that sometimes the blockchain does have to bleed back in you'll see scattered throughout occasionally a wait for total catch up function basically what this ensures is that you're getting a good view of the market currently you haven't thrown way too many transactions at it everything's in sync that kind of thing so it just ensures that you're more real world modelling rather than trying to ensure that you can get everything in one nanosecond through so I'm going to now talk through building one of these agents, a really simple one that's just going to do a bit of training I guess and then we'll break again and hopefully you can put together your own a bit I try to find where your okay so for anyone who has it set up we've got this run simple agent Fagusium reinforcement run simple agent I'll put up on the screen after this to pass to these files and then in agents we have our simple agent so what run simple agent does is it sets up a scenario I can discuss the actual arguments with people but which we call curve market maker and that basically that contains to ensure that there's a nice shape market and a few who just randomly buy and sell so they're very dumb theoretically you should be able to change it which is what hopefully we're going to build that runs this simple agent that we've got together here so what I've put together is a few nice template functions without having to worry too much about how many of these works you should be able to cut and paste these into our step function which is down here we've got the simple agent and you can see it's got the structure as described earlier we've got our initialize where it's finding the market finding the asset and minting itself an asset and then a black day submit a market order check our positions it is currently always going to buy so 50-50 chance of making money okay so we're down to the really just the hardcore workshop participants you're the ones who prevailed with us despite python and its foibles and the slow internet so I'm going to crack on if you're coding, keep coding if you have questions there'll be time to ask questions and what we want to get to now is just to show you a little bit more questions that you have there the market maker et cetera that you're interacting with and then move on to the reinforcement learning part so as part of the environment if you just set it up there is nothing it's the empty vega blockchain as it would be when it launches live no assets, no markets everything has to be created through governance and once the asset is created once the market is voted through which the python sort of expedites for you you will still have just an empty market no liquidity on it no activity so effectively the code provides various versions of market makers that are based on some stochastic control problems that are pre-solved, coded up and basically running and then some sort of hacks where the optimal solution of the stochastic control problem is a heuristic and then you put something on top of that and you can have liquidity trader, you can have liquidity takers which just means they place random market buy and sell orders you can have informed traders what does it mean you just tell them what the price will be at the next step maybe they have inside information or whatever and there are the momentum traders through a library you can trade on all the patterns that these chartist people have there are these vomiting camels I don't know what else you can code up agents that trade on that