 The workshop that we're doing is using the graph in Dune to power on chain actions. My name is Christian, this is Matias, and we're from Component. You can find us at component.fi. And we'll quickly go through the agenda so you can get an idea of what we're doing here. So first things first, I just wanted to mention that this will not be a code along. We wanted it to be, but some of the dependencies required. You need pretty good Wi-Fi to download them, so you'd have to run to the basement and come back up. We don't want that, so instead we'll just discuss some of the things that you would have coded. So one, we want to get to know each other. I think that's why a lot of us come here is to find like-minded individuals. Two, we're going to do an overview of the Web3 data stack. So give you an idea of like the different layers of the Web3 data stack and how you can use them. Three, we're going to do essentially an example that goes through that you would have coded along, but instead we'll just kind of show you how we did it and can give you the information so you can do it yourself later. And that'll be data fetching, so the data fetching process, data processing and modeling. So what you do with that data after you fetch it, and then transaction execution. So essentially a way to grab data, process it, model it, and then have that do an on-chain action, a transaction for you. And then at the end we'll do breakout sessions just at your tables or whoever you're standing next to. Instead of the coding element, we'll just essentially design your own automation speaking with your table. We'll walk you through it. We'll do some engineering instead of coding. Cool. So this is actually my email. I don't know if you're familiar with Questrade. It's a Canadian company for trading stocks and using derivatives. I got margin called a lot, as you can see. It wasn't that much money, it's okay. But I'm not a good trader, basically. And that's kind of my motivation for what we're doing because I got margin called so much. I'm a bad trader, maybe automating things and having logic do it for me is probably a better idea. So that's specific to DeFi, but of course there's many other ways that you can take what we're going to show you and use it across anything that's built on the blockchain. Okay, so we want to start with everyone just kind of talking to each other at their tables. You can get to know each other, maybe introduce each other. But here's a prompt based on what I just showed you from myself. And that's a time when things didn't really go your way in Web 3, so that could be you lost money trading derivatives, not in Web 3. You can talk about that too. A time when maybe you suffered a lot of impermanent loss or you can say good things too. Or a time when you should have sold the NFT at the top, that kind of thing. So yeah, take five minutes and chat with the folks around you. So we're going to give a little overview of the Web 3 data stack. There's a lot here and we're not going to have everything. So absolutely if you're a data protocol, please reach out. Maybe we can add it to our slides. So we'll start with this nice little visual from Andrew Hong on Twitter. This was made about eight months ago. And so here's a breakdown of a lot of the different sort of layers of the Web 3 data stack, mostly relating to Ethereum and EVM chains. So obviously you have your sort of base layer where you're actually running clients, interacting with the Ethereum blockchain state directly, reading logs, call data, etc. You have nodes as a service, you don't have to do that yourself, but you're still operating on those sort of raw data streams. And then you have a layer on top of that. You can think of that as the query and data mapping layer. There's a little more structure there. Often these things are indexed. They're potentially, DUNA analytics takes your ABI's and does some contract decoding. So you can actually read the parameters and the function names instead of looking at a bunch of hex codes. There's flip side and the graph and everybody knows those. Then there's sort of that layer on top and this obviously encompasses a lot of things, but they're higher touch, much more customized ABI's. The data might be a lot more contextualized and a lot more useful for specific applications, but potentially a little more obtuse. Some APIs are that data processing is happening in the background and you don't necessarily know. So those are things like Covalent, EtherScan, some manual tagging systems like through Nansen. And then we have a few that we use pretty regularly, so we wanted to make sure they made it on there. And a lot has changed in the last eight months. So people know crypto stacks, Dave and Mihai, they run like crypto fees and L2 fees and all of those different front ends. They've built a little data processing stack to make it easier to build front ends with subgraph data. The DUNE is releasing their API. We mentioned the DUNE, but API changes things pretty significantly. We use the Misari standardized DeFi subgraphs pretty much on a daily basis. They've created an open standard for different kinds of DeFi primitives across chains. If you want a little more control, ETL and Trueblocks are pretty great. They're both open tools allowing you to index from your own node. And even the ETL data is available in a BigQuery database. You don't actually have to do your own indexing if you don't want to. DeFi Lama, they pump out a ton of new APIs and new protocols. Their adapters are a pretty cool system. And then we have another one, spec. They do allow you to sync your own Postgres database with TypeScript Transformers, bringing in transaction data, indexing it, and then bringing it into your own Postgres database to live beside your own data. Cool. So this is the process, as I mentioned, that we're going to be going through. So first thing we're going to do is we're going to do data fetching. And for each of these sections, we're going to talk about the alternatives to what we're doing. So to give you a little bit of flavor, so you're not stuck with one thing. And then we'll run through the example that we actually did. But yeah, just an overview of data fetching, processing, modeling, transaction execution. So our example is using Python to collect data and write a simple prediction model. So yeah, the three steps are pulling ETH price data from the graph in Dune, just to have kind of two data sets to compare and to show you two different ways that you can do it. And then using a autoregressive moving average model to predict price movement. I wouldn't use that approach at home, but for this presentation, it works. Basically, if price go up, swap USDC to ETH on Uniswap, that's the transaction that will automatically happen. Obviously, this was built originally as a code along, so we tried to keep it as simple as possible. A lot of this can be expanded on pretty thoroughly. So obviously we went over that data stack and there's so much there. So many different choices to be made, and like any other choice you're making when building a product, it's mostly about trade-offs, engineering choices. So we wanted to talk about some considerations to keep in mind. There are definitely others, but some big ones are available fields to actually have the information that you need to use. The data format are going to have to wrangle all this data to have it usable in your automations. Scope, does it cover the chains and protocols that you want to use? Are you going to have to merge different data streams in order to get information from cross-chain, cross-protocol? Is it event-driven or is it pulled? Flexibility, can you extend the model? Stability, are they going to change their interface on you while you're running your automations and break everything? Because for some reason we still don't version control APIs everywhere. Latency, if your application requires real time, you can't be operating an hour behind blocks. Trust, reliability and redundancy. Openness of the actual transformation function, so understanding how that data was transformed from raw inputs to the data you're looking at now and then openness of the data itself. If you want to be able to inspect the internal state, can you just get a dump of the database and check it out? Those are all some pretty important things. So in our example, we are just fetching ETH USD price data from both graph and dune. Of course, this could be done through most of the APIs that we mentioned. We like working with both tools and so this was just a nice little example of using those. Yeah, we'll start with dune. So very simple query here. We're actually just using a dune created or user created table, which gives you prices of different ERC20s on Layer 1 Ethereum. And here you can see we're basically just pulling in the time, which we're changing to hour because we want hourly price. And then we're doing the average of the price of ETH over that hour. And then we're doing the max price and the min price during that hour for the last 30 days. And you'll see this query as a query ID here. That's going to be relevant in a second. Cool. So yeah, as we said, we don't have access to the dune API right now. If someone from dune is here, please let us know. We'd love to use it. But so instead we're using this open source SDK that grabs essentially the data from a query. It's limiting in the sense that you can't actually run the query using this. So you basically, you know, every time you want to do this, you'd have to go back to dune run the query and then come back here and it'll update. After you run this function. But yeah, basically it's just grabbing kind of that static data from the query the last time it was run. Then, yeah, if you just scroll up a little bit to data, data 1, 2, 3, yeah, just quickly mention this, it's not that significant, but we're basically just changing the JSON format that we get in the response and we're turning it into a pandas data frame. Are people here familiar with pandas data frame? A lot of nods. Okay, cool. So we're going to go back to the rows and columns. Cool. And then, yeah, I'll pass it over to you. Yeah, there's a little bit of processing here to get, you know, the data wrangled so it's consistent across both dune and the graph. This one is, you know, the graph data pool. So we're specifying the endpoint. We're just using the chain link prices, a sub graph here. In fact, this data isn't actually hourly data. First, you know, we were using graph QL to query the graph. We're just looking for the ETH USD asset pair ordering by time timestamps. So we get the newest data. We're ordering descending and we're grabbing the first thousand prices as well as the timestamps associated with that. Then we do the exact same thing here except we do one thing which is resampling so that we only end up with, you know, one time stamp every hour or one price every hour instead of instead of multiple. And here's just an example of the outputs. You'll notice that there's actually inconsistency between price and price here. That's because the dune table actually does this in decimal format while, you know, the graph here, this sub graph specifically looks like, yeah, eight decimal points. So the fact is we're running this model separately so it doesn't really matter for this example. But it's just one of those things to pay attention to. Obviously your data needs to be consistent. Yeah, and one thing I'll mention about that as well is it's actually great to use more than one data source just so you can actually check these things. You know, it's hard to know what the source of truth is. So, you know, using different data sources, check them against each other, you know, maybe before you run something is always a good idea. And another thing is we didn't really mention this, but with the data we're pulling, you know, this is a simple example, but the really cool thing is that, you know, anything on chain is fair game in terms of data that you can use. So think, you know, LP in lending protocols, you know, on chain derivatives protocols, NFTs, you know, it doesn't have to be defy related. You know, you can do some pretty sophisticated stuff with different types of data. So, yeah, moving into this section now so data processing and modeling, you know, now we're at the point where we have the the dune query in the graph query pulled in using Python, and now we want to model it essentially. So before we jump in back into the code. There's just a few considerations, kind of like just what I said with like there's so much composability when it comes to data. There's also infinite compose composability when it comes to modeling. You know, you could use instead of using a time series prediction you could do conditional logic like, you know, maybe there's someone you follow that trades really well and anytime that they make a trade you want to mimic it right away you know stuff like that. Or if you know someone like me is not a good trader you can take the opposite position. Then of course there's there's so many types of machine learning algorithms time series predictions. The list goes on and on for the types you can do. And again I know we're focused on on defy for that presentation this presentation because that's mostly what we interact with but you know if you're really into stable diffusion, and you want to see text generated images. You could always grab, you know, text from your lens protocol interactions and generate an image based on it and mint an NFT on chain if you wanted to something like that. You know that's really up to you, because all this computation is done off chain the sky is the limit. Cool so quickly just on the, I guess the modeling we didn't we already did the processing I guess earlier when we just reform it out of stuff into into a pandas data frame. So here's it's quite simple. If you're familiar with like a little bit of data science. I'm not a data scientist myself. But what we do is we basically split the data that we pulled in so the dune data and the graph data, and we split it into a train and test set just basically 80% train 20% tests, not that sophisticated. And then what we plot it and we'll show, you know, we'll show the results after and then we run it through a armor model so auto aggressive moving average model to essentially predict. So we're doing this by our, and the goal here is to predict whether the price of ETH in USD is going to go up in the next hour. So the logic here is basically just, if the next hour price is greater than the last price, then we want to buy if it's less than we want to sell or short or what have you. It's that easy folks. Not financial advice. Cool. Yeah, these are the results. So as you can see, the, so the red here is the test set. And we are basically, you know, the prediction is basically saying it's trading sideways. Maybe I think it probably goes down about $10, $10 roughly. If anyone knows the crab strategy on and squeeze this would be a good time to use it I guess. Yeah, these are just the results. Most web three relevant, probably the most complicated step in automation here is transaction execution. So there's a number of ways to do it. So bringing these insights back on chain. So there's, there's tons of considerations to think about which chains are operating on is the environment adversarial. So, you know, if you're working in defy, then you're probably looking out for Mev and sandwich attacks, anybody trying to exploit your model. If you're, you know, minting NFTs for art purposes and not for financial purposes, it's probably less of a big deal that someone front runs you. What kind of timeline are you operating on, you know, some things have much larger delays than others. Key management, do you want to manage your own keys, what kind of permissions do those keys have. What should your logic be transparent. So, you know, if you're running a proprietary training algorithm, you probably want that to be executed off chain and just have the results posted to the chain. But if you're running something that you want to be transparent, you would likely want the, the, you know, computation to be visible on chain. You know what happens if a transaction fails, if it's, or it's congested. So there's a bunch of options that are annoying for manual transactions, but if you're not taking them into account when you're automating transactions, you can get something stuck in the momentum pool forever. You can potentially end up not being able to to transact if you're not, you know, replacing that transaction and replacing that nonce. So there's there's a bunch of options and there are almost certainly others as well. And these are not necessarily mutually exclusive. They're just kind of ideas of part of it. You know, libraries that wrap on top of, you know, the wallet APIs that translate JSON RPC. Ether is Web 3 JS Web 3 Pi, which we're using in this example. If you're running something like that, it's probably in some sort of function that you can either be running, you know, locally or on a cloud. You can, you know, use traditional cloud service providers or, you know, tenderly has their new Web 3 actions, which is effectively lambdas for for Web 3 interactions that can be triggered by events. Keeper networks. So you can some keeper networks support like more complex off chain logic. Some only support triggering, you know, transactions with a function signature. There are transaction execution services. So something like open Zeppelin defender relay and in fewer transaction services. They handle a lot of these questions that are kind of annoying for a small development team to deal with. So things like key management, they allow you to hook into a KMS system. They also handle, you know, gas, figuring out gas fees. So if a transaction fails and the first transaction is is waiting too long in the mem pool because it doesn't have a high enough gas price assigned, then it will actually ramp those up progressively and it tries to figure out, you know, how much gas we need to be able to execute this transaction in a certain amount of time that being the variable you set. You can, you need to make the decision, you know, if you're not actually doing execution, and you're just posting data to the chain, then you might want to execute more like an Oracle model. So just posting data, you're not necessarily thinking about execution, then you're more thinking about the on chain execution that is you're more thinking about how do I get a reliable, like reliable posting of results to the chain. You can go through a multi sig if you don't trust your automation, you could have them as one of the signers of a, you know, one of two multi sig, where they know basically are queuing up that transaction, send yourself a notification and then accept or reject. That might be the most thing that most people are more comfortable with. You could potentially run things, run the actual run some of this logic on an L2. So what's the name of the. The Oracle. The machine learning model and start. Oh yeah, there's a company called modulus labs. They're doing some pretty cool stuff. So they created this thing called rocky bot. That basically does what Matias was saying, where they're running, it's on it's on start net, and they're leveraging that and just basically posting the result to the chain. So it's like a neural network on chain, it's it's it's very cool. Definitely check it out. Yeah, and then I'm not very familiar with MEV or, you know, like, being a searcher so I'm sure there's other people who can expand on this but in that case you're probably using something more dedicated to your building bundles using the builder API believe. And for this example we're just using, you know, web three dot pie. This is probably the thing that people are actually the most familiar with. We're fully we're just, you know, setting some. Some variables were selecting our chain being girly. We're grabbing an RPC provider. There's some middleware that's required for girly here. We're grabbing a private key from environment variables, which of course means we have to manage that ourselves. And then we are. You know executing transaction in this case we are swapping on the swap v to v to router. So we're. Grabbing the API using that to encode our function signature and, you know, setting some parameters and posting that to the chain. Cool. So before we go into this, just wanted to recap so we used sub graph to pull to pull each USD price data we use Dune to pull each USD price data and we processed it. We're actually using Python pandas and then we used web three dot pie to actually send the transaction. It's actually what we're supposed to show just putting it all together. It's again, you know, three steps here really grab graph and doing, you know, formatted data check with the model whether we should buy graph by based on the graph model, whether we should buy based on the Dune model and, you know, if both are true, we are buying. Yeah, and one more thing to say here is. So this makes it look easy. And that's actually what we're working on a component. So working on something called blueprint that abstracts away all the stuff we just had to do. Although it's really fun to learn how it's done and you can definitely do it that way. And, you know, to see how it's done under the hood. But what we do with that product is essentially try and take you to this step without having to do, you know, all the other stuff. Okay, I guess it's me. So you guys saw what we came up with. Now it's your turn. This might be a little long with the 10 minutes, but I'm sure everyone who came here has something in web three that they're passionate about that thing probably is generating quite a bit of data. Probably available through one of the many services we mentioned. So we'd like to take a little bit of time, you know, explore some data sets, think about what kind of automation you could build using those as your input. You can work as individuals or as a group to start. In that match, we're going to sort of have little 30 second pitches to everybody else in your, on your table. And you'll decide on one to sort of design as a group. Yeah, so just to, you know, give you an idea of what you could do. So anything in defy where, you know, you're triggering, you know, based on a prediction model or based on some conditional logic could be to sweep the floor of an NFT collection. The really anything you want and any data that comes to mind that's particular to you and what you're doing. And remind you to kind of be thinking about those three steps where I'm getting the data form how I'm going to process it and then how I'm going to actually generate something actionable from it. Yeah, go ahead. All right, folks. We're happy if a few people came up with an idea and like thought about the process. And I think if you did anyone come up with an idea. Raise your hand if you did you want to you want to talk about it. So we think that kind of defy investing has been quite risky forever. And so the idea is kind of a generalized pressure release valve. So if a condition is met, whatever investment you have, if it crosses over that threshold and pull, pull that, you know, pull out to wallet essentially. So you could think of a liquidity pool. You're invested in there but if the two assets start deviating beyond a certain range, then you exit that position. Thank you, Greg. Anyone else. You look like you have an idea. Okay. Quick picture. Let's hear it. I'll try to make it very quick. Yeah, what I've learned throughout a lot of a very horrible loss experiences that you have to optimize your bias. That's a strategy for centralized finance. I mean centralized trading, but I did. It does work for defy and LP and it's basically spotting when the price hits hits a threshold and it when it's oversold within the volunteer bands like mean, like in the, in the lower part of the Bollinger bands, and you want to do that, you want to identify that by signal throughout different temporalities, temporality sort of like the weekly timeframe, the daily timeframe, and maybe not no shorter time like the timeframes, maybe the three day time frame. I don't know, but identifying by signals throughout different temporalities using the Bollinger bands those that's that's basically it. Oh, it's working. Yeah. You look like you have an idea. No, okay. You have an idea. Okay. So we discussed an idea of auto rebalancing crypto index. So let's say you want to build a simple index that's like 50% wrapped BTC 50% wrapped ETH. And that balance is going to be different 50 50 of the price of Bitcoin increases, for example, so maybe using some some kind of feed like that to calculate the balance of your portfolio and rebalance every hour week, whatever. So yeah, rebalancing I think is a is a very good use case like in, whether it's, you know, just a simple spot trading portfolio or it's you know something a little more complicated like LP or landing or borrowing that kind of stuff. We have time for one more, maybe we have a little time. Okay. Anyone else. So we were like, none of us were actually in device space but we actually thought about some other use cases. So I'm working on a reputation on computation protocol based upon pairwise trust like a whole trust to him by how much, and then I'm looking my output would be basically the number country would be like, after all that recursive calculation like each peer in my network is assigned a trust score right. And he's actually from the space and we talked about how we can actually use this this type of algorithm in like voting versus like a civil prevention. Typically, my algorithm is pretty good at identifying civil civil clusters who actually trust each other but like none of the other real people actually trust any of them. So maybe we can actually ingest this data and then do that number crunching identify those civil culture and then his now could actually admit something onto the contract. So it's like those accounts as potential civil maybe maybe like avoid them. You know, things like that. Cool. Thank you. Maybe one more. If anyone has one. If not, it's okay. Anyone keen to share. You look like you want to share. You're writing your idea right on your phone. Okay, later. Okay. Cool. Yeah, that's it. So we'll just wrap things up here. Yeah, we don't have time for that. I don't think so. Well, we'll just go to the last last slide I guess. Yeah, if you want to run with any of these just remember to think about like we have the basic idea right now, but there's a lot more to it. I think adversarily what where the attack vectors, how could your strategy be be manipulated, you know, be defensive. Yeah, and then just to close things out. To mention what we're doing again. So I component we're building something called blueprint and all of your ideas, except for maybe the last one that sounded really cool but it was a little over my head, but very, very cool. But for the defy related stuff at least and would love to talk to you about what you're doing to see if we can incorporate that in the stack so if you have any ideas that you actually want to execute on or you have, you know, friends that like want to build these kind of data driven we're building tools that basically make the entire process easy. So you only have to worry about the logic of what you're building. So yeah, that wraps it up. Thank you everyone. Thanks a lot. I know it was an unconventional workshop, but I appreciate that everybody was talkative and, you know, engaging instead of just hanging out on their phones and things like that. That's great.