 Welcome, everybody. Let's get started. We've got a full day of program, so I think we should get going. I'm Aaron, and this is Tim, and we're going to give you the first session, a basic introduction to the Swarm project in general, and also more concretely what the team is doing, what the roadmap is. But it is my pleasure to start off by just reminding you what the big vision is for Swarm where it fits into the ecosystem overall. So you may remember that when the Ethereum project started, it was not just about the Ethereum blockchain. We had three components. They're called Ethereum, Swarm and Whisper. Ethereum is the blockchain that we all know. Whisper is a communications protocol allowing two people to anonymously communicate, and Swarm was meant to be the data storage part of this triad. And the Ethereum and Swarm connection was that Swarm would be used, that was part of the original vision, to store in the Ethereum state. The Ethereum state that's growing larger and larger, we know that's a problem, thinking full nodes and so on. Well, the idea was this would be the storage layer for the state in the blockchain. And I'm happy to announce, and as Piper's cheers have indicated, there is progress on that front finally, and Piper will be talking at 11 on the progress on storing Ethereum state on Swarm. That's kind of exciting, so that's all I will say about that right now. The other half of this triad is Swarm and Whisper. Now Whisper was developed, but it's not a very efficient protocol, because it was aimed at secrecy and anonymity. Swarm itself has a layer of transporting data, and as such we figured it can also be used to transport messages. So Swarm has encompassed a messaging protocol, so Swarm and Whisper together, you can think of the original mission for those. Storage and communication are now part of the Swarm project. And so we like to say Swarm is about storage and communication infrastructure for a sovereign digital society. I'm going to talk very briefly about the storage. Yes, it's about data storage and serving data. Communication, because we can talk peer-to-peer across the network. And the sovereignty. So I'd like to point out that this is really about reclaiming something that's lost on the internet. We as citizens of the internet, as a digital society, have lost sovereignty over the digital commons. Everything that's out on the internet is owned and controlled by third parties. And if Ethereum or cryptocurrencies in general are about getting independence from the banking infrastructure, then Swarm in this context will be about getting independence for the internet itself. That adapt deployed on Swarm will be controlled and owned by the people who use it. And it will be sovereign in that sense. It's not controlled by third party, and we're not beholden to Google or CloudFare or any third party to host or control, and we don't rely on them to run our data. So that's why this is really... So we talked a lot about this slogan when we came up with it as a team. So it's important to us this idea of sovereignty, of reclaiming something that has been lost. So that's Swarm in the large vision. But let's get back down to Earth. We need to talk about how we're actually going to go about delivering on this vision. And on that note, I'd like to introduce Tim, who joined our team earlier this year, and he'll take it from here. Thank you. Thank you, Ara. It's funny, I have still trouble to describe my role correctly. So I started to join in March 2019, but the story about me being involved with Swarm started last November at DEF CON 4, having the great opportunity to have a long, long evening discussing a lot of topics with Victor. And it was quite impressive, and for me impressive, conversation. And I got inspired by Swarm. I really got hooked up by the idea that we have the storage layer, which needs to be delivered to allow us to have the sovereignty with our applications and to provide an Internet of the Commons. So very important for Swarm, obviously, and for its success as a platform project, are the strategic partnerships. So I'm really pleased that we made progress on that front. And when I say strategic partnerships, I mean really the organizations which put in a lot of money. And it's not doing it totally just because we have a lot of contributors in the community, which are not now named here, but which have been little contributions, bigger contributions over some period of time and then dropped off again. But those two, the Ethereum Foundation and IOV Labs, are at the moment the biggest supporters of Swarm. The Ethereum Foundation had been for a long time. We had, along the way, again and against all from private donors from the Ethereum community, which allowed us to make progress. Nevertheless, to operate on the scale we need to operate on, compared to Ethereum, is very similar. And this project needs similar resources and scale to operate for full institution. With IOV Labs, we started the track system in Swarm, which means that we have different tracks, delivery tracks, where we focus on different areas of delivery. And IOV Labs is the main sponsor and providing full-time employees and resources for developing, maturing and incentives, which are required to have as a public incentivized network. And I'm excited that we have great progress on this, which will be presented later about the first implementation of Swarm. We have the core track, which is really focused on the co-stability of the Gold Client, which is a sponsored by the Ethereum Foundation. And we have the Swarm Communication Track, where we identified some blockers which we need to resolve to move this forward. But we have multiple strategic partners, which have a big interest in using PSS as a follow-up development, as an improved development, which can be way more than what Whisper could do, which can scale. The goal is to run into a deployment where we have a public testing, which we can try, and we will focus on delivering the Swarm mainnet at the end of 2020-21. Obviously, the world is moving fast, and so we need to move fast as well. Review 2019, what are the things which mainly happened? We changed the way how we do development. We started to improve our user stories and ethics. We started to have different levels of abstractions to allow our partners and our community to better understand what we are doing. We established a IoT Labs partnership. We claimed our own release cycle because Swarm was with the GetTheme, and we had the same release cycle with the GetTheme. Now we have our own, and it gives us great independence to move forward and the speed we need to move forward with. We established a script process in July, which is the Swarm improvement process. It's like the EIP, it's pretty much the EIP process put in for Swarm. We started the Ethereum accreditation process. The goal of this is to make Swarm more competitive, which means that we are not internal as a team, which is just paid by the foundation, but now we are in a current negotiation step-by-step. We deliver milestones and ensure that Swarm is competitive in the ecosystem, competitive in the sense that we have a high execution speed. One of the most important changes very recently is the change in the management structure about the lead squad. Management restructuring, let's speak about the lead squad. The lead squad is an acronym, like Swarm is very acronym-close, close to acronyms, you will always find them. And its legal entity is an executive team, as it stands for Swarm Forum Advisory for Decisions. I think this is the part you need to remember, that is, I think, very important. And the mandate is to mature the management structures to ensure that the organization structures are given, so we are speaking about a Swarm Foundation, to secure the partnerships and the funding and to set the environment to deliver Swarm 100-0. Do you like to add something? To deliver Swarm and to have the structure to develop further after. Yeah, absolutely. To guard the protocol and to move further after the mandate launch and to ensure that the ecosystem came from. Just because of lead squad, all those people which are now in the lead squad in managing companies, in creating companies. I'm the youngest of the lead squad. I'm super excited to rely on those many years of experience from Krieger. We're sitting in the audience here, Victor, and Javier from Epic Labs. So I'm really, really happy and thankful to work along those three people. Regarding the roadmap, I mean, I said we have the tracks. This is kind of how we plan it right now, but you will see that we will have more improvements about how we plan our roadmap in the future as well. And please don't spend too much time here. I will just walk through it. I mean, you see the point when we want to deliver. It's like the pin-in, for example, the PSS, Cademia Improvements Group APIs, all the things. We have with the film stand on Swarm at 11 the update about what we have there. I'm quite excited that we will see the progress here. You see it in the communication part, also a creative improvement if you want to have a conversation about it. Speak to Lucy is the right person to speak to. We actually got a push sync with this, which means you can synchronize chunks with over PSS in a network. You see also the slips coming up. There's also the post-slip slips, for example, for insurance spend protection, which was written by Daniel. I'm also excited we have progress there. But I don't want to go too deep in what we planned, but this is what we delivered. So this means with the 0.5 release, we actually have the first time in the Swarm history all the elements we think are needed to start really to pivot with Swarm. Because you can have basic guarantees which means you can host your content on your local node and ensure it's not dropping off, which is necessary to ensure that you can have your static website or whatever you want to ensure that it's not dropping off the network. We have a lot of experimental features which will be matured in the 0.5 release line. And that, namely, is like pushing the ability to distribute chunks via PSS is the progress bar, which means that you can know when you actually successfully uploaded a file. If you want to speak about it, Safur is around, and Safur, do you like to show your hands? Thank you. We have the Swarm, which is the first step in incentives. There are many people to speak about. It's like Victor, you can speak with Fabio, you can speak with the IOV Labs team, like Winka and Wojtek. Do you like to raise your arms? I'm super excited to hear the presentation later by Piper and Stephanie. Forecast 2020, this is a little bit vague at this moment, but you will see that being more concrete over the next month as well. Strategic partnerships, strengthening, so DEF CON is all about increasing our collaboration and drawing new ones. Obviously ensuring that we mature the management structure to ensure that we can execute with all those partners together. And focus on stability and developer experience, so we get more adoption. 2020, the Swarm's back for having the party client implementation, the public incentives, testing and the creation of the Swarm Foundation. I think that's all what I have to say at this given point. Thanks for your attention. I will hand over back to Aaron. Can you make pictures? Yes, just to let you know. In the center, oh, you will. Thank you. Let me drop this slide now. If you want to talk to us if you get in touch with us, there's a few canonical points to contact the team. We're on Twitter at ETH Swarm and we have the ETH Swarm Reddit, which we're hoping to use more and more as the community grows. For people who want to get into the development in our day-to-day chat, we have the most instance on behive.ethswarm.org and as for the domains, well, we have a new domain and I'm kind of happy about that because I wasn't sure we'd get it when the ENS launched short names, but our new homepage is swarm.eth, so if it was previously the Swarm that was to get over the seven character limit. This is now our new homepage. Anyone can access that as long as they have Swarm. They don't yet have Swarm and you need to use the old boring internet and you only have HTTP or HTTPS. Well, Swarm.eth isn't going to work for you. You're going to have to add erium.org to see our website, but that will also work. That is the same content. So, at DEF CONVAC today, what do we have? Let me mention, we've got these talks in the morning. That's where we are now. We have three workshops in a row. They're about 70 minutes each, an hour and 10 minutes, and then breaks in between to get set up. The first one is about getting started as if you want to develop your apps on Swarm. It's very basic how to do uploads and downloads and manage content. Just the fundamentals. The next one, well, we have more advanced functions, like, I guess, PSS messages and, you know, Swarm feeds. All the features you might use if you want to really tweak your Swarm app. And this third workshop is about incentives, which is the one where we actually try within this room to run a fully incentivized Swarm network, that is, where all the nodes start paying each other micropayments and sending checks and doing all that stuff. That's exciting. We tried it with a few people the other day, and we checked and it sort of worked. It did, I mean, without a few nodes, but we're trying something bigger here, so we're trying to get our whole room wide. So it's a big step, it's a historic step. If you want to be part of that, that's at 2.40. And then at 4 in the afternoon, we have a few more talks about entanglement codes. It's actually a really interesting way of getting redundancy in the data and Swarm. And they've done some great work about mapping these sort of redundancy codes directly onto the Swarm data structures. And Daniel will talk about persistency, how to make sure that data persistence isn't lost. And then at 4.40 we'll have an AMA team ask me anything, so everyone from the team, because here at the time, will answer your questions in the room or on Reddit at East Swarm. So we really do want feedback from the community, and that's the session. We'll have a whole big discussion with everyone. And now on the schedule it says next is Victor, but he's asked me to switch things around. So we'll actually move Anton and Raphael, I think, to come up next. So they'll talk about large-scale Swarm deployments and then Victor will come after that. Thanks. Anton and Raphael, please. So good morning everyone. Yeah, so I'm Raphael, this is Anton. We work for the Swarm team. We are kind of mainly responsible like for DevOps related tasks and also like testing Swarm. And testing Swarm is not that easy, like even like a big part of it is like observing how Swarm is behaving. And this presentation will explain some techniques that we apply during our testing process. So I'm going to start a little bit with observability and basically cover the tools and techniques that we use to develop Swarm. So most of the observability tools like blogs, metrics, tracing them parts of the Goetheorem codebase for a long time and we've had them over the past few years. So what we've done over the last year is basically include them in our automation tools so that they're available to everyone who develops on Swarm and they can get meaningful signal out of them. So yeah, the first part is to set the logs. Swarm pushes logs to the standard output. You can set different verbosity and also include the call site location of all the log lines. It's quite useful to bump the verbosity up from the default value of info so that you get also like debug and trace logs and understand what your Swarm node is doing when you're running Swarm. Metrics are basically pushed into time series database where we aggregate them from in the distributed environment and we can actually visualize what a Swarm network is doing not just one node. So basically we aggregate logs within our deployment suite elastic search and visualize them with Kipana. We'll be showing this later in the presentation so that you see how we do this on a day to day work. And Metrics, we aggregate them within InfluxDB and we have dashboards in Grafana where we visualize basically different counters and timers so that we have an idea of how performance form is and what part of the protocols to optimize and whether things are just, we are working correctly. It basically gives us an overview of what our nodes are doing when we run different test scenarios on them. We also have tracing and basically the flags that you need for that are the tracing flexes Swarm. We're using the open tracing framework. Most of our internal APIs are already instrumented so you have the possibility of adding traces and log lines within any part of the code base and then have them visualized in this central system for which we use Jagger from Uber and that also gives us another system to basically have visibility of our networks, private deployments and yeah, testing in general. This is what Jagger looks like. We just visualized an HTTP request which goes through the internal APIs of Swarm and also goes to multiple nodes. So from node 16 we communicate to node 3 and node 29 and basically the timings of those requests. So next time I'm going to give back the talk to Raphael to tell us a bit more on what we basically improved on the last one year. So deploying Swarm, like in the past, so what we did is many times we deployed Swarm on our machine. We reduced for example containers to deploy like multiple nodes on a single machine and then of course you have the bottleneck of your machine. You don't have enough memory, you don't have CPUs, at one point in time it gets slow, you are just able to provide a cluster of, I don't know, 10 nodes and that's it. Also like on our public gateways in the past we've used like some virtual machines on like public clouds, AWS and we used Ansible to configure these machines. This didn't like really scale very well. There were like many manual steps that we had to do like getting VMs and then running the Ansible scripts manually and so on. So we have improved this I mean here are the problems mentioned that we had and basically what we did was we started using Kubernetes. Kubernetes, for those that don't know it's just like a container orchestration tool. So you just have like a bunch of servers and you just orchestrate like applications, you say like run 100 nodes of these and that's mainly it. So it makes like this whole automation process easy for us. And in Kubernetes you have a thing they call it Helm. Helm you can see it as a package manager in Kubernetes. So imagine if there's like a package swarm which we actually did develop this. You can like install swarm in Kubernetes and just configure it as you want. And so this swarm Helm chart that's what it's called. It provides a lot of configuration and it has like everything integrated that we've mentioned before. This means like the metrics stack like influxdb and perfuner, the tracing stack with Jager and also like the logging stack like it's all in one chart. So it's just really the one single application that you deploy and configure and you will get like everything out of the box on your cluster. Additionally we also have some smoke tests. Smoke tests you can see this as some like essential basic tests that in the case of swarm they just periodically upload some files and retrieve them and we even got like metrics from these so that we can see over time how the cluster is behaving if we keep consistently like uploading files to the cluster. Yeah, as I've mentioned like this is what our swarm Helm chart looks like. So we do actually have swarm itself as an application there. And then we have like all these dependencies that Anastas have to have everything working like from scratch. The metrics stack which is influxdb and perfuner like to visualize and Jager like for the tracing stuff and actually also now since we have swap and like incentives it's not here but there's another component that we've added like for testing purposes which is there should be also what is it called? a ganache. So we use ganache like to simulate the blockchain and connect our swarm nodes to it. Now I'm handing back to Anton he's gonna show you actually a live cluster that we have deployed and he will go through like things we've mentioned. About the Helm chart that Rafael mentioned all this is open source so anyone is free to check out the repos and yeah basically what I'm gonna show now everyone is basically free to run it on your own clusters or even on your local machine to have a look. So let me see I guess I need to make this bigger so that the internet works. So basically I'm just showing the parts of the deployment that we started yesterday and we see that it's like a 50 node swarm deployment and we've been running some periodic small tests on this deployment and basically in order to get this what you need to do is basically just provide one YAML file which basically identifies that you want to make a swarm deployment based on the swarm chart which is open source the number of replicas that you want and then base to this deploy swarm for you. It also comes bundled with the debugging tools so I'm gonna show you those the first one is Grafana so we just pour forward to the Grafana instance on the cluster takes a while and then we can open it up in the browser so we basically see automatically we get all the dashboards that we are using in the swarm team you don't need to import anything or set of data sources they all just come here and this is the default swarm dashboard which gives us an overview of the warning stairs debug lines, the up times of the nodes, the HTTP requests etc we can check also some other dashboards for example the smoke tests so the smoke tests over the past let's say 12 hours we can see that we are periodically uploading and syncing with also the jumps that have been synced across deployment runs and the other useful information for example the stream dashboard we have information on the syncing protocol the main pool syncing protocol swarm the number of peers per node the received chunks the offered chunks the wanted chunks and basically a very nice overview across the whole deployment of the various protocols so that's pretty much what Grafana gives you it basically visualizes all your metrics and the nice thing is that it's available immediately when you run your home chart and it comes bundled with all the dashboards that we are developing on a daily basis so the next one I'm going to show you is Jager so basically this is the distributed tracing so we're going to port forward to the Jager service and for that I also want to see the last smoke test that run so that we can visualize something that just run and we see here that we uploaded successfully to node 0 and then we fetched successfully from node 4 so I'm going to just check this request from node 4 and this broke because I just killed the port forwarding we go to Jager we look into basically the request that we are kind of interested at let's pray for the Democrats all yes, thank you and we can see that the request is like from 9.50, 9 minutes ago and we can basically see all the buffered reads on this H2P get request the reason it's sequential it's because we support range queries and you can actually just ask for a chunk of that content but we are uploading and downloading the whole thing so we see all of them and within those individual read buffers we can see the remote requests and also the local store requests so the local ones are those chunks that we have on the local nodes the remote fetch ones I'm not sure if you'll see that are basically those that go to remote nodes so in this case we see that the node 4 ask node for which chunk and we can see for which chunk it asks when that happens how long it took etc so as an overview we check what happens within a deployment and we measure 95% thousand, 99% thousand across the whole deployment in Grafana with the metrics and within the distributed tracing we have a more in-depth look over specific requests so much about the distributed tracing the next thing I want to show you is the login setup which basically we use Kipana and Elasticsearch to aggregate the logs and again I'm going to port forward to the Kipana service and there we can filter for anything that basically is mentioned within our logs on any of the logs so this setup is aggregating logs from across our whole cluster which means that all deployments are sending logs so I'm just going to look at all the logs of the last 15 minutes within the swarm private 0 node and we see all the logs here why this is useful if we search for example for HTTP request we can see for example this request and search for one individual request based on its request identifier and we can have again all the information about the specific request to a specific node that we have identified as interesting so we have the cost stack location and the request so that's quite useful when you're developing the protocols if you want to debug what's pretty much going on within your deployment and it comes out of the box when you deploy a ham chart so with one command you get those three systems and it helps to build on Thomas own pretty much it you can also show the ingress ingress you want to show it what's the command QCTL get ingress so also with another thing we have so swarm has the HTTP interface and we automatically have like a reverse proxy that generates like these host themes and they are publicly accessible so we can target any of these nodes like through our browser easily and yeah it's just targeting one of our nodes or we can just load balance through them between them and everything out of the box now you can play we mentioned like large scale deployments we just showed like a 50 node cluster but this also works for more nodes we we demoed I think it was 1126 nodes so that's it there's also a list of useful links we put here so there are our ham charts if you know about Kubernetes you want to try this out just have a look at the ham charts and yeah that's all thank you very much this is deployed on Kubernetes on Amazon so we're using Amazon for our development but it can very well be on Azure or on any other cloud provider Kubernetes you can deploy it on your local machine as well or on your own hardware it's quite transferable across any type of hardware yeah thank you for that this is actually a really big deal because when when we are running a single swarm node and things don't go right it's sometimes really hard to know what's going on in there when you have like log dumps and you can try to figure out what's going on but what these two have done is not just made these logs really nice graphs you can see things going on what they were just showing with that distributed tracing is you get to see what's happening in the entire cluster somebody makes a retrieval request and that gets passed from one node to the next to the next and you can see the logs go all the way through this so that's a really important thing because when we're developing something like swarm peer to peer dynamic network you're not only debugging what a single node does you have to get the emergent behavior of the entire network correct and that's hard because you get really sometimes very subtle things changing, having big influences so that's why this was really great so if you're into DevOps look at what they've done, look at the repo you get a lot of really cool tools so yeah we rescheduled a bit, that was Anton Raphael Victor's talk in 1040, the next talk coming up is from Voitec and Rinka, they're right here about the incentives track good morning everyone my name is Voitec Shimatka and I'm joined with Rinka with Rinka Henriksen here who's who's we're here together with IOV Labs who is partnering with swarm to build this distributed storage incentives so before I dive into it, actually Rinka is the expert here I will just very briefly introduce who we are, so IOV Labs is an organization that is building RSK rootstock and RIFOS so RIFOS is a one-stop framework that is adding the layers that are needed in between blockchain and dev developers that's storage that's level 2 payments, naming system gateways service marketplace and things like that so we partnered in July with swarm to integrate swarm into the RIF storage system and for the last three months we've been working together and by we, I again mean Rinka Fabio who's here Ralph Marcelo and Santiago and Diego Massini who are all members of this joint incentive track that we're doing together and we've been building this swarm incentives and I think we're launching an incentive test now soon so I'm going to give a word to Rinka who's the expert on this one my point of work so hey everybody thank you so much for being here my name is Rinka Hendrikse and as Voetic already said I work as developer on swarm but I'm employed by IOV Labs and I'm also having a background to say ok so a bit of a background because I realize this is like the first maybe some general presentation I should might know but I actually want to question this who here says he's comfortable with swarm like alright little bit less than half so let's just give a brief overview ok so swarm you have a chunk and a chunk is a very tiny piece of data about 4000 bytes and these chunks you can upload them to swarm and based on a certain algorithm they will be spread throughout the network like you really as you upload something you don't have to say where these chunks come they're just going in a network based on the orders, based on the hash of the chunk and the orders of the nodes so they're spread in a network and the idea is that if I as a node or like a user of swarm wants to download those chunks I can retrieve them upon demands again magic happens and these chunks are somewhere assembled and they're being showed in the browser or whatever anyways like this sounds very nice right but any problems with this well there's no incentive really because it sounds nice that all these chunks are being spread throughout the network and people just host it but it's in the real world if you want to build like a really robust and consistent and scalable network you don't have to pay somehow you need to denote in a network that must be compensated for hosting those chunks so this is the incentive strength and that's why incentives are so important so meet Swap Swap is a swarm accounting protocol and it's like the base layer of incentives in a swarm and really the core idea is that instead of paying for everything which happens in the network account of what you're doing with your peers so in a sense nodes keep track of balances with all the peers and a balance can be either positive or negative and now if the balance tilts too much to one side so if we are if the beer which we are connecting to like so if the balance tilts too much to one side pay our peer and if the balance tilts too much to the other side we expect payment from our peer and this is like very high over what a swarm accounting protocol is now just introduce a bit more in detail for this I have some graphical something so here you see two nodes you have seen nodes B and on D and below here we see the trunkometer well the trunkometer is not the official term but we just like to use it but anyways this you can see it as as like the balances so here we have the balance right now the trunkometer is at 0 and you see here in red this is what we actually don't want to be so this means if our peer doesn't pay it's okay if we are here but if your peer doesn't pay we'll reach 120 we disconnect so that's the trunkometer now we're going to use swarm we're going to send chunks back and forth so now we send one chunk and you see also the trunkometer just the balance stills a bit to the other side but now there's like an evil consumption right we're both using swarm and there's not high variance so next time the chunks are being served to the other side and the balance stills a bit to the other side and that's how it happens no payment happens it's all we just keeping track we just accounting basically now there comes a big chunk or he's downloading maybe streaming a movie and then you see that the trunkometer tails to the 100 and then we do a payment also the trunkometer is being set back to 0 and this payment can really be in ether or in RSK or RID it's a peer-to-peer network so in principle it should not matter in what currency you pay you pay you know the only thing that matters is that you can pay your peer yeah okay so how to do those payments that's actually the next part of my talk what I already said peer-to-peer nature in swarm makes it possible that swarm is payment agnostic what I said the only thing that matters is that you can do the payment and how it doesn't really matter but of course we have an initial implementation this is what we all advise knows to use when they boot up swarm and this is the checkbook is basically an on-chain deposit against which checks can be written and checks they are like checks in real life they are redeemed more as time in the future and of course you can save a guest cause if you accumulate checks so you don't have to cash all the checks like you would do with a real life check but this is a commutative amount so you can save guests so this is a checkbook so here we have a check and on the check some information is the contract address you have the beneficiary so to whom are you paying and the commutative amount this is a smart contract and you have the commutative amount and the signature but well we're not cashing this check we want to save a bit of guest cost so we get another check this check lists exactly the same contract address exactly the same beneficiary we get another one the amount is 300 now the peer who is actually the beneficiary of this check thinks okay let's cash them out let's send this piece of information with this signature let's send it to the checkbook so it pays us 300 now we continue it's some accounting protocol and the payments happen now we have a check of 400 and we can decide to cash this out we want to get a bit of more security so the checkbook is a very simple smart contract in a sense but the nice things about it it's basically a layer 2 payment system but the nice thing about it as a user of Swarm you deploy one checkbook contract you fund it with money and there's no limits to how many peers you can pay out from your checkbook smart contract so that's really beneficial for Swarm because once you boot up a Swarm node actually I don't know how many nodes are we expecting as a Swarm node but let's say it's 100 so it would be really not a good idea to open a payment channel with 100 people because maybe you even don't have even so you really want to deploy a checkbook contract one contract and you can serve as many peers as you want there's a nice trade off between security and costs because if you want this to be super secure you cash out if you have a check you cash it out directly so you don't save on guest calls but you cash out the check directly but if you think ok Swarm there's like a repeated dealings with all those peers so you have been dealing already for months with one peer and I think ok I don't want to cash out this check directly but I trust this peer and that's why it's maybe until I have accumulated 5 checks or 500 and then you have a bit less security and then what we also have there's the no-eath entry into Swarm this feature which is on roadmap will be developed very soon but that's basically you as a node you boot up Swarm you don't have any ether like this trade from UX perspective right because most of the real users don't have an ether account but what you could do if you boot up Swarm you can basically say you flag ok I don't have any ether but as we are incurring Swarm and you send checks to me please instead of just paying this directly to me can you deposit a smart contract for me and I will compensate you for doing this that's the principle of no-eath entry into Swarm alright and then of course there's also some hot topics for research like the Swarm incentive protocol or the incentive layer is like we are the first ones to do this I think in the blockchain space so there's like a lot of hot topics for research and for this we invite all of you just to think together with us because they're not really answered like we have ideas but we are implementing them so it's like really hot topic so one if is setting prices so what will be what is like a reasonable price for using Swarm how much does a chunk cost well that's one question the other one is supporting multi-currency Swarm so as I already said in principle you don't matter that if you do a payments to one node and the node wants to pay you in Rift it should be okay right that's the principle but of course there's some side effects and we have routing assumptions and a lot of interesting questions but okay that's not a hot topic the other one is incentivization for long term story what is really about incentivizing content which is hot which is topical which is being downloaded frequently but what if I have like a photo album and I don't often download it but maybe in 10 years I want to download it right so it's not really it's long term storage and yeah you can contribute I don't know if you can read it but we have swips repo put this on the ether sphere slash swips and there you will find swips for all those topics alright okay so I'll close this presentation with roadmap so in the immediate future the first thing we want to do is really polish the bandwidth incentives so this is the swap implement some sensible pricing oracles and discovery of your prices payments through multiple ways for example through Rift postage stamps which is actually a spam protection and a way to ensure that your content gets pushed to the network then later on the big topic is storage incentives so this is the persistent storage this year we want to tackle a centralized way of doing so pay to pin kind of a little bit similar how IPFS does this and later on in the decentralized fashion so this is the swear and smiddle as described as the sw3 paper incentives gateway is another thing we want to build so providing a simple tool for the third parties to just run their own note and to actually be paid for running no one providing access to swarm and eventually building a storage light notes which will allow us easy browser integration and could be necessary for being mobile friendly and one last thing I'm going to show you a short demo but this is actually a content that we're going to do together at 2 o'clock in the workshop but I'm going to demo a little bit right now so the first thing I do is I'm just running a ganache which is just a little tool chain here then I'll be running one note a boot note of swarm and I'll connect few other notes to it even right now I've created this simple website which should actually show me the content of the note so this is my note that is running I don't yet see the Ethereum address of my note nor the checkbook address but I already see that I have two peers that I connected to I see that I have zero balance between those two and that I haven't sent any checks but let's start a few more notes and see what happens so they get added here now I have some four notes that I connected to I'm going to create that's not the one I wanted to send I'm going to create some random traffic on the network that's hopefully going to so I'm randomly uploading and downloading to the different notes and as I do so the balance is to change maybe we're just very unlucky here and actually nothing hits the button that would explain it so now I'm just uploading between the first note and the first note and I'm uploading 16 kilobyte files so the balance is to change over time and we can already see that as these notes communicate they've already paid the check here and I can even have a look at the check so this is must receive check from a peer one and I got already paid because I directly actually cashed it out some 1.9 times 10 to the 14th so everyone I have decided that this is like I'm talking to you today about essentially knowing what was a promise from the very early days when Ethereum was launching about the idea of let's use Swarm to store the Ethereum state my name is Piper Merriam I work for the Ethereum Foundation I lead a team of Python developers and one of the main things we work on is a client for both the ETH1 and 2.0 network called Trinity since Trinity is written in Python we have the second-fest language for everything and we have had a bunch of challenges because of that we're not as fast as go, we're not as fast as Rust so we've had to come at things from different angles which has been hard at first but now it's actually kind of cool because it's caused us to have to innovate in areas that other clients didn't necessarily have to earlier this year I went to the Swarm Summit with the express intent of getting this effort started and the Swarm team was incredibly receptive it's been a really cool last however many months it's been but first I want to tell you a little bit of base knowledge that may or may not be things that you already know so let's first look at the data structures that the Ethereum blockchain uses I'll get these kind of split into two categories there's the actual chain data itself this is the data that you need to rebuild the entire state of the Ethereum blockchain and the state data this is the data that is used during EVM execution the chain data looks roughly like this there is a chain of 8.6 roughly million blocks there are 500 million transactions and receipts it comes up to about 100 gigabytes total compressed similarly we've got account and contract data 76 million accounts about 10% of these are contracts this comes out to a total of 375 million tree nodes and this actually accounts for about even though it's only 40 gigs of data when you sync an Ethereum this tends to account for more than half of the time that you spend syncing the main reason for this is that one the protocol has some inefficiencies in it we have to actually request these all individually and they're really small pieces of data so these requests are very inefficient we'll talk about this a little bit more all of this data is content-addressed this is the structure of it in general you've got the header chain this traces back 8.6 million headers that all reference their parent each header has these four tri references in them trees, transactions, receipts and uncles and the state route which represents the entire Ethereum itself, the contract storage things like that all in all this comes out to about 1.5 billion things I'm trying to get a sense for what any Ethereum node has to do to give you a picture of sort of like where the challenges are in serving this data over swarm if we take a naive content-addressed approach we will have to request 1.5 billion things by their content address and that's a lot guess syncs in 1 to 2 days which means that in order to do that we would have to do approximately 10 to 20,000 things a second the idea here is that there is a ton of individual pieces of data that we need, we need to get all of it and if we go at it very naively we have to do a lot of stuff per second to actually be able to sync this here's a little bit about tries they look something like this this is how we store the states this is how we store account data this is how we store contract storage and the important part here is that the only important parts really are the state root at the top and these green nodes at the bottom all this stuff in the middle can actually be rebuilt from the nodes on the bottom so we use the top as as like the reference point for what is the state of this but all we really want is the bottom stuff those inefficiencies I was talking about are that in order to get at this bottom stuff currently we have to work our way down through all of this middle junk which adds up pretty heavily to a lot of extra data and a lot of extra processing that has to happen it's also really unfriendly to your hard drive it's a lot of very small reads and writes again, 1.5 billion things that we have to get and one more thing this was talked about a little bit earlier but I think this is one of the important parts here is that swarm nodes like data that is close to them the swarm network has a topology chunks hashed to some area of this network and if you, a swarm node are in a specific area that the chunk hash is to you will prefer to hold on to that one but you will also prefer to hold on to things that are being frequently requested so if something is popular swarm nodes will tend to want to hold on to them cool, so on to the good stuff we are working on building a bridge between the ETH and the swarm network we call this BZZ ETH and our goal is to use swarm as this like alternate data source I have this word alternate highlighted here because it's important to note that it doesn't matter if the data is guaranteed to be in swarm we have a backup source to get this from which is the current Ethereum network the idea is use the swarm network for what it's good for so as long as most of the data is there or the data is there to some reliability threshold, that's great if some of it's missing, not a big deal we can always fall back on the current paradigm we came up with a plan earlier this year it looks roughly like this we tried to come up with something that was like minimal and sort of delivered value as fast as possible so figure out a way for them to talk to each other we did that over the devP2P protocol we have a new BZZ ETH protocol that lets swarm nodes and Ethereum nodes communicate with each other the two things that we knew that we could do right away that are essentially one-to-one parity with what the Ethereum network does which is serving block headers by their hash and serving tree data by its hash and there's some unsolved things that we need to figure out which are more efficient ways to communicate transaction receipts and uncles just entire tries of data I was really excited to see this in the release last week the 0.5 release includes this, I'm a little embarrassed that we don't have a Trinity release that actually supports this but this should be very soon around the corner so I spent a lot of time trying to figure out how to explain the concepts in this and this is one way that I came about it which I wanted to highlight that at first glance fetching this data over swarm is worse than fetching it over the ETH network and that isn't a hit on swarm swarm does what it does very well the reason that the ETH network can do better is that it has contextual awareness of chain data so you can ask an ETH node give me the block at this number and ETH nodes know that but for a swarm node to know that for a swarm to have a concept of canonical block numbers a swarm is all about content addressed by its hash so we have to come at this from sort of a different angle than we do in the ETH context similarly you can ask an ETH node for a set of transactions for a given block you can prove that data once you get it and the reason that an ETH node can serve it to you again is that it has context it keeps this extra metadata information about the chain that is sinking but it doesn't make sense for swarm to know that information swarm should be sort of serving data by a content hash so I really like this graphic this is what it's like to come online as a new ETH node full nodes have a maximum capacity that they can serve a total number of teats I guess and most of the time all of those slots are full and as you know there's that kid over here on the right who doesn't can't get any data and this is sort of the general state of the network you stumble around you try to find some ETH nodes that aren't completely filled up and maybe you get to sink from them the thing is that not all ETH nodes are alike Carol comes online she wants to sink maybe she connects to Alice maybe she connects to Bob however if Bob is partially synced then you can be able to respond to about 50% of Carol's data requests Alice fully synced likely responds to about 100% but the idea here is that not all ETH nodes can be like swapped for one another maybe Bob is broken and Bob is like stuck at some weird like fork and never got off of it and Bob's just holding like a pierce slot you keep asking him for information he maybe serves 50% of it but there's a bunch of it that he's not going to be able to give you but we're interested in here is that if Alice is a fully synced node and Alice is piping her data about the Ethereum blockchain into the BCZ network then Carol can get that data from Alice like normal but Carol can also request that data from any BCZ node and it doesn't matter if the one she's requesting from is the one that Alice is piping the data into and the other node will then be fetched the data from inside of the swarm network and get it back what this allows for is Alice just a single fully synced pier piping her data into the swarm network to serve thousands of pierce or previously she might only be able to serve 20, 30, 50 I don't know but this action is automatic kind of horizontal scaling that are amazing in that popular content will naturally spread itself around will naturally be available and so the idea here is that swarm doesn't do what it does best while it may not have this contextual awareness of the blockchain of the state of the chain of this kind of like deep information it can do this one thing it can do a bunch of things really well but it does this thing really well in this content address and the powerful part here is that we don't have sort of like a chicken and egg situation where we need a bunch of nodes piping data in to get a bunch of data out one single eith pier piping their data into the swarm network can serve thousands of other pierce as long as there is adequate capacity in the swarm network to handle that so this allows the network or the concept of an eith pier that comes online to sync it allows the network to handle a much higher capacity we could have thousands and thousands of nodes come online all at the same time and under the current network they would struggle to sync they would connect to eith pierce and be partially synced and then other ones that are trying to sync would connect to them and they would only be able to serve small percentages of the data but if they're connecting into the swarm network like we're tinkering around with all it takes is one fleecing pier to serve all of these new pierce coming online this is where we've gotten to so far we have a communication channel that eith pierce and swarm eith nodes and swarm nodes can communicate over we have two way transmission of block headers so a system where a swarm node that wants to can get notified of essentially like the tip of any ethereum blockchain and then trace their way back to origin fetching all of the headers as well as ways for an ethereum node to fetch arbitrary headers from them up next we're looking at two way transmission of tree data again this is kind of the inefficient version of tree data retrieve or retrieve it node by node again these are like easy things that are just straightforward content addressing and then we have experiments that we need to figure out and solve problems for how to efficiently retrieve transactions and receipts and so that's what comes next we're looking at how we do block data transactions and uncles and receipts the header again remember it stores a reference to a try and that try is essentially all of the transactions and given a set of transactions and the statement for that try you can verify that they are the same thing but in order to actually request this data naively it's way less efficient an eth node you say give me the transactions for this block and it just hands them to you and then it's kind of up to you to verify that they are what you wanted but if we request these the other contents address over a BZZ node then we're looking at kind of OEM complexity instead of O1 because we have to walk down the tree fetch every single one of those nodes and then rebuild that list of transactions from that data so there's an inefficiency here that we have to work out and this comes back to that 1.5 billion things a geth node doesn't actually process 1.5 billion things it uses these efficiencies of being able to collapse what would be 300 requests down to a single request in order to not have to process 10 to 20,000 things per second it's actually needs are much lower because it uses sort of these shortcuts and things and we have to figure out how to do that well within the context of Eve Network talking to Swarm Network so one of the first things here is ways to have more efficiently retrieved tree data remember I told you earlier that the way that we currently do it is get the state root, get this thing, get this thing, get these things and then finally we get to the data that we need one of the things that's actively being worked on are protocols and ways for us to fetch these things in bulk and only fetch the data that we need we've called these prefix queries is one way to think about it or range queries which are give me all of the nodes between these two values here or another way to think of it is just like give me the thing at the bottom I know what I want I don't want to trace my way down the tree just give me the give me the leaf node at this key and so one of the ways that we're looking at doing this is giving Swarm some like native understanding of the tree structure this isn't native context of the blockchain but this is more of a way to have it natively understand this tree structure and thus be able to return information in a more rich way another thing that we're looking at is just screaming retrieval of all of the leaf nodes for a tree so rather than like give me part of it it's hey just give me the whole thing I want all of it or give me all of the pieces from it that I don't have yet but it's ways to efficiently retrieve all of the data out of the tree without having to request each of the nodes individually another thing that we'd like to do is leverage this feature of Swarm called feeds show of hands Swarm feeds cool I am potentially going to bastardize this a little bit but the general concept of Swarm feeds is that it is a time series database for a key value store so you had a key topic you've got a public key whoever it is that is publishing to this feed and you've got a value that's updating over time and the idea is that you can retrieve this value give me the current value because you can also work your way back through history and say give me what it was before give me the book before that or give me it back at this point in history so the idea is that a ETH node could feed this sort of subjective time series data into a Swarm feed which then lets nodes retrieve data that isn't exactly content addressed but they can consume this feed and say oh okay I already synced from this node earlier I know that they gave me good data I see that they have a feed of all of their blocks that they've received I don't even need to connect to Alice anymore I can just consume her feed out of the Swarm network now in order to sync the blockchain so this lets a full node a good like reliable full node not even need to connect to a peer that comes online and still let that peer hold data that they're essentially feeding into these feeds we've been working on this thing called beam sync in trinity it's sort of like a fundamentally different way to sync the chain remember where I told you we had to innovate to figure out a way to do things because well Python is slow and the general concept Jason may hate me for some of this if I'm bastardizing it but the general concept is that geth has this thing called fast sync or like fast sync is sort of a current way that you sync an ethereum node remember there's 375 million tree nodes and the idea is in fast sync you download all of this state and once you're done with all of it then you start executing blocks on top of it this takes a long time this is a lot of data to pull the important thing sync is that a block may only touch a handful of nodes in the tree and the experiments show that it's somewhere between one and three thousand tree nodes they get touched in a given block and the idea is that we download this data on demand so we spin up the node, we sync the header chain and once we know what the tip looks like and we know what the state route is we get the next block and we just start executing it we run into a place in our state database where we don't have the data that we need we fetch it on demand over the network once we've built that all of that up we've executed the block just like we're a full node but we only had to download something like a hundred thousand times less data so what we've validated here is that we can have a syncing strategy that takes a completely empty database and takes you to essentially full node status in about one to two maybe five minutes so that takes the sync time from two days down to five minutes which is huge there's a lot of asterisks in there what I mean when I say full node and there's this entire backfill thing that has to happen but the general concept is that that we can go from zero to sixty or whatever you want to call it way faster than you can in the fasting world what we're looking at is ways to make this even faster block witnesses are this concept of what are all of the things that we need to execute a block and block in, we touch all these red ones I'm blocking this one, we touch all these red ones and essentially if you just have this data and a little bit of extra metadata so that you can prove that it's the right data then you can execute that block if you package all that up, we call that a witness in current beam sync we actually have to spend some time working our way down through this to fetch all this, we make like something like actually three thousand requests the idea behind witnesses is that it turns that into one somebody else is packaging up a witness for a block for us and maybe putting it in a swarm feed then we can consume their stream of witnesses come online immediately start pulling witnesses from them immediately and start executing blocks immediately only having to download like one couple things per block so stateless clients are another term for this we have not been calling this like stateless because the idea is that we do want to sync to a full note to throw this data away at the end we want to bring more full nodes onto the network after which we can do some of the more fun things like stateless nodes and things like that so the idea here is yes, a stream of witnesses we can use swarm feeds a reliable node that you synced from in the past you can jump online you don't even need to connect to them you connect to MPCZ node and you can pull this data out another thing that doesn't exist very much is archive nodes archive nodes are heavy you need terabytes a terabyte plus a bit of hard drive space in order to sync an archive node an archive node is an ethereum node that doesn't throw away old state it means that you can go back to block 2 or 10 or a million and execute things at that block and it holds onto the state there the standard mode of nodes is to throw away a state that's older than maybe 200 blocks or so and an archive node has the entire historical state that you could possibly have of an ethereum node what we want to do is go sync an archive node as we do it pipe every single one of those tree nodes into the swarm network then we have all of that data there we know that some of it over time is going to disappear off the network because people aren't asking for it but the nice part is we can just work our way back until we find a block that does have a consistent full state and then re-compute everything forward from there to refill that history so using swarm as sort of like the global archive node makes a lot of sense because the data access patterns for all of this old information you don't need to get at it super fast it's not something that people use regularly but it is valuable to be there so that you can occasionally when you have special needs roll back time and see what the hell was going on a block 2 million so the other thing that we're looking at is how we can store some of this extra metadata so when you sync an ethereum node remember earlier we have about 100 gigs of chain data about 40 gigs of state data but if you look at the footprint on your hard drive it will be about 200 gigs there's a bunch of extra metadata remember there's 500 million transactions and your node stores these mappings of this sort of concepts data transaction hash to block hash things like that this alone is like 20 gigs of just index data 20 gigs of metadata and every single node computes this every single node stores this on their machine and it would be ideal if there was a nice way for us to fetch this data over the swarm network it is one of the problems that we're still working on figuring out how to solve a pretty standard thing is to ask your ethereum node give me this transaction give me the transaction for this hash until you've downloaded the entire chain state as an ethereum node you don't know this and there is nothing in the ethereum protocol to retrieve a transaction by its hash which is why nodes have to store this metadata but it makes sense for us to try to figure out ways to store this in the swarm network itself it gives us a way to sort of like share these indexes or contribute to them as a group another thing that ethereum nodes have started doing recently is sort of considering certain chain data as ancient blocks like one through a million are like not going to change really blocks like anywhere head minus a couple hundred are very unlikely to change and we can think of these sort of as just like a pand only files you can package them up into chunks store them as some sort of ancient chain data and then again if you have a node that you've synced from in the past you know they were reliable which really only means that it's low likelihood that they're going to be briefing you if they actually publish hashes for their ancient chain data we can download these things in bulk all of this data is still verifiable you're not trusting them in any way the worst thing that they can do is give you garbage data that doesn't validate locally and then you throw it away and you go somewhere else to get it but we can speed up a ton of stuff why just download it how long does it take to download a hundred gigs of data on a decent internet connection it doesn't take a whole day so if we package this stuff up into chunks of bulk data we can download the entire header chain and the entire ancient chain data in a couple of minutes rather than what currently takes hours or days another thing is actually using swarm for scalable like line infrastructure in the current model there's no BZZ in here and all LES peers rely on full nodes to get their data if we have all of the state data in the swarm network and if we have all of this information in the swarm network then now all of a sudden LES of like client nodes would be able to get that data from other ETH peers from BZZ peers and potentially from each other and a lot of this is just ongoing research advancing the like client protocol for Ethereum clients but the majority of data should be retrievable over the BZZ network people piping it's been really cool working with this swarm team on this I've wanted to collaborate with them for a while and they were so receptive to doing this and it's really exciting to see these networks bridged into each other something that we'll be actively working on over the coming months and excited to have other people join if you want to awesome so you can already promote this running here but we don't really know what's going on we see some blog notes there see that we already know about 15 nodes that means that like 5 of you were able to successfully connect because we are running 10 if you have WebSocket and JQ installed what you could do is you can query your WebSocket interface on local cost the default port is 8546 and we're just sending a WebSocket request for our JSON RPC with the BZZ Hive BZZ is basically the API enabled and Hive is the function call and then we just parse this result grab the result that we have printed on the screen and this gives us the the table of our local node and I can see here that my node knows about 20 nodes within the network and I have 6 spheres for some reason we're not able to connect to basically we are connected to the nodes on the right side and the right side is the nodes that we are aware of we still can't connect to this node who seems to be close to our node if you don't have those tools installed I think we have this WebSocket tool that can be a product Does anyone have a local copy of those two optional utilities? No, we don't unfortunately but we will actually what we are going to do now is we will upload a website to Sforum and we will share the hash of the website and you can use that to actually get this information that we are seeing here on the website so you don't need these tools just give us some time so as I mentioned we are using this experimental pushing software right now which by itself doesn't give you much guarantees on the content I mean if you start shutting down their nodes content won't be available so if that doesn't work we might need to switch back to the default state of Sforum and we are using the default state of Sforum the first issue that we see right now is that there are nodes that we are aware of who we are not connected to for example this one 5C E0 and then three other nodes here and it's not immediately clear what the problem is with connecting to those nodes I think that Sforum hasn't detected the interface and it's announcing a different IP so if you have like a few network adapters on your machine like the Wi-Fi the Ethernet etc Sforum is probably not detecting your IP for the Wi-Fi and the announced addresses don't seem to be correct Sforum works with Wi-Fi sorry it's supposed to have internet I'm on the same network sorry so yeah maybe we don't have internet so currently on our network that you guys are using you are connecting but the problem is happening basically your nodes are announcing their IP address as being like 127.001 which is local host and of course other nodes can't use the IP to connect a trick we can do to fix this so when you start your Sforum node there is a flag minus minus not I f not if and here space and then you put your network interface work interface so this would be your Wi-Fi I don't know WL feeds to something or if you don't have internet but it was like the ETH0 something like that so if you know how to figure out what the name is of your network interface on your Sforum node rerun it and add this not if egg and the name of your network interface the Wi-Fi this should then fix some connectivity issues that they are having oh yeah right on my node I don't see them because I've connected to the boot node and since everyone else within the room is connected to the boot node we can see this behavior on the boot node so that's one of the things if you look out with Sforum if many of the nodes within the network are behind the map it's a problem so they have to become peers the idea is that those nodes within the network should have connections between them and if both nodes are not dialogueable you can't you need to come up with a solution to relay information between them and we don't have that functionality so that's part of the next steps we need one participant that was able to start the Sforum node with them not if like we just want to confirm that can people raise their hands if they already restarted their node with a not if like and they were successfully able to configure their network interface to just edit the interface so we have four people and I think we should re-do this then so many more people successfully connected the Sforum node how can we be sure that people will come if you run your Sforum node just to repeat this is where you get your binaries so here is where you download the binaries to run your Sforum nodes after doing that you need to run your Sforum node with a not if flag to ensure that your Sforum node is discoverable in the local network what you do is minus minus not if you use your network interface your currently using which is your wifi password so to know if you did it okay instead of calling the VTDI function that you call previously we can call admin underscore node info because I don't use it and there it will say your enode address if your enode contains the of your interface then you are good so if it doesn't contain you are good so if you can do the RPC call admin underscore node info and see an enode that has a sexual value if you are not able to do this it's also not too bad because some of you are able and the network will kind of be available it just doesn't work if all of you are not able to do this okay so here we have the processing and here we have so if you have a macOS you can run the network you set up this all the ports to show the interface you have and then you name take the name from your Wi-Fi repeat if you have a macOS you run the command in the shell and you set up minus dash this port of your ports so and then and then you can run the command to get the network interfaces for those of you that were already able to start it with the NAT interface we updated the instructions to include here a link to a website that we just uploaded to this swarm cluster we have here so if you can try like to open it you should see this little thing with just the connect button if someone is able to see it let's that's just a small html application where that connects to the web suffix of the local node and displays some of the information that alternatively you would need a web suffix and some JSON parser to view so you can just see it we basically see the VZ address the network ID our peers out of my game was someone able to open this website it's on the workshop instructions you have to refresh it and on this step like the optional one piece in your demo table if you go to like the last sentence there's like a link working working working just to clarify if you have problems with it use also the IP address of your local interface that also works you can hard code the IP address that you want to announce your network and I'm just going to add it like here so instead of it not if you have external no it was not it's not right yeah it's not and that would basically set your to announce the IP address that it's within the network so just to summarize you have two ways to ensure that your Swamdorf is discovered in the network one way is to set it to not if command your local interface which is connected to the Wi-Fi and depending on the operations you can use different commands to list those interfaces for like if something is about that install hardware and for Windows it's IP config but you could also as an alternative part use the minus minus not equals IP double point and then the IP address of your Wi-Fi adapter and both should work but importantly you need to restart your node to setting one of those two flags and defining it correctly I will get around if I can help you FYI I had to remove to delete the data directory when I restarted the node because otherwise it was just like messing up the program okay so you said it's good to remove the data directory before restarting the node because it's not messing it up so maybe you like to do a clean slate before restarting the node can I help somebody with this someone need to know were you able to access the Wi-Fi so the word then okay that's great so I guess we can then move on and if you have any problem just shout we have some time maybe you like to summarize what we want to show what we have shown okay let's summarize what have we done so we started a cluster you guys all connected initially we had some experimental feature running which is like push syncing which allows you to see like this progress bar it worked for some of you unfortunately it didn't work for all that's why it's still experimental and it's not running on our mainnet currently on our testnet then we all switched back to the default mode which is just pulsing and some of you started re-uploading files for example we uploaded this big bunny video we shared the hash like on this workshop like on the very top you guys should be able like to access this unfortunately we didn't like get to the end of it but let's just like go through what would be the next steps so downloading the file you have done that there's like uploading and pinning a file pinning a file just kind of guarantees that your storage what you upload stays like on your local node so that it won't get like garbage collected over time and it's really simple to use when you upload just provide like the minus-minus-pin flag and you can also list the pinned content and unpin so you have kind of all the APIs here and there would also be the option of for example uploading a website it's just a bit it's also the up command just a bit different with some more flags where you provide basically the directory that you want to upload at the end and like the default part would be like your index file and the flag recursive which uploads like every file within that directory I've seen that some of you have experimented already with this BCC raw so what actually happens when you upload a file on Swarm the hash that you are getting back is not actually the hash of the file that you just upload it's the hash of the manifest and the manifest actually contains more information about like what's inside for example here we have a manifest of something it's a file, a single file that was uploaded like an image and the actual hash of that file is here so this is just like for information purposes it's also interesting to call the BCC raw on like a website so that you see the manifest of the website to see how that looks like on file yeah using ENS so Ethereum name services we couldn't go through with it unfortunately we if you checked if you paid attention on the comment line flags there was like an ENS API part which is pointing to the ENS contract and it's we were just using Rigby just pointing to Infura the internet is bad here so we cannot like do this name resolution but basically what this means is that you could like have your ENS domain pointing to like content hash like you could upload your website and instead of like sharing your content hash with people you would actually share like your ENS domain yeah then we would also have like some examples like using encryption how to encrypt something is uploaded how to protect it with then like a password so when you share like the hash once the other like your friends access it he would actually get like a pop-up on like the browser asking for the password so this is kind of all built in and yeah there are then that would be actually the end we didn't get there and I guess we would have need like 15 more minutes yeah and we had some bonus parts with some advanced encryption stuff and I think that was currently we just shared this locally because of the internet but once you have internet you can yeah either sphere slash defcon file I'm gonna write it down actually the account if you have any questions feel free to reach out to us on Gitter or by email or any other channel that you might like repeat one thing once again the push thinking is not enabled on the public test net so if you try to use the form gateways gateway to make sure that you don't provide the sync mode flag then you don't have the progress bar and you don't really have feedback on when your content is synced to the network probably with one of the next battery releases we will make sure that we include it in the default form configuration if you actually connect to the main net like you don't need all these legs like the default flags they were just fine yeah should I repeat there is the presentation so at the moment we support a few different APIs that you can interact with your node HTTP interface there is the web socket interface there is a javascript client that you can use which makes all of these things very simple mainframe the company is working on that it's called Airbus so this is something that developers would use to build a decentralized application that easily uses the features of form so that you can let's say upload or download files to javascript on your web app or even use vits and other features so there is Airbus so the CLI is actually using our API for HTTP I think this that's a javascript that's a javascript provides APIs that are easy to use so use that are there more questions or for these guys any more questions do you have 5 questions 3 to go so if we'd like to compare this one technology or protocol with IPFS is there any source of comparison between them can I say that they almost the same well they both provide content that rest storage IPFS does not provide any incentives so the incentives part is kind of bagged in with swarm and the incentives track will have a workshop later so you're going to hear more about that IPFS is solving that problem with another project and there are quite a few differences in the way the cadamia table works within IPFS and swarm both you and cadamia yes but different cadamia distribution there so basically the DHT we use for different purposes there are quite a few differences so both they could provide the decentralized storage but they solved it in a different way but I'm afraid we already have time to go into details now we can talk later and I can yes so okay another question is about when I establish my own ethereum node several minutes one of them is to be close and do transaction as close as possible number two is to mine and have ethereum having a swarm node what's my benefit is it about only be close to data or close to files or also there is some sorts of benefits the contracts that you mentioned there will be incentives and incentives tracks for that in the next workshop how about swarm explorer what about that to find files yes I think like you remember when you have no Google people created lists of like files and where I lie and so on so in theory you could create indexing services on top of swarm but these are services you will need to be built up on top of that so swarm provides the links to the chunks and to the manifest which is the collection of chunks and potentially to feeds to manage like changing content but in general this need to be built up on top of it so the swarm explorer will need to be an application on top of swarm X this is correct because we can talk afterwards and we can go thank you so much cheers we have a meeting with Joe thank you are we going to continue or do we want like 2 minutes to air out our thoughts continue continue good so this part will be the advanced usage of swarm basically and what I'll talk about that translates to using pss which is the messaging layer on top of swarm and then using swarm feeds which is a way of referencing changing content and swarm using the same identifier using the same hash and it's going to be pretty geeky the whole thing I'm not going to go into like details about architecture and so on and so forth mostly hands on terminal beauty sort of thing and hopefully a bit of interaction as well now I prepared a small repository of my local machine please get clone this it contains all of the script files and other supporting script files that I'll be using in this walkthrough I've also bundled there is for all the scripts most of the scripts there is a javascript version a python version as a build version and I've bundled the node modules directory for the javascript version and hopefully that could alleviate some dependencies although I don't know if there was supposed to be bytecode in there let's see if it works or so please take a moment and clone this you write I guess you all know what to do and if I ask does anybody not know what to do it's not so cool to raise your hand maybe subtly ask your next door neighbor in case you have a problem with this right so let's have a look at the contents of the folder can I remove this by the way because I only have one screen is everybody home somebody didn't phone write it down first of all let's talk about pss so basically pss as I said we have a layer over swarm and using the same when we pass around content and swarm chunks to a certain address on the network we could use the same cadamely logic to root ephemeral pieces of content as well which is messages the addressing scheme we use for the messages is the swarm overlay address the address that the node is running on but every node will have a different piece of identity as well the public key of the node is running so to send messages in pss we can do it in three different ways we can do it in an unencrypted way which just references the overlay address of the node we can do it in asymmetric encryption or public key encryption which means we use the combination of the overlay address of the node and its public key to send the message and lastly we can use symmetric encryption which basically then both of the sites have to know the symmetric key to encrypt and decrypt the contents so what we'll do here is I'll show you first how the communication with the underlying socket the json rpc json rpc content on how to send these different kinds of messages then I will start a monitor on my machine so that you can use the scripts to send contents to it that's basically the idea and that will be the pss part of it and as I said it's going to be pretty geeky so please bear with me so in the repo that you pulled there is this readme .text sorry about that it describes all the files that are in the repo and what they do so let's get started so pardon me for sitting down but it's awkward to stand and do this stuff so I have some notes running on my computer obviously so let's see and right so the first thing we'll do is we'll connect to the unix socket of one of the notes I have running on my machine there are some shell scripts these shell scripts basically just output some json rpc command that you can use to get information from the socket sorry I can paste that in and there I get the contents out right so this is the bcc high one that gives us the high table this is the pss method call for getting the overly address of the note so as you see the response here the overly address of the note have this overly address basically I can send it's all I really need to send a message to the other note so let's start the socket monitor I'm not expecting to do this I'm just showing it right so that's all the monitor of the other note something too long to see this so this exchange here this is the send raw method call this sends an unencrypted message to the note parameters the first parameter is the address of the note the overlay address that we got previously with this base address call the second parameter is a topic so all messages on pss have topics there are 4 bytes long used to disambiguate like is this message do I have the code that I need to handle this message it's the contents themselves hex encoded we can send so let's try to send using the public key encryption first of all I would want to get the public key of the note so there is the method for getting the public key there is the public key as you see then let's send something to this note using the public key sorry I changed this last minute right to send to a note using the public key first of all every pss note keeps an address book of the notes it knows on the network so first of all we have to combine or connect the public key of the note from the sender because the sender doesn't know how these things are connected so that is a separate call it's called pss setpuk here public key it has the parameters public key topic and address we pass that to the note to send it and as we see this is the method for sending a public key encrypted content it has the arguments public key obviously the topic again and the payload on the message now we don't include the address here because that's already registered in the address book right so let's get this rolling so here we see it showed up on the other side scheme as I have to say is the symmetric one so it's a little bit more involved potentially but I'll try to simplify so first of all as with the public key we also have to register the symmetric key to a recipient because you can't infer one from the other so first of all we call symmetric key which I'm sorry about this base 64 but I actually noticed preparing to this there is a bug in the API and DSS which will be fixed very shortly but I didn't dare to do it now where so this first this first parameter is the symmetric key itself it should be in hex format instead then the topic and again the overlay address of the note so basically that's exactly the same as the public key registration right so for convenience right now this is actually the same you can't see it but this is actually the same as the BCC address so we use the BCC or the overly address as the symmetric key just for simplicity sake so let's register there let's then send a message again so this is then the send symmetric version oh sorry so now actually I have to supply also the key itself or wait a minute why did that happen oh yeah I didn't actually pass it through the note right so let's try to do that first oh I did sorry right so yeah we see sync symmetric key it sets the value and you get in return identifier here 32 by identifier this is the identifier that the database no database uses to reference this particular symmetric key address combination so when you want to send the message using that symmetric encryption you also reference that ID you use the ID instead of the key itself in the API call so that's then the send method call it's the symmetric key ID first the topic again and the message payload again so let's try to send this and yeah you see it show up on this on the other side this is the symmetrically encrypted part I realize that she took a little bit now because actually it shouldn't be able to receive it on the other side and the reason why is that it doesn't have the symmetric so both of the sides actually have to symmetric key register in their database but I guess I already did it for this node I don't know but you would have to use the set symmetric key called on both the sender and the recipient in order for the other side to see the message so is that pretty clear so far then yeah cool so this is obviously maybe not your everyday usage of you know how you want to use this so and you might not be running Linux for that matter so instead of using unix sockets we can obviously use web sockets as well I guess you already did some web socket work here but so basically I have the monitor of this node that I'm running running here so theoretically you should be able to send messages to me using the scripts I have provided now let's try a little bit of interaction let's do the javasol actually let's do the python one try that one first so there is a python script called ws.py it takes two arguments the first argument is the message that you want to send to me and the second argument is your web socket web socket address this will in I guess your cases will be localhost colon 8564 in my case it's not but 8564 are you running web sockets or the application was web sockets? 8546 yeah 8546 so as I wrote it not as I said it sorry about that so as you see I can reach it from I have several nodes running on my machine so I can reach I sent one through this so if you try to then and the reason why this works is because in the repo there that we pulled there are two text files there is a bcczadrub.txt text file and the puffkey.txt and they both contains my address and my public key so nobody's sending me a message or did you try or I sent a message and it's instant dependence but it shouldn't be should be local network so that should be fine 8546 it says it gives me a debug root jsonar it says send rock how do I say that so you can also try to use the javascript one that's just endo ws.js and the same two arguments or you can also use the go one which is go run and ws.go and the same two arguments no what you're doing is you're sending the message through your note with your web socket the web socket interface is not exposed by default you can't just connect to an arbitrary one and do this obviously it would be a security risk because the note is running with the private key and everything so I seem to have do you receive it from me? no I don't so I seem to have the note is the same level so I'll restart this note and maybe that'll help just in a second so then we can use admin and admin that'll add me to you if I make this one too certain so I'll have to change that to me this is the message where it's in you should see our message appear it appears very same so that's zero up it says it contains population of two so it's like tonic loss oh yeah oh right yeah sorry it was all scrambled only two notes so I guess they are just connected to each other so I'll restart my little close cluster yeah it would be better than tonight sorry please I'll I'll host it again of course I'm probably scared I can I can maybe maybe I don't have to push it worse ah so that's it yeah sorry about this did you attend to the first workshop? yeah exactly I was sleeping in class so okay that was in 25 see I even visited the I even visited the page I have it in my history so I was getting started workshop right and oh Jesus you changed everything yeah right okay so and then let's do thank you for the support DevOps Gods then we go then we do that two two which one was it it's what what was it oh thanks do you remember my old page I'm getting some coffee there what's up oh ah shit this was the you see but at least the segregation software is working so that's good normally this is my own let's see what's up like I've done with Gnash if you want to answer this is not related to us but if this one this one this one I'm sorry I've got a different question if this this is it here like I'm trying to call the HTTPA the swarm node on in like the special argument what is the argument HTTP under and then 0000 HTTP back thanks man so yeah see that 61 61 yeah stop that apart from the fact that yes that we can't build the FCSJs dependencies we couldn't vary now we do this yeah there's just one though that's very interesting oh there we go slowly make sure like one more I second if I just on the right it worked so so we tried to send the messages if anybody is still not sleeping that was the same message I sent the message it's not from we need yes he said you might get because you are using a different network ID so you like visit like for some time how long let me restart I can just it'll be better or do I have to restart the node as well everything let's yeah yeah yeah yeah you don't know shit that's how you want it to be no you'll like this you'll like this still the same there's a different one still not so many reload it yeah that worked I said I said oh you mean that can you try to change the content of the message yeah I changed it pushing or are we running pushing pushing oh you can account for this kill off so we do I see I mean it's supposed to be like anonymizing that work so I didn't really ask to expose you right well at least that's kind of proof that it works sort of right with all the connectivity and everything running so this is how to use it so basically all of the files here that are in that are in the repo the intention is that they kind of constitute the tutorial how to interface with swarm of PSS on a low level using unix sockets or using web sockets and there is them to just to recap quickly the ws dot something files that are for the raw message ws.pub which is for the public key sim for the symmetric key there is also and I will show that now I will go on to feeds because this took a long time the ws dot in is a monitor that receives messages for you just to put it to the terminal and the notify pi thing is python script that uses the GNOME desktop notification thing that you saw here that makes the messages pop up yeah actually what I didn't mention is what I didn't mention is the receiving nodes in order to prep process the incoming messages you have to register you have to register code handlers to do that otherwise it won't do anything so under the hood there is a subscribe RPC call that does this bit you find this then in this in the ws in thing and also the notify pi just look at that it will show you the RPC call that you need to use in order to be able to receive or to handle incoming messages I won't go through it now it should be pretty self-evident so let's go on to feeds do we still have time for something or I don't know you're on this one I think you still have 50 minutes really time for we started much later of course well let's start in an end but you have to I need to know what is the final call so I mean so we know right that part of the problem with using content to rest up is that every time you update something the address of the content will be completely different so illustrate this once again by swarm upping this file swarm of why am I looking at the screen like that swarm upping this file obviously how are you going to reference if you wanted to have one web page let's say that reference that content that's not so easy so I guess we didn't show anything with the yes so far because it didn't work right but Ethereum offers a name system kind of akin to DNS in which it's possible on-chain to register to a static identifier the address of content or actually a multi-hash reference to content basically that means you change the website you do an E&S transaction on chain and then you wait for 15 seconds in the best case and then somebody can visit your page right and first of all you pay for every time you do this so if you're a cheap skater or poorer for that matter maybe you don't want to do that the other thing being of course that it takes 15 seconds which in the digital world is a hell of a long time so Swarm provides a method to do this under a persistent identifier and we call it Swarm Feeds now Piper touched briefly on this on his talk I think he called it a very fancy name something like a time-based key value store or something like that time-serious database exactly kind of nice lingo so and it's exactly what it is so basically in the repo that we pooled there is a shell script file called updatefeed.sh this file shows you all of the steps that you perform in order to create a feed and update that feed with two different types of content and then pool that content and display so I'll go through the moves and you can have a look at well first of all let's run it and see that it actually works and I hope it works oh whoops so what's that man now oh yes okay so first download yes so this file takes this script takes three arguments it takes the first content you want to post the second content you want to post and also the path to your data directory so it's not really meant for public consumption it's kind of a workshop demo thing so bear with the UI philosophy here so and then go back to the directory hang on oh wait just copy all of it from the last file just a second so I'm not necessarily expecting you to make sense of what's here but basically I just wanted to run the script and make sure that it works ultimately it does so here is the first content that we pulled and here is the second content that we pulled so now I'll go through step by step what this script is actually doing so call again so the identifier of the feed is a combination of two things that's a combination of a topic which is a 32 byte identifier and the amount that signs the update which is the same basically as you're passing through your bzz account argument when you invoke so when we are running a swarm node and we're using the command line interface as we're doing now we will be using the key that's running the node is running on to sign the updates so that's also why we have to pass forward now let's upload the text foo and we get this swarm hash for that content so let's verify that this is correct if it's not then we have had some sort of space-time-continue thing and we might have to correct why so that I get at least those wrong yeah we can do it so so we get this question huh oh I see so should I just continue continue just that you know you tell me when you feel like I've gone too far or good so where were we yes we have the hash so now I want to create a feed first of all we use a swarm create feed create command we pass it topic we pass it our our account that we will be using this is the public IP of the node this is yes the public ID of the node is derived from the same key used for different purposes but ultimately it's the same seed so now we have created a feed actually you don't really create a feed I'm kind of cheating you what you are creating though is a manifest did we go through manifest or oh actually we didn't oh so I was supposed to do that too so basically a manifest is basically a manifest is a directory pointer get into some metadata about what a pointer is basically so you could I guess the best allegory is a directory right but anyway let's have a look and see what's in there of this identifier so oops let's use this wonderful little tool called jq so this is how feed manifest looks like basically it has a special content type and it has those two elements that I just mentioned the topic and the user so basically the idea is that when you use this manifest it provides a shortcut so you don't have to supply these parameters as query spaces now so we created this now we want to update the feed with yes not for me oh sorry I wanted to ask the mod time it just looks like 0 does this indicate the beginning of the feed that mod time is not used in this context so I'm not even sure if the mod time is used of the BCC manifest thing I think it is but for the sake of feeds it's an irrelevant field maybe we should even remove it to change the mod time cause it's a lot of heavy so we remove it actually 0 right so the command for updating a feed is a feed update whoever thought of that aim we have the same parameters the topic and the user in fact we don't use the user here because now when we're doing the update we have to sign the update so basically the chunks that represents the feeds on the network only get passed around between nodes if the signature on that chunk is valid so this is why we have to sign it so we so the user is not just inferred of course here because we're signing it so it knows what to do and then we need some data to add to that so let's add the swarm hash of the first content that we post like so and we have to provide our password of course to sign it probably we should have some feedback on that other than just zero return value but in principle that should work fine then let's see to retrieve it it's a special scheme called BCC feed and to BCC feed you can provide topic user I mean not can you have to and what to do wrong I get the swarm content hash as the account so let's not do that and there we have the content it's a nice binary output but let's pass it to next dump and verify that it starts with 57, 18 blah blah and yeah so the content that we posted with 57, 18 blah blah blah ends with 1807 here is 1807 it looks pretty good we're too lazy to to confirm the rest so yeah as you see then you get the content now what we did was we quietly feed about the latest update that's on the feed and we got back this content from the swarm hash that it references right so because this manifest has a special content type the note even knows it can perform some magic on this so as we saw before when we got the clear text content from this swarm hash then we got the contents behind it back like food right wouldn't it be nice if we could just reference directly feed itself and get the content instead of the hash right and you can actually do that and this is kind of what this manifest is for this is why we call it parade of feed because with this manifest this is possible so instead what we do is we reference use the BCC scheme that we already know and used right and we pass it the hash of the feed hash of the feed manifest in fact pretty close and the question is where the hell did I put that now because it's not so easy to read anything here I think this is the one there we go so as we see now then now BCC this was the manifest hash and we got back food so far so good but still not really magic right magic comes when we update this feed so let's go ahead and do that now so we have bar text let's see hides oh bar let's now update the feeds with this the command was swarm and we needed the data there because we're cheating and feeds update very good thank you we need the topic oh actually if we're really really lazy and now a bit risky because I'm not sure if I even tried this before so you see the swarm feeds command actually has manifest flag which takes the parameters from the manifest instead of having to supply it on the command line of this so let's let's live dangerously and try that and maybe it's going to need a 0x I'm not sure and then far so good update found I guess maybe this doesn't work oh wait there's a 6 missing okay let's do it the non-lazy way it's 64 right because it's 32x2 and then to find somewhere in this terminal the user ID I think it was this user no fuck we don't need this sorry pottymouth visit again visit again we do visit again and oh f598 and d8f6 I don't think that's what we had last time let's just verify and see that oh my god here we go oh wait was that actually the content hash though oh yeah so it wasn't that impressive there we go so that's like yeah so far so good is this a good time to do some questions are there any yes great 5 or versioning so right now you can see let's imagine that in the url you've got what version equal let's say 2 or 1 or 3 or back 2 versions so instead of retrieving born are you full is this allowed or is this in your roadmap well so actually ironically the original implementation of this allowed for exactly this I am a bit unsure how if it's this actually is possible now so there is a possibility of hinting the algorithm to a starting point on where to look and maybe this is a good time to bring up some theory about feeds but the theory about feeds well anyway so but I can't I'll sit here and guarantee you that this actually works in this way anymore I don't know Victor for example can you remember if this works out we didn't use time stamps these two versions just simply give the version that we treated I think now yeah exactly so I think if you actually know them if you actually know them you know the time of the previous version if the theoretical problem here is that you can't really you don't really keep up the previous version you can't really know when the previous version was was the issue therefore it's very difficult to maybe if we use the get methodology how gets doing the vergening then maybe we can apply it to right there is also a question because you know this whole scheme is kind of a little bit of a hit and miss scheme so basically so let's just take a look we can do the theory in this but maybe there are some I don't know simple questions first yeah so right now you can only one user can update the feed or at least only one address and is it in the roadmap more than one or is it like not possible for you multi-seeker yeah the multi-seeker is not implemented it's not in the roadmap concretely but yes you point to a very valid point of warning in the sense that if you do two updates from two different nodes at exactly the same time you will maybe have a fork and in the network on what was that actual update so you should not do this currently either you know just publish from one node or it's up to you to make sure that you don't publish at the same time or at the same key at least for different nodes a little bit of a comment here so the difference between feeds and updates on the EMS is precisely this that with feeds we don't really have a consensus we can't handle concurrency the same way the blockchain handles concurrency well so this is kind of the core difference here and because of that if you imagine some large scale application let's say a something like Reddit then you will have eventual consistency through the blockchain but you cannot like if people are posting comments through feeds where they're publishing comments with their own keys with their own feeds what you can do as a client is you can monitor the clients the other feeds that are currently active with that feed and you can have a sense of like real time changing content so that's how you see your applications that all the real time stuff happens in feeds but then you have some sort of eventual consistency through the blockchain because that's where you can handle consensus this is not a consensus it is not very specific so because they will be updated on the same on the same time so that will be through different points so the chunks are distributed so once you see a chunk with a particular address then whenever you see that the address will give you a very important assumption of course what it is and therefore that if you want several people to update the feed then obviously this will be the same private feed not very good idea and also the other feeds basically are that you can't guarantee that they even if they use a different types then it's the most likely I find that it's really not exactly the same same time stuff but they can't guarantee that that they did the same last one so maybe it's a it's like it's like everything they only update the same time as long as you can potential and that is why they so all kinds of streams which we are signed whatever scheme you find then they will be possible to come through it's kind of it's nothing to guarantee the eligibility but if you want the private feed you can screw it yourself but so if you want to update the feed the same time stuff but it's going to be screwed so that's important assumption and then in this situation when the use case is kind of that several parties think they think the same way to base you know best practices a bit glorified it's always hardly working but the recommended pattern to use is that everyone updates their contributions any kind of index that needs to contain items updated by different people should be an aggregate index should be a service which aggregates the content from various sources the the simplest method for this is I will do that also so for example if you imagine a Facebook feed then it's a more realistic scenario because the number of people who really want to monitor is finite and sport so you can actually pull that feed and then we construct the commands on your feed so basically you can use Facebook by the client side aggregating feeds of the source and so you have to come up with this how you reference the commands to your posts and there are use cases when this doesn't apply when the customer read its scale and there's thousands of potential users commenting on the post in these cases you have to centralize the aggregate index which which can only decentralize properly if you have a source or whatever that challenge or verify the index basically someone promises to change the contract to index correctly and if they don't index your command then you can use them and they can use that because that's what it is okay I think I have a question that I can ask you later do you mind if I switch up? do you mind if I ask you do you see that you need to be proactive or can you understand yeah I'll show you a bit actually if you get the question let's just briefly touch on this maybe so the actual the feeds themselves are chunks but they are not content addressed obviously then it wouldn't work so we have to have a predictable way of constructing them how to reference the data and this is done by concatenating the topic the address of the user account the time stamp of the update when it was made and something called level we bundle this together we take the hash of it and the last one this is what it's actually asking for now the level thing needs a little bit of explanation so if we were just using the time stamp we would have a resolution of one second per update which would mean basically that you would have to ask the network for every second from now and backwards until you find a hit this is obviously not so optimal it takes a long time so we have a second level of categorization which is basically to bundle together exponential durations from a second so it works like this the higher up you get the more time it represents and you fill it from the top and down so here zero is one second each of these is one second one is two seconds two is four this would be 16 seconds but as you fill this down like this you have a logarithmic way of looking back into time without having to fill in all the different seconds in between that's clear right pretty much is it clear? great so as you see level six would be a minute level 12 in hours 17 in days 20 in weeks 22 in months 25 in year and the highest level is 31 which is approximately 68 years and the idea was to just have one look up back to 1970 first of January the contents of the chunk themselves so in order to be able to verify the signature of the chunk we need to have some sort of reverse look up of what the metadata actually is so we have a version field a few bytes for future use and the topic in the address the unique time step and the level which we already saw this is the prefix of the chunk then we have the data and the signature at the end that's what's inside the chunk when we ask for that address and the signature is constructed from the digest a digest of the data which in the digest is the version so basically everything down to here except the signature so to your question then obviously you have all of this information right so you could sign this as long as you have the private key outside of the note somehow and I don't think I'm going to go through like the whole body of the script that I prepared for this because you guys still want to show something right with some real world to use it for this I could just briefly refer to in the repo that we pulled there is a file called update feed js it references a npm package and please show mercy it is my first npm package and it's kind of sort of dicey pack of providing access to methods to actually do the swarm feed stuff client side so it has like a very very primitive wallet implementation and it gives you the digest the digest production and also that's the signing and you will have to pull this module of course to use it and then this script will show you how to to put this together in order to make one single update totally client side without dependency on the command line tool and stuff so and yes I wanted to before I gave it over to them I wanted to also point you to in the readme so where is the readme here it is too many windows hang on at the bottom of the readme I have a list of other resources that are available for this so I earlier wrote DevP2P general DevP2P tutorial in GoLine which shows the whole DevP2P stack and how to create an Ethereum and so on and so forth and it also covers PSS there is also a very simple proof of concept messenger DAP that we have on our repo that's written in I can't remember what frame but it's browser javascript so have a look at this, maybe it's a reactor or something I don't know here is the one I mentioned and yes as Victor alluded to Paul and the guys in mainframe wrote the Erebus library which doesn't do only feeds it does feeds, MPSS and Swarm in general it is type script though not javascript so but you gave me a reason to learn type script so I thank you for this well I guess it didn't learn well enough then because I had some trouble with the transition but yeah then the second one, some cleave yeah I'll just read through the rest what's the read chat in PSS oh don't get me started it's very unfinished but it actually tries to do group chatting using PSS and feeds and it's a plug into the IRC curses client program we chat so it's not finished but it actually illustrates I guess some of the stuff that Victor was talking about too about aggregating different feeds into one timeline and blah blah blah and so on and so forth so it kind of demonstrates but it's unfinished work somebody wants to pick it up and run with it then please feel free so I think I'll leave it at that for this part and then if it is and Stan my previous question you asked about the you know somehow standard scheme is used for example I update at least 10 seconds or the most accurate question contains at least 10 seconds then you can always construct the address and then you can actually read the chat read the chat you know this is the work so this is the way it is because you would have left it you would have left it so the standard scheme is not directed so thank you so yeah I'm the diffuse I'm the city of Datafund we empower users to reclaim to own and manage their data and we want to enable developers to create daughterage applications with privacy by design apps we are fair believers that fair data economy up to us and that's why we started working a library called mds.js which enables us to do different kind of applications really fast on one of the things we started doing is a concept called fair drive which is our attempt how we will give users storage and means and give data back to the people the specs can be found on the github everybody are welcome to contribute really like welcome if you are not willing to contribute we also started to give a campaign so you can give us some money but basically that's it we will slowly now go to the apps that we built using the library one is the swimming pool which will be difficult to show because there is no internet here and then another one is fair drop so small hole is drop you drop a file and you send it fair drop is a little bit more advanced it has user interface you can send store files to other users and that's chatty it's kind of a developer sandbox where we use it to test our ideas and how things can work of all this we created consent management which is a GDPR compliant consent management editor and viewer compliant with cantara specification and on top of it you can sign send consents over swam and sign them on blockchain and both parties can agree or revoke and work on these consents so yeah I will leave the talk now to them to show you the awesome power of FDS library ok so I want to go right to the top level of the staff now we decided to build these applications that are indistinguishable from web 2 that are completely web 3 under the hood and to have a stable infrastructure so how did we made these the kind of distinguishing factor is that we've got we're using both blockchain and swarm network using kind of blockchains, polyfill elements that are unavailable in swarm yeah like the multi-sync swarm aspects and we've prepared a short workshop right through very quickly so for anyone who wants to follow along you can go to this address so it's 1 1 2 1 6 8 1 3 3 8 8 that's got all the information you'll need ok so FDSJS is an MPM module so normally we just in it the new and at the the dependency we can't do that right now so we've actually uploaded to swarm something that we've already built the dependency for you so if you can just navigate to that address you'll see this and you have to be on this Wi-Fi so on swarm workshops Wi-Fi navigate to that address I'll follow it go into the directories and start a new NOS Ripple we've got this flag here just gives us a way of sensing the group level so just because we're using NOS we're going to add a stub file class because that's normally available in the browser and it's such a FDS with empty arguments but I'm actually running all this locally so you can see we've got DNS config the force here that gives out free testnet ether, an 8 gateway and a swarm gateway so just you guys will need to change this to be specific to whatever you want to do just create two these names and just instantiate the FDS library make new objects with that config we've just added and now we're getting to the actual FDS since that's itself so now this is super super high level so I'll give you a quick idea of what's going on here what just happened and there was we registered the sub domain with the DNS implementation that we've got on the testnet and registered the private key there we also registered the address of a contract that we call multi box public key public key public key and it's great and the address of the contract called multi box which is a contract that we're using to essentially polyfill the feed to multi-sig situation that we were talking about before so this contracts allows another user to come to you and say okay we made a swarm feed for messages for you and it's at this location that's one again the other again send the file so it's just a simple stub file this is where we send it and that path at the end is for multi box so you can have multiple different paths you can send stuff to depending on what application it is and that enables interoperability across applications there's also a whole layer of client-side encryption it's not simply relevant to this everything gets encrypted and the riders are in FDSJS so when the cyber breach is swarmed it's already encrypted so that's the messages consent and then this is the other side where you can go and check and what this does is it aggregates all of the feeds that were indicated in multi box all of the feeds retrieves them decrypts them based on the public key in ENS and then aggregates them into a query so see there we've got one message excuse the user details underneath just into that file and check and it'll say what's the message that was sent actually so actually anyone who's created an account can use their message to send in between each other and send messages so that's the super high level bit and then I can show you a little bit what is going on under the head there so for example this is the swarm feeds bit of the library and this is what I've reduced down to a simple get and set interface so you can see it's got this two-step process the update resource relates to the feed and this send request relates to the actual file itself the data so in FES there's like three levels it's the initial pointer that points to the manifest of the data and the manifest itself and then the raw data stored in the library so you can see the first two levels here and it's got a bit at the bottom of the thing where I'll show you a little bit how how that kind of manifests itself into the swarm itself so that's just getting some dependencies available there's word3usls then this is using a lower level interface that's related to this initial account to send the topic name test to be a value test to start digging into that a little bit so that's the address that we've created for Alice this is what the topic ends up looking like once we've passed it to hex and padded it with some zeros then we can put that together to have a look at what has actually ended up in the swarm there so this is basically the encryption step just so we can see what's going on go to bash so you can see this is what Louise just showed us using BCC feed Topic ID that I'll just generate up here after he's redressed for Alice should give us back hash then we can use BCC or either we send up here yeah I'm not sure what I'm sending up here it just ends up being there sometimes so there's you can see that's kind of that's two steps to what is actually quite complex serve interrelated data that ends up with this fair drop situation where we're holding multiple feeds and then aggregating everything into a set of messages from different users that's it the same university today we talk about what Merkel met entanglement so in this talk I will talk about redundancy systems about our collaboration with swarm and the prototype that Racine is working on use of site redundancy and then the impact of redundancy and market risk so our motivation is to store a document for long term in a system do you think you can recover a file after a decade or more with current redundant mechanisms? It's a bit of a challenge so this is one there's many many cute bees but these bees may decide to be free at some point and some may fail or disappear and so it's a failure the failure happens so what happen if you have your file in that note that suddenly disappear you will be very sorry for that so I will talk about three different approaches to achieve redundancy in a business basically they are like well-known mantras of how that's I call it the more the better and the less the better and then I will talk about my work this was developed during my PhD and then on those codes so let's go to replication. Replication is very easy just copy the file, it's a straight forward but it has expensive space requirements because you need to copy the whole file and it's not good for long term storage especially because for instance in so on when you are syncing the file or in any peer-to-peer or when basically when you are copying blogs, it's your copying blogs that are popular but in long term storage or something that is a document that is your personal or nobody cares you can use that device but still you need some some redundant method that is appropriate for long term storage so here is a block that is missing so then in this way with replication basically you have with three copies you have two different small ways to record and when you distribute this file and these three replicas there are some nodes that participate in this and when one of these disappears so this node can give you the file that is missing without any decoding just a straight forward in a racial coding I would talk about m out of n blocks so basically you have the file and instead of just storing the file you split the file in m blocks and with some magic base of arithmetics doing it in a lot of fields you create more redundancy with this redundancy you can save with this approach you can save as space requirements but this is very expensive with repairs and with the maintenance in the system because you always need to read m blocks to record the file so again you have some other ways to record the file but these are import many blocks and here in this case we distribute the blocks so instead of the original replica that we have in replication we split in four fragments we generate some redundancy so there are more peers involved in the repairs and you see that if one of these is missing you have to found four other peers that are available to record the file you can increase so in this previous example you can tolerate up to three failures then how do you increase for tolerance? you can subdivide this initial fragments and then you generate more redundant fragments but in this case we are with the same space overhead and now there are more nodes involved but the problem is that we are also increasing the number of peers that need to be available to do the repair and this is a problem in the repair network for instance if the system is under attack or if they have low availability this is not good so what to cover with the antagonist the main idea is that you mix the file with other files in the system or you mix the blocks of your file with other blocks of your file and how do we do it? this is based on XOR operation so it's much cheaper operation than the Gallo fields and it's very simple like XOR into your block with alpha other blocks and that will generate alpha blocks but because this is part of a system that is all entangled together then you can use other blocks out of this alpha and alpha other blocks that can be used during the repairs so the good thing about the antagonist is that repair costs is constant no matter the alpha that you use you will always need two blocks for repairing one single file and the storage overhead is given into the alpha blocks that you are generating and for the ones closest potentially with these alpha blocks because now when the block is missing you will have this path of two blocks or this path of two blocks actually alpha path of this but then you will have more and more as a larger path to recover the block if there is a bigger failure that is affecting the system Brazil will talk about tanglements first so in this case now we have an equivalent block that is equivalent to the replication and now if the block is missing because the VR is not there so we will use only two nodes and you see now that we have glue that are the direct blocks that were generated during the tanglements but there are also many other blocks that were already stored in the system or that were stored later in the system that can be used during this repair process so that increases that increases the number of piers that can participate during repairs and the possibilities and we are distributing more than information and that is actually good and also if we want to increase even more the involvement of the piers in the system instead of using these sides we can use a small block so that we split again the original block for fragments and then in this way we are using the same as the previous example but now we are basically distributing the file in many many more piers and then the repairs can be parallelized better, can be done in a more efficient way so in a nutshell so here are replicas replicas you always need so like PR availability requirement is something that is not good so replicas are the best because it's just one pill that you need in entanglements you always need two piers and in the traditional erasure codes it depends but S is M but we can also as I showed in the previous slides we can increase the block distribution in the large number of piers which is something positive but in the case of erasure codes in this way growing in this way requires more storage overhead but with entanglements we can use small blocks and without increasing the storage and Rassim we will talk about this demo so I will go through the prototype very briefly and then show you a demo of the repair algorithm and show some results so in a nutshell this is how the traditional or parallel to swung is now the entire file is sent to some swung node it might be your own swung node and then you chunks the file into 4k and the file is synced across the network so what we did in our prototype is that we implemented an additional layer and goat which spits the file into 4k and adds the encoded blocks according to the redundancy scheme and this is sent to the swung node in the same way and synced across the network but when once the nodes receive the file the user gets the manifest hash which it keeps in our collection because this collection is then used when the user wants to restore the file because it has the information about what is the data what is the properties so I will show you the demo here so this this shows all the data blocks in the whole file so total 115 data blocks and then the coloring down has means the red ones will be unavailable and unavailable means that it could not retrieve the file within one second so it doesn't distinguish what kind of data we cannot get the data within one second it's a failure then the blue ones it's repaired and green is downloaded so it starts off by this coloring curve as you can see there is some which is already red but when it goes to number number 59 so this is event 459 it request a parity block for number 25 which we will use to repair later on but also what can happen is that the parity can also fail so the algorithm keeps running and number 41 you see that the one block is repaired using these two parities on the side and then if we go to 160 yeah so here you have a case where there is two parities at each other which is failed and what the algorithm does here it first repairs the parity which is further away and now it can repair the one it needs and then now it has the sufficient information to repair the data block and this keeps running until all of the blocks are grey and then it terminates so to go back to the presentation here we also have done some evaluations and here we inserted errors of 5 and 15 percent and we used one prototype with entanglement codes and one prototype with grid sormon and again is this one second failure that you cannot achieve the block within one second it's a failure this is what we see in the data so this is the results for a file of one megabyte and as you can see for the baseline sorm the entanglement codes are performing better and we believe that this is because we are only connecting to one single sorm node when we are retrieving the file and this has some limit on concurrency but with our prototype we are overloading this because we are issuing all of the requests in parallel the results with grid sormon they are also not too good because this is like we need to do some more work on the implementation and we also think this will be more reduced if we ran sorm nodes ourselves but the main takeaway here is that we can implement user side redundancy and even with relatively high amounts of failure it doesn't have any cost in the download time so next step is redundancy so what's happening if we move redundancy to the network the system what we are interested in now is how these market rates are affected also with redundancy and so then when you have if you are storing data off-chain like if you use one to store data off-chain usually this is the the root of the market in the blockchain or in Ethereum so here we have the data block at the leaves of the tree and all these other blocks are basically hashes and this is interesting because we can do an efficient verification also if we want to know that this block is part of this market rate this block is part of the data that I go to download so I need to only modify with some more blocks hash blocks from the market but how how this market rate is modified when we introduce redundancy so when we are using replicas this basically is not modified because unless we are inserting some tag or some label to this replica I mean the replica is identical so we have exact hashes and what will be exact so no modification when we are using erasure codes basically the market rate grows to other directions because first we will have more leaves because we will have the encoded blocks and then we will have more hashes and then basically it also increases in height and there are also some other challenges that we have seen in previous implementations in Tahoe and also in Bitcoin when they are using erasure codes I need to be careful because there are some kind of attacks that you can like for instance include a second file here in the codes and they will have the same the same route balance so let's know how to discuss this anyway with entangled blocks the thing is that because in the way that the process is done and coding is done basically we are creating a kind of chain of entangled blocks and this chain is basically if you absorb every two consecutive blocks these two consecutive blocks will represent exactly the original data block so as you see like in this case the first one is equivalent to this one but this one can be regenerated with two blocks and this one with these two and this one exactly and we have actually multiple layers that you can do like this so basically you can play in the implementation so to avoid modifying the particle free because of this property of the entangled block we didn't go further than this but we are thinking in this for next implementations so concluding remarks Swan is ambitious to provide long term storage so it needs to combine different redundancy methods to provide the service to provide different service and so at the moment it only adds replication at the syncing process and it's considering implementing the erasure coding and entanglement so towards this goal we show how can we add entanglement at the user side and we show some measurements about this and then we expect to complete this continue with this work and apply entanglement after ficing it inside the node so that's all for it for today Any questions? So could we go back to the slide with the marco tree? So the green the green blocks, the green chunks which form the marco tree are they in any way protected against loss because they are just regular the swarm network does not distinguish between blue and green chunks it doesn't know which one is green which one is blue so do you are you considering some redundancy for the green ones as well? Yeah that's what we discuss when we went to Madrid that we could also include made entanglement at this size but at the moment we are not sure if that will be necessary I mean the extra storage of our head for doing this entanglement adding redundancy here it will be not much but it adds complexity to do this no? but on the other side because these are well protected you can regenerate all these because basically this is a hash so if you can recover this because you have a very high tolerance here because you can't find them that's how you retrieve a file you basically start unpacking unpacking from the entire package tree if you have the lowest level what they're saying if you're just asking for a hash they don't know what it takes to regenerate it if you just ask for it give me this hash and it's missing you access the file from the root hash that's how you need to unpack it you take the top chunk with the root hash yeah that's true I mean for for locality you access from here and my talk here is the verification of this of if this block belongs to a sort of file or if it is for locality you need you really need to add these extra results we are running already late so let's go for the last question yeah so I just kind of assume that the context that we've been taking is like what can be entailed with what is like it's as soon to be like it's a rick and not necessarily like a global it depends how do you implement one way could be like they know that uploads the content just has in memory some last like for instance with this example that we did is like 15 blocks in memory and keeps doing the entanglement with these last blocks and keeps always in yeah that's one way to entangle with blocks that are in the system then you need to download like free blocks for or you can keep it like maybe you don't need to download if you have a pool of like blocks that you were already syncing before and reuse this pool to do the entanglement automatically you are entangled your file with other blocks in the system yeah so I guess my question is if like I'm a note on this network the context of the entanglement of what I'm storing or will I just not know well this can be encrypted this is is that what you're asking did you we can discuss we are running thank you so much again a little bit of overlap incentive talk that was I can wait in the morning so there you will you will probably recognize some of the themes but I would like to approach this set of problems from a slightly different angle so what am I going to talk about first I'm going to outline what the challenges of persistency are so basically we may network of freely participating nodes where every single node is operated potentially by a different entity following its own interests what are why does it how does it become possible to nevertheless ensure that the data that we upload to this very very diverse and constantly reorganizing network how why it can still stay there and remain retrievable then we're going I'm going to talk about swap incentives so this is the part that is a little bit overlapping with the morning presentation but I will more focus on the emergent behavior so I will talk about what incentives we have in place and what we expect to happen because of those incentives then I'm going to talk about protection against spam and I use this word spam in the broadest sense possible that's how people at Google use it so basically it's any content that you don't want so it's not necessarily advertisement it's and anything that is that is unwanted content that takes up space takes up bandwidth and so on and then in the last part of my presentation I will go through a few use cases possible use cases or use cases that people are currently already working on and see how they play out with with the kind of tools that we have or that we're going to have so let's start with the challenges so the challenges are basically come from four different sources the most important one is churn so nodes are coming and going they take some data with them when new nodes join the network then some content which they don't yet have is being searched as of them and so on so the fact that nodes are leaving and joining the network it does put a stress on content and it does more challenging to retain it and to locate it and I went to talk about how we address this the second one is spam so what spam is in this context it's content that it potentially crowds out other content so even though we expect that swarm will have many many nodes but no matter how many nodes swarm has its resources in terms of storage and bandwidth will always be limited and if there's no bound on the content that is being uploaded and there are some problems with the incentivization of that then we can expect spam crowd out useful content and I'm going to talk in detail about what we're going to do about it and what we're currently doing about it then there's opportunity costs so again because of the limited resources in this case of one particular node so they have some storage they have some bandwidth and they are making decisions individually each node or each nodes operator is making decisions of how to use these limited resources and by not using them for one purpose or sorry by using them for one purpose therefore going the opportunity of using it for some other purpose and then finally we have incentive misalignments which are again well known problems in game theory and economics so I will shortly go through them how they affect swarm in particular so sure when nodes leave it's obvious how that is a problem because they reduce their redundancy we have the same information in the network stored on fewer nodes nodes joining swarm they require time to what we call sync so in swarm lingo syncing means obtaining all the data that will be eventually searched at your node so in swarm the addresses of nodes and the addresses of data come from the same address space it's a distributed hash table that's what it means which means that your address kind of determines what kind of data you should store what kind of data is going to be requested from your particular node and what we call syncing is the process of obtaining that data and if nodes join it takes time while they sync during that time they are still they cannot actively participate in the network they cannot they cannot serve requests so from these two things it's apparent that any changes in membership be it nodes joining or nodes leaving else some kind of data movement something needs to happen because nodes so as nodes come and go data needs to be shifted around and that is that particular that particular process is what we call pool syncing in swarm so when you're reading swarm documentation pool syncing is the special kind of syncing that happens because there are changes in the membership of the network and also we have this concept of the radius of responsibility which is the kind of data the kind of distance of data for which a single node is responsible so if swarm consisted of a single node then its radius of responsibility will be infinite so basically it would encompass the whole data space but as more and more nodes join the network this radius of responsibility is getting smaller and smaller and and as nodes leave it's getting bigger and those changes those changes are the ones that need to be handled by the pool syncing process spam so there are two kinds of spam there's malicious spam and there's accidental spam so in case of malicious spam and I will talk about when some nodes might engage in malicious spamming so it's an attack it's an attack it's performed by an attacker who has a specific selfish goal of obtaining some benefit at some other nodes expense and because of that they are pushing content to swarm they either want to do it for the purposes of censorship or they want to to get paid for something or and so on so there can be malicious spam in objective and there's accidental spam so basically if uploading doesn't cost anything then people might not be expected to be sparing with their upload activities and they just might casually upload a few gigabytes not caring about the consequences and the load that they place on the rest of the network and one of the interesting challenges that we faced and I think we did a pretty good job of resolving it that it's really difficult to tell uploading from syncing so in both cases data is being moved but in one of the cases when you're uploading data you're uploading content into the network that has not been there initially in the other case in case of syncing you're just moving content around as necessary and of course the difference between the two is that syncing is something that is necessary for the continued retrievability of data from swarm whereas uploading is a service that in which for which the network which will take more resources from the network and therefore somehow the network needs to be compensated for but from a single nodes point of view a single node has a very limited a very limited view of the whole network it is only connected to a logarithmic number of piers logarithm of the total number of piers from a single nodes point of view it is really difficult to tell whether a chunk that is coming in is syncing or an upload and as you can see from the arguments that I just highlighted by misleading the node about the nature of the incoming chunk you can really influence there and so if uploads are free then they might crowd out important data and also censorship resistance spam resistance they are kind of at odds with each other to some extent because if it's easy so if there's some spam protection mechanism then one avenue of attack a censorship attack is to somehow specify the content that you want removed or suppressed inside the network is to classify it as spam so on one hand the network needs to be like it needs to be difficult to remove data that is that is uploaded legitimately on the other hand it should be easy to get rid of spam and there's kind of a trade of here and I'm going to talk about how so now I'm just highlighting the challenges but then I will talk about how we are actually meeting these challenges so there's opportunity costs I already talked about them so basically there are two kinds of opportunity costs one is storing swarm data versus something else so you have a limited disk space and you want to allocate some of it for swarm and other parts of it for other things or even inside the swarm inside the space that you allocate it for swarm there's a choice whether you store this data versus other data and these all imply some opportunity costs and there should be a clear way for each node to make these decisions in such a way that if they follow their own rational interest the emergent behavior will be that they will actually work in the interest of the whole network so the misaligned incentives are so the problems caused by misaligned incentives are selfish storage which is kind of an analogy with selfish mining where basically by maximizing your profitability you somehow are doing unnecessary work so in selfish mining if you recall it's for example things like mining empty blocks and also there's a tragedy of there might be tragedy of common situations when there's some behavior that's in everybody's interest so the benefits are shared for the whole network but the costs are kind of private to the node that is engaging in this behavior and in this case we can expect this kind of behavior will be under provided so these are the incentive misalignments that we want to avoid so the one incentive system, the most basic incentive system that we already have implemented and tested and you have already listened to a very detailed presentation about how it works and now I'm going to talk about how SWAP all by itself addresses some of these concerns so just a quick recap what SWAP does on a very high level so in each case if one node wants something from another node they eventually pay that's kind of the bird's eye view of the thing and of course the option number one is that they pay by reciprocal service so typically they don't pay by something checks or making on-chain payments the typical preferred way of paying for services reciprocal service if the accounting of mutual services gets this balanced then it can still be settled by a payment and if it gets this balanced and there's no payment coming forth then eventually the sanction is to disconnect the node that is running in the red so I don't want to talk too much about it since you already heard a presentation in the morning but the interesting part which I would like to spend a little bit more time on is what these things incentivize like once we have SWAP what can we expect to happen first of all nodes will try to retain the most profitable chunks so they want to retain the chunks they want to store the chunks it's in their best interest in terms of SWAP to keep chunks which are requested from them very often typically these are the chunks that are nearest to them which is wonderful because that's what we want them to do so SWAP all by itself incentivizes that you kind of follow the DHT protocol correctly because if you are storing chunks that are far away from you then they're not going to get requested from you and that's then wait for you the other thing that SWAP incentivizes is in caching so if there's some popular content that is being requested from you all the time and even though it's outside of your radius of responsibility but somehow it gets requested very often then you can earn caching because you kind of buy it once you store it and each time it's requested of you you're selling it so actually SWAP does incentivize caching so we have some caching code in SWAP nodes and it's not worth for somebody to take that code out because that caching behavior of the SWAP node is actually in the interest of that node through you know through being paid for it by SWAP so one consequence of SWAP being in place is that we can expect popular content to be replicated very broadly and persisted so if there's some very popular that something like I don't know like wallets, like my crypto or some other that everybody uses then we can count on being available forever because SWAP actually incentivizes both in storage and it's caching now there's one thing that we encountered observing what happens which what we call the donut effect is that if some content is really popular and it's requested all the time then it's always served from cash and it's not actually served from within it's radius of responsibility which means that selfish nodes that basically keep a tab on every little chunk and count how much that chunk earns them they might decide that even though I'm the closest node to this chunk I will still garbage collect this chunk because it doesn't make me any profit but that's okay it's okay because while the content is popular it's served from cash anyway so the users of the network will not experience any loss of content and if the popularity comes down then the sinking process will kind of gravitate this content back to where it belongs so the content will still persist so the donut effect does not affect persistency even though it's an observable behavior however sinking itself the fact that content is being pushed towards where it belongs is incentivized only weekly by swap basically there are two reasons why you want to sink content one is that you want to make room for other content so instead of just deleting something you rather sink it before you delete it because there's this common good thingy that if we all do this then the network will work better and we will kind of all benefit from it and it doesn't cost much more or it shouldn't cost I will talk about it so it shouldn't cost you much more to sink something and then delete it and just simply delete it so the fact that you know you're earning money by swap therefore you want the network to function correctly yet you want to make room for stuff that is profitable for you these two things together weekly incentivize that you push content that is far away from you to where it really belongs but it's a weak incentive it's not a strong incentive and garbage collection again notes can be expected their garbage collection strategy to maximum profit and there's this heuristic that we have implemented it's the garbage collection queue so garbage collection queue is consumed from the bottom whatever is at the bottom of this queue gets garbage collected and chunks that are have been accessed recently they're moved to the top of the queue which means that if something gets requested from me so people actually paid to me I don't want it to garbage collect soon so I move it from wherever it is in the queue I move it to the top that's one strategy however it's very important that sinking does not insert into the top of this queue because what's at the top of this queue is kind of proven content that has already been requested from you the stuff that is being forwarded towards you it's unproven content you're not sure you cannot be sure that it will ever be requested all of it can be spam and you don't want sinking to crowd out profitable content so what happens in practice and the way these warm notes are currently implemented is that sinking actually sinks somewhere towards the middle of this queue and what would really help and we're going to have that and I will talk about it shortly is some credible signals some signals that are really costly to fake that somehow marks important content that distinguishes between garbage and spam and on one hand and important valuable information on the other hand because if we have something like this it would actually advise collection strategy in order to be profitable for the note that does it so we have arrived to the third part of the presentation so how do we distinguish spam from important information and for that we have come up with what we call a postage stamp model so it's been inspired by the international mail delivery how like paper mail is being delivered across the globe for like at least two or three centuries now two for sure so upload should cost like there should for somebody needs to pay for uploading because uploading does impose new costs on the whole of the network but it's important that only the uploader pays for it because otherwise if those that are syncing uploaded content would also would also need to pay for it they just won't do it so somehow we need to mark chunks as okay this chunk is paid for and that should be recognizable by looking at that chunk and so the unit of accounting here is so the dimensionality is value over information over time so that's one example et age over chunk over second but it could be anything it can be like dollar per kilobyte per year or something like that so the dimensionality of accounting is this and a very important insight is that because the hash function distributes chunk uniformly across all the nodes actually the local information that they note has is actually sufficient it's surprising so basically you don't need to see all the chunks marked with the same type of postage stamp and yet you can have a pretty good guess of how many there are you know how large the network is because you have a logarithmic number of pairs so up to a factor of 2 you know how many nodes there are in swarm and you know how many chunks with that particular postage stamps you observe and you simply multiply that by the number of nodes in the network and that gives you an accurate estimate an accurate enough estimate of how many chunks have been marked with that postage stamp moreover the inaccuracy that you have stamps from the inaccuracy of your estimate from the number of nodes in the network but that's the same for all types of for all types of postage stamps meaning that in order to compare two different postage stamps your accuracy of estimate will be still will be better still because the error that you make with the number of nodes estimate will be the same for both of them and it will kind of cancel out so the simplest the simplest implementation of the postage stamp and that's the one that we're going to implement first is the so called proof of burn so proof of burn means that in order to put postage stamp on a number of chunks you need to prove that you have spent either so proof of work wouldn't work proof of work is a very natural candidate and we could just take a page from Whisper which uses proof of work for this very purpose but the problem is that the delay the amount of time that you need to spend to generate that proof of work would be proportional to the file size so it would be a real drag on the performance of the network and the other problem is that proof of work has a huge specialized attacker advantage so if some attacker is specializing in attacking this particular this particular feature then they have a giant advantage over the regular users who are using general purpose hardware for just using it so specialized attackers with special purpose machines or special purpose software have a huge advantage and that would actually become a denial of service vulnerability because they would spend the network at a much lower cost to themselves than what the legitimate users would have to expand so proof of work is as attractive as it is unfortunately it's unworkable so instead of that we have proof of burn which is basically a digital signature so proof of burn is a digital signature on a chunk with a public key which is registered with a smart contract in exchange for actual payment which is locked up in the smart contract and not released so you burn some ether in exchange for that your public key with an expiration date is registered by that smart contract that anybody can check with an access and therefore they can check whether you have deep burned enough ether in order to put that signature on that particular chunk so you pay once on chain to this contract and then you can sign a whole bunch of chunks but the individual nodes in SWARM will know how many chunks you have signed with that particular with that particular key so they can divide it so they can tell how much you actually burned in order to sign that one chunk which is nice because you do only one on chain operation and you can do millions of chain operations so we are using the blockchain sparing of course it raises some privacy concerns the privacy concern here is that if you're doing an upload and you're signing every chunk with the same key then it kind of indicates that all these chunks belong together so even if they are encrypted you don't know the structure you don't know in what order they follow after one another but still you know that these chunks they originate from the same upload and that can be a vulnerability for example for censorship so if some of these chunks turn out to be whatever offensive content that some powerful entity wants of the SWARM then they can say that okay, remove everything that's been signed with this key or else and we want to avoid that so we somehow want to solve this problem and we actually do have a solution I will not go into details so we will have this Ask Me Anything part in which I will explain this in detail it involves a cryptographic primitive cold blind signatures I will talk about them later one-on-one during the Ask Me Anything session and the next stage after the proof of burn is the storage lottery which is even better than proof of burn because it actually allows storing nodes to use the data that they are storing as lottery tickets so each chunk that has been signed because of a payment to a smart contract automatically becomes a raffle ticket in this lottery in which the postage that was paid for uploading all those chunks will be raffled off to those who are actually storing these chunks and this way you can actually strongly incentivize syncing so at this point if we implement storage lottery syncing will no longer be weakly incentivized there will be a very strong profit motive in storing what you need to store because we can have these very easily enforced rules that whoever submits a chunk to claim their winning in the lottery if somebody who has an address that is closer to this chunk also submits it they will actually win so they can challenge that and win the prize so the closer you are as an uploader the closest uploader is the one that wins and because of that a lottery ticket is suddenly worth much less for the notes further away than for the notes that are closer which means that the note that is closer is willing to pay for it is willing to buy it for a price for which the seller is willing to sell because the value for the two is different so there is a mutually beneficial exchange transaction that can happen over swap so you can use swap to pay for these and suddenly you don't have that problem that who pays for sinking because notes actually want to buy the lottery tickets which give them a expected expected and calculable so an expected winning that they can actually calculate they can statistically forecast it and therefore they know at what price what price they are willing to pay and because of the way it is arranged the notes that are further away they are willing to sell for that price which creates a market for actually sinking the chunks and also it solves a large part of the unpopular content issue so that's the one that has been addressed in several talks and I will also talk a little bit more detail about it in the last part of my presentation so unpopular content is the one that is not asked from you and therefore swap does not in some device its storage but this lottery to some extent does so in the last part in the concluding part of my presentation I will go through a few use cases two minutes okay that's enough so the first and easiest thing is sending large files so it's nice because the persistency that it requires is on the order of magnitude of days you definitely get feedback on the beginning of the availability because that's the progress bar feature that has been talked about already possibly the end of the availability so you can have some estimates of the costs involved and therefore people at DataFund have already implemented FairDrop for which the kinds of incentives that we have in place seem entirely adequate the next slightly more involved use case is hosting static web content and most importantly dApps, web-based dApps so here for popular content swap is enough we have already discussed that for content that is not yet popular so you have a startup dApp but nobody knows about it so you are exerting marketing efforts to make it popular but before it gets popular it still needs to be available it still needs to be persisted so what can you do well the easiest thing that you can do is you can pay for it by swap you can pretend that it's popular you need to download it yourself and keep paying for it and therefore those who are incentivized by on the swap will host it that's actually the like the step zero of content pinning so this is something that we are going to have very soon when you can store stuff that you want swarmed to host and you just keep re-uploading it and downloading it and pay all the nodes over swap to do that and because of the garbage collection the way the garbage collection queue works it will provide the adequate incentives for swarmed to host it then there can be it can be delegated so you pay someone else to do it for you or eventually there might be smart contracts around which those organized that are doing this for for others so this is again something that is very well within reach and we are hoping to get here this year persistency against censorship so here you want to force to play and lose welcome all games so basically you want it for the publisher to be easy to re-upload and for censors to be difficult to get it off Tony short question we are 11 minutes over the time oh sorry I have one more slide so okay ask me anything storing ethereum state on swarmed we have already discussed this storing encrypted personal data so this is like this is the holy grail of persistency this is the hardest kind of persistency so imagine you have your own encrypted backup nobody will ever download it accept you and probably you will not download it ever so for these we have these things which are a little bit further down the road so this is the functionality that probably swarmed will not have this year but this is where we're shooting for and the very last thing that I would like to just mention and because this is a question that I very often get like how can people how can we delete from swarm and we have this fundamental issue that we have now we can do it the web 2.0 way so what happens if you delete something from facebook it gets marked as deleted well we can do that too so basically what what deleted means that it's no longer accessible from a given UI or given API also it might become undiscoverable from the current Ethereum state now since Ethereum state is is a eternal record it might still be discoverable from a past Ethereum state but also as we have incentives for garbage collection if something is not being paid for the incentive structure that we have in place will mean that swarm will slowly but surely forget it so basically these are the things that we can do about deleting content so that's it thank you so much for your attention and thank you for staying so long, it was a long day not only for you it was also a long night for the preparation and yeah thank you for being here also thank you for all the people who come to us today which are contributors for swarm for a long time thanks for the entanglement again this was at the Madrid Techathon when you wrote the entanglement group of contents if I remember it right in Madrid you mean? so like in general it's awesome to see those innovations happening and it's even more interesting to see how we can bring them the mile to production ok let's get started are there any questions regarding the talk of Toni which are directly related? can you please explain the relationship between group of burn and churn because when I'm hoping in and out from the network I cannot observe the postage coming in and my estimating details ok so postage stamps are transmitted together during the chunk during syncing they're not transmitted during the request so if somebody wants to download content you do not send them the postage stamps because they're not interested they want the content but if syncing is going on with syncing you send all the postage stamps together if nodes are leaving then syncing will gravitate more chunks towards you and you will observe more chunks being signed with that particular postage stamp and because of that your estimate will remain accurate and if you come back then we get the stuff that you didn't have a chance to get when you were offline so you should get eventually so it's the syncing that keeps your estimate accurate if you have a question just speak up it's a rare opportunity to see us all on one place it's a time stamp part of the postage stamp data which is signed so is it possible to tell by looking at the postage stamp how old it is? yes so whenever you pay so imagine so the analogy is that when you go to the post office you buy these huge rolls of stamps and then you can stick them on all your letters so obviously since it happens on the blockchain there will be there will be a clearly marked time when you did that payment and also it does have an expiration so postage stamps have a validity period and that validity period is publicly observable on the blockchain so what you've been talking about now is like the location of the content is based on the logical location of the DHT right? what happens if you have files that are being accessed mostly from a geographical location but they like in the logical they would be uniformly distributed but that isn't necessarily over geographical and it might be super inefficient to post a file on one side of earth when the other side is where it's actually being accessed so this is one caching so this is when caching comes into play so as you have seen the further away the node is from you the more selection you have to whom to connect so given the cadamia rules you're connected to the same number of beers in every bin but the bins that are further away from you you have more opportunities and you can pick those with which you have the shortest bins so with those you will have a better connectivity that kind of follows geography not necessarily physical geography but network geography and because of that since caching is incentivized by swap what will happen if something is continually requested from one area of the network that even though it is stored somewhere where it belongs because of its content address it will get cached along the route on all those nodes which are further away from the content but closer to the nodes that are requesting it and swap actually incentivizes that kind of caching so it isn't the interest of the node so it's not just that we program this into swarm there's actually an incentive to keep it that way so you cannot actually make money by removing that code from your swarm node but even the closer node logically aren't necessarily closer geographically right? no no but they are eventually swarm works this way so even if it's not efficient to store it in the logical location it will definitely restore that way so the geographical location might help in the efficient distribution but for our ICUs they will actually be stored in the other part of the world potentially yeah but specifically for a popular content right? that is regularly requested from a geographical area for that kind of content all the caches in that area will be full with that content right but how do I know that I will get connected to nodes and nodes because with nodes that are a little bit further away from you further away the more choice you have so for example consider those nodes with whom even the first bit of their swarm address is different from yours have the nodes in the network so from those have the nodes in the network you can pick the ones that are closest to you with like lowest latency close latency yes so you pick those of course the truth we told it has to send that it's a little bit theoretical because we don't have this picking algorithm but it's doable and it will be done a short question to just give room for other topics and other questions is there something a reference you can point out where things are written after purchase? yeah so the Swip number 8 I think 8 is the post-pitch nodes yeah is there another Swip which is related to the topic well there's the Cadamlia Swip which I think is it's not yet there well it will be there I think it's already has a number 16 or 18 or something like that so just in general to package this knowledge and to make it accessible we introduced also the Swip process which is close to the EIP process to have discussions and have like say compartmentized focus knowledge gathering I'm position that 8 and also Cadamlia is defined in there it's not yet in traffic mode but it will be soon and this is also a place to ask questions and to follow the development we have to say that the Swip process is primarily aimed at more mature components to talk about stuff that we already can't spec out and with more uncertain topics that we're still researching or unsure about that there's a research form that we maintain that's where discussions about things happen these kinds of things that we haven't even implemented yet it's more like it we discussed the research form you can find the research form by going through our website the Github repo for the Swips yeah obviously Github repo www.github.com www.github.com www.github.com yes wonderful do we have for another topic questions I mean you could ask everything from business over incentives over tool development whatever you have on top of your mind there's a discussion about deleting components where it's like is there any possibility to restrict access to hash I mean if both shares with Alice something yes and publish that fact in blockchain yes so so in general access control so reading access is provided by the ability of the encryption so for those kinds of access control you encrypt data you don't share unencrypted data with a specific set of users so there's a whole subsystem of swarm which deals with access control and if you want to revoke access what you can do so there's a process by which you can change a few things on the swarm like you don't have to re-encrypt the whole content you just change a few things and then when you keep updating that resource or that website then those who have been left out of the access control list they will not get updates like you cannot force them to they can't have a backup of what they have read once so if you have access to something it cannot be taken away from you it's impossible you know you can make a backup copy of anything that you ever had access to right? so that cannot be restricted however if that content changes in the future then those who have been excluded from the access control list will not get the updates they will not be able to decrypt them right so although that's true that's like the past content that was once accessible to an agent who cannot be revoked because once seen they can just store it and access it often legal environments and regulations pretend that this is not the reason and people should have the right to get their stability so I usually have an answer to that which is somewhat hypocritical in a similar way that Mark has deleted answered which is basically for legal legal context you have to pretend that access is so access is actually modelled by the current access control status of 1.5 so actually if you delete someone from the access control list then that's considered that the person has no legal right to access it so you can pretend that things can be unseen and then you simply pretend that to model that with the current access control so if you actually remove someone from the access control that's how you define their right to access something it's kind of a bit of a bit of a time it's like Mark has deleted that so it's too long I kind of have a philosophical question about serialized of all the so I'm not sure about this like technically but there's a lot of database like the blockchains and stuff you know we can do consensus on like adding data of course so we can add all the data and then the delete is kind of like Mark has deleted and then the rest of the map sort of tends it's not there anymore you know you can't delete physical history but the question is like philosophically should we should we build a system that's intended to store anything forever or should we build a system where it's built to be able to delete it? Swarm forgets so swarm is not a system which stores forever and this is a very specific difference to like a blockchains which just depends and it has a huge archive which might need to be accessible and there may be some philosophical argument why you walked here in all of history or maybe it's a technical limitation I'm not sure so that's why it's a philosophical question about technical and I wanted to get a survey like who who wants to build even if it's not swarm maybe something like once more who wants to build something that archives everything forever and who wants to build something that's like more let's keep that short quick show of hands who wants to build something which archives something forever almost so to I would demand it everything of interest exactly on a system everything that someone is interested to pay for whatever cost everything of interest it's a minus but it's a cost and if someone is willing to pay for that cost then it should be I would answer the question also with nature like in nature everything is relevant as requested right so things which are relevant there is a need for it and they stay relevant because there is a need for it and everything what is not moving what is not accessed anymore is that right and when we think about incentive systems and the need that this needs to be economically by interesting well as long as we make a tacit assumption as long as the resource is scarce so if storage is basically so much so much available like air it doesn't have cost of course the whole thing is pointless but as long as it's scarce resource then the cost drives it's the direct interest and it's of interest to link it to another topic which is censorship if I may I raised the hand but I was also hoping that the other questions we asked because I would raise the hand again so I don't think there's a uniform answer it's like a lot of times when we discuss data and then we try to find like this general answer it really depends on the context so it's like who should forget should be forget as society should be forget as individual should the machines forget it's like a lot of asking to this having said this in the domain library of Alexandria that I want to give in the public use one day so I don't think it's like a black and white question and it ends it's very difficult if you actually implement something that is able to forget that you can enforce that okay right now we should forget about this this is a very difficult topic because then if you make such possibility to delete then you're only able to do our percentage as well let's move forward with this question I have recently announced that they will start test nets pretty soon and they plan to launch main nets sometimes in 2020 what do you will make of their technical progress or their approach what we've made about like how what do you think about their progress and their launching and then we're very interesting I would like to forward a question to you Boiteng because you compared a lot swarm and IPFS in the last months and I would love to hear your opinion about this yeah so we looked in both swarm and IPFS and can you say what you said we mean IOB labs IOB labs yeah so we evaluated those two options the thing is that it was some three months ago and Fallcoin still was not really clear if it's going to happen and it has been soon to be released on a test net for the last two years as far as I remember so I would also keep that a little bit with the skepticism that being said I didn't for the last months I didn't really kept up to date with that but Fallcoin however when we looked at that what wasn't clear was how can we actually integrate with Fallcoin because Fallcoin and IPFS is a very complex system that's just very closed to any outside let's say integration or anything like that whereas with Swarm we had clearly this opportunity also I think Fallcoin I mean it's if I'm correct it's a separate blockchain right but I think this is like not necessarily a good design because why would you set up your own blockchain and only for one use case basically whereas Swarm we were seeing in Ethereum and I don't know potentially we can maybe use Fallcoin it's also like if you compare Apple's with like other fruits like the architecture is totally different from both systems and also the mention or say the opportunities maybe we can move to a less competitive questions or do you like to add something? sorry I understand Fallcoin is this kind of mining mining for so it's mining basically it gives positive rewards for what you're showing that you will start by which means that it basically has the same program to tragedy of common problem so basically it might be right 99% of the time exactly what it was 1% or when it's when you lose your backup it's when you when you start that's why we from the start kind of emphasis that most probably we need the solutions for ensuring advice with a potential cumulative threat to a particular individual negative incentives first shorty with Fallcoin to earn money for storing your files but nobody loses money if they lose your files and we need both sides if you want to have real guarantee because some other file pays more in the future they can just delete yours there's no penalty for deleting except an opportunity and that's really what we're targeting with our storage incentive let's move on to the next question some related question but not talking about competition Danny in his talk you mentioned that the incentivation like a priority is to provide a reciprocate service does it mean that then Swan wants to create some kind of collaborative storage system like more than other than having people that want to put just a storage note and people that don't want to have any storage note just pay it Swarm all three of these options are available so I think that the greatest inspiration for Swarm both technical and philosophically has been Victor so I think that Victor is a very very fundamental development on the internet it's a real innovation that happened in the early 2000s and the way they have this stick for that protocol that you can only pay by reciprocal service and that is what ensures the high output of Victor and downloads and basically we took a page from their experience that it's really nice that basically in order to download you have to open up your upstream bandwidth which you don't need anyway while you're downloading and you set a little bit of storage and you can do the same in Swarm that the fact that you have some available resources it will be easier for you to just put those resources to use for others in the same network but if you don't have those resources you can still buy them so it works both ways and if you're producer and you don't want to consume you just have a big data center the FAT pipe you can put it to use and you will learn internet without spending any and that's how Swarm components are like for some extrovert that's exactly where where blockchain can turn out to be like really, really a fundamental change in making sure that these systems are economically sustainable and there are a lot of participants that are not selling consumers themselves so it's like you don't have to fulfill tools at the same time for the system to print it allows for to be separated you can have an operator which allows for download and can be a consumer without ever offering anything and similarly you're not forced to pick us for example if you have a cell phone you want Victor and movies with your cell phone because it will drain your battery it will drain your account and so on so then you just want to pay and download and you have that option with Swarm we need to unfortunately clean out the room and the conference time is ending so give a warm round of applause to this one team the most important thing for this project is cooperation and having the red partnerships and bring it to usage so thank you all for taking part in that journey thanks for that