 Okay. Thank you Hi, my name is Sam Macbeth. I work for a company called clicks. We build a web search engine and a browser fork of Firefox I Usually work on privacy, but I'm going to talk about something else today And that is that protocol in the browser. So this is something we've been experimenting with on the side What can the browser bring to empower users and we're quite inspired by the beaker browser Which is the electron-based browser that put the dot protocol Inside it So this is kind of a story of the process of trying to get this idea. So I put a d-web protocol inside Firefox So firstly, what why do you actually want to put d-web protocols in the browser? So why is d-web good for the web? I'm going to give a few examples there are many but a few that are relevant for that So the first kind of biggest thing is that you get rid of servers. So in the case of that I can just say type in the command line that shares some folder And then it's going to give me a link and I can give that to someone else and they can load my web page This is really powerful and if we integrate that in the browser You could just have a button to create a website and you don't need to have any infrastructure any servers No man in the middle. You have a website Second it's offline by default, which is really handy. So If I've loaded a website, I can reload it without internet This is super useful and now the web has lots of mechanisms to try to Achieve this So bring it back on top of the normal web stack with service workers and all this But this is really complicated. The d-web just does this by default It's also super transparent. So we're familiar with With view source on the web you can look at all the front-end code But if there's no servers There is no back-end code that you usually can't see it's actually all has to be Inside the front-end source code so you can view source of the server as well It's also self-archiving. So if you take a data example, and there's this API called data archive and I Can open some address and I can just look at the history of edits to that page And I so I can request a specific version so you can have proper Permalinks which are permanent because they to a fixed version that we're not will not change So quick intro on what that is So that is a protocol that was developed a few years ago. Firstly with the intention Aiding data sharing in academia between research groups And as I mentioned that the beaker browser kind of gave a bit more attention for the web use case with the With their browser a couple of years ago We can describe it as a specific file space space single writer data synchronization protocol So there's a CLI you can install with MPM And then just type that shares and folder This gives you an address on the network and then on any other Computer in the world I can just write this and it's going to pull the data directly off me Underneath there's two main components of the dat stack there's Data structure which is in the hyperdrive module and peer discovery in discovery swarm So what is hyperdrive? This is a file like abstraction on top of append only logs So underneath you have these append only logs which gives you the Verification of that only one person can write this data and the file like abstraction is nice is a good high-level way of Organizing the data So the way this works is I can just so the top snippet. I just create a drive on top of some storage And then I can use Simple file commands like write file But this drive also has a key Which is this address of the data and then this magic method called replicate So I call this method. It's going to give me a stream Now what I can do is on a different device. I can create a new hyperdrive and Use this key that the first one that was generated in the first type of drive And now what I just need to do is get This other stream so this could be Over a socket or any other kind of web piping data between two devices And I pipe these two replication streams together and now my data is replicated So I can now call and make API calls on my hyperdrive and retrieve this data from the other peer Discovery swarm is For connecting to peers for a topic So I have a swarm and I just give some topic name It doesn't matter what and then it's just going to give me events with connections to other peers very simple Behind the scenes this works on a few different things like DHT and MDNs discovery So yeah, so we just put these two things together Yeah, so we just put these things together So what the hyperdrive has is what's called a discovery key, which is a key derived from the actual address key This means that your actual address is hidden from the network So I join and then I will get a connection from another peer and then I do this replication piping Okay, so moving to how to implement that in Firefox So what we're aiming for is basically that you can have in the URL bar just put an address in That is a data address and the browser is going to load it like a web page So there's kind of two requirements that come out of this First we want to actually run the dot network code in the browser. This at the moment is all implemented in in Node.js And secondly want to make the browser understand that Data is a protocol that should be handled by this bit of code that we're running With the dot network So firstly you think what where do I implement something like that in a in a browser like Firefox? So first option might say native code. This is obviously where the HTTP HTTPS implementations are at the moment Which would be possible, but there's a few challenges there. So firstly the Native code in Firefox is very complicated is extremely high velocity project And we have to interface with Node.js anyway, which Would be tricky to link up with the existing C++. So at least for the first step for prototyping this this is Not a good approach So secondly we could do a web extension. So web extensions are standardized APIs for extending browser functionality This is this is the first attempt project is called DatFox was dat in a web extension And the reason I think this is possible is because there's this API in web extension where you can declare a protocol handler and they also Mozilla white listed these dweb protocols so you could have a protocol called dat and you can register it And this is how you declare it in your manifest file But you can see the problem here. So the last thing we have here is a URI template This is based specifying Where to go if I if I get a doubt URL Where to redirect it to? So this is just a protocol switch. It's not a protocol handler And Because I can only switch to HTTP and I cannot run a HTTP server inside the browser extension I Can't actually serve this request content from the browser So even if I could do the first requirement of running that in the extension. I cannot link it up with the protocol Despite that we have there is an extension you can install for Firefox But what it does is it requires you to install a secondary Executable and Node.js process which will then do the dat part. So we read out we do some tricks with Proxying so that when this Initial redirect happens the extension picks up and roots it via this helper process to load data from the dat network And the image doesn't work But this so this Works okay But a few issues here so the dat protocol is not displayed in the URL by it says HTTP It's also a bit cumbersome to install because you have to download this extra process and Some manifests have to be set to link these things together Also, there's some quirks with this protocol implementation because it's not designed for this use case So only works on top-level loads if I have a absolute URL for an image or something it's not going to load properly So is there someone that something else we can do to Get a better integration well, actually there's so in Firefox there's this these so-called experimental API's which are the way you implement a new web extension API and About a year and a half ago. There's some resilience you came up with this project called LibD web and These this were were experimental API implementations designed to aid Or help people build LibD web build D web applications or D web extensions and This has implementations for protocol handlers TCP sockets UDP sockets and so on So what this looks like for example for protocol handler, I can just write this in my extension code Register protocol and directly have a handler there. So this is exactly what we need Likewise, I can just create TCP sockets and send data across So now we have all of the kind of fundamental requirements to do what we want to do which is run everything inside the extension But there's some kind of still some dots to be joined Because so in the discovery swarm module somewhere deep in the dependency tree, we just have Some modular that does require net or require dgram. So They just expect these no JS APIs to be there So we have to make that link to kind of give an alternative implementation so it can use that instead So this is a library called LibD webify that I put together but This was helped a lot by contributions in the community because people came up with a shim implementation of net and D gram on top of these LibD web APIs And we can put that all together into a browser five plug-in so what browser five usually does is Sees these core node libraries and replaces them with shims and things like net would just have an empty shim What this does is it instead of replacing with the empty shim uses these new ones? to To give this extra functionality on top of these APIs So yeah, this is extension called datwebexed which Does this so it loads? dat web pages and it has the protocol correct and is loading everything internally in the browser. So now that instead of running on V8 inside node. Yes, it's running on spider monkey inside Firefox So yeah, so now we have native like protocol support And we run the full stack There's also some other nice benefits that actually this data is in the user profile directory So we're not just storing in some random other place keeps people's data separate We also found that so the new Android web view implementation based on gecko called gecko view you can also run this in that and then you have support on mobile But there's one main disadvantage is that for these experimental API is you need privileged context So you need to be signed specially to say you're allowed to use APIs This means that this cannot be offered on the add-on store has to be bundled with a browser But luckily as I mentioned at the start I work for a company who does build a browser So we're currently offering this in beta, but soon to be on release of the clicks browser And also an upcoming new version of the Android browser would have this will have this in as well So I do a quick demo. So if you've noticed that I'm putting these slides in the browser in on a dat page if you have a that browser or Just want to I'll just have the Cli you can just download this and see what's happening at the same time But yes, so this is the web so I can Inspect and I can run commands in the page. So I mentioned at the top this dat archive API that was created in beaker browser as a way of allowing sites to Modify themselves essentially using dat front dat commands so We can oddly that can also do that here is that big enough So I can create this dat archive for this current page that I'm on and I can get some info about the current site that I'm loading You see there's some some metadata about the site so it has this address So this is the actual address of the dat site and on This is the amount of data for the whole thing And then what's important is this thing? So this is is owner is true. This means that on this device I have the private key for this dat archive, which means I can write it So as I can write it I can just call write file And write a new filing So now if I go to new page well so This file I just wrote But I can also also delete it obviously and now for our fresh It's gone and if you were thinking you would be good to get these changes in real time from my device And Yeah, and also we have history of course So you can see a list all of all the edits I've made to this page So you see I tested out the demo a few times beforehand But then I can show that we have this idea of version URLs as well So at this version this file doesn't exist anymore, but it existed in a previous version So I can just put the version number in the URL Which version was it? Okay, so at version 38 This file existed. So now this is a permanent link to this version of this file, which will never change All right. So what are the next steps? So we Got reasonable support so far But there are a few challenges come up so The first major thing is you want to try to upstream libd web Because outside firefox call but the implementation is the experimental API is use internal firefox API is which are constantly changing So this is quite a maintenance burden to keep having to fix this with Every other firefox version So take one example this service discovery implementation in libd web the Internal classic used was just removed from firefox a couple of versions ago Because there was no test that says this is we need to keep this so they thought I'll just clean out this old stuff So now as a ticket to bring it back, but this is the kind of thing we can avoid by upstreaming this into the Into a Mozilla central so that they will have tests for this and these will these APIs will keep working There's also some improvements to do in the protocol handler integration. So Because the browser is not so This implementation is is have a bit of a hack and It mostly works the most most parts of HTML and JavaScript, but it's the browser doesn't understand it necessarily as a HTTP page and certain elements break and certain storages break So that's something needs to be fixed Also, we don't have control over other parts of the experience. So this info bar So the one next to the URL which is meant to show you some information about the page We're not yet able to add information there to explain what this is what this protocol is Also so far the Dart network has not been tested at scale This is something we'd like to do to see how this discovery mechanism scales up if you get Several orders of magnitude more users on a dat site And also to test different network environments. So lots of users will be over behind different types of nuts and Whether they can always escape that and make connections to other peers There's also that version too. So a new version of the protocol is is Coming up soon and there's some extra challenges and getting that to work In this context largely because of new crypto that it's trying to use which is not doesn't have an implementation for the web yet And yeah, then we want to build more things on that. So more applications that users can use to publish sites or Upload things and send them to their friends Obviously goes about saying that all of this stuff is open source So these are just a load of links to different projects here. So that project stuff hyperdrive Lib dweb and These extension implementations at the bottom It didn't all fit on the slides. There's another one, which is the browser, but yeah, so thank you any questions We have five minutes for questions. So Almost five Hello, thank you for your could talk So currently beaker browser experience many extensions to the unworked garden project with many more features Sorry, I can't quite the beaker browser gets many more extensions Regarding the on world garden project is this will you try to implement this as well? Yeah, we'll have to see I'm kind of waiting for when the finalized version of that comes out Ideally this would work in user land. So we just provide the base support but we see Also considering exposing kind of an API so other extensions can also talk to this extension to do kind of peer-to-peer stuff as well So expose some of the lower level APIs for building applications for that So I wanted to ask what's the main difference between that and IPFS The main difference. Yeah, any difference because I am I don't see really What's yet the difference basically? okay, so One main difference with that is that It's scoped to a particular address. So I'd make one hyperdrive. I have this address And this is a separate individual swarm. So I'll only ever talk to people For that address IPFS is going for more global network So you put any file in IPFS and you talk to everyone in the whole network to tell people that this file is here That is also not content addressed So you just have a key pairs and address and then you put data into that folder IPFS content addresses everything so an IPFS when you put a file in the Address for that file is based on what's in the file and this is then immutable so that address is always that file But that has this mutability on top because it's not tied to content addressing So I missed one minute of your of your talk. So sorry if you've covered it in that one minute But my question is I know there's a JavaScript implementation of that. Yes. Yes. Does it and but it's marked as unsupported How how unsupported is it the the old one? So there's a few different approaches and that that ecosystem to the web So Hyperdrive itself is agnostic of how you find and talk to other peers and this has always been browser-ifiable So you can always put that in browser because the Networking the discovery swarm part is tied to being able to open TCP sockets Obviously that doesn't work in the browser. So Then there's a few different ways you can get around this The dat.js was using webRTC to talk to but then it's in The problem with that is it's an incompatible network So you have like one swarm which is that in the browser people and then another swarm which is that node people and they couldn't talk to each Other so then there's a few other approaches. You can for example open a web socket to a gateway and Synchronize data with them and they can bridge to the rest of the network there's also other ones that act as just a Other types of gateway which just act as a discovery gateway. So You just stream to this and then we'll find other peers and then proxy those streams So actually that that gateway doesn't know anything about what you're looking at What we're doing now in This is Doing a combined approach so we because we can do the normal networking But we can put webRTC alongside it to help with discovery because sometimes it doesn't work over sockets Okay. Thank you very much. We do not have any more time for questions. Thank you again for for giving this talk