 The title of my talk is Scalable Secure Scuttlebutt and my name's Dominic Tarr, so what is Scuttlebutt? So Scuttlebutt is a nautical term for gossip, a butt is a barrel that water is stored in and Scuttle is to open the barrel, so it's to make a hole in it. So this way Scuttle is also to sink a ship because you open the ship. So this means gossip for pirates and stuff like this. So I got the term from a part of the Amazon paper called Flow Control for Anti-Entropy Projects. It's like for anti-entropy protocols, so it's eventually consistent gossip protocol that's used within a trusted system. And then I made Secure Scuttlebutt which took the same idea and then added security. So the idea of the pair is its public key. The data model is just append-only logs where each pair appends to it, has its own personal log that's signed. Pairs can relay other messages but they can't insert, modify or reorder messages because the signatures would then be invalid. So this gives you a really simple data model that can use to re-represent a bunch of stuff but it's completely free to run, like signatures, you can make thousands of signatures a second on an ordinary laptop, the current implementation can receive 2,000 messages a second and validate them like it's very, it avoids all kinds of bottlenecks that you do if you have a consensus layer. So this, then there's a few, it maps quite closely to social media kind of models like Twitter and whatnot because you follow the feeds that you want to replicate and this solves a whole bunch of neat problems so this solves a civil attack because the human, it bubbles up to the human layer and they just unfollow the nodes they don't want to talk to or don't want to hear from. It also provides discovery so writing a crappy pair to pair Google would be really, really hard but writing a crappy pair to pair social network actually can still be really awesome if there are awesome people on it and you can still discover new things so it's a way of discovering things in a decentralized way. And then because it replicates messages in order it's easy to think about building a database on top of this because the things happen, you receive things in the same order that they happened so that was secure Scuttlebutt and then my recent work has been to make it much, much more scalable so scalable secure Scuttlebutt is the same thing again but optimized for bandwidth and latency so the big deal here is that the overhead, the networking overhead is proportional to the feeds updated and I'll explain why that's important so part one is optimizing the network typology. So the simplest, most naive design would be just to connect to everyone and I had a really cool animation of this that you can't see but perhaps I'll just like, I don't know, can see you want to see something kind of happening? So this of course is like inherently centralized, it goes out from one point but it's actually really efficient because each message is transmitted exactly once to every node so the amount of people that receive messages is exactly the same as the amount of messages that need to be, that are sent and but the problem is this means that actually there's one node that does all the work and all the other nodes do nearly no work and then you know about, you know about why centralization is bad. So another solution is you just have a random network, just pairs, every pair connects to a couple of other random nodes and it's really easy to make a fully connected network like this, like if you connect to more than two other peers, the chance of being completely connected is like, as the network gets bigger it increases so once you're past two it's like the chance of being fully connected is like pretty much certain and but the problem is that it's a trade-off between bandwidth and latency depending on how many extra connections you have. So I have another animation here, so it's like, looks like a noisy mess and then you might see some like red, all the red messages that are flying around those are the extra redundant messages and if you have like, I think this one has just like five connections or something, I've got a little table here, yeah so once you have like five connections per things then you're getting like 10 times extra redundant messages that you don't need to send, well you kind of need to send to make sure they arrive but you've sent 10 times as much as you needed to but it's super robust. The simple, and it's super simple to implement as well, the role is just send new messages to everyone that didn't send it to you, which is also being don't resend messages you already know, so just when you send a message, check if you already know about this message and if not send it to everyone that you're connected to and then that will flood the whole network, pretty efficient, like very robustly, like if some nodes get taken out it'll just be routed around otherwise. So there's only like a couple of cases we need to hand think of here is like one pair sends it to another pair or the second pair sends it to the first pair and the other thing that can happen is they both send it to each other where there are two paths for that message to get to Alice and to Bob and they both received it and within the time that it takes to send one message, that's the only case where you get both pairs passing the message where you get a redundant connection and the bandwidth complexity is like it's proportional to the messages times the spanning connections plus twice the redundant connections because each redundant connection gets two messages sent on it so the redundant connections are like much more expensive than the minimal set of spanning connections and once you've got 10, when you've got 10 connections per pair you have like 95% redundant messages so that means if we can get rid of the redundant messages we can just transmit one message per pair which is optimal so the better solution is a spanning tree network so spanning tree is like we get a graph and we just only have the connections we need to reach all of the nodes so we can take the flooding gossip and we can use that to build a spanning network in a decentralized way by just whenever you receive a duplicate message you just deactivate that connection. I didn't say disconnect and I'll come back to that in a bit so you just deactivate that so now you're sending like just the number of messages. The only problem is that trees are really fragile so because now you've basically just added a whole bunch of single, you've made every single node into a central point of failure so there's one path between any two random nodes and if any of the nodes in that thing fail then those messages won't get through so the yeah so it's bandwidth optimal but it's not robust but then there's this paper I discovered a while ago called Epidemic Broadcast Trees and this combines the best parts of both flooding gossip and spanning trees so how it works basically is you have the connections are in two modes there's eager and there's lazy and so the redundant connections get put into lazy mode so in eager mode when you receive a message you transmit it immediately and when a connection is in lazy mode when you receive a message you just send a really short note that you know about this new message now so all of the main all of the spanning connections get the whole message sent across it so that's that lets the message propagate as fast as it can and then the message the redundant connections they just have a very short message being sent across so it might only be so in the current implementation of secure Scuttlebutt and the average message size is like 700 bytes and it could be up to 8 kilobytes but the note would only be like 80 bytes I think and that could be like binary optimised down to probably less than 10 so the redundant connections still exist but you just send a waylier starter on them and then they also the robustness still exists because when a message when so when you receive a message for the first time you like rebroadcast and so on but if you receive a note about a message that you haven't that you don't know about yet then you send a you send a request back to that back to that back to that peer asking them to turn back into eager mode and then they'll send you the whole message and now the network has repaired so as the best of both worlds yeah and then there's only a small amount of state that you need to keep track of per pair for per feed you just need the ID the local sequence number the remote sequence number the local request the remote request and the local and remote modes so that's only that's you know we're under like depending on how long the ID string is we're talking about like just like tens of bytes that is okay so then the other so that's up to that's all about optimizing the network so the second part is optimizing the handshakes so secure Scuttlebutt had this so original Scuttlebutt had this thing called a vector clock so vector clock is just like a list of all of the things so per so you have a whole bunch of a whole bunch of friends and when two friends bump into each other they say oh when did you last hear from Bob I heard from Bob on Tuesday when did you last her from Alice I heard from Alice on Monday and if I say that I saw Alice on Monday and you saw her on Sunday then I can then I know I just need to send you the messages from Monday well from after Sunday so it means we just exchange this one thing and then I just know immediately what you don't have and by arranging the messages in a linear order order it makes that comparison like completely simple if you have like more complicated orderings and like if you have some kind of more complicated structure it becomes much much more difficult to do a remote set comparison but if you may have it linear order it's like dead simple so just like compromise towards the simplest structure that's easier to be optimal about so that but that has a problem and that if you're having like the original paper that describes this said that it probably you know it was designed for used for one cluster within a data center so you'd have like a DynamoDB instance that they said might have like 300 nodes which is maybe a lot for one you know cluster in a data in a data center but it's not a lot for a p2p network we want to have thousands and thousands of nodes so if you have thousands of thousands of nodes then just sending the vector clock like really adds up because you might it might just be like a public key and a sequence number so that might be like 80 bytes but then if you're sending this for like 3,000 people that is like into like hundreds of kilobytes so that means like each time you connect you send you might send like half a megabyte of data both ways and if you can reconnect and reconnect you could be sending like tens of megabytes a day just to keep the network alive like even if no messages are being sent so that's not really that's not very good at all but one of the other Secure Scatterbug contributors Sal Charles Linhur came up with this this idea to skip some requests so what you do is you just remember the vector clock that the other person sent you and then compare your new vector clock to the stored one for that peer and then just don't mention all of the things that haven't changed so if I if we meet on Sunday and we exchange news about Carol and then we meet each other again on Wednesday and neither of us have talked to Carol since then then we just don't mention Carol and because we know that if so if the other person did they would mention her so that means that you can just leave out most of the vector clock and you just don't mention most of the things that have changed that all the things that you think probably haven't changed and then if you find out if some if the other person does mention it then you just reply with like your partial vector clock of the things that that didn't didn't change yeah so I had yeah as you know this is this is this is really important because when you have like up people using any pretty much any kind of application you power laws are really typical so you're really likely to have some users that are like really active you know use every day or posting every you know constantly you'll have like some users that are like daily a fair amount of like moderate users and then like heaps and heaps of like people that only use it occasionally or even signed up and if you used it again and you don't want to be wasting bandwidth on people that haven't done anything in a while so this straight this simple technique loves you to completely cut out all of that stuff and that basically gives you in the end so basic basics about the message over here per connection was like the messages plus the feeds and if no messages have changed the messages might be zero but feeds could be thousands but secure Scuttlebot is messages plus updated feeds and updated feeds could be much much smaller number usually it would be like if you reconnect frequent like frequently number could be like will be close to zero if you reconnect months later it will just be you might get individual users might have posted hundreds of messages but the overhead is just one for that updated feed which makes reconnections very cheap the only time you have to pay the whole send all of the feeds is when you have a when you when the first time the two pairs connect yes that's very good and the latency complexity is just you can you're replicating in either one or two round trips so one round trip if you've said if you sent the whole clock and they then send the messages but if they send a part if they saw or if they don't or if the messages that you haven't mentioned haven't updated then it's one round trip and if you send a message if you send a partial vector if they send a partial vector clock and then you have to send one back then it's two round trips but my so some background since now basically improvising so like I became interested in this because I was living on a sailboat in New Zealand and I was used to like like my normal thing was not having very good incident at the time at the first when I first became interested in this I didn't actually have any internet in my boat and I was like why can't that be like why does that mean that I can't use my computer like you should be able to use your computer also grew up on a farm in New Zealand where the best internet we have we had was like a satellite internet that had like a one-second round trip which is like you know it shouldn't really take that line that long that long for light to travel but it has like I don't know lots of switching packets and that sort of stuff so I was really determined to like latency is really a killer like if you and if you live in like a fancy place that has fiber and that sort of stuff then latency you might not better notice it in a couple of extra round trips might not be that bad but when latency gets bad it gets really bad and if you have a design with that sends a lot of packets back and forth then it's going to be really really painful to use in that situation so it's determined to have like a constant number of round trips so one or two round trips I think is like about how much you can I can tolerate for a replication protocol logarithmic round trips not not acceptable that's going to be really painful in in many cases so the part three is I'm glad they've arrived comparison to IPFS I was sure people would ask about this so secure scuttlebut basically optimizes for the worst case and secure in IPFS so secure scuttlebut is designed for like long append only chains and this would be like the pathological case and IPFS so they're trying to do completely completely opposite things but similar and that they both replicate data without global consistency so they're kind of like an infrastructural role for if you're building like other peer-to-peer apps and so on so neither again neither have any like cryptocurrency or thing like directly built into it but if you're building stuff like that you could move your data around using a combination of these things secure scuttlebut streams updates in order per feed and IPFS has appointed to the latest message and works back and if you want it to have it relatively efficient you'd have some kind of tree or some sort of directory you find the things or just not have very deep structures have relatively shallow ones so IPFS is probably better for like arbitrary partial data sets like wikis and sort of static sites such things all like directories of data and that sort of stuff whereas SSB is designed specifically for like social applications or social applications with something strapped on the side so secure scuttlebut is basically graph database and IPFS is file system yep okay that took less long because I didn't get to show you the animation stuff but I'm that that's it okay if you'd like to if you'd like to learn more you can go to a scuttlebut.nz that's two T's and get hub.com slash SSBC slash patchwork there's like a UI for like a real real working application that you can use today and if you go on that you'll meet lots of load love lots of load lots of lovely people that will tell you all about how things work yeah sorry the slides didn't work got still got eight minutes any questions or heckling or anything oh scuttlebut.nz yep cool thank you very much