 Hey folks, this is another research notebook video. I haven't done one of these in years and years But I have a point I mean because most of my videos are happening over on the t2 tile channel now But I have some stuff that I've been working through this robust local synchronization idea that has some kind of interesting stuff I thought and I thought I would take a little bit of extra time and try to present it research notebook style It's still probably be kind of long and kind of confusing But hopefully it will be of value and the very least it will just be helping to document the arc of my own thinking as well as Project in design development. All right, so a Little background as quickly as possible, you know so the idea of synchronization is everything worked together and in digital computing It was the heart and soul of the digital computer revolution that the original CPU and RAM the whole model there was a clock and The clock had to be distributed to all the circuit elements that were involved and it had to reach them all at pretty much the same time So that when each individual circuit element decided what next state it should adopt based on the things It's connected to the things that it was connected to would all be showing their results from the previous state and then Chunk the clock comes and everybody updates their state and things all go crazy all over the world But then it settles down and then the next clock edge comes and so forth So globally clocked architecture has been the backbone that has powered the whole computer revolution basically in terms of the individual machine and When you have that the goal is to get this single deterministic execution path the idea being that you know once you know the previous the current states of all the things that are input to you and Then you get the clock edge you produce a guaranteed output that you know I am the and of my inputs or the or of my inputs or the sum of my inputs, whatever it is And as a result because each individual step happens on the absolute beat of the clock boom boom boom And each each individual step is absolutely predicted by the previous step The pro the prediction of the entire program overall is also absolutely predicted deterministic execution now The bigger the number of circuit elements that you've got the harder It is to actually get the the clock out to all of them all at once I mean even if you have a you know a clock in the town square and in the in the church going ding ding ding It takes time for sound to spread from that from the bell over the whole village And so the bigger the village is the less synchronization you've got and as chips and circuits and computers have gotten bigger and bigger and bigger They've actually gotten smaller smaller smaller But as they've gotten smaller smaller they have more villagers on the more individual elements that need to hear about the clock Distributing the clock signal to everybody Has become more and more of a problem and more and takes up more and more real estate more and more energy and so forth so much So that most of the the computers smartphones and so forth that you deal with these days Actually are not no longer globally single globally clocked instead. They're called globally Asynchronous locally synchronous and and what it means is they have a bunch of you know sub villages They have a bunch of small clock domains that there's a clock for the CPU And there's a clock for the memory and there's a clock for the communication buses running between them But the clocks those clocks are not Coordinated as some higher master level by anybody so that instead now when you have one clock domain meeting another clock Domain you have to have a synchronization Border you have to have some additional mechanism to deal with the fact that this one might be halfway through a tick Well, this one is now firing and you know if you just let this side look at the signals on that side They would get chaos So there's all this extra stuff that you build in between the clock domains to add additional Synchronization but still in the spirit of single deterministic execution the machines that we built today using gals globally essential of cylinders Are designed so that even though the different clock domains may technically be running at different clocks And so this guy might get a couple extra clocks in before this one gets a clock and it might be different different times We execute the program we design the whole thing so that you still get a deterministic result hopefully Once the program is done now I hated all of that I mean, you know, I started out and computing in the 70s and it was great fun And I learned tons and worked on it for decades, but eventually this whole Control freak top down everybody do what I say or else do nothing I got tired of it It really, you know, it seems number one It was kind of played out and number two was a security nightmare and number three It just wasn't very much fun anymore and it seemed like there was many more Aspects of computation that really weren't covered by the top-down king of the universe I'm the best just do what I say sort of attitude and so the whole best effort Computing thing that I've been pushing on for the last decade or two as well as the robust first stuff that leads to this videos Happening right now is all about saying let's give up on the deterministic result We'll give up on the deterministic execution path So now hardware and software both are gonna have to be able to deal with failures and deal with stuff coming and going and disappearing And still do somehow make some useful progress So the suggestion is you know the past was single globally determinist Globally clocked architecture now we have multiple clocks, but we're simulating a Deterministic result in the future. We're gonna give up and we're gonna need a robust local sink And that's what I want to talk about today Okay, so if we're giving up on a global clock, what do we replace it with well one candidate to replace it is this thing called an Asynchronous cellular automata and that's what the movable feast that I talked about in the demon horde sword video and the whole TZ tile project is all about Is you know a spatially array a spatially distributed array of very simple processing elements that do not have a global clock They do not go kuchunk kuchunk kuchunk all at once instead They fire off sort of at random or we don't even know how they fire off But we do a little work so that when this guy is looking at his neighbors There's nobody else also looking at those neighbors now if we start with something like that and how are we gonna compute with it? It's a whole different picture and that's what the T2 tile project has been about But there was this one particular idea saying you know well I have a trick. I have an idea I'm smart saying how I could take this asynchronous Events flying all over the place and synchronize it so that we can get rid of the asynchronousness and now we can get back to a kuchunk Kuchunk kuchunk everybody can look at what their neighbors used to be and decide what they want to be now at the same time in quotes and This is an idea that goes back to Nakamura in 1974 has been reinvented a couple of times since I am just going to call it the Ntn algorithm Nakamura Tuffoli, Nihana And Let it go there and I'm going to talk about a slightly simplified version of it and the basic idea the most important part about it is there's this sort Of three-state loop which I'm going to call red green blues typically what it is and we use this idea to say How do we synchronize an asynchronous CA is we have each individual little site is going to look at its neighbors? So each site is going to be a color red green or blue at the moment And it looks to the neighbors and lets the neighbors catch up with it And so here's the picture If we're in state red We look at our neighbors and if we have no blue neighbors that means everybody's caught up We can move on to green and if we're in green and we have no we have no red neighbors down here We can move on to blue if we're in blue and we have no green neighbors. We can move on to red Okay, so if I'm sitting in red say and I have a blue neighbor that means someone hasn't caught up And I cannot make that whoops that way I cannot make the transition. Okay, so if everybody obeys this rule Then we we can get this cycling going. So let's take a look at an example here All right Okay, so here we've got a grid full of stuff I am gonna flood the world with This stuff So here we are now Suppose so they're all blue at the moment and so so the rule for blue said The rule for blue says if there's no green neighbors I can move on well So this blue fellow here he can move on so he becomes red now red can't move on Until there's no blue neighbors and in fact he's got all blue neighbors So even if we click on him again nothing happens, but if we click on the neighbors they have no They have they have no green neighbors so they can make the blue to red Transition and and you see how it all spreads out and now these guys in here like this guy here He's got no blue neighbors So he can make the transition from red to green and so on and we can spread this out a little bit more Here and now this guy in the middle He's got no red neighbors and he can make the transition back to blue So you see how it works and if we just let it rip It's kind of trippy But you see what it's doing. There's We have Red chasing blue green chasing red blue chasing green and it's kind of zooms in on Random little spots that just because of chance happen to be taking longer And so that's the guy that we have to wait for But then they catch up and we move on it's a nice trick. So What can we do with this? So this is the principle of always wait for your neighbors now The first thing to think about this is you know when we talked about global clocking. It's not like everybody here went from green To blue at the same time far from it I mean there so you know this blue this blue this blue these various blues are not even Necessarily the same step in the computation Unless you actually traced it all around like sort of you know contour lines on a map an elevation map all right, so The ntn algorithm does more than just the wait for the neighbor cycling that's the backbone of it But if we're going to be able to update Synchronously in effect synchronously we have to do more and so the idea is Instead of just having my internal state that other people can look at I have two copies of it I have my internal information That nobody but me can see and then I have my published information that's out in the world and we deal with it on different transitions So when we make the red to green transition We update our internal state But we don't tell anybody about it. No, we make the green to blue Transition we tell everybody about it. Now. Why is that a safe? Well, because the cool idea is when we make the green To blue transition we know there are no red neighbors So that means we haven't got anybody who's in the process of computing their internal state At the time we are updating hours and then finally on the blue to red state We just do nothing we wait that's essentially just to keep things separated so that we can tell which way is forward And that's the trick so it takes twice as much state because we have to remember both our internals and the stuff that we've published When we update our internals on the red to green transition we look at everybody else's published state So even if we see it at a different time than somebody else We know that nobody has updated based on our stuff because that will only happen on a green to blue transition. See very tricky Okay, and now once we've done that we can make a whole new set of rules a sort of virtual universe on top of the physical universe Taking advantage of the synchronous rules and we can do that here as well So let's flood the universe again Uh, okay, but now let's change the display One here on the back and add a number two in the middle. Okay, so now of course Dave and a snow storm So now you see the background is all blue and that's because that's the they're all in state blue in terms of the red Green blue cycling but the front is all white because they're all in the synchronous universe. They are all White ones. So now if I make some A little pattern like that say, okay And if we let it go it's kind of cool the way it works So you can still see hopefully the red green blue cycling happen in the background But now on top of this there's this whole black and white thing going on And so the rules that are happening in the synchronous universe here are the conways game of life Uh, certainly the most famous pretty sure the absolute most famous cellular automata set of rules And this particular pattern is called the glider It's the most famous pattern in the game of life And it's you know, five little black states that essentially toggles back and forth between an l shape and a zigzag shape Made out of four and then one thing in the back that kind of zooms back and forth driving the whole thing forward And you know and this thing is working fine and you know pretty soon. It's actually going to get up to the top of the universe here and It's going to turn into a block because that's what gliders tend to do when they run into the edge of the universe and there it did There it is turned into a block And so now the synchronous universe is still updating the asynchronous universe is still updating but nothing's actually happening Uh, uh, so that's really cool. And that was the basis of the berkeley prof who was thinking, you know Hey, we can just continue to do synchronous cellular automata like conways game of life and use the ntn trick underneath to Just deal with the asynchronous business and not worry about it. Uh, uh, but That's fine as far as it goes Uh, uh, but now the question comes up if the point of the making an architectural decision That we're going to get rid of asynchronous ca's entirely and use the ntn trick to present a synchronous universe So that programmers only have to deal with synchronous stuff Then we have to say, okay. Well, how big is this synchronized universe going to be and in the cellular Cellular automata. It just keeps getting bigger and bigger and bigger So what if something goes wrong? We haven't talked about it But sooner or later if we make the thing bigger and bigger bigger something will go wrong So let's hit it with an x-ray. We got a little button to simulate that And what happened? Uh, uh, well, we lost some of our some of our universe just got blown out entirely Those aren't black states. Those are actually holes in the universe. We'd like some way to fix those, but we don't have it I mean, let's hit it again And now look at this The entire universe has come to a halt Why is that? Well Let's take a look Um, so here's our rules again But what if you happen to come up with a state where the red guy if he's looking in a little three by three box If that's what he thinks neighbors are Um, he says can red go to green he has no blue neighbors. He's got a blue neighbor So he can't go to green and green needs no red neighbors But he's got a red neighbor so he can't go to blue and symmetrically blue can't go to red unless he has no green neighbors Each one has a neighbor of the other two colors mutually in the same neighborhood So none of these three can advance So it's a deadlock Now see one of the big tricks was when I started this up I flooded the whole universe with blue. I mean it didn't matter I could have flooded it all with red or flooded all green But the point is it's all the same color So there's no cases of all three colors mutually in the same neighborhood when I started it up And as long as you follow the rules like this, none of these states will ever come up But if there's any possibility of failure You get screwed So bad That the entire universe the entire synchronous universe grinds to a halt and that's what we're looking at here Exactly if we look in close here Actually, if we change the middle, okay There it is right there green red and blue all Within a neighborhood of each other so nothing can move on like that And you know, I could try to fix this up. I mean, you know, well Let's not take the time because the point is the only way I can fix this up is by using my bigger knowledge Not just looking at the local neighborhood about where the problems are and say, okay Well, if I could color this one red then That this little problem would go away and that might be true But there's a bigger problem and the bigger problem is is that you know in the synchronous universe Has that transition happened or not? I mean the reason it mattered We were relying on these transitions in order to tell us when to do the updates And supposedly in the architectural world, we were going to guarantee That the uh synchronous universe was going to update deterministically And even if we could repair the rgb looping that Caused deadlock there. We cannot reconstruct what the state of the synchronous universe was supposed to be So bottom line Deterministic Architectural sync this idea of we don't have to worry about a synchronous because we'll just use ntn and get back to synchronous And everybody will be happy fails Uh, and so the general idea of that, you know, we we started out using global clocking But it was just small little computers and as the computers got bigger We went to multiple clock domains and interfaces between them But we tried to produce The effect of simulated global sync the ntn idea is another way to produce simulated global sync And it just doesn't scale up as you scale up Eventually there will be a failure somewhere and in this case the cost of the failure is the end of the universe So let's change the game and that's what this is all about that, you know, if we're saying no We're not going to make a we're not going to decide for the programmer to present To hide the nastiness of the asynchronous world. We're going to let that come through But we can still make additional tools that will help the programmer Get things done, you know, so like I said, I started out thinking, you know, sync is evil Don't sync everybody should just do whatever they feel and, you know, be, you know, anarchist hippies and everything will be fine And that's great. But as I worked through The t2 tiles and started programming more and more complex stuff Then, you know, you figure out pretty quickly that if you want to do something bigger than yourself Then you need to be able to coordinate with whoever you're working together with and that's a form of synchronization And so the idea is is that what you should do is synchronize However much you need to get whatever job you're doing done and don't synchronize more than that So because synchronization has these significant inherent costs So synchronize as much as you need to get the job done and no more and that's the idea of local sync program sync And so that's different than assuming it's going to be done as a deterministic architectural assumption In particular if we know that we're programming a local synchronized area, then there will be edges. There will be corners We know the thing at any given moment is going to be finite And we can take advantage of that in a way that the deterministic architectural sync that was saying, I'm completely open ended All you have to do is tell me about the neighbors and I can do with it Can't do so we can do a different design approach that exploits the fact that we're doing finite Program to sync so that we can take advantage of errors. I'm sorry take advantage of edges might take advantage of errors So in particular what I've done is I say here's the approach Let's take whatever we're dealing with and somehow define a notion of upstream and downstream And build a coordination mechanism on top of there's a headwaters of the stream And there's the end of the stream at the ocean whatever it is and they need to talk That's the coordination thing. All right. So let's look at the second example here I'll get rid of this guy Okay, so let's get uh, all right get this guy going labels again So well, first we'll just let this guy run a little bit. So you see what it does Red green blue yellow And then back again. So it's doing some kind of cycling It's it seems like it's a four stroke engine rather than the three Cycle of the antian algorithm, but let's see what's going on if we look inside these guys One here And what I want to call your attention to is right at the bottom if you can see it Got this thing called ringo state that has an m upstate That's false here and m downstate. That's true. And then over at the end We've got m upstate true and true and so now what happens is Well, if we uh All right, so here here we go So now if I do an event here, this is the end of the stream. This is downstream So I do an event here and now we the upstate is false and the downstate is true And the idea of the rule is is the the uh tail the most downstream Be in says, you know, I want my downstate to be the same as my upstate. I am an empathic I want to support you you say false. I will say false and so his downstate changes to false And now it works its way back up and eventually it gets up to here And the head the root of upstream the source of the stream discovers that they're false but the root of the tree is a jerk The tree wants the upstate to be different than the downstate So it goes back to true and then we go back down and so on So what's happening is the head of the of the tree the root of the tree is saying I want my upstate to be different than the downstate The tail of the tree is saying I want my downstate to be equal to my upstate And everybody in the middle is saying I don't want to be involved in it If the upstate if the guy's upstream of me say I'm just going to listen to what the upstream says about the upstate I'm going to listen to what the downstream says about the downstate and stay out of it otherwise And so that's what we get so now the jerk keeps flipping things so that they be different that goes heads downstream and the empath switches to make it Match and it goes upstream and we get this local clock That automatically runs between whatever counts as the root of the top and whatever counts as the bottom That's the idea empath in the jerk And you know, I implement these things and You know, it's always the last step. I mean, I actually implemented this idea of having one side that you know Communicating pair where one of them always wants things to be different and the other side always wants them to be the same And they keep going back and forth in an oscillation. That's actually what the t2 tiles the actual t2 tiles use To communicate with each other across intertile connectors. Well one side the north north Northeast or North north sorry the well three of the connectors are the upstream guy the other three connectors are the downstream guys So whenever you put two of them together you get one upstream and one downstream And one is trying to match and the other one's trying to mismatch as a result They share a clock between them and that's how they communicate information through the intertile connectors so It was only in the last month that I finally figured out what I had implemented first Using the on the t2 tiles themselves And then in software that I was just showing you and it's what's called a ring oscillator And a ring oscillator is a signal loop with an odd number of inverters So, you know, here's so this has got four elements in it But this only got one little circle on it That's the inverter the rest of them are just buffers that just copy the signal from one to the output So if we power this thing up, who knows maybe this thing sees a one on its input just as kind of random So he copies it to its output and so on it gets all the way up to the one that actually is the inverter The inverter says, okay, my input's a one there for my outputs of zero And the buffer copies it. Well, that means now the one that used to be zero one is now zero And it just goes around and around So this is exactly what we saw Or very similar to what we saw In the demo that we just looked at Where the root the upstream the head Is the jerk the jerk is the inverter and everybody else is just trying to say, okay Whatever you say, whatever you say like that And and so it works fine And ring oscillators get used for all sorts of actual, you know in low level electronics They get used to actually make clocks and to synchronize different clock domains all kinds of things So this is a software Ring oscillator is what I've made That's what what the way to call it. So and you know, just to make sure we understand, you know So what's wrong with having an even number of oscillators? Now we got a circle here and a circle here And we clock it and That's what's wrong with it. It doesn't actually oscillate a signal loop with an even number of Oscillators doesn't ring. It's a memory. Oh, and that's one point actually about the name ring oscillator There's an ambiguity in the word ring You can fall afoul of it means both the idea of ring like a wedding ring. It's a loop And also ring like a bell ring like a wedding bell Where it goes? Because it's ringing because it's going one zero one zero one zero Depending on context the ring and ring oscillator might mean the fact that the signal is going around a ring Or it might mean that the behavior of the signal rings up and down between one and zero Okay, uh, all right But in the general case, uh, I'm dealing with things that are not necessarily just an individual line But they are, uh, you know arbitrary shapes and how are we going to define upstream and downstream If there's more than one thing that's upstream and what I use is I just say Don't copy wait until upstream all agrees if upstream all agrees on what the value of upstate is Copy it if downstream all agrees on the state of what downstream is supposed to be copy that Otherwise just stay with where you were And we can take a quick look at it. Oh, it's taking so long We'll take a quick look at it because it's kind of what the point is All right, we'll get rid of these guys So we'll start over and we'll make a seed ring. Oh That's the line that we just did what we what we want is a here we go All right, uh, uh, so now we're doing, uh The same exact same algorithm except now everybody that's either above us or west of us north of us or west of us That's our upstream everybody who's east or south of us is downstream And anybody that's on our same row or column is neither upstream nor downstream. So we don't look at them Um, and we insist on unanimous consent. Uh, everybody upstream says, uh, true I take true for upstate and so on and this is fantastically robust. So, you know, let's hit it with some x-rays Uh, uh, can't see squad Uh, let's get the x-ray tool here Okay. Well, I killed the I killed the plate. There's an underlying plate that's doing this, but we can actually make it get more robust than that We make So this is you know, this is beyond Okay So this is what's called an l2 plate But uh, so now if these guys individually, they're still using And look at this. They're using the ringo ring oscillator all the way ultimately from this state up here All the way down to this tail down here and They've got tons of other things going on. They're much more complicated But still in the middle there, you can find the upstate and the downstate Are working exactly the same way and the root is trying to make them different while the tail makes them be the same Um, and you know, this thing we can we can just plow all kinds of problems through it And it recovers extremely well because number one the uh, the little individual plates are sort of, you know, Replaceable and the neighbors rebuild them But the key for this and this is what was works with the ringo approach that did not work with the ntn approach Is that the red green blue cycling? Recovers like that and even if we do something worse like we you know, just put a big old blob in the middle of it Paint a You know something like this All right, there's something like that You know, so we actually, you know made a big mess out of it. Um, now you can see what's happening here is in this case This particular Square right here It's got no upstream Uh, nobody upstream of it that counts the the white wall is not relevant. So it doesn't count So it in fact now it becomes one of the jerks. It in fact says, oh, I want to have my up the opposite to the down and This guy over here also thinks he is uh, the root because he's got nobody above him So he also thinks about it. So, you know with this big block Just tearing a hole in our in our program in our big data structure we End up thinking that we've got two roots, which is bad and depending on exactly what we're going to do with it It might be terrible But it doesn't even spread because now because we use the consensus rule. So there so we saw this this All right, so that's gone green But nothing else happens. So it goes red real quick, but it doesn't spread further So once we get far enough far enough into it where we don't see unanimity Then the guys in the middle they wait for consensus. They wait for unanimous from the upstream and go So this still does the right thing And my feeling these days is this approach the ring oscillator approach the generalized distributed ring oscillator Is so robust so naturally spatially sensibly robust Let's get rid of this guy That You know if Whether you like it or not um, this makes sense to be the definition of what a distributed shared local clock does And you know, okay If there's a risk of it having multiple roots then that we're going to present that in the programmers documentation for how to deal with it rather than try to pretend We know best we can force synchronization. We can guarantee everything. No, what we want to do is reflect reality In a way that makes sense And this The empath and the jerk I like it a lot Okay, so yeah, and so and then once again when I finally do the related work when I finally study stuff in order For example, make this video This the idea of don't update your state until every all your inputs agree It's called a c element. It goes back to 1955 It was first used in the iliat 2 computer and so on and so forth And I love that. I just think it's great because and and now okay, you know, it was used there It's used in the t2 tile hardware signaling and now it's used again In the ring go protocol that I've just developed in the last month And you know on the one hand I used to think Um, you know, that's bad I'm reinventing the wheel instead of reusing it. But now I'm convinced. No, it's actually right That's what we want to be doing at each scale level We want to be reapplying the techniques of robustness as appropriate to that scale And you know, it's taken really long. It's going to be a 40 minute video. I'm sorry. I'm sorry Well, we'll see if anybody wants to watch it or watch a little bit of it but one last point The ntn approach cycling with the red green blue. It takes two bits I mean, it really only needs three states but representing it in the digital machine You end up with two bits for it The ring go upstate downstate thing also takes two bits It uses all four states, but it's completely different model And you know the the idea of saying, you know in the ntn thing We take those two bits and we strap them together and treat it as a binary number And we say that those two bits are owned by the each individual little atom And it makes the decision about when to go from red to green green to blue and so forth It's the object oriented approach and you know, all it's trying to do is sync and you know So the idea of coordination like sending messages from one end to the other. That's somebody else's job In the ring go approach those bits aren't really owned by the atom that they're signaling I mean they're equivalent in the ring oscillator signal. They're equivalent to wire You know the ones at the top and the root and the tail they're like computational elements the jerk and the empath But everyone else they're just like i'm just making my bit the same as the upstream bits I'm making my downstate the same as my downstream bits. It's not my business And so object oriented this thing called an intrusive data structure and then the ring go approach uses an intrusive data structure and it seems much more interesting to me that The way to do Sync the way to do local coordination is to grant part of your resources to Reflect The world around you to let signals propagate through you Like that and This seems like a nice example either way you slice it It's two bits but with the object oriented approach It was an unending nightmare of trying to get it to deal with things when things went wrong again when everything was perfect It was beautiful And with the ring go with the intrusive state. It's just it's so right that even when it's wrong It's right Okay The advocacy is The top down master of the universe is Good for tiny things, but it doesn't generalize. We should get over it. We should get over it now And understand and embrace the member of the team the bottom up the smallest beautiful Let me stop there Thanks for stick sticking with me if anybody did Here's more information about the t2 tile project itself The t2 demos a github repo it doesn't quite yet the the code that's up there right now is a little bit old But it'll catch up soon And you know, this is all being supported by living computation foundation. So that's it See you next time if there is a next time in the meantime If you're interested in more stuff check out the t2 tile project. There are sort of bi-weekly more or less videos there Thanks for watching