 I think we're there a little bit later than I was hoping but well we have an hour to work with We'll just run back and and we'll cut it and the folks that aren't here live will never know the difference. It could happen It's been two weeks. It's hard to believe time goes so fast. I am completely stuffed up in my sinuses Simultaneously incredibly stuffed and incredibly dry. So hopefully I can get through this Without losing my breath and and so forth. We'll see what happens This is the last t Tuesday update until December 7th Taking November off to well partly to work on the grid But also just to do other stuff that has been waiting and waiting and waiting So what this episode is going to be mostly about is Resilience and robustness in the L2 plate stuff. That's what I've been working on for the last couple of weeks And there's been a lot of progress and you know to me Coming up on the November break. The feeling is is that you know the story of the summer and the fall anyway of 2021 as far as the t2 tile project goes. It's about plate and about L2 plate that are you know, they're they're clunky and obvious and Non-organic looking And all of the things that that we know about that But you know, it seems to me absolutely the best shot We've had so far on the path to utility that we could actually build up stuff to do that and part of doing that is To try to take seriously robust first that you know, we can't just say yeah Yeah, we'll care about fall tolerance, but not actually deal with it. So two weeks ago I was X-raying some of the face plants and so forth I'm sorry two episodes ago a month ago now and that's what I'm also doing here today so You know when you say you're gonna allow non-determinism when you're gonna allow errors The temptation is to just say well the only errors that I'm gonna allow are gonna be ones That you know are independent random single bits flipping and so forth. That's not what we have we already have at least four categories of Different kinds of errors, you know an atom being cleanly erased leaving the site empty That's the easiest kind of failure. We deal with Packet drops Simulating what goes wrong or what may go wrong. We're still not sure between the actual t2 tiles Cosmic rays is just you know bits flipping in the background No reason no rhyme or reason random numbers as far as we know x-ray bursts is focused stuff That may actually have some, you know space time Coherence to it In all cases the goal of the computation is first dynamically preserve its pattern That's you know for the a life folks That's the definition of life that we use Well that you know that I offer for the take on How to get artificial and look in natural living systems together and the process is Blast the system until it does something wrong try to figure out why it did something wrong see if we can't fix it and go around again this cartoon from Jeremy the hand waving comic came by on Twitter a few days ago This is about what my last two weeks have been like on the left, you know, and nobody actually wants to see that But I did do something that I did in the very beginning 2018 the early days of the t2 tile project. I started a script that does a screenshot once a minute day and night ever since a week ago Monday and Just to kind of get a flavor of the mess of the whole thing and we're trying to take a look at it real quick here You know with things being kind of confused. We'll see how this goes, but do that We're gonna do do the version on the right where we just dip our toes in and have a little simulation of all of the false starts so That was the idea. I started out using the x-ray tool that we saw in the opening demo Where you can just go in and you like use the airbrush to Spray false spray bit flips onto it and seeing what happens. So those yellow black lines are Sights that upon loading them it turned out there was a Parity error error correcting code detected a problem and that's what causes those things to show up And you know, there also can be bit flips that lead to problems that aren't detected by the error correcting code And that's of course just the beginning of the entire can of worms that we have to deal with but you know, they reap they they clean up remarkably well and My goal in doing L2 plates was really to do larger-scale stuff But they've proved, you know extremely resilient at the level of Reestablishing the L2 plate itself now a lot of times You know the content the L1 plates inside each L2 site get lost when an inconsistency is developed Discovered and the L2 site gets erased and rebuilt and it's gonna have to recreate itself So, you know, this is not any kind of magic. This is not just preserving everything magically This is let's keep the L2 plate structure intact as best we can Dynamically preserve its pattern and then have L1's guys deal with you know Well, maybe they could have some redundancy in neighboring things and they could transfer some information in so there's an example I don't know if you can actually see it, you know, right down here where there's a little extra L2 plate because the thing is coming out a little bit too small But it cleans itself up and that's the typical of what sort did so I would just hit it with more x-rays Until it did something that was wrong So this is an example of something that goes actually wrong now We've got an L1 plate that's got no L2 plate around it at all and no L2 plate in within reach Able to reseed it and then the L2 plate off somewhere else. This didn't go anywhere. This one was a trapping state This was a failure and what I would see a lot of is L2 plates L2 plates spawning off extra L2 sites that really shouldn't have been there and I looked through that and Eventually, you know, I got it down to a specific case here that you know, it's supposed to be three by three So so that X should be a three there and not a seven But it had a bit flip it had the the four bit flipped from a zero to a one which turned a three into a seven and that's what it's doing And so the next plate over still thinks it's a three by three But this L2 site thinks it's a seven by three and this is one of the tricky states where that error Actually leads to a legal configuration in terms of the plate relationships The seven by three plate says I contain you your entire three by three You're just inside of me and you know, in fact when the error first happened It was only in one L2 site that was complete was claiming this and if it had truly been a seven by three Containing plate it would have been in all of the L2 sites along that edge and it wasn't but the L The other side had no way of knowing that locally speaking It just says okay that looks like a plate that's containing me So no act no actual error when it was looking from one L2 site to another but As you know I messed with it some more and it recovered some more and it re-grew and respawned from one side to the other and so on Same thing would happen if it had been cosmic rays happening rather than me actually messing with it But an interesting thing happened is that you know They were both sides the seven three plate and the three three plate were trying to Regrow well the seven by three this is seven by threes and the three three as that and as a result Eventually, where does it happen here? Yeah, right here that eventually in the same plate So now we're in a single L2 site and we're looking at adjacent L2 Well one L2 plate we're looking at adjacent atoms and now one of them says my L2 information says I'm on part of a three by three L2 plate the other one says I'm part of a seven by three L2 plate. That's an inconsistency all The L2 atoms of a single L2 site should agree on the size of the plate and their location in the plate So this is where an inconsistency is detected or well It wasn't detected But it turned out to be fairly easy to write another little bit of code To say if you know, we are on the same plate So there's me and there's a the other thing that I'm looking to say are we consistent? So if we are on the same plate in an L1 sense, which means we're in the same L2 site Then we ought to have the same My size Does that here we go. So my size And it should be the same as a size my position on the L2 plate should be the same as L2's position and so forth And if not, we have detected an inconsistency and we return true now What actually happens when we detect an inconsistency? Okay, we found it But how do you know which one is right one says three three one says seven three? How do you tell? Because you know it could be either especially when you just have a little teeny people view The answer that I've been adopting more and more as I go forward that I like pretty well is Whoever discovers the inconsistency assumes they're right They They're self-confidence and they tell the other side to die And you know in fact, they don't really tell it They set the death bits on the thing to initiate the death cascade on the other side The problem with this right is you know, which side happens to discover the inconsistency Depends on whether the three three Adam gets an event before the seven three Adam or vice versa And that's random by the design of the machine So how are we going to have any sense of knowing whether the right person the right site? The right answer is going to win and the answer to that is we don't know But over time the odds favor the locally consistent choice and and that you know It's it's been kind of a joy for me that says I work through this and I keep thinking You know from a correct and efficient point of views Well what I have to do I've discovered an inconsistency what I have to do is to get a bigger picture view You know look further and further find out what's going on take a census take a vote send messages To say you know am I right or am I wrong so I don't make the wrong mistake But so much better to just say heck I'm right Tell the other guy to die And if I'm wrong if I'm not right then he's part of a locally consistent region that I'm gonna run into another one of them soon and I'm gonna have to win a flip against them and It's the basic thing is that the consistent as long as we have redundancy local spatial redundancy Like supplied by the plate the next guy over his position on the plate is one bigger than mine I could compute his position on the plate, but every plate Adam records their own position. That's a ton of information That means that if I have a cosmic ray that comes in and flips one site That it might be able to take over and convince the whole world that yes, this is a 7-3 plate That's the correct answer, but it's gonna have to win These showdowns with I think it's 3-3 7-3. Okay. I win next guy I think it's 3-3 and the odds are the locally consistent choice wins It's really Extremely fundamental and it feels like it's part of the very bottom-up nature of the whole thing I mean, you know fork bombs are the that just reproduces fast as possible are the absolute basic dynamics of these kind of systems And if you have like a red fork bomb and a blue fork bomb and you plop them both down They're both expanding as fast as possible They quickly run into each other and start overwriting each other and you say which one is gonna win you know is it gonna end up being all red or is it gonna end up being all blue and You know it could go either way, but the basic principle is you know You should bet on whichever color has more area right now other things being equal if red is surrounding blue Then it's more likely that blue is gonna get go extinct and vice versa if blue has more area For this exact reason that the minority has to win more Longer sequence of battles, even if they're completely random coin flip In order to actually succeed and they just not likely to do it could it happen sure is it likely to happen don't bet all your money so I've got you know that so these ideas are some of these things like the 7 3 and the 3 3 we're at the L 2 plate level That was a fault. It was a one-bit fault originally, although it did spread for a while before it got wiped out Was one of these things where it flipped it into another legal state a state where the L 2 plate was Containing another L 2 plate and so forth same thing. I've I saw it several times happening with the L 1 plate And I think I have so you know here's top of the minute you can see my thoughts working out over time So here's a case where you know it the two of the L 2 L 1 plates up There aren't filling the L 2 plate. Why not? Well because the plate that they were being reproduced from down below actually consists of a smaller L 1 plate and a Mutated edge that says it's containing that smaller plate and when the L 2 plate spawning process says Let's make a copy of the let's seed the L 1 plate from a copy of what I have it seeds Whatever size it sees and that's the wrong size to actually fill the plate and those things come out that way and once again, you know We could say that Every L 1 plate should completely fill the L 2 plate and then this could be detected as an inconsistency And something could be done about it about it But this is a trade-off and you know my feeling is is that maybe we want to have complex L 1 plate Relationships which are then surrounded by an L 2 plate. So we're not going to necessarily be able to detect whether this is right or wrong just based on local information and this goes to the Fundamental principle of this whole thing that you know allowing L 1 plates to be nested in various ways and then have an L 2 plate puts around them is much more flexible Then just saying the L 1 plate has to completely fill the L 2 plate or it's inconsistent And but that flexibility that ability to be more programmable Means there are more legal states that can be confused that you can have some type some You know cosmic rays the x-ray burst whatever it happens to be That messes things up and takes us not to an inconsistent state but for a consistent state of a program that we're not running and in general what we would like to do is be as Inflexible as possible be as hard-coded as we can get away with Given that we're down in this very low-level stuff, and we want it to be all strong from my money the key point is that exalting universal program ability is It's almost criminal because what that means is is that you know there's this unbelievable myriad of legal programs nearby and when we get beyond just Cosmic rays flipping random bits, and we get to malice then we can leap to one of those other states one of those other legal programs Trivially alright So I fixed up a couple of those problems like putting in that little bit of extra code and then I also had this one where The the simulator crashed when I had background cosmic rays going and I was looking at Twitter or Sleeping or whatever it was I would come back and That the engine itself had crashed and it actually took quite a while to run down for Good reasons and bad. Oh, and and and by the way just for the record I changed so the cosmic rays the way that they originally were working up until a couple of days ago Was when an atom was being written back from an event into the tile storage It had a one in a hundred one percent chance of being corrupted And if that one was chosen on a one percent chance then each bit of it had a one percent chance of being flipped So a one percent chance of being chosen for corruption and then a one percent chance per bit of a flip And I've now changed that to one in a thousand so It's you know ninety nine point nine percent of the time it will be written back Faithfully, but that remaining point one percent the time It'll be corrupted and when it is corrupted It'll be a one percent error and you know what it boils down to it's like just like that in the mouthwash You know it says ninety nine point nine percent of germs are killed, you know, and there's plenty left You know ninety nine point nine not such as a gigantic number really one in a thousand There's still tons of errors that are happening in this thing the problem and you know And most people that are studying fault tolerance They're talking at errors of the one in a million one in a billions, you know much much smaller than this This is really quite aggressive But at one in a hundred the three by three l2 tiles would sometimes they you know They would just get wiped out just by bad luck, so I want them to have a chance of surviving so I did back it off All these stuff I've been through I'm not gonna taking too much time, so I'm gonna skip it Here's another one where it grew a little extra a little, you know webbing between two of them and eventually I managed to figure out what was causing the Simulator to crash it was terrible because I would leave it running with background errors for like hours and hours and it was fine But then I would come in and I would mess around and start Investigating what's going on and 20 minutes later it would crash and what it was was look up a long element type from Adam That's the fundamental low-level thing where he says tell me what the class is what code I need to run for this particular atom and at that low level the atom is supposed to be consistent It's supposed to have been error-checked ready to go and that code is not ready to deal with it being Inconsistent being messed up And why are we calling look of Adam element blue type from Adam we're calling it because we want to find out what color the atom is Well, that's what we use every time we're drawing it But in this case we're calling to find out what colored is not in order to draw it on the screen But for an atom view panel, it's we're trying to figure out that blue color like that and That drawing of the atom view panel was done in a separate place in the code then the drawing of the tiles So there was nothing catching that error. That was the problem. So we have fixed that up Fixed a whole bunch of other things and as we saw in the opener I'm just gonna do this relative. I'll take a couple of minutes to do a demo and then we'll wrap up since we did start late You know, I always imagine that things will go much faster than it was and you know I wanted to get the L2 plates moving. Uh-huh. So move these machine and they're still not moving But you know, all of this robustness stuff was worth it Taking the time to pound it all out because you know, this seems like the the absolute Basic stuff that we're gonna be building on and we want to number one make it as solid as we can and number two We want to understand it quite exquisitely So feeling good about that but if we want to make things move we need to be able to coordinate at a whole L2 plate level and Just like we did on the L1 plate one way to do that is to do a rock paper scissors Sort of thing to coordinate You know the upper left changes color and that flows down to the bottom right and then the bottom right changes color back And it flows back up and down But doing that at the L2 plate level is new in this update So let's see if we can Do a demo here. All right, so What is that seed All right, I think that's right. So Okay, so this is a RGB sync Demo so it doesn't do anything except it's an L1 plate and it's going back and forth So once the upper left is green It changes blue and once the lower right is Red it changes green like that and we go back and forth and this is very nice But we can mess it up because that's part of the game here unlike, you know, and again, this is a variant of the Basic trick that's long been known in cellular automata fields about a way to put the synchronous stuff on top of a synchronous stuff This is the classic one But it's related and you know as I repeatedly Emphasized when talking to people that throw that result in my face saying oh, there's no reason to do a Synchronous cellular automata because we can always lay a synchronous one on top of it and do that Well, that's only true if there's never any errors and so if we come in here and See it survived We'll we'll mess this up Whoops see I killed the whole thing Now in a way, I'm kind of I'm kind of stacking the deck against myself because This particular thing, you know Let me just put some X-rays on to this and let it go for a second. All right So So you see that kind of grayish color in there So there's in the classic synchronous on top of asynchronous There's just three states which we make our beat red green and blue Here we have a fourth one which is sort of like an initialized state that we go into in certain conditions when something's going wrong And so that's part of why I wiped out again part of why it's actually able to Survive a lot of the messing around I'm doing with it It's doing really well Which All right, well, maybe we won't be able to do it. So, you know, this is sort of Praising with fake dams, I guess Maybe if I just draw some RGB sinks in there somewhere That was inconsistent too well, maybe I won't be able to show it Cuz Did the job too well Well, it turns out well, I suppose we could do it. Why don't we do that? We'll try to do a An actual L2 plate demo instead Oops, all right, okay, so there we go So this one actually locked up just right out of the back out of the back Because it's got this weird little case that the solid entire plate doesn't have and so the L2 plate has to jump The the color signals have to jump from one L1 L2 site to another and that means they have very limited Communication and they can get into places where they don't know what's going on and they end up locking up So if I get rid of these by hand There we go, okay, so now that should Finish and then cycle back Like that So this is nice. It works. Well But if we turn on background errors writes fault, where is that? It's behind here. Whoops Turn on writes fault here It won't be long before we will manage to get The RGB sink into a state where it stops Cycling because it reaches an illegal state now again I've done a bunch of work in the last several days To make this more robust than it was out of the box and I have ideas about how to make it still more robust But we're not quite there yet. I want to make it more symmetrical. You know how it is Let's turn these back off Okay, so now it's messed up again, but we can do surgery and Get it going if we just get rid of those and so part of the problem is is that the When red is heading down, it's got this buffer of blue But when green is heading back up, it's right against red And so I want to introduce a fourth state like yellow that will go between green and red when it's going up And blue will go between green and red when it's going down. There still can be additional Inconsistencies that won't magically fix everything, but it will help So so this is pretty good. I feel I feel good about this And that that'll be the the demo that we wanted to do So All right, that's it In November I want to make progress on building the damn grid I think a fair call is to have the grid all assembled in 2021 it may all be working it may be working to some degree it remains to be seen but The thing will be built and we'll find out if it's not being not not running will have more information as to why So we're gonna have a camera a better camera. I ordered it. It should be coming in in the next week That's gonna hopefully get us much much better Pictures of the running grid then we see during the waiting room for example here and so on and we'll be good Just going from there All right So that is it. I don't know what the heck happened with the camera when I started up. It's a different camera But Step-by-step we've also got a new workstation a more powerful workstation with a better graphics card and so forth that is going to be coming in Well, we've got it that by the next t-Tuesday update. I want that to be running all of this We'll see how it goes. I Hope you're going You're doing all right. Thanks so much for checking in whenever you manage to Come around to see these incredibly nerdy Little progress reports. I feel okay. I feel like you know It was fun Finding these cases and discovering hey There's a pretty easy way that I could actually catch that and watching that the L2 plate getting more and more Resilient as a process That's it. I will see you in December. Have a good November. That's it