 The T2 Tile project is building an indefinitely scalable computational stack. Follow our progress here on T Tuesday updates. Distance 7. We'll come back to that. So last time, it was about the common data manager that I've been struggling to get it to do what I wanted it to do, and had this little bit of a breakthrough that was easily understood in the ideas of Stigmergy, that you want to not keep stuff hidden away in your head, but have stuff out in the world, and then everything you and others who are involved in collectively getting something done, react to what's out in the world in a way that makes the next step automatically possible. And got a lot of progress, got a demo to kind of work. There was some strange things about that, I was happy that it worked at all. But the speeds were kind of weirding me out. I mean, so these kind of numbers like 1.6 kilobytes per second, 1.7 kilobytes per second, those I was familiar with. But you know, I was also seeing these things, where was it? It was somewhere here, you know, 5.9K, 5.8K, that was like more than I was usually familiar with seeing. And put it all together, and it still came out that CDMSS03, that's the MFM distribution, that's the big one, 15 megabytes in 2 hours, 2 minutes, 10 seconds, 2.2 kilobytes per second. So that was about what I expected, and 2 hours is a big drag. But you know, what was going on with the 5.8K, the 5.7K? Well you know, so I looked into it further, and it actually turned out that there was a combination of bugs. And the way that the transfer of data works is the one that's getting the data asks for the next chunk, and remembers what chunk it asks for, and then when the chunk arrives, it checks and makes sure it's the one that's expected, and if so, it puts it in its buffer and writes it out to the disk in the appropriate time and so on. But there was another part, which was the background, you know, timeout mechanism that was supposed to kick in and, you know, give everybody a bump if there was some kind of problem. And that hadn't been tested, and the, well, it had been tested, but it hadn't been tested enough. And the net effect was that it was actually waking up and saying, hey, what's going on? We're still looking for block number, you know, 109, and it would send out another request for it, so that eventually the first request would get handled, then it would get handled in a normal way, and then we'd say we're looking for 110, and then another copy of 109 would show up, and the way the code was written, it was saying, no, no, no, I don't want 109, I want 110. So it would end up taking what was two copies of 109 and turn that into two requests for 110, one of which was going to get ignored. And then when the timeout would come in again, it would inject yet another one and another one. So the 5.8 kilobytes per second, the 8.0 kilobytes per second was almost all repetitions of multiple copies of the same block, which were getting tossed away, but additional requests for it were going out. And the only time, the only reason that ever stopped at all was because eventually the K-FIFO, the first in, first out data structure inside the Linux kernel module that we wrote a while ago, finally filled up with all of these packages, with all these big 180 byte chunks of the CDM, the file that was being passed, and the kernel finally said, you know, the heck with you, I'm going to start throwing these things away. And that was what was slowing it down, because eventually the requests and the replies would sort of run out by going into this space. So it was like, OK, well, you know, maybe we should fix that bug number one, but maybe we should take advantage of the idea number two. So I rewrote the thing so that instead of just waiting for the first packet to come back, it would have a notion of it could have a certain number of requests in flight simultaneously. So it could ask for 109, 110, and 111 before it actually saw 109 reply coming back or whatever it was. And that's what max MFZ data in flight was. It's the number of bytes that you're willing to have on spec that you're asking for. And I said it, at first, I said it to three times the packet length. And it started to work, and it started to work a little better. So eventually I went to six times the packet length, and I ended up at 12 times the packet length. So in theory, you could have 12 packets, each with, you know, a chunk of 180 bytes plus metadata in flight at once before you actually stop waiting until you start waiting for the reply before you go ahead to the next one. That I also needed. I had another parameter, which was how much time around the loop we would check to do outputs and inputs after I kicked that up a little bit. I mean, originally the whole point of CDM was to keep it as a background process so that it didn't take up too much in the machine. Still want that to be the case, but I have to admit that during development, it really helps CDM moving MFC files reasonably quickly. And when you are moving a lot of, you know, the fundamental things, changing the laws of physics, changing the MFM engine itself, changing the T2 infrastructure, are you really going to be running simulations that you're trying to collect it for? Well, maybe you are, but it's not going to be for a while. So I sped it all up and it, oh, and when the other wonderful thing was, you know, I finally, finally discovered the Perl debugger. I'd never used this in my life. It's great. I mean, it's got a little quirks and everything like all debuggers do, but man, it helps out a lot. So all of this wouldn't have even happened except for the fact that I've now ramped up a little bit on Perl debugging. Big artist started up again and now 15 megabytes in 45 minutes, 40 seconds. More than cut it in half, 5.8 kilobytes per second coming in. Now, so that, taking that in combination with getting rid of all of the debugging symbols from the MFM package, cuts the MFM package down to like 7 megabytes. And the time to push it out is like 25 minutes, something like that. And it can push to an entire power zone in a half in under an hour. Not so bad. Now, of course, there was the whole problem. Oh, and I updated the CDM control, the program to manage the deleted files. You know, this is what it used to look like. The names of the files are deleted as of this time stamp and so forth. But that all got changed because now we went to slot stamps, which, you know, a two-digit slot number in hex, a six-digit time. So that got redone and now we've got, you know, now it's, you know, slot 90 at 51675 is deleted, slot AO is deleted and so forth. So, but we gave up on backward compatibility to do this. And so now how am I going to get the software? I can't use CDM to get the software into the middle of the grid. I can plug into the west edge of the grid using an Ethernet cable, but I can't plug into the middle of the grid. So I was either facing, disassemble the entire grid yet again or what. And I came up with a what. I used the serial port cable, you know, the serial cable, this thing, that I can plug into any tile that I want to as long as I can get at the little slot on its lower right side. And I used Z modem. Is anybody old enough to remember Z modem? We have exchanging files like over telephone lines, over UARTs, over serial lines. Well, that's what this is. So I used Z modem to send down a tar file and to send down a hack, a little script that I made to unpack it and set it all up so that with those two files down, if I ran the script, then that tile would be ready to go with then CDM could be used for the rest. And I sat there and I went through each tile in the grid, like three minutes apiece. And I got it done. And there it is, the Z modem saying it's uploading the files. I would have thought it was downloading, but you see, things are different now. And we got the thing updated. And there it was. And so, you know, here they are. This is a whole power zone, all powered up. It's got an ethernet cable sticking out the side. You know, I was thinking about when the power, when the whole grid gets bigger, it's not necessarily clear that there's going to be a wire near ethernet nearby. I mean, you could just stick in one of those little dongles, Wi-Fi USB things, but the tiles don't have any USB ports accessible. The USB stack that's built onto the underlying board is blocked by the West ITC. So I was even thinking about, you know, well, I could get a Wi-Fi to wired connector, you know, and then I looked into these things a little bit. And they're pretty cheap, but oh, man, it looks like they're an incredible security nightmare. And then I thought, again, well, wait a minute, maybe I should just, you know, eat my own dog food. And so I said, you know, this is much, this makes a lot more sense. I will just plug in a tile. I will let the tile update using CDM. So there it is. It's pulling, you know, 03, 516, 98B. You know, one of the things, I kind of like these timestamps. So these six-digit timestamps are hexadecimal, base 16. So each one is four bits long. So 516 represents 12 bits and 98B represents 12 bits. And these are five-minute chunks since 1970. And the way it works out is that the first three digits, 516, lasts about two weeks, kind of like the current T Tuesday update period. And in fact, most of the time that I was doing this development, I was in the 516 era, which was kind of special to me because it was one of my phone number area codes when I was a kid, you know, 51675129, you know, on Suffolk County in Long Island. They've rearranged the area codes. I don't think 516 goes there anymore, but it did when I was there. It was kind of nice. So now we're actually on to the 517 era, which is nice too, but I enjoyed 516. And it seems like a good pacing, at least for someone of my age, you know, about two weeks isn't bad. All right. So anyway, so I would collect up the packages on one of these tiles and then power it down. So there it is, 99% for the O2 package, which is what we were dealing with here. And it's done. Now, it's actually trying to pull in an MFM package too, but that's still going to take a while. So I said the heck with it, you know, the grid that I want to send the T212 package to already has it. So I just power this guy down, boom, and I carry him over to the grid with a ribbon cable, I plug him into the grid. He boots up with the power from the grid. And actually the first thing that happened was he resumed picking up the MFM pile. This, the Stigmergic solution for CDM, it's really nice. You see, you see the files going in opposite directions at the same time. Oh, you need O3. Oh, you need O2. Sure. Let's swap. You see things where, you know, one file starts being fed in one direction and then eventually this guy gets ahead and it turns around and starts feeding the remainder of it in the other direction. It all just works. It's nice. And so there, and now the grid is starting to pick up O1 and 90, these various things they come on through. And here comes O2 and it moves on through the whole thing is good. I realized that, you know, maybe it would be worth actually getting like a little 12 volt battery pack of some sort and making a little portable tile that had its own power supplies and used the power inject the data only connectors, the DOs, so that you could actually carry this guy around and he could be live and you just sort of plug him in. He'd go feed, feed, feed and so forth. May still don't know was shopping for it a little bit. All right. So that's the state of the CDM version three pipeline. It's pretty well done. It's in production as far as I'm concerned and I'm kind of moving on. So let's do that too. While I was doing that, you know, if you remember about a month ago now, I had the power zone was flickering and disappearing. It was all terrible and it turned out to be, you know, lousy wiring. So this little loop of wire was touching the other side and so forth. So I said, I think really I should try to find so I made it better. I did it much more carefully. It's all beautiful now for the ones that I hand built, but I also started looking around to shop for these, you know, commercial ones for a Y connector so that we plug it into the power supply. We'd have two heads coming out and we could put two power injectors on it and that turned out to be really handy because that way you could put one down and then put the other one down. And if you needed to get at either one of the tiles, you could move the other power injector without losing power to the whole zone. So I wanted to have Y connectors for everything. I found some cheap ones on Amazon. They came in. I tried them out. I plugged them in. Things seem fine. It was working everything. All the stuff that I just showed you that picture with the yellow ethernet cable was using one of these Y commercial Y cables. Now, when I was looking at it, this was an example. So look at this guy. He's got big green arrows coming in from five directions at once. That's a CDM running and pulling packets from five neighbors at the same time to build up whatever it's looking for. I don't know what it is at this point. And that was pretty cool. I was spending some time just kind of enjoying it. It's working pretty good. And then I'm like, you know, well, you know, wait a minute. Why is the voltage on this thing 10.93? The power supplies are supposed to be 12 volt power supplies. We've specked out a power zone so that it should be less than, you know, 50 to 60 watts. The power supplies are supposed to be good for 65 watts. What's going on? Looking into it, it's like, wow. And here's another example. 10.87 volts on these two and this one, you know, so 10.90 volts here, 11.95 volts here. What's the difference? Well, there's actually a frontier here. These are two different power zones. This one is running off of one of the existing power supplies that I had from way back when using one of the homemade DIY splitters. And this thing is running off of a sort of a scratch power supply I made using one of the commercial Y cables. And the Y cables were hot. So I'm really not so sure about those Y cables. I was worried about them for current carrying. I'm still worrying about them. Yeah. And you can almost see the power injector frontier there between the pretty much nominal 12 volt side and the 11 volts and downside. So at the moment, I'm happy with these lead mo AC adapters. And I'm going to stick with them, but I think I'm going to stick with the handmade splitter cables unless I can find some that have heavier wire. So that's it on the hardware front. One more issue after getting the CDM kind of working. It was the, you know, the next pain point for as far as developing and going forward was a way to avoid having to click every single tile. When I wanted to switch to see the CDM, what was going on and then go back and click every single tile when I wanted to see what the tile menu was. It was showing and so on. So it was time to implement flash traffic, which had been, you know, we'd had it running in an earlier version when it was the MFM T2, the direct port from MFMS, the simulator before that was proved hopeless and got all torn out. And we've been working our way back towards that as a sort of second system put together better flash traffic reserve for messages of extreme urgency. Brevity is mandatory. That's what we're talking about. And it's been in the spec for ages. You know, there it is, the flash traffic all right up with the MFM traffic and the intertile connection negotiations and so on. Going through the various still there, the ITC changed its design. That's when we go through a couple of months ago when we were trying to get the intertile connectors to talk to each other. And eventually it got fleshed out. The flash command has a command and index a TTL checks on what's that. So the command is going to be a single byte saying do this, you know, put up the tile menu, the index is supposed to be a way to disambiguate so that you know that this command is fresh. And so the idea is that each index, each command should have a higher index than the previous one. Unfortunately, of course, there's only one byte. So you can't count very high before you're going to have to wrap around. And that's a big problem in general. It's a big problem in distributed systems. There's a related thing called the count to infinity problem, which is mostly known specifically in one particular routing context. But it's really a more general problem of trying to come up with a way, a distributed system for describing freshness without dealing with the fact that the numbers get eventually bigger than you want to deal with. So this is one byte when it gets to 255, it wraps back around a zero and we have to deal with that somehow. TTL is a time to live. When you put a packet into the grid, each time it takes a hop, in our case, from one tile to another tile, it decrements the time to live when it gets to zero, the packet stops. Check sum is just a little bit of redundancy that kind of mixes together the bits of the rest of the packet. So you have some idea whether something has happened to it. In the current iteration, this is now C code, C++ code. We've got the packet header, one byte, the command, one byte, the index, the thing that's supposed to count to infinity, one byte. We also have the range, which is the initial time to live so that any person getting it can not just know how many hops it's got left on it, but it can know how many hops it had when it started. And then it's got this B point. What the heck is the B point? It's the XY coordinate where this particular command originated. Because one of the big problems with this is, okay, so this guy says, he wants to tell everybody to show the tile menu. Then somebody else says, no, everybody else show the CDM menu. How do you know which one is supposed to have priority? Well, the index is supposed to be going up. But what if they both happened before neither of them had seen the other? They're going to have indexes that are, who knows what? Who knows how they're going to relate? A better way to do it would be to say, I'm only going to compare your index to your previous command. So in this case, now what we do is we keep track of where the packet came from just in a relative way. You know, I start out at 0, 0, and by the time I get to the other guy, I'm at 3 minus 2 or whatever it is. And I say, okay, what's the last index I saw from this guy? And if I only compare indexes from the same origin, then I don't have to worry about the inconsistency of the commands coming from different places. I may still have race conditions where multiple commands come in from different directions and they will be executed in different orders at different sites, but that's a separate problem. We'll have to deal with it. So we pass along the x, y, it's a signed single byte, which allows us to go 128 basically in either direction in x and y, which totally limits the range of our flash traffic. The maximum size of our fleet is going to be some kind of 120 by plus or minus 120 relative to us and so forth. But of course, that was already limited by the time to live, how many hops we can go and so forth. So all that's there. And then there's a little array going from the directions of the compass north, northeast and so forth to how the B point, the x, y changes. And it's a little bit surprising, but it ends up making sense that if we look at the east and west offsets, when you go east, you actually add two to the x. When you go west, you subtract two to the x. Why is that so that when you go in the brick wall, when you go up to the diagonal up, you go plus one, plus one, if you now go diagonal down again, you go plus one, minus one, you need to end up at two zero. So if you went straight across, you need to increment by two, not by one. So there's this coordinate system where moving left and right on a horizontal row increments by twos in either direction, going up and down on the diagonal, goes up and one and down on the other. And you know, okay, and that works pretty well. And so here's the code that says, when you've got a packet and you're getting ready to propagate it onto someone else, you take what they thought the origin was and you subtract out this offset that we just talked about based on the direction that you're sending it. Similarly, you decrement the time to live and you update the redundancy so that because whenever you change any of the arguments to the packet, you have to recompute the checksum. I did some early design for what I wanted the flash traffic menu to look like, boot everybody, turn them off, show the tile. Eventually I realized I wanted more commands than we're really gonna fit on one screen. So I developed, well, so it turns out the graphics library that we're using never had a radio button concept. We never implemented it before. So I implemented radio buttons so that you'll be able to pick one of these commands, show the tile menu and the other one, show the CDM menu will pop up automatically, that kind of thing. And I got it all implemented and it was like, what's gone wrong here? The whole thing, it's got all of these different coordinates coming in. This was all coming from one command that I sent out. What would it turned out to be? Oh yeah, I had a loopback cable. So you go out the southeast, you're at plus one, minus one, you come in the northwest and you're at plus two, minus two and you can go around and around. It's exactly like standing between two mirrors and saying there's that guy and there's that guy and there's that guy behind him. So that's kind of a pretty inherent flaw in the idea of being able to locate relative to you based on the hops that you pass, which is a problem on the one hand, but on the other hand, I am not going to stop because of loopback cables. This makes total sense that this loopback cable would cause this illusion. So we're just gonna live with it. And I was gonna show a demo, but we're really running super long. Maybe we'll pick it up next time. So the flash traffic stuff is starting to work and I feel good. I feel like a lot of progress, that the CDM stuff in particular was solid enough that I could build on it and stress it and it would work great. And so that's it. Don't know if anybody's counting, but next week is gonna be the two year anniversary of these T Tuesday updates, two years. So gonna have to do something special. I'm not sure what. And I will stick a little bit of some of the video that didn't get to use on the end here just so people can watch it if they wanna hang around. Thanks for being here in any case. Hope to see you in two weeks. All right. Well, I was trying to catch the first try of this and I forgot to turn them actual start recording. So we missed it. So this is a redo of the new flash traffic system. So like I was saying, you can still pick these outer commands and set things like what we view, the different menus on the titles, but now we have this control panel in the middle where you can issue commands to the grid. So let's, for example, say, everybody should show the common data manager. I'm gonna send it to the grid. I'm gonna send it out to distance eight, which is measured in terms of hops. And I counted it out before and I think eight does it. One, one hop, two, three, four, five, six, seven. So eight gives us an extra stop, in fact. So there's no calling this back once more. We've sent it out, so here we go. It took a while, but there it is. And we ought to be able to do it from anywhere. Display the tile, distance seven. It's really surprising how much variation there is. The way I designed it theoretically, it spreads out from the center and it ought to execute from the outer back toward the center. So like the captain, the one that gives the order goes last, but it doesn't really work out that way, or it works out that way sometimes, I suppose. Now, let's see, let's go back to the site display, then try one of the things that we have not tried yet. So this will be a first. Now, what happened here? This guy just failed to get the order, completely failed to get the order, huh? Why don't we try it again? We'll try it from a different guy. Display sites, grid sites plus eight. All right, you, there we go. You got it that time, and this guy got it as well. All right, so this is the new one. So let's try control physics seed one to the grid. Engage, uh-oh, oh, what that did. But nothing has actually happened here, so I suspect it didn't work. Ah, yeah, there we go. Worked fine, I just forgot that I hadn't written it yet.