 Computers keep changing the world, but their power and safety is limited by their rigid design. The T2TILE project works for bigger and safer computing using living systems principles. Follow our progress here on T Tuesday Updates. This is the 46th T Tuesday Update. Let's get into it. For the last couple of weeks now, it's been about getting intertile events working. And that is really the heart and soul of the whole idea of indefinite scalability. If you're running a program on a computer and it can make changes inside itself and inside the physical machine, but to get outside the machine it has to do IO, it has to do something different. But if you turn the whole thing inside out and you say you can automatically, by the way the architecture works, have an event that sort of starts out on one piece of a hardware, one TILD, what we say, and have effects on the neighboring TILD so that in some level, except perhaps for timing issues or other reliability issues, what have you, the computation just moves from tile to tile to tile. And that's what intertile events will get us. And we're getting close. The essence of intertile events, the way we're doing it here in the T2 is it has two parts. It has a locking step where you ask the neighbors not to make any more changes in this part of the program grid so that I can make a change and be able to believe that you know about the results of it. And in order to provide the results of it, you have to be able to talk to the neighbor and do a cache update. So locking plus cache updating. Two, three weeks ago we actually got the locking pretty solid and we've been working on the cache updating. This past week it's been a huge battle for getting cache update working. I'll tell you how it worked and how it went. And not to spoil the ending, our plan for next week is to have the entire shebang going, the actual intertile events going, no matter how slow, no matter how lamely something, so that we can actually see a computation begin on one tile and move to another tile and keep on going. I think it's in reach. We'll see. In addition for next week, a plan for the next five weeks, why for the next five weeks? Well, not sure if anybody is really noticing, but that is going to be a year. That's going to be 50, we're almost at 52 episodes of T Tuesday updates. I would really like to have a pretty major result. And who knows, maybe after five weeks from here, we're talking series finale. Who knows? Everything's up for grabs. That's why we need a plan. All right, let's get into it. Okay, yes, the E&O division. What is the E&O division? It's the new name of the propaganda division, education and outreach division. Why is the propaganda division being renamed? We'll find out in a week or two. So this week in the E&O division, we have another podcast and it hasn't been done yet. It's actually going to happen today, later today, which is one of the reasons why I got to get this thing out. There's a book. It's a really good book by Brian Christian and Tom Griffith called Algorithms to Live By and it sort of sums up a tremendous amount of computer science as it applies not just to computers, but to people, to living systems, which is very much in the living computation spirit. And apparently they are doing sort of a follow-up of Algorithms to Live By where they're having a small little podcast and they actually, Brian Christian invited me to come talk about the T2 Tile project as one of the episodes of their follow-on podcast. So that's pretty exciting. We'll find out how it went next week. In addition, oh yeah, so Andrew Walpole got a setup with a new actual location, t2tile.org, go check it out, see what's there with this great new logo. And in fact, he has gotten it made into stickers and there they are. They're pretty glassy looking. And he says he'll send some out to you if you're willing to go over to the Gitter and ask him for them. You'd have to give up a physical address, which who knows in the modern world. Maybe that's too scary and maybe we just don't care, but I think I'm going to get a couple of them because I think they're cool. In addition, the place that he got them from, the Sticker Mule Place, they were running a sale, which is one of the reasons he did it and he put the link to it and so I tried it too. So I got some stickers as well and here they are. So here it is. It's a MFM t2. I tried to make it something that was, it's got a little Unikifying Text 244. That's last week's update because that's when it was that I ordered it in 2019. This is going to be a commemorative 2019 Software Guy Lost in Hardware Land Sticker. Never to be repeated. There's 50 of them. I'll give you, if you ask and you can somehow get me address. They actually have three, you know, three little inch of tile connectors on them. So if you actually get more than one, although I can't imagine who would be entitled to that, you could actually tile them and have them kind of overlap with each other. Just by purposes of comparison, here's how they compare to a real tile. They sort of fit in the display. They're just really very nice. So that's kind of fun. And supposedly they're high quality vinyl, whatever. I don't really know much about stickers. All right. So the main event for today is the report from the Software Division about trying to get cash updates working. Failure plus deadline equals drama. Well, that's that's David Mamet's recipe for how to write drama. The deadline was, I really wanted, I mean, I really wanted it last week. And all we got was one little teeny cash update packet sent six bytes sent, which was a lot more than we had. And this was the setup with using a big long curve around cable so that the key master in white could be talking to six different tiles all at once to see what would happen. And we got to the that six greater than one, six less than one. That's a single six byte packet being sent. I guess it's sent to the east and received from the west by the neighboring tile. That was last week. So, yeah, so we successfully shipped a cash update packet. And that's, you know, what's last week's making lemonade out of the fact that entire events were not working well. So this week it's a whole new thing. So I went right back to work on a week ago Wednesday almost. And so as usual, what I do is make lists of things and see if that suggests what to do next. And then maybe I can start chipping them off. And so the real question is, you know, got this whole existing structure in the M M M M F M S simulator that has its whole idea of here's a model of a tile. Here's a model of an entire tile connection. Here's a model of a cash. Here's a model of a channel. Here's a model of one end of a channel and so forth. Those are all little pieces of bits of software that have been built up over the years to represent a tile and intertile event. And of course, part of the challenge is that the T2 tile, the actual real tile, which did not exist when M M F M S was written, doesn't work quite the same way that the model does. So there's going to be sort of a mismatch. And one of the big mismatches was that the M M M S simulator thinks that it has six different individual point to point connections from my northeast cash processor to the neighboring tiles southwest cash processor and so on. Whereas the way the Linux kernel module on the physical T2 tiles works is all of the intertile packets are delivered in one place slash dev slash itc slash M F M. And from the first bite of each packet, you tell where it came from or where it's going to when you sit when you ship it. So that was going to have to be changed no matter what. And so I've been moving the software design up and down making these changes going around and around circles driving myself crazy. Trying to figure out what's the right way to adapt the M M M S structure to the T2 tile hardware. So here's an example. Could we come in a little higher feed packets down to the CPs? That's the cash processors, the things that take care of sending the cash update methods, messages back and forth between the tiles. And so how could, well, maybe we should try that. So I decided to try that last Wednesday decided to try it. How can we proceed on that? So I make a sort of a to-do list and sometimes I actually managed to kind of knock them off done, done, done, done, sort of done and so on. Now, a lot of times I don't check them off quite as cleanly as that. In fact, you know, so entry number seven was 939 entry number 11 is 940. It's a minute later. I was just knocking them all off when I came back. But then it moves on, right? So now I ended up building a whole mechanism that here's the other problem. The flash traffic where we had packets where you said like grid reset, grid clear and so forth. Those are traveling over slash dev slash IDZ slash MFM, the same channel as well. But they have to be diverted out so that when we're reading a cash update and we get a flash traffic packet, we can dump it someplace to be handled a moment later. So we built a new abstraction packet FIFO first in, first out FIFO where we could throw a flash traffic packet in there when we got it. And then later on when we had a moment to process it, we pull it off the other end and deal with it. So we built it. We built these FIFOs a number of times now over the years going all the way back to 2008 and the IXM tile, the honorary T1 tile. Here's another one. It was untested, but in fact it appears to actually pretty much work and so forth. What I finally realized was that when this is all in the MFM MFM package on the T2 tiles and to push out a new MFM package to the grid takes a really long time because the inner tile bulk traffic, common data management process is a background process. It's slow. It's like 1,000 bytes a second or something like that. And the MFM package is megabytes so it takes an hour. So really it was time to take the Keymaster, not to be trying to inside out to get six different tiles to talk to it but to put it back on the loop back, to put it back on the cable so it could talk to itself and we could debug it more conveniently. So here it was. This is when it was actually sending updates to six different neighbors at once. I actually, then this was an interesting little physical test. I said, so can I remove the Keymaster from the middle while the rest of the grid is running without having any of them reboot? And I did it. And one of the reasons I could do it is because I have now a Y cable for the power connector so I can have two different power injector heads. One that I could plug into the Keymaster, one that I could plug into the rest of the grid so that when I pulled out the other ITC connectors the Keymaster didn't lose power. So it actually did work. It was ungainly. It wasn't real clean. And then I blew it because I set the... And this is kind of a remaining problem, right? We have all of these naked exposed pins and I set it down on something I'm not sure what. So after I'd gotten it all out the power supply glitched and I had to reboot everything anyway. But so now it is. And now here it is wired around to itself with three inner loop back cables connected east to west, northeast, southwest, northwest, southeast. And so forth. It was a real kind of a pain because I couldn't actually see the screen very well anymore because it was like covered with bandages. Eventually I realized, oh yeah, I could actually wrap them around the back because it's just one tile so I can have access to all sides of it. And there it is. And okay. And now it's working. And this is what I've done. All the remaining development this past week was on the loop back. So the key master is talking to itself, but it doesn't know that. So it makes debugging a little easier. That doesn't make it easy. So again, now we're to last Thursday. I'm asking myself questions. I'm making guesses about what I think is going on. I mean, like this is software. It's supposed to be completely well-defined. We're supposed to know really what's going on. There's supposed to be correct versus incorrect. This is the way software actually works. It's so quickly, so complicated that who knows what you know. And you make these guesses and so forth. And so, you know, I was getting punchy, issue one. It's an update and being received and handled. I went into the debugger GDB, which for some reason has got some problems. I have to be very careful about how I use it on the T2 tiles or else it gets some internal error. But it's still valuable for getting information and being able to put breakpoints and say go here, stop, let me look around. Eventually, I started putting in more and more printfs, more and more output messages while you're running so that you can look at it afterwards and see what's going on. Things rapidly become so complicated that I actually sit down and I write annotations to describe to myself what's actually happening. An event begins. We grab a lock. There goes the begin packet. There's the six bytes equivalent to what we saw last week. But now we're actually sending an atom. 15 bytes in each atom update packet. We send a bunch more. We send an end packet. An end packet. It's super cute. It's a little T2 header byte and one more byte saying E, meaning end. And it shipped. And now, of course, it shipped and came back to itself because of the loop back. And now more stuff and more stuff and so on. And we say, you know, wait, that's bad. We sent no atoms and we didn't send any packet and so forth. So you start to see where it falls off the rail. And then you try to make inferences about what might be going on inside the software that caused you to see that behavior. And then, you know, once things have gone off the rails, it's really hard to learn much. And finally, we actually die. And so I was doing a lot of this the last several days. And then, you know, sit back and ask myself questions. How can I explain that data? Does that interact? Now, one of the real problems is that, you know, this guy, he's got the display thread and we actually ask the tile to stop running events when we display it. And then we ask it to start up again. And that's a sort of asynchronous process. The timer that's saying it's time to display is different than the thing that's actually running events. So there's a lot of complexity there. In fact, if we had a re-implementation that was just for the tissue tile, that would be something that could almost surely be simplified away. And it would be the sort of thing that would all go much faster. But again, it's not just about going fast. It's about going fast in the MFMS style code base that we can actually go back and forth between the simulator and the tile and have some relationship between the dynamics, the behavior of the two. All right. So doesn't pausing just mean doing this? I don't know. That's different. There's a whole other level. The tile driver and the tile driver has a, in fact, has an advancing case where it advances the model of the channels, the things that actually ship the bytes. And once I talk to myself enough to realize this, then it was like, well, you know, maybe the thing to do would be to go all the way down to that model of the channel sending individual bytes and replace, move in there rather than trying to go higher up like I was doing earlier in the week and replacing a lot of the software. Maybe we want to go really low level and replace the very low level thing. And the advantage of that is that you preserve much more of the existing code. The disadvantage of that is that you have to buy into more of the MFMS mentality. In particular, we would have to simulate the idea that there's six different channels to talk each ITC to talk to its corresponding neighbor rather than one place where they're all pooled together. So we'd make a separate layer and we'd ship any outbound stuff. It's couldn't so forth. Long and short of it. Yeah, and that now I'm already starting to go, this is Saturday now. We're starting to think, yeah, I'm going to do it. All right, so we really want to try it now. This is early Sunday and I'm basically starting over. I'm taking, say, you know, in fact, I took all of the software that I had hacked up and I moved to the side. I got a fresh copy of an earlier version and started working forwards again with this idea of going all the way down to the bottom and preserving as much of the existing code as possible. It's going to make things a little bit slower, but it's going to make the simulator and the T2 tiles be more similar to each other, which is good. And hopefully it'll allow a somewhat reduction in the amount of debugging necessary to get events working on the tile, because it'll take advantage of more of the debugged stuff from MFMS. And I had this thought. Let me see. That's one real danger signs I had this thought. Preserve the existing lock and go all the way down to long live locked, which is one of the very low level things. So I started over. I said, okay, we're going to intervene at four places, lock a choir, pack it outbound, pack it inbound and lock release. And then we're going to have strategies for dealing with these things and so forth. And I started into it. And, you know, don't worry about efficiency. Just try to get it working. And, you know, getting copies of the logging output, annotating them to see what's going on and so forth. And I made progress. But, you know, now log file is Tuesday. So the current state of affairs is we can run actually a bunch of events with getting locks from the neighbors, which is actually getting locks from the self. But as far as the T2 tile knows, it's getting locks from other tiles and lock requests and so forth. And it can actually send cache updates back and forth. And do all of that stuff in a way that all looks plausible, except every so often for some reason it seems to fail to deliver, fail to release one of the locks before it goes on and tries to get it again. And then eventually it's holding like four locks, three locks, and that's completely inconsistent. It never needs more than two locks to have an event on the T2 tile. So close, but totally out of time. This is my new slogan, software guy lost in software land. Here we are. But we have these long traces of all kinds of stuff that actually is happening. We have a new pitcher. Last time we had that six greater than one, a single MFM packet. Now look at this. We have a storm of MFM traffic. 87 bytes in 11 packets were sent to the Northeast. 109 packets in 109 bytes in 11 were sent to the Northwest and so forth. With this, I'm pretty confident, famous last words, that we'll be able to get to intertile events plausibly kind of working by next week. And that's it. The next update will be out in a week. Have a good week.