 The T2 Tile project is building an indefinitely scalable computational stack. Follow our progress here on T Tuesday updates. So, this is the new Common Data Manager. It's actually the Common Data Manager status panel in MFM T2. It's not the Common Data Manager itself. Common Data Manager doesn't actually have much of a user interface. But this is running in some super fast time-lapse. It's like two hours running compressed down into half a minute or something like that. But it's working and a lot of new stuff is there. So, I want to talk today about how we got there and some of the challenges had along the way. This is what it looks like now. So, this is a couple hours later after the Common Data Manager has finished moving the MFM engine, which is like a 15 megabyte file from this guy, this Tile up here to its Southwest and its Southeast. And, you know, it took forever. It took hours. But, you know, the CDM is not the main event. It's the background. And in fact, at the same time, all of that was going on. Oh, here, this is the new interface where we have the CDM status panel, which is what we were looking at. But at the same time, in the background, there, look at that. There was a Type 1 beam wandering around this little universe the whole time. And that's the point, that we're going to be willing to take extra time for the CDM to do its business because, in general, we want the underlying laws of physics to be changing relatively rarely. And this, once we climb that far up the stack, is going to be what it's all about. So, that's where we're at as far as it goes. I want to drill down into it a little bit just because, you know, of course, I didn't expect to spend as much time on it in the last two weeks. I did some other stuff, but did mostly this. But it's back, and it's better than ever. Even at the moment, kind of the original point of doing the whole CDM revisit was to do pipeline transfers where it could flood through a whole bunch of them at once. We're still heading for that, but first it was time to clean up the original version and make it something that was more suitable for some software engineering. And this happens a lot. When you're trying to build a whole new stack, you're going to have this process, or at least I'm going to have this process. I do have this process of trying to reach for a bunch of behavior in a spike, in a quick program just to learn the issues, get the lay of the land, see what I really want, what I actually can have, what I need to give up right away, and what maybe I can have if I try some more engineering. And so the first version of CDM, the first two versions of CDM, in terms of the underlying wire protocol that CDM is using, this is now version three, but it's, well, let me show you. So the original, this is the little directory of the first two, well, the previous CDM. And CDM.pl, that's the whole thing that does the exchanging data, the announcing files and seeing who's got newer and older ones and so on and so forth. 43 kilobytes of Perl code to do it. And what it looks like now is this. There's still a CDM.pl file. I don't know if you can see it up here right there. And it's 784 bytes, but there's all this other stuff, which is in total 100,000 bytes, 100 kilobytes, 104 kilobytes of stuff. So it's somewhat more than double. But everything is broken out as best I could. I tried to be really squeaky clean about it. So, you know, if there's a different function, make a different module, make a different class, express the relationships as best you can. And it's already been paying off. It's already been, you know, when there's a bug or when it's clear that we need to be doing something else that we're not doing, much more often there's a specific place that ought to be responsible for it. Not just where was it happening to be implemented, but it ought to be the time queue. It ought to be the transfer manager traditional. The TM traditional is dealing with it or whatever it is. So, in the original code, even though, you know, it had a bunch of stuff there, it also had a lot of things like this. In the whole, you know, load up the list of deleted files, right? When you're sharing common data, one of the real challenges is how do you delete stuff? And normally, if you just delete a file and someone else says, hey, here's another copy of it. So, we had this scheme to keep a list of all the files that have been deleted so that once we delete them, everybody would consult the list and they would not take a new copy. They would not offer a new copy if it was on the list of deleted files. And in fact, that list of deleted files gets itself packed into one of these MFZ files and passed around. But the original CDM, the CDM that we had until two weeks ago, you know, when it was time to update to the deleted files, it printed a little message. So, there was a bunch of stuff that wasn't really there. This is the new code. It's got an express place where it's supposed to run hooks for when files have been loaded into CDM or when files have been released to disk. We have, when the command's CDM deleted file has been loaded, we run the hook, we register the hook on our hook manager, you know, try to be as careful as I could, everything in its place. And then all through it, you know. There's all the various stuff that you have to do. If a file's coming by, we need to check if the thing has been deleted. If there's a file that has been sitting in pending, which means it's just been received by somebody else and we're going to now promote it up to common, the CDM common directory, which is where everybody is supposed to agree on what's there. We shouldn't promote it if it turns out it's deleted or if it got deleted in the meantime, et cetera. And the file itself, each individual file actually has events and timeouts and it is able to sort of be active and it's really emerging. But this idea of you have, you know, sort of interacting agents, kind of like actors for people who are familiar with the kind of programming model called the actor model. And then you have timeouts and everything has to be designed so that if timeouts come at a weird time, then they will either not disrupt things or everybody will be ready to tolerate it. So it all works out. And CDM control, which existed but was very limited, is now much more limited. It still only has a few commands. Delete, add another file in it, help list deleted. And we can, you know, if we look in the guts of it, oh yeah, this is a nice trick. Didn't exist until this past two weeks. CDM deleted.mfc, that's the name of the file that contains the list of files that's been deleted. So here's a nice little piece of code in CDM control today. It doesn't let you delete CDM deleted.mfc, which, you know, have I ever actually done it? I haven't, but now I can't. So a lot of it. And now we've actually got a few files that I put in the thing that are deleted files. So newstyle.mfc was something that I was using for testing. Most of these are testing. CDM Distrib T2GFB is something that goes back over a year. I was actually unable to get rid of. We're not, you're not using GFB. There was a thing for doing graphics that ended up not using. But, you know, I would think I had gotten rid of it and then I would plug in some old tile that hadn't been updated since forever and it would start spreading Distrib T2GFB around because we didn't actually have the deleted stuff working. Now it's in the deleted file and so on. And, you know, so newstyle as of, you know, 15 bbbb, T7 as of, you know, so what are those numbers? Those numbers are the number of seconds since midnight, January 1st, 1970, I think. Something like that. And the point of it is, is that when we delete a file, we're deleting it from some specific moment in time. We're saying, because the file named test.mfc might have many incarnations. So when we say I want to delete test.mfc, what we are really saying, we want to delete it as of this time and earlier. Anything earlier than that is we don't want it. If there's a new one called test.mfc that has a newer timestamp associated with it, then it's a separate issue. And that's what those numbers are. Numbers of seconds since the epoch since 1970. You know, is there anything weird about those things? I mean, aside from the fact that they're big, you know, 1.6 billion seconds, it seems like a lot of seconds, but then 1970 seems like a long time ago. And, you know, every single.mfc file has one of these things, inner timestamp. There it is. There's another one. Is there anything weird about that timestamp? You know, they're all about the same, 1.6 billion because we're all in the same year. There's some from our little pink boxes that we saw a week or so ago. Again, you know, 159. Anything weird about them? See anything weird about them? I never saw anything weird about them until the last two weeks. And there's another one from the last video. So, I'm going to tell the story of one of my bugs. And by way of learning a little bit about the MFC file and how this kind of stuff works, as well as just because, you know, this is the T Tuesday update and this took some time. So, the way an MFC file works, it's based on zip files. That's what the Z in MFC stands for. Moveable Feast Zip. And the way it works is it takes all of your files and it wraps them up in a zip file called the inner zip. And that's what inner zip is for. And then it uses your private key or in the case of the stuff for the tile, it uses the special key that is a source that is compiled into the code to be accepted so that all the tiles will accept code that has been signed by this particular handle. And it signs the inner zip file and then it wraps that whole thing in another outer zip file and then writes that whole thing out, okay? So it's like, you know, Russian nesting dolls. It's your files that in fact are wrapped up in a tar file and that whole thing is wrapped up in a zip. The zip is signed and then that whole thing is wrapped up in another zip file along with the signature information and some other information about the whole thing. Now, the new version, the make inner zip, it's the same thing as before, but here's what it is. I wanted to have an announcement packet. This is one of the things I realized was kind of wrong about the previous version, especially when going to pipelining was that you could announce this file saying, you know, I got a file, it's this long. Here's my timestamp for it. Trust me. And in particular, don't trust me just until it gets to you, which is the way the one hop, the traditional stuff, works. But trust me just because I say the check sum of the first whatever is this, you should just go and do it. So what I wanted instead was to be able to say, here's some information, name of the file, the timestamp of the file, a check sum of the file, summarizing it all up. I wanted that to be signed by the same authority that was signing the inner zip. But we were going to take that signature and we were going to smash it all down small enough that it could be in a single packet so that when we announce from one CDM to another, saying, hey, I've got this thing, we're actually sending something with a cryptographic signature in the single packet that the receiver can then check and say, oh, okay, well, you know, maybe someone stole the key and this is all being messed up, but I'm no worse off trusting this announcement packet than I was trusting the whole MFC when I got it one hop. That's what's going on here. So in order to do that, I needed to be able to predict what the inner timestamp was going to be so that I could put the same timestamp to be in the announcement packet that came in the thing. So the idea was, was that the inner zip code that I wrote, I changed it to also, you know, give me back whatever the inner time was so that we could go ahead and when we make this announcement packet, this was the whole new feature, put the inner time in it, sign the announcement packet, give me back the announcement packet. And then when we make the outer zip, we include the signed announcement packet in it as well. And that was great, except I couldn't get it to work. It worked sometimes, but not other times. I couldn't figure out why I was going crazy. Eventually I came up with a workaround, but I didn't understand why it worked. And so, you know, go to the documentation. I am using, this is written in Pearl and one of the reasons I'm writing it in Pearl is because it has so many libraries of all kinds of great stuff that you want to use. This in particular, I'm using IO compressed zip along with a bunch of crypto stuff. And, you know, you read the documentation more closely and say, hey, time number, set the last modified time field in the zip header to number. Hey, that's handy. But hey, I don't even need to do that. The field defaults to the time the IO compressed zip hop which was created. So that's what I did. I would just get the time myself, the number of seconds since New Year's 1970 and then use that as the time. And, you know, as long as the compression thing didn't happen in the next second, it ought to be all right, but it wasn't. It would work sometimes and not other times. So finally I broke down and said, okay, well, I'll use the time equals whatever and I'll get the time myself. I'll pass it in and say, use this. And again, it works sometimes, not the rest. And eventually, and so here it was. So here's my new revised code where I picked the time by calling, you know, give me the number of times and moments and since the seconds is the epoch, pass it into this thing, pass it into everybody and it didn't work. The hack that I finally figured out was that if the assigned time was even, it worked. All of those times, all the times in the pink boxes, all the times in all of the timestamps and all the verified stuff, I'd never noticed it before. They're all even. And so once I made the code, always do even timestamps while when it worked. And finally, after I diagnosed it, then I could find out by, you know, googling around what's going on. Zip entry timestamps are recorded to only two second precision. This reflects the accuracy of DOS timestamps when the thing was created. So there it is. There's the answer. In the land of DOS, every single bit was expensive. So the timestamp was recorded the number of seconds in the day and the number of days. And the number of seconds in the day was recorded in two bytes, which is 16 bits, which can only go up to 65,000. But there's 86,000 seconds in one day. So they use two second granularity, which, you know, okay, history is always with us, just like the evolutionary history of all the crazy stuff that we have, because that's just the way it used to be done. There it is. Wouldn't have killed the documentation, folks, to mention this somewhere. And so there it is. 1597, 884, 526. All of our identifying numbers for the life of the T2 tile project are going to be even. So, okay, that's mostly what I was doing. Ben, our latest LCF nerd actually joined a while ago when things got kind of messed up. So I apologize for that. It's possible that we may have a mechanism to improve our email flow for the future. Who knows? We'll see. We'll find out about that. I thought also I just wanted to run past a few of the other projects that are out there, because folks might not know about them. And there's a lot of very cool stuff happening. So, oh yeah, this was the latest thing that Strider, who I've known since a long time, has been working on MFMS on the actual major code base in part to get it converted to SDL2, a more recent version of the graphics libraries. And he had a report. So, I got the MFMS now, uses the SDL2, and I wrote a really simple little synthesizer app that plays frequencies online now, and you know, the 440, because I think it looks notes, but you can drop these things on there, and they play their little sounds for their process. And so, yeah, it was kind of happening with the way it came out, it was funny. So, that's pretty funny, I enjoyed that. I'll put a link to the video below. You know, what exactly to do with Adam's making sound? Who knows? But after we get stuff working, we're gonna need to get stuff interacting with the world, because that's what this computational model is about. It's not about the Platonic input output mean nothing. It's about being part of systems that are coupled to the actual world that they're physically located in. Also, Luke Wilson has been working on this sand pond thing. Let me show you a little bit of his stuff. Luke Wilson posts stuff on, I see it on Twitter. I guess it's on Instagram also. This looks like 2D. He's doing like parallel event windows, but the underlying engine is 3D. Let's see. I'm not exactly sure how this works. All right, there's sand. And we could, you know, melt it with lava maybe, because I'm not sure if that's melting it or not. Laser, yeah. There we go. Yeah, there's some way to grow that. Oops, that's not it. Ah, here we go. Yeah. Take a laser shower, buddy. So very cool. And the latest stuff with the event windows, he's kind of looking towards, I think, GPU stuff, which, you know, is really great for folks that try to speed up little finite scale demos and can't not look at the community without talking about MFM Rocks. So we'll end with that. And we can't forget MFM Rocks. Andrew Walpole is the great grand bean with accessible MFM-like simulators. It's been an essentially continuous development for, I don't know, a year or two now, and at least these are the cell membranes that he's got. There's a new thing you can play with. You can put them all together. Super swapworm. I don't even really understand how that works either, but it sends back little pulses to shorten up the end. Here, I hit the times 10 button here to just get it so it becomes a little jerky, but it goes faster. I don't know what the heck happened. But there's an awful lot you can do to train your eyes just by playing around with these things about how asynchronous bottom-up parallel organizations can exist and form and live and die. So some of the things going on around the community, folks, thank you, everybody. Thank you for coming in and taking a look. This is number two of leaning into nerd, giving more detail, being absolutely specific about what's going on. If other people find the project, that's great, but we've got an awful lot of history, the folks that have come along this far. I'm hoping next week that we'll have the pipelining, not just working, but working, and we'll be back to intertile debugging. We shall see. Have a good two weeks.