 The T2tile project is building an indefinitely scalable computational stack. Follow our progress here on T Tuesday updates. Hey folks, thanks for checking in. It's been another two weeks. We've got a really nerdy one this time. Uh, uh, I want to draw a big lesson about Stigmergy and, uh, how it works in, in programming or I think how maybe it ought to work more than it does. And it connects to indefinite scalability and it connects to best effort. Uh, and really this is my way of dressing up the fact that the CDM pipeline and still the topic of discussion, we have not yet moved on. We are not yet back to intertile events, but we have a pretty good progress. So, um, last time I was talking about CDM next generation, I didn't actually use the phrase, um, and it looked like it was making good progress and it was. And it was an important reorganization. Got a lot of good stuff out of it, but it was still full of bugs. And the more I tried to push on it, I mean, I could get three tiles in a row, uh, to work, uh, and spread stuff through no problem, but in more complex configurations when CDMs were getting restarted and so forth, it was a mess. And it all boiled down to, you know, the, the program starts up and it reads the files and it infers a whole bunch of facts about them, like makes maps of checkpoints, uh, of different points so that when it's sending the file out, it can give other, the people at the downstream people saying, when you get this far, you ought to see a check some kind of like this. You ought to see a little, you know, marker that you can make sure that nothing's gone wrong. And, you know, when the files would change or more stuff would come in or I would restart, uh, there was just, you know, the other guy, the guy downstream or the guy upstream, he didn't restart. And he was still going to be talking about things that was no longer true for me. Like when I was starting up, I would make little short little tags to, to refer to these, you know, big, long CDM MFZ files that, you know, the file names could be like 50 characters long or something like that. So we made these little teeny tags at, at startup so that we could refer to them back and forth and that when you're exchanging data, when you say which data you want, you could just use one of these little tags instead of the whole file name and so forth. But trying to get the case of figuring out what everybody knew and what they can trust and what they should count on versus what they should not count on became a bigger and bigger and bigger mess. And in the end, I did a major rethink that I want to talk about briefly, uh, and kind of started over version three. Uh, but you know, I get to pull in a lot of the pieces from version two. So the big picture that I got to about this is, you know, when you write a program that you, you start accumulating state, you read an input and you'd say, now I've got, you know, the, the client's name is Dave. And I have to remember that so that when I print stuff out later, I can say, you'll be the envy of all Albuquerque Dave. And managing state every time you pull something in, that's a representation of something that's out in the world or out in a database or whatever it is, you have this problem of what, if the world changes out from underneath you. So the state that you've got is no longer good. And so the idea of this Stigmer G, which I'll talk about more in a sec, is that you use as little internal state as you can get away with. And instead you rely on the outside world giving you the cues that you need to know what to do next. And so minimize internal states, maximize the tangible goals. It's like, if you want to remember to bring something with you when you leave the house, you know, put it in front of the door. So when you go through the door, you know, at least there's some chance you won't just carefully move it out of the way and walk on through. And this all ties in with this basic principle of don't think you're accomplishing the goal. Don't think you have this master plan to take over the universe, just try to make things better. And the programmer, the software engineer, the coder, their job is to arrange things out in the world, out in other computers and in you and so forth, so that if you just make things better, the goals of the larger computation will be served. And the way to do that, the slogan I had in my head is, the terrain is the map. That's the goal. We should just be able to look out in the world. And what we want to do is obvious. And that is this idea of Stigmergy, that that's a picture of a termite mound in Australia or something like that. It's it's taller than people are high. And I think there was one that had some person and it's like 10 feet tall or something in the termites, little tiny little things. And this thing is, you know, it's a famous marvel of engineering of, you know, nature's marvel of engineering. These mounds have all these kinds of internal structure that, you know, they regulate air and temperature and protection and all this stuff. There are versions of these things where they somehow connect the termites can somehow detect the compass points. And they already alter orient themselves north south to get the most sun or the least sun, whatever it is. All of this stuff from these little teeny incredibly minimal state, you'd think creatures. And so these are the examples of Stigmergy out in nature. The mechanism of indirect coordination through the environment between agents or actions. Yeah. So Stigmergy comes from Stig, meaning like mark or a puncture and and Ergon, I don't know what is energy. So the idea is you make a mark of the work that you do and the mark of the work should cue the next piece of work automatically just by virtue of having done it rather than having some arbitrary symbol like a stop sign that could mean anything. If you have a gate that bars the door, that is a direct stop like that. So answering one of these examples of these sort of mediated things where by laying down chemical sense that they can trail, they can say, you know, there was work done in this direction because I went in this direction. So you should go in this direction too and so forth. So the idea is, is, you know, every time when you have a computation, when you have a program, you're trying to write, you're trying to have this some big extremely complicated thing that you want to have happen and in order to have that happen, you have to break that down and represent it in simpler and simpler and simpler terms until you get down to stuff like and gates and or gates or three plus two is five or go left at the light that can be executed directly by whatever machine you're running on. And that's what Stigmergy is. It's just breaking the boundary of what's inside and what's outside the machine. So people talk about, you know, why can't programs be more like cancer lessons in Stigmergy? You know, this was a thing when I got when I was Googling Stigmergy today. The idea is, you know, programs used to be all organized hierarchically, but now in the age of internet, everything is all distributed and people can kind of just jump in and fix stuff anywhere they want to. Yeah, I suppose it's possible and possible. Stigmergy ensures tests are executed in the right order. You know, these are one of these things where they're so gaga about the phenomena, they always over exaggerate. It doesn't really ensure it in very narrow circumstances and, you know, ants, they do not find the shortest route to food in optimal time. The ant colony optimization algorithms are basically a form of distributed hill climbing and they do really well and they're really simple and that's great, but in any kind of optimality, you know, no, only in very simple circumstances. The Grasse is the guy that did it in 1959, something like that, mark of work. Yeah, there it is. The stimulation of workers by the performances they have achieved. That's what the goal was trying to be. I found a reference to high level Stigmergy programming language, which sounded kind of interesting, but it was behind a paywall and I was not motivated enough to try to get around it. Um, so, okay, so Stigmergy, how does that play into the current situation? So for CDM version three, which is what we're on now, the Goldilocks version, you know, not too tiny and disgusting, not too blown out and big, but just right. At least we hope number one, the tough decision was to ditch backward compatibility completely. And in particular, that means giving up on supporting the traditional one hop set of packets that we had before, in addition to trying to overlay the pipeline network on top of it. Could it be made to work? Sure. Was I going to take the time to make it work? No. Uh, one of the virtues of building a whole new stack is that you have quite a bit of plasticity for quite a while. You know, the, this is the flip side of not being a success yet, but you can change stuff structurally and still be okay. So we're going to pipeline always and we're going to try to fight the state, minimize internal state. And the way we're going to do that is by take what would have been internal state and put it out in the world whenever we can, so we can just react to what's out there rather than trying to remember what we had learned and keeping up to see whether it's still good. Um, so that had a bunch of serious costs. Number one, I got rid of file names. You know, CDM, T212.mfz. We're going to flush file names for purposes of the common data manager. Now I am making a segregation for all the stuff that's working for MFMS, the simulator. None of this touches directly. But when it's time to take code that you ran on the simulator and got it and you're ready to try it out on the grid, um, then you're going to have to do some different stuff. In particular, you've used a CD make common data make instead of just a regular make in order to build the MFZ file, but it's not that bad. And in particular, uh, it's going to have to be signed by me at the moment, uh, who's got the private key that the grid will recognize. Uh, so goodbye file names. Whoa. Uh, goodbye, two second timestamps. They're not really gone, but they're not the most important thing anymore. Uh, goodbye. The whole idea of starting up at runtime and reading in the map, the MFC files and building incremental checks on maps. That's all gone. And goodbye. The whole thing about coming up with short little tags so that when we exchange data back and forth, we can just send the tag and not have to send the whole file name. That's an important efficiency thing because each packet can't be bigger than 256 bytes, 255 bytes. So if we're going to send a 50 bite file name, uh, uh, then we've used up a significant amount of our budget and that's got to go in every packet. So that's the idea. Instead of file names, we're going to have slot numbers. A slot number is a number from, uh, one to 65,000. It's two, uh, I'm sorry. It's 256, one to 255. That's right. We only have 255 slots. That's it. Uh, and we're going to end. So for example, slot number two is the T212, uh, package and slot number three is the MFM package and so forth. And, you know, at the moment, we really need like maybe five, four or five, uh, slot numbers for that, for everything that we've got going on. Once we start shipping out lots of different laws of physics, lots of different code from people's programs to try out, we might have dozens more. 250 doesn't seem so bad. We're going with five minute timestamps instead of two seconds. And with that, five minutes, we can take three bytes worth of time. That's 24 bits, uh, three bytes worth of, of timestamp at a five minute interval, plus one byte of a slot number and we can refer to, uh, a specific version of a specific MFZ file in 32 bits without having to make up a tag. And remember, is the tag valid or not? That's the reason we got rid of file names and went to slot numbers, went from 50 bytes to two bytes. Now we went from 50 bytes to a one byte. Wow. I mean, I can't even believe it. Anyway, and, and went from, you know, a bigger timestamp to a two by three byte timestamp. And, and then here's the other thing is, you know, we've been building it, reading the MFC files and checkpoint going through it, you know, with a digesting algorithm and say, coming up with a little, you know, here's a checkpoint for this point, here's a checkpoint and so forth. Now all of that map gets built when the MFC file is getting made and the map is permanently associated with the MFC file and we never have to compute it again. It's just like the idea of putting your keys by the door. Now we have a new format for an MFC file that takes the old MFC file and moves it back 1,024 bytes, one kilobyte and puts a new chunk at the front, which is all of the information that we need, the map of the checkpoints, the length of the file and a cryptographic signature of just that first kilobyte. So that as soon as we get the first kilobyte, we know pretty much everything we need to know about it and of course we get the first kilobyte first, just like you can't walk through the door without opening the door first. So that's the idea and I'm taking too much time even with our new nerd thing. There's what a file name used to look like right in the Emacs mode. We have a special mode to look inside MFC files, you know, and you can see the stuff that's inside it. There's, you know, there's the inner zip file and we can go into that. That's got a tar file inside. It's a different kind of archiving thing and inside that there's files and files and more files and more files. It just goes on and on and on and that's one MFC file. But if we look at this, not with this all human friendly version, if we look at the underlying byte by byte representation, which we can do with Emacs, it looks like garbage. But look, you know, there at the very beginning MFC parentheses, 1.0 parentheses, new line. That's the special flag, the magic signature of an MFC file. And, you know, right after it, there's a PK and some more stuff. That's the magic signature of a zip file, because that's what it is. An MFC file is a zip file slightly warmed over. There's the MFC thing that the MFC dot zip that was all presented nicely before and so on. By contrast, now files look like this CDM SS SS stands for slot stamp slot and timestamp the 32 bits, the one byte plus the three bytes that represent the MFC file within the CDM system. CDM MS, CDM SS 02, the T212 and 516388, whatever it is, 516389, those are five minute increments. They're not always even anymore. And inside it looks the same. There's a there's a inner zip, there's a tar file and it's got files after files after files. But if we look at it literally again, now it has CDM 10 at the beginning, that is the special signature of the one kilobyte map of all the information we need to know and noting the split check sums that will allow us to pipeline this on through. So we just start feeding it to somebody and once they get the first kilobyte of it, then they have a map for all the rest they need to do. They don't need to ask anybody that the tag numbers can't go bad. And if we look and we have to reduce this, there it is. So if we zoom out far enough that we can see a kilobyte all on the screen at once, there's the MFC 1.0, the original file that's all still there untouched, just pushed a kilobyte later so that we'll always remember to take our keys before we leave the house. So that's it. Let's take a look at it. I've got it running here. I don't know. Can you see this? Hopefully we can see this. So this is our main server here. Um, you really can't tell what it's doing because there's no actual representation of serving stuff. But here we've got representations of receiving stuff, 80%, 1.6 megabyte, 0, 2, 5, 16, 3D2. Actually inside that map, there is a label. So T212 is in there, but it's not part of the file name anymore. And it's only it's restricted to 16 bytes. There's also this 57% 0, 2, 5, 16, 3, 69. What's that about? That's an old version of the T2 tile package, because the same slot 0, 2, that got obsoleted by a newer version that I sent before to even reach this guy in completely. And, you know, really what what officially will happen, but some of the code isn't quite there yet. We've got to win a dominating version. So 3D2 dominates 3, 6, 9, same slot newer time. But we wait for it to be complete and be checked. And once it gets complete and checked, there is a garbage collection process that runs in the background that will delete this thing. But for the moment, it's still there. So we're at 80% on 3D2. And up here, we're at 78% on 3D2. That's the pipeline. Yeah. And we can see this guy is outputting 1.6 kilobytes per second. That's pretty low to the northeast, which is here. This guy is outputting 5.7. Yeah, you know, that makes more sense to the northeast. That's why it looks like this guy is catching up. So here's my idea. Again, I mentioned before we've done one and we've done these little straight in a row line things before. And it was always once there was more complicated structure in places for multiple things to go where things went into trouble. So I haven't tried this yet. I'm going to try this live with y'all right now. I'm going to take these two, which are running and they're all happy by themselves. They're not sending anything to anybody because they don't have anything to send. I'm going to plug them in and we'll see what happens. OK, we'll go connect them together. Let's get this guy in there. I don't know which way to connect it. I guess I'll connect it to the guy up top first and then to the West. All right, so now they're connected. This guy should soon up. So he sees his southeast. He sees his Northwest. OK, that's good progress. Eventually, this guy should get around to offering 0 to 516 3D2 because because he's got the the map at the first kilobyte and he knows all of the stuff that he's got out to 78 percent is good as long as the checksums haven't been messed up. So he hasn't really sent anything quite yet. Let's put in the scary connection here. All right. So now this guy should pick up an east connection and he did this and bam, he's now he's heard about 0 to 516 3D2. I don't know where he's getting it from. Actually, I guess we could tell from the the loads input from the Northwest. It looks like he's getting it from Northwest and it looks like this guy is sending stuff to the Southeast. The two numbers in the I and the O is sort of a fast time scale and a slow time scale to kind of get the impression of how much is actually moving and it didn't die. And so now the question is once this guy gets beyond the first percent or two, he should now be willing to feed it to this guy and then we really see this pipeline starting to move. Now, I don't know if it's going to happen or not. We're over 20 minutes, so I should probably be thinking about letting you all go as if everybody was just glued to the screen. I feel like it hasn't, you know, it really the the T Tuesday updates the last month or so have been really a little bit on the boring side. And I really appreciate folks who are stopping by. And, you know, the the nerdiness of it is good and the nerdiness is helpful as far as just sort of getting more information out about how the whole thing is designed. But building a whole new stack the base of the pyramid is big. The base of the pyramid has to carry a lot of weight. So I think we're doing OK. I feel pretty good about the Goldilocks CDM version 3 is going to be enough to get us through to distributing software on the grid. You know, it's never going to be fast, but it's going to be reliable. I think we'll see. I'm a little scared that we still don't have any action going here like why didn't this guy get around to announcing it? At the moment, it's all random. But we'll see. All right, so that's it. I've updated the trailer. It doesn't say noon mountain time. It doesn't say best effort noon mountain time. It says on Tuesday during mountain time. And, you know, today it's already afternoon mountain time on Tuesday. So this is going to be coming out late in the evening. It is what it is. Folks sort of dribble in during the week and a half, two weeks, and there's usually a little, you know, a little extra traffic, you know, shortly before the next one is due out. Oh, look, look, look, look. It did it. It did it. All right. I don't know. This guy got to 2 percent somehow. I mean, it really is just random. And now we've got bites that are moving from the this is the key master, as it happens to this guy. Actually, it looks like he's this guy is now getting it from the West. Look at that. There's one point one kilobytes a second, one point two kilobytes a second coming from the West. So he started out. I think he started out coming from the Northwest, right? But now he's hopped over and he's now feeding down to his southeast. He's feeding one point three off. And there we go. All right. I hope you had a good two weeks. Hope to see you next time. Stay safe.