 The T2 tile project is building an indefinitely scalable computational stack. Follow our progress here on T Tuesday updates. Okay, lots to talk about. Last time we got to the point where we put together PowerZone 2, but we hadn't actually tried it, so let's take a look and see what it is, and then we'll talk about some stuff. This is our two PowerZones. The gray up here, yeah, you might be able to see it, that's, this is PowerZone 1, the original one, and right here is the DOs, the data only intertile connectors that share ground wires and signaling wires, but no power. So the idea is we should be able to power Zone 1 up, leave Zone 2 cold, and bring it up separately. Let's try it. I've had some issues, but here we go. All right, so that's going to take a minute and a half still, how frustrating to come up, but the point is, let's, this guy ready to rock, get PowerZone 1, to actually get the screen that it turns, it's a matter of turning off the lights and letting this stuff show through. Actually why don't we do that now, and so we could try to see some stuff happening and then we'll boot up these other ones and we'll call it a day. That's plenty dark. Got a few, got a few beans running around there. Heck, let's get, so seed 2 is the splits at the end of the universe, guys, so if we give them some time, which I'm not going to do right now, they'd start to fill things up. Modulo, the continuing bugs, all right, so here we go. It's power up zone 2, all right, so power zoom 2 comes up and we see the frontier between them as the intertile connectors negotiate and manage to figure out and get in sync with each other happening all over here and happening between power zone 1 and power zone 2. So now, there we go, there we go, okay, so now all across the frontier, the intertile connectors have opened up and we ought to, for example, be able to send seed 2 right through from power zone 2 into power zone 1 and so forth. So that's the good news and let's just go back and talk about it a little bit. So lots of issues, let's see where to begin. So this is what it looked like when I was putting it together, sharp eyes will notice that the new power zone 2 is on the left in this picture where it was on the right in the video, I rebuilt it a couple of times, but this is just to show you what it looks like. There are the back plastic frames that connect to the overhooks and then they connect to each other. These are male and female overlap and squeezed together. They're just press fit, but they're, I did them pretty tight and they really seem to be pretty reliable or pretty, I mean partly you don't want them to be incredibly strong actually they need to give a little bit so that everything else can shift into place and in general it's kind of like crush zones. You want to let them absorb some of the energy so clicking it in all the way down the line then you put a tile in to cover them and then you just need a DO connector on one side or the other well all the way around to avoid making a single electrical path between the two power zones which could destroy the power supplies which is one of the reasons to really avoid this and way back in 2008 with the first generation the Illuminato X Machina when I was teaching with those students often well more than zero times destroyed power supplies and could have had bigger problems by connecting multiple power supplies through a bunch of T1 tiles that were actually shouldn't be connected to the other and so here it is. So here's the two of them almost all of them in and it immediately brings up another problem which is the content data manager that we worked on ages ago and it's really just been doing its job okay is conservative and what it does is it has a MFZ file which is like a jar like a zip cryptographically signed by the Keymaster so that there's some reason for these tiles to trust this particular new code because this is low level code this is laws of physics code that comes in in the MFZ file so if you change the MFZ files you're performing magic you're changing the laws of physics and so that's where we put in the cryptography and all that stuff at the upper level at the atoms and the elements and so forth that's where we want the custom bespoke reality to fend for itself and it's not about cryptography it's not about fragile brittle systems and so forth the problem is is CDM going through these intertile connectors is slow and we want to reserve most of the bandwidth for the upper level for the MFM events so in fact to send the MFM package which is big especially because of the moment I've compiled it with all kinds of debugging options which makes it bigger it's like a 15 megabyte file which doesn't sound like much but it actually takes over two hours like two and a half hours to go one hop so if we're going you know one two three four five six seven hops to get across two power zones times two and a half hours that's a pretty slow workflow so I spent most of the time this past two weeks working on a new version of the common data manager CDM the pipeline version of it so that rather than have the destination wait to receive the entire file so that it can verify the checksum and the signature before it sends it on further which is the safest thing to do now I'm having the origin compute a bunch of intermediate checksums at like 1% 2% 3% of the file and pass them along and empower the receiver once they get to a checksum match they can then announce that much of the file downstream so if we can get this to work right we might be able to go across a relatively large grid in not that much more than it would take to do one hop that's the power of pipelining and in my little test you know so all right so we have a new version of t212 that's the package that has all the infrastructure in it everything but MFM and that's where the common data manager is so we're using the old version of the common data manager to release the new version of it and that's a little tricky that's like rebuilding an airplane while you're flying it and I really hard and some parts of it worked but so when we're updating t212 we're using the old one hop at a time thing and that's the sense starting to send to sending last of that's the hallmark of the old style and and this guy got it and he installed it and he all of a sudden knows about running version one there only was version one before nobody ever said it that's because he's now running the new version and he gets back in touch with the key master now oh and the down and up shows that in fact the guy that are northeast the key masters northeast actually rebooted after installing the t212 package now he's running version two and on my desk you know so here's the key master at the bottom and the guy that we just saw at the beginning and another guy up at the top and you can see bulk data moving through all the connectors at once that's the pipeline working and so yellow is the guy that's at the top he received the the t212 package he installed it and now he's announcing it oh and now that he's getting a new version of the MFM package actually MFM to I just made it up for testing but these prefix extended those are the hallmarks of the pipeline where every time we get a new checksum for so much we then announced to the rest of the world that I have this far if you don't have this far I can give you pieces of it so that all worked pretty well and it's all ends up hanging on how do you know which to t212 you want which MFM t2 you want and there's a little pink box at the bottom of the tile display that probably can't see very often because it's really a tiny font but it's got these numbers these are not serial numbers these are Unix time numbers seconds since 1970 something like that and within a name t212 we just believe a cryptographically signed timestamp that's newer than the one that we've got means we should take it the cryptographically signed MFM t2 that's newer and so forth so we can watch this we can see it happening so here I'm not 440 484 I usually look at the load or digits because they're kind of rare that's the old version and here look at this so up here he's got the new version too 54 84 but his MFM t2 is the 32 526 and these guys up here have a different 94 562 and in fact all three guys around him have the newer version of MFM and this guy is downloading the MFM package from all three of them at once and it's not quite bit torrent because well for one thing we do it sequentially from the front and for another thing we don't go three times faster because we're trying not to kill the tile and leave some performance over so it doesn't actually speed it all that much up but it makes it more resilient balances the load better so that's the good news about the pipeline the bad news is once we take it out of the nice three little three guys in a row it's got bugs it's got bugs that are causing it to to make not sufficient progress to actually hop across the thing so more work needs to be done on that okay also there were hardware issues and then we're going to go a little bit long today but there's been a lot of stuff going on and you know what can I say this is the news so what's wrong with this picture if we know anything about the t2 the tile panel which is what we're looking at here that has a bunch of powerful things like reboot turn yourself off crash the engine quit the engine similar but different and over here a whole bunch of physical stats 11.72 volts that's the grid voltage that our entire power zone is seeing 98.4 94.8 degrees is the temperature sensor in the middle of the circuit board minus 128 degrees Fahrenheit is allegedly the temperature at the edge of the board so there's two temperature sensors on the circuit board unfortunately the temperature sensors built into the CPU of our particular unit don't work that well so we're doing it indirectly by having these temperature sensitive resistors mounted nearby this one I've been seen forever it was like a manufacturing defect or something like that these are the kind of things that happen if the board got flexed so the solder cracked a little bit or something and so the temperature sensor is not making good contact this one I was totally familiar with but however in the past two weeks I ran into this one where both the center and the edge temperature were off and that was bad the edge temperature we're not currently using for anything but the center temperature we will actually start slowing down this a thousand M that's the one gigahertz clock speed that the CPU is running at and according if the center temperature gets too high we actually throttle down the CPU to lower speeds to try to cool things off it doesn't make all that much difference but we try so the fact that this thing is reading minus 95 degrees it's going to be perfectly happy running at one gigahertz which is the max speed for these things no matter what's going on under the hood so that I am just living with at this point I suspect I could you know get down to the circuit board here and probably just touch the contacts with the soldering iron and maybe fix this up but it just hasn't reached them to the level of putting the time in for it on the other hand I had a tile that would not listen to me I would type on it tap on it to bring up the menu display it would not bring up the menu display I finally did debug that and it's it's the touchscreen is dead and it's touchscreen was not originally dead it originally worked for some time but somewhere along the way it has died this is what hardware does so I marked it out I pulled it out I went back to the spreadsheet where we have the processing unit and the display the fridge thing and marked out you know 3182 rest in peace and it got replaced with 3181 because that's one of the ones I just had lying around so step-by-step in addition this tile was persistently running hot so it was stopping down the speed all the way down to like 300 megahertz and so I which in fact when you when I have the the splits at the end of the universe running you can see this guy was becoming a jamming place that the they were all clogging up in the middle of nowhere now it's going what's going on here well it was like this guy was running 300 megahertz while all the tiles around him were running at 1 gigahertz so I took a look in this was one that I had built early and it didn't have a heat sink in it so I opened it up and this is a chance especially for folks that have come to the T2 tile relative the recently might not have ever actually seen close up what it looks inside this green board is the seed studios Beagle bone green a particular version of this Beagle bone which uses a Texas instruments processor there it is here's the circuit board that were that we designed for the project and one of those things right there th1 that is the temperature sensor for that's the center temperature sensor there's that gives you a sense of how big it is and then where it's located so when you put the thing down below it when you flip the Beagle bone green over put it on there's still a gap but it's fairly close it's sensing the thing and in fact there's just enough of a gap that I found these little heat sinks these little stick-on heat sinks that I have now been using when I do all the assembly so when I built the power zone to a couple of weeks ago they all got the heat sinks on it well now this guy's got a heat sink on it as well board goes back on you have to really push to get 96 pins to set and then I put these screws and you know it's not like the this board is gonna vibrate out of 96 pins but step by step just be careful put the screen back on put it back in the case that we've seen plenty of and then plug it into the hole in the grid where it came out of and put in the in this case PD connectors power and data plus one PI the power injector one okay one additional thing we're gonna take our 20 minutes it's like the good old days I had all of a sudden power zone to flashed and would not come up and it looked like the power supply had died I had made a new power supply put a new thing on it from to try it out because I was having I needed two power zones on the two power supplies for the grid and I had my third one for my own little guys this was a why splitter that I had made so that would be two options so that from a single power supply you could plug into a single power zone in two places well why do you want to do that well it's because you know just like we saw if you want to remove a tile that's got the one and only power injector on it you're gonna have to put the turn down the whole power zone in order to get at it whereas if you're plugged in in two places you can always unplug one of them to get at a guy so that's why I'm going to be using why splitters on each power supply so that each power zone has two landing points for power from the power supply this was one of them it seemed like it was okay got closer no look at that a little bit of wire a little loop of wire kind of coming easy so they kind of work around it was shorting out the power supply and the power supply thank you very much was detecting it and cutting the circuitry so nothing in fact was damaged and I and so I did it I did a new I redid that much more carefully like it's pretty good at this point and then I went shopping to find one to two man appropriate male to female splitters what one challenge is most of these splitters are limited to three amps of current and I'd prefer to have five amps so that's why I was making my own we'll see what happens going in one interesting thing while I was shopping I was also look reading this book called Systematics which is it's kind of a corny book about systems but it's got a lot of really fundamental truth in it as well any large system is going to be operating most of the time in failure mode yeah well that resonated for me last few weeks oh and another one of its little pieces of evidence applied system antics loose systems last longer and function better that's kind of a whole slogan for best effort computing for what we're actually trying to do here so that was nice to come across all right that's the hardware story the few the proud the LCF nerds thank you folks Martin from France is our latest LCF nerd also I just want to shout out again to Isaac and Andrew because they're actually doing monthly contributions and I see it every month when it comes it's like oh no you know what have I done so thank you I thank you folks so much step-by-step next week got to get the pipelining working and use the pipelining with the improved debugging to get back to intertile events that's it I'm gonna say sorry I ran so long but I'm not gonna say that there was just a lot of stuff to talk about I hope to see you next time I hope you're doing okay have a good two weeks