 All right, so hacking a time machine kind of sort to maybe What do I mean by a time machine and he guesses a clock? Obviously a clock a clock is obviously a time machine right because it's a machine and it tells you the time So it's a time machine Yes perfect, let's talk about time A long time ago in a galaxy far far away wait that Star Wars never mind I wrote you guys a little nursery rhyme for this You can read most of it. I'll just read the last bit out to you because it sounds better when it's read out Oh, McDonald had some clocks. They weren't in sync. Oh, no Right, that's a trouble with clocks. You have all bunch of clocks in your room. They've not been here because it's not a school apparently But you have clocks normally clocks Have some problems one batteries run out, right? You have to go around replacing batteries not fun Synchronization is annoying. You have to make sure your watch is reasonably accurate walk around Synchronize everything down to the second and then press go on well I'll press but like push the little stick in that makes the clock start turning again annoying lots of work The other thing about clocks is you can't really monitor whether it's running out of battery or if the time's wrong You kind of have to send someone around at Some specified interval to every single classroom to look at the clock waste of time if I could tell which clocks weren't working I could just send the person to the four places that the clocks had problems instead of sending him to like Here are and number of classrooms go to every single one of them And the other thing about clocks in general across the board is that time drifts time drifts a lot less in Slightly less cheap clocks shall we put it that way, but it's still drifts And you want to deal with that so we're like, all right We want to do something slightly different from what we normally do in a school I'm sure some schools already have this set up had this setup when we installed it two years ago, but either way we were like, right I Found this online. This is a clock From Shanghai Global Time It's powered over powered over at the net know much as I was like is this my savior. Well, let's look at it Doesn't work for my use case Do I have batteries? No, it's powered over the net Does it have time synchronization? Yes, it has NTP. That solves the issue of like Hey, I have to go to every clock and set it to 13 0104 or something monitoring and We'll build something. It's okay. We can get around it. It's it's kind of like on the internet Well on the local network, so I'm sure we could do something, right? How do you tell the IP of a clock like this? If you were a reasonable manufacturer, you would have the MAC address on the box on the device Maybe a multiple parts of this device. Well, there is no MAC address anywhere on this device I'm sure you could take every single clock plug it in and figure it out, but You can't of course you could have the device reported to an IP address How do you do that with an analog clock spin spin spin stop at 10 spin spin top at stop at 21 Where's the 21? Okay from right and you could use seconds, but then IP addresses each Octet goes up to 255. How do you get that with? 60 right it also has more problems It really is just a motor glued to a standard off-the-shelf regular wall clock Which has a double a sock a double a? Battery socket into which they sold it to very flimsy cables. It was amazing looking at this and I was completely freaking out Also when you set the time I hope you like watching needles go in circles It has to if you are two minutes ahead of where you should be you have to go through a full 12 hours to get to the Where you're supposed to be not fun All right do a little more digging these lovely guys global time My savior 2.0 What horrors lie beneath your facade? Well, let's find out does it work 2.0? Yes POE Yes, NTP monitoring. We'll figure this out at least we can get the IP address now You power this thing on and it starts off and you look at it very hopefully because it goes 9 9 9 8 8 8 Down all the way to zero zero zero zero zero and then it goes the ACP and Then it waits and then it goes one nine two one six eight one two and you're like that's not We use ten slash eight was why is that there and then it goes DHCP now? It actually worked ten twenty one blah blah blah and you're like okay cool at least we get the IP address But that's all generic stuff on a more concrete level. What do we actually want? Let's talk about features. What features do I want from a bunch of clocks? I'm going to have across an entire campus and don't want to be running around touching ever Not that I'm going to be the one touching it But I don't want the people whose job it is to be running around poking these things to be going around poking these things if they Don't have to So there's two mandatory features that we need and something like this. Oh Apple power out Yep, all right clicks Two mandatory features. I would need to be able to set the time zone because when it ships from the factory It's at UTC Now I love UTC. I wish the entire world used UTC. I wish it was instead something like you start work at 00004 or like, you know at like 12 in the morning 12 in the morning UTC you start work That sounds lovely to me to most people they're like, what do you mean? I want to start work at midnight like but it's 8 a.m Anyway, we need to be able to set the time zone because kids don't get time zones Especially because this is going to be the same from like kindergarten all the way to 12th grade 12th grade is they know They're time zones. They still don't know UTC. They know GMT, but GMT is a pointless time zone The other thing we need to do is to be able to set NTP servers The clocks ship from China Their default NTP server is in China. The latency to the NTP server is roughly 700 milliseconds NTP is designed to deal with latency on a network I don't think it's designed to deal with 700 milliseconds of latency on a network And then there's a bunch of features that I like to classify as I really want this But I'll make do if I can't get it features I'd like to be able to check if the device is configured and I'd like to check if the time is drifting All right, let's talk about configuration. If you want to configure a clock Well, how do you configure a clock in your mind the most ideal way of doing it? Well the HCP Most of you know DHCP is the thing that assigns you an IP address. Well When you work on a large network you use the HCP for a lot more than just the IP address There's a whole bunch of information a configuration information that gets delivered to your device over the HCP and RFC 2132 Thank God has time offset and time server options and boom the clock should configure itself, right? But we're Shanghai global time We do things our way and this is our way It's a lovely little app if you're wondering why the colors look a little weird If you look very suspiciously at the Mac bar on top and you think it's because I'm running it on wine It doesn't matter running on windows. It looks the same Quick rewind how many clocks are we talking about? That number's a bit small. Let me make it bigger for you That's how many clocks I need to configure. Am I gonna sit there poking at a GUI app? Nah, what do we do? We sniff it Obviously talks to the clocks over the network. We sniff the network We set the time zone, right? And you get a bunch of stuff and in that time zone Setting giant bunch of packets eight packets. I think We look for the string. Oh wait. Oh and in hex all this is next. No, it's not there, right? So clearly it's not there Maybe they send it as a string because we're Shanghai global time. No that that string is not there either How about the number of hours between I? Don't know but like, you know eight hours. Well, how about that in minutes? Nope. How about that in seconds? Nope Well, okay fine. Let's try again NTP service These should be easier to find because there's no really weird way to encode these things, right? There are only two possible ways you could encode it host names or IP addresses neither of these are in the packet congratulations is There a method to the madness. Well, this is where I went down a rabbit hole For maybe two days and then I went does it actually matter? Do I actually care? Are we getting ahead of ourselves? You have a bunch of packets. You don't know how they work What's the first thing you try? Send the packets again try replay attack. Does it work? There's your answer. Lol. Yes, it works So initial setup is fixed Initial setup is fixed Not really We have three hundred twenty one clocks The firmware locks up if you don't send it the exact eight packets it expects if you send it seven It waits for the eighth if you send it nine while it waits for the next seven until they come in And it just completely stuck so you can't really do anything that might be freaky also, there's a timer and if you send too many packets and they reorder because it uses UDP It overflows this network buffer and the firmware crashes and kills itself It's great. Trust me. Thankfully they put the network part of it separate from the actual clock part of it So the network can go down the clock still lives Also, these things are spread across campus and we don't use a level two broadcast domain across campus because that's done So I need a way to I can't just say send this to like the entire broadcast domain and the clocks will configure themselves doesn't work So What do we do the ACP to the rescue again except not the way you might expect by sending good options or something No, I look at the ACP server logs and I go all right. Is that a clock? That's a clock configure it and you do that for every single clock you see It's kind of annoying, but it works. So initial setup fixed, but wait. What about monitoring? Well before we do that Let's take a little detail neat tux. This is tux If you encrypt tux You get that It looks very pixelated there. It really is just that anyway But if you use a specific type of encryption called electronic codebook ECB You get tux Can we see the penguin? Can we see the penguin? It's kind of small for you guys, but Just going based on shape alone if you're closing up you realize that a lot of it looks very very similar So going back to my earlier question. Is there a method to the madness? Well, is there one? It looks like some kind of sort of codebook It's possibly a simple XOR operation, which if you think about it is an ECB with a block size of one byte so That brings us to the question. How is the key determined and that brings us to the same answer I had earlier after spending two days done some rabbit hole. Does it actually matter? Well, in this case, does it actually matter? No, because I see a whole bunch of 4d's here a whole bunch of 79's here a whole bunch of 22's there What if those are the keys because the packets are mostly null bytes? Well, you get that you get a whole bunch of zeros. All right, that looks kind of reasonable now the next question Can we verify the keys? Correct. Well, there's a magic number to start of all the packets that lines up. It's kind of good news It's actually magic text GT for global time B spate B zero one and God knows what that is because that changes depending on type of packet Either way, we can now make sure that a clock is configured by doing two things get the config Verify the config to get the config we do the same process We sniff what the crazy windows application sends Which is that but this is in Python a little more condensed mostly because I didn't want to show you guys a giant block of text Interestingly enough this 5f I can't actually change this padding that it uses without also changing these two because these two will need to Be zero after being exored together. This number somehow magically changes There is some craziness to this and I couldn't be bothered to figure it out. So we try it We got a deal for skater config. We see things we like We see things we don't like Default passwords by the way lovely Right. So want to verify it fairly simple. We pull the config We check if the bytes 0 8 0 are there my first suspicion was correct for how they encode the time zone By the way, it just turns out it was exored Make sure that everything lines up great. It's configured. So looking better review of the features time zone set We can do it ntp service that we can do it check your device configured. We can do it. What am I checking drift though? That's kind of annoying, right? So can we pull a clock time and then be able to go? Hey is it set up or not if it's within some Buffer that you like you go great. Leave it alone. Otherwise you tell it to re-sync, right? Once more with feeling we repeat the process. Where is the time in this madness? Here is the time in this madness This slide intentionally left blank What about our old friend the windows application tab number one? I don't see any time here Tab number two. I don't see any time here tab number three our clocks don't even have Wi-Fi What are you doing here tab number four here? I can change the password. Oh wait, that doesn't really help me So even the monstrosity can't pull the time. This is how things are looking right now So we can't really check the rip, but it's all right. We'll live the clocks run ntp, right? What could go wrong? well Weekend we get reports the time is wrong and Things break right so when things break you go into a place like this you grab a new clock you plug it in But if you've got an important step configuration There's a whole bunch of technical solutions like I could have come up with for this But the simplest one was a non-technical solution You take a new clock you want to put put a new classroom you plug it in on the IT floor The server configures the clock you can now plug it in anywhere problem solved right policy solutions Well, are we done? We think we're done, but then someone comes up to you the next week and goes the time's wrong But only by a few minutes. It's five minutes off, but that was the entire point of the exercise I didn't want different classrooms in different times. What the hell this guy? I Thought there was ntp. This is supposed to solve everything Unfortunately, the only ntp had boot in the did drift What do window solution reboot? So what do we do today? I'm running low on time. So I'm going to speed this up check of a floor's clocks in the IT floor If it is great configure it Any clock that goes off the network it probably still works. I'm going to assume it doesn't you're a dead clock Someone's going to come around and we're going to power cycle you first remotely We can do that because PoE power of earthenet, but if that doesn't work we'll send someone down to fix you and To keep the sink problem in check every semester every holiday actually we just reboot the clocks because they sink again It's lovely right now everything works So the trouble with the original clocks we originally had three of them really solved time-drifts kind of sort of solved I mean it gets kind of bad right before the holiday then it fixes itself and Then you end up with 321 of these guys all in the same time much unlike this picture That's all I have