 We'll get started now, Julian, talking about time and synchronization. Right, so I'm going to run through this pretty quickly because it is a talk that's actually meant for a slot twice as big as I got, but I can do that. So, clocks, of course, a classic thing we've been developing over many years. The first real time reference we had is, of course, something that happens about one pulse per second. Heartbeats. Now, from here on all the rest are commercial clocks, essentially, but it's odd that the resting heartbeat is one pulse per second. Or, of course, it might just be a convenient number. So, over long-term rest, sleep, you actually get within about plus or minus 10% of 60 beats a minute. So the first level of commercial clocks, basic mechanical oscillators. This is one from the OLPC's educational resources, so they're good for more than just laptops. It's a simple mechanical oscillator, accurate to about 1.1%. And then you get better optimized mechanical oscillators that go to about 100 parts a million. These were used in the early days of radio as a frequency reference, being that clocks for a very long time were only usable as long-term references, not short-term. So they needed frequency references that were a bit more useful. Next level of accuracy, we're getting into the power grid. The power grid is an interesting case. Over a very long term, a couple of days, it's accurate to about 10 parts per million. But the power grid's slightly odd. The way you indicate that the power grid needs more generators is the frequency slows down. The way it indicates the frequency scales up. This is just an artifact of AC synchronous motors, which is what essentially every generator is. One of the classic horror stories of power operations is people who forget to phase lock their generators or don't notice that the phase lock lights had blown, hit the sync to grid button and watch their generator wretch itself off the mountings, which may be several inches of steel. Rather neat, you can go to dynamicdemand.co.uk and witness the live frequency of the UK grid. It's kind of interesting to watch over a couple of minutes. You can see just how variable it really is. So the Australian grid is specified to be plus or minus 0.08%, 99% of the time. Next we get into the actual useful things, crystal oscillators. These are the bread and butter of just about everything. You probably are walking around with a dozen or more of them. In a lot of devices, mobile phones, they're used as the CPU clock and as the startup for the mobile clocks. They are later synchronized from the mobile network. Your laptop has them. Pretty much any digital video device has them. They're everywhere. These are about one part per million on average. You're losing a second every week or two. So the interesting thing about these clocks is they're generally quite highly temperature affected. This is where servers get a real benefit because despite having really cheap oscillators, because why on earth would you buy a good clock for a computer? They happen to be in temperature-controlled rooms. And even when the temperature is a little variable because of, say, daily load patterns versus air-con capacity, over time that ends up averaging out really well. And a server in a good air-controlled room can be surprisingly accurate after a long period of aging. So now we're back into the old clocks. A very good watch. This is, of course, the classic mechanical watches. It's about 0.1 parts per million, three seconds a year, which is just phenomenal. The very best mechanical clocks, the laboratory or more observatory clocks, really, grand clocks, are about 0.01 parts per million, losing a second in three years. At this point you are isolating for temperature, humidity and pressure. You need to understand the concepts, mass, friction, metals, how metals interact, escapements. You need to understand vacuums and how they interact with metals and mechanical systems, and, of course, suspension from things. What they actually are then is gravitometers, which should make it pretty obvious... Oh, sorry. And this particular clock is the short-free pendulum clock, which is believed to be the most accurate mechanical clock ever made. So, of course, being gravitometers, all they're measuring is our next better clock, which is the earth. In theory, average it out long enough, and you're at a part per billion. In practice, you're about an order of magnitude worse, and that change is why we have leap seconds, which I'll explain that. It's not really regular, it is slowing down, but if you average long enough, and that's the thing with a lot of clocks, if you average long enough, all this stuff is surprisingly good. Now we're heading towards the lab standards. The next level is still a quartz oscillator, but it's just inside an oven. Because if we put it inside an oven and insulate it properly, we can get far better temperature stability than even an air-controlled room. This is a HP 1011, one of the most common oscillators out there. Each side is about a 10 centimeter cube there. These were very commonly used in the earlier days of cellular networks, or early days of digital cellular networks before high accuracy atomic standards and GPS standards became practical. These range in price and size from about the size of a matchbox all the way through to pretty much that's one of the larger ones actually. So the next step is we move into the atomic clocks. Those clocks, I can't even remember which company owns them these days. At the time those are Ephratom, Elpro, Rubidium clocks. These days they go for about $60 each after coming out of US telecoms. So that's about Yay Beak. I had one here, but you're not allowed to bring radioactive devices on planes. And I forgot to ask some of the people who are driving up the rocketry stuff to bring it for me. There's more. A lot of lab equipment where you need really accurate frequency standards, you can buy a Rubidium clock option. They're pretty neat. And the next step up and coming to the atomic clock, the classic thing is Rubidium is an atomic clock, hydrogen is an atomic clock. Acesium beam tube, and I actually have to qualify that as of last week, is the atomic clock. Acesium beam tube is the standard for a second. It's, depending on how you say it, by definition a working cesium clock is correct. This is a HP 5071A. It is the most accurate commercial clock ever made. HP sold off all the timing and measurement to Agilent or Splitoff. They now got bought by Symmetricom. This product, who in fact already had their own atomic clock models, this was designed in about 1997. It hasn't been beaten. It's an incredible piece of engineering. And we can go better. The next step up is the hydrogen maser. This is a lab one, not a usable office standard or lab standard. Okay, well you don't all have mad scientist labs at home. Just some of us. These are better than a cesium over short term. They're not as good over long term. No, basically. Every single person will have different definitions. Short term can be anything from a couple of an hour or two up to a couple of days, and long term anything longer. But it always depends on exactly what you're meaning as to what short term and long term is. In some cases... No, short term usually means at least an hour, which is the smallest I've seen at reference to. Now, something new came out last week. That is a Symmetricom. That is a cesium clock. The coin on the right there is a vertically flipped US quarter to give you a sense of scale. That is a cesium atomic clock. Because of some differences in how it's presented, it is not a cesium standard. It's just a cesium clock. But it's still better accuracy than any of the commercial quartz clocks and in just an amazing size. If you want to buy one, they're about $1,500 US each. It's kind of crazy that people are seriously talking about buying a bunch of them for their own experiments. Oh, and by the way, the HP clock, those are about $30,000. The cesium tubes inside, if you're lucky will last for 10 years and are about $15,000 for the good tubes. So we've got the clocks, next we synchronize them. Starting out we had master clocks. You give them a pulse on the hour to jump forward. You have a grand master per city, a master per site, a submaster. You basically stagger whether you sync on the hour on the half hour. Essentially you just hope that the clocks aren't out by far enough if you've missed pulses that you can't just use a big fad electromagnet to pull them in. That's all they were. I suggest you read Frac, P-H-R-A-C-K, in Classic Hacker Magazine, issue number 30, which talks about the history of the Western Electric Clock Company, the Western Electric Priority Clocks, and that clock, it was one of them and is made by the self-winding clock company of New York City. The Frac issue also goes into the fun had for daylight savings changes because they only sync the minute hand, not the hour hand. So the actual standard today is pulse per second. Same basic idea, we just do it every second. Usually you set a negative delay on the sending device to compensate for cable length because you're synchronizing it at nanosecond levels. The next step is something else called IRIG-B, which is 100 hertz signal, or 1k hertz, depending on which variant you're using. You get the time, you get the day of year. NASA used it quite heavily and developed it for their launches. If you saw the video recently uploaded to YouTube of all the wonderful footage of the shuttle at 400 odd frames a second, that was IRIG-B timecode, they burnt into the images and they mentioned that in the video. I have a lab clock at home which is plugged into that for my desk display. The next, but unfortunately I don't have any rockets to launch off it, 10 megahertz and 5 megahertz signal square waves are commonly used as lab frequency standards. They're not true time standards because they don't give you a position of the second. 10 megahertz is generally newer gear, 5 megahertz is older gear. You can actually find gear which is 10 kilohertz for the very old gear, but that's really odd. Often you'll have a 10 megahertz input for frequency and a one pulse per second for time. There's a variety of time codes, SIMTI, audio timecode, VRTC, video interleave timecode, audio video film production, accurate to a subframe which can be up to a hundredth of a frame and a frame is usually 24, 25, up to 60th of a second. There's also techniques known as bi-phase and tack for speed which is where you record phase offsetted sine waves or similar signals and basically you can electrically identify how much to speed up or slow down multiple recorders to synchronize them. Word clock, SIMID given to digital audio. It's an AES audio signal. You can think of the SPDIF digital signal you use. It's just all zeros. Stuff we care about, NTP, network time protocol. This is an old protocol, 25-odd years old. You can get accurate to less than a hundred nanoseconds. Long term, with an LAN. You can get below a millisecond over the internet if you're not going over ADSL links. ADSL links basically ruin your ability to get time synchronization. There's a bunch of high quality public servers, pull.ntp.org. If you have bandwidth and you have decent clocks, consider joining the pool. NTP is not related to the daytime protocol or to the time protocol. We just have to have three different protocols. You might have heard of stratum. So in NTP stratum we have a master clock, which is stratum zero. Stratum one clock is the system running NTP, which is one strata away, et cetera, et cetera, et cetera. The problem is NTP stratum bear no relation to the system they were developed of which is a telco stratum system, where you have stratum one as primary reference clock which means essentially an atomic reference. 10 to the negative 11 is only achievable by lab grade atomic references. If you need it long term it's achievable by GPS clocks. Stratum two is a secondary clock, stratum three, so on and so forth. The accuracy reference is actually in holdover state. So except for stratum one which is a primary clock, everything else, the accuracy only matters when they can't talk to the primary. If anyone has to deal with these, hello, please hire me, that would be cool. Sonnet, SDH, cable networks, mobile phone networks, all this stuff uses it. It's kind of interesting. So PTP was developed as a replacement for NTP. Version one, you could only use over LAN over single hop. Version two, they made it routable. But in the process it's not much better than NTP. It doesn't really have serious deployment outside labs, although I'm seeing it gain some traction in the telco space. That's kind of neat. All right. So what do you actually care about? Why have accurate time? Well, users will probably complain if your clock's 10 minutes out. Kerberos breaks if it's more than five minutes out. Well, let's face it, users will complain if you're more than a couple of, more than a minute out. Users will complain if time in email bears no relation to the email. The fact that it bears no relation to what they think is a different matter. Why am I getting email from the future? Log analysis can really, really hurt. We have some big squid clusters. We need to have systems within less than about 15 milliseconds because a single page of log data from one of our squid hosts covers about 30 milliseconds. And that's using my excessive 150-odd line terminal. So the hard-locked systems are the easier it is to correlate log data. So there's a lot of time stands, it turns out. Wilma GMT, which doesn't really exist anymore, it's actually UTC, which is the civil standard. There's UT0, which is astronomical time. UT1, which is mean solar time. UT1R, which is smoothing out the tides. UT2, which smooths out the season. UT2R, which smooths out the tides and the season. And TAI, which is atomic time. And it's basically UTC, but with no leap seconds added. Then there's time zones. There's a couple of databases of time zones. TZ database is the canonical one, used by just about everybody except Microsoft, and parts of IBM. Keep it updated because daylight savings time changes multiple times a year. In nine months working at a company, I had to change daylight savings time settings three times because of Australian government changes. This was for Australia-wide deployment. You can just pull it automatically. There's a bunch of things there. Daylight savings, usually in our forward and summer. Lord Howe Island uses 30 minutes. Changes of 20 minutes and two hours have been observed. And there's silly time zones out there. Adelaide is half an hour out. We have 45 minutes out. We used to be fair, that's the worst of the current time zones. We used to have 20 minutes. We used to have 25 minutes. We used to have 44 minutes. But actually that wasn't correct. We used to have 43 minutes and eight seconds. Exactly. I chose the most amusing out of the Wikipedia page. There's about 30 or 40 ridiculous ones out there. So if you actually want to deploy things, what should you do? If you want to have accurately. Have two or three physical hosts with good clocks. You can buy cheap arm. You can buy the little, secure boxes. Sync them off a nearby stratum one or two NTP server. Don't use between three and five servers. NTP's algorithm goes weird otherwise. Perhaps use a GPS. Perhaps use a season or rubidium clock and a GPS. Always have a source that is either GPS or NTP from a trusted source. Because otherwise you may miss leap seconds and similar. There are appliances that's a TS 2100. They're neat. They're expensive. If you have more than a thousand-odd clients, you may need to do something with distribution. Older NTP appliances may need about 50. Again, sync the three to five masters no more. You get the idea for slaves. Most NTP configs have a local clock. Turn it off. Your masters have a justifiable reason to leave it enabled, but only your masters. If you use virtual machines, you are screwed. This is a general consensus. You're lucky if you get less than five seconds skew over 24 hours. It's getting a lot better. These machines are not timing references. I think I'll leave it there. If people want to learn more about managing time in applications, come and grab me during lunch. If people want questions, come and grab me during lunch. Otherwise, I think that'll be it. Thank you very much.