 One of the dangers of working freelance, as I have done to the last decade, is the temptation to start viewing yourself the way that your clients tend to do as a wizard, ensconced in your high tower with only your grimoire in your palantir, working in beautiful isolation to bend reality to match your will. The danger lurks in how this view can feel quite accurate, 90% of the time, but sealing yourself off like this all the time can block even the most experienced attacker from realizing some of their best work, which requires not just showing up at conferences and such, but truly venturing outside of one's comfort zone, taking the initiative to reach out and ask for knowledge, access, or perspective that will never come through a mere search query. In November 2015, I noticed that the flight radar iPhone app, somehow new ahead of time, that the plane I sat on would take a slight left turn over Atlanta, implying that the app had access to commercial flight plans. In a flash, I saw that if I could somehow get this data to, then I could write my own app, one that would use weather data, such as that used by NOAA to generate maps like this to predict turbulence, location, and intensity during flights abound to nervous flyers like me. And so as with any project, I began by retreating into my workshop, assuming that I could apply all my usual methods of isolated study and experimentation until the day I shipped, just as I would with any client-driven task. Obviously, I would need access to two data sources, one for weather conditions, and one for flight routes. My idea being that for a given flight, I would just run the ladder through the former and show you the numbers somehow. And my natural introverted starting point was, let's find the public APIs. Surely there are public APIs. I started with the flight plans. And to my surprise, I didn't find any obvious APIs offered by the FAA, even though I had a notion that flight plans were public information. I did quickly find a commercial API offered by a company called Flight Aware, which offered me exactly the information I wanted for the low, low price of 2 tenths of a cent per query. OK, so noted. Put that aside for now. NOAA, they have public APIs. One even involves air turbulence. It is a lot of fun. You can give it a path of geographical fixes and a timestamp, and it'll send you XML telling you all about the air vents along the way. Air vents look like this. They're big old or regular polyhedra floating in the air with floors and ceilings defined at certain altitudes. They define a 3D space that contained an increased likelihood of atmospheric conditions for aircraft passing through them, including turbulence. Sounds perfect. And I drafted a prototype out of it. And some friends tested it, but it was clearly too broad to be a useful predictor of anything from a passenger's perspective. What I wanted was something like this. NOAA's graphical turbulence display, which I've used as part of my own pre-flight meditation for many years. And friends, I sunk many serious hours into the puzzle of downloading and analyzing these images, pixel by pixel, before I even began to consider the possibility of doing something with those contact links down at the bottom. Now, my wife is a public servant. That's her on the left. She works as a librarian for the US Navy. And upon hearing my image analysis woes, she said, that contact form is probably monitored by someone who cares deeply about their work. Seldom hears on its curiosity about it, and he would be absolutely delighted to hear from a member of the tax-paying public for whom they ostensibly labor. This struck me as counterintuitive, given my own typical private sector response to unsolicited requests for attention from strangers. But I like to think I know when to defer to experts. And so I filled out that contact form. Feeling optimistic about the AirMed API, I asked if they made their graphical turbulence data available in some plaintext format. XML, maybe JSON. And I got a response almost immediately. And they said, JSON or XML files, yeah, sorry, we don't have any data like that. But we do have an anonymous FTP directory full of up-to-a-minute weather data using a format that meteorologists use called GRIB. And here is all the information you need to get at it. Would that be useful? And I said, I will let you know. And I retreated to my tower, which that particular afternoon was actually the Blue State Coffee House in Alston, Massachusetts. And there did work. I've no doubt most anyone in this room could have done. And I had a great deal of fun with it. Does there already exist a suite of open source tools specifically for working with GRIB data, yes? Was it already packaged up for Mac-O-S Homebrew, yes? Did it feel indescribably refreshing to work with open source tools that process data for a purpose more focused than data processing's own sake as so often feels like the foundation of my career? God, yes. And my discovery of this territory would have been impossible had I not jumped the track and contacted someone from outside of my usual sphere. And so I ended up that afternoon with this. And I felt jubilant. And I posted it to Twitter, jitterbug being an early code name for this project. And my friend Jenny said, is it a tumor? And I was like, no, Jenny. But yeah, OK. I didn't really provide any context, so that's fair. But no, it's conus. It's the continental United States, or rather a slice of the atmosphere above it, or rather forecast of clearer turbulence intensities at a particular altitude at a particular minute on January 16, 2016. I was able to read the data well enough to create this image. This is the moment where I knew this project had potential, which left a problem of the airplanes. I had enough of that given a takeoff and landing times and a line of navigational fixes, I could estimate what conditions surrounded the plane at any given point on its journey. But asking users to look up and type in nav codes of their upcoming flight versus just a flight number was a non-starter. As an aside, I learned a lot about aeronautical navigation points for this project and how there's no good public database for them all. But you can fake it well enough by downloading the free trial of the super nerdy X-plane flight simulator and rating its data text files. But that is a tale for another time. Back to my tower, which I wasn't ready to leave again just yet. I returned to my old friends at FlightAware, they have the commercial API. And I wrote a screen scraper of their web app, which, among other things, turns flight numbers into navpads. This would not scale to production. But it did let me complete my proof of concept of which me and some friends were the only users. And yes, that's plotted against an Apple Maps screen grab. It was there and it was good enough. But now progress slowed down. Never have I felt so close to another soul, writes Randall Monroe in XKCD 979. And yet so helplessly alone is when I Google an error and there's one result, a thread by someone with the same problem and no answer last posted to in 2003. Naturally, I wasn't the only jerk on the internet who wanted FAA data and indeed found many instances of people asking but no clear answers. I did find some tantalizing almost rumor level pointers in multiple years old forum posts that link to the same URL. And clearly, there used to be a website there. But now it just held a 2011 press release describing a random policy change like a fading concert flyer tacked to an old warehouse. However, it had an acronym I had not seen before, ASDI, and a reference to something called the FAA-ASDI data feed. And that gave me another handle to search on. And I found, for example, this document listing users of ASDI, whatever it was, which included my friends at FlightWare. OK, right track. I'd also seen mention of this class 1 versus class 2 designation, which would come in handy later. Finally, I found this PDF, sitting in a directory containing no index files and on an FAA website otherwise dedicated to flight delays. That search engines had indexed it, bespoke a dignified pass for the document, and I could only assume its continued existence in organizational oversight. I did not let that stop me. I had learned my lesson earlier. I wrote that email address printed on it. Like a tourist speaking from a phrase book, I pieced together vocabulary I had encountered but not necessarily fully understood. And I said, hello. I would like to request information about obtaining class 2 direct access to the FAA's ASDI data stream. A week later, I got a response. And it said, ASDI, we retired that. Have you checked data.faa.gov? You know the website we have that is literally covered in giant green buttons, the very first one of which is labeled en route flight and related data. And I said, how is your robots.txt file? You know what, never mind. Thank you. And friends, I hit that button. And within days, I had an assigned contact within the FAA to help set me up with its full fire hose of flight plan and aircraft positional data. And there began a months-long adventure of personal growth both as a hacker and as an entrepreneur that lies far outside the bounds of this talk. At the end of it, and indeed at the end of last year, I launched bumpyskies.com, braiding together all of this work. The service feels like a stub to me full of potential. And I don't know where it'll go next. But I do know that I wouldn't have gone anywhere had I not forced myself needing access to tools to just ask for them. It is a trick I look forward to repeating sometime. And I hope someday you can make use of it as well. Thank you very much.