 Good talk, good luck with that. Tag teaming civic data, I'm Liz Rush. And I'm Singh Weisoo, and we are members of the first cohort of Ada Developers Academy, which is a non-profit code school in Seattle focusing on training women making a career shift into tech. It focuses on web development using Ruby on Rails, as well as general software engineering principles such as software architecture, test-driven development, and agile. It's a year-long program split into two six-month phases, the first phase being in-class instruction, and the second phase is an internship at a Seattle, a local Seattle tech company, which ends October 27th, at which point we'll be available for hire. Our six-month classroom experience was based around Rails projects that we did in short one- to two-week sprints, culminating in a final four-week-long self-directed capstone project. Singh Wei and I had previously worked together on a two-week-long Rails app, and that's when we started to discover where our individual interests and strengths lay. Liz likes front-end, JavaScript, and UX, whereas I'm more back-end-oriented and must mess around with databases and APIs, and we both really enjoy the challenge of architecting a project together. So we decided to pair on our capstone, an application that would use the city of Seattle's open data initiative to provide users with a way to discover street parking restrictions in their area. So I know this isn't much of a problem here in Portland, but if you've ever tried to park in Seattle, you know this game. Yeah, our street parking signs are nearly impossible to decipher. In this particularly egregious example, it appears as if every sign contradicts the next one, and as a final cruel joke, they tacked on west of here when, even though the direction the photo was taken, we are all east of the signs. So where does that leave us? Yeah, so after we hit this issue ourselves, we thought it might be an interesting problem to try to solve for our capstone. We had been granted a week of free play in ADA before our four-week-long capstone started, which I spent doing a research spike. Simply enough, I figured I'd Google to see if somebody else had fixed this problem for us, but there were no mobile apps. There wasn't really anything out there that was very friendly to use, but we did find out that the city of Seattle has a parking restrictions map on their website. So we also investigated the different data sets that were available through the open data initiative, and we also managed to get a contact at the city of Seattle, not in their transportation department, but a data engineer on their web team. So this is the way the city of Seattle government implements a map showing parking information. They use ping overlays on map tiles to show different parking categories. And as you can see, it's not exactly the easiest to read, and the categories aren't really obvious, and also the data is static. We thought we could do better. Our idea was to use the open data to display the parking information in a much simpler and more human-friendly way, for example, rendering just green lines for available street parking and red ones for no parking. So we were pretty excited. There was open data available. We had a contact in the city, and we had a starting point to improve on. Our morale was pretty high. We had discovered our manifest destiny, if you will, and we solid forth. But knowing that this would require some help from the city of Seattle government, we called our contact. We explained our idea, what we thought we could do, hoping to get some feedback or tips on our journey. And he said, well, if you think you can build it, you'll be the first. Good luck with that. Ha ha ha, ha ha, ha. And then hung up on us. So, in spite of this ringing endorsement, the guy did provide a link to their ArcGIS server, which was what their web application was using to generate the parking maps. Though we had never used an ArcGIS server, the Esri website told us all about the tools ArcGIS provided, which included a way to dynamically generate the KML files that we needed for our project. So, this is perfect. Except... Turns out our perfect resource wasn't really so perfect after all. As you can see, there were quite a few implementation problems. Almost every informational link was a 404. And many of the links produced empty files that we couldn't use to extract any data. At this point, not knowing what we were doing, we reached out on Twitter to see if we could get other developers to help us. And we managed to get back a link to a fully implemented demo ArcGIS server from a developer at Esri. We found out that there were actually a lot of other parts to an ArcGIS server that we didn't have access to, such as projection tools, coordinate conversions, and more. Since we didn't have this on the Seattle server, we called up our city contact again, and he told us that the person who implemented this version no longer worked there. They had no idea what was going on with the map server, and that they'd be unable to provide us with any guidance or help. So as we mentioned, the city of Seattle used ping overlays for their web app. But in order to implement the features that we wanted, we decided to use KML files or keyhole markup language files, which are an XML type formatting for geodata that's used by Google Earth. When we did try to get KML file from the map server, this is how it actually rendered. So, yeah. So at this point, with not much luck getting the KML files that we needed from the map server, and with two weeks to go, or a long way from Oregon, we were starting to get worried that we wouldn't finish our capstone project in time. So we compromised on our original idea, and decided to use the ping overlay instead of the KML files to get a minimal viable product. We thought that this would be much easier to implement since this is what the city of Seattle government did. So here's the ping overlay we managed to generate from the map server. We were working on this project from downtown Seattle, and that's an overlay of downtown Seattle, and it took us a little while to struggle with the query parameters, but we were really excited to see that we were able to successfully generate this ping. We forged ahead, we spent the next week building out our backend, trying to architect a solution to store all of these, and then we eventually discovered that the graceful fail looks exactly like the success. So, yeah. So faced with this problem, we decided to snoop around the city government's code to see how they handled the same issue, and the answer was, they didn't. As you can see, they simply put in a comment in the else block that says, we didn't get any parking results either, so let's all just go home. Here's my favorite comment. What do here? Toodoo! This is silly. We're not trying to tear down the city's code, but at this stage of our project, after so much frustration, it was actually sort of comforting and validating in a way to realize that even real developers don't always know what they're doing. And here's one last gem. This caused great grief. We feel you guys. We feel you. So we eventually got to the point where we were getting the correct ping overlay on the map that corresponded to our bounding box queries, and we decided to use the Google Maps API as the base layer. However, we ran into the problem of the overlay not lining up with a map. As you can see, all the lines are skewed, and this was pretty perplexing since we were working under the assumption that whatever Google was using to display their maps, everyone else on the internet should be using it too, right? So it turns out that's not true. There are many different kinds of map projections, and different regions often use different systems. We ended up on a choose your own adventure sort of trail, researching different kinds of projections and spheroids, and learning that many other civic map servers used flat map projections, which are perfect for a smaller area, but which become increasingly inaccurate the farther away you get from some central reference point. So the Seattle government, for example, stores their geo information using a flat map projection specific to the Washington North state plane, whereas the Google Maps API displays their information using a spherical Mercator projection more suited for global geo data. So we were basically trying to lay a flat map on top of a round map, and lost a lot of time trying to figure this all out, and even when we did finally get the correct spatial reference ID, we couldn't actually figure out how to solve the skewed overlay problem. So after beating our heads against the wall for a little more time, we finally figured out the solution. By accident, I had just happened to be messing around with the query, and I typed in the wrong spatial reference into the right box somehow, and we ended up getting exactly what we wanted. And this was really confusing because it was the exact opposite of every single example we had seen for similar ArcGIS queries on the internet. So now that we were getting the correct ping overlays properly projected, we had to figure out how to store them. Since we were anticipating building out both a front end web client as well as an iOS app, we didn't want to overload the city's map server with requests. So we decided to separate the project into actually two separate Rails apps, a backend API that would actually make the request and cache the data from the city, which would then serve the front end web client. We used the carrier wave gem to cache the ping files in an Amazon S3 bucket, but since I was taking around 10 to 15 seconds to complete, we set that as a background job per request and then sent the client a temp file. So then any other requests made by the front end client with the same bounding box parameters would receive the cached file from the cloud. So at this point, we've wrapped up the four weeks we had for our Capstone project. It's the end of April. We've done the first six month portion of ADA in classroom. We have the sixth month internship to go. We've basically reached the chimney rock of our ADA journey to becoming developers. Yay. And here's what we came up with for our minimum viable product to show up on presentation day. As you can see, we're using the same ping overlays that the city was using, but we made our web client mobile responsive and all of the important information that tells you what these categories are can be found in touch responsive JavaScript tool tips on the side of the legend. So these tool tips also include information about what these categories actually mean. So if you see something that says time limited parking, you can actually click on it and find out if you can really park there. Even though these are small improvements, we found that they made the data a lot more accessible to the average user, especially since there are explanations of what it means. We've even heard back from a few actual users that this has already helped them navigate parking in Seattle. We presented this app in front of Ada Investors, our mentors, future bosses at our internships, as well as friends and family. Additionally, we were also invited to present along with two other students who did civic related projects at a local meetup for civic technology enthusiasts. Unfortunately though, we didn't really have the greatest experience and while most people responded really positively to seeing our work, we did get a little bit of discouragement from spectators who were really interested in telling us about everything we were doing wrong with our entire lives. We also presented at another local meetup where a guy in the audience actually tried to actively break our app during our presentation by overloading it with requests, which on the one hand kind of rude, but on the other hand, it was as if we were being viewed as an equal to a professional developer. So it was as if we were in this awkward transition phase between beginner and junior developer and it was hard to figure out the balance. One thing that we found extremely beneficial, not just for this particular project, but in making our transition from students into the real world development of our internships was mentoring. We were lucky enough to have a really great contract iOS developer who was extremely enthusiastic about our project, her brother, yeah. My brother, John Rush, who we paid solely in top pot donuts and coffee in exchange for his computer science knowledge and experience developing and planning iOS apps. Because we knew that this product is really a mobile product, we knew coming into the project that we had a large blind spot when it came to iOS apps. John was there to guide us while still letting us decide the direction of the project, the architecture, as well as the design and user experience of the mobile app. Additionally, I had two mentors from Substantial, Mark and Robin, who were mentoring me during the month of the Capstone project and who have remained my mentors for the past three months as I'm interning. Having the continuity of mentorship over the course of a larger project turned out to be extremely valuable. We had experienced developers to whom we could ask questions like, how do we use MongoDB? But they were also there to put our setbacks in perspective and give us real constructive criticism. We used them as great resources instead of just asking one-off questions to other developers on Twitter. And we found that it was really valuable because they knew the whole history of the project and they also knew what our current skill level was. But they were able to help push us in the right direction while maintaining distance and never touching our actual code. We believe that the support from the mentors that we had at Substantial, as well as the mentorship from my brother on the iOS side, helped keep us motivated to pursue the project after Ada ended and it also made the internship onboarding go a lot more smoothly. However, Sing-Way had a different experience. Right, so while Liz had a lot of support at her internship, at my company, lack of structure, management, and support were more the norm. So my company, which is EMC Isilon, essentially does no web development whatsoever and it's used to employees who come from traditional CS backgrounds. And even though they have an established intern program, they also draw from CS majors from around the state and the country. So it's pretty hard to navigate the transition into a completely different area of tech and to also be a representative of an alternative learning paradigm that was in its first year and still trying to figure out how everything would work. So the learning curve was a lot steeper and it kind of felt like I was starting from square one and I had less support. So I would sometimes just steal Liz's mentors instead. So for example, one of the members of my team when he came to one of our presentations, his first reaction was, that was better than I expected. So how does this all relate back to this project? Well, we discovered what we believe is a really hard lesson for beginning developers. Code is actually a really small part of what makes development difficult. We had to start to figure out what we wanted our product to look like, what we thought success would look like, and how to work together without a hierarchy. Part of this involves scheduling actual work days to meet and work face to face, using Pivotal Tracker to ensure that we were making progress and the most important lesson we learned, which is don't code when you're hangry. We learned this one the hard way, as you can see in this action shot of Singh Wei and I. So now with Liz firmly established as my boss and also equipped with the mentality that we might actually be becoming real developers, we wanted to continue with phase two of our app following our original idea of manipulating the lines on the map overlay itself. So we were finally able to locate a KML file for the entire city on the open data website and managed to parse it in Ruby only to discover it didn't contain all the information that we needed. We did discover that there was a different file format that did have all this information and these were shape files, which are another Esri tool that stores geodata in a digital vector storage format. But it's not exactly human readable and it required a third party library to decipher. So we tried asking around for which gems would be best suited for the job, but no open source ones, for example, RGO and GeoRuby were really sufficient for what we needed it to do. And while there's another tool being developed by Esri, it's apparently not ready for release unless Davey says otherwise. But finally we did find another solution and it was? Python. So it turns out that Python libraries are really good at math and handled both reading the shape files and converting the data to the correct projection. And though we wanted to keep our API a Rails app, we thought we could make this work and integrate a Python script as a separate cron job on the server itself and feed the shape file data to Rails. But it turns out that our data was bad anyway. We were told that the data was being maintained by the city and updated every day. And this was true on the map server side, but on the open data initiative website, it turned out the last time that they had actually updated this information was in 2011. So, so much for that idea. So we decided to go back to the ArcGIS map server. That's right. This guy. So in order to connect the public data to the city's map server, we decided to use some of the information about the geometry objects provided in a CSV on the open data website, such as unit ID number to query the ArcGIS server directly. As Singhway mentioned, we're now implementing the solution which uses the open data initiative and the map server together. But what this means is that even though we started out our project trying to architect a way to play nice with the city's server to avoid hitting it all of the time, now when we want to get up to date information, we're going to have to scrape for that data by querying until we hit the result limit. This means that we're potentially going to have to query 2,300 times in order to ensure that we have the newest data for every single block face in the city. But, you know, should work in theory. So, we have a demo for you today to show you what we've been working on. This is our iOS app. And as you can see, you just kind of land into the map. There are these lines that we are writing based on the data. Red means no parking, based on the time of a query. Green means you can park there. And we also have a yellow color if there's like some ambiguity or you need more information. But, you know, kind of the basics of any regular app. You've got your GPS. You can look up a location. We have a favoriting system so that you can store some of this data locally for quick reference. Or if you just happen to not be good at remembering where you parked your car, then you can also favorite it and look it up later. And one of the greatest things that we think we've come up with is this idea of filters. So, you have default filters when you search, but you can also adjust those based on your current needs at the time of the query. So, as you can see, when you change the time from 10 hours to one hour, all of those areas that were previously red turned green. And that's because they were two hour parking. So, now you can base what you query for on your needs. And we also have, we're building up a profile system where you can have a default filters. So, every time you query, you've just got your defaults already in there, as well as a section for putting in residential parking permits. So, in Seattle, if you were in Capitol Hill and all of the parking appears red, if you happen to have a residential permit for that area, you could put that information in and all of those previously unavailable parking areas would turn green and become available for you. Additionally, the next feature we've been building out is an ability to tap on any specific given line and have a pop-up that will bring in data from a separate data set from the Open Data Initiative that is the actual text on every single parking sign in the city of Seattle. It's something like three billion data points. It's just totally crazy. But we've been able to match that up with the data that we're scraping from the city server. So, now you'll be able to query for what you want based on your needs at the time of using the app, as well as be able to tap on it and see the specifics on the block face you're currently at. So, that's kind of our basic app so far. As you can see, we're still in development, but we think that we are getting closer to an actual market-ready product. So, some concluding thoughts. Even with only a few months of training and having never really programmed before starting ADA, writing code itself was never actually the hard part. It was working with data that we couldn't control and which had little or no documentation that were the biggest challenges. Not only that, but it became increasingly clear that there is never actually one straight path to a solution in spite of how simple an idea might seem at the get-go, which is how she fooled me into working on this. Basically, we didn't really know what we were getting ourselves into, and no one else did either. So, because of this ambiguity, we had to redefine what success meant. Part of this learning process involved the discovery that if you wanna build something new, you aren't necessarily going to be able to look up the solution as simply as clicking on the first stack overflow result. So, while this was frustrating, it was also really empowering for us to see that other so-called real developers with real jobs don't always know what they're doing. And someday soon, we're gonna be those real developers with real jobs who don't always know what they're doing. And originally, we wanted to make this app so it would transform the way that this open data was being used. And this meant something interactive, human-friendly, and mobile. However, because we had only four weeks to complete the project within the scope of ADA, we had to settle for an MVP that wasn't exactly what we envisioned. This is not to say that our four-week MVP wasn't something to be proud of, but on the contrary, it was a little bit terrifying that after just five months of learning to code, we could not only build something like this, but we have a vision and a path to an even better product. Yeah, and even when we spend hours and hours on code, we don't actually end up using, we also realize that investigating potential solutions and figuring out why they may not work could also be counted as a win. And in some ways, it was just as productive as checking in production code. This was actually a really hard lesson for us to learn, the fact that all of us in ADA were making sharp turns in our careers to go into tech made us feel like we had to catch up to where other junior developers were who had traditional backgrounds. So anytime we were writing code that wasn't used or ended up being the wrong approach, it seemed like we were wasting time or taking steps backwards somehow. But persisting through even when all roads seemed to lead to a dead end really deepened our mental and emotional stamina and gave us a much better idea of how real-life code actually evolves. So seriously, if anyone can help us wrangle this data in a more efficient way, please come talk to us after this talk. That would be fantastic. And once again, I'm Liz Rush, and I'm Sengwaisu, and we hope you enjoyed our talk. Thank you.