 Okay, we'll go ahead and start the stream now. And then, yeah, just let me know when we're live. Okay. All right, we're live. Great. Hi everyone, welcome to episode two of the 2020 Wikimedia technical talk series. Today we'll have Mateo Santos software engineer at the Wikimedia foundation. He'll be speaking today about Wikimedia maps and its challenges. Mateos is finished. We'll have a short Q&A period for folks raised questions, either in the Wikimedia office IRC channel or in the YouTube live stream. Thank you in advance for your patience. We're all working remotely right now with various internet speeds and equipment. And I'll be monitoring the chats as best I can for any questions that come up. If we are able to answer one of your questions during the live broadcast, we'll be sure to follow up later. So now I'll hand it over to Mateos to get started. Thanks Sarah. I'm going to start sharing my screen. So welcome to the map stack talk. My name is Matheus Santos. I work for the product infrastructure team under the product department. Since I joined the foundation in June 2018, and since then I've been the main engineer working on the maps maintenance. So that's why I thought it would be a good idea to explain how the stack works, how we have been dealing with the challenges for over this almost two years. And I hope you enjoy. So I would like to start with some history and context. Maps is, it's an old project inside the foundation. The early stage of the project was between 2015 and 2016 with the discovery team. They were responsible for prototyping it and making it available in production. So it was a really huge amount of work at that time. And a lot of features were developed to make the maps as it is today. And at some point, the, we had a community wish list because of this great work deep on maps. The community really enjoyed how it was being presented on the Wikipedia. And they, on the community wish list of 2017, maps were one of the top requests from the community. And because of that, and because the discovery team wasn't working on this anymore, the collaboration team was formed and worked on on this maps and enhancements for about two quarters. And after that, finally, the handoff happened with for product to the handoff of the maps happened to product infrastructure, and that's where we started the maintenance mode. So the early stage, as I said, the release of the beta version was around March 8 2016. And we would have a stable flag really later on 2019. The service was working well, but we had a lot of maintenance and server issues that we should address and I will talk more about this later. So as I said, that was the discovery team that works to create maps and deliver it. And one thing about the community wish list team, which is up to 2017. It was that maps was chosen, but the scope of the, of the, the, the wish was too big for to be working on work by the community by the collaboration team. So if you go to the maps improvements to 10 team page only tech, you will see a really nice backlog of information, a lot of reports from different times regarding maps development on this on this on the on those two quarters. And the thing about this that I would like to highlight is the complexity of the the stack. And that lead to a goal, a specific goal for this, this period of time, which was ensure that the stack was stable and could gain a wider audience. This is really important to know and I will explain more about the complexities that everything that touching maps encountered over these years. And that's just fun fact about the community wish list, because it was too big, only half of the of the tickets that were created was finished. And we already we had some we have we have some tickets out there that are stale because of the platform, the platform state of art, we have some orders that can be done but needs volunteer work. And that's something that we are trying to address as a community with maps. And now, after the handoff on 2018, the we are the current stewards, the property infrastructure, and we have the help of operations, is specifically on the day, which helps us a lot on the off side of maps. And in private infrastructures Michael and I, right now, being the main maintainers. And, okay, so what is maps as a Wikipedia product and how it, how it is available for the, the community maps has like has two main products. If we can say this, one of them is the extension, the cartographer extension. And I think this is the mainly used by the community is where we can integrate maps with Wikipedia, the wiki pages on every Wikipedia project. And, but we also have the cartoteria entire server which is is a product if we have terms of use it's open to third parties. And, and, yeah, that's, that's basically how we show maps to our, our community our customers. And so cartographer is an extension to show an interactive map on Wikipedia page. And you can also, you can have an interactive map and also a static map depending on the, on the settings that we have on the, on the, on the, the project on the Wikipedia project, but basically the, it wants to deliver. And cartographer is based on a leaf on leaflet which is the engine behind the cartographer interactive map. We use a specific version which is a map box version of leaflet. So it's kind of updated and we have some work right now. We're trying to migrate this version, which is awesome. And, but basically leaflet is open source and mobile friendly and it's the top. I think it's the top library that we could use right now to deliver interactive map maps, and it's very easy to customize and write plugins for it. And cartographer is an example of that because we have some built in plugins to interact with leaflet and a specific Wikipedia and architecture. So, what is, what we use as data, OSM data is I think everyone, or most of you must should know OpenStreetMaps, which is was considered the Wikipedia of maps, and it's basically OSM is a map of the world open freely freely to share an open license and that's really aligned with our mission and it's really good that we have this for our infrastructure. Right now we use a set of plugins to load OSM data into our specific, in our specific hardware. We use OSM to PGSQL to download to put data on a post degrees database, and we use osmosis which is a also, but the both of them are set as our tools from the OSM community, but we use them to load data, load data daily, hourly or immediately depending on the configuration. And so we can keep the OSM, the data of the maps updated. And what uses this data, so we have cartographer as an extension, but we also have the title server as I mentioned, in the title server, the title server is called cartoetherian, and it works with another sort of software that we call cartoetherian is basically, oh, and yeah, let me mention this cartoetherian accelerator use mapnik as an engine because we do rendering, we do static image rendering, we do binary data rendering for tiles, and mapnik is the engine behind this render, so we can, so mapnik is used by Tylerator when we are generating pre-render tiles and it's used also by cartoetherian when we are generating tiles on the fly. So cartoetherian, cartoetherian is a Node.js server, and it's vector-based because we use mapnik to generate binary data and compressive data so we don't need to store images on our databases. And cartoetherian uses the data generated by Tylerator and deliver tiles for every client, every Wikipedia client using cartographer or any other third-party user. Tylerator is what backs up cartoetherian vector tiles, I will explain more about how the host stack interacts, but Tylerator is the piece of software that will get the OSM data and generate it in a tile way and distributed regarding different parts of the map. So it's Tylerator because it comes from tile and I'm not sure if you know, but the way that we serve maps, the way that we serve maps on a server is tiling the whole planet and depending on the specific region that we request, you get an image. And Tylerator do pre-generation of all of this so we can have a lightweight and a performant API so we can attend the Wikipedia requirements for delivering content as fast as we can. And okay, so also with cartoetherian we have another part of the service which is snapshot. The snapshot is a way to render images and serve static images and the snapshot is important because we can also render data from commons or wiki data and put on a map, on an image on the map, on the map so it can be served to non-JAS clients on Wikipedia. And that's the main reason for the snapshot service to exist. Another thing about the cartoetherian service is that we collect external data from wiki data in commons so the community can create with more flexibility freely licensed content. And we can use this on our maps. So geoshapes extract those data from commons and wiki data and make it available as a jail jason. So it is used internally by cartoetherian and the snapshot service, but it's also a exposed API so we can have this data as a jail jason at any time. So, okay, I think that's that will make easier to see where all those software feeds on the on the stack. And I think what we have is Tylerator as a pre-generation vector of vector tiles engine. It loads data from postgres it use Redis to consume a job queue and post jobs to generate different tiles of the planet. Tylerator has also a UI where we can schedule jobs and do maintenance work or even monitor what is happening with the tile generation jobs. So if the vector Tylerator generate the vector tiles, it will start it in Cassandra and make it available as an a fasted way possible for cartoetherian so it can serve tiles through varnish and get to the client. And also OSM is the data that we are storing and being consumed and use it to generate all this information. And this is another diagram that is cool to show you because it, it kind of explain how a one of our clusters work and I mean our clusters work like this we have two clusters codefw and a q amp. And this is a diagram that shows more or less how the user interacts with our data. So, basically, the process as I said to is Tylerator generating tiles cartoetherian serve them. We have four machines with each one with a postgres master or slave and a Cassandra or master or slave instance, and they are both served through varnish. And I think, I think that's basically the how we deal with with the technology and the the hardware to show map to have maps in our infrastructure. So maps is awesome and complex. And I'm going to explain a little bit more details on that because that's what we'll see before we start to working on it or maintaining it and when we go deep deeper you see the, the needs of the of the project itself. So, we entered maintenance mode, and in the new q1 2018 fiscal year, 1819 and the state of maps that on that time was like this. So cartoetherian Tylerator has three three dependency really large. It's a lot of components. Most of them are map box components that are open source and you can use it, but cartoetherian Tylerator use a lot of software to keep it running. And at that point we didn't have cartoetherian or Tylerator in Garrett. So we the cartoetherian was supposed to be a community for itself, but it that didn't happen. And later on the maintenance work we brought it to Garrett and start to do our, our work on on, we figured out the search code as being on our infrastructure. And you can see a list here cartoetherian cartographer Tylerator which I mentioned but there are two more components because every data needs a style to be in the style on how to render it. And we have OSM bright, which is a project with a fork from a box style, and it's basically a carto CSS kind of carto CSS project where we have the rules on how to render the maps, how we are going to paint roads, how we're going to paint lakes and, and stuff like that. BrightMap is supposed to be the new style that we're going to be deployed on 2018. But this this it didn't happen because it was we didn't when we started the maintenance we didn't have much knowledge about the stack we were afraid to deploy and not being able to keep it running or because of the lack of knowledge that we had about the new style. So we keep it OSM bright running and this is the state of art today. And the source of the the way that we we the style sees see the source is the set of rules that will get the vector tile and be presented to the style, the style guide is those two projects OSM bright TM to search, which is a set of different SQL scripts to SQL queries to the data from the OSM database and turn into a vector vector style, and metal was supposed to be the source for the new style. It was a work for Polenormal, and it was supposed to also help with different loading of OSM data. Unfortunately, unfortunately, it didn't happen. But that's, that's what we face it when we started doing maintenance. And when we were developing, and when we are doing maintenance here, I'm going to show you a couple of projects that I created and or are created to do maintenance and help me to create it. And Carter tools is the first one. It's a small collection of tools that, at some point I need to investigate or read logs from Carter three and two investigate if we have slow carries running on post SQL. How I had to collect data to see how the dependency deprecation would happen and track this over time to see if you were keeping up with the platform evolution of the stack. So it's available on GitHub, on my personal GitHub right now, but there you can see different tools that I, some I created some I created, and it really helped it on the maintenance flow, at least. And another one was Carter doc. And that's, that's the, a really interesting project because I'm a first week in my first two weeks. I was supposed to work with maps, and it took me these two weeks plus onboarding to finish the first setup. And I created a Docker compose setup to run all kind of, all kind of scripts that I that we need to use with maps. And now I reduce it this time to anyone can create this create a setup to work with the maps with the committee foundation in 30 minutes, or more depending on the size of the database of the OSM data you are downloading. But basically this Docker compose setup has Cassandra reduce post SQL with post just running titerator and criterion. I also needed a to have map books to my box studio. So I could debug some work on style, because we were using because we were using old, old carto CSS, which is kind of deprecated on maps nowadays, as a style, as a style framework for maps. And, and that's it basically the, the, this project also supports OSM to be just SQL and info SM three, which is ways to load OSM data locally. And that really helped on the to share knowledge about the, the particularities of this tech or the development between our team, and I hope this can be also of help to the community, or volunteers. And finally, every maintenance mode has to deal with technical debt, and I'm going to show you some stats about what how we deal, how we dealt with technical debt and how we're continuing to work on this. So, that's, that's, that's the technical is that technical debt stats that we had before the handoff and at that point we had outdated the penises so I, as I say that and when I explained in carto tools, I created a script where I could see every package that was that were being used by Cartot Tyrion and editor and see on their source, if we are we were we were using a outdated dependency of or outdated version of that package. And the, the reality is that 52% of Cartot Tyrion was outdated 62 of director was outdated at that point. So we didn't deploy the new styles. And there was, there was still some prototype code running on the platform. And at and looking on the setup and how we had to work with maps we saw that it wasn't a volunteer friendly project so it's kind of hard for the community to work technically on maps. There are some experienced developers which are volunteers and are submitting patches for maps, but it's, it's not like the, it's not the same presence that we have for media week as a whole. What happened so what are the technical debt stats now. So, we have a lot of peanut versions that we can't upgrade on Cartot Tyrion and editor, and the outcome of this is the deprecation will increase over time. So we are setting a lot of plans on how to work around this, but that's the reality right now. The update outdated events are increasing. So the new style is not deployed yet and shouldn't the tech, the technology involvement is outdated now. If we're going to do a move with these tiles or how we render tiles I think we should move to a different technology more edging cut, let's say, like mapbox GL, which removes the need of mapping on the stack and make it more feasible to maintain over time. And so it's not a volunteer friendly project we try to mitigate this problem. So, at the beginning we had, as I said, like 36 repos on under Cartot Tyrion organization GitHub, we gather all of this and make it Mono repo, so we can have a better source control over Cartot Tyrion and editor. So we moved this now new Mono repo to a Garrett mean to to a Garrett instance on our infrastructure. And that was a really good thing that we did because that for the first time maps started to have CI tests. We didn't have because it was we're all in GitHub and we didn't have any CI setup for maps at that point. And now we have a clear, at least a clear CI setup and how to interact with it. We tried to document it all process that we could. And the, the SR, SR 18 did a great job creating cookbooks for postgres for when we were doing some maintenance on the server on when we were we need to depot servers and refresh the database for some reason. And I hope that with the Docker compose environment, the, the difficult to install install when work with maps will be reduced. And, but our work also created more technical depth is not always good because as I said, we needed to pin some versions specific versions and because of those versions that we need to pin, we also had to fork some projects that weren't being maintained anymore and we, we needed to for them to create and develop and do some changes so we could, for example, move to note 10, which was the case of note map Nick, and because of this, this, because of that, we now have like, we have a good setup with that been stretch to note 10 for maps, but it's not going to be easier to move forward, if you need to go to note that note 12 or our first, for example that buster so we're we're going to have more work on this in the future. I'm sorry this is duplicated and so the maps achievements so far on this almost two years of maintenance was that we did a OS migration at the beginning of the work. It was really difficult. It took about two months to understand and finally deploy the last word that the finally deploy maps on a Debian stretch cluster for both for both cluster actually. And because of that we learned a lot about the full planet tile regeneration we have a report that's really nice to to that that has nice information about benchmarks on how long it will take to move to another to do this again in the future, and how long it will take to regenerate tiles over the over for the whole planet, and all our learnings and of our failures on this. So, I will have a link in the end for that too. We did a migration to note 10 and at that point it was the first service to go to note 10 other than I think methoi was the first and then we moved. Cartesian titerator it was a nice thing for us, because it's really it was really difficult we did we had to. We needed a lot of work around on this to make it happen. So, move it Cartesian to greet, it started to have CI test for the first time for maps. And a lot of improvements on documentation and operation sites and on development side as well. And the status of the project is maintenance still. We do modern regular maintenance. I'm not sure. But we do have a lot of knowledge share about this. A lot of, we don't have written plans on how to do that, but we have some new work that can be done on our roadmap in the future. And we can do infrastructure modernization if you need to keep up with with the newest version and a better maintainable infrastructure. We need to move to Kubernetes at some point. And we have a lot of reactor re architecture that we can do on critical modules to make maps more easy to maintain. More documentation. This is always welcome. So, yeah, I would like to shoot out to the volunteers that help that has been helping with maps for this whole time I learned a lot on the interactions that I had. And I hope that we can with with we can share more knowledge about this and can move forward maps because we have the research that we have nowadays is strictly to maintenance and I know that's that the community deserves more. But I think the volunteer work is really trying to fill the gap and even do new features are trying to evolve the platform with us. And yeah, I think that's it. Here are some sources that from what I showed on the on the presentation. And yeah, I would like to thank you. I think we now have open room for questions. And I hope you enjoy it. Thanks to tennis. That was a really awesome talk. We actually do have a couple of questions that came in on IRC. The first question is, do you have plans to move map box dash gel, given its lack of support for a number of languages. Well, as I said, we do have a lot of knowledge on how and I did some prototypes with map box gel, trying to move at least the snapshot service to it. But the investigation showed that it would be a really, it wouldn't be an easy effort to do that so we don't have a plan. I plan. So what we want to do is read write a plan so maybe we can do this together with the community. But it is feasible. It is, we, we have room for that because cartoon is really modular modularized in its architecture. So we can pull some modules out and and get some other modules in. If possible. The next question is, has there been any thought around moving the compute the compute heavy parts of maps tie later and. I would say this is kind of theory and but not postgres Cassandra to Kubernetes. Well, this, this, this work really just started on the necessary side. The, the first challenge that we have is how we're going to make Cartesian and tell a rate of working with the CI pipeline and the CD, the future CD pipeline. So we didn't know, we don't know yet how we are going to make it work for now, because Cartesian is different from other service. It's a mono repo. It wasn't developed in house, but not in house, but it was developed on GitHub, supposed to have its own community so it has different patterns that we are now integrating. We're trying to react acting. But so I don't, we still need to make sure that we can deploy Cartesian Taylor later on Kubernetes. And after that, I'm not sure how this is going to be. It's not going to be so good, but we do need a separate instance for Cassandra and postgres. And well that's a really challenge that I think the operations team operations team will have. Let's see. If you have other questions, please feel free to put them in IRC and checking the live stream on YouTube as well and it looks like we don't have any questions there. And we'll give it like, just a minute to see if anything comes up there. Just a lot of thank yous from IRC and also from the, from the YouTube stream. So, I think, I think that's the last of the questions. I'd like to let everyone know that if you do have questions after you've seen this talk, you can feel free to reach out and we will pass those questions on and make sure that they get answered for you. And then I just like to say thank you for meeting with us today it's really awesome that you're willing to do this talk it was really great. Thank you to friend Brendan Campbell Craven who does the AV for all of these. And also just a reminder that we do these talks monthly. We're looking into members of the technical community and you don't have to be a look me just ask you can be a volunteer. We're always looking for people to share what they know about technology so if you're interested. Feel free. Oh, I have one more question. Do you want to answer it. It's called can, can we bring back, can we bring back maps beta. Do you mean by maps data do you mean our data on Wikipedia like in commons of wiki data or it's on the live stream so it's a little bit behind. I think it was beta beta. Yeah, sorry, did I not say that clearly. Well, I don't know. I think maintaining the map stack was really challenging to not say it's really difficult but really challenging and I hope that we can make it more stable than it is. I think I see the community needs a lot of new features to to also feel the gap that maps has created on interactive content. And yeah, I'm not sure if how we're going to do this, making it more stable but that's the best to go. I hope that answers. I think that's it for for the questions that are coming through. We'll wait one more. One more minute. I think that is the last of them. Awesome. So, thank you for answering that. And again, if anything else comes through. I'm more than happy to pass it along to you. So, thank you very much for joining us today. Thank you and to everyone feel free to reach me on the internet if you have more questions.