 Okay, hello again everyone. So next up we have Ernest Durbin, who is also Python's chair this year. And he's going to talk to you guys about running with edge software, 5PI's aging code base. Alright, can everybody hear me alright? Excellent. If you are watching the sort of thing that's blasting past on the screen, you might have understood the theme if you can read it quickly. So this is running with edge software, 5PI's aging code base. I'm Ernest W. Durbin III and a little bit about me. I'm a software person. I live and work in Cleveland, Ohio. Since about 2012 I've been contributing Python software to the infrastructure team. I'm an avid taco enthusiast, but recently much of my free time has been involved with vehicles, which is led me to believe I may be a massive business mechanic. And that's because I make every attempt possible to make this car my daily driver in the fair weather. This was my first car, like the first car ever. It's a 1967 Saab Model 96. It's one of the last two-stroke powered cars in the United States, which is a fun fact. And I've been servicing, repairing, and trying to improve it since I started driving it around 2003. Driving a vintage car every day can be a pain. Things will break. Cash will generally be the easiest solution. Experts in parks may not be readily available. Engineering assumptions from the 1960s may not be valid anymore. So is this a lost cause? Should we just scrap the vehicle? I could get something far more reasonable and give up on the cross. I don't think so, and that means that there's only one thing to do, and it's get to work. Oftentimes we just be pulling things apart and cleaning them up. Can we pose you for a moment? Sure. Okay. I do not know. There we go. Sometimes it's... Oh, wow. Sometimes it's simple telemetry that needs to be repaired. And once in a while there's need for more invasive surgery. During a recent conversation on Twitter about my Saab, Hinnick saw the need to interject. He posited that perhaps my being tied up in vintage Saab was indicative of why I might have been tied up in one of my favorite ways to contribute to the Python community. I wanted to consider this carefully, and I concluded that Hinnick is not wrong. So let's talk a little bit about PyPI. It's one of Python's largest pieces of core infrastructure, and it's short for the Python package index, also known as the cheese shop. And I've been one of PyPI's caretakers along with a couple other people since sometime around 2012. During that time we worked with Donald Stufft and Richard Jones and a handful of other volunteers to improve PyPI's availability and keep the lights on. So a little bit of history. PyPI's predecessor was this thing called the Vault of Parnassus. And it was a hand curated page full of links to submit Python stuff. But that's all. It was just links to other pages. It even included a lost and broken links section. In TrueGlorious 1.0 fashion, it was pure hypermedia. Links were king. It was just, you know, here's where you go to find out more. And there are archives that are still floating around. This is from archive.org, and there are a couple others that are out there. I really encourage everybody to go take a look at it because we've come a very long way in sort of what's available in the Python ecosystem. And a lot of this stuff is still valid. So that was great. And it provided a lot of value. But, you know, there was a need for improvement. If you're not familiar with a PEP, a PEP is a Python enhancement proposal. And it's sort of the way that Python and Python ecosystem changes in an official manner. Like most core parts of Python's ecosystem, PyPI is now a handful of PEPs. And we're going to go over just a couple real briefly to talk about what they are. So PEP 243 described a distutils extension, if you're familiar with distutils, to provide, to upload metadata about your package to a central server. It was ultimately withdrawn and replaced by PEP 301. This also described a distil extension for uploading metadata. And it included classifiers. And it was eventually accepted and became the PyPI we know today. For the first few years, it hosted, darn it. So for the first few years, it hosted no files and acted simply as a metadata server, much like the Vault of Parnassus. Packages were hosted by the maintainers in whatever way they chose. The goal was something more specific and built into the Python ecosystem than the Vault of Parnassus. Uploads in packaging and package hosting were added in the sprints of Python 2005 in Washington, D.C. And eventually, in 2013, PEP 438 made hosting mode for packages explicit. So you can tell if a package was on PyPI or hosted externally. And finally, in 2014, PEP 470 deprecated external hosted packages. So if you want to install something with PIP from PyPI, it had to be on PyPI. Over the years, PyPI has seen a number of changes, but the largest has been the sheer volume of traffic. So PyPI today is Python Package Index. More than 100,000 unique packages, more than 700,000 total releases, about 175 million requests per day, and about 20 terabytes of data per day are served. And there are like three working tests. So, and I think I deleted two of them the other day. So this is a chart just to sort of give you an idea for growth. This goes back four years since we started using Fastly since it was the most accessible. And we've served about 95 billion requests and about 10 petabytes of data. It's an incredible amount of stuff. So where did this code base come from? The initial commit inversion control started November 1, 2002, and the first package was actually released just a few days later on November 6, 2002. This is Vintage Software. Remember how I said it was just initially, you know, a disk-util extension and a metadata index? Over the years, it's just added features and sun-setted features and changed a bunch. But interestingly, most of the basic parts of the code base have lived on and been extended or modified to do what we need to do. On top of that, it was built on a one-off Whiskey implementation, a web server gateway interface implementation, which we might scoff at today, but you have to remember back in 2012 or 2002, dang. Options were incredibly limited. Trying to operate Vintage Software every day could be a pain. Things will break. And when they break, people will talk about it a lot, as we remember. But not everything's great, and here's all the tweets that contain PyPI down since 2009. There are hundreds of them. Running a service that becomes a core piece of infrastructure for developers who love Twitter can be a challenge. They will speak out and let you know. Tools were built which expected and required PyPI's availability for deployments, continuous integration, and workflows. So what do you do? This is a chart of over the last, since 2009, the frequency of PyPI down in tweets, which I'm pretty proud of this graph. Let's go in the right direction. So, you know, we sort of had to do something to provide better, or not we, but in general the community had to do better to provide value. And so PEP 381 introduced mirroring infrastructure, which made a lot of PyPI going down less of a problem. There were some problems with this, though. All of these were community hosted. Users would basically say, I have a mirror. Can you give me a C name? And so now you're giving out C names to python.org subdomains to pretty much anybody that just had good intentions at the time. And clients had to change to support it. So you had to explicitly point yourself in a mirror, and that's not always the most convenient. Cache is generally the easiest solution within its software. The single biggest impact on PyPI's availability and getting better in speed and all this stuff is vastly. They provide reliable installs. You can always just point at pypi.python.org. You don't have to switch to a mirror, so client complexity is lowered. There are powerful features for us to change the way that things happen. I really like Vassly. So Vassly. Vassly. Okay, said enough. So Vassly made a big difference, but there was a turning point back in 2013 where things really just sort of came to a boiling point. There was one VM running in a data center somewhere else. It was manually managed so people things had changed and installed everything. It was running on a bespoke, donated piece of infrastructure. And all the support for PyPI was 100% volunteer. Experts in context might not be readily available. Volunteers can't always just say, oh yeah, I'll spend X amount of time talking about why this is the way it isn't such. So eventually we decided to get a little bit more real. We brought PyPI sort of into the current age where we're at now with configuration management, high availability storage, high availability databases, multiple web servers, and we were hosted on a hosting provider with an SLA. So whenever we had an issue with infrastructure, we could say, hey, why isn't networking working here? And somebody would answer. And finally, engineering assumptions from 2002 might not be valid anymore. Over the years, PyPI grew a bunch of anti-features, in my opinion. Most with very good reason, but we're kind of past where they made sense. Again, the custom, you know, whiskey framework, it works, but you know, it leaves a lot of edge cases to be concerned about that, you know, experts in HTTP can probably solve better. There were SSH uploads for a while to get around the fact that TLS wasn't in the sort of golden era we are now talking about some crypt and other solutions, which meant giving shell access to anybody to upload a package. That is not that great. The real-time download counts are fragile, flaky, and hard to believe. So those have been replaced with a Google Bigtable. If you'd like more information about that, which a lot of people do, they missed their stats, let me know, and I'll point you out to an announcement on the Bigtable and how you can access it. There was an OpenID provider, an OAuth provider, and for a while used some broken cryptography, now broken cryptography. So a lot of these things have been removed, all but one, which will be removed, I think, next, like two weeks. So is PyPI a lost cause? Should we just throw it away and give up and do nothing and you can write all your own software? A little, yeah, but not the last part. You should still rely on experts and other places to provide that value for you. You need to be able to install things. One of the biggest drawbacks to PyPI's current code base is that onboarding new contributors can be a huge pain. A lot of the things we spoke about, the complexity, custom frameworks, very few tests, it's not a very inclusive environment to, like, say, I'm going to jump in, make a change, and have it reliably deployed. Dependencies, database schema, and custom web framework, all just make that worse. And they make it hard to solve problems and do new work. So over the last few years, there's been an effort on a project called Warehouse. Warehouse is the next generation implementation of PyPI. It's a full rewrite with extensive test coverage, documentation both on its functionality and developer documentation. It uses modern best practices in the web world and it's built with caching in mind. The original PyPI is not, which we'll talk about momentarily, which makes some things very difficult. Donald Stuft has been working tirelessly on this development and things are really progressing. If you've uploaded a package to PyPI in the last month and a half, I think, it actually went through Warehouse, not through the legacy code base or vintage code base. But these things don't happen overnight. As a matter of fact, Warehouse has been progressed for more than two years. So it's time to put the old thing on life support so that we can focus and build out the new thing. So I'm going to talk about a few things that I think are critical to make running the old PyPI while we develop Warehouse possible. So a few warnings. So think concepts here. None of these specific implementation details will necessarily just be ready to go for your purposes. There is going to be some code on the screen. Don't focus and try to rush to read it. As it comes up, I'll highlight it as we go. And I also want to note that the goals here are slightly different than your average production system. We're very sensitive to behavior changes here. We need to maintain compatibility with a lot of concerns internally for Warehouse. Excuse me, I'm just going to catch my breath. And there are different goals. And so if you're building something new, a lot of this advice is valid. But I'm really focusing on what it takes to continue running this. For more information on what to do for something new and great features, there was a great talk in this room just previous by Brian Pitts. Capacity and stability patterns. Check that out. The single biggest enabler of effective change for PyPI legacy or vintage PyPI, especially given a limited time and resources, have been metrics. These are all about answering questions. Often you'll notice a focus on system metrics like CPU, memory disk, et cetera, which are fantastic. However, there's a welcome trend over the last few years in tech towards application level instrumentation. System level metrics are great for answering questions like, what is slowing down or breaking? And did my change help performance or hurt it? Application metrics can tell you things like, where are my efforts best spent and what features may no longer be worth supporting. So here's an example. We have a very simple method that calls another method so it's nice and abstracted and stuff. But it's called a search method and we want to instrument it. We want to know how often it's being called and how it's performing. It's really easy to change. There's a library called perfmetrics and it makes it almost comically simple with decorators. Import perfmetrics and you decorate your method. Magically, you're suddenly getting these stats D metrics submitted that tell you things like a rate, how often it's being called. The count, which is just the raw number of counts per time slice. Account, again, because of the way stats D works, aggregator works. And then you also get these really helpful things. Lower, mean, upper and upper 90 by default. Lower being the fastest time in the slice that it executed. Mean being the average time. Upper being the absolute slowest. And a really important one, upper 90 being the 90th percentile. So up at that very top level, like how bad are things. So here's just a really simple, thing you can do with this. I can say what are the five most time consuming methods for this subset. And it's kind of nasty in graphite, but you can take these metrics and do some math and take the count and the average and say, how much time are my backend web servers spending doing X? And so now you can say, well, there's quite a bit of deviance on this pink one, which is a call to get the number, like the URLs for release. So we should maybe focus on making that quicker, more time for the backends to work. Another example, this is another part of search. And you'll see we look at the start time and the end time. This is a little bit more complex, because we can't just decorate it because it's not in a method, but it's like a subset of a method that we want to measure. So we can start at the end time, submit that. Yeah, so we do the work in the middle, and now we get this graph, which is, again, the same thing. It's the timing of those calls. So we can look at it and say you set a timeout. Yeah, so here we actually time out the request. And so when we're tuning that timeout, it's really important to keep an eye on, specifically, upper and upper 90, so that we know if our threshold is too low, we're going to have all these timeouts, if it's too high, we're not actually going to catch failures. So you can... Oh, there, my slide was set. So anyway, so you have the timeout. So now we can chart and say how often did things timeout and tune things more effectively to keep an eye on that. In the same vein, but not necessarily directly related to metrics, capturing and looking at your errors effectively has also been really helpful. PyPI has a lot of errors of all varying types. And more importantly, since it's hard to reproduce sometimes and since we don't have good tests, it's actually really helpful to have a stack trace and the variables in context and such to help understand what exactly went wrong. So the next thing I want to talk about is scaling with what I'm calling tactical updates. So it's really easy to look at a problem and say we should just throw it all away, rewrite it, and move on. As I've mentioned before, there are very few tests and so that's a very dangerous proposition. Your chances of deleting code and then re-implementing it perfectly are very low without tests. So here is PyPI's search method. It's a big block of code, but what it boils down to is it generates six separate SQL queries that all execute. And if you know anything about SQL, this is really bad news. So things will catch on fire, it's really bad. So if search is kind of a problem, then why don't we use something a little bit more oriented towards search? So this is re-implementation about the same number of lines of code and most of it's just sort of massaging things back to the way that the things that call this method work. And because of how naive the search implementation was, it was actually really easy to implement a naive solution in Elasticsearch that performs much better and the accuracy is within acceptable territory. People complained a lot at first, and then I had to learn about Elasticsearch, not just throw code at a problem, but we got there. And the payback was really huge. So this is the upper, so this is the worst case search time, went from somewhere like one and a half, two seconds to like 250 milliseconds, which is really nice and more importantly, the number of errors that we were serving at the edge went for timeouts and just back end overloading dropped off the face of the planet. And so there are some scenarios where you look at a chart like we serve so many 500s, we should stop doing that and there's no way around making like a concerted effort to do like a tactical change that changes the behavior within acceptable amounts and improves what you're doing. The next big thing is caching. Caching is a tricky business, and HTTP caching in particular is probably the most accessible to most people, no matter sort of what you're doing. HTTP caching is sort of built in to the HTTP protocol with cache control headers. There are many options you can sort of adjust and tune to your parameters, and it's generally sort of bolt on. HTTP cache between sort of an HTTP client to server almost any time. This guy, Matt, how do you say your last name, Matt? Robonode. I just know him from the internet. Has a pretty good talk, cheating your way to web scale. The talk's a lot of HTTP caching. You should check that out if you want to know more. The long story short of HTTP caching is to get a CDN. You can generally sort of shop this problem out on relative budget terms cheaply, and if you're going to get a CDN, I might recommend Fastly. They're really, really nice to the PSF and PyPia, so I'm just going to keep saying their name. Another form of caching is internal stuff. This is where things get pretty tricky. It switches from out-of-the-box solutions and vendors you can pay to having to think a lot more. So as an example of why caching is hard, this is a piece of puppet source code that was in the puppet code base for many, many years. It's recently even changed. It was about 18 months ago, but there's one particular problem with it, and it calls the XMLRPC, which is a sort of old interface to PyPia, to get the current name and version of releases for a given package. So a little bit more about XMLRPC. This is a valid XMLRPC call. It's kind of wordy, and it's very well specified because XML. But there's one really, really bad problem about this. This is a totally, this is absolutely just reading. It's a read-only call. You just want some data back, but it's doing a post. Posting is sort of a problem because it makes your HTTP caching concerns much more complex. You have to do smarts to try to cache a post. It's dangerous as well. In general, maybe this wouldn't be a problem. Puppet's a pretty popular configuration management framework, but I run it on a couple servers here or whatever, but Google or somebody at Google's size or similar size has a ton of instances, and they all want to call this method, and it's hard to cache. So we can't do it in HTTP, but maybe we can do something a little bit more intelligent. So here is that package releases method in the XMLRPC. And so the implementation of caching, again, is just sort of a decorator that uses something at least recently used cache to look at the method signature and cache that for as long as it's appropriate for the package name, and that way we can, without querying the database, we can return the same response and just cheat. This was a pretty big win, too. So this is the hits or blue misses or green on that. So it provides a lot of value because we're not nailing the database, and it goes faster as well. Another thing is rate limiting. Rate limiting is super important during that your users are happy. If a single user can come in and ruin everybody's day, it's no fun because not only will that happen, people might actively attack you and leverage that. So some things that rate limiting can protect you from is spike in traffic from one client, misbehaving or malicious script. You can use it to prioritize traffic, and you can use it to shed load. So you can say that these things are not as important. Don't do those. I'm going to talk about the first two since it's what's most common for PyPI to see, but the other two are really, really neat to think about. And Stripe did an excellent blog post about rate limiting. Stripe.com slash blog slash rate limiting. If you search Stripe rate limiting, you can read it. It's super solid. So spikes for one client. If you know anything about Linux systems, there's a scheduler called Cron that will run a script at a given time or a given frequency. Again, if a single person is doing that, it's not so bad, but if you have giant racks of servers all running something at the same time, a single IP address or a range of IP addresses can come in really quickly. So rate limiting spikes from one client can be super helpful. It generally manifests as a single IP address or a few IP addresses that all seem to hit you at the same time. And more often than not for PyPI, it's sort of that loads some XML or PC interface. I think only once I remember where it was literally like thousands of servers that were all running something at the same time every night. And that's a lot harder to handle. That's effectively a denial service or distributed denial service attack. And we'll talk a little bit more about how to handle that in a moment. And oftentimes this can... diagnosing the denial service attack can be much more complex than just saying this IP address is nailing me for a long time. Another thing with rate limiting is dealing with misbehaving scripts. So your new PyPI crawler is awesome and after hacking on it all night, you go to bed, push new release and wake up in the morning to find out that all you're getting back are 429s or the HTTP status code for slow down, you're hurting me. What went wrong? It's a common issue with tools that are built with really good intentions but end up calling routes to the back end that are expensive. Again, we generally only have to deal with this for XML or PC. So if you want to build a PyPI crawler and get data from PyPI, you should use the JSON APIs, which you can find out a little bit more on the... there's a link on the current page to tell you about that. So rate limiting, sort of the easiest solution is just to let your web server do it. You can say in Ginex, it's pretty simple. You set up a little configuration for your rate limiting, where it's going to be stored and then you... for each location or each URL, you can say this is exactly how I want to handle it but how much people can get. For application level rate limiting though, it becomes a little bit more intense because like I said, we're getting requests, we need to introspect a little bit to get an idea of what's going on. So this is the actual work of this method and this is just the code that does rate limiting for the XML or PC right now. So the code is actually relatively simple but there's a bunch around it and so if you're going to do rate limiting, you need to be kind of careful and so it's set up as a context manager so it can be called in sort of a simple fashion. You can say with rate limiting, or with rate limiting as limited or something, a Boolean, now you can yield that Boolean and based on the response of this, you can say, okay, drop the request or continue. It's really important to log these things. You want to sort of understand when somebody comes to you and says, hey, why am I being rate limited? You can tell them exactly like it's, well, these are the reasons and this is when you were rate limited from our side and fail safe. If your rate limiting code fails and like stops everybody from getting to your service, it's actually a big problem. And then metrics. Again, as with anything, you want to know how effective your change was and so you should measure in some broad sense how things are going. So here's PiPI's metrics for rate limiting from, I don't know, I think a week or so ago. The blue line is how often the methods that are rate limited are invoked and then we haven't had anybody over or enforced so I wasn't able to find a good chart. And this is another part of measuring the errors for the fail safe. How often we have a problem with the rate limiter and we can note that we're not serving 500s to go along with this. I don't know what this is. So, a brief aside on ban listing. Automated systems for handling not abusive but misbehaving clients are great but occasionally things are just so broad or it's the first time you've seen it that you have no idea what to do. And so this is where you've got to do some log diving. You have to sort of put your detective hat on and find out is this intentional, is it an accident, is there a pattern to what's sending me all this traffic. So occasionally, PIPI does have a couple switches to make it safe for us to continue like serving traffic without shutting down for everybody or failing. So FASI provides a thing called edge dictionaries so we can just send a we can just put a piece of data in a dictionary and FASI can look up like is this IP address banned right now. And so this is really useful for banning a single IP address really quickly without having to think too much about CIDR notation and such. For more intense cases they also provide edge ACLs so you can say I need to ban this entire subnet. They're really hurting me right now. And ruining features. This is another thing that's really super duper helpful for running software. It's really super fun. It reduces your security and error footprint. And it also reduces the scope for the new software, the thing that's going to take over for your old and busted current system. And it also gives you a migration strategy around that so as you're removing features you're able to also maybe reimplement features and start working from the new system along the way. I'm super jealous of Alex Gaynor. He found a like giant JSON blob in the codebase so he now has the number one for deletions. And the last thing I want to talk about is simplifying architecture. This kind of goes hand in hand with removing features. But the architecture of re-application is also a large part of your maintenance. The number of systems you depend on and the way that you run them can cost you time. That might be better spent elsewhere. So it's easy to feel like you're running through a murder mystery mansion when you're in an older codebase. And they tend to be these like giant sort of downtrodden castles. But sometimes it's important to step back and think what are your core needs? Maybe a more modest solution will suffice. So these are a couple things that have been done to reduce the architectural overhead of PyPI in the past. And also at the same time help us in co-deploying warehouse. So we moved file storage from GlusterFS which was like a cluster of servers that represented a file system for multiple servers to share like to serve packages to S3. What's nice about that is now we can run old PyPI and new PyPI wherever we want and S3 is a relatively battle proven piece of infrastructure. Also killing a lot of download counts was huge. We were running like two or three servers that were just dedicated to handling like counting how often things are downloading and doing it poorly. So now there's a much more rich set of data in Google BigQuery including like the TLS version that your client dealt with, the Python version, PIP version all sorts of really good things that allow you to understand how your users are using the packages you're hosting. And then the self-host of Postgres was occasionally a problem and so we just shipped it off to Heroku because they're really generous and offered us some credits. So to review some things that make running an adventure piece of software possible. Metrics, all the metrics making changes with like forethought and specificity rather than just like going nuts and rewriting everything. At once. Caching, removing features and simplifying architecture. So real briefly I want to talk about how you can contribute. It's pretty much the main reason I want to give this talk. So here we are. The number one easiest super duper way everybody should do it if you're in production I guess. But use pipi.org. So pipi is going to move from pipi.python.org into its own domain for a number of reasons. But pipi.org is the new thing now. That will be what replaces pipi at some point. And you can use it today to browse, search, etc. And you can also use it for your installs. If you're using it you can also go look at if you run into a code issue that you know is specific to the code base. You can go check out the warehouse code base and file an issue there or also contribute. warehouse.readthedocs.io has developer documentation, how to get up and running, how to use it, how to be excited. And more generally we recently launched a meta tracker for just general python packaging problems. You can also check that out. And lastly, if you're filthy rich and just want to throw money at it or your employer does, you can check out donate.pypi.io It is a super great way to contribute. But the big thing is definitely check out the code base. If you're able to code python or you want to get started there are tons of issues that need to be addressed before we can sort of go live. We're starting from really simple fixes to complex things to CSS, to templates, etc. So definitely go check out the issue tracker. There's pretty much something there for everyone and that's how you can help. So thank you. I'm Ernest and are there questions? So the question is I had the review and the question was is that indicative of priority should address those things? I'm not sure. The metrics are definitely what I would call priority one because they can be implemented without affecting the behavior of your code base, optimally. You could have a syntax error or something but generally they're safe. So statsd and datadogs, statsd protocol and most metrics and Prometheus for instance, they all run out of band and so you're not making a call to submit them or it's over UDP so it's free. So generally it won't dramatically affect the performance or behavior of your application so it's almost a free win. So start measuring as much as you can to inform your decisions down the line and that'll actually be able to help you change the prioritization there. So the question is, can I elaborate on the decision leading up to rewrite from scratch? I wasn't involved in that decision directly. I was kind of just excited to talk about with the infrastructure and I think it's well guided. There was a large attempt by Donald to refactor PyPI's code base into Flask Live and the value just wasn't there because it would have been an uphill battle against code coverage against all sorts of other understandings. I don't know if Donald's written like a blog about it but I feel like he did. Does anybody know? Anyway, I think it was the right call primarily around test coverage and just building something testable to begin with because yeah, what was there was if you haven't read the old PyPI code base I highly recommend it. It's very large files and there's if statements for routing and such. It's interesting to say the least. So the question is how is it balanced like old running old PyPI and developing or running new PyPI? Sure. So primarily it's balanced around feature implementation. So uploads for instance are now feature complete for warehouse and they're working appropriately and so it was a really simple way to say this functionality now moves. In general the balance has really been by the way that Donald and I focus our time right now. So my main focus has really been on hardening and continue to operate PyPI proper and Donald's focus has been on his time when he contributes on driving and helping guide the warehouse development. So I mean if the old code base is what's currently taking traffic it requires a little bit more care and feeding obviously and I think the next big phase for where I'm going to be considering thinking is starting to load test and canary traffic over to warehouse. So the question was the question was thank you for running PyPI for us thank you for using Python and PyPI so still more time for questions. So again for uploads what's that oh sorry the question is when is pip going to switch over to the new URL. So for uploads recent versions of pip and setup tools are already using it for uploads. For installs I don't know I mean I assume at some point once we're ready we'll switch it or not we'll. The setup tools and pip maintainers will switch the URL but I think the real plan is just to basically take care of behind the scenes with the CDN and start diverting traffic because the PyPI.python.org domain is not going to go away anytime soon. It's baked in I'm sure to all sorts of places. So the question is why is the new domain better for you and what does that help. Primarily security concerns you know again subdomains and domains and you have to be very careful with cookies and all sorts of other concerns around TLS certificates etc and so that's the primary driver and also simplicity we don't have to sort of contend with other DNS records in the in the broader Python.org scope Sure. Like I said github.org So gitub.com pypaiw werahouse You'll find an issue tracker there and it's like completely full of issues. There's everything has been very well documented. Donald has done an excellent job, Donald in the community and users have done an excellent job of documenting it. So it can be something super small you know I'll make a change to to some limit in like, you know, the rate limit for the current implementation, right? The threshold might be here. Donald is like, you need to make sure you write an issue to make sure we don't miss that. So pretty much anything from like the most minor to the most major has been documented there. And so any issue there, even if it feels really trivial, is a great way to contribute. So if you check out warehouse.readthedocs.io, the base requirements right now are running docker environment. So the entire, so the entire sort of local development environment can be spun up with just a single docker composed invocation. It's really awesome work by Donald. All right, I think I'm done with questions. So. So.