 My name is Jimmy Lai. I'm a staff software engineer at Carta. Today I will talk about Python linters at scale. So in this talk, you can expect to learn about the different scaling problems when using Python linters and learn about what kind of tools you can use to solve those problems. For each category of problems, we provide a checklist to help you navigate through the solutions in a more structured way. So we start with some context. We have many Python code bases and one of them, we call it monolith. It's a large Python code base that has a lot of code. In our case, it has more than three million tonight. But in the meantime, we also have other smaller Python code bases. They are either some microservices, GRPC service, or some Python libraries, and we also have hundreds of developers. So all of those comes with different scaling problem. People will start talking about the problems, so they could get some Python popular linters. Here are the linters tools we use at Carta. Black is used for code formatting, and it can format the code into a consistent way, like turn the red block code into the green block code to make them easy to read and write for developers. The configuration of black can be added to PyProject.tomo in a section toward that black. The default millions is 88. You may want to extend it to a longer number, considering everyone has a white monitor now, or you may want to specify the target version, or exclude some generating code. We also use iSort to sort our imports. So the different import statement from the same module can be organized together and sorted in a perfectly. The configuration can also be added to PyProject.tomo. When use iSort with black, you may want to use black profile to keep the format consistent with black format. We also use PlayGate to check our code style, syntax error, and box. It can provide a suggestion like undefined name or end-use variables. Unfortunately, PlayGate doesn't configure to PyProject.tomo. You can use any of those config file when use PlayGate with black. You may want to ignore some rules from the black aid in order to avoid the conflict. Otherwise developers may feel confused. We also use myTie to type check our code, which is really useful to catch type errors while developers editing the code, instead of finding until the code was released to production. It can be configured in PyProject.tomo. And we usually recommend configuring myPy by enabling those configs to make the type checking and type annotation more strict, which can make the myPy suggestion more useful and more specific. So far we can see we want to use a lot of binter and each binter has a lot of config. And in general, we usually configure our binter by using our version control system like it. So we will add our binter version and binter configs in our code base. We can use a specific Python package management tool like pip or poultry, provide the binter version. In this example, we specify sort version in a requirements file so we can use pip to install it. We also store the config in PyProject.tomo. So when we run dinters, we will use that specific config in the code base. And yeah, that's the common approach. So once we have the version and config in the code base, we can do run them in the local environment where developer edit the code. The goal is to help developer detect dinter errors early at development time and interact fast. We can use the mentioned Python package management tool to install the package or install them in the darker image. And in either way, we can also configure the git commit hook to automatically run dinters at commit time or integrate with IDE through FireWatcher or language server to monitor IDE events. So by doing all those, we can help developers run dinters easily. And the developer can also choose to run them in a hard way through COI command. Another environment we want to run the dinter is the continuous integration environment, which we want to always make sure the code merged to the main branch don't introduce new errors, test error or dinter errors. So whenever a change is pushed or proposed, we want to run all the dinters by starting a BI runner and in order to fast finish the dinter run fast, we will want to pre-install and catch the dependencies in the CI runner either through remote cache or using darker image. That way we can immediately start the CI runner when a commit is pushed. Okay, let's start looking to what are the scaling challenge we are going to base and solve. So the first challenge comes from the large code base we have. In our example, we have more than three million sites of code and more than 30,000 Python files. When we run the mentioned dinter in this code base, black can take more than 10 minutes to finish. MyPy can take 30 minutes to finish, which is too much time for developers to iterate back. Another challenge of scaling comes from having too many code bases. We have many code bases and initially different code base enabled different dinters with different version and different configs. So a lot of inconsistency that code for developer experience and in order to make them consistent with the then best practice is take endless effort to upgrade the dinters and configs. And the third challenge comes from having too many developers all of our developers they constantly submit requests to make changes for our code base. And it's not easy to understand how dinters work for them for each developer, each code base and each code changes. It's just too many of them. And before we having merge queue, we also occasionally has dinter errors merged to the main branch, which cause issues, blocks other change and the simple dinter situation if it show up too many times, it can slow down a lot of developers. We want to help them faster. And also so far we only talk about expected dinters or Python, but to development, we also use other tools. We use GitHub, we use Docker and many other tools. We actually also need best practice for those tools. So we summarize our challenges in two categories. One is the dintery also another one is the full developer experience. We develop and try to solve the solution over the past couple years. And here are some of the different approach we take which worked. So for stating up the dinters, our strategy was to avoid unnecessary co-manages on large number of code. So here's the checklist which summarize different approach. The first one, if we can only run the dinter and updated file, we can make the run fast. When running dinter locally, we can try to get the updated file from it like doing a github or if you are integrating with your ID editor, you may use a file watcher to get the list of updated file and provide them to the dinter. That way dinter only analyze the updated code to be fed. And in CI, when developer propose a change, they only updated a few files in their progress. So we can get the updated file list from the github or API like this. Even for request, we can get a list of updated file and provide them to the dinter to run faster. Another way, a tool that you can use to only run an updated file and run different dinters in parallel is the pre-commit tool. So it's a tool that provides several features. It can run on a committed file, help you configure your pre-commit hook easy. And when you configure multiple dinters, it run them in parallel. And it automatically set up a for sure environment, install the dinters in the environment and reuse the environment. So when you run pre-commit multiple times, it can reuse the environment. So the configuration is something like this, pre-commit config in mobile by adding a config like this. We add a black version 22.10 to be configured in the pre-commit. So we can run the pre-commit command or just commit our code, then the dinters will be run. And when you configure multiple dinters, it will run them in parallel. Another strategy to make dinter run fast is to reuse the prior results. Especially for large copays, some dinters like MyPy, they which require the knowledge of the entire copays and entire dependency graph, which can be a huge analysis. If we cache the previous run results and store them remotely, we can download them in local environment or in CI environment to be reused. So the workflow to use remote cache is like this. When we start the dinter run, we will check if the remote cache has a recent cache that we can reuse by looking up the revision we are currently at. If a cache is available, we can just download the cache and run the dinter and the dinter can reuse the previous analysis result. At the end of the run, we also update, upload the updated cache back to be reused later. In our large copays, MyPy used to take more than 20 minutes, but by using remote cache, we can improve the runtime to be less than five minutes. Another strategy to not only reuse the prior results, but also try to find a faster implementation is so, an example is like graph. Graph is a rust implementation of Flakey and I-Sort. It can parse the code and run different analysis. And the implementation rust make it really fast. It also built some cache features. So it can provide pretty good performance and it's getting really popular this year. So now let's look into the second category of the scaling problem, which is the war developer experience. We talk about inconsistency and the entity's effort and hard to know what's going on. So the strategy for solve all those problems is to build linters for any back practice we want to apply and provide auto-fate to help developers fix any issues in a productive way. The check is include the imagery as the linters and auto-fate. The imagery is the way to help us understand what's going on. So we can collect metrics from the CI and local linker runs to understand where our linker run in which environment, which code base, what kind of suggestions it makes and how it runs, what's the latency, are there any exceptions based. So we use data doc to collect metrics like this. We can know MyPy make a unused type ignore comment, a full request at specific file, specific line. We can also know this MyPy run, it takes how much time and aggregate the latency metrics for us to work on speed up. So with the telemetry, we can understand what's going on. Then we can try to build more in terms to, and also provide auto-fix to help our developers. When we build custom Python linters and auto-fix, we can use a framework, it's called fixit. Here are some built-in rules that are implemented with fixit. The first best practice is to always, it's pretty easy say frozen when you declare a data class. That way by default you may want to make your data class immutable by making frozen equal to true. So this in turn can detect when frozen isn't specified and warn the developer and the suggested auto-fix is automatically act frozen equal to true. There's also another rule, such as use AppStream when a percent format is used. So this in turn help you build suggestions and auto-fix for Python code easy. And in order to provide better digital experience, we also integrate with GitHub check, which allows us to mark the check as required to protect our main branch and allow us to provide notations to show the data errors in the context of code to help developers understand the data error easier on their request. And also we mentioned we may have some other non-Python linters, which we want to apply some custom expertise, for example, before we have a merge queue, we found errors sometimes merge to the main branch when two requests, they have some logical conflicts in order to reduce this issue. We have a best practice linear tool, which is a bot to check every full request if the full request is behind the main branch too much. It will make a comment to recommend the developer to rebase, to make the rebase very easy. We also provide an auto-fix. So as it mentions here, developer can just add a rebate label on their full request to automatically check the rebates. And when, because of the telemetry matrix we have, we will know when an error show up on a developer, if the same error also show up, when the error show up on the full request, when the same error also show up on the main branch, we know this error is not caused by the change on the request, so the bot will make a comment like this, saying the error is from the main branch and a fix is already available, so they can just rebase to get a fix instead of wasting their time on investigating a related issue. We also built some other custom Python linters to help us be able to incrementally solve linter, solve errors incrementally, especially when you have a large code base that has a lot of tech depth. In this case, linter will start with making subscriptions to prevent new usages, but in the meantime, we also want to collect the existing errors to fix them incrementally. And in this case, the strategy we mentioned collect telemetry matrix and build custom linter will be useful. With this approach, in this example, we try to build a tool to deprecate some library. So this type sample is a target we want to deprecate it, but we want to stop the bleeding by blocking new usages and we know it still has a couple hundred existing usage from our metrics. So we continuously collect metrics by analyzing the code base so we know what's the existing errors and we can just help the engineering team incrementally fix those by providing this dashboard to them so they can easily find the action items and the owners of those items to collaborate with them and see their progress over time. Another tool we built to help out the developers across different code base reused the workloads, like the GitHub check creation, the dinters, so we used, we built a framework. Okay, we built a, sure, well, we built a reusable framework which it provides a simple API. So anyone can use those API to build their custom linters and autofix and they can focus on building the custom linters logic and the framework automatically collect metrics and generate the GitHub checks and without annotation. So the last thing about autofix is automatic refactoring. Probably you already tried to use dependent but to automatically bump your dependency package version. We built our custom autofix to fix some trivial dintter errors so developers don't need to spend their time on fixing them. We used the CSG to build custom comat and use a GitHub library to create requests. And so I had another talk last year talk about how we built the automatic refactoring framework that create requests and manage their life cycle to get them merged to solve the large tech problems. If you're interested, you can find my talk in EuroPython 2022 and also check my blog post about how we use it to solve type of notation problem. As I mentioned, we have a lot of autofix. So here are some example for Flakey complain about used import a lot. So we provide autofix that can use the editor but also the request. So we integrate with GitHub check. So GitHub check can show a autofix button like this. When developer click on the fix Flakey button request, the bot will pick up a workload automatically push a fix. That way they don't even need to open the editor or run it coming. Everything will be generated automatically. We also build autofix for some my pie errors. For example, one common one is non-return. When a function has no return statement it should has non-returns connotation and build an autofix which can automatically do that for developers. We also have a custom autofix. For example, the request has them being reviewed after it's created. So while in a large organization you may have different teams and different code is owned by different team. It's not easy to figure out who owns the code you modify. So we provide an autofix, notify reviewer teams the developer can have this label on their request and we will base on the call on nerds and the GitHub stack mapping we have to find the call on nerds sectional automatically send a reminder for the developer to get their request reviewed. Here's another example, release on merge automatically release, make a new release when the request is merged, if the label is added. So as a result, we were able to develop a framework that makes building linters to support custom linters on autofix or want to developers in more than 30 different code bases. And with this, we make running on the internet with the same calculation and that practice is easier. So far each week, the internet has run more than 10,000 times and provided 25,000 suggestions and the autofix also been used a lot of times. So to recap, we provide a checklist for the linters to speed it up and we also have a strategy for improving the developer experience. Thank you for your attention. If you are interested in more of our work, you can check out our engineering block. If you are interested in our job, we are also hiring. Thank you. Thanks a lot for your talk, Jimmy. It was nice hearing from you and it was very useful to hear about linters. And maybe will you be available on Discord maybe for remote people to maybe ask some questions for the next minutes or something? Thank you very much. Yeah, thank you. Have a nice day, Jimmy.