 Thank you for coming. My name is Raphael Gomez. I work at Octobus. We're a small consulting company specialized on Mercurial and I'm going to talk about how we interface Python and Rust within Mercurial to well, you'll see why and what are the challenges that we faced and how we fixed them, how we didn't yet for some and yeah, so for those of you who don't know Mercurial is a version control system in the same generation as Git. It was made in the same month of April 2005. It's written mostly in Python. It has C extensions for performance reasons mostly. It handles huge repositories for companies like Facebook and Mozilla with millions of files and revisions and it has a very powerful extension system that is super interesting and I have no time to talk about it today. So maybe check that out for yourself. So why did we choose Rust? I just said that we had like 40,000 lines of C code. So why move to Rust? Why use Rust in general? So as a recap, Rust is a low-level language with a very powerful type system. It has no garbage collector, which is quite important when you're trying to interface two languages. If both have garbage collectors, you run into some very tricky problems, so that's nice. It has a lot of compile-time memory safety. Some people might argue that it's too much or too little, but it has more than C does and it allows for simpler parallelism than if you were to write the same code in C or Python for example. So compared to C, maintainability is a lot better. It has a better signal-to-noise ratio. That means that the more of the code that you write actually is the algorithm that you're trying to do and has less to do with just freeing up memory and getting segmentation faults. You have better compile-time guarantees, which boils down to the same thing. You have a standardized and modern tooling in the form of cargo and auto-formatting and a test suite, for example. And it's saved by default by some definition of safe with a few escape hatches, which are very easy to grip, like the unsafe blocks and a few others. There are many reasons to choose Rust, but one of the main reasons is for performance. It's comparable to C for sequential code, kind of. Parallel code is much simpler to write and to maintain. So for us in a version control system, it's a very good asset. And it allows for optimizations impossible for C compilers. I'm not saying that Rust is faster than C as it is. Sometimes it is, most of the time it's not, but it's pretty cool. So there was an experiment made by Valentin, who's a software developer at Jane Street, and he did a very small subset of the AG status command that was written in pure Rust on a big, big repository that they have. And as you can see, the performance is quite a lot better than what we have in the Python and C version. It's orders of magnitude better. So that sparked a lot of interest in the mailing list and in the community in general. And the plan to put Rust inside Mercurial was already kind of in the works, but this kickstarted a lot more work, work that I was a part of. But just before I joined, the Mercurial project chose to use Rust C Python because there are many ways of bridging Python and Rust. And Rust C Python is the one that they chose for many reasons. I really don't want to get into them. It compiles on Rust table. So there you go. It's composed of two packages, you could say, crates in the Rust ecosystem. It has a low-level crate for binding to the C Python ABI. So basically you would never interact with that. And a high-level crate to interact with Python so to expose a module that looks like a Python module to Python, create functions, classes, and execute Python from Rust, which means that you get like an eval function. You could just put Python within Rust and just execute within a context. So we have the following structure within Mercurial. I'm only talking about C Python now. So for people using PyPy, I'm sorry. So the pure Python code, of course, talks to its backend, C Python. The C extensions do the same thing. And Rust is split in two crates. The first one being HGCore, which is a standalone library that has no idea whatsoever that it's talking to Python. And HGCpython, which is one of the crates that can glue HGCore to a Python executor. It's the main one that we're working on because C Python is so common. And, yeah, that's the structure. So I was very excited to start working on Rust at first, and it was not super fun for many reasons. The first non-trivial thing that I started writing was about twice as slow as the reference implementation, so that was not super cool. And the reason why is because of friction. When you're trying to piece two languages together, no matter the languages, except if it's like C and Rust, you basically always have issues at two levels. You have the developer level, so with the complex interface code that you have to write that is basically completely orthogonal to what you're trying to solve, you're trying to do a problem, and then you have to take your data structures and move it to and from the other languages, which you really don't want to have to think about, but you really do. And exchanging data in general is costly. Because you have, at the very least, you're moving memory around. At worst, you're allocating a lot of it, which is very expensive, and then you have to loop over all your data. So, for example, if I am statting 100,000 files on my laptop with hot kernel caches in parallel and Rust, it takes about 30 milliseconds, which is pretty cool. And if I give the results back to Python, it takes about 300 milliseconds for anything useful, so you just add up one order of magnitude more than on top of what you were actually trying to do, negating the entire point of doing it in Rust. So, we have a few possible solutions. You can just exchange less data. So, you move up one abstraction layer and you, for example, instead of giving the bytes of the file, you give the file name and make Rust open the file, that kind of stuff, which is pretty much the same as doing more in Rust. And you can also communicate with C directly and not go through the Python layer. So, if you go back to the structure that we had, there's this arrow that we're trying to use. And it turns out that in the Python standard library, we have something called capsules, which are Python objects that encapsulate, that's the typo, function pointers. So, their main purpose is to allow for a shared C API between extension modules. So, that's what we're trying to do because Rust targets the C API, A-B-I. You can open your favorite Python interpreter and look into the data module and see a capsule object. So, there you go. That's in Python. But moving up abstraction layers, you need more powerful abstractions. And there were some missing features in our abstraction layer, which is Rusty Python. For example, there was no way of dealing with and creating sets. So, Python sets are a very useful collection to use. There was no support for capsules. So, we kind of had to do that also. There is then more hairy stuff. Inheritance for classes written in Rust. So, what that means is if you write, if you create a class, a Python class from Rust, it has a Rust backend, if you will, and you try to inherit from that in Python code, it will completely just crash at an interpreter time telling you that it's not a valid base type. And it has to do with the fact that if you try to inherit from this class and you forget to call in it, then what happens to the memory? Rust needs its memory to be initialized. It doesn't just work like that. So, you can't really extend a Rust-backed Python type. So, you have to use composition over inheritance, which makes some people happy, but it makes the performance really unhappy. And it makes me unhappy because they have to do all of this manual encapsulation, which is slow and cumbersome. Properties don't have properties yet in Rust-C Python. They're very useful. And if you're trying to build a drop-in replacement of a class that you've written in Python and just trying to rewrite it in Rust and expose the same interface, you have to, again, do the same encapsulation mechanism to have properties that call your inner methods in Rust and blah, blah, blah. Set adder is not in Rust-C Python, which means that you cannot have instance attributes, which is very limiting. And the last one, but not least, is when you're trying to build an iterator on Rust collections. So, I'm going to talk about this one a little bit more interesting, and we've found a solution for it. The main idea is that you want to have a Rust collection, let's say a vector or anything that you can iterate on. And you want to pass one value at a time when it's being iterated instead of just taking the whole collection, moving it through the FFR layer, having two copies of the same stuff and trying to synchronize it, and it's just there. So, you want it to behave exactly as a Python iterator would. But to do that, you have to tell the Rust compiler that it really has to let go of the memory that it's trying to hold on to, and it basically revolves around sharing the references to the same object between the two languages. So, these are not my slides. All right, that's okay. There's supposed to be a slide here talking about the work that we've done, but basically, my colleague did a proof of concepts in early June last year, and then I upstreamed the first, I would say, non-trivial data structure version of it in Mercurial about a month later, and it's been upstreamed this week in Rusty Python, so that's pretty cool. So, upstream work, that's not up to date, because PySet has been done by my same colleague, George. Thank you, George. PyCapsule, same stuff. You have a useful macro to define capsules. Properties are being worked on, I believe, by, I don't remember who, but someone is working on them, and iterators and Rust collections are also a thing. So, this is our target. This is basically the lower bound of what we're trying to achieve because we know that this is unrealistic. It does not do everything that status does, and it's very focused on doing one thing exactly, and it's all very tested, et cetera, but it's a lot faster, and we know that we can do a lot better than what we're actually doing currently. So, where are we now? I have two cases, the first one being pathological cases in our favor, something that favors the newer code. As you can see, the new code using Rust is faster by a good margin, which is nice, definitely, but for a more realistic case, we see about 50% improvement in performance, which, again, it's very nice. It's cool. It compounds nicely if you have a CI system or something, but it's very far from the thing that we're trying to achieve. So, there's a lot more stuff that we can do to make this go faster. I talked about how Rust makes writing parallel code a lot better and a lot easier. We could do more things in parallel. There are basically three main loops within status, and I think only one of them is running parallel as of now, and it's really the simpler one, and it's not really the most expensive one. So, just running those things in parallel, so say if you have 100,000 files, you don't want to check the first file, do all the things, then second one, et cetera. It's very easy to parallelize most of the stuff, especially now that most computers have a lot of CPUs. It actually gets a lot faster. Better conditional execution. That has to do with the fact that Mercurial is a 15-year-old code base with a lot of features and a lot of bug reports that have added little lines here and there, and you don't want to break the backwards compatibility that you've had for 15 years with the huge tests that we have. So, you want to work incrementally when you're trying to put Rust inside of Python code, and one of the things that I've done, because I have mostly been working on status, is doing the naive thing, just figuring out if it's any faster, if it's any faster, it's good enough, and then work in a different iteration to get it to go faster again. So, you can maybe think about the things that are not useful to do in any given situation. Maybe we can optimize some paths and maybe some not. We can rethink the order of execution. That's something that Rust allows us to do because it has such a strong type system that some invariance or some constraints of the problem we're trying to solve can be more easily fixable and reasoned about than if we were trying to use Python. For the very more complicated parallel stuff, maybe we could do two loops at the same time and do two things that weren't possible before because it was just too much to think about doing. Of course, fewer exchanges between Python and Rust. That's pretty obvious. If you're moving fewer things and defining better interfaces between Python and Rust, maybe you have less overhead for the foreign function interface that you're trying to define. It has a better approach for maintainability in your project. It has a cost of defining strong interfaces. That way, Python can do its job, its very good job of using extensions and being the very productive language that we know and Rust can have its own job and being the very fast language that we know. Of course, the usual suspects of optimization, so basically allocate less in general, make fewer allocations even if you have to allocate the same amount of memory, using memory alignment, all that kind of stuff, and something maybe a bit more controversial in this room, not start Python at all. So, I'll get to the next slide. But Python has a start-up time that can take tens, even 100 milliseconds sometimes. And if you remember the slide I showed you, that's about the time that it takes to run status on 100,000 files. So, maybe we can completely bypass Python in some cases because Python has a very good place inside Mercury and it made Mercury what it is now with its extendability and metaprogramming capabilities. But if you're trying to run a CI, for example, and you're always asking for the status and the log and the diff and whatever, and this is not very... You don't really need customizability. You need something that is very fast and as fast as possible. And the start-up cost of Python actually costs a lot just running Python at all. So, maybe for some aspects we can completely counter it. I don't have time in this talk to talk about that. Maybe we can discuss it later. Working so much with Rust has given me a renewed appreciation for Python. Code is very easy to understand. Even with a very weird Python that we have to write in Mercury because we're really at the edge of performance in Python, it's basically writing pseudocode most of the time and it's such an achievement in programming language theory to just have a language that most people can read without having to bother themselves with a lot of details that they don't care about. You get something that works very quickly. There are some extensions in Mercurial. I said I wouldn't talk about them, but just a little bit. There are some extensions in Mercurial that are 100 lines, 150 lines that do a lot of work and it doesn't do it super fast at first but it does it at all and it's so great. So, it allows for experimentation. You can figure out your interfaces later. It's very rare to break your entire project by just changing something somewhere in Python. Usually, you can get away with it, whether that's a good thing or a bad thing is up to you but it allows for experimentation and it is a lot faster than Rust code that you are not done writing because you have the 15th error message that tells you the same thing. So, thank you. Do you have any questions? Do we have time for questions? Do we have time for questions? Sorry, oh, sorry. Hi, thanks for the talk. I have two questions. When it's related to starting Cpython, I wondered if the code integration really worth it or would it be better to just have Rust-only comments and Python-only comments and just start one or the other depending on the Mercurial comments? So, if I understand correctly, the question is, is the interfacing of the two languages worth it or should we just bother with Rust or Python depending on the use case? I would be very happy if I didn't have to do any foreign function interface work but it's the best case you have because having a complete rewrite of your software is usually not a good idea at all so you want to work incrementally and that starts with just, you know, taking a small bit of your code and putting it in Rust and as I said, both languages have a lot of advantages and you don't want to give one up, I would say. So, I'd say it's worth it. We still have time for a couple of questions. Yes? Yeah. How do you test... How do you test newer code that doesn't break previews? Oh. So, how do we test that the new code behaves the same way? It has to write new tests for Rust. Oh, yeah. How do I write tests for Rust? Yeah, and make sure it doesn't break anything. Yes. We have, most of our test suite is integration tests so they talk to Mercurial... Quiet people, please. They talk to the command line and we see the side effects of what happens. So, most of the test suite has no idea of what the implementation is. Whether that's a good thing or a bad thing is sometimes, you know, kind of tricky but it allows us to change the implementation and just see if it broke anything. We also have Rust-specific tests for, you know, smaller functions but most of the test suite has no idea of what's going on in Mercurial. Yep. Where's he? You could just read the question. Yeah. So, actually related to the first question, to avoid the FFI, would it be an option to make a server in Rust and then talk to that server over socket, for instance? So, the question was, would it make sense to, instead of FFI, to use a server client with Rust and Python? Yes. And in terms of performance, it does not hold up. We're talking, we're trying to shave off milliseconds of runtime and going through it like an HTTP stack or a socket usually does not make sense but maybe, I don't know, maybe in some cases it could work. Okay. I think that's enough.