 Okay, let's start First I briefly introduce myself My name is Mitya Trofimov. I work for JetBrains. I am team leader and developer of PyCharm IDE But today I won't be speaking about PyCharm in this talk If you want to discuss anything about PyCharm, come to JetBrains booth PyCharm team will be there during all the conference ready to answer all your questions Don't be shy mostly I Developing Java in Python Java has been my primary development language for the many years And Python for me is more than a language that I use It's like a subject of constant investigation like a snake that I examine Which step of scales it has which teeth and what is inside of it how to rebug it how to profile it and Also, I'm curious about new programming languages and a half a year ago. I started to play with Rust a bit I knew nothing about it at that moment still I don't know much about it now, but I hope that I will be able to introduce it to those who are not aware of it at all By the way, who have you have heard about Rust? That's pretty much and how many of you have tried it already develop some stuff. Oh, that's cool For those who have tried it, it will be I hope to be also interesting because I will show some corner corner cases Yes, so what is known about rust? Yes, and answering one more question Why why did I pick a rust? Why why did I started to invest my time and learning it? I found it interesting and also I wanted to develop a profiler for Python and to make it work fast. So, okay, I will tell about it Rust Rust is my Zilla project and they actually are using it already for the new browser engine called server The project started in 2010 as a site project of my Zilla employee grade on Hoare and The version one was released on May 15 this year. So now it is 1.1 and Before version one things were changing very at very high pace in the rust breaking compatibility And now it's still changing standard library. It's not polished Yet and ecosystem around is just starting to emerge. But now it's Has bake what compatibility and this allows to develop production applications and rust So what is the rust? What did I know about it when I started to learn it? What exactly kept with my eye? They told that it is fast Prevents nearly all six-folds guarantees threat safety close to the metal. It has zero cost abstractions pattern matching and type inference And that sounded very cool. I thought it would be very interesting to learn it First I started to listen some talks on YouTube nothing became clear for them Then I found some specification and turned up that it is already outdated the language Had just changed and then I found rust book For all of you who's interested I recommend this book The starting point it's online and we're well written and I don't think that a talk can help you to teach the language That's why today. I won't explain basics of rust at all. I don't have a goal to teach you the language But to give you a feeling of it Okay, so let's start with a small but real problem and As this is my advantage of rust let it be computational problem like computing primes So the problem is to compute prime numbers between two and n and prime number is the number that has no Devisors except itself and one so like two three five so on and we will solve our problem with the help of an algorithm called Repertance sieve Algorithm is simple the work with this way. We first take all the numbers from two to n and Then iteratively throw away those who have devisors so something like that We start to take two and then we throw away all events and then we proceed To the number three we take as a prime throw all the multiples of three and Then it will be five and all multiples of five are thrown away and seven and so on So here we see a Python implementation of our defense sieve It's quite beautiful. Isn't it here? We initialize our non-primes as an empty set and then we to rate through a range and If the current number is not in the set we increment our counter and update all the Multiples we put it into the set as a non-primes and then return it from the function Okay, so let's run it This is our function and also we have here main function Which takes command line argument as n and then it executes our function and prints output Okay, so it's something like Okay, so we have four between two and ten and Four one hundred We have something that seems to be correct Okay, let's measure the speed the time of this program We will comment out the output because it's always slows down the execution and Okay, and for One million make something like this one. Yeah, it's It's More than half a second. It's pretty fast But what if our Task is what we do care about speed. What if our task is to implement the the algorithm as Efficient as possible one of the obvious solution solutions is to use C programming language because everyone knows that C is very fast and it's efficient and Very well very good programs are written in C for example C Python and written on C So let's implement this algorithm and see so the program now is a bit longer But it's pretty much the same. So we have our defense function Unfortunately, there is no set in the standard standard library of C So we use array of primes and we where all items mark as one This means that is prime and when it's zero, it's not prime and then we iterate through it and if it's prime increment our counter and Then update of the multiples of zero and then we put our array Into the result structure and unfortunately there is no tuples in in C So we need to have this result structure which holds a counter and our array So and here we executed and we print the counter and Then we want to print all the primes We first put them into the another array, which is the length of of the counter So we iterate again. We put them and then we print them. So Looks quite quite similar. So we need to compile this and We Run it for ten. Okay, it works. Correct. Now we run it for hundred oops and We have segmentation fault Anybody knows where the the error is Yeah, I know that it's difficult, but let's see it Managed to print the Count of primes. So probably the error somewhere here. Let's examine those lines again. So we have all primes array and Which is the length of the total primes and then we iterate through our First primes array and if the number is prime we increment this counter and put it here So probably we could think that we have we go out of range here, but it's impossible because we start from Zero and we increment it With the same condition where we incremented here when this is true so it shouldn't go out of the range and Here we just print the data out of the array. What what is the problem and Actually that that kind of kind of problem that that could be called like a beautiful journey in C because you have no idea what's going on and I'll tell you what says the problem problem is not here that the problem is here because when we Returned our primes array we thought that we are returning array But actually what we do in C is that we return the pointer to this array So this is the pointer to array. It's not array itself and array was Allocated in the beginning of the function on on the stack and Actually, it was valid only in the scope of the function and when we after we Return the pointer to this array and we go out of the scope This array it's it's just no more. It has ceased to be it Expired and gone to meet it maker So and we still have a pointer to do it and that is the problem very common for C programmers. So that's called A dangling pointer pointer that points to know where to some random some random memory and That's why we get it would have a segmentation fold here So our task was to get our primes as fast as possible solution was implemented in C But C is not a solution. I Know that there are programs implemented in C and probably there are people who are convenient with C and who know How to use C efficiently and probably they don't make such errors But something still tells me that sometimes they do and I personally after years of Java and Python just can't imagine how to live in the world Where you can suddenly become a Pointer to a random date in memory So let's carry on to Rust finally, let's implement the same in rust What we will what we will do now is Just to Reimplement our C program, but in rust and we will see how Rust compiler handles this station. So here we have Our C program and here we have quite the same rust program if you don't understand little syntax details It doesn't matter because I just want you to to understand one basic concept For example What we do here we have the same structure and Actually, it denotes the same as in C. We have counter that is integer and this is the The pointer the pointer to array and then there is no Pointers in the rust but these denotes like it's called slice in the rust so it doesn't Holds the data doesn't hold the data. It just points to some data external for that structure So it's actually the same as in C and Then here we allocate our vector initialize it and we iterate and increment the counter Then we do The same as in C and here we return our vector as a slice Okay, so let's compile that oops I messed up with typing Yes, and we have compilation error and Rust compiler tells us that primes does not leave long enough so what it tells us that Exactly what we have here that hey, man, you cannot Compile that I won't allow you because you just want to return the pointer to the memory that will expire after we leave the context of this function and Actually This seems to be very strict, but what is better to get this error just in time before you run your program or To debug some mysterious Segmentation fold just in the weeks after you deployed your program on the To your users for example, I Think this is much better But let's let's run it. Let's make it work. We won't Fix this exact copy Because it doesn't make sense instead of that we just to implement it from scratch in more idiomatic rust because the rust has set And it has tuples like python. So we have much shorter solution and it resembles us python and Okay, let's run it Okay And for one million Something like so yes, it's 20 milliseconds. So it's like 25 or 30 times faster than python and the concept that that Helped rust compiler to to deduce the error that we had is called lifetimes if you are interested about the it read first book So concluding our comparison Python is 25 times slower than rust and And See doesn't work just but so rust is fast and safe But that is exactly what they told us in the beginning nothing new and Returning to our main topic can rust to make my python shine Yes, but if you search in the internet about communication between rust and python You'll quickly find some tutorials about foreign function interfaces You will even find examples like this These Examples are quite Clear and simple and they work if you try So this allows you to call rust code from your python code But it's not enough What if I want to access see python internals from rust what if I want to convert python string object to rust string What if I want to return a complex object from rust what what if I want to make rust library? as a module in python and Actually, that is what is needed in the real applications For example a python profiler by the way, hope you have ever used that profiler for python Yeah, that's cool But for those of you who hadn't I'll tell what profiler is profiler is a Program that measures frequency and duration of function calls of another program and Normally the less overhead it has the better So let's make a python profiler and rust and to see how it goes actually what that was my Initial idea idea when I started to experiment with the rust to try to make for example, it's simple tiny python profile There are two major types of profilers tracing profiles and sampling profiles also called statistical profiles and statistical profilers they periodically capture frames for any program and Normally it has less overhead and tracing profiler which traces all calls in the program Let's see how to implement statistical profile and trust but here We won't go step by step implement all the program because we don't have So much time we'll just focus on two important aspects and maybe we'll learn something all the way and the aspects are periodically and frames so How to run task periodically there is no no time in rust and the library yet But there is a wonderful library called me or metal IO me or is a lightweight library providing effectively different operational system abstractions like timing and We just create a event loop Set up a timing event in it and then we run a new thread and We pass there our event handler and what is interesting here is an event handler Our handler will capture frames and save them to statistics map That is a sampler object that we create it's called sampler and As our timer works in a separate thread that means that our sampler is a resource that is shared between different threads So it's a shared Mutable resource which is believed to be very dangerous As everybody knows so that share shared mutable state is a root of all evil But not in rust rust guarantees you as a safe shared mutable state which sounds like a lie, but it's true What we do is we just put our sampler into mutics Immutal exclusion primitive useful for protect a protecting Shared data when you create a mutics you transfer ownership of the data into the mutics immediately giving up the access to it and Then any access to the data through the mutics will block threads Waiting for the log to became available Thus making the data accessible only through them through the mutics by one thread at a time and To pass the reference to another thread we wrap it with the ARC ARC provides reference counting through atomic operations And it's also safe between threads and having done all that rust compiler guarantees us That we won't have any race conditions never It's just impossible and Not having done that well You can't access that object from different threads Rust compiler won't allow you to do that. It won't allow you even to pass This mutable data to another thread so it will be single thread usage only So compiler guarantees you that you your program will work and to understand this better Read about ownership and rust So capturing current frame For simplicity we'll capture only current execution line as we are not interested in in call three at the moment There are three pieces of information file name function name and line number That we will collect at every tick of our timer In Python there is a function in module C's that is called current frame Under the hood it use function by thread current frames looking into the C Python internals we will find out that The structure that we need is called actually underscore frame So we have this underscore frame structure that points to some To some pike code object that we also need and What we need now we we need to convert it some somehow from from C structure to rust structure to be able to use it in rust and That could be sometimes hard because some C types are Not very obvious how to map to rust types. There is no No, no strict mapping no direct mapping because they are just absent in idiomatic rust so There are special rust types for that to fulfill that gap For example C void is analog of void and asterisk mute is a special type that reflects C pointers and Normally there is null in rust at all but To check this row pointers so we have special method is null so knowing that We Write our code and remember that when One tiny thing but which is important When you're calling C function or using row pointer rust can't guarantee safety anymore all such expressions should be within unsafe block and I think That is what they mean when they say that rust prevents nearly all sick falls because that is the way The word nearly means that it doesn't prevent sick falls in unsafe blocks when Europe is sick or So knowing these mappings knowing how to use unsafe blocks we just create our structures in rust and Seems that we are very close There is everything that what we need but how to convert Python string to a rust string funny, but that was nearly the hardest problem I faced because Actually it was difficult and at some point I came up with something like this so it's just to convert The Python string to the rust string at the last line and it did work sometimes It didn't handle some differences between byte strings and unicode strings, and I was already I already started to implement that but Then I came across a library called rusty Python by Daniel Greenwald and my life became much more easier That is a beautiful library and I highly recommend it actually it appeared that The most things that I needed to communicate rust and Python Were there I only need to to add Some details for my specific case and also it's very good example of rust code For example a string conversion using this library looks like that and also It handles all the cases with unicode string representation and also it provides very very important abstractions like special log Corresponding to the global interpreter lock and Python Yes, and this is this is how you can expose your native rust library at the Python module using Using the rust macros. It's amazing. We read the source. It's very very cool written, so it's Just this line. There are a lot of more under the hood and they are very interesting So Enough it was much code And much very much details, but I'm not finished yet Now as we have a couple of minutes, let's see how our profile of works so So we're profiling this Py that we have written At the first time Okay, that's not interesting. We need to Wow, I thought that it will happen at some point It's impossible to make live demo without fails Yeah, so we comment out our print pythons make matter Um, yeah, let's start with Okay And this was fast. Let's make it for 10,000,000 Okay, it's very simple now. It's very basic, but whatever to tell us that the 85% of our time went in the light eight and 14% and light seven actually Nearly 1% was for output of this line So line eight is this one So what we are doing here is we are dating our set in array and What we are doing here 15-14% is very incremental counter inside the array. It looks believable. So it looks very Very logical. Oh Yes, and if you Still care about Performance in your Python applications, but you don't want to dig that deep into the native code I recommend you to listen to the talk of my colleague Katerina Tuzova that will take place on 23rd of July and Thursday She will show you how to write performant Python code Without using any C or REST. Thank you for your attention questions Is it possible to distribute REST code as a Python package install which can be installed using people? Yes, I've came across little library that allows you easily to Write your setup dot pi this way that it will compile your REST code if you have rust compile that installed Maybe it even downloads it. I don't know. I never tried it. I Wanted to try it but had no time before before and you just type setup setup dot pi Be able to install and it builds your REST from scratch like for C So type pi allows you to install six tensions. So there is a library that allows it for REST also Thank you