 Okay. So our next speaker is Nikita. Please give him a close and... Hey, thanks a lot. And first let me introduce myself and before that let me say that it's always great to see so many people in this room, people who are interested in Rust and the language itself has started about two years ago, a little bit more than two years ago and there were maybe a couple of hundreds of people who were interested in that language and now it's really great to see that it's getting traction and see a lot of people in this room who are interested in this topic. So introducing myself, my name is Nikita. I work at Madesafe and Madesafe is a company that develops peer-to-peer network technology and I think we have one of the largest code bases that are written in Rust. So today I want to share my experience developing one of the libraries that comprise our stack. And today I want to talk to you about parsing. So parsing is one of the crucial steps in the topic of compilers and the compilers is a complex topic by itself so it will be almost impossible to cover all the things that go into compilers in 20 minutes. So this will be more like an observing talk. So I will cover the basics and my goal here, my goal today is not to give you the entirety of information about compilers but to inspire you and basically to pick your interest. And actually compilers is a complex topic and a lot of people are scared about it. They open the dragon book, they see that it consists of more than 1,000 pages and they close the book and never read it again. But actually compilers are a very interesting topic and if you take one part of it, they can be immensely useful and they're not that complex. So let's start from the basics. So how parsing can be useful for you? So what is actually parsing? So if you don't know, it is one of the steps in a compiler which allows you to extract meta information about your code. So you can actually imagine the compiler as a kind of pipeline that consists of several steps. And the first step in the compiler is the lexical parsing, the lexical analysis. The lexical analysis is a simplest step that gets your code in its text representation and transforms it to a set of tokens. A token is a very simple thing and can be thought of as a word. It can be a keyword, it can be a string out of your code, it can be a number, it can be basically anything. A return is a token, a fan is a token. But having just tokens, we can do many useful things with them, right? We can do many useful things with just code as text. So what we do here is we put tokens in a token, in token trees. And out of those token trees, we create, out of those token trees, we create a data structure that's, that is called abstract syntax tree or AST for short. AST allows us to extract meta information out of our code and use this information to do basically anything. So how do we apply ASTs for our practical code? You might think that ASTs are usable only for the Rust compiler itself or for building some, I don't know, some utilities or things that belong to the Rust compiler. But actually, ASTs have many everyday applications. So you can extract meta information from your code and based on that information, you can generate code in other languages. This is very useful if you want to, for example, if you deal with FFI or foreign function interface and you want to create a library in Rust that you want to make usable from other languages. And it will be very tedious and complicated to write this code in other languages to work with your Rust library. That's why we have ASTs. You can basically scan all of your library's code and extract that information about the functions that you export from your library and generate code in other languages based on that extracted information. And you can also transform your code. It is practically when you want to apply some kind of refactoring to your code and I believe the next talk will be on this topic. So for example, you have Clippy that shows you many compiler errors in your Rust code. And it is very useful, but it will be very nice if it could do automatic transformation or automatic refactoring of your code, right? So that's what you can do with AST. With AST you have full information about your code and you can also generate new code out of ASTs. So basically what you do here is you're taking the code of your library and you're generating new code based on that code. And as an output you have a refactoring code. And there is also the metaprogramming aspect. It's basically related to macros and macros are very similar to how ASTs work. So macros basically employ the AST framework that you have in your compiler, but the macros are accessible to only within the scope of the compiler. And with AST you have the full information about your code. So basically that means that with metaprogramming you can generate any Rust code based on the full information that you have extracted from the AST. And of course you can use ASTs to compile your code to a binary, but thankfully the Rust compiler does that part for us. And of course you need to know that the use cases are not limited to those that were listed on the previous slide. There is a lot more use cases and they are basically limited only by your imagination. But of course ASTs by themselves are pretty absurd concepts. How do we actually apply them to Rust? Because this is talk about the Rust language, right? And in fact Rust libraries, multiple Rust libraries already apply that concept and we use these tools every day. And Rust FMP is also another great example of code transformation because what it does is basically takes the AST that it has extracted from the source code and outputs this AST with a formatting that follows the preset rules about how indentation should work, how to put the brackets, where to put the brackets and so on. And Clippy is a similar beast. So it extracts the meta information from AST and based on that extracted information it checks whether your code has any bad patterns in it and suggests how to fix those bad patterns. And there is no magic in it. You can basically write your own Clippy based on the compiler technologies. And finally if you use IDEs or text iterators with a Racer plugin installed you are also using ASTs. ASTs extract information from your code and you have code completion, syntax highlighting and all the nice things that IDEs provide you. So IDEs basically a compiler that extracts information from your code and makes it more usable and convenient for you to write your code in the user's interface. So how do we actually use these ASTs? And of course you don't have to write your own library or you don't have to write your own code to extract this information or parse your code into ASTs. You have it already in the Rust compiler and you can use the lip syntax that is an integral part of the compiler to basically extract this parsed information and use it in your code. There is a caveat though that lip syntax library is available only in the NITIL version of compiler. And Rust use it to parse your code and it provides it basically as an API but if you can't use the NITIL version of the compiler an alternative version will be syntax. That is a crate that has been imported from the lip syntax to a separate crate. It's basically a code that has been taken from the Rust compiler and put into a crate that can work on a stable branch of the Rust compiler. And it has the exact same API so you can basically use it interchangeably in your library if you want to. If you are not allowed to use a NITIL version of compiler you can basically just link this library and it's a drop-in replacement. But there is another caveat in that this library has been deprecated and it's usage mostly confined to legacy projects. Well, you can still use it. It's still available on the crates.io repository but there is also an alternative and if you're starting a new version of your... basically if you're starting from scratch and not from some kind of legacy code you can use an alternative library that's called SYN, S-Y-N. And it is used by multiple projects too and this syntax crate is used by Rust FNT stable so it's basically a matter of preference although the authors of this library would prefer you to not use it. So how do you actually... to put it in practical terms, how do you actually use that? You basically import the syntax which is part of the Rust compiler as a crate. You add a separate feature flag that says that it's allowed to import private parts of the Rust compiler and you basically import it and use it as a usual library. So then you provide a file name of a Rust file that you want to parse and as a result, you get a vector of items. So items are actually enums that represent all that you can use in your code. So it represents functions with their arguments. It represents statements. It represents all the expressions that you have in your code and a nice thing about it is that it's an enum and as you know, we have a very nice language construct in Rust that works with enums that is called match. And by using pattern matching, it's very simple. It's very easy to actually extract the information from those items. You basically match on a type of your AST item which could also carry all the relevant information with it in its enum variant and you use it however you like. So that's how it could look. It could be a function declaration which has its inputs, its arguments together with its types and it has also an output type. So it goes on like that with other kinds of items with other kinds of ASTs. So it's not a rocket science so you basically can continue in the same manner. And finally, there is a code generation aspect. So code generation has lots and lots of useful and practical applications. For example, when Mozilla started to develop Serbo, they stumbled upon an issue that there is many libraries that are not written in Rust yet. So to use these libraries that are written in C and C++ they got together with a library that's called RustBindGen. So what it does, basically it uses C-Lang to extract the meta information from C++ code. So basically you can think about it as an analog of that lip syntax from the Rust compiler but it works for C++. So it extracts the meta information from the C++ code and generates code in Rust. So that way you get a library that can talk to a C++ library but you don't need to write it by hand which will be, of course, very tedious and error-prone. So then there is a serialization aspect. So if you work with protoboffs or ARO or other frameworks that work with other languages they basically follow the same pattern. They have a description of data structures that you might need to use in different languages and they basically use code generation to provide this feature and in Rust you can do the same and I believe there is already a bunch of libraries that provide this thing for you but if you want to make a similar thing you can use code generation as well. And finally, why do we use ASTs? Why do we use parsing instead of just going by macros? Well, first of all, macros don't have access to the environment that you run your programming. So macros are kind of isolated by the compiler context. They can't access the input output. They can't treat a file from your file system. They can't go to the network and so on. And with parsing, by parsing your code you can basically have all the power that the Rust compiler and the Rust language provide you and not only the Rust language and the full power of your environment. And the second aspect of it is that macros 2.0 that should be available in the Rust compiler soon should cover at least a part of these use cases so you might want to follow this closely. And finally, how do you actually generate your code? You can use a tool that utilizes the builder pattern. You basically call the functions. So to construct a function you will call itemFN and provide the context of that function. You provide the function name. You provide the function arguments and you provide the function body. And as a result you will get the AST item that will contain all the relevant information for your function. And from that AST item you can convert it into a string and get the rescue as a result. But it's not very convenient. So whatever you can use instead we can employ the concept that is called quasi-quotation. It is widely known and widely used in the LISP language and it's widely used by some functional languages like Haskell for example. And the concept is quite simple. It's a very resembling of macros but actually instead of outputting just a string it provides you with tokens and the AST items that you can use further. So there is also a Rust library that's called quote that provides a bunch of handy macros that you can use to generate your code. It looks like this. As you can see it's almost the same as you would have written it if you have dealt with macros. So you can just create a new identifier and place it in your code instead of placeholder. And as a result, if you convert it into a string you will get a resulting function, a resulting source code. And you can now put it into a file or do whatever you'd like with it. And finally a small case study about how can you use parsing in your project. So at Matesafe we dealt with our task was to provide a library to be used basically in multiple languages. We have a library that talks to a peer-to-peer network and we want people to use it from basically any language that exists and we provide this library as an FFI interface so it exports a frame function interface and it's available as a C library as any other C library. But as we wanted to be used by JavaScript or from Python and from Java and from other languages we... And it's a very tedious process to just write that code by hand to write the language bindings code by hand. We came up with a solution of parsing, basically parsing our entire library and looking at the exported FFI functions and generating Java and C-sharp and Python code based on that. And besides that Java wouldn't work simply with that code so we also generate JNi bindings in Rust so we generate Rust code that exports JNi functions that Java understands and that's the entire code of that library is generated by basically as a result of code generator and parser. So that's it and if you have any questions please ask. I have a question. Is it possible to manipulate the ASD during the compile time? I mean if I'm compiling my source code, my library and I want to replace some structures during the compile time of the same library I'm compiling is it possible to do that? Well the question is can you use... basically can you transform ASTs in the compile time? So when you're compiling your library can you transform the ASTs to have some other representation? And of course it is possible and you do it with macros so basically that's what macros does. They take the AST3 and they produce another AST3 so that is sold by macros and also there is a procedural macros thing and there is also a macros 2.0 so Rust provides multiple ways to transform your code to some other representation and you can choose whatever suits you better. Well specifically I was thinking about like generating a mock object and replacing it with original instead of original structure so that during the specific test case I would be using not the original structure but the generated one so that I wouldn't need to define traits for that. Yeah so the question is can you basically do a mock testing in Rust by replacing structures with some other versions with instrumentation and test functions and so on and yeah I believe it's possible if you use the compiler plugins and the procedural macros and if you use the derive key word that what it basically does and that's also how the Serda library works it derives your structs so you just write derive serialize and the rest the Serda library takes that instruction and generates the serialization and deserialization code for your structure so I believe that can work in a very similar way for the mock testing and the mock features so for the mock version of your structs. Any other questions? Yeah so the question is is there a project in Rust that is basically compiler generator that takes a description of a compiler and generates a compiler from the BNF grammar or whatever it's called. Yeah there is a bunch of such projects I believe one is called LALR and it is based obviously on lower parsing so yeah there's a multiple projects and you can basically find them by googling Rust compiler generator and there is a prominent once that are being actively developed so you can find many of them. Thanks all for your attention.