 So, hi everyone. I'm Chris. I'm super excited about procedural macros as you may have noticed by now. And I'm going to talk about a particular procedural macro I've been working on, what lessons I've learned from it, and hopefully give you some advice that you can apply when you write your own procedural macro. So first off, what is a procedural macro exactly? Well, in a single sentence, I can say that a procedural macro extends the syntax of Rust. Let's unpack that sentence for a bit. So when you tell the Rust compiler or cargo to compile your code, you start with the source code and the first place it goes is into the parser, which takes your source code and converts it, or parses it, into a form that's easier for the rest of the compiler to understand. Magic happens after that, which we won't go into for this talk. And after all that, we get the finished binary, either an executable or a library crate. Where a procedural macro fits in, then, is inside the parser. You can see the bagel there. It's a bagel, by the way, not a donut. A bagel. And what the procedural macro does is whenever the parser sees a call to this macro, it takes whatever is within that macro call and hands it over verbatim to your procedural macro. Your macro will take this input, process it in its own way, and then hand back generated Rust code to the parser, which will understand it as normal Rust. In that sense, a procedural macro maps a custom syntax into Rust code, thereby extending the syntax of the language. This might seem a bit abstract, but in fact, procedural macros are everywhere. You've probably heard about Surrey for serialization, diesel for databases, or the Rocket web framework. But in this talk, I'm not going to discuss any of these libraries. Instead, I'm going to talk about something called mod. What is mod? Good question. Put simply, mod is a procedural macro that enables you to write HTML in line within your Rust programs. And yes, it is named after the My Little Pony character. It was a while ago, please don't judge. Moving on. To help understand how mod works, let's take a look at this example. This program generates a bunch of HTML and prints it to standard output. And as you can see here, it calls a macro called HTML. And inside of that macro call is completely custom syntax. The Rust compiler itself doesn't know what this particular syntax means, the thing in purple. But what mod will do is it will take this piece of syntax that's given by the compiler and convert it into normal Rust code that the rest of the compiler will understand. And this is what you see when you run that program, by the way. The string hello pinkie pie within paragraph tags. Now, if you were to run something like cargo expand to view the code that mod generates, you would see something similar to this. The real code is a bit more complicated for various reasons. But this is pretty much the gist of it. As you can see there, all it does is create a new string object, push a bunch of literals to it, and this rendered to method does HTML escaping. But other than that, there's not much to that code. If you were to try to write an HTML template by hand, you would be hard pressed to write something that's simpler or more efficient than that. And that's one of the biggest advantages of using mod or any other procedural macro solution, the really low runtime overhead. We can see this in the numbers as well. A while ago, I wrote a bunch of benchmarks comparing different templating engines just to get a feel for it. And as you can see here, mod is by far the fastest one. I was expecting some applause here, but never mind. But also, if you look at the label on the bottom, it says render time in nanoseconds. Like, if you've done much web development before, you'd know that an average web request takes around 100 milliseconds, which is about thousands of times bigger than even the biggest bar on this graph. So in the real world, most of the time, the speed of your template engine doesn't matter that much, but I think it's still cool that we can create something that is that efficient. Now, another advantage of mod would be static type checking. And in fact, I think this is the more important advantage. If we were to take that first example we had and replace that Y with an E, so it's his best poem instead, the compiler wouldn't let us run that. And in fact, this is the exact same error that you would have gotten if you had made that typo anywhere else, because at the end of the day, we're just generating normal rust, right? And that is in comparison to other templating engines which might parse and run the templates at runtime such that you wouldn't know about any typos until you try to run your app and everything falls down. Now, I've hopefully convinced you all that mod is amazing and you should all use it, but how did I come up with this idea for this library? How did it all start? Well, it started like many things by reading the official documentation. This is an excerpt from the old Rust book. It has since been rewritten, but this is the old one. And one of the sections in that book is about macros. And in that section is an example of a macro that generates HTML. And when I saw that, I thought, why don't I take this concept and develop it into a real-world practical library? Now, this isn't the first place that I saw this idea. I had already known about this general concept from other languages like Hamlet and Haskell, Slim and Ruby, and Razor and C Sharp. In that sense, mod isn't really original except for the fact that it's written in Rust. And I don't think that's a bad thing at all. In fact, it's a good thing because you already know that the idea has been proven elsewhere. And the nice thing about Rust is even though there are existing libraries which are really high quality, just because Rust is so young as a language, there are still a lot of places where the library support isn't that good. There is still a lot of low-hanging fruit around. And so if you have a concept that you're really passionate about but can only be found in another language, I think it's a really good idea to take that idea and try to import it to Rust. You could make a really good contribution to the community that way. So I got started developing the library, but early on, I ran into issues developing on Rust nightly. Because the thing is, even now, most of the procedural macro API is still unstable. There has been a big effort to stabilize much of the API lately, but the stuff I use is still nightly only. And the funny thing is, if you look at the history of Maud, you'd notice that its first release was in the beginning of 2015, but Rust 1.0 was released in the middle of 2015. And the key night among you would notice that I started working on this thing before Rust was officially released. And in a sense, that led to a sort of trial by fire, I guess. I didn't really have a choice to figure out how to work on nightly. I had to learn, well, I was forced to learn to deal with these breaking changes coming in all the time. And that gave me some really valuable skills for working with procedural macros. Here are some of the strategies I use to manage all of these breaking changes. Most importantly, I scheduled a daily build using Travis CI. The exact kind of continuous integration doesn't matter that much. The important thing was it was a daily build. Because the thing about Rust nightly is that it happens every night, right? I know, I'm putting out the obvious here. That means that if you have a daily build and you notice your build breaks, then you know with relative certainty what exact change broke your build. And that narrows down your search quite a bit. Research, Rust commit history, blame and PRs. The nice thing about the Rust development process is that it's really open. I think that was a major theme in the keynote this morning. And so, if you find something that breaks, it's just a matter of a few minutes of poking around and you'll probably find out what changed and why it changed and what you can do to fix it. And finally, subscribe to relevant tracking issues. This builds on that whole Rust being really open thing. All unstable features, including the procedural macro API, have a GitHub issue that tracks all of the work that is done on that feature. So, it's generally a good idea to subscribe to that particular tracking issue. Now, one other problem I had with developing mod was keeping the library simple and well-scoped because it was largely a one-man job. One way I managed this complexity was in keeping the syntax really simple. In particular, I put distinguishing markers at the beginning of every expression within that syntax. One example of this is in the let statement in Rust. If you've done any work with Rust before, you'd know that the let statement starts with the keyword let. And I know I'm saying a lot of obvious things here. And the nice thing about that is from a pausing point of view is that once you see the let keyword, you know that the rest of that statement is going to be a let statement, no matter what. There's no way that you can go halfway along that statement and suddenly realize, oh no, this is actually a while loop instead. And that means a parser can avoid backtracking. That's what it's called. Having to go back and parse everything again. Use the limiters, parentheses, square brackets, and braces as much as possible. One of the few things that the macro API gives you is that it groups these delimiters automatically. And so if you build your syntax around these particular delimiters, then not only will it look more consistent with the rest of Rust, it'll also make your job much easier because you don't have to match these up yourself. One example of where these two principles apply is in the syntax for substituting in dynamic content. So if you've used the templating language like handlebars or moustache before, you'd know that to include the value of some arbitrary expression at runtime, you would do something like the example in the middle. You would wrap an expression in braces. And I had to come up with a similar syntax for mod as well. The first syntax I came up with was the one on the top, starting everything with a leading character. And while that followed the first principle of having a leading marker on it, it didn't follow the second. And the problem with that was, well, imagine there's more stuff around this particular expression. For example, suppose there is a pair of parentheses after that carrot thingy. When the parser sees that, how do we know that that particular expression ends there and not somewhere after that pair of parentheses? Like this could be a method call, right? Or if there are square brackets there, this base could have been an array, and we could have been trying to index it. And so I ended up coming up with really arbitrary rules around where a particular expression is supposed to end. These days, we have libraries like sign, which let you factor out this sort of parser. But it still makes life a bit difficult. What I eventually settled on was the syntax on the bottom, wrapping everything in parentheses. And now this follows both principles, and it's really easy to pass. That's what I still go with today. And another issue I had was, well, I guess a piece of advice would be, don't fear the heap. By the heap, I mean structures like box, string, and vick, which let you choose an arbitrary size for a particular data structure at runtime. And the nice thing about Rust is it lets you avoid heap allocations altogether, if you want to. But that doesn't mean you have to do that. And in fact, zero allocation code can be much harder to read than the equivalent code that does allocate. And without benchmarking, it's hard to tell whether avoiding allocations actually makes your code faster. I think sometimes it makes it slower, I'm not quite sure. As you can imagine, early versions of more did not follow this advice. And I tried to write a version of the library, which didn't do any heap allocations. Problem is, when you're generating an HTML string, you have to put all that data somewhere, right? If you're not allocating a new string to hold all of that, you have to end up writing all of that data to an existing buffer. And to support that, I had the macro instead generate a closure, which took in an existing buffer and wrote to that instead. If you know about returning closures in Rust, you would have heard of import rate and not being able to name the type. There are a lot of issues around that. And in particular, if your closure ends up referring to anything in your list of arguments, that means that you have to annotate the lifetime of those particular parameters explicitly. And that's the apostrophe R thing there. And when I saw this, I thought, this is way too much effort just to generate a bunch of HTML. And I came to my senses and just the idea. Now, so far I've covered more user facing aspects of more, like the kind of things that you as a user would see. For the rest of this talk, I'm going to go more into implementation details. Things that you might not see as a user, but still help you, help you structure your procedural macro internally. First off, using an abstract syntax tree or an AST. By that I mean, rather than taking the input to your macro and directly mapping it to the generated Rust code, you should instead take that input and rearrange it into a sort of tree, a more explicit or simpler or more abstract form that is easier for the rest of your library to work with. Early versions of more didn't do this. You might be sensing a theme here. And actually, I thought that was really cool at first because, well, everything is done in a single pass. But the problem was the moment I tried to do any sort of analysis or optimization, everything just fell down. And I ended up rewriting it to use an AST anyway. So even if you don't think you need it now, please use some sort of syntax tree because you'll thank yourself later. Another thing is keep the spans. A span is a particular procedural macro thing that tells the compiler where a particular piece of code came from. By that I mean the source file, the line number, the column number and various stuff about hygiene. And the nice thing about, well, the reason why you should keep the spans all the way through is that it improves compiler errors. Imagine you generate some code and that generated code has a type error in it. If you follow this advice and keep the spans all the way through to the output, then when the compiler encounters this type error, it will understand that this generated code originally came from this exact location in the user's code. And it would express this error in terms of what the user originally wrote. Whereas if you didn't keep the spans, the compiler would just throw up its hands and say, well, the type error is somewhere within this file. Good luck figuring it out. And it also avoids weird scoping issues. Like, if you've heard about hygiene, this is probably it. I can't claim to understand it. But if you keep the spans, it should all just work. One example of where using an AST and keeping the spans is really nice is in a particular error message that Mord gives. As you may know, in HTML, elements can have attributes attached to them. And one of the rules around attributes is that you can't define the same attribute twice on the same element. As Mord is in HTML template language, we have to enforce this rule as well. And if you have a closer look at this error message, you'll see that it explicitly marks out the parts of the original code that are relevant to this issue. Exactly. And this wouldn't have been possible if we didn't keep an explicit structure that represents the user's input. And it wouldn't be possible if we didn't keep track of the locations of every single syntax node in there. To cap off this talk, I'm going to touch on where Mord is going to next. Generally, right now, Mord is pretty much ready. It's usable. People are using it in production for some strange reason. But it's pretty much okay now. So most of the work at this stage is polish. In particular, I want more diagnostics, more error messages and warnings. In particular, Mord doesn't warn about invalid HTML elements and attributes right now. And I would like to warn in the future. And also, well, no opener. Hands up if you didn't know about well, no opener before I mentioned it. Wow, that's a lot of people. Everyone who raised their hand just then will benefit from this particular thing. And working on stable rust. This isn't a really high priority right now. But as more of the procedural macro API stabilizes, somewhere along the line, it'll tip the scales and it'll be worth pursuing. So that's all I have for today. Thanks for listening and enjoy the closing keynote.