 Hope you had a good lunch and you have a good conference so far. Today I'm going to talk about abstract syntax tree. So to start off, I'll introduce a little bit about myself. Right now I worked for LinkedIn Learning. Just in case you don't know, LinkedIn Learning is a platform where we offer video courses in fields such as technology, creative, and business taught by industry experts. It started as linda.com and then later it was acquired by LinkedIn. At LinkedIn Learning, I get my first exposure in Ember. Before Ember, I was working at this startup called Spoke, where Spoke was trying to help employee to work more efficiently by using machine learning to manage internal requests. There I was working full stack. I used tools such as Node.js and Angular. So to start off, I want to share two problems that I encountered this past year that really leads me very interested in this topic. So one of the problems I encountered this past year was a problem to book edits syntax on a scale. For example, I was trying to wrap around all of our requests wrapped around in this global error handler. Two, when I first started working on this project, my first thought is I'll just do it manually. But then when I started looking at it, there were thousands or maybe like hundreds of requests across our code base. It's going to be a lot of code that I need to like edit. And it's going to take days and it's tedious. So I thought about a second solution, which is to use regex. And I tried it and I wrote the most complicated regex in my life. And it solved about half the problem. For other half, regex can just simply cannot do it. And regex especially can't handle the situation where the lines, the kind of a request call that outlasted a couple of lines. Regex does really well when single line, but when it's like a couple of lines, it doesn't do as efficient of job. So that's when I started looking into code mod and it made my life so much better. And I was able to transform a lot of the code in our code base using code mod in just a couple of days of work. So that's the first problem that I encountered. The second problem I encountered started from observations. I realized we have a lot of code conventions in our team. And you would see that code reviewers or one review code, we're always trying to enforce those conventions. Sometimes I feel bad of pointing them out. And as people who submit code, I also feel bad of making those mistakes. But sometimes they're just hard to catch, especially when I'm trying in a rush, it's hard. So I was wondering if there is a way to automate this process. And that's why I looked into creating a custom linking rule. And then it also made my, I think my team's life so much better where we can identify problems before it's been pointing out to other people and shorten our kind of problem solving cycle. It was great. So from these two problems and those two tools I used, I realized both of them somehow have this very similar underlying way to solve a problem. That is involved code manipulation and transformation. And that's what I wanna talk to you about today, is to how to do it, how to just use that tool and make your life easier as a developer. So to start with talking about kind of that underlying concept behind them, I'm going to start with a simple analogy. If you're ever trying to learn different language, you would always start with probably one or two things. First thing you'll learn about vocabulary. Second thing you will learn about grammar. You will learn things like, Apple is a noun. I is a subject. You have a sentence, you need a subject, verb and noun. If I say a sentence like I, Apple, some of you just don't understand, like what does that mean? Understanding grammar behind a language enable people to communicate more efficiently and enable people to understand each other better. And how does this relate to our daily life as a developer? Similar to speaking a language, writing code is very similar where if you understand, if we can understand how the code are working behind the scene and be really familiar with the grammar behind it, we are able to do so much more things with the code that we write every day and thus become a very efficient developer. So that's what I'm going to talk about today. To help you understand better of the underlying syntax and the underlying structure behind the code that you write every day. So to start off, I'll go a little bit more in-depth about the tool, the product utility tools that we might encounter in Amber and from there, I'll point out to you this underlying concept abstract syntax tree behind them. And then I'll talk about how you can apply abstract syntax tree and use it in your future work. So to start off, I'll go a little bit more in-depth into some of the tools that I may have mentioned before. The first is Linter. What Linter does is it catches error and point out errors as you're developing. It can often seen as a code editor plugin or like as a pre-commit hook. I'll start with an example of disallowing unreachable code as part of the Yes, Linter rule. So for this rule, if you have a line that's written after the return statement, like assigning a value, the code editor will barf at you for making this mistake. This seems like a simple like problem to solve, like you just check what comes after return and that's it. But it's actually a lot more complex because if both the same line, return value equals to two, how would the Linter determine that it's after return but not before return? How is that position determined and how does Linter know the particular syntax like the context around that particular line? That's some questions that will be answered later. But for the next thing, I'm going to look at another tool. Code mod that I mentioned a little bit before and I wanted to go a little bit more in-depth into it. So code mod enable us to bulk edit syntax within your code base. A lot of us may have went through the transition of upgrading Ember 3.4 and trying to get to the Ember Octane. To do that, you need to convert a lot of the curly braces syntax in your template into angle brackets and that's what code mod can do. It automatically not only convert the curly brackets into angle brackets and it adds add symbols in front of the arguments and it skips attributes such as class. Code mod here is really impressive because it not only edit things in bulk and it's able to distinguish some of like a very similar syntax. For example, in the original code, the curly braces version in 3.4, you would have syntax such as curly braces country.name and code mod is able to distinguish this curly braces with the curly braces that's wrapped around super dash select curly braces and they decided to convert the curly braces right next to super dash select but then skip the curly braces around country.name. That's a hard problem to solve. In order to solve it, code mod also needs to learn to know the context surrounding those two lines and similar to linter, those are very similar problems that it needs to solve and I'm going to talk in a little bit later of how this problem is solved. For the next thing, I'm going to talk about the third tool, transpiler. The reason why I want to look at it is even though this tool, you may not be actively using it but it's probably somewhere in your dependency. You probably have this as a package in your project. What transpiler do is for, let's say for example, what it does is it convert ES6 code into a backward compatible version that can run any browser such as Chrome or IE. It saves us developers so much more time from writing code that's particular towards your browser. So I want to look at a specific example that Babel does. In this particular example, Babel is kind of converting the fat arrow syntax into a regular function syntax. And in fact, not only from the fat arrow to kind of backward compatible version, transpiler are also able to do the other way around. It can convert older version of syntax to the new version of syntax. This is definitely sure a very beneficial tool to be included in our repository. But similar to code mod and linter, transpiler also needs to solve a lot of problems in order to get to where it is. One important problem is also very similar is how does it know where is the ending of a syntax that is trying to convert? How does it know that when it's converting a fat arrow to a regular function that N plus one at the end of N plus one, that's where it's supposed to wrap around the function around instead of this whole entire item. All right, so I've been throwing a lot of questions at you and maybe you've been curious, what's the answer Sophia? All of those problems have, all of those tools have very similar problems to solve, which is to identify a context surrounding a particular line of syntax. How does it do it? The answer is abstract syntax tree. Abstract syntax tree is basically a tree structure that highlights most important syntax of a language. To understand abstract syntax tree well, we had to start with the most basic concept of tree and then see how the concepts evolve from a tree to abstract syntax tree. So what is a tree data structure? Tree is basically a bunch of nodes that are connected by edges. If you are not familiar with this concept, I linked an article at the bottom of the slide and feel free to check it out after the presentation. So basically tree is very useful in computer science and it's always adapt to, it's widely used and it's made adapt to very different forms for particular problems we're trying to solve. So for example, we have heap trees. We use heap trees to store memories. In order to solve the problem of encoding decoding, people come up with binary trees because that's the kind of data structure that solves that problem particularly well. What if we want to solve the problem of representing a language? That's what it comes parse tree. So you can think of parse tree as basically making a grammatical structure of a sentence onto a tree. For example, when we have formula three times four plus two, if we map it onto a tree, it would look something like this. So it's a literal translation of this grammatical structure. You can see in this tree, there's a lot of math terminologies about what each part of this line does. For the example purposes, I'm not going to go in deep of what each this math expression means. But here I do want to point out the specific term here looks really scary, but it will change depending on the context. If it's a natural language, say, I ate an apple, then the different note here will probably be replaced by none or verb and then say if it's a handlebar template, each of the node will be replaced by things like mustache statement and different kind of statement or hash values. So basically the different kind of a node is determined by the grammatical structure of the language. So when we look at the parse tree, it actually looks kind of complicated and it's not as efficient as it can be. If when you're looking at this tree, there are a lot of nodes on the tree that actually contains repetitive information. For example, if we look at three, there are three parents here and as a tree is not as efficient as it can be. And that leads us to abstract syntax tree. Abstract syntax tree basically have the best part of parse tree, which is the kind of the structure, or medical structure for a syntax, but it highlights the most important information. For example, it does so by kind of condense the chain nodes together and then it take away extraneous information in this case, brackets. And then that's where we get to this new tree, which is the abstract syntax tree. The way I think about it is the abstract syntax tree is the tree I would think about, that would be how I actually think about in my head. And then I would think about the tree on the left as a way to kind of reproduce the original format of this tree. As you can see, if I combine the information from the middle tree here with the tree on the left, I can get to the original expression of three times four plus two. So as you can see that this tree hierarchy really provide a clear way, like a very hierarchical way to get the surrounding syntax. Because if I look at four here, I can clearly see that the plus here is taking precedent over the expression four plus two. And then the multiplication is taking a precedent over the plus. So it's kind of at a higher level. So kind of abstract syntax tree, this is how if we have a code that people are able to know the context around it is by this kind of hierarchical structure that provides a lot more information about the context. So yeah, we just talked about the concept of syntax tree. And then how it provides this context through this hierarchical structure. How does it relate to the code that we write every day? This is what I'm going to talk about next is a step to step from the code that we write every day to abstract syntax tree and translate it back to the code. So typically to go through this step, there are three steps. These three steps are pretty common for tools such as compiler, translator, oh, not translator interpreter. So the first stage of the code manipulation is this parsing process, which I'm going to get into it soon. So you can think of a parsing as a process converting a sentence to abstract syntax tree. In order to make this happen, we first need to know what are the lines we're reading. So for example, to convert from the three times four plus plus two, you have to finish reading it as a machine. So that's the scanning step. After you finish reading it, you have to find a way to break this statement down into a certain way. So that's what the tokenization does. Tokenization basically break down this particular expression into smaller chunks such as three, four and two. And this whole process of scanning and tokenization are called lexical analysis. So let's look at a real world code example. Let's say I have this block of code. After I scan it, this is the kind of token I would get. It's like div, class, ember, com, h1, title, and next line. So this is some of the example tokens that I would get from this block of code. After I get this block of code, the next thing to build up the tree structure is we want to put them together into a tree. And then this process is called syntactic analysis. So in order to help this process of building a tree, we use a tool called AST explorer, or I use this tool pretty often during my development. This is a really simple tool, actually. It's a website developed by Felix Klim and I have linked the link to this tool and his name at the bottom of the slide. So to use this tool, you just simply go to this website. The interface will look something like this. At the top left corner, you would have the code that you want to transform. And at the top right, it will output you this abstract syntax tree with the token that's already parsed. So here I want to use a glimmer parser to do the job of actually building the abstract syntax tree. This will be the output of this abstract syntax tree. As you can see, each one is this element node. And within this element node, there is mustache statement that is this dot title block. And then it has the value, aka original this dot title. So this will be the actual abstract syntax tree coming kind of the output of the parser. So the parsing steps are really done by existing packages. If you were to write a compiler or like a new tool, you usually don't need to kind of write your own parser. Usually there's already parser coming from each language. For example, for JavaScript, there are already parsers exist like Babylon. And for handlebars, there are parsers such as glimmer parser. There's also parsers such as handlebar parser. And all the different parsers works a little bit different. And therefore the parsing results is also a little bit different. The second stage of the code manipulation is the transformation step. If you ever want to write your own kind of code mod, or if you ever want to write your own linter, this will be the bulk of the work since the parsing step is already done for you. What the transformation step does is help you actually manipulating the code. For example, looking at this particular block, say I want to add a helper function called beautify to this in front of this.title. This is the kind of code I would write to actually transform it. As you can see, the kind of to the transformation usually have this visitor pattern for this particular code block. If you're not familiar with the visitor hook pattern, I've linked the link at the bottom of this slide if you want to look more into it. So basically within each hook, I write an expression that matches the kind of statement I want to modify in. In this particular case, the kind of code block I want to modify is this.title. It's with mustache statement. So that's the reason why I have this modification inside the mustache statement method. And here I'm just time checking if it's a path expression, then I would add beautify in front of the value. Just FYI, this code may not actually work, but this is an example of the kind of manipulation I would actually do to manipulate the code at a scale. An interesting thing about this is if I were to write a linking, link to rule, I would do exactly the same thing, but except instead of actually manipulating a value, which is what code mod does for linking, I would do type checking. For example, if I want to create a link to rule that's trying to check, it's each mustache statement, aka those values that's kind of wrapped around a bracket if they have this. I would do it here and check node.path.original have contains this value. So the last step of the code manipulation process is code generation. Not all the tools that we created or you're going to create need this step. This step is helpful for tools such as compiler, transpiler, code mod, but maybe not as much for linking. But what this code generation step does is it reprints the trees that we have just manipulated in the case of the beautified example, because in the last example, what we did is we changed the value within a particular statement. And what the code generation does is it detect the change and then it reprint out the things that we have changed. And then this would be the ideal output from the code snippet before. So just now I went over a concept such as abstract syntax tree, what it is. And the process of how it's applied in a lot of different tools. I hope you have a base understanding of these concepts and the tools around it. There are a lot more cool applications can be developed using abstract syntax tree that hasn't been developed yet. And I hope with more understanding of these concepts, you are able to empower to develop your own tools. And I'm really looking forward to see what you can do with this information. At the end, this is a list of resources that I found being useful for me when putting together this presentation. And lastly, I really wanna thank my team for their feedback and support and for the conference organizer for really sticking through everything and make this happen. Great job. This is my contact information just in case you can still contact me. Thank you everyone. Hope you enjoy the rest of the conference.