 Good morning, I am so as Vijay introduced I am Raj Shigar. So this is my first time in Jays for Pune and this is my first time in Pune also. So I have been in Mumbai for I think 5 years but for Pune I am coming here for the first time. So my Hindi is very bad also and when I went into Mumbai for the first time for my job. So initially I learned a couple of sentences in Marathi. So the first sentence I learned is, I do not know if I pronounce it correctly, Gathiro that poode I. So let us forget about my Marathi proficiency, let us talk about code mods. In this talk we are going to discuss about a new type of tooling called code mods which will fundamentally bring a change in the way you do large scale migrations or refactoring migrations with your code base. I work for a company called FreshWords as a front-end developer. FreshWords provides simple and innovative SaaS products for the CRM business for all kinds. So if you want to know more about FreshWords come talk to me later on the session. Once again hi, I am Raj Shigar on GitHub and Raj Shigar underscore C on Twitter. So if you have any questions around the code mod topic or anything you can shoot me in Twitter my DMs are open. So let us talk, set the stage first before diving into the topic. So let us talk about paradigm shift. What do you mean by a paradigm? Paradigm is like a lens, an imaginary lens or a conceptual lens through which all the information you perceive gets filtered through. Paradigm shift is an important change that usually happens in the way that is replaced by a new and different way of doing something. So let us see some examples from history and science to understand the concept better. So let us talk about the geocentric theory. So geocentric theory was put forward by Tolmi which says the earth was the center of the universe. So Tolmi's head centered model had lasted without serious challenge for 1500 years. Then in 1543 Copernicus put forward a new theory called heliocentric theory which says the sun was the center of the universe not the earth. So this change in mental model, this change in understanding led ultimately to the classification of new planets, the reclassification of moon as a satellite rather than a planet and a different understanding of mankind's place in the universe. So if you look at large scale code braves migrations from the same perspective we still have a lot of old ideas and mental models to which we are sticking with for a very long time. Let us see one by one. So the first one is regex pattern matching. Whenever you think about making changes to code, naturally you tend towards regex with pattern matching. So you find and replace your text editors or code editors find and replace underneath works with regular expressions I guess. And then you always think about manual grunt work for doing large scale changes. And then you do add up changes every time and this ultimately led to breaking changes. So you are not confident when you make changes with large scale code bases when you are doing upgrades or migrations. So how do we address these things? So let us take a simple example where your regex might not work. So this is a simple code. This is like kind of hello world example for code mods or why regex might not be suitable for large scale changes. I have a variable hello world and a function say hello that is a log hello world. I want to replace the variable name hello with world which is simple change. So after applying a regex here, so now you can see the output the hello world has been replaced with world all over right. So this is not what we wanted we just wanted to change the variable name not the not anything else ok. So the natural tendency in problem solving is that you pick the first solution that comes to mind and run with it. The problem here is you may either run off a cliff or into a worse problem than you start at with. So regular expressions often suffer from a limitation that they do not work with context anymore like they are not able to differentiate between a variable name, a function name or a string literal. So context matters a lot in these areas. So let us see some other examples where your regexes might not be suitable if you are upgrading to new language syntax features like ECMAScripts 6, 7 or 8 something like that. Let us say for example take aero functions. So before aero functions we used to do this you know we assign this to a variable and use it inside the function context. But after the aero functions we do not have to do all those things because aero functions have a context and regex might not solve the cases here because you are deleting code, you are wrapping code, you are replacing something everything works with context and scope. Let us see one more example with view js upgrades if you are upgrading from view 1x 2.0 the attached callback life cycle has been going into mounted and you also have to wrap everything inside it with a mixtick function. So now you know that the regex patterns are not suitable for making last skill changes because you miss you always miss something when you you know work with regexes. So let us talk about the manual grunt work part. So how do you increase throughput in a large scale migration work large task which will take weeks or months like this. What conventional system tells us that there are only two ways to do that get faster people to do the work or get more people to do the work. But in reality getting faster people is more of a subjective thing and it always does not work out and getting more people to a project is like seems like a classic mistake in software development right. So the most sensible thing to do here is to automate the task. So it is time to shift paradigms. So let us see how we can address the old paradigms which we were talking before how we can bring in new paradigms so that we can scale our migration tasks. So this is where code mods comes into the picture code mods is short for code modifications. It actually works with something called AST transformations. They can be automated and they can be reusable. Code mod is a tool or library that assist you with large scale refactoring or code basis it was developed by Facebook and later open source grid. So what is AST transformations. So let us first paradigm shift from rejects pattern match to AST transform. What do you mean by an AST and AST is an abstract syntax tree which is just a tree representation of your source code right. So the ASTs are primarily used by compilers to do syntax analysis, semantic analysis, consistency checking and it is also used to generate optimized code. So how your compilers create AST it undergoes a series of stages like lexical analysis, syntax analysis at the end of syntax analysis stage you get the AST. Let us see what is there happening in lexical analysis and these kind of stages. In lexical analysis there is something called the scanner which scans your code and convert it into a list of tokens. As you can see here you have a function hello world console log. So all these individual things has been converted into tokens even your punctuations and you know delimiters are also converted into tokens. In the next stage the parser takes the tokens and construct a tree data structure. Okay so this is your abstract syntax tree so it starts with program body and function declaration identifier inside the function declaration body you have block statement and the inside all this stuff. So if you are like me or someone who don't get ASTs the first time so I will give you a simple analogy think about it like a DOM of your web page you know DOM right document object model it is nothing but the tree representation of your HTML markup. So AST is similar to that it is tree representation of your source code right. So how AST transforms really work so there is a stage called parsing stage which will take the code and convert it into an abstract syntax tree we saw that here earlier right and then you manipulate the nodes in the abstract syntax tree and then you create a new node or update or delete node you modify the tree and you print back the tree by generating the code from it and you get the modified code. So this is how AST transforms work at a very high level so we will see a tool called AST explorer which works with your AST so through AST explorer you can visualize your ASTs and you can inspect the nodes and stuff. So this is a sample code I showed you so the variable declaration and function declaration at the right you have the AST it is represented in JSON annotation you have objects you have file program and body under body you have two types of nodes variable declaration and function declaration and a variable declaration you have type the node type is variable declaration and it has declarations which is an array of objects and again it is a function node variable declarator. So why it is an array of objects is that it is identified by both identifier and literal because you can have multiple declarations within a single statement and you see an example if I say another declaration you get two variable declarators here. So two variable declarators can be part of a single variable declaration and then function declaration is identified by a node type function declaration and inside it you have the identifier which is the name of your function identified by the name and you have params which is an array of objects you can have multiple params so that is why it is an array. So you have identifiers like which is identified type A main and inside the function body you have a block statement. So which is a expression statement your console log is an expression statement which is itself a call expression with the member expression as the callee your object is identified by console and the property is the log. So anything comes with dot something is a member expression in AST spec ok. So you have the arguments here identified for the callee expression. So this is about AST explorer so you have seen what AST is right what it can be capable of so where they are actually making use of ASTs. So simply put it is used everywhere in your JavaScript ecosystem of tools and libraries it is used by Babel, ESLid, your webpack, parcel, so your bundlers, so your linters everything use makes use of ASTs for there and then the second params shift I was talking about is from manual grunt work to automation how you can automate these changes. So there is a tool for that which is called JS code shift this again created by Facebook by Felix Kling it is made up of two things one is called the runner which is actually a CLI command line interface tool and the second one is a wrapper which is a wrapper around a library called recast. So the CLI tool you have to install it with the npm package npm install JS code shift and you can run it with a parameter passing t transfer sorry stands for transform where you give the transformation logic for your AST nodes. So how do you want to your nodes to get transformed so you put that logic in the file and give it to JS code shift and point out the source and input path like SRC or something and then you run it it will give you status like what are the number of files processed it can also capable of spawning multiple threads simultaneously to work with files and then it also gives results like what are the different files that has been modified or unmodified or status. So that is why this becomes a de facto tool for running your code mods in your code basis and then recast is a library created by Benjamin Newman he is also from Facebook. So recast is actually an AST transformer library so all the AST transformation logic your node transformation create update delete are given by the recast library. So it is also a non-destructive pretty printer which means it keeps your source code formatting intact and then it is also generate source maps. So this is a high level overview of how your code mods will work at the first stage you have the parse stage which is taken care by recast and at the end of the parse stage you get the ASTs created and then you have the JS code shift coming into play for manipulate the nodes find you can find create and date nodes and then finally you print it back to the source code using recast library. So I give you an analogy AST and DOM right previously you know reinforcing the same with AST explorer and JS code shift you inspect the DOM using your browser and manipulate it with jQuery right you manipulate the DOM content by manipulating the jQuery sorry the nodes of your DOM. Similarly you inspect AST with AST explorer and manipulate it with JS code shift. So if you understand this one thing you are an expert in code mods you are expert in AST transforms ok the last paradigm shift we are going to talk about is moving from ad hoc changes to reusable code mods. So every time you do large scale refactoring or large scale migrations with a code base you cannot simply reuse the same thing or same kind of work to another code base or another project right how your code mods address these kind of issues. So there is a tool called code mod CLA it is written by Robert Jackson who is a core maintainer of EmberJS framework. So anybody heard about EmberJS framework ok which it does not count here ok. So code mod CLA is actually a tool to scaffold and generate a project boilerplate for your code mods you just say code mod CLA new project it will give a project with all the batteries included for your code mods you can directly go and type in your AST transform logic and then it supports multiple transforms what I mean by transforms here is just code mods you can manage multiple code mods within a single project and you can auto generate documentation with examples something like how the code is before and after change code mod CLA helps you to generate automatically these type of documentation in readme files and then you can test your code mods it comes with just a test runner I guess you can compare the snapshots with your code mods and then you can also publish your code mods to NPM and you can give it to your other teams or other projects or share with the community also. So let us revisit the new paradigms what we have been talking about so far we moved from rejects pattern matching to AST transformations we moved from manual grunt work and using JSCodeShift for automation and then instead of doing adult changes we use code mod CLA for creating reusable code mods and then finally this eventually leads to from breaking changes to make changes with confidence. So if you want to know what are the different code mods available out there so put to be an awesome list of code mods which is having more than 200 plus code mods for various purposes for various frameworks like react ember, vjs, angular, preact and it is also used by companies like Facebook, Shopify, LinkedIn, Intercom and Square for the internal code base migrations or refactoring kind of work and for a framework like ember.js code mods are the natural migration path which means if you want to upgrade from one version to the next version of ember the core team itself will provide code mods by providing a clean migration path how do I address breaking changes or how do I address deprecations raised in the new versions. So there are more than 20 plus official code mods for ember and you run it with the tool called ember CLA update which will automatically give you a list of code mods you can choose and run. And first of all it is low ember code mods. Recently we did a migration large scale migration by moving our test suits from MochaJS to QInit it is more than 5000 test cases and 100 plus code mods and we did it in two weeks and two months two weeks to get the tooling ready like writing code mods correcting all the use cases and each cases testing the code mods publishing it sharing it with the teams or squads and then it just took two months for the entire migration work you just run the code mods and run the test and verify. So keep in mind that your test is always implied so without tests you are not supposed to run code mods ok. So imagine that if you do this same kind of work manually so how much time it would have taken right. So now see a demo of creating code mods I worked through a simple example which we saw earlier so the replacing the variable name part right. So this is a st explorer so you have a lot of language support also apart from JavaScript so what we need to do is we need to replace the variable declaration hello with world right so inspect the variable declaration node you go into the variable declarator node and change the name to hello. This is what we need to do so let's see how we can do this using JS code shift and a state explorer so you turn on the JS code shift transform by default you will get a default transform which will basically you know reverse the identifier names as you can see at the right all your identifier names has been reversed it just takes and reverse the names. So we will remove this one and write our own AST transform logic so basically it is replacing with a new node actually the identifier it is finding all the identifiers and it is replacing the path value name with world ok. So now you can see we are still making the mistake of not looking through the context we are just finding all the identifiers right we should not do that instead we need to find the variable declarator because identifier comes for both your variable declarator and function declaration also so you need to find the right type of node so that is the first lesson in writing code mods so I am putting a filter here now with the identifier saying that if my path name is equal to hello then you replace or update the name so now you get the right transform right so you can also do it differently so I am just updating the value here path.value.name so there is one more way to do that so instead of finding identifier you can also do it in a different way so you will see that so you can also pass the filter directly in the find function instead of doing a separate filter here so you can just say name hello and again it will the right node so now we are going to find the variable declaration node so not the identifier node so this is just a more advanced version right so the first one was more primitive so now we are finding the right type of node so instead of finding identifier we are finding variable declarator so which will fetch only the variable declarator nodes not the identifier nodes and again see it works the same so with your variable declarators so it is variable declarator I guess it is not variable declarators and there is also one more shortcut way we call something called collections and js code shift where you can directly find the variable declarators instead of using the generic find function right so the define function is generic so here you can directly find variable declarators and you also have helper methods just say rename to hello world you can also write your own collections and like this you can find the function declarators you can also extend js code shift with collections and path nodes everything this is the demo so now you know that writing code mods is about finding the right type of nodes right so the second thing is you manipulate the nodes you create build most of the time the nodes or you update the nodes or you remove the nodes so for these things I put together simple tools which will help you assist you know to find the right type of node so one is a tool called AST finder which will find give a find the API js code shift find API by looking at the source code so if you say let variable it all also gives variable find declarators and there is another tool called AST builder which will help you to write the right builder API with js code shift just you have to just give the code I'll show a demo so now it's time for the last parent shift now we come to the climax so we have seen a lot of tools right to write code mods so about finding the right type of nodes and building nodes so what if there is a tool which automatically does it for you a tool which sees the source and destination code so we have source code is going to change into destination code by looking at these two quotes it will generate code mods automatically right so this is the tool which is called Jarvis okay so let's see a demo for that so this is a default one so you are replacing foo with foo bar here so this is the code mod for that so it is which is automatically created so this is a source code from the destination code so Jarvis looks at both the source and destination code identify what type of AST node we are working with and just put the relevant find and replace APIs for them so since this is a call expression we are finding a call expression with the cally name foo and replacing with the new call expression but the cally here is a number expression so which is foo dot bar so anything with dot bar as I said earlier is a number expression and this is the output okay so it supports different type of node operations with something called opcode so this is the replace node operation you can also do remove insert before so this is the remove operation you just have to remove it and insert before so inserting the new code before the previous the source code and insert after okay so Jarvis works under the assumption that it always tries to find and replace with new nodes it does not like write code mods like what humans we do right we just update the properties as I saw earlier just we are updating the value name right so let's see the same example with that so I'm just going to replace the name with the world so here you can see it just replaced a new declarator node so instead of that there is something called smart update which will eventually do what we humans how we you know approach AST transforms so like this so instead of building new nodes it just updates the node value okay so we have seen what code mods are how powerful they are so what is the take away here so the first take away is that we should be building transformation tools to make large scale refactorings you know which will eventually boost our confidence so that we don't introduce any breaking changes all right so the second take away is that code modifications with tools is greatly improve the way you well and maintain large scale code bases it's very easy with code mods instead of doing everything with manual work or projects transforms and thank you so over to you Vijay because I think I finished before time you're fine okay so we are opening it for the questions even so I have two scenarios the one is the first one is let's say I have a config file JSON file from which I'm reading in certain variables and then I am creating the same variable inside JS file and the second scenario is let's say I have a karma test in which I'm using the same variables that is inside the JS file and I want to replace that variable in this both cases will this work okay so you're saying that you're taking from a config file a variable name into a code file and then into the test file correct yeah but the thing is there is a redundancy so in a config file let's say there is a variable name hello okay and it has some value and in JS file I'm using the same variable name and assigning it to the same name variable okay question is whether the code mods will touch the config file and the test files no it need not to because you are pointing the right directory to them so usually JS code shift works with the JavaScript and T script files I'm sorry type script files it always looks for extensions with the dot JS and dot TS so your config files if it is a different distance extension it is safe I guess but the karma it won't work right spec file if you want to run the code mods only for your source code files you can just point out the source code directories alone you don't have to point the entire root directory you're getting me yeah but if suppose if I wanted to because say otherwise it's a pointless because your karma you want to do it both in the test file you can give the root directory the top level directory for both your source and test files so or you can run it separately it's up to you how you run the code mods no point is not how I run the point is if I want to do it does the code mod understands the syntax of karma file the way basically a stick and render identifiers from the karma file okay your karma file is written in which language is written JavaScript yeah understand yes yes see the code mods and a street transforms works with any parser or compiler or transpiler like your babel or type script which will just export three APIs like you have parse API transform API and generate API so if the compiler or transpiler exports these three APIs and gives a you know solid EST speak then your code mods can understand very well