 So this morning I did a presentation about making your own languages with Racket and about how it could be useful for creating data structures and data languages but are useful in a specific domain and this afternoon I wanted to take a bit more time to make you try by yourselves making a language. So I wanted this session to be a bit more interactive and so if you have working computers you can try live and see if you can do something. The code is on this repository on GitHub at umoe-slash-first-dem 2019 talk and there's a workshop directory. So if you pull that repository and of course you also need to install Racket. It should be in all the good package repositories around there. You can get to to this file once you pull it and and we'll see how we can make our own language. The first thing you usually do when you create a language I show I have already shown those lines this morning so this is the most simple way to create a language in Racket. The way it works is Racket has modules and every time you import a module it executes some code on this module before importing it into your runtime environment and running it and during this time you have the ability to read the code you are importing and modify it on the fly before it arrives into your into your code your running code. So the way it works is that any Racket module you create you just create a file and by itself it it can be used as a reader for over modules. So any Racket file have the ability to read over Racket files and this is the kind of module you want to make so there's a the magic is happening here. This function it's actually a macro but so this macro is the macro that is being called when you import a module and in this file we're going to change it so that it does what we want. So we are modifying the stand our way Racket works by putting our own code into this function. So the way it works is we just rename the function while we provide it. So providing a module is the inverse of importing it. Importing a module is done with require you just say okay for for this module I need I need this I need this I need this and providing is the opposite so it declares all the all the the functions and the stuff you're creating your file and you want over over files to use. So if you provide something that always already exists it's going to be replaced by the one you provided. So here we create our own module begin and we provide it as this name so it's going to replace it into the environment and the module begin we write is really simple it's like only those lines. It's just this is the way to declare a macro in Racket and in scheme. So we declare a macro that is called module begin and takes an expression as an argument and we call the actual the real module begin with our own code here. So this is the part where we we change the way module begin works is we just call the old module begin with our code and the code is doing only one thing. It just provides some variable that I called the ranch. Yep so there you go. You got it. So it just takes the expression that is inside the module and puts it into a variable and provide the variable. What's happening here exactly is I'm going to show you interactively in a moment is that the data file I want to process is this one. So this is the this is an example of a simple language I just made. I designed this language so that you can create a ranch and have ponies in it and describe the battle cries of every pony and so this code is going to generate some function that I can call. So I'm calling them just right there. You just call the ranch and you give them the name of the pony and it should print their battle cries. So if I do this just there I call racket on my main file and as you see they're just trying out loud what they want. So the way it works is that I want to take this data and transform it into actual code. The easy way to do that is through a macro. So I just match for I just create for this example I create only one macro the ranch macro and so the ranch macro is going to run and analyze stuff in there and generate some code and it happens just there. The macro is there so I define a macro called ranch that takes a syntax and so this part is a bit hairy but you get used to it when you work with a syntax boss. The macro can have two keywords ponies and pony. Every time it stumbled upon one of those keywords it just doesn't care. It's just therefore as an information that okay we are in this structure this is the pony section that's it and the way it pattern matches the data we saw is there. It looks out for a section called ponies in which there are any number this is any number the three dots there any number of the section starting with pony then a name as an ID and a battle cry as a string. So this line here describes the structure of our data. If we can look at our data again we have a structure starting with ponies and then any number of structures starting with pony. So here we start with ponies and any number of this stuff and we want to replace that. Oh yeah sorry and you want to replace that with code. So the replacement part is happening just there. I just create a function that takes a pony name and switches over it and if it equals the pony we just met then it returns the cry. Over rise this pony does not exist and we'll be able to see in real time what this macro is doing step by step using a very handy tool which is called the macro stepper and if I step it's gonna open up yeah. So this tool allows us to see in real time the macro over the code. If I step it's gonna define a module it's gonna define some stuff for the racket system and then at one time it's gonna reach our code and there okay. So this is before the macro hits and this is after and we can see that it transforms our structure into a lambda with a cone and an if statement for every pony in the data structure. So in C or C++ this would be really efficient because you're transforming the data structure into actual code with direct inlining of the data. Here the condition is directly in line for every piece of data. So you actually generate code and a big if and it happens that the cone keyword is actually macro so if I step a little bit further we're gonna see the cone macro transforming our code into if statements. So here the cone defined into into racket is transforming to a bunch of if. So once this is compiled into actual binary it's extremely efficient it's not it's not data it became code it became compiled code. And so when I discovered this I thought wow this is like the the best advancement we have in data oriented programming programming systems. It's like we can write data and have actual code at the end without anything in between. It's like your data is your code. And so I started to to look at how I could create languages for for my needs. And so that's why I'm doing this talk right now it's because I thought it was really exciting to just write just express what you want from the program from a data point of view and get actual code in the end. So this is the the the trickiest part to understand it's the macro here but there's a lot of resources on the internet from over racket programmers. There's one that is Greg Henderson he has a really great blog about writing macros in racket. So you should really check that out. And at first I thought I would stop there and make everyone try this kind of stuff but four minutes before the the talk started I got something else working here. It seems you might be missing package. Maybe. Yeah so you have an issue with. Yeah. Okay. So he's wondering if there are any dependencies missing. And I thought I would I was using known but maybe I'll check that out later. Yeah. Oh yeah that'll work. Yeah. And so right now the language we created looks like S expressions. So it's the standard way of writing least languages. It's just parentheses everywhere and atoms between between them. And racket can go further and provide you here I'm using the default SX reader but you can actually use any reader you like you can create your own readers for syntaxes that are not S expressions. And so I created another example that the other example requires dependencies external dependencies. I changed the language into that. This is completely arbitrary. I could have used any any way of writing this. I just thought it was cool that way. So if I want my language to look like this and then make it so that it generates this then generates the code we saw. There's a step further. We can write a reader and a parser. The thing we wrote right now was the expander. So the expander is using macros to expand the code into bigger stuff. The bigger functions as we saw in the macro stepper. But there's over layers and one of the layer is the parser. And there's actually a language called Bragg which I'm using here. And this language allows you to define a syntax for a language and use that syntax as a parser for your language. So here I wrote a syntax. And so I wrote that a wrench can be an optional new line then a marker for wrench begin then a new line then a number of ponies then this is yeah this is a typo. Then wrench and then new line. And then ponies is any number of ponies and ponies is any number of spaces then extra extra. And so this syntax is actually a rocket language and this gets processed and transformed into a parser. And it generates exactly what we saw there. It generates this kind of stuff. And the way it works is we have to define what wrench begin is as a token what pony begin is. And so it is done in the reader. And so the reader overwrites an over part of rocket. Instead of overwriting module begin we overwrite two over functions read and read syntax. And those two functions are called when you actually read something from a file when you import it. And so you can define written text to be a parser. And here what I'm doing is so this is the parser I defined. When this gets compiled it is compiled into a function called parser called parse. And in the reader I require it so I get parse. And as you can see we can visualize the dependency. So parse is coming from this parser. And so I can call this function to generate tokens from my file here. And I define the tokens here using a lexer. This is a new line. So I define that every time it finds this character it generates a token that is a new line. Every time it finds this suit of characters then this is a wrench begin. Every time it finds this this is a wrench end. Every time it finds two characters that are alphabetic two or more characters that are alphabetic this is an ID. And anything from double code to double code is a string. And so by defining your tokens there you can then use them here to define what the syntax of the language is. And it will take this and process it into this. So this is a really powerful system that allows you to define anything you want. Any language you want. Do you have any any questions about what's going on or is it okay? Yeah, so here what I'm doing is oh yeah. So how am I using the parser here? How am I requiring it? So one of the issue with the Bang-Lang syntax is that it allows, I cannot put directly something like that. Let's say my reader.orca team. It doesn't work like that. Only actual modules are authorized here. So I need to create an actual module. And I make it here. Let's see. Okay. So this file is just declaring that the reader is workshop slash reader. And so workshop slash reader is our reader here. So the way it works is you just have to define a special module that is called a reader. This is something a bit specific that I had a lot of difficulties working with this at first. That's why I'm showing this as an example right here. And I hope a lot of people trying to do this will not have to struggle with this as I struggled. But yeah, right now I declare that the module Pony when I require it, when I use it, the reader is going to be my reader. That's it. So this is all I can show you about this Pony language. So I made a lot of over language if you want some examples. More complex ones. And I encourage you to take this example and push it further and make your own language out of it. You're free. Yes, there's a question. Yes, I tried it. So yeah, I'm going to repeat the question. Is it possible to create some indentation based languages like Python, for example? So I tried it. And it's actually possible, but it's a bit complicated because Bragg, as I showed here, will not be enough because indentation based languages are not regular grammars. They cannot be computed using the kind of grammar I showed here. You need a recursive grammar. And you need to keep the state of what is the number of indentation. And this is so this is getting a bit more complicated. It's possible, but it's more complicated than the example I got there. So one of the languages I made with this was it's one of the first languages I made with this was a language for generating emulators from CPU description. So you just describe what the CPU is, what are the operations, and it generates an emulator. I called it virtual MPU. And let me see. I'm going to show one of the one of the files. Let's see. So this is the kind of file you can create with this language. You declare an MPU, you declare a name, then you declare the registers. Okay, so VCPU has an A register, B register, a status register. The stack pointer register is 16 bits, stuff like that. You can precise, you can declare that the status register is SR and that all the bits mean this. So there's one bit that means it's the carry, one that is the overflow, et cetera, et cetera. You declare the interrupts, and then you declare all the operations. And I made my own somewhat macro system inside this operation stuff. The operations are starting here with the branches. So those are the different styles, the different kind of branches in the emulator. Then the moves, load A, load B, load the stack pointer, their stack manipulations, math, add A plus B, stuff like that. And some of them required, some of them add exactly the same way of working for a lot of stuff. So I made macros, for example, there, subtraction, addition. And you can declare them on the top. So here, subtraction and additions are described here. You just say that you want a result. You want the result of adding moves together and then you change the carry, you change the have carry, you change the overflow, you do a bunch of stuff inside the CPU, and you get your result. So this is an example of the kind of language you can make. In this one, I prefer to use S expressions and not a special language. But I also did, in the same project, I also did another language, which is actually an assembler. And so I created a language that passes assembly language. So this time, I needed a special reader and stuff like that. But this code right here is parsed and transformed into an S expression and transformed into racket functions and then executed. And it's just assembly language at first. So for this one, the reader looks like this. There's like boost tokens. There's like a token for numbers. So numbers can have a dollar before them. They can be in hexadecimal format. There's comments. So there's comments with a semicolon and stuff like that. There's a special section for data. You can put data in there and stuff like that. And the parser looks like this. So an assembly is just lines, any number of lines and new lines between. And then a line can be an instruction or an assignment or data declaration. An instruction has a tag, an optional tag, then any number of space, a mnemonic, then any number of space and operands and stuff like that. So it's like 20 lines and it parses assembly language. And it took me, I've been working on this project for maybe one month and I knew nothing about how BRAG works before. And I just learned how it worked. And in 20 lines of code, you can generate a parser for assembly language. What can I say? Yeah. What is actually expanding into like the assembler? Does it expand into packet code that works? So right now it's assembled, it's expanded into binary. So the way it works is it parses the assembly and then generates opcodes for all the assembly stuff and then put that into a file and so it compiles a binary. And I put it there, multiple options. You can generate the Motorola SREC format or the Intel format or stuff like that or direct binary and stuff like that. So yeah, it's a working assembler for the CPU I wanted to emulate. Try to stand. So Racket, so I wasn't in the personal talk. Racket is like a variant of the list which defines your own language. So yeah, Racket is a list language from the scheme family which allows you to define your own language. So you are now defining your own parsers for this big language. In this case, assembly, you parsed it and you're compiling it to what? So he's wondering how, I'm generating Racket code, but how does it compile to something else after that? So in this case, I think I can show the expander. So I'm going to show up the GPU. The interesting file here would be, there, okay, the assembler. So it is compiled into structures, Racket structures. So the, yeah. So at first it compiles into Racket language and then when the Racket language executes, it generates a binary. It generates a binary from the assembler. Like the assembly language is assembled into a binary so that it works on the target machine I wanted it to work. So what I'm doing is I'm generating code that is executed to do what I want. Instead of just executing some code, I'm generating code that then it executes. So I parse the assembly into a Racket structure. Then the Racket structure is parsed into Racket code. Then the Racket code is executed and the main function of that execution is write to a file. The first function that is called is write this to a file and then everything else is the data that I scrambled and mashed together. I could have written, I mean, it's a way to write an assembler. I could have written an assembler in any language. But the way it works in Racket is interesting because you define your program as a language instead of just a program. So instead of saying, okay, I wrote this assembler, you just run it on some code and then you get a binary. I wrote language that is actually an assembling language and so when you use it, you assemble into binary. So it's a bit like changing the way you think about programs. Instead of using programs, you create languages and you use those languages. It's the way Racket promotes. Any other questions? So what's the benefit doing this? One of the things I like with this is that you define, instead of thinking in terms of I need this program that handles this data this way, instead of thinking into pipes that you put together to achieve a result, you think into how can I best describe my problem? How can I best describe my issue? What is the thing that I want to do and how can I describe it so that it's obvious when you read the code what you want to do? And so Racket allows you to open a file and say, okay, so my problem is about bakeries. I want to generate a cell bread and croissant and stuff like that. So I have a bakery and it says a lot of stuff. So it has products and there's croissant and there's bread and baguette and stuff. And you start by thinking about your data and then you make it so that this data is actually executed. It's actually a program and actually makes what you want. So the benefit I find is that instead of thinking about making programs that takes data and outputs something, you just write your data the best way you want your data to look and then it executes into something. And that's what I found a bit magical about this way of working is that it redefines the way you program. It changed a bit my way of seeing programming and languages in general. I don't have to think about, okay, in Python there are those kind of data structures that I need to use or I remember with my colleagues we had really long talks about we should using this kind of classes and then this class inherits this class and then it generates this interface and we need to change the interface so that this customer can work correctly with the interface and this class interacts with this class. And it forces the way you think about programming is you have tools around, you have, okay, I'm working with C-shop for example, you have classes, you have reflection, you have a lot of tools around, but I think only those tools in your hands, you only think about how to use those tools. The only way you think about resolving your problem is using those tools. So if instead of thinking in terms of tools, we think in terms of what is the actual program and the data that I want to process and what do I really want to do, you write the language that gives you the best tools for this task and this could be a different language depending on the domain. So you still have to write the tools to translate your data into actions and what I like is that the way you write the tools is through a language. So they are not just tools, they are a language that you can make evolve and that you can document, that you can write specification for that language and that you can share with colleagues and that you can promote and that is adapted to your company and the product you're working on and the clients and it doesn't change completely the way we work. We still write tools and those tools are executed so that we get what we want. But instead of having some executables that no one knows how it works in a shady place in your repository that no one touched for 10 years and there's no documentation for that and can you change it so that it handles the new stuff that just came around and you look at this legacy code and you're like, OK, it's not going to make it. I'm going to need to rewrite this from scratch. And instead if you have a language, yes, you will need to change it, you will need to adapt it, you will need to modify the way it works to adapt to the situation but it's a complete language. It's not just tools around. And so it has consistency, it has a logic behind and that's what I like. Anyone want to try Racket now? Raise your hand. Great. How do you use Racket in your work? I mostly use Racket in my free time because in real life I'm actually doing JavaScript. But actually I'm doing something that is in relation between JavaScript and Racket recently. I've been working on a project that has been started by a student at the University of MIT and he's been working with Racket a lot and he has written a lot of thesis about a lot of stuff in Racket. And for his PhD, he invented a language that generates JavaScript from Racket. And this language is really interesting. It's called Erlang there. And so this language uses Racket and generates actual JavaScript. So this is how it looks. So you write something like that and it generates this kind of stuff. And this is really interesting because this is similar to the work of the team at Babel. The way Babel works is that it's a generator for JavaScript and it pauses the syntax of JavaScript and it understands what you need when you want from your JavaScript code and it transforms it into other JavaScript code that runs on any browser. And Babel is an awesome open source project. And this is the kind of stuff that allows to do the same thing as Babel but using the power of macros in Racket. So everything here can be used as data, can be parsed, can be modified, can be analyzed so that you can generate different JavaScript depending on the situation so that you can generate JavaScript depending on the browser or stuff like that. And it happens that the Babel project has a system called macros that they wrote from scratch by hand and it works mostly the same as Racket. The only difference is that Babel is like a really big open source project with a lot of contributors so it's way, way more developed and there's way more features. But one interesting thing is that using Racket here, you don't have only access to the macros defined as Babel defined then. You have access to all the Racket environment and all the other languages. So one of the things I was showing in my talk this morning was an example from a language I'm making. So this was my talk this morning and there, okay, I'm going to put it a bit smaller. And this is an example from, this is something that I try to achieve. It's not ready yet but I'm trying to make something like this. I'm trying to make a system in which you can write API servers, web servers and this is, for example, a response from an API and inside you have code that fetches in the database. You have code that generates HTML, you have code that generates CSS, you have code that generates JavaScript, everything in the same syntax and in the same function. So you just focus on writing your complete website as data, as function, as just one function and then it generates everything, it generates JavaScript, it generates the CSS. For example, the style system here could be a system like less so that you can have variables inside your style and generate your style dynamically. And the same thing for the JavaScript part, it could be something like Babel. So I'm trying to use Erlang inside this part there. So I generate JavaScript but inside the JavaScript you can put some data, you can modify and generate some JavaScript code and stuff like that. So this is the kind of way of programming I'm trying to, I'm exploring and I'm trying to see where it goes but I think it's kind of cool to be able to forget that there are different languages and just write in one language with a lot of syntaxes to generate bits and bits from the server. Yeah. Is it more complicated if the data is not a tree? It's like a graph? It can be, sometimes it can be, I know some people in the racket community that are working on that kind of structures with cycles inside and stuff like that. The racket parser and the racket expander and the macro system is extremely powerful so it handles the cycles and stuff like that. I don't know in details, there are people with PhDs that would be way more able to answer this question but yes, it can handle data structures with cycles and stuff like that but I guess it would be a bit more complicated and you will face some issues. If nobody has any other questions, I think I'm done.