 Hi there. This is a talk about creating your own programming language, specifically a language that doesn't suck. And for many of us, the language that does not suck is the first language that we learned. And for me, this was the language that ran on this computer here, the Apple 2e, which was a computer that my parents gave to me in 1983. And I just took that picture today. It still works. And it runs a language called AppleSoftBasic. You can still find documentation on AppleSoftBasic to this day. So here's, for example, an introductory manual. And if you look inside that manual, you can see what a basic program looks like. And you can see it consists of a bunch of lines. Every line has a line number. And on every line, there is a command. So it's none of that methods and classes and functions nonsense that we have today. It's just straight up do this, do that, jump to that line. And it was a lot of fun during the day. So we're going to use the Racket system to implement that. And Racket is great. It's an entire toolbox that you can use to create your own languages. And it comes with many languages. And it's just a lot of fun. And if you want to play with the code for this talk yourself, I put everything on GitHub in a repository called mikesperber.rc3. And that also links to a download link for Racket and has all the code for today. So anyway, I have to warn you, this talk is pretty heavy on code. So that's pretty much the only thing that I'm going to be doing is I'm going to write code in front of your eyes. And if you know, that's just too much for you today, I'll understand. And I won't be mad if you go see another talk. So but if you're in for some really serious hacking, then this might just be the talk for you. So Racket, when it starts up, comes up with a blank screen, such as this one. Well, it's not quite blank. Each file in Racket starts with a hash lang line that tells Racket what language that file is written in. And the system actually comes with many, many languages, one of which also happens to be called Racket. Now Racket's okay language, but it also has some annoyances. So we could print something out. By calling the display function. And if we run that program, well down here, you can see the output, that's fine. But sometimes we also want to display several things, for example, a little bit of explanatory text, along with a computation. So this here, well, you can see that the default Racket language is a Lisp dialect, which always has parentheses around compound forms, and has an operator always in front. So you don't write, you know, one plus one, you write the sum of one and one. Now we really, well, hopefully we know what that should print out. But if we run that, we get an error message. And that's because display only takes a single argument to print out. And the second argument is supposed to be a port, which tells display where the output goes, if it goes to a file or the repl or whatever. So really, if we want, you know, several pieces of output, such as this, then we need to put in several calls to display. So there you go. Well, that's super annoying, right? And even, you know, Apple soft basic at the thing where you could just say print, and you could just put in several things. So you could do this thing that we were trying to do with display in the first place, right? But of course, you know, run this print is just not built in in the shape that we would like. So we're just going to define it ourselves. And to do that, we're going to write a macro, which means that the this form with print at the front is going to get translated into code that racket already knows about. And so we want to translate this into these two calls to display up here, right? And well, one restriction that we need to remember is that, you know, a form always translates into a single form. So when there's two forms here, in order to turn these into a single form, we just put begin at the front and parentheses around. So this is the translation that we want. I'll comment that out, just so that we have that at our disposal. And in order to write a compile time function or a macro, we say define syntax, we're going to define a new piece of syntax. And that piece of syntax always starts with open parent print. And the compile time function that implements the macro always gets called with an argument, which is exactly the piece of syntax that the compiler saw. And we're going to call that form. Now, in order to translate it into this begin form, we need to take it apart a little bit, identify the arguments, we need to parse it. And for parsing, we will use a racket library called syntax parse. And I'm going to import that as a library. So it's not a default feature in the core of racket language. And since we're going to use syntax parse inside a function that runs at compile time or at syntax expansion time, as we like to say, we're going to say up here for syntax. So we can use it inside a define syntax form. So we write, you know, syntax parse here, we want to take apart this form thing. And we will write a pattern that describes the shape of the form. So we'll start with a very simple version of that pattern. So it starts with an open parent print. Then there's an argument, and that argument has to be an expression, which is denoted by putting colon expression there. And now we want to translate that into a different piece of syntax. In order to create syntax, what we do is I put in hash, back quote, we want to translate that into display arc for now. So we'll just do a single argument version for starters. So I'll put that here, I'll run the code. So there's nothing in the REPL now, but now I can say at least hello world, very exciting. And that works. But of course, it will only work with a single argument. We want to use it with several arguments. In order to do that, we can just say, well, we don't just want one arc, we want several. And to do that, we put three dots here. Almost looks too simple to be true. So it just says, well, there can be as many arguments as we want. And now, as we said above, we need to translate into a begin form. So we want to say like this, right? And then we want to have as many calls to display as there are arcs. And we can also do that by putting, you know, three dots here. And Racket will automatically figure out that that means we should have one call to display for each argument. And make sure that all the parentheses are closed. And let's try that out. One plus one equals plus like this. And sure enough, that works. So now we've added a new syntactic form to the Racket language. Pretty simple as it goes. And we'll use that as a small building block for our basic machinery that we're going to implement next. So, well, I guess it's a good time to save our work. Just to make sure that we're not losing anything. I'm just going to call this file basic dot racket. And there we go. So, okay, well, we got that print thing, which is similar to the print in basic, which is nice. But I don't know if you've ever used basic, it looks a basic program really looks like this, right? It says, you know, print hello world. And then you can see that there's a line number in the beginning, and then you go something like, yeah, go to 10. So it's always the first program that just prints something repeatedly. So, yeah, well, this is obviously not there yet, right print. Well, there's no line number. There's also still the funny parenthesis syntax. So if you create when you create a new language in racket, the way to do that is, well, you start with a parenthesis syntax. And later on top of that, we will implement the actual, you know, line based basic syntax that we might remember from our Apple two days. So, so to that end, we'll just start by saying, not, you know, 10 print, what we'll say open parent 10 print, you know, hello world. Eventually, we're also going to have to take care of the lowercase print here, and then say 20, you know, go to 10, something like this. So that's what we're going to start with. Again, we have to think about how we will translate this into code that racket recognizes. And well, you know, this program, it doesn't do anything yet. It just stores, you know, the print command, for example, it just stores that into into storage of the of the Apple soft interpreters. And I can then start the program by saying, well, I want to start the program at line 10, or something like that. So it's, it's not enough to just sort of translate this into a call to print, we will also have to deal with the line numbers somehow, so that also go to 10, for example, makes sense. So in order to store a piece of code, we can just give it a name and racket, right? So the idea is that we will translate it like this, we will say, well, we'll call, we'll just define a function, we're just going to call that line 10. And we will make that one call print. And we will we will then define a function called line 20. And well, so go to, of course, in racket doesn't exist, but we can just go to another function by calling it, right? So I'm just going to call line 10 here. And well, you can already see that eventually, this is hardly going to work as line 10 will never proceed to line 20. So eventually, we will also have to put a call to line 20 here, so that the code moves in the right direction. But we'll start simple, we're just going to start with a single line, and then we don't need to call that next function. How does that work? Well, we again say define syntax, right? And in this case, we're just going to define a piece of syntax that we're just going to call basic. Again, it takes an entire form as its argument, and we'll start just with a single command. So I'm going to call syntax powers here, pass it the form and say, well, basic, and the way basic commands work is there's a line number beginning, which is an integer. And then there is a command, which is, in racket terms, is just an expression. So, well, what do we need to do? Again, we need to translate it into another piece of syntax. So, well, we can see up there, right, we want to translate it into define. And then, well, there's an open parent, we're going to have to cook up that line 10 name. Also, we'll defer that problem for a little later. And we will just insert in here the command in the body of the function. So, okay, so that leaves the problem of computing the name line 10 from the line. So we want to compute that name by sticking that, you know, line minus in front. And to that end, well, I want to call a function called format it. And format it, well, so it takes something that I'll explain in just a moment. It takes a pattern, which is sort of like print f, except, well, the pattern language is slightly different. So the tilde a says we should insert something in here and we want to insert the line number. Now, I'm kind of tempted to put in line number here. Now, I need you to understand one thing is that, you know, as syntax, a piece of syntax that represents a number is not the same thing as the number itself. So, for example, if the line number is 10, you know, down here in the REPL, I can type 10, well, I get 10 as a result. But if I type this funny syntax thing 10, then it says, well, there's a piece of syntax and it contains that number 10 in there. If we want to stick it into the name, we really need the number 10 in order to extract the sort of the actual piece of data that is represented by the syntax, I can call a function called syntax E. So if I call syntax E of, you know, here, hashback quote 10, then it gives me that 10. So really, you know, that's what that line number is. So I need to call syntax E here. Moreover, you can see here that I left three dots here, three dots here. Format id also takes an argument that is called the syntactic context. Now, the macro system in Racket has great sophistication when it comes to identifiers. If you've seen macro systems in other languages, such as in C, they're very simple about names. And you can introduce name clashes very easily and Racket avoids that. That's not going to be the main subject of this talk. So suffice it to say that as the context, we want to define the line numbers are at the same level as the basic form that appeared. So we'll just stick this here. So just a little piece of magic here that we're just going to have to accept for the moment. So while this thing, we want to stick that in here, right in here where the three dots are. And to that end, I will just give it a name. I'll call that thing name. And I want to stick it in here. Now you have to understand a very subtle issue here, which is that, so for example, you know, when I have a pattern variable here that appears in the pattern, I can just stick that in this syntax. For example, here I can just put in command and Racket will figure out that I really meant this command here. Unfortunately, well, I mean, there are reasons, but that does not work for this, which is a regular binding. It's not a pattern binding. In order to tell Racket to, well, you know, stick the result of evaluating that piece of code here, I will need to put in hash comma name, which says, well, here's an expression, you know, stick the value of evaluating that expression in here. So that's, I think that's confusing at first, but you'll get used to it. So remember, when there's something in here in the pattern, you can just stick it in here. And if you have regular bindings, or if you have a piece of code that you want to stick in your syntax, you will need to prefix that with a hash comma, which is also called unquote. So let's see if that works. We'll run that. See here, I already ran into the first trap here that I just told you about. I called syntax E on the line number, but the same kind of kind of goes in reverse. So if I can't just use a pattern variable in regular code, instead of instead, what I need to do is I need to prefix that pattern variable with hash back quote. Okay. So, and then it knows, okay, I need to look among the pattern variables for this. So if I run this, well, then it says format it, it doesn't know about that. And that's because format it in racket is in a separate library. So I'm going to need to jump upwards a little bit and say here that for syntax, I don't just want syntax powers. I also want a library called racket syntax. And that hopefully contains format it, go back to that code, press run. And now at least it's silent. And now, well, I could try doing that by saying basic, you know, 10, you know, print hello world. And doesn't do anything, right, as it should, it should just stash that piece of code. And, but if I now type this, it executes that first line of code. So hooray, we've got a very basic, basic version of basic here. That works. Okay, so now we have a single command and a basic program. But of course, we want to have several of them. And so up here, we already know how to do this, or at least how to tell syntax powers about this, we just put three dots here and says there can now be several lines in a, in a basic form. So, but of course, we need to translate that differently. Right now, our expansion just has a single definition for a single line. And now we need to expand that to do several definitions. So to that end, we will record as a functional programming language, we do a map. So we apply a function to all the line numbers and commands and the function always returns a piece of syntax for a single definition. Let's try that. So I'm going to say lambda here, I'm going to put three dots here, because there's a subtlety that I'll explain in just a moment. And now the question is, what do we map over? Well, we want to map over all, you know, all the line numbers and all the commands, of course. So if we have line number here, well, you know, it's not just one line number, right? It's several line numbers, one for each line in the program. So we always have to use it like this, right, that we always put the three dots behind it. Otherwise, this racket will complain. And moreover, line number again is a pattern variable that comes from this line in the syntax parse form. So, you know, the pattern variables, they really only work in this hash bang, this hash backcode form. So that's what we need to write. And also we do the same thing with the commands. And we're still not done because, well, you remember, you know, hash backcode returns a syntax object. But what we really need is a list because we're feeding that into map. So, racket has a handy function that does that called syntax to list. Also, here is syntax to list for the commands. And that allows us to map over, well, here the line numbers and the commands. Now, here's the subtly. Suddenly, because we now have line number and command be regular lambda parameters, they are no longer pattern variables. Okay, they're just regular variables. And this, I think really is the most subtle aspect of the of the racket macro system is sort of you understand this issue, right? The pattern variables come from up here. And everything else, everything that is not here in a syntax parse form in the pattern part is not a pattern variable. So in this case, these are now regular variables. And that means they should not appear in hash back quote anymore. In this part where the line number is also means that a racket is not just going to replace the pattern variable by the expansion down here with commands. So I've got to push hash comma here. So if you figure that out, if you figure out that distinction, I think you'll be fine. So let's try that out. I'll run the program. Thankfully, there are no error messages. Let's try again that basic program. Oh, it complains again. There's another trap. Well, not a trap. Really, it's something that we've seen before is map returns a list. And you would see here, so it says your quote open for N, that means it returned a list. And that means, well, map produced a list, but racket really expects a syntax object. You know, if you're coming from lists or from closure, you expect sort of lists to be usable as syntax objects, but racket makes a strict distinction here. And it uses that distinction to track source code locations to give you good error messages, and also to track hygiene to keep track of lexical binding, which is not the main focus of this tutorial, but just so you know. So anyway, we need to definitely stick hash back quote here. And now we have several definitions. And because we have several definitions that we want to stick in one single form, we use begin just like we did with print before just going to indent everything nicely. And of course, now we want to stick the result of evaluating this in here. So I've got to put hash comma here. Okay. And unfortunately, there's still another subtlety. If you look at the expansion here, you can see that it would, well, hopefully you can see that it would expand into something like this, right, you have begin, you know, from here, and then you have the open paren, which is from the list produced by map. And then inside those open parenns are the definitions of the various lines, like, you know, line 10, line 20, and so on. And you can see that there's one pair of parentheses too many. So we really don't want the racket macro expanded to stick that list in there, we want it to stick the elements of the list one by one inside the begin. And to do that, we need to put a magic character here, also sometimes pronounced unsplicing. So, and that does exactly what we want. So it's just like hash comma, but it expects a list, and it will stick the elements of the list there effectively removing one pair of parentheses. So let's try that again. Oops. So here's that basic form, at least it doesn't air out anymore. Let's see if we can start it. And it says, hello world. And we could also now extend it, right, and put another line here. So right now we only know print. So do that. And so we can start line 10. And you can see that it, well, line 10 runs, but really we expect the program to go on to line 20 after line 10, line 20 is there. But we haven't yet linked the lines together. So that's going to be our next product. So what we need to do here is, well, after the command, you know, associate with the line happens, we need to stick in a call to the next line, right? Now, how do we get that call, right, in order to produce the call for the next line, we need the line number of the next line, but we don't have it here. We just have the current line number. And to do that, we're just going to map over an additional list, which is going to be like the original list of line numbers, but we'll shift it by one. So along with the current line number, there's always the next line number. And to that end, I'm going to call the list function, CUTR, which just, you know, removes the first element from a list. And so that gives us that. And now, of course, map would complain because the lists do not no longer have the same length, right? This list is one shorter. And so we need to append an element to make up for the last element. And since there is no, since there is no line after the last line, I'm just going to pass false, right? This is racket syntax for false hash F. And we'll need to make sure that after the last line, we don't try to stick in a call. Okay. So here we are. Ah, we need to, of course, make sure that now that we're mapping over three lists and not just one, I need to add an additional parameter here, right? And now we could say call next line. And I could just, you know, stick in a call to format it like this, right? And then pass next line number. But that would be poor abstraction. And well, it's kind of copy paste code. So we could pull it out into its own function definition. That's a useful exercise. So I'm going to take that code, copy it one last time, stick it up here. Now there's two subtleties. Let me get rid of this obsolete comment here that we have to pay attention to. So I'm going to call that function, you know, make line name. And of course, we need to pass in the line number, this thing here. But we need to pass in one more thing because this basic here that you see here, remember that it referred to this basic down here, right? It used to be here, but that's no longer in scope because we're outside of its parentheses. So we also need to pass it in as an argument. And we'll call that, it's called the context used by Rackett to make sure that the definitions appear in the right place and are referable by their name. Okay, so we have this great. And then we can take out those two calls here and call make line name instead. Remember that we now need to call this basic thing that was there before and line number. And the same thing over here, I'm sorry, it's called make line name, probably noticed. And we'll do the same thing here, call it make line name, remove all the crud here, let's duplicate it. And now we have a nice call. Of course, well, we'll get to the fact that it's not a call yet in a moment. But also, I want you to think about one more aspect of macro expansion. This definition up here of make line name is just a regular procedure definition. But we want to call it from the macro expansion process. So essentially at compile time, where macro expansion time, but this is just a regular runtime function, it will not be available until runtime happens. And in order to tell Rackett, no, this is a function, please make it available to macro expansion, we need to replace the define by define for syntax. So define it so it's available during syntax expansion. Okay, so one more thing. This is of course, not a call yet. In order to have a call, we need to have parentheses to the parentheses around the name of the next line. So I'm going to put do this and put parentheses around it and then stick this in with hash comma. And that would be the call to the next line. Well, except it isn't quite, because remember, next line number can be false. And then we would not have a function to call there. So we really need to make a case distinction here. I'm going to say, well, if there's a next line number, then this is a great function to call. This is the function of the next line. And if there isn't, well, I'm just going to call void, which is a function that does absolutely nothing that's built into Rackett. Okay, let's try it out. Let's give it a whirl. Well, oh, it says next line number, reference to an unbound identifier. That's because I put a typo here. Let's try that again. Okay, now it at least goes through. Let's see. We had that program that had two lines in it. It doesn't air out. That's great. Call line 10. And, ah, okay, now it runs both lines. That's wonderful. But we see that we made a small mistake in the implementation of print, at least in Apple SoftBasic, when you call print, it prints return a new line at the end. So why don't, as a last act of this particular task, we go up here and stick up here, they called a new line. That should work. Try that again. And now we get the output as we've wanted it from the beginning. So now we can have a sort of a basic program that has several lines where the lines are run consecutively. That's great. So what's next? Well, here we can already see what might be next, which is the infamous go to command in basic, which just jumps to another place with no memory of where it came from or what was going on beforehand. So if we look at our code here, we can see that go to is going to require special treatment. We can't just implement it the way that we implement print because, of course, when there's a go to, there should not be a call to the next line after the go to that would make no sense because go to just transfers control directly. Now, well, things are getting a little bit unwieldy at this point in this, in this longest function here. So I will put the code that translates a single command into a separate function and we'll see how that goes. So here again, remember that we need to say define for syntax. So it's not a macro, it's a function that is called from a macro definition going to call that function translate command. And well, so we pass in the pass in the actual command. So that's the, that's, if you will, the original source code from the basic program. And we need to translate that into a racket code. And well, and the idea is that we use translate command to produce the body of this function here. So of course, that means we need to also pass in call next line. So let's see. Let's pass that in. Great. And for now, we're just going to keep that very simple. We're just going to going to keep the definition from up here. Now again, remember that when we produce syntax, we can only produce a single form at a time. So there are two forms here, but we know we can always combine two forms into one by sticking a begin in front. So we'll just do the abstraction for now without adding additional functionality. So instead, we will say, okay, translate command of command and call next line and omit this to our way with that. Make sure that all the parentheses line up. Yeah, it's getting getting to be a lot of parentheses and see if that works. So run that and your program runs as before. That's great. So we want to give go to special treatment and in order to do that, well, we need to inspect the command here and see, well, if it's a go to command, then we give it this treatment. In all other cases, we just translate to the code that we already have. So I call syntax powers call pass the command. And now I put here a pattern where it says, well, if it's a go to command looks like this, go to there's a line number and that line number is an integer, then I'm going to do one thing. And in all other cases, I want to do something else. And you can see here that, you know, this is now a syntax parse that has several clauses, and it just tries them in order with the compiler. While it does syntax and expansion, first tries the first one, then the second one, the first one that matches gets to be run. And this thing here is just a wild card pattern that matches anything that you want. So that's the fall through clause at the end. Okay, so in this case, well, we have go to now here's another subtly, in order, we really want the word go to to appear there. But remember that syntax parse does pattern matching. So whenever there's a name here, that name is just going to match whatever is there in the input. It's really subtle issue. We really want the word go to to appear here. And another subtle issue is that we want that to be consistent with the rest of the binding structure of the program. We'll ignore that here. But in order to do that, we need to put something like this, a magic clause here at the beginning that says, well, go to whenever it appears here, means that we really want the word go to to show up there. So for all of our specially treated commands, we need to put that in, we put them, you need to put the name of that command into the literals clause up here. Okay. So now, when there's a go to, we don't, you know, we just want to generate a call to the procedure that's associated with that line number. And we get the name of that procedure by saying make line name. Well, and remember, make line name now needs that context argument up here. We don't have it here. So we need to add another parameter, call that context up here, we pass basic. And okay, got that here. And then we can call make line name with context and the line number. Okay, so got that here. And that's just the name of the procedure. We still need to make a piece of syntax with a call to it. So we put hash bang back quote, put a pair of parentheses. And in the middle, we need to put hash bang comma to actually stick that name there. Okay, well, let's see if it does anything. Oh, error message. Well, again, I fell into the same trap that I always fall into line number here is a pattern variable. So it really is only valid inside a hash bang back quote. Well, it is inside a hash bang back quote, but the hash bang comma kind of undoes that. So we refer to it as a regular variable here in order to refer to a pattern variable. Again, we need to put hash bang back quote here. So it must seem confusing at this point. But if you just do a bunch of stuff with macros, you will soon get the hang of that, I think. And then you'll recognize the error message right here. It says pattern variable cannot be used outside of a template. So line number is a pattern variable. Now we can see, oh, that nicely, that occurrence here of line number nicely refers to that one. Let's see if it runs now. Okay. See if we can do this. So, well, here's a rule that I forgot about, but fortunately, it's easy to remember because there's an error message says literal is unbound in phase zero, right? Phase zero has to do with the phase zero is runtime. Don't have to worry about that too much. But it just means that go to has no definition that it refers to. And we're not interested in a definition because we're translating calls to go to into something else anyway. So I'm just going to say put in a dummy definition here to false. And that just means if there's any reference to that go to, then, you know, syntax parse will recognize it. Let's push the wrong button there. Let me try again. And, well, at least it doesn't error out. Let's see if we can run the program. And that's pretty neat. And now, well, I guess it's time that I put an actual basic program here into the source code. So I don't have to keep re typing it like this, you know, like, like this, for example, and well, how are we going to see that it works? I'm just going to put a line 20 there that says go to 40, make that line 30. And put line 40, which says, you know, the end, right, so that we know the program is done. Let's see if that works. We could call line 10. And indeed, it says how world the end and it skips hello again. So go to works now. All right, what's next? Well, we can just keep adding commands, right? But of course, some cause more trouble than others. And one that causes a little bit of trouble is assignment, right? We can have variables in basic might look like, well, in basic, it would look like this, right? A equals 42. And, you know, that would establish a variable called a, and we could then use it, right? We could say something like this. And I'm going to use a syntax slightly different. So of course, it has to be parenthesized. The operator has to be in front. So might look like this. And I'm going to use colon equals to avoid confusion with the, with the equal sign without a colon, which is the equality operator. Okay. So we'll just use the same technique that we used before. So I'm going to just like go to where it needs to be a literal, we want another case to match only when there's an actual colon equal sign there. Well, here's a variable. And that has to be an identifier. That's how we talk a tel syntax parts that needs to be an identifier. And then there's a right hand side, which can be any expression that we want. Okay. So got this here. And now we just need to generate a piece of code that works with it, right? And in, in racket variable assignment happens with an operator that's called set bang. So the exclamation mark is pronounced bang to denote that there's an evil side effect happening here. So, and we can just use these two pattern variables. We can just use the right hand side here. And okay, well, remember that trick that we needed here with go to, we need it with colon equals to. So we need to provide a dummy definition. And now, well, here's a little problem, right? It says a is an unbound identifier down here. And that's because, well, racket distinguishes between binding an identifier. So you're establishing that the identifier exists and then changing the value in the storage cell associated with that identifier. Those are two separate things, right? And so we could make that error message go away by saying, you know, define a to some dummy value. But then of course, that's no longer basic, right? It's, if we would need that additional definition and basic, we can just introduce an identifier by doing an assignment to it. So somehow we need to generate the set of variables that appear in assignments and make sure that we also generate definitions for those identifiers, dummy definitions at the beginning. So, well, we need to generate those definitions. And for generating those definitions, we need the set of variables that are in a program. We'll start by just collecting the variables that are in a single command. And to that end, I'm just going to define a new function here to call that collect variables. And I'm going to pass in the command. Okay. And well, not just the command, but we will want to collect a list or rather a set of those variables, because of course, a variable might occur in several assignments. So we will use something called an id set. And we'll define that in a minute. So a racket comes with a library that allows you to manage sets of these identifiers. And we will need to import that library up here. So call that syntax, that's called syntax id set. And that allows us to deal with those sets. Okay. And of course, we need to look inside those commands. So we're just going to use the same syntax parts, essentially the same syntax parts that we had up here, right? So we have various literals here with go to, of course, there can't be any variables introduced by a go to, we're mainly interested in the clause that has to do with assignments. And well, how does that work? So in that case, what we need to do is we need to say, free id set, add, and call id set, and add the variable here. And anything else, well, we're not going to do anything, I'm just going to call void, the function, the built in function that doesn't do anything. Okay. Okay. And now we need to call that function, collect variables for each command. And we need to do that up here, from the basic form, right, where we have all the commands at our disposal. So first of all, we need to create an empty it set that we can add to call it it set. And so we're going to call a function, it's got a cumbersome name mutable it set. Oh, and not just that mutable free it set, because again, racket makes subtle distinction between identifiers, depending on whether they're free or bound. Don't worry about that. This, this is going to work. And then what we do is, well, we want to call, collect variables on each command and passing in that it set. And, but remember, we don't just have a single command, we have a whole list of them. And so we need to do this. Well, we take that command dot, dot, dot, and of course, need to wrap that hash back quote around it, convert that into a list, we've done that before. And then we can use a built in function called for each, which just calls this function on every argument on every element of that list, right? It's sort of like mapped, but that returns your list of the result, whereas for each just doesn't bother with the results and throws them away. So now we have a set of identifiers. We just need to generate definitions from it. And these definitions need to go here, right before the actual code for the basic program. So again, we can use map, right, to map all over all the variables. We'll figure out in a moment what we're going to do with those. And well, we'd like to map over its set. Unfortunately, its set is not a list and map wants a list. So we are going to use a function called free. It's set to list to make it a list, right? And so for each variable, we want to generate a dummy definition. Again, we need to generate a piece of syntax says define, we stick that variable in, and we'll just define it to false to make sure that things don't get mixed up with regular basic values. Okay, now remember again, I mean, you can see here the other called a map, right? We want to make sure that this gets spliced into that form here. So again, I call that I use that same funny magic thing here hash comma at, which splices all those definitions into that begin form. Okay, well, let's see if that works. And now it says, oh, that variable is already defined. That's a good sign because you can't have several definitions for single variables, and we put one in ourselves. So I'm going to delete that. And okay. And now we can see that there's a variable here, right? It's bound to false as it should be. And let's try running the program. Well, not much happens. Why is that, right? Why does the program not produce anything? Well, we could look at a is now 42. Let's see, let's go up here. And we can see here the translation of colon equals and you see, you can see I forgot something. This just sets the variable, but it does not go on with the program as the rest does. We need to put a begin here and then put in the call to the next line. See if that works. And now line five. And that actually prints out the value of that variable. So now we've got variables in our program. That's wonderful. Next up are conditionals and basic. So how does that work? Well, basic has a statement called the if statement might look like this, right? We compare a with 42. It's kind of silly. We know what the result is, but just to demonstrate the feature. And then depending on whether that's true or false, we could whatever set B to one, or set B to two. And we'll just print that out along with the A here, just so we know what came out. All right. Well, we need to translate the if command, of course. So we need to go up here into our translate command function. We need to add if here to the list of literals. So we can recognize ifs. And it looks like this. Well, we have some tests. That's an expression. We have a then branch. That's an expression. And we have an else branch. That's an expression. And well, we're just going to translate it into rackets if, but we need to make sure that the then branch and the else branch also get translated. So we use hash back quote. Again, this is not the same if as here. This is now rackets if. We're just going to leave test as it is. And then we call translate command. And remember, we want the result of translate command stuck in there. So we need to put hash comma in front, pass in some context. You know, again, you know, we need to pass pattern variable here. So we need to put hash back quote in front and we'll pass call next line as before. And do the same thing here, translate command context. And call call next line. Okay. Well, let's try running it. That doesn't work, right? We get an error message says B is unbound. Why is that? Well, we're generating definitions for all the variables that occur. But we, but so far, we're only looking at assignments that occur at the top level. We're not looking inside and if so we need to make sure that we also extend collect variables in the same way that we extended translate command. So we also need to add that to the list of literals here and just put the same. We can just copy the pattern from up here. Copy that here. And, well, we just need to call collect variables recursively on the two branches, right? So we call collect variables on then with the it's set. And we call collect variables with the else on the it's set. Well, let's see if it gets any better. Well, at least it doesn't air out. Let's show up our program again. We could call it starting at line number five. And that's not bad. So we now see the it says Princeton number one, which comes from this assignment here, right? And so B is assigned to one. If A is 42, when A is indeed 42, we could also kind of set it to 41. Try that again, line five. And we see that it now prints A2 because the branch went the other way. I'll just change that back and save. Great. So we'll add one more feature. That's actually not just one command, but we'll need three commands. And that is basic sub routines. So we'll help. What could that look like? For example, in line 1000, we could say we're just going to print the B variable. And then in line 1010, we just put a return statement. So AppleSoft basic doesn't have functions in the way that we know them, or methods. It just has these sub routines. And so we could call, we could output B by just saying, you know, whatever line eight here, and we could say, go sub line 1000 like this. And the idea is then that it will go and print B, right? And return and go on after that with Hello World and whatnot. And we could also call it several times, of course. Okay. And yeah, we'll just leave it like that two commands. Okay, so go sub and return. What should we do? Well, we go up here. And obviously, we need to do something with translate command, we will just add go sub and return. And these are not built into Racket. So we need to have go sub false dummy definitions for them to do that. And so how does that work? Well, go sub is kind of like go to accept the program goes on, right? So while there's go sub line number, which is an integer. And of course, well, we need to generate a call just as we do with go to, right? So going to put, well, I'm just going to duplicate that from up here. And the only difference is after that call is done, right, when that finishes, then after that is done, we need to continue with the next line. Unfortunately, we know how to do that because we've got the call to the next line sitting up here. So I just stick call next line here. So that's the only difference between go to and go sub. And of course, we also need to implement return. And well, what happens with return? With return, nothing happens. Also return is similar to go to and goes up in that it doesn't go to the next line, it just returns. And when there's an active go to it will just go back to the line after that it will go to that call to call next line here. Let's see if that works. Okay, so at least we don't get an error message line five. And well, how do we interpret that output? I kind of forgot what the program was. Well, here it said goes up 1000, it printed B, and then it went on with the program as before. Arguably, the program should finish here after line 40 and before line 1000, but I'll leave that for you as an exercise to implement. So now we have enough basic there to at least give us a fair idea of how to do this whole thing and maybe how to add the rest. And I promise that we would also get actual basic syntax and we only have a few minutes left. So let's get to it. In order to do that, we need to make sure that the functionality that we've implemented in this file is available in other files. And so we need to add what's called a provide form. That's like an export or public annotation or something like that. And we just need to provide the basic form that we implemented, the print command that we implemented, go to, you know, all those things that we added to implement our basic syntax return, there we go and save that file. And then we're done with that part. Now, for the actual basic syntax, there's no way around writing a proper parser. And we don't have nearly enough time to do that. It's not that hard, but it takes a little bit of time. So I've prepared the code to do that. It's just called a reader. So here's a file called basic reader dot racket. And if I run that, well, you can see here, for example, there's a function called parse line takes two arguments, one of them is called Zerts. That's just context information about what file this code resides in and so on. I'm just going to pass false here. Later, that value will be provided by the racket system. And the other argument is an input port, which is just a reference to an open file or something like that. But we can also use a string by just saying open input string here. So if I say, you know, 10 print a plus one, and call that function, you can see that it returns list structure that corresponds exactly to, you know, the syntax that we the parenthesis syntax that we implemented earlier. We just need to hook this up to the rest of the racket system. And for that, we need to write a little boilerplate function going to call that basic read syntax also takes a Zerts and an in argument. And that function calls parse program, which does the same thing as parse line, but for an entire program do this. And we want to embed the result of that in something that racket will later understand. So to that, and we need to create a module declaration for a racket, I'm just going to call that module basic. My name doesn't particularly matter. It imports the racket language. And it also requires so it imports the code that we've just written, which is in the racket dot arcade t file. And now here's the result of parsing the program, we still need to put a basic form around it and splice in the result of parsing the program. So that's that. Now you might have noticed that the back quote up here and the comma add here doesn't have a hash with it. It doesn't create syntax structure, just creates an ordinary racket list structure. We still need to convert it to a syntax object. And for that, we use a function called datum to syntax. And just pass false here for the context. That's enough. Make sure that all the parentheses line up. And then we're good to go. Well, almost good to go. We still need to export this function. And we need to have a different slightly different name for it. We're just going to call it read syntax on export. And that's a magic name that racket will later recognize. So I'll save that file, and then we are good to go. So in order to try this out, I've prepared another little example file called basic demo syntax dot rkt. This is what it looks like. And you can see the hash line, lang line up here declares that we want to use the reader that we just implemented, basic reader dot rkt. And what follows is very much Applesoft looking program. And while I can run that by just typing line 10, like we used to do that, and we can see that it pretty much works. So there you have it. I mean, Applesoft is just about as different from usual racket as it can be. And so if you can do that, then you can implement just about any language you like with the racket system. And it's not just for toy stuff. It's a great tool for organizing your software architecture for just generally generating pleasant notation for doing things like documentation in the racket system. There's lots of publications and lots of examples on that. And of course, we've implemented the same, we've used the same machinery to also implement teaching languages and to iterate quickly, improving those languages. So I hope this has motivated you to try out the racket system. Again, please check out the GitHub repository if you want to look at the code or send me email or contact me some other ways. If you're interested in this stuff, and I'm looking forward to your question and a little bit of discussion, we'll see. Thank you. Thank you very much, Mike for the talk and to for introducing us to racket and to show us how to implement a language there. I already saw there was already quite some discussion in the chat, but we have still some questions for everyone. And we start with a question from someone and the question is why you use these macros instead of simple functions because he thinks you can also use just functions to produce print. Yeah. So that question was asked just after I introduced print, right? So it doesn't refer to all the rest that we did there. And indeed print could be a simple function, but and indeed there's a lot of things that can just be functions that don't need to be macros and racket, but and print is one of them. But then I couldn't have used it to demo a single simple macro, right? All the other macros that I introduced are a lot more complicated. So I figured I'd do a simple one first. Very good. Let's go on. There are all the differences between bindings and operators. It would be entrepreneurs to quote unquote, it would be something I wouldn't it would be great if you showed them more in detail, but that's more common. Yeah. So so it is a little bit confusing that there's two kinds of variables, right? And I hope throughout the talk, there's enough of them that you can see what the difference is. The reason for that is in that this is sort of an expert level macro system that we're using there. And the simple macro system that's in racket, the construct for that is called syntax rules knows only pattern variables. So if you deal only with syntax rules, then it's very simple. But if you want to write more complicated macros like the ones that we did, then you naturally sort of incur both the pattern variables and the regular variables. But there's only these two kinds, right? And you get you get the hang of it by playing around with it. So then there's a comment by Cinehonia who says all the switching between hash, back quote and hash kind of reminds her of existing and entering math mode in latex. Yeah, that's that's a that's a great analogy, I think, right? In latex, when you put a dollar sign, you enter this different syntax, this different world there. And then you can exit that again by writing, you know, backslash text RM or something like that. That's exactly that's exactly the same, the same idea that you have, that you have a racket, right, is that you enter the syntax construction mode, and that you exit it to regular code, and then you can re enter it. So you kind of can pop, pop, pop, push, push, push, you know, like the like the infinite turtles, I guess. The next question is, if there is, if you have control over the runtime of your language with a racket? Yeah, so I see the question goes on a little bit that and suggests that memory management might be a subject. So, so racket gives you great control over the runtime, because it has one of the most flexible runtimes in existence, it might be the most flexible one in existence, specifically when it comes to control structures. So, so macros is one aspect of the flexibility of a racket. Another one is that it has something called delimited control, which allows you to, to implement exceptions, to implement threading, all kinds of things in terms of the primitive constructs of the language. It does not give you control over memory management directly. I mean, you can influence the way that it deals with memory a little bit. It does not give you control whether it's interpreted or compiled, but if you want an interpreted language, you just write an interpreter. So I guess it gets your control over that. But there's a great many examples that come with a racket language and its package ecosystem of different languages. So one example is that there's a Haskell-like language, for example, that comes, the only difference is that it has parentheses. Or I think the racket-based system comes with a language that does functional reactive programming, where each value changes over time. You can see it change live in the rebel. The next question is from Decoder. Did you implement a whole basic interpreter in racket? And how does it compare in speed with a more traditional interpreter? So, well, I implemented a little bit more than I showed, but I didn't, you know, I didn't put all the features of Apple SoftBasic and the graphics in there. You notice because it's implemented in macros though, it is a compiler effectively. So again, it's compiled into racket code. And I think it would compare quite favorably because racket has a pretty, pretty efficient compiler in the backseat there. So, it's kind of, it's not apples to oranges, it's kind of apples to 40-year-old prunes. So, and racket obviously doesn't run on an Apple too. But I think the Apple SoftBasic implementation that you're getting there is going to run quite fast actually. The next question is from Proless. Do you think racket with its extensive macrosystem is particularly well suited for building languages? So, it is the best system in the world to build languages, right? And when, so, and when in my commercial work, right, whenever we need to design a DSL, then racket is at least involved in the prototyping stage. And it could very well be involved in the production stage as well. Specifically, you've noticed that racket is a Lisp descendant, I guess in many ways. And of course, if you get a more traditional Lisp like Clojure, it also has a macrosystem. But the macrosystems of traditional Lisp are significantly less powerful than the one in racket. And there's two differences that are relevant. One of them is that the syntax in traditional Lisp often doesn't track source code information. So you don't really get good error messages, which are important if you really have domain specific languages that are exposed to domain experts. And the other one is that it deals with hygiene. We didn't talk about this in the talk because there's lots of other talks and papers on the topic of hygiene. But you know that sometimes when you write a macro, you need to invent new names that are just supposed to be temporary. And doing that requires extreme care in a traditional Lisp system, whereas in racket it's just automatic. And so pretty much the subtleties that I've shown, that's a pretty complete set. I don't think I deal with more subtleties on a day-to-day basis. And in a traditional Lisp system, you would also need to deal with hygiene issues, which you don't in racket. So that gets you sort of one level of power above traditional Lisp in terms of how funky you can get with your DSL.