 Welcome to Emerging Languages Camp, 2010. Ioki and South by Ola Bini. Okay, hello. Yeah, my name is Ola Bini. I work for a consulting school in Hogwarts, and I'm an amateur language designer and language implementer by night, basically. I had 20 minutes to introduce two different languages here, so we'll see how that goes. I'm not sure if there will be much room for questions. I'll be around, so if you have any questions, just cast me out. Most of the talk is going to be about the first language, because the second language is kind of new, like three weeks of implementation or something like that. So I'll just get started. So Ioki is a language experiment. It's a language experiment specifically to investigate expressiveness. When I say expressiveness, people ask me to define it, and that's pretty tricky. So I've kind of given up on defining expressiveness, and instead talk about this from the angle of what I feel is expressive and what I feel gives you the right tools to communicate. For me, programming languages is about communication. It's about communication with your team members, with yourself, with the compiler, with your team in five years and so on. To me, expressiveness is about how can you make it possible to communicate as clearly as possible your intentions without actually embedding anything that is not about your intentions. My view of expressiveness comes quite a lot from the Lisp family, but also from Ruby, one of the German core developers. That's kind of where I am from a language standpoint. Ioki is a dynamically typed language, just like those. It's closest to parent is IO. It's actually really similar to IO in the basic syntax and the basic runtime model. If anyone is interested in major differences, I can talk about that later. But right now, I'm basically just going to talk about a few features of Ioki. Ioki is a very small language core that allow you to add quite a lot of stuff, which I've done. So this is a long section of stuff that I'm probably not going to cover at all whatsoever. Instead, I'm going to focus in on three different things that makes it possible to write expressive programs. A big part of that is the way the core language macros work and so on. But yeah, I mean, both of this stuff is implemented in language itself and it's just there. So going back to the basics. The language is dynamic, strongly typed. It's prototype based. When I designed language, I wanted to... For me, I wanted to have the core as small and simple as possible. To me, having instances in classes, the two different concepts, it's fine, but just having instances is much simpler than having two different kinds of things. So that means that language core is kind of halved in size just because of that decision. Everything is based around message passing in the manner of small talk, not in a manner of actor-based languages. It's also a very mutable language. It embraces mutability. Everything is mutable. Since it's a language experiment, I could get away with not doing anything about concurrency, for example. So you can't actually do anything when it comes to concurrency. That's not really what I wanted to explore. Which means that you can do totally unsafe things all over the place and there are no real problems with doing that because you can't share to a different thread anyway. So the JVM is the home of this language because I didn't want to write all the other stuff. I wanted to focus on the language, the language semantics themselves. And the garbage collector in the just-in-time-for-power-off Java just makes it very simple for me to do that. It turns out that it was really simple to port to the CLR. So the language also runs on the CLR. Actually, it runs on a Mono2. That was maybe two weeks of work because I have a fairly extensive test suite of the language. So I could just basically run the tests until the test passed and I knew that I had a working watch of the language. There are quite a few different features of the language that I like. One of my favorites is the condition system. The condition system is basically a protocol for reasoning about errors and problems that can happen during your program run. It's kind of high-level comparison to exceptions. You can implement the condition system using exceptions. I like it quite a lot because it makes it easier for people to talk about protocols. What should someone do when something goes wrong? How can I fix this? Can I do something about this? And if so, I want to specify a protocol for how people can actually fix this. Because in general, the exception system is something like Java. You throw an exception when something goes wrong and then the user of that library can catch that exception. But at that point, it's really too late to do anything about it. So what you do is you just put the whole thing to a log file and forget about it, because there's really not that much you can do about it. The condition system allowed the library authors to say, okay, this can go wrong here. I'm going to signal a condition, but I'm also going to give you three different ways of getting back from this condition and actually continue execution from that point on. And then you can choose. As the library user, you can choose, do I want to unwind the stack to this point or do I want to continue execution by fixing this problem this way? That's something I like a lot. It's got runtime macros. Those are not like Lisp macros. Lisp macros are load time or compile time. These are runtime macros. It's kind of an in-between between call by name and regular macros, because you actually get access to the messages that are sent in, so you can modify, transform them, and execute them and stuff like that. IOKA has more syntax than Lisp, but less syntax than most other languages. It's also part of the whole of making languages accessible and core small and simple. So this is a small example of the sugar syntax, because I wanted to have the core syntax be as small as possible. Operator precedents and things like that is actually something that I add on top of a canonical syntax. So this is a sugar syntax that kind of shows some of the examples. I got some math here using regular precedents, and I created a new list of stuff. I assigned something into that stuff. This looks very much like Ruby style. It's almost exactly like Ruby, actually. This one creates a new method and adds stats to add on a slot on text, and then you can call that method. So you can change everything. Everything is open. If you desugar that, you will see that the syntax is quite similar, but there are things here that are changed. So the canonical syntax, equal sign, is a method call. The same thing is true here. You can desugar enough the operators, equal sign down here. So square brackets is also a method call, and square brackets sign them and so on. So it takes the idea of assignment into places just like common list. Quite seriously, which means that you can do things like this, where you do an assignment into a place that is specified by the first argument. Yes, actually, I had that on my slides. Anyway. There is also an internal syntax that you generally don't see. The canonical syntax you will see, but the internal syntax you will not see and the reason it's ugly is specifically because literals is also a model that's message sending, which means that you can override it and do stuff like that. That's pretty horrible from a performance standpoint, but it'll allow you to do some pretty expressive things when you want to do DSLs, for example. Okay. So that's kind of the basic syntax. I have a typical example where I use the features of the macro system to kind of fold code into something that expresses the pieces that are necessary, but not the pieces that are not necessary. This is something that actually happened with the implementation itself of Ioki going from the first release to the second release. So I want to show you an innumerable method called find. The behavior find takes, comes in three different forms. One form where you don't have any arguments at all, one form where you have one argument that is a message send, one form where you have two arguments, where the first argument is the name of the argument and the second one is kind of a place that uses that name to do something. All of these kind of do the same thing. They find something that is where the thing in the argument list retires true and then retires the original value. So here 42 is the first true value, we'd have done that. Here we find the first square, the first doubled value in the list from 1 to 100, average on that value, which is 44. Here we apply a regular expression to each string and return the string, the first string that matches. And finally here we return the first cube that is larger than 100, which is 5. So this is find. Sum is a very similar method. The only difference in sum instead of find is that sum will actually return the result of the execution on the right-hand side instead of the original value. So here, when you do a regular expression match, the first regular expression match that matches will return a regular expression match, but the other ones will return nil, because that's how regular expression match works in IOC and Ruby also, if that matter. So the same thing is true with indexing into a string. If you index out of the bounds, it will return nil, otherwise it will return the numeric character of the place where it indexed. So this one will only work for full bar, which means that it will return the a lower case. So these methods are extremely similar. They all take the three different arguments. There are about 25 methods like this in the core of enumerable that all do almost exactly the same thing. Or rather, they do the same thing three different times, but they're almost the same. Now, the initial implementation of find, this is not the one you will find in the core, by the way, because I've removed everything in IOC that has a documentation string. So you associate documentation with everything in the system. I've removed that because that makes the code examples quite a bit longer, but everything else is exactly like it was in the first release of IOC. And what you can see here is that we get the arguments, we check the length, and then we go through using each, and then if cell n is true, then we return it. If length is one, we evaluate the code on the current ground, which is kind of a flexible context outside of the call, and then we return the n value if that is true. And finally, if we get two arguments, we get the block using the x name or the argument name, and then we evaluate that block and return n if it's true. Now, the other implementation is some. And if you look really carefully, you will see the difference now. Did anyone see it? It's very small. It's kind of spread, it's here, and it's here, and it's here, and the difference is that we return, in this case, we return the same. In this case, we return the evaluated value and the same thing here. We return the result of evaluating the block. So this code is almost the same thing, but there's all pieces of code that kind of change inside of it. The new implementation in IOK, the one that ships right now, excluding documentation string, looks like this. It's using a macro called innumerable default method. It takes three arguments. The first argument is what to do before iteration. The second argument is to what to do on each iteration. And the third argument is to what to actually do after iteration if nothing returns inside of here. So we return nil if we can't find anything, return false if we can't find anything. Otherwise, x is the value of the transformation and is the value before transformation in this case. So here we can see that the difference between fine is that we return n, the value before transformation in this case, but we return the value of the transformation here. And that's the whole difference. And these three things are the only pieces that you need to kind of parameterize most of the innumerable methods to do something. And notice that you couldn't really, in a good way, use lexical blocks or a closure or something like that because we actually want to return from the enclosing thing the generated method. So what this looks like in implementation is like this. This is mostly a common-wisp style code template except that it's not common-wisp because it's actually object-oriented. And you can see here that we have something called a desyntax. Desyntax just takes several arguments and destructures them. And returns a new template, which is in this case a demacro. You can see that this is almost exactly like the implementation originally a fine as-up except that we have these places where we insert the repetition code, we insert the return code and the init code. Same thing here. Init code, repetition code, return code. Init code, repetition code, return code. And this implementation is used in about 25 different innumerable methods. And I've been keeping adding to the innumerable methods and the benefit of this is it's really easy to add new stuff if I want to. On the other hand, the whole cell x, cell n thing is not a very good plot on the API, which is why I'm not showing this that obviously, but this is a latch bound macro. So it's only available for the implementation of the innumerable methods. It's kind of like an implementation detail. Okay. My final example in IOK is of course a combinator library. This is example usage. It looks kind of like BNF. This is executable code that will return a parser. And the way it's doing that is first taking kind of the, all of this, it takes as code, it transforms it into this, where it gets the precedence right and so on. And finally, when it's done that, it uses the fact that you can override literal creation inside of a specific context, obu.org of it to take any literal that you find like this and just return a parser for that literal instead of the literal itself. Same thing is true here. We changed all the regular names like letter here and put a prime inside in front of them. The prime method will just return a parser parser that delegates parsing to the parser specified by the name letter here, for example. So this is what a parser context would find. In total, create text is the method that is used to create a new text element from a literal. So we get a raw argument, create a new text parser, set the text to be the super call so we actually get the real literal and we do the same thing with number. The hand down here is the implementation of the prime symbol that takes a name and then returns a new parser parser with the name of that name. And then you can combine these in a purely obituary way. So this is just an implementation of the dot dot method, returns a new range parser, returns a new raw parser and so on. And these are available in the sub-context because these go-ups are all obituary. Okay, so that was exactly 15 minutes, which I'm pretty impressed by myself. I didn't actually expect that to happen. So the new language I'm working on is called SEF. And you will see some code samples. Most of them don't run yet. I'm pretty close to some of them running, but most of them are at this point kind of goals instead of actual real running code. The goal with SEF is to take Ioki into the real world, and that means handling stuff like concurrency, performance and other details. Ioki is pretty slow, like 10 times slower than Ruby, maybe. Something like that. I haven't spent much time optimizing it, but it's pretty slow because that's not really what I've been interested in looking at. SEF, at this point, Ioki is about 18 months old, and at this point I felt that I'd learned enough from Ioki that I could start going away from Ioki and see what happens if I add some new things, mix around, stir and see if I can make something that really actually is useful for real programs. And it turns out that to do that I had to steal from lots of languages. I kind of have a differential feature list here that is the features that are different from Ioki. The first one, every object is immutable. The AST is immutable. In Ioki, you can change the AST that you're currently executing, which is horrible from a performance standpoint because it's really hard to do just entire population. For example, in SEF, everything, every object in the system is immutable. It's still a prototype based on Victorian system, but everything is immutable. But local variables are not immutable. They are mutable, so you kind of have the escape hatch of having local variables. And lexical poachers in a way that can mutate the original lexical scope. So you have a big gaping hole of mutability that you can use if you want to, but hopefully the other things that I steal from other languages will allow you to not do that. So, lightweight threads in the style of Killim are going to be added, so you should be able to do things in a similar manner to the way you work with message passing in our line. That includes doing proper TCO, which will be a performance hit, but I think it's worth it for the expressiveness. I'm also stealing closures, conclusive primitives, and also, I wonder, yeah, I have it on that slide. And also, full numeric tower. I didn't mention this, but IOK doesn't actually have floats at all in the language. Instead, everything is a big decimal because it feels like everyone is misusing floats so much these days. I've seen so much Java code that uses floats to represent money, for example. It's horrible. Okay, I decided to do away with that, but I'm probably going back to having floats in SEF because I do want to have real numeric tower with complex numbers and so on, and actually have that be reasonably fast. So, another way of looking at this is all the stolen features and ideas from different places. Most of it, as you can see, is from closure. I steal some ideas when it comes to the module system from IOK, the numerical tower is straight from Java. Mostly everything else is from IOK, basically. So, just some code samples. Paul Graham has this example of how expressive your language is based on how small you can make an accumulator. Now, this is kind of the SEF version of the accumulator. The hash symbol creates an abstraction that is both obi-scoped and lexical-scoped. That actually works when you have a beautiful audience because it's always easy to see which reference is what. A top-level name is kind of followed by a colon symbol. There is a reason for that. In IOK, the name followed by a colon is used for keyword arguments. And the top-level, as I mentioned a little bit while, a top-level in SEF is actually it's basically a method called to something that creates a module. So, anything that uses a keyword argument should be exported. This is the lightweight threads version. Kind of the Erlang style way of doing the same thing here. So, here I create a local method. This one is only accessible through lexical-scoping. So, it's not exported from the module. Yes. Tail recursive call to itself and also start a new process and saving the pit. This is extremely similar to Erlang. Finally, tail recursive sets kind of an implementation of the ideas from Guy Steele's paper about how important TCO is for object-oriented languages. The module system I would actually have time to talk about but the basic idea is that modules are just regular SEF objects and you have to specify what you import. It's not taking the new-speak idea of having a platform object, but basically everything else is similar. Some resources and that's it.