 This is yet another Alang demo. Okay. Yeah, can't give you too many of these Hope you're not too bored by them already so What I plan to do here is to run a system showing some of the properties of the Alang system so on this window up here, I've started up an Alang and I'm going to start up a system that comes with it called observer It's a tool for introspection So I can look I can look at the system here. I can look what's going on in the system, etc, etc So I can for example look at the load and If we look at the load here is doing very little is the only only load you got the load serve itself So now let's try and run an application here So now we're going to start up another system here and this one. We're running in LFE Okay, partially anyway, so now I just have to load in some commands This is just defining some local functions in the LFE repel this is the this is Down here, you can see what the LFE mug looks like in ASCII ASCII graphics, right? The one out on the desk looks much nicer. How do I make this thing bigger bigger plus right? We'll see this work showing Is that better? Yeah, on the other on the other shell. I was just I was just starting the observer system So there's nothing much there to see. Can you see? Okay, so now this is a space this little demo with spaceships, okay? We will do one thing So yeah, we'll go here So now I can start the system So now I'm gonna I'm gonna We're running in the LFE shell now So now I can start the system I can say how big I want to how big I want the universe to be so it's 500 sectors by 500 sectors and we'll try running 2000 spaceships on it as yet You don't see anything because I actually haven't started the the timer running to start to simulate this now I can do a start run here so I can run this and We do a timing ticker 75 milliseconds. That's what this means and now I'm starting the application here and now suddenly all these 2000 spaceships pop up on the screen and they're running So each one of these is an Alling process The logic is implemented in Lua Running on it. So now what we want to do is now we want to look at this through observer So let's see we go here and now I do nodes I have to connect the nodes and after this connect to the sim the simulator which is called sim at the end up there and Now I can connect What happens? Yeah, now we go so now I'm looking now my observer which is running on one Alan node is looking at introspecting on the system running on the other Alan node and this is the actual Spaceships simulator here so we can see we're actually we're running a bit difficult to see but we're running eight schedulers one per core and The system is automatically load balancing between them Now these ships are very simple For them and I can do a lot of stuff here I can for example Decide I would actually like to run on fewer cores So now I can I can modify the system now Spell this correctly So now I can say I want to run it on six cores and if we go back here and look It's now closed down to cause and move all the processes from those two schedulers over to the other six schedulers And if you notice you didn't see anything on the side while doing this and I can bump it back up again if you want to now This is pretty boring. So now we can do on some other ships as well here So I can I can set the first we make this a run ship I'll explain what this is in a moment. So I'll set the first 1000 ships to become run ships and now we'll see if you can see this hopefully you can see that some of the ships are turning green Like I'm afraid I don't really know how to make this any bigger here and the green ships have a different have a different Way of moving the original ones just went in a straight line until the edge of the universe and they bounced off The run ships here They have a different strategy. So when they come out to the edge of the universe, they start following the universe around So you can see all the the green ships coming around here and If we go back to the load here, for example We can see there was a spike in the load when I was loading in the code and Reconverting all the ship functions. But if you notice there was no time None of the ship stopped everything just kept working There was a slight increase in memory because I've got new code running in here as well That's good. I Can look at more I can look at more things here in the system as well For example, I can look at all the processes I'm not going to I can click on a process here and this is a ship process I can get information about this process and Again, the observer is running on one node and it's looking at a system running on another node That one yeah That's so this is the this the load. We're looking at here is Of the our system itself running on the schedulers So they're running in about a bit over 50% load here And this is the one down on the right is the load of the machine itself So I'm running 2000 processes each one is ticking every 75 milliseconds and it's recalculating It's trajectory what it's going to do now the thing is the logic of each of these Ships is written in lower So each so each ship is now in process and it's getting information and time of ticks and stuff like this for it We'll see if you're getting information from other ships as well But it's all the all the the logics programmed in lower So we're now running three languages at the moment So we've Alan we've got LFE which running in the shell and we've got the logic written in lower So yeah, and I can still I can change things Well, we can we can go back for example, we can go back to go have eight eight cores So I set the cores to eight here and now we know as it pops up the other two cores move processes across those using all these So all this load balancing done automatically by the system Now this is a game and it's pretty boring right and one thing with a game is you want to kill things I mean, that's really what a game is all about right killing things, right? So let's makes let's make some ships Into what I call a tack ships and we will make the last we'll make a hundred attack ships So we will make say we'll call the last 1900 to 2000 the last 100 ships will turn into a tack ships now they're gonna turn red And a tack ship moves straight ahead. We're running when it runs into a ship in front of it. It kills it It zaps it So what we're going to see here one if we're going to see explosions I am not a graphics person so an explosion looks like a tiny yellow square. Sorry, I Don't do graphics But we will see a lot of output coming in the shell here because every time a ship zaps another one It's going to write out zap and every time a ship explodes. It's going to go boom I don't have a sound interface either like so So now we can do this So there we see red ships and now you see the red ships where they're getting ships before they're killing other ships They're killing each other, too. We don't discriminate. We kill anything That's the our language We just kill happily kill everything and now you see the ships are being zapped and boomed and stuff like this as well for and If we start looking at the load now We can go up and look at the system here and now we can see instead of the 2000 about 2100 processes we have we had we only have about 1700 at the moment So we killed about 300 ships already and If we look at the load charts, we see the memory usage is going down because every time you kill a ship you're killing the process So we're killing our link processes here killing the ship for it So the memory usage is going down. We see that the load is going down for it I can now for example go back to six cores if I want to we could probably go back to four cores make it even more more adventurous and It just keeps going so now we've turned off two more cores and now we're running on four cores And this will just keep on going killing ships until basically almost all dead. So yeah, that's just a bit I want to show So any questions we have time a Quick question perhaps if anyone has anyone No It depends it it depends if one can kill the other one before the other one can kill it then then it'll die, right? Yeah, so this whole thing is running completely completely asynchronously So I do have another example for those who want to see later What happens if you start doing synchronous communications between ships you let it run for a while and everything locks So of my two thousand ships over have that one ship that's not locked everything else just stop. Yeah a question Yeah, it's not come online because I've been working occasionally to try and clean it up, but I'll try and put it on When it comes it'll be on my github So it just keeps I want to make it slightly more diverse so you can program the ships in other languages as well So it could then have some ships programmed now like some ships program LFE and some ships program in Lua And that'll be fighting each other Yeah, the load slowly decreasing for it. Yes, and interesting thing is I mean this whole thing with that the load balancing using multi-course This is baked in to the virtual machine into the beam. I am not doing anything To tell it to use the course the system doing itself. I'm controlling it So many course I wanted to use but when I change the number. It's the one that's moving processes between the courses formula It's the one that's handled handling the load balancing For example, I'm this is not something I have to worry about. I can worry about if I want to for example I'm setting them of course here if I don't the system just does automatically It will also if the load goes down far enough actually start to turn off course if the operating system allows it Trying to be energy energy conservative here So now we don't dying pretty pretty We need some we didn't wait a minute. We need some more attack ships Just sorry or just change that and we'll go don't leave over so we'll set 1500 They won't be so many left. So I'll just change these so now we make more attack ships now We'll start killing things more again, and if we look on the load here, we see it was a slight memory bump And I'll put a new code in. Okay I'm plug myself here okay, so How many people were at the Idris talk yesterday? half-ish, okay So in the in the talk we had there was an example of Writing type safe printf Yep That's better So there was an example of a of a type safe printf So Idris doesn't have printf built in but that doesn't matter. You can write it yourself So it's kind of an example of using dependent types to do metaprogramming So you have to believe me that this is an implementation But because we've got dependent types, we don't really need a unit testing framework or anything We can actually just write examples in our in our application and have it so that If it compiles then our our tests path. So here's an example We can have Say printf of hello world equals hello world and Idris will find proof and this will only compile if that's true If I come up here and delete this We get a compilation error saying that's not the same So it's actually running printf at compile time. So I could also do Example of Actually having a format string So anyone want to guess what the output of this is? so That's that's printf working and we could also write and give it Something else. This will be hello 42 and therefore just means that these things are equal so We've written types and we've proven that those types are correct those types being just that these two things are equal So we've got unit tests But by using types and by by providing proofs and our program will only compile if those tests pass So if we change the implementation our program won't compile anymore Okay, so Whenever I show the printf The first question I get is this only happens at compile time. So it doesn't work with Any strings that you get given at runtime? We can actually make it work at runtime and we do get benefits from it. We do get dependent the benefits of dependent type by Even even when we're getting a user user provided strings So let's Let's take a format and if you look at our printf it returns something of this interpret format thing which generate the type for us You don't really have to worry about that. What we're going to do is do some IO based on the printf function I'm going to get Idris to generate a heap of code for me so If we pattern match on the format We will find out that the that the printf function In the case that that our format string was empty We will find out that what we have that what we have is a function. That's just a string. So we'll just output a string, right? We can actually verify this by putting in x here and we'll get a compilation error saying that So x is a string So we actually know that x in this case because we pattern matched on it We know we've refined the information about this interpret format So we know that printf is going to give us just a string in this case So what do we want to do in the case that we get a string from printf? If we want to actually run this printf function print it okay, so that compiles. That's right In the case that we get Like we've got a percentage D in the in the string What do we know about the the function the printf function? What's the type of the printf function in the case that you got a percentage D at the beginning of the string Into something yep, so if we can you can look at this down here So it actually yep, it's it's an int to something So what we can do is get an int from somewhere so we're gonna say something like Give me an int Get an int. This actually be a string in this case. We have to we have to do a little bit to make it into a Into an int And so we know that this function this x here is a function And so we want to we want to recurse because we want to handle the rest of the full mastering so What we have to give it is The function after we've provided an integer so there's this dodgy function in in Idris good cast Which doesn't crash. It'll just if you provide something that's not Represented as an integer, so if it's not just numbers what it will come out as is zero It's a little bit dodgy, but we'll go with it Okay, so that compiles Anyone want to guess what we want to do with strings if you've got a percentage s and what do we want to do with this? What what is this function? What's the type of this function string to something yep, so Just give it the string that we give it and in any other case We actually don't care about it because we know what what's the type of um I Know this is probably not easy to guess what the type of this will be in the case that we get anything else But it's just the rest of the format string We don't actually change we don't have you don't have to provide anything to it if it's anything else other than percentage d or percentage s So we can just recurse here there you go, so This is a function that will take in the format string So the representation of the string the underlying representation of the string and do some I owe and By using dependent types we've been able to know what the type of this function is based on the format So every time we pattern match on this format We know a little bit more about this function and in each of the bodies here It's actually slightly different type the type here and the type here of this x is slightly different in each of the bodies So we can do is write a main function and run it so we need to Take the string get out of format string based on that and we can give it the printf function Interest will also infer a value for this so we can actually just Put underscore there and you just will figure out what the hell we meant. Okay, so now that we've got that Let's like we've written dependent types and that's all cool But we want to actually write a program that works so we can we can compile the program and Run it anyone want to give me a format string something a little bit interesting What is that gonna do? No? So ask for a string, okay? Yeah So um, so at runtime it's it's figured out the it's positive percentage S is like a format string going through the program that we wrote which was to To every time we see a format string ask it to give me a string And then we can run the printf function because we know in that case that it's a function from string to something And so we keep running it we keep running through the format string until we get to the end of the program And we print it out so we can do This is not just made up I can say percentage D percentage s And there you go so you can actually use like depending times We can actually do things at runtime and it's not just a theoretical thing that we can only do certain things at compile time We can actually use dependent types and get benefits at runtime Thank you. I have two things to show one is around one is specifically meant for beginners like in the morning Like I meet a lot of people who were just get started in the function programming So from my personal experience I failed twice when I started learning No surprise like that's what everybody does so but Once I find out that aha moment when I finally figure out what if everything is all about then I actually put that into Some like kind of three steps like if you know these three steps You can cover pretty much everything about functional programming. Those are the three steps. I'm going to reveal today. Okay So function programming that we first start with you know modeling the domain Like whatever domain that we are in we are we just model that with the constructs Which has been offered by the corresponding language if it is any type of functional programming language like we just use The corresponding record type discriminated union choice some type and everything put together We define the domain whereas in case of dynamic functional programming language We use hash map and all those constructs to define the the domain once we define the domain then we define Functions which operates on the domain data which we call it as lambda x we define just functions Which will operate on the data that you know created and that this function is going to operate on the data that you Have created in the previous step the third and most important is we just Define the system itself by composing all the functions which we define in the previous step those Different different operators are the the key things that most of the people get confused But the logic underlying logic is just we just we are composing together and coming out with the System so let me showcase that by showing a real-world example kind of so We are we are asked to model a chat application And I'm going to talk through one of the one use case called where we are when a message got posted We are persisting persisting that in a database. Okay, so for this particular use case. What are all the domain information? Hashttp request which models the recipe request which is being posted the message itself the error which may happen in the system in the process and a result type Which is when we are doing some operation it can be either success or it's a failure in case of success the message itself is The result in case of error. We are going to kind of know we are going to find out whether it is a which error it is going to be and The final thing is HTTP response now. We modeled all the types pertaining to this particular domain Next step is we need to define functions Which operate on this? The first thing is read request which is going to read the HTTP request and it's going to give you the message from the request In case there is any HTTP read error. We will be getting error Then another function validate which is going to take the message and going to you know Say whether the message is through the validation if it is case of any error will be populating the error field So the create in DB is another function where it takes a message It again it's going to give you a result either it can be a success again In case of success it is going to give you the message again back or in case if it is an error It's just going to return an error The final thing is write the response back to the HTTP endpoint where we take the result and put that into a HTTP response. I'm not getting into the details here I'm just defining you the functions which will you know do the operations in the pipeline Once we did all those things all we need to do is just compose Those operators are different functions. We call all the terms M M word and all those things will comes into that picture Like but don't worry about all those things basically what we are doing here is we are just changing everything together In one form or the other and then we are defining the system and at the end of the system is nothing But the HTTP the function itself is a function which takes an HTTP request and gives the HTTP response So this is the functional programming Like for a very very new be a new beginners like this is the three things that you need to identify like how do you define your domain? In the underlying language, how do you define functions which operate on your domain? Then the last pieces which is the most complex speed is come you know putting all these things together and coming out with the solution Okay, this if we crack these three things, you know, you can easily very well understand any functional programming language Okay, this is around for the beginners and Since there are a lot of in the even the previous demo that we are we have seen the the power of types Like you can point to any data source and get something in F sharp We have a concept called type providers where you can point to any data source and you can you know get the strongly date type access and I'm going to showcase some of the examples so There is a type provider for Jason like I'm just pointing this Jason provider to is it visible? Okay, I'm just pointing this guy GitHub type to the remote data source where I'm just pointing to the GitHub API So I'm just executing this now. I have GitHub API type in my system. So if I get The sample out of it For all the HTTP. This is the sample HTTP response Which is written being written by the GitHub API for all these things. I will get intelligence see here Emails URL emojis you are what these types are automatically created from the response I didn't do anything the library has taken care of all this compiler magic behind you just point to the data source it will create the types for you for free and This is like we can extend this to point to different data source also like fine for example I'm just pointing to a different endpoint where we are getting the user information. So I'm just Creating that type now this type can you know fetch user information From the GitHub in a type safe manner. So I'm just have created defining a function where You can you just give the username it will create the particular strongly based type for this function so now I'm creating a Type for a don't say who is the creator of your shop. So his GitHub and this design So I'm just executing that Okay, I didn't executed this. So now I have this function Now it got so I can get his company name He's working for Microsoft here also, I'm just Getting all these things his bio all this information. I'm just pointing to a remote data source From the Jason response. I'm getting all the types for me and these types for example data, but I guess Okay, this is intelligent enough to track if it is a integer It will give you a integer type if it is a daytime It will automatically convert to a dead type if it is a GUID again. It will create all these things exactly Exactly so here the for the very first time when you are providing this right, this is compile time So you cannot do this in this is runtime. I Created the types in my compile time. I'm Executing I'm using those types in the runtime. Yes, that's where you need to when you are creating the type You need to point to the right schema Like you in the when you are providing that this URL what you are telling to the compiler is hey This is my data data, you know resource URL go fetch the URL and produce the types And then you are you know the type system is available then Yes, if the Jason response has changed Yes, but it If it fails that could write like this and that means the system itself is it has to upgrade to talk to the new version of the API So that's a thing that we are getting here Yeah, the most API will have the versions. Yeah, that's another thing when the when the version is free the response won't get changed and we can extend this to any Data which has a fixed schema like Jason. I'm using a CSV provider. I'm having a data here And it just shows ID name agent email. So I'm just populating. I'm just pointing to this data source Now I have the CSV provider with me. I'm just getting a sample out of it. If I want to print all the age I'm just using this function. I'm just printing it so it Which sequence? Yeah, now it print everything twenty eight seventeen and thirty When that is oh, okay. So what the very first time you need So that will the two for the type generation for the subsequent iteration. You don't need that actually one Yeah, it's it's at the compiling time. It maintains its own like it said compiler magic is happening in the beginning So this one again. This is also type safe if I change age or maybe age To different field say age to I'm getting a compiler error here Say the age has been changed. So I need to put age to even I will get interested also age to So this is the yep. Yeah, there are two ways if you are very sure that this Like I pointed out if there is any fixed version that the under this version this Jason response is not going to change at all You can point to a version or there is another option where you can create a sample response and put that in as part of your code And refer that sample response in your flight system and then use that in your subsequent building Okay, and this can extended to My like you can think like how many of you will write a regular expression without referring to any sort of Google It's very hard. So what we have come out in F shop is like we have a rejects type provider all you need is just feed the Regular expression into this rejects type provider So this is just to parse the The phone number here. I'm when I'm parsing the phone number. I'm just assigning some labels The first three digits pointing to the area code the last Like three and like some last seven digits are phone numbers. So I'm just using the type match I'm feeding all the values and I'm putting dot They put phone number that value It will give me the phone numbers, which is the second part if I put not Area code again. See you see the intelligence is coming up. So I will get the area code say if I change area code to say Code I will get a compiler error So basically if the underlying data schema is fixed it you can you can create type providers for that Here this fixer is only it's going to be the code and the phone number You just if the schema is fixed that I'm just feeding that to the type provider The type providers creates a type for me. I can access this in a strongly type That's it. Thank you so I thought I would just do a quick demo quick check for you What I'm going to show you is not the original Haskell version and that come class and I wrote a paper about a long time ago but rather It's a little okay So rather I'm going to show you the version in Erlang that our company so Now we can see that right? okay, so Suppose you want to test the reverse function on lists really simple example Here is how you might write a conventional test In Erlang so this just calls reverse on a sample input and then matches the result against the expected answer Okay, so this is this is how the vast majority of tests are written in Industry right where you you perform a sequence of calls and then you check that you get the expected result and if I Go to the Erlang shell now Okay, can you see that? Yes Let me just compile that module and then I will run My test and yes it passed so What I'm doing here is matching the results against the expected value if it doesn't match for example if I would put a 2 here That'll cause a mismatch then We get an exception so my test before I screwed it up passed So that's all very well except that you have to think of the test cases you want to run and that's not hugely interesting So instead what quick check let us do is write one test Except that we usually have to express the test differently Right, so it's going to be difficult for me in general to predict the result of reverse without rewriting the function I don't want to do that So what we do with quick check is think of some general property of the function Like the fact that if we take any list of integers and reverse it Twice we end up with the original list. So this equals. It's just going to make a comparison between this and this Okay, and now if I start quick check Then I'll run that and I'll just give it that property as an argument. It's just an Erlang function definition and Sure enough now it's passed around a hundred tests if that's not enough for you After all a hundred tests don't necessarily mean that it's true But ten thousand tests do don't they? so Now we have a lot of confidence that's really really holds and if we take a property that doesn't hold Like this one that just says that if you reverse a list you get back the original list Then as I showed you before I run this one By not working at all. What do you mean? That's true, but then then this wrong property would pass, wouldn't it? Let's see if it does No Okay, and here you see this is the randomly generated list fail the test And you get some output as well from the equal saying look they weren't equal and after shrinking we get the simplest case So that's a very simple example demonstrating this behavior, but I thought I want to show you something a bit more Substantial So I have implemented a very simple key value store So let me just demonstrate it to you show you that it works This is Erlang so often that I have to start things start the key value store Let's insert a key of one with a value of a and a key of two with a value of B Okay, now Of course, I should be able to look up those values So let's look at one It's a Let's look up to it's B and the other thing that we can do is Delete things so let's delete the key one Okay, and now if I look at one It's gone. So Erlang but this code just returns false in that case and if I look up to It's still there So as you can see the code works Now something else, which I can't show you in this time is I've built a specification In Erlang a quick check specification that specifies how this should behave But what I can do is run those tests for you so let me Think that's what I called it. Yes Let me test the property There we are passes a hundred tests and Here I'm getting a distribution of which commands appeared in the test So you can see we tested insert about a third of the time look up about a third of the time and delete about a third of the time The start and stop are outside the things that I'm putting statistics on So that's cool, but maybe I should run a few more tests Let's test for ten seconds. Oh, it doesn't always work Now you can see Up here This is the first test that failed randomly generated it's actually 77 calls long and Something went wrong in this test case a Post-condition failed this last lookup returned false when it should have returned a now Here you can see both the strength and the weakness of random testing The strength is that we found a bug that was certainly not obvious from an initial quick test And the weakness is that we've got something to debug that is 77 calls long nobody wants to do that hence Quick check shrinking after shrinking we get this test Which does five insertions a deletion and a lookup that returns the wrong answer Okay, our five insertions necessary. Let's run it again. Okay This test also does five insertions a delete and look up it's a different different data, but Of course the initial random test was very different, but after shrinking. It's surprisingly similar. Let's run it again Okay, this this time we sometimes get this we needed six insertions a delete and lookup and five insertions a delete and lookup so this five insertion case this really is the smallest kind of example that fails and So now you've seen shrinking in action and producing something much more interesting Can you imagine writing a test by hand for your test suite that would look like this? I think not So we might ask how often does this property actually fail and I've got something called How often does it fail? Which just lets me measure that so I'll run ten seconds of tests and now I made a very slight change to my code and In a moment it should tell us here. We are this property passes 99.54 percent of the time So it's less than half a percent of the time that actually fails you need to run 200 to 250 tests on average to find a failing case So that's how hard to find this bug is And what is the bug? Well, I've also defined something that will just rerun the last test But show us what's happening. So it's a key value store. I said, but how does it work? It uses order binary trees internally so Here are the trees that we constructed and this every time I change the tree. I'm drawing the tree before and after So for that last test case that failed. This was the tree. I Wonder if I can make this big so this is the tree that was constructed with five insertions and That's the tree after doing a deletion and I think the value we deleted was 36 at the root So when you delete the root for binary tree, it's a bit tricky Actually, you have to move some other value up to the root and this has moved zero up And then it has to put the 34 somewhere and it's put it Here to the right of 37 That's why look up doesn't find it Okay, so this breaks the invariant on order binary trees and so look up doesn't work So this is quite a complex bug deletion is hard to get right. We could now look at the code. We won't But you can see that quick checkers just immediately put in front in front of us a mineral example that provokes the bug and Then we can take that test case and analyze it and it's really not that hard to find the problem Which could have been very hard to find without that help so That is Okay, that's pretty much them. I wanted to show you of course all of these were still small programs So you might wonder how well does the approach scale? I want to just quickly tell you about the largest project we've done which was for Volvo cars and Involved testing out of our software. This is software that goes that that runs on every processor In the car we were testing the so-called basic software. It's like the OS and This was a project in which cubic our company collaborated with Volvo cars and With them the Swedish standards checking body So they certified that things meet standards and with our help they certified that various implementations met the outer star standard for the basic software We read 3,000 pages of specifications. The standard is long We turned that into 20,000 lines of quick check code. So that that's how much formal content there was there We tested more than a million lines of C code from six different suppliers and we found more than 200 problems of which more than 100 were problems in the standard itself and On those occasions when there was some comparable test code to compare against ours was 10 times smaller so The point of this is to answer the question does quick check scale Yes, it does So that's it if you'd like to know more Then I will be running a workshop on Sunday that will go through the tool that I demonstrated today and Showing in a lot more detail how to build quick check specifications that can do this kind of thing Okay, thank you So this is all in the wrong order because my next talk in this same room I'm going to be introducing APL and now I'm going to be showing you some code instead to begin with But anyway, the what I'm going to show you is something that came out of the APL workshop that we did last year Where at the end I said, you know, we've looked at the language as anybody have something they'd like to try and you know I have me solve live and my rash had this data That was the the JSON file containing the schedule for last year's Conference so you can see there I've I've loaded some code on the first line there And then I've done a quad n get we call it where I read all the content of this text file into a variable And then I've done a 10 100 reshape of input So we have the first thousand characters in that file formatted into 10 rows I made a 10 row matrix just so we could see what's there So you can see it's you know, Jason deeply nested Jason showing the What was happening that year and then I think the challenge was you know format this in some kind of Visible representation, so I wrote a function which did this So you call it with the left argument Which is the day of the conference you want to look at and that Jason on the right-hand side And then it returned actually a four column matrix APL array where the first column was the starting times and then what's happening in the three slots There were three slots last year So what I'm going to try and do now is trace this function, so I'm going to go back up to that line and hit control enter And that brings up our tracer So we can see the code down here And I'm going to single step through it and then try and Show you some of the intermediate data just to give you an idea of what it's like a day in the life of an APL programmer I think this took about an hour from start to finish when we got the data until we had the finished program Okay, so my I have a function here called format which I've called the right argument Inside the function is called Omega the last argument. So here I'm saying from Jason Omega Dot into that and get the the object within that called conchedual Sessions indexed by the left argument, which was the day so I'm extracting a piece of of information out of that So just type shed here We don't sort of have a variable watch window because in APL it's more common to just type expressions So you can use the APL language not just to see the value of something but to to execute expressions on it And see what's inside. So this is just a Jason object. That's not very interesting Now if we look at the first one of these and then I call something called nameless class 2 so give me all the The variables in there. Why didn't that work? So that's all That's just going pear shaped Anyway, I was just gonna look at the first element there I can't figure out exactly what's gone wrong in the demo here But I have I've written a little helper function here this curly bracket thing is to extract All the properties of a given name from what is an array of of the the schedule slots So here I create Five variables will take a look at one of them time slot for example That's all the times for the the slots that were scheduled in that in that session I've got something called track Which is this list of strings track one track two or a star if it went all the way across And I'd like to translate that into something numeric. So here I say look up Track one in the list track one track two track three this IOTA symbol here says take this data and do a look up Returning the index of each item found and then the for residue here is going to convert anything Which isn't in that list into a zero So that line of code gives me Now my variable track looks like this. I've got a zero if it spans all the all the tracks Otherwise, I have a track number Now I'm interested in doing a little bit of work on the times so I get I convert the time slot variables into numbers Using minus or hyphen and colon as separators field separators And then I extract the fourth and the fifth column of the result that I get out of that So if I just hover over HHMM here, you can see it's a two-column matrix for the hours and minutes and Then I think this is actually unnecessary, but I thought at the time I wanted to do some arithmetic on it So I convert that using base value representation. I say interpret. This is a number base 2460 and give me the number of minutes since midnight, so I now have Numbers since minutes since midnight and then I take the unique values of that and Sort it so that I have the sorted start times Minutes since midnight and Now I want to initialize my output matrix. So I take the number of starts that I have and the max reduce of the Track number Plus the span which is how many tracks a session Spanned because I think there were some that spanned to where two Tracks were combined into one room so output now is a Three-column matrix. You can't really see that here. It says in very small letters here It's 21. We had 21 different start times and three was the maximum slot number Look up the start of each session in the list the sorted list of starts and that gives me the row number where I'm going to insert some data I compute the column by taking the Maximum of one and the track number because I don't want to get an index error by trying to index with zero and Then I have the span here where I do iota each which generates the numbers one to N for each one of those and Then I add those together. So I end up with something called Col if we look at row We can see that there That the the target row and column that each one of the slots is now going to into go into I Pre-initialized the matrix by inserting Left arrow ASCII art into the matrix for the sessions that span More than one so we're sort of starting where they start and then plus however far they go I've actually put a bit too many of these in but it doesn't matter because I'm going to overwrite them here By indexing into the output with the row numbers and the track numbers the 20 fold each of The titles which were also in one of the fields and now we can see we're starting to get the end result here So we have a 21 by 3 matrix Finally I want to prefix them with the times and I do a minus 8 take of each of the time So that gives me if you remember what that looks like. It's ISO formatted date So I take the last 8 and then I do a negative 3 drop of each of those so I lose the minutes So times becomes this thing here and then the result of my function is the times catenated with that matrix and we're done and So that took about an hour to Develop and there are no loops in that. I think you'll agree. It's a very different way of thinking about data But once you get used to it, it's it's very very highly productive way of doing software development Yeah, so that was that demo any questions about that Otherwise I have a video I can show you of an APL application in action I think those of you who were here two years ago may have seen it already Should I do that or or any question? Yes What's it used for? That's a good question the the people who pay money to use it significant money to use it are nearly all doing financial applications a typical APL application is A portfolio management system So doing risk analysis and reporting on large collections of financial instruments and the big advantage there is that and this code May look strange to someone who has a background in computer science But to somebody who's coming from a mathematical or engineering background and has learned math And we'll look more at that after the break This is actually a more natural way of sort of translating mathematical problem statements in into programming So most of the people who use APL are Not software engineers there. So I think I said two years ago any kind of engineer other than a software engineer It's a little bit weird because the typical APL programmer I think is a physicist or a mathematician who got hired by a company doing financial software and retrained to To work on finance rather than physics, you know, the I think the algorithms in in a lot of physics are identical to To finance they're pretending that you can use the same algorithms to describe what happens in the financial markets as you get No energy loss and stuff like that. Yeah No, I mean, there's there's a derivative of APL called K KX KDB Which is a reduced instruction set APL which comes with a very high performance inverted column column or database That's used for real-time trading Our APL system is used for you know Systems that need high performance this kind of software scales very well You can deal with very very large quantities of data because there are no loops even though it's interpreted I'm this same code would work on a schedule with you know hundreds of thousands of items and That allows our interpreter to to do a very good job of it because it's not going around interpreting I think the the highest performance system we have is curiously in Sweden also it's a Patient journal system, so I think about half of Sweden's hospitals by patients Count are using a system written in APL For the the real time, you know in the hospital in the emergency ward and the reason that system works so well is again It was written by a nurse Who had a friend who could do a bit of programming and they approached it from the user perspective? Rather than starting with a star schema database decomposition So they have a system with sub-second response times rather than you know taking hours just to log in which I think is typical for hospital systems Yeah Yeah, I yeah, I should stop both Roger and I are doing talks later, and there's a full-day workshop tomorrow You want to see the video? So this this is not a typical APL application I have to admit But it's a lot of fun it's written by a Finnish user of ours who Got hold of the EU released public data about mapping made it free And then they had a competition for writing applications in the public domain Well, no the data is in the public domain. You have to pay to use this. It's a game essentially So he wrote this game in APL And we need do we have sound Our user meeting last week Celebrating 50 years of APL. We had a 3d motion engine a simulator hooked up to this game in the Out you know outside in the break area So people were riding this thing in a speedboat doing 50 knots He claims to have the best wave simulations of any any game any maritime game Working with a professor at the Sun University of Helsinki to get Baltic waves exactly right Now of course that wasn't entirely written in APL the shader is running on the graphics card But at the wave, you know all of the all of the physics was done in APL just the renderings Done on the GPU, but that that's not a typical APL application and also do that. All right I'm not so confident about the whole blow your mind type of goal, but I'm gonna try right So how many people have just have tried our line? How many people have Not used it in a real project because it doesn't have the libraries you need Okay, so most I've been doing it for about ten years I got exposed to around ten years ago and most projects where I've ended up not using Erlang at least in the initial days were because I needed stuff that was available in other languages very very easily and Had to be written from scratch In our line and I've definitely come across a bunch of people with that problem There was somebody a gentleman yesterday giving a talk with exactly this And this issue so my goal here is to just show a trivial way of Literally bringing in other languages into the Erlang system, right? And the way it works is is this So in in Erlang there's this construct called a port And it allows you to essentially spawn a sub process an OS level sub process that You have control over and can because you're spawning it you have control over the standard input standard output But you also have control over, you know other file descriptors or typical approaches that you Choose file descriptor 3 as input to that external program and you choose file before as output from their external program, right? So now with very little code and that it's about a hundred lines of total code, right? If you put that in place then you get access to all sorts of libraries and all sorts of languages and In some cases it's worth to trade off in some cases. It's not All right, so if it's if something's in your critical application performance path Then you probably don't want to jump off to another runtime and come back But then there are several use cases where The developer productivity is worth doing that, right? So if you can if you can get all the nice distributed systems concurrent Programming benefits of Erlang in most of your application and whenever you need to jump off to something and still save time and develop the whole thing It's kind of nice, right? So this is the idea You can do this for any number of languages. I'm going to show it in ruby And the way it works is there's a There's an Erlang part written an Erlang process that is managing or Controlling the external program the external program also needs a little module that's interfacing with the Erlang part, right? and This There is a predefined program protocol in Erlang around how you should do this so The the pre definition in that protocol is that if you open such a program and Then you can send it messages in two ways you can send it messages as packets or you can send it messages as a stream and the approach I found successful I found to be successful for this kind of setup is Is packets and when you send those packets you can say that every packet is Prepended with the length of whatever your messages, right and in my case right now the messages are going to be simple RPC calls right call this procedure for me and give me the return value back And I'm going to block on those calls. All right to start with at least But what I will do though is I will do this in the JVM and I'm running the J to be jar So I can run Ruby inside the JVM just because I don't like the verbosity of Java Alright, so so let's let's do a little example and that should you know get us started That takes a little bit of a second any questions about the general idea so far jar compilation can okay, so the example I'm going to do is Let's start with something as simple as Math So in this scenario what I'm doing is first just starting a little port program Which does my math for me, sorry, which sets up that interface for me So what's going on here is I started I Started a Java jar which in turn has my little program right that I wrote and So it started now and the next thing I'm going to do is Do a simple math Example right, so I'm adding two numbers and The math is happening inside Ruby so I'm invoking the program from our lang the math's happening inside Ruby And I get the result back Never do this in a real system right there's no reason to But it was it's a decent example is sort of get things started So okay, it worked right and I could repeat it with other numbers and it would still sort of do the trick right so Once you've written those hundred reusable lines that I were talking about Then doing this for any number of use cases looks just like this right so So there's some little code going on in the background that's enabling this but The function I wanted to expose was add I Wrote a little module called my math port. I included a mix in in Ruby that I've already written and I'll show you in just a second And I wrote a little function that allowed me to add Numbers together right So this worked right But let's make it a little bit useful things that you can't do in our lang or you would have to write libraries to do in our lang and There are libraries available. So let's try and use some of those. So let's maybe try PDF generation right so there's a Ruby library called prawn which allows you to generate PDF documents, so let's try doing doing that so My example Is going to look very very similar All right, so in this case I'm now instead of starting my math port I'm starting a PDF port, which is it just that other program that you just saw and now instead of Instead of calling the math command, I'm going to call a PDF port command. I think my function was called generate Let's say I want to give it a file name And I want to put some messages in it. All right, so first time you run this it may take a little while More than usual because this is when the new process is being spawned But after that, it's pretty fast so I can do this repeatedly with different values and do that pretty fast Let's just quickly see if it actually worked All right, so this is my hello.pdf just so you guys know that I'm not tricking you I'll do this again There's my hello.pdf. Let me just open it right and there's a fun con So that's kind of cool, right? So Now I can if Erlang has a PDF library But if you go look at it, you don't get a lot of confidence It was last updated like a few years ago Nobody's maintaining it on the other hand the Ruby libraries being actively maintained being used in a lot of production projects You get you know a little bit of confidence about about using it and it's a useful tool when you're trying to convince team members were You know who make that argument of well, why are we going to the system where we have to write a whole bunch of libraries, right? So let's take a let's take the example Sort of in line with the goal of trying to blow your mind. Let's let's get a little bit crazy this scenario So so because I'm calling stuff inside the JVM the JVM can run so many different types of programming languages, right? So I'm going to pull one in with the Rhino runtime that runs JavaScript And my goal here is that I have some data in Erlang And my user has given me some JavaScript to run in the context of that data, right? And now to do this in Erlang I would have to write a full JavaScript interpreter, right and Or I give up on Erlang and I say okay I'm going to just live in Java for for this particular setup, right? Either of those scenarios But this is middle ground and it's most of the time acceptable. There are some There are some performance trade-offs in terms of going to another runtime But in scenarios like this one where your goal is to do something sort of external to the core of the system or Well, let me not say that I am making assumptions But but in my case it was a it was a peripheral entity that didn't it wasn't time critical it wasn't critical that in a Certain time window this should get executed, right? So that's an area makes sense. So what I'm going to do now is is execute a JS port that I've written right and I'll show you the code in just a second But this time I'm gonna start a JS port program and this time and I will run a So this this line of code is interesting. So what I'm doing here is Okay, the the piece of JS code is a into two, right? What is a is data from data that the Erlang world knows about right? So somehow I know that a is hundred and I need to send that context over to Ruby And then over from there Down to the Java world and then from there over to JavaScript right and And then run that code in that context. So let's try it. Let's see if that works and I will show you how Sorry, I keep switching right so it did seem to work All right, so what what did it take right to pull this off? So let's look at the port programs. We looked at the math program already was very simple as this is just dependency type things, right? the key here is this mix in which is bringing in that behavior and that mix in defines this start function, which what it does is So remember the Erlang program Started writing stuff on file descriptor three and reading output from file descriptor four and what the start method will do is it will Interact with those two file descriptors set them open open a pipe and then start dealing with them, right? So So that's what's going on underneath. Let's look at our PDF example Again same idea I mix in the RPC port Mix in and then I just define it a little function I had another example here to draw a little graphic, but you know, I forgot about it and Then this is the JavaScript code right and what's nice about this is that it literally is You know 15 lines of code that I can set up and now I have a javascript runtime I can occasionally call out of my program when I need to right and Of course the hard part is done by the folks that developed Rhino, but it's nice to use right So what what does it so that's what our programs will look like and you can end up writing a whole bunch or a library of these if you need to So what does it take on the ruby side it needs a It needs a little mix in the way you do mix in the ruby is that you you define this module class methods Thing and you define a function which is invoked every time the include instruction is called inside a module right and so what that does is it sort of mix in mixes in every method in that module right and Remember we call the start method the start method is essentially opening in IO on file descriptor 3 and opening in IO on file descriptor 4 making sure they're in sync If it receives a sig pipe I stop stuff and the reason for this is that that's what Erlang will do If it's dying right so if it's dying it will send a sig pipe over to that process it's controlling right and Then I just loop over And this this stuff is just sort of so at the two ends I one end serialize my RPC call the other end deserialize it and You can use whatever protocol you want to do to do the serialize these deserialization I found message pack useful because it's available across languages and seems to work most of the time unlike a lot of other alternatives But you can use anything you want so this is about you know 60 lines of code on the Ruby side And if you switch over to the Erlang end of this It's another 60 70 lines of code and what it is is a little process That is doing It's following certain standards Erlang has defined around monitoring this little port program So our when stuff goes wrong You'll know because you'll get an exit message or you'll get an exit status message right those kinds of things are handled here down here the crux so that RPC call is over here where I send a message over to So whatever data I have I pack it with my serializer and I send the message over to my To my port right over here and and then I sit and wait for a response until I get a response So which is why I said I block right? I don't have to block right and that's my last statement and I'll leave you at that So a more complex version of this could essentially set up Both these parts Both these parts as routers right so JVM has actor libraries Erlang's already processed different driven. So you can spawn a little actor manager Which based on whatever message that comes in from the Erlang and routes to the JVM routers right or JVM actors And the JVM actors can respond back to a central Person which is again the port program and that program When the message is received on the Erlang side you route based on some kind of addressing standard or address standard Right, but once you have that set up you essentially have a slave JVM That you can do whatever you want in right database connection handling. So we use it At our company to do connections to Haroop for example So we do Haroop jobs from time to time and Haroop libraries are mammoth. You don't want to reinvent that wheel And it's nice to be able to go do that on the JVM and come back and be in the Erlang work. And so that's That's My little presentation. I hope it's useful And gives you some value in arguing that point about there are no libraries. Thank you If you're not that okay Only a couple so that's great because there are a couple of demos which I'm going to run and first one is going to be replay of what he showed in the Alang user conference last month Which was kind of it blew me away. I'm not committing that it's going to be the same impact But it was kind of very cool what Alan could do I mean, although it's it's obvious what it can do But from a user point of view, there are so many things which are Which are available, but needs to be connected and only once the pieces come together. It's kind of looks awesome so Hope the net connection holds up, but Yeah, I'm going to Part of the screen is cut off. I don't know why So what we have is a kind of download stuff. Let's see if I can make the net happen But I have one pre-recorded session. I can run through it if it doesn't work but the reba 3 is a very cool tool for Building stuff and packaging for our line You should definitely try it if you had a longer or if you want to try a lang after this demo So reba 3 is this tool along is already installed on my system What do you need is a What is that? I hope doesn't get too big So what what I have here is a conflict a simple conflict which this talk about the plug-in. This is a copy of Fred's Work which he had done and I kind of asked him after the demo for my presentation over there But we said, okay, he gave me all the permissions. I believe we should be okay me using here as well so this this this is the plug-in basically what it actually indicates is You can you have a plug-in system where you can have templates and you can import templates which which for convenience basically Which I'm going to use so there are only two two files in this folder. One is the reba 3 application Or the script which which will help us in building stuff and using the plug-in. The other is the basic reba 3 configuration so let's Try this so let me Make a new crowd project Yeah, it's it's what is that? If I move my window, I will not be able to see on my screen. That's oh, okay. Okay. Is it better? Is it okay? So I just created a crowd project. I'm going to Create a handler You can add handler Say some fruit Doesn't matter Let's create another one Let's stitch up the New handlers into the router Let's build it That's when it's checking out all the dependencies automatically based on the template it downloaded Which I'm going to show you in a while. What what are the dependencies for this project? so there are numbers wave ways you can actually define the Dependencies one is you get up with the other is local file repo. There are other you can do any kind of one version control Primarily this this tool is pretty smart to go to a whole new level of packaging and releasing Also for embedded kind of deployment which I made one of project Unicernel for I use the ramp on Unicernel for packaging a lang making a lang microkernel So rebar 3 was again helping me in packaging which I'm going to show you in part in part because that's a big demo So I don't want to show the whole part, but just of it using elixir in the background But yeah, so it's it's getting all the dependencies building it So there's nothing Okay, so there is some C part for optimization, but you could use alternative libraries for the non-f portions of the demo So we have the project ready now So let's just run this It's up and running. Let me just pull up browser 8001 So it is the familiar swagger that you have automatically for the resources that we created So as you can see The bunch of them you can make queries Which say you want to get I Don't know if you can okay, let me pull that screen down so that you can see them side by side So the response there's no four of four because we didn't create one. So let's let's go ahead and add something there Okay, the content is created We can do a get again as you can see the screen below you are seeing whatever the action is happening live action in In the logs for cowboy. So now you get the response for body I mean like the similar thing you can delete delete the resource as well, which is simply just Yep, so the content is deleted Now I try to get it again No content Yeah, great. So So this is pretty cool, but So why stop here? I mean like you can You can do multiple stuff and one of them being running tests which is Definitely Very important. So I didn't write any test. So let's see what happens by the way behind the hoods if you were to check out this Repo you will find out the when the project is created. So it's using proper which kind of based on quick check It's using the quick check We have property is testing just that it that is open source So but everything comes so like you can see there are the tests are running and So if I were to open why shock you could see that there are some messages going back and forth in the local host And see that in action, but yeah, I don't have my shot installed Okay, so without even writing a single line of test is able to get 100% in routers and even as you can see the various others up or like you can see the hundred percent model coverage Which is kind of neat. So for any kind of rest deployment This is kind of really awesome and you get everything and if this doesn't stop it here you can actually go further and look at Matrices, which is another very important piece of Oh, I'm I'm not running the So there's none because I just I stopped it earlier. So let's let's do some stuff again See how that gets updated So you keep creating some content So I have something here and you can look at what happened Yeah, so this is one and if you Do it again It changes I kind of need you Suppose something with orange It means it dynamically expands. So this is pretty much anything then you have statistics It's using Folsom library in case you're wondering what okay Okay, so The whole so by default the system is actually getting giving a much more matrices than you actually need But yeah, you can choose to disable some of them if you want to but this is kind of very critical in a production system, so you can There's one more I'm forgetting that you're Okay system and memory It's kind of kind of lot more verbose and capturing a whole lot, but This would be better So these are along memories and processes for that matter the how much memory processes are taking the the amount of memory used by atoms and code so if you want to just pull it and Capture it in your measurement tool and add you to anything for that matter So I had more stuff to show but unfortunately we are running out of time So if you want you can catch me Okay, maybe a last bit which I probably already have set up which is kind of Micro-colonel stuff, okay fine. Yeah Yeah, okay fine that that's okay So I just wanted to show an elixir demo which was using the line micro-colonel, but it's going to take a while So we can do it later on I hope will blow your mind and like a glimpse of the internals of the language So the first factoid is that Julia is secretly a list That's that list You've got a list prompt. It's called Pempto list It it is what the parser for Julia is written in So we can invoke the part So that's a Julia expression passed in a flist And as you can see the expression is secretly a S expression so This is the Julia Apple If I do one plus two But if I don't want to evaluate it, I can coat it with this intact And it returns me an object which represents the expression one plus two And it has a head which is called and then arcs Which is plus one two plus is the function being called and the rest of the arcs so Since we can represent Julia code inside Julia, we can obviously have hygienic macros I'm going to show you one of them That I just made up I mean that I usually show Because it's easy to understand so What's happening here is this macro is taking an expression and Keeping the first argument as it is and reversing the red Right, and it's also keeping the head so The way you call a macro in Julia is with the act prefix Because it's kind of nice to have the separation between when you're calling a macro and a function so What should be the result of this? It should be any guesses flip to minus One yeah, that's correct. So what it's basically doing is it's a flipping the arguments two and three And passing it to one the passing it to minus so we can see what it expands to we using this function called macro expand so we can see it's three minus two So that's how Expressions are represented inside Julia And the next thing I wanted to show is how does one plus two point zero to make things figure actually work, right? So if I look at the type of one, it's an integer If I look at the type of 2.0, it's a float 64 In most languages the language spec itself would define a way to promote these numbers to float 64 and then add them But in Julia, this is actually defined As part of the standard library and I can look at how it actually works So I can I have this nice macro called at edit to which if I give any expression It's going to take me to the method that is invoked when I call one plus two One plus two point. Oh, that's what I wanted to call. So this is The method that gets invoked for an abstract type to two number types for which a more specific plus method is not defined so it just says that Promote the arguments and then do plus on it There's actually a nicer way to step through code In Julia using this package called gallium It's the debugger So I can say enter one plus two point zero and It will take me to that line It takes a couple of seconds in the beginning because it's compiling everything So that's the line. Let's step inside Again the conversion mechanism is just a generic function called convert which takes the type which is the promoted type of T and S and then Value and then converts it to the promoted type. So that's how promote works Let's step in actually will not step in further But I want to show you that there are many steps involved in that getting done, right? But the Julius Julius just in time compiler. It's smart enough to remove Remove most of the abstractions That that were created to just make the promotion mechanism work So if I say at code underscore LLVM, it's going to show me the LLVM IR. It's going to execute So if I if you see here, it's just a fired instruction Which is the fastest it can do so I can Also see the Coordinated Expression so Julia sort of allows you to dive down into your code to Down the rabbit hole and actually look at what instructions get executed So another cool demo I had Is okay. I'll just take like two minutes on it's an APL in Alright. Yeah, so this one's not I guess it's a APL interpreter in Julia, so we have string macros So you say APL and pass it a string and it's going to evaluate that using the APL interpreter and Nice thing about this is that it just generates an AST and then you can call that with Julia values like this and If you look at the code LLVM of that, that's also as good as just doing it in Julia and There is this Compute operator in APL so I can say minus commute and it becomes flip of minus so even if I do minus Commute it's going to generate code which is aggressively in lined in the sense that Over here if you see it's percentage zero percentage one here It's percentage one and percentage zero and it's not doing any Any more allocation to do the flipping and and the and this Inlining applies till like six levels of the flip operator. We thought me having to do anything in the interpreter itself Yeah, so I guess that's my demo. Thanks