 The third thing that is strange about our conference never heard of keynote that to Robert studying early, but we can do that All right, so it's actually a real honor to have Robert second year in a row He was here last year took the train from Mumbai to Bangalore was there enough to do that and survive He's right someone said take first-class air condition. I did that was fantastic. So yes, so I can recommend it seriously So basically he survived and this year he's back again We'll see where he goes, but anyway without much to you, I'm gonna get Robert up. Okay. Thank you So Yeah, thank you. So yes, so I'm Robert Verding. I was one of the three original Alang developers together with Joe Armstrong and Mike Williams So it's it's one third my fault And if you watched Alang the movie, which I'm not gonna show you if you watch that it's Joe and Mike who are talking I fix all the bugs So, yeah Yes If you're gonna watch that movie watch the sequel That's that's better. You have to have seen the first one to understand the sequel, but the sequel is much better So yeah, so I'm gonna talk about something completely. Well, not quite It's a pilgrim's progress to promised land. This is what this is what it's all about Yes, it does okay, so what's this all about so well You're here You've heard the prophets and their followers extoll extolling all the benefits and virtues of the Alango Alexia and it's the true path the promised land of concurrent systems Look even more land of concurrent fault tolerant scalable systems Yeah, no more buzzwords here at the moment, but there are other bugs as well And you decide to try and make this make this pilgrimage What is going to happen is you're going to hit a number of different issues We're gonna talk about the various stages you get to You're gonna hit these issues and the thing to make sure is I don't want to make they should not become problems There will be issues you will hit them how to make sure that when you do hit them they do not become problems So this is what this talk is a bit about and the first thing to ask yourself, of course is Are you following the right prophets? I mean a lot of prophets out there, but are you following the right prophets? We're not saying here that they've been lying to you because they probably haven't been lying to you But is are you listening to the right ones? It's just really where you want to go So yeah, have you chosen the right path is our language here the right language and system for you So we might be talking about languages here, but it's actually they're both cases They're a language together with the system you cannot just get the language You can but you don't want to the language you get the system itself the way of building things etc like this So is this really where you want to go? I Am not here going to get into discussions which one is better our language here Take your pick choose it. I prefer LFE this flavored Alan how you cannot write things with parentheses I can never understand, but that's the absolute the best one for it But I'm not going to get into that discussion and in many ways It in many ways. It doesn't make it doesn't matter So so in some cases you'll end up in the same place in some cases Different languages have different features which might be very interesting for you, but that's I'm not going to worry about that So the question is is this the right thing for you? What's it all about? So what we're going to do now is it took a little bit a little history I'm not going to make this very long to try and try and explain why do they look like they do I Mean there are awful lot of languages out there with different backgrounds and different ideas Why do these languages look like they do and the other thing was here that? When we were developing our language originally, okay, we came from Ericsson, which is telecoms company and we were looking at better ways We thought better ways at least of implementing telecom systems and one of the things we realized is that Controlling the switch is not the problem Because there was a switch there that there was already software for controlling the switch You would send the switch a few few signals and to and to Two users would then start talking with each other That wasn't a problem. We'll send them if you I'd send a few more commands to it and the phone would start ringing Something like this. That was not the problem These were the type of things that were the actual problem for designing these type of systems You have things like very large number of concurrent activities Yeah, well looking at some of the more interesting and others these 12 points come from a thesis beyond a Decker Who was at the boss of the lab wrote later? About these things and there were these 10 features here for you described as fundamental to this type of application So the some things were more important than others They're the ones I've marked in red or more interesting than others directly from the outside You have things like a very large number of concurrent activities That was fundamental to the problem You might have a large switch with hundreds of thousands of connections You might be running tens of thousands of calls at the same time plus everything the switch is doing And you have you have all these things going on at the same time to have an awful lot of concurrent activities We have timing Timing constraints things have to happen at a certain time or within a certain time We have things like distribution I Think French I don't know French and Francesco mentioned it this morning about distribution and Joe So Joe Armstrong says if you want to make a distributed system You need at least two computers to make it fault-tolerant distributed system if you if you want otherwise If one machine if you have one machine in my crash the system goes down So you need some form of distribution Interaction with hardware in very large software systems. That's nothing special Continuous operation over many years So again, this was a requirement from from these type of systems that they should not go down And you should not have to take them down if you're doing upgrades You should be able to upgrade while the system is running Same thing with software maintenance and things like this for you have They have to be fault tolerant Things will go wrong in the system, but the system is not allowed to crash That's all there is to it So how do you make fault tolerant systems, and I just want to point out here? What we're saying here is that is that not that there won't be errors there will be errors When errors occur the system must be able to handle those errors control them and make sure the system can continue afterwards And these things were not Optional if we could not design a system which can do all this it wasn't interesting That's all there is to it right and what you find is when you start looking at this There's absolutely nothing about telecoms in there. It's a certain type of system with a certain type of properties That's what we're looking at really Unfortunately Seeing our language felt or deemed to be a telecoms Language in a telecom system. I think some you scared off some users who weren't doing telecoms But there is nothing about telecoms in there And this this makes a few other things as well this So this was the problem So where did we end up about this? Change my glasses. I can read something here So we're what we end up with in designing a language and a system around the language So they were designed at the same time with these properties in it So we had lightweight massive concurrency. This was built into the basics of the system For us it was almost important as being able to make a function core is to have lots of processes communicating with each other very very efficiently There has to be support in the very basic system for building fault tolerance system These are things you can't put on top as a library. They're so important They have to be the primitives have to be in the very basic system. We have to be able to handle timing constraints Which means amongst other things the system must never block It's this is just I mean the system can't stop for five or ten seconds doing a garbage collection because it's just not allowed to do that So we had to we had to think a lot about designing the system Partially the language and often about the system and the implementation of that to make sure the thing does not block and If you look at the current beam today, it is very difficult to make it block You can do it, but just put an awful lot of effort into doing it Just your normal code will not make it block It's like an operating system You can have an operating system running with a lot of process doing heavy load It might slow down the system, but you will not block the system. Well, you shouldn't anyway. It's the same thing in the Alan systems as well Continuous operation and continuous maintenance. There is support for doing code handling for loading code while the system is actually running And it's very well defined. Well, one of the problems doing that is what happens if I redefine a function which is actually in use So some process is using it another process reloads that function This has to be very well defined so you can do that That's one of the properties of the Alan system for handling code is that it is very well defined What happens when you load a module reload a module and We need primitives for distribution So yeah This is the type of thing Alan and Alexia which runs on top of it of course is designed to do But we're not good at everything We have never made that claim If you someone says that says that Alan is good for everything. They either don't know what they're talking about or they're lying and they're a typical number of things is We are not good at you say heavy number crunching. We can do it, but we're slow So if you want to serious heavy number crunching, you would not write it in Alan words here If you've got something where global shared mutable state is the best way to go we don't do that We just don't do shared global mutable. That's it Again, if that's something as fundamental to your system or your solution of the problem, then you have to go somewhere else However, however, however, we are very good at interacting with other systems So from a very early time, we were sort of polyglot in our way of thinking You would have things written in other languages talking with Alan So if you need to do heavy number crunching write it in Fortran and then control it from Alan That's a typical way of doing it other things as well I mean typically you will have you will have things for your system written in other languages So why use Alan to interface it and we're very good at doing that So this is the type of system that Alan and hence elixir it's designed to run on This is the type of thing is designed to do and also some of the things is not designed to do So the that's that's the first stage of the system. The idea is am I following the right prophet? Is this what I want to do if it's something else go somewhere else The second stage here is a joy of the first experiment. I've made some experiments in my system I've made a simple tool. I downloaded the systems and tools and I built my first simple experiment and I'm happy because it works It might have been Simple our language Alex yes over or a phoenix web page. I can knock one of those up in an hour and it works I'm really really happy about that. That's great. That's the first. That's your first stage. So now you feel Driven to keep on going to working with this because I want to do more and The next stage you're going to get through I call the village of confusion Because you're going to hit you will hit things that just don't seem right depending from where you come from and The first one the village of confusion The system is working but it's behaving strangely right in some ways You've really have you really listened to these prophets when they're talking about this about this and When they're describing the language in the system, did you really understand what they meant and The trouble what you can what you will very quickly run into is Especially from the Alexia side it may look like Ruby but it's not Ruby and To be seriously honest Alexia users who know what they're talking about and their developers will not say it's Ruby running on the beam They will say it is Ruby influenced Be very aware of that they're not saying it's Ruby and it's not Ruby. It looks like it, but it's not Ruby. It's influenced It's not an object-oriented language You can do object orientation on it But it doesn't quite behave as other object-oriented languages do If anyone has questions I can talk more about that afterwards. I've done some work with that as well, too It has a strange way of handling data if you're coming from a normal OO language and has a very strange way of building systems It's not the same you will hit this you will hit these stages for it and The first one the first one you'll hit after village of confusion is the hill of functional It's only a small hill. It's not a big hill, but you have to climb it eventually and the sooner you climb it the better and the thing here is now Yeah, simple understanding is not high. It's a functional language They are functional languages with the standard typical functional language properties. There's just no way around this It's immutable data. That's it and this is not a language feature This is actually all the way down in the implementation the implementation knows this and can use this efficient to make things more efficient We have things like pattern matching in there Pattern matching is a fundamental property which is used everywhere and once you get to understand pattern matching You'll love it, but until you do it's just really weird We have things like recursion And you're if you're coming from your typical OO language. You've been told don't use recursion because recursion is bad We're saying use recursion because that's all you have Literally you don't have anything else So you have this problem that it's a functional language and in most cases what you'll find is it's just a standard functional language Everything you'll find in the Alangalexia sequential language. It's a standard functional stuff There is one exception that's binaries, but I'm not going to mention those That's the first one. The second one is the mountain of concurrency get over it get up this mountain as quickly as possible understand what's happening here and Here this is something we have to we have to climb up and understand to understand how to build systems using this Because it is different. It's not it's not high But it feels very different just a different way of structuring and building systems and until you understand this Your system is not going to behave like you think it should you're not going to get all the benefits from the Alangalexia system for and we have things like Well isolated processes So yes, we have processes. We can have lots of processes. We can literally have millions of processes, but they're isolated They don't share there is no sharing a data That's it. There is no global data. That's it. There are things that look like global data But when you start looking at them and say no, they're actually not global data That's all there is to it How do you communicate between processes asynchronous messages? That's it at the low level That is all there is We are not scared of creating lots of processes as I mentioned there are products running millions of Alang processes on one machine products in there for the money And we aren't scared of crashing things This doesn't come without problems, but we don't get that. So yeah, we'll talk a bit about crashing things later as well So the next stage you're going to get to I call the mist of frameworks And this is nothing really our language is specific. It's using most frameworks as well. So when you're using frameworks They can often create a deep mist Around what's actually going on in the system and to understand that you actually have to go in and look at this mist and see What's going on there So So they can they can be very helpful. I'm not not saying anything against not using frameworks at all I'm not they can be very helpful. They can encapsulate behavior They can hide lots of boilerplate code. They can provide default handling They can hide internal system structure and in many ways, this is all very good Because it makes it much easier for you to build your system makes it faster to get it get get it things up and running But there's a big but It makes it very difficult to understand what's going on because you have to understand what's going on underneath But the frameworks have hidden it for you So they made things very nice by hiding it. They also made it very difficult by hiding it and Especially there to choose especially two cases where you might need you might run into problems with this I get an error I Get narrow in the system. I've written something something's going wrong I'll get a report from something I have not written for starting process I have absolutely no idea what they're doing and things like this for because the framework has done all this work for me and that can make it finding errors difficult and The second one is what happens if I've got my framework and I want to combine it with another framework Therefore they have to start talking with each other and if that was not something that was originally designed in the framework Then I have to understand what's going on internally so I can make this connection between things In a typical case with framework frameworks is they will have a set of things They're designed to work to do to build to build a system that does this and they're very good at doing that That provide an awful lot of help for doing it as soon as I start wandering out a bit I have to start thinking myself. I have to start understanding what's actually going on internally So I call that that's what I mean by the mr. Framework Again, this is nothing Alan. You'll exist specific at all. This happens with any framework you're working on what's actually happening inside and One of the mists that come from this is well, it's the Marsha fault-tolerant systems So you have this thing about building fault-tolerant systems Which is one of the fundamental reasons for coming to this environment at all This was because we can do it and we have proof of doing it I mean system with five or seven nines reliability. I think five nines reliability is about five minutes per year or something like this downtime That level and there are systems. I do this we can do it but You have to understand what's going on and it's a big marsh. It's a definitely big marsh. How do I do these things? There are a number there are a number of standard ways of doing it OTP is one of them, but it's very easy to wander off and get just trapped in the bog and not get anywhere So it's some ways. It's very easy some ways. It's very difficult to do this and yeah We need to pick a good one and of course OTP is the obvious one to start with There's a lot of support for it. It was designed for building these type of things So if you start looking the OTP code, you will find it's actually quite simple a lot of it Because this is what our language designed to do for. I can write a supervisor in one page of code Why because that's what one of the things our language was designed to do for it just fits very nicely together everything Many of the stuff we've been using most of them already part of OTP So if you're you you're in Oxy and using packages, they're built on top of OTP They're already using it all The underlying system what's underneath Alexia is built on top of OTP So it's all going all the way down. It's used everywhere for Yeah, so use it and extend it and the thing to be aware of is that it is actually extensible Sometimes you find people Discussions on the net where people get very work very scared of this. They put a lot of effort. They sort of see OTP is some form of System built in concrete with the steel around you can't do anything with it. That's wrong. It is extensible Alexia is a typical example. It's added things OTP compliant things it's added them for because you can do that The system itself is quite open and it's all implemented now So you can if you if you need something else which the OTP system doesn't provide You can write something which is OTP compliant follow the rules. There are five rules for doing it It's very simple. There's support in libraries for helping you do all this stuff. You've done that You just plug it in and just works So OTP is an obvious choice. It is extensible. You can write your own things to it other systems to it but There's a but of course It does require a lot about thinking How your system should behave when things go wrong Okay, we know we say when Because we assume you're going to get errors in your system So what how do you how do I build my system in such a way that when things go wrong the system survives? That's the whole cold the whole thing for it right and that means you have to start thinking before beforehand or in the early stages of what is important in my system So if I'm using these packages or frameworks, they will do that for you in many ways They will tell you yeah, we'll do this. So if something goes wrong, this will happen You just don't have to think about it. So but sometimes you just do have to think about it yourself So you have to sit down and sort of think about okay I've got these different types of processes going on if one of these crashes. What do I need to do? Maybe I can just ignore it. I don't care Maybe I just maybe I have to clean up after it might have allocated resources. I have to make sure that these resources now free Or maybe I have to restart it Maybe it's something which is providing such a service in the system whereas where if this is not running this I cannot say the system is working So therefore I have to restart it And there is support for doing all these things inside OTP, but I have to sit down and work out what I actually need to do And when I do that I have to structure my system to do this and when that works Then I don't have to think about it anymore So I've talked to people to developers and they say that they have never thought as much about errors as when designing our Lang systems because you do it up front and As I say when it works I can ignore it in most of the cases I can quite happily write my code as if everything works because when things go wrong the processes crash and The system knows what to do Again it works. It's a surprisingly simple mechanism, but it actually works So that's that's another thing Another one you will have seen is a double-edged sword of metaprogramming. So yeah I'm a lisper You cannot write this without having macros and doing metaprogramming. Well, you can but no one does So I'm very good off work If you're running Alexia, we're running in our language in a small sense But if you're running especially running Alexia, you've been using macros quite a lot quite a lot of the system is implemented using macros and That's good Because you can make things quite clear you can provide the right macros to make things more usable and more Understandable and more readable So macros have the power of doing good, but also becoming extremely evil. So what do I mean here? so if I They are a powerful tool that they help me write code that Build code at runtime, but used wisely. They can be really helpful I can write macros which say very Understandable very nice and tell me what's going on and things like this Unwisely they can write can generate completely incomprehensible code So I can write a macro that does it does ten different things in different places, and I've absolutely no idea what's going on And you get extremely complex macros for that's the bad size of it and they can make things really difficult to understand You're already using macros So if you're if you're running, I mean deaf module deaf and deaf be their macros a Lot of things standard stuff. You're using a macros. There's nothing strange about this And they're they're pretty understandable macros deaf says define deaf module says define the module We're containing these things that's nice and deaf says define a functional at least one clause of a function Contain this it's pretty straight. They're straightforward But you can write very bad macros as well and a common rule Well, some people say very hard out of say When it comes to macros don't use When it comes using macros don't I think that's a bit over hard I think a more reasonable one is don't use a macro if I can implement it in a function If I can write a function does the same thing don't don't do a macro the macro does not give me anything The thing to remember of course is macros who evaluated compile time So when you're defining your own macros here and sometimes using macros you really have to keep track of sometimes where when things are being done That can lead you to very strange behavior if you get it wrong So one thing that can happen you can think your macros being evaluated at compile time That's actually been evaluated runtime because the way you've defined it or vice versa You will think that things will be evaluated runtime, but they actually evaluated when the macros expanded instead Be slightly careful there So it's there. They are a very nice tool for doing things if you use them correctly. Yeah, the next stage here Is you find out to your horror that the language heaven is actually multilingual? Well, it's not one. It's not one language. So, yeah And yes So yeah, you've heard tales. It's multilingual And If you go on read on the on the web you'll find there's a bit of squabbling between the various languages in this heaven here And they're fighting with each other and which language is the best and the other side is wrong, etc All I can say is don't despair in this sense for it. So what we have here What we actually have when we start thinking about it is something we call the Alang OTP the eco the Alang ecosystem And this is what it's all about We have a system so the language of heaven Language having a problems land is the Alang ecosystem and all the languages are running on top of the Alang OTP system So we have of course for example, if we just got a pen in as well Where's the pen button? Yeah, so we've got the Alang OTP here We've got the beam at the bottom. We have Alang OTP providing the whole OTP system in a large large number of libraries Then we have a set of languages running on top of this We have Alang of course we have Alex here. You've got LFV Yeah, we have a prologue as well if you want to do that. We have allure and other languages as well Now the benefit of all this is that why? Because if they follow the rules which they do They can only openly interact with each other. I'm not locked down into one of them There's no there's no requirement of me to use one And if they do they do it properly the system as a whole can be more powerful than any one language provides Just because I'm using one of the languages doesn't mean I can't access libraries written in other languages So a typical a simple example. We have Phoenix and Ecto for Alexia for doing web servers They're written in Alexia Okay, and they tend to be used from Alexia However, they're running on top of a package called cowboy for handling connections and cowboy is written in Alang And just know it. There's just no problem interfacing Nothing strange about doing this. There'd be no sensible reason for say rewriting cowboy just be just because it's not done in Alang Or sorry not done in Alang not in Alexia So this is not something you really have to worry about This provides a lot of a lot of benefits. I just point out one here For example, if if if you have a something written in the ecosystem, which can talk to other languages in this case for example, Java Any language in the ecosystem can use this and talk to Java as a base they can build their own stuff on top of it There is there is something in the in the Alang libraries called the J interface for talking with Java It makes Java look like another Alang mode in this set of classes in Java for handling Alang data structures There's also a little added feature here, which I very much liked unfortunately. It's not really been productified There's something called our junk writing on running on the JVM. It's an implementation of Alang on the JVM And it is a very good implementation of Alang in that basically can take your Alang code and run it straight off there and it works For some things it's faster and some things are slower you still have problems as heard earlier today about the JVM garbage collection Doesn't get around that, but this is an alternative just an example things for doing it We're working on something now. We call Peerlang which is for is for a system for having Alang talk to Python So you could write you can have things written in Python You can interact with them with an Alang Alang can send messages to Python and Python can send messages back to Alang interacting with these things And this gets back to another thing I was mentioning slightly before is we have this very polyglot view of the world And where there is support in the ecosystem for talking with systems written in other languages And we don't find anything strange about that If you have something Settled libraries or procedures they're written in another language say Java or Python Which you find very useful and fundamental part of your problem fine use them Trying to rewrite the whole thing and Alang would be stupid use them interface with them and talk with them And do that we've been doing it from the very beginning So the telecom switches were talking about they were written in some weird assembler Ericsson had but we didn't have to We didn't have to worry about that. We just send messages to it and get things to work One here I called the pool of introspection that's the next stage So I don't know how many people have actually started using the introspection tools inside Alang if that is common or not at all This is more Alang elixir hasn't got quite that far yet at the introspection, but they're getting there There is a set of tools So you can sit down sit to the pool and look at look in the water and see reflections of yourself and the systems What's going on internally for it? And you can look at the system while it's running And now I'm talking production systems Test systems are one thing but now we can start looking at production systems introspect on these get information about what's happening while the system is running So we can do things like well, there are a number of standard tools One I was talking while showing a bit earlier today when I was With my spaceships we have observer which is just part of the standard Alang release for just introspecting and getting some Information out of the system while it's running We can look at load we can look at low distributions a lot of other things going on there that of course works directly with elixir And any any other language running on it This is not a problem We have a number of third. We have a number of other tools. There's one called DBG Well a quick push There is a number of tracing. There are actually two functions for doing internal tracing of the internal system while the system is running So you can look at things like you say which processes I want to trace what do I want to trace inside those processes I Can trace for example message passing in and out. I can trace function calls Of specific functions. I can even trace things depending on which are which functions I'm being called by which arguments they're called with so I can do different tracing things on top of that as well I Can trace other things I can have a process tell me when it does garbage collection for example if I want to start tracing garbage collection There are for example in the standard Alang system three profiling tools. They all work on top of these tracing bits There's a two tracing bits If stands for built-in function do this So DBG that's part of the standard Alang release. That's just another interface into these tracing bits Which provided a bit easier to run sort of The person who wrote those like short function names So if you start looking you'll find a lot of functions with one One character names two character names three character names There are some long ones with four character names as well, too And to understand what they're doing you will not find you will not understand that from the from the function name itself But they're short to write from from the shop DBG is meant to be run from the shop There are other tools as recon and red bug a common third-party tools Again for doing introspection looking at the system not just the tracing as well We have a commercial product Wombat OAM for looking at systems and looking at distributed systems and networks Trade getting information out of that as well to talk to Francesca about that These all work in Alang, but the mapping to LX series very straightforward We're just talking about looking at data structures. And yeah, they look slightly differently But the mapping between them is very straightforward for so here You'll see something written in our own data structure if you're coming from the Alexia world. I can see the app This one of these is no big problem. There was all one very funny Tweet about that. We have it. We have a map data structure So if you write that in our line, you write it as hash curly braces with the fields in that happen That hash happens to be a comment in Alexia So in Alexia, you write it with a percent curly braces, which happens to be a comment in Alan So, yeah, it's true But apart from minor details like that the mapping is very straightforward So it's no big deal. No big and these things are meant as I said, these things are meant for introspection on running systems And it is very easy if I have a running system I can have a shell running and from Michelle I can do anything I Can introspect I can load code I can do whatever I want while the system is running And I have these tools to help me look at the system as well So, yeah, that gets us well to the final stage at least in this part. It's a joy of arrival We've actually built our system. It's working It can handle everything expected to do it can survive errors as you hoped it would and it just keeps going So you can run your web server It will run there might get a few areas you might lose an occasional connection But the server will just keep going it will not crash everything will just keep recovering as you want it to and All I can say is rejoice and the path actually wasn't as difficult as it first seemed The trick is to realize you will have these issues and be prepared to meet them and The longer you push them the longer delay than the worst they'll get So I mean the simple first case. Yeah, these are functional languages. That's it get over it Once you get over that and get used to it everything else works until you get over and use it you're going to get very strange errors So just the same thing with start building the systems as well, too And yeah, that was basically what I was going to say. That's me questions clear as mud