 Can you hear me back there? Yeah, okay So thank you for having me here, and I want to thank PJ and Johnny who just left for Inviting me here and making this great conference. I really enjoyed yesterday's talks, and I had a lot of fun meeting many of you So looking forward to today. So I got a warning up front. I have this tendency to speak pretty fast Sorry about that But I want to give you as much as possible try to pack as much as possible in my talk So hope it's gonna be worth your time and also during this talk You may get this impression that I'm trying to sell you something and I just want to state up front that this is not what I'm trying to do The position I'm taking in this talk in all my talks is that of a happy user I said this by the customer and this is really how I think of myself when I Consider a leak sear and Erlang I've been using both languages for many years now And I'm really really happy with them and the OTP framework and the associated ecosystem third-party libraries Frameworks and a lot of great tools that we have and I really feel those things help me do my job Help me focus on the core problems of my job But still there is absolutely no doubt in my mind that like the most important piece of the puzzle Here is the runtime layer itself the virtual machine the official one which we call beam And this is the thing that I believe takes this whole system and puts it very far apart very be very very different And special compared to anything else that we have available today And it is a thing that I feel helps me the most with some of the most difficult challenges I'm about to face regardless of the precise nature of the project. I'm gonna work on and I just want to state that Beam as the runtime layer is actually not great for every type of software Which is in fact precisely what makes it great in my personal opinion Because rather than trying to be a jack of all trades and therefore a master of none Beam is built with a strong focus on a particular class of problems a particular class of software And I like to call such software a software system And this is a kind of software that you build you put it into production You start it once and henceforth it has to run continuously constantly for like a long time many years But maybe decades even and it has to do many different things at any point in time to serve the needs of many many users Depending on the system like 24 seven every day of the year and of course it goes without saying it should do those Things within some reasonable time constraints, whatever that means for each particular use case now Think web servers back ends of any kind Crawlers chatbots we seem to have a lot of interesting emerging examples in the embedded area as well So quite a lot of different software falls into this category that I'm labeling the software system And then there's a lot of software which is not a software system And if you're building those other kinds of things like especially some more complex things on a larger scale It's very likely that beam is not the hammer to that particular nail But if you are building a software system regardless some kind of a web-facing system Regardless of a business domain complexity or scale there is no doubt in my mind that Beam-facing languages are your best option available today Precisely because the runtime layer itself is fine tuned for the problem, right? It has some properties it makes some strong guarantees and some trade-offs are made in implementation Which are very well suited for the class of software systems, and this is what I'm going to talk about today I'm gonna cherry pick only a few examples and Try to explain. Why do I feel are they relevant in this particular kind of software? So it's gonna be technical, but nothing super deep you don't need to worry about that it's more like a high-level look at our lowest layer that we're using and For the purpose of this talk, I'm gonna assume that the thing called nif does not exist Now nif for those of you who are not familiar stands for native implemented functions It is a mechanism that allows you to write your own custom native code like in C C plus plus Recently people have been doing it with rust And to run that thing directly in the beam operating system process and when you do that then all the guarantees are off Right like you can crash your entire system You can completely paralyze it leak memory resources and whatnot. So it comes with a lot of power and a lot of bad stuff can happen And this is why I usually say it should be like your last resort It's perfectly valid if you want to want to implement some parts of your system in some other third-party languages Which do not compile down to beam But there are some other safer options which are not as performant but way safer and this is what I would recommend first So whenever I say something is guaranteed or something cannot happen It's really isn't true if you take nifs into the equation, but I'm gonna remove them from this discussion Okay, so let's get down to business Right, so you have your system powered by beam like let's say it's written in elixir. It's as a mixed project You start it whether it's locally with a mixed tool or in Production which in my opinion should always be an OTP release The story is always gonna be the same. So you get your own operating system process Your own instance of beam and in there you're gonna have a couple of threads running Called schedulers by default one per CPU core, but completely configurable There will be some other threads which I'm gonna ignore today And then you have a bunch of these what I like to call little yellow boxes This lightweight beam processes and you're gonna have way more of them like this is idiomatic Erlang and elixir I usually like to describe it as run different things separately You want to run different activities in separate processes and this quickly leads to explosion of these processes like in smaller systems You might have hundreds of them larger systems tens hundreds of thousands millions of those things Could be running around during some peak periods So this is what's gonna give you a lot of superpowers and the first thing I'm gonna talk about is context switching In the schedulers right so schedulers are the ones which are actually running those things like a scheduler pulls a process The process hits a scheduler and now it has some CPU time it can run some beam instructions and During that time if it doesn't need CPU anymore like it's sleeping or maybe it's waiting for a message to arrive or maybe it's Waiting for an IO operation to finish then it's gonna implicitly yield and someone else gets a slot But if it's highly CPU bound like for a long time say computing pie to billions of decimals Then it's gonna be switched out and this is gonna happen Usually it's gonna happen pretty quickly like in less than one millisecond and I say usually because the actual Implementation is not based on fixed time slices. So under some condition Arguably arguably not very likely it can happen that a single process holds on to a scheduler for much longer Like maybe a couple of tens of milliseconds But I'm not personally aware of any kind of elixir code Which we can write that could hold on to a scheduler forever, which is why I like to Say that for all practical intensive purposes the scheduler is preemptive and what this means, right? So we have like preemption and a very frequent preemption very frequent context switching what this means for our system it brings us a property which I like to call responsiveness and Instead of trying to describe it. I'm going to try to demo it with a simple Through a simple site that I built here for you. It's powered by By cowboy and plug I'm gonna start the thing and to make things a little bit more explicit and clear I'm gonna explicitly stated. I want to have just one scheduler thread and Let's start the thing Okay, and let me start the observer thingy Okay, so we can visualize stuff right so you can see here Hopefully that we have a one we have one scheduler thread It's already a little bit busy because the observer itself is implemented in play in our lungs So whatever it does it also uses this scheduler, right? So every single beam process of this beam instance is going to use the scheduler thread So now let me show you the site do not be intimidated by the complex user interface. I'm going to walk you through it step by step Okay, so what what does it do? I give it a number It gives me a sum of first and natural numbers and if I give it a larger number. I'm going to get a larger sum Which is hopefully correct. I actually have no tests to back this claim But it doesn't really matter in fact because what does matter here is that the implementation is completely CPU bound, right? So there's no yielding sleeping message passing file operations and whatnot So basically goes from 1 to n and accumulates the sum along the way So if I give it a large enough number, then of course it's going to keep this single thread Scheduler thread busy and this is what we're going to try. I'm going to give it the magical mystical 9 9s 4 5 6 7 8 9s, right, so hopefully that's right Let me check it out and when we when I go to the scheduler Utilization hits, you know now we're fully maxing out of the scheduler got a move quickly here So the site is still responding right in a separate step I can run stuff and I can get my responses pretty quickly this thing here is still ticking and it's just finished Right, so what you've just seen is the long running computation was running and it wasn't really too much extended Relatively speaking, you know, so the running time is in the ballpark and the short running Requests were pretty quickly certain. This is what I like to call responsiveness That's what let's talk a little bit more about that But so what you get is let's say that you have like a web server on this machine with a single core You have single scheduler thread and you have five requests pending like tasks that need to be done All of them CPU bound the first one quite heavy 10 seconds of CPU time That's quite intense the remaining four fairly short ones one milliseconds each and what you're going to get Could beam is like after five milliseconds you have already served four out of five requests Then you're handling the rest of the long computation and even while you're doing this You can still take more work and serve it pretty quickly Assuming it doesn't require a lot of computational time itself But so this is what you want in a system because you're juggling with needs of many different independent people now in contrast with Cooperative scheduling and I want to say that many of popular technologies today are going to give you this type of scheduling When the task hits the scheduler or this thread then it doesn't let go for as long as it needs CPU So for the first nine thousand nine hundred ninety nine milliseconds nobody gets their response right latencies All the latencies are ten seconds plus just because you have one super heavy long-running computation So this is not really quite good and I don't want to say the corporate scheduling is universally bad but I don't think it's a good default or a good choice for a software system and You can suffer from this problem Even if you have multiple threads because the story is roughly this like you're going to have to deal with a bunch of Bunch of logical activities right like thousands or millions maybe like handling requests doing some background processing and whatnot And you're going to have a fairly small usually fixed-size pool of operating system threads because those are heavy resources So you cannot have like millions of them So what you need is just to have a couple of these tests there's a small fraction of your system being super CPU busy for a little bit longer and Most of your system is on hold business of one or few leads to paralysis of many or all and this is definitely not good I have seen the situation in production myself. Have I seen this thing take down the production? so technically speaking it was working but Basically all the threads were busy and nobody was able to get their response within any kind of acceptable time So for all intents and purposes it was down not only have I seen it I have actually did it myself Not intentionally but by mistake I make mistakes from time to time like probably on a daily basis and this was a great example of how Seemingly low-level technical property Amplified my mistake and led to production being completely down, right? So that's that's pretty pretty bad in my opinion now beam is way more forgiving towards such types of mistakes Like if you have a couple of these things then your system is ticking as I have shown you and of course if you have more of these Long-running tasks, then they're gonna put a pressure on those schedulers and Your latencies are gonna increase your performance is gonna suffer But this degradation is gonna happen gradually gracefully so you get a time to spot that something is often to Maybe Understand what's the problem and fix it before it completely blows your production up, right? So this leads me to the next point the next property I want to discuss and this is the fact that Activities of your system powered by beam are first-class runtime things now what I mean by this is As I've said initially you want to run different activities and separate processes and these processes are Implemented at the runtime layer, right? So therefore your activities are runtime entities I'm I'm stressing this because there are some popular lightweight concurrency implementations Which do this on the library level and therefore they are not runtime entities, obviously another part of this puzzle is that activities have identities rights of these processes have identities, which is again not always the case the most notable example is go Basel CSP concurrency model where activities are anonymous and communication channels have identities now in beam you have basically activities as runtime entities with identities and what this means is that You can manage and manipulate these activities by talking to your runtime directly even if they themselves refuse to cooperate So this is pretty important, but sounds quite fluffy and weak So I'm gonna try to demo what I mean by this. So let me head back to this site I'm gonna show you bug you're gonna immediately know what's the problem But let's pretend that we don't know and let's see where it takes us, right? So like if I enter a negative number here Then this thing is gonna never gonna return right so you can tell here that it's just spinning and if I go to the scheduler Then it's completely maxing out, but it's never gonna return like even if I close this tab It's still computing there. So it's maxing out my scheduler and of course as I've shown you the site is still responsive So that's cool But if we get more of these things running, we're just gonna do a denial of service and essentially the system is not gonna be Working so we need to find what's wrong and assuming that I have some like proper Monitoring and alerting system in place. I should get some notification like my CPU Utilization is way off and now as a developer. I want to go to the production and see what's wrong I understand what what's causing this problem. And this is what we're gonna do. I'm gonna debug the thing Of course, I'm doing this thing Locally, but whatever I'm showing you holds for production You can tune into a running production without needing to restart it set some environment flags or God forbid Re-deploy a tweaked version of your system You can hook into running production and you can get a lot of useful info just by talking to your runtime about your activities And this is the point I'm trying to drive Right, so I'm gonna use like low level stuff. There are some you actually don't want to use this in real production You have some higher level tools Which are easier to use and a little bit more production save But I just want to drive this point and keep it fairly lightweight, right? So we want to find out who's burning our CPU and this has to be clearly one or more processes So what I'm gonna do first is I'm gonna list the processes That I have running in the system, so I get a list of pits and now I can go through each of them and I Can Ask my runtime something about them, right? So in this particular case what I'm interested in is a property called reduction count Or reductions and I'm gonna keep it very simple and stupid It is a number the higher the number the busier the processes and kind of dumping it down here and skipping some fine points But that's that's the idea. So we're I'm gonna look for the ones which have the highest reduction count Those are the one which should be the busiest So what I'm gonna do here is I'm gonna pack this thing in a tuple So I get a list of tuples reduction count plus a pit and now it's pretty much straight forward It's a smooth sales basically I need to sort this thing in a descending order And then let's see okay, and then let me take just top end of them Like say five and then I'm gonna For the better UI UX I'm just gonna print each one on a separate line Okay, so we have top five Reduction counts and you can tell that the winner is way busier than the runner up and therefore this process here Is the cause of our problems right so we were able to find the thing that's causing the problems Now I'm gonna we're gonna drill into it and we're gonna find find out what it does So let me take this pit into a variable right so I'm taking this identity because you know this thing That's think that thing has the identity and the first thing I can do is I can take a look at the Current stack trace right like a snapshot and keep in mind that this thing is completely out of control It doesn't cooperate with me, but I can still find out a lot about it But so I get the stack trace and I can tell that it's like cowboys So it's a web request. It's a router so I can narrow it down It's some so now I already know what type of the request the file name the line number and I can go back to the code Stare at it, you know like contemplate meditate Maybe something comes up and I can see some potential problem But why stop there, you know, I can I can get much better than this I can actually ask my runtime to trace me the activity in this process alone out of all millions of them running around the Production I can hook into this one little thing and I can see what happens and this is what we're gonna do I'm gonna use a DBG module because it's out of the box It's kind of gnarly or cane, you know a little bit cryptic, but it can do a lot of stuff, right? So what I'm gonna do is I'm gonna start a tracer and now I'm gonna say that I'm interested in tracing just this process and In particular I want to trace only function calls You can trace a bunch of other stuff like garbage collections messages being sent and received and whatnot So when I'm gonna look at just dynamics of this particular process and then I'm gonna set the pattern I'm interested in local calls of every single module. So every every function of every single module This is gonna generate quite a lot of output. It's gonna basically Hog this shell session So what I'm gonna do here is a cheap trick and just gonna sleep for a single second and then I'll Stop the thing because I hopefully I didn't make a mistake here because it's gonna mess with my dramatic build-up. Let's see Okay, looks fine and stop Yes, okay, so we have one second trace Thank you very much. Thank you. Do not do this at home, right? I'm trained professional So we have one second trace of the activity of that particular process and you can tell that it's just infinitely recursing And now I pretty much know what's the problem like like in 99% of the cases the root cause of the problem Is that I was trying to be too smart about too insignificant thing, right? So in this particular case, I was trying to make some as a tail recursive Implementation so I'm recursing passing three arguments the start the finish and the accumulated some every time I recurse Increment start by one and I'm gonna stop when start and finish are the same Which is never gonna happen for a negative number for a floating point number probably some other edge cases around now I can go back to the code. I can fix the thing. I can even hot patch it, you know in production without needing to restart it But in the meantime, you know this thing here is still it's still consuming my CPU, right? so we're still maxing out on the scheduler and We can do a little bit here to help the production. So what I'll do here is I'll stop the thing, right? So I'm gonna send it an exit signal and in particular I'm gonna send it a kill exit signal Which is special because it can be trapped. It can be intercepted. This is kind of like kill minus nine, right? So we are brutally killing the activity and I'm gonna do this thing and you can see the log entry and going back to the scheduler We're now back to the normal utilization, right? So what you just witnesses I approach the system and I was able to quickly sift through like millions of things running around the system to Find the rotten piece then drill into the rotten piece to find why is it rotten and then take it out without disturbing Anything else in the process and I'm not aware of anything else that comes even remotely close to this level of service Like even this seemingly simple and naive feature of being able to forcefully terminate an activity gone rogue It's gonna be mission impossible for the vast majority of popular technologies today like the best case that they're gonna give you Is something along the lines of this, you know You politely ask the thing to stop itself and then you hope and pray that it's gonna stop itself Which is not gonna happen in the case you've seen right so what your best and only option in that case is going to be to to restart the whole operating system process and That means that you're gonna have to take down Thousands or millions of perfectly well functional healthy activities down as a collateral and that's pretty shitty if you ask me Right so in Erlang you can get much better You can get this thing I can ask the thing to stop itself But I can talk to my runtime to make some stronger guarantees I can ask my runtime. Let me know if that thing dies no matter how that so I set up a monitor to process And then I asked the process to stop politely and now it has the chance to flash its work say some final last words And then I'm waiting for the corresponding message and now no matter how the process dies, you know, even if it crashes before my Request reaches it. I'm gonna get this message. I'm gonna know that it has terminated. So that's one big thing That I get here and of course me being a separate process I can wait for this down message to arrive for some time if it doesn't then the other thing refuses to stop Maybe it's busy. Maybe just ignoring my request and I'm gonna brutally kill it right so I'm gonna send it to kill exit signal and Then I'm awaiting again for the down message because the exit signal itself is asynchronous So I need to wait for the down message but now I'm pretty sure that it's gonna happen because this is brutal termination and what you've seen here is a Sketch of how supervisors terminates its children, right? So polite, please stop followed by a brutal brutally die if you refuse The only difference is it doesn't use a special message. It uses an exit signal, which is not kill, right? But that's the only difference. So this is kind of the core idea another interesting thing we can do Because we can forcefully terminate our activities. So this sketch the snippet that you see is a Computation with a cut of time right with an upper time limit So this is kind of like here I'm the manager process and I start the computation in a separate process. It is a task. I don't care about the boundness It can be IO CPO doesn't matter So now as a manager, you know as every self-respecting manager. I basically what do I do nothing, right? So I stare at the clock and wait for the result to arrive if the result arrives I'm gonna take the credit, you know do something with it and yay But if the result doesn't arrive within some given time constraints, then I'm gonna terminate the worker But so I'm in walking task shutdown with the reason brutal kill under the hood This is gonna do what you've seen on the previous slide. It's got a new process exit kill followed by It's gonna then await for the down message. I don't know if James. Are you here somewhere? Yeah, it's kind of out. There you are. Okay. Yeah, I'm not sure why are you using brutal kill there? But under the hood, it's a special reason kill process exit with a special reason kill is used Okay, so a lot of these stuffs that I've shown you again I just want to drive this point the first and foremost thing the property that that makes it possible is the fact that we can talk to a runtime of other activities So the runtime knows about those activities and they have identities. This is this is a big deal for me The next thing I want to talk about is share nothing concurrency. This is pretty famous property And it's a great example of a trade-off made in the implementation of beam So I usually like to describe it as these processes are essentially separate programs No, they just happen to be running in the same operating system process But have nothing in common really they have their own flow of execution have their own stag their own heap share no memory and Because of this property you have to pay the price when those things communicate like when they when you're sending messages Across process boundaries the message is being copied in most cases. There is one optimization path But for the most cases it's being copied So therefore you have to pay the price in some memory, of course and especially in some time The good thing about it though is that the the cost of this message passing is explicit It's obvious it's up front and therefore when you're tuning your system and your benching it when you're optimizing You're gonna see this cost and you're gonna optimize your messaging protocol Like you're gonna reduce the chatterness between processes and you're gonna remove or avoid sending needless pieces of data across process boundaries And the cool thing about that is that it makes it easier not easy But easier to spread this little yellow boxes around multiple machines because your messaging protocol is already optimized, right? So that's one good thing. Another good thing is that by paying the price up front. You don't have to pay the completely Unexpected surprising non predictable and sometimes very expensive price of stuff to world garbage collection so we don't have the thing in the beam and What's great about that is that you get more predictability Predictability like you're tuning your system. You're benching it and you reach some targets desired performance and you put it into production and You can expect more consistency there You can expect less gatchas in surprises over time, right? So that's another nice thing But another part of this puzzle that I actually want to talk about here is The fact that resources are also owned by Processes so what I mean by this is like stuff like file handles and network sockets ETS tables Monitors races and whatnot all of those things belong to some process Like usually by default one that actually creates them like if I open a network socket And I'm the owner and I can give it away to you and then you're the owner But someone is always the owner. That's the point Excuse me Now what this means is that when the process terminates no matter how it terminates Everything is properly cleaned up Like memory is reclaimed and files are closed and sockets are closed and so on and so forth And even that means even if the process is brutally killed and this is important because brutal termination means you don't get to say final Last words you cannot catch the thing you can do try after or anything else So still the stuff is going to be properly cleaned up and I'm just going to briefly show showcase this so I'm going to head back to the site and I'm going to do it from the command line. I need to curl This site just a second. Let's Okay, so let me try the good case. This is cool. And I'm going to do the minus one thing again So again, I'm running a request that runs for Infinity now we're going to kill this this thing And this time I'm going to do it in the observer to make things a little bit quicker So this one is the top the top one and this is the process that I'm going to kill That's the request handler and it owns the socket, right? It's going to send the response over that socket and I'm going to kill the thing and Immediately here I get an empty reply from server because the process has died and the socket has been closed on the server And therefore the client is now received the proper information, right? So that's just a demo that even when you have brutal termination the thing is going to be killed Okay, so finally I want to discuss how OTP supervisor actually sits on top of this seemingly simple low-level Properties and give some guarantees of its own, right? So OTP supervisors are usually discussed in terms of Restarting failing activities. This is arguably the most important point but feature but it's not the only one The thing I'm going to talk about here is our supervision trees So a supervision tree is basically the way that allows you to like you have this thousands and millions of processes in your system and with the supervision tree you organize them into Services of subservices of microservices of nano services of Pico services and so on and so forth you just drill down, right? So that's kind of the idea what you see here is an example of a supervision tree That I had a couple of months ago when I was demoing Phoenix at a local conference in my city So we have like a top-level supervisor. It's a system and when I start the system It consists of a back-end service and the front-end service So starting the system means I'm going to start the back-end service fully Then I'm going to start the front-end service fully and then the system is considered to be started and so on and so forth You get the idea now When you have such a system, you know, I just wanted to say think about it Is it is if it's kind of a unit system for your beam? it's like in it D if you will and Now when this thing is running there will be some various situations when you want to stop working service There are a bunch of reasons for that like sometimes you want to do this explicitly because of the flow in your application other times you want to do this You specify Rest for one or one for all supervision strategy and this essentially means you're telling to the supervisor if this thing terminates then Terminate all of these other children because there are somehow deeply connected. They are tightly coupled. So sometimes you do that There are another another frequent cases when you have a bunch of restarts in a single supervisor And then the supervisor gives up it says I cannot fix this problem I'm going to terminate my complete subtree and I'm going to delegate the thing further up the chain right so bunch of different reasons these are all the only ones and Basically, like if I want to stop a front-end system, I want to get that right like You would expect to have the proper subtree cleaned up with everything with all the associated resources That's what you would expect and that's this is what the supervisor is going to give you No, you can mess it up But it's an opt-in you really need to look for trouble to get into trouble right to the by default It's going to work pretty pretty well and the reason why it can guarantee this is because it sits off top on these Simple guarantees so frequent context switching with preemption means that no matter what the workers do even if they're super busy Hugging up your CPU and scheduler the supervisor is going to get its CPU time to do its work Right, so that's one important property and because we have first-class runtime activities That means that the supervisor can work with its runtime to manage those activities It can ask to be notified when things terminate and it can also ask to forcefully terminate those things if they refuse to stop themselves and finally of course share nothing concurrency, especially resource ownership means that You're going to get a proper cleanup files and handles and network sockets are going to be closed and whatnot now These are the foundation of the supervisor and take any of these properties away You know pick anyone take them away and things might look the same at the surface level They might even work well for a lot of cases, but there will be some non-obvious very treacherous Ugly nasty problems lurking you might end up with dangling processes Which could lead to re-entersy and race conditions and you could have leaking memory and resources and you can have excessive Needless CPU consumption, which could lead to self-denial of service. You could like completely block your Scheduler it's a bunch of nasty problems Basically, the system is not able to self heal properly in all situations anymore and sometimes the attempt to self heal is actually going to aggravate your problem So that's you know pretty pretty terrible the fault tolerance layer itself becomes faulty and that's pretty bad in my opinion because fault Tolerances I believe one of the most important property of any kind of a software system that is supposed to run for a long time So if you are aware of these risks, you can of course mitigate them You can work hard you can follow some best practices and do thorough code reviews and whatnot and you can improve your chances But no matter how hard you work how much you try you're never going to be able to fully eliminate The risk because the risk itself originates from your runtime and this leads me to The main point of this talk like this super big and important message. I want to send to the world So I have this feeling this wrong feeling that as the community of software developers many of us not not I'm not talking here But you know like in general when we're choosing our tool for the job and comparing different tools and discussing them It seems to me that we put a huge amount of emphasis on these things on the right side Sometimes I'm almost kind of get this impression that the success of a software system and its reliability depends on whether you have curly braces and semicolons and out of the box one line deployment commands You know and I'm not going to say that these things on the right side are not important They're of course important. You want to have some Some reasonable syntax and you want to have some flat learning curve and approachable Community and you want to have of course West echo system of libraries frameworks tools and so on but The thing is that if you have those things sitting on top of an inappropriate runtime, which doesn't give you some foundational guarantees then those things are an empty shell right there like a shiny facade on top of a shaky Structure and in contrast if you have like a properly designed runtime with the respect to the kind of problem you're solving then it's straightforward to support different syntaxes and to to flatten the learning curve and to evolve the echo system and of course a great example of this is elixir itself Because you see the runtime is basically your foundation It is your ground right and when the ground is solid then you can move with a lot of confidence at the predictable base Never looking back never regretting your past choices you can build your system and watch them grow and it will be on your original plans and imagination and Basically, this is why I believe that when considering your tool for the job, especially for a software system I would say that you should ask not only what your language and your library and your Framework can do for you, but you should first and foremost ask what your runtime can do for you because it can make a really huge difference Thank you