 Hello, everyone. I am Sujata Hemadi. I currently work at Redbus. Since two years, I am an airline developer. So, today I will be talking about my journey with airline at Redbus. More than my journey, I believe it is every developer's journey who is trying to move from object oriented language to airline. So, along with the journey, I will try also explaining some of the core concepts of airline, so that it becomes clear. So, to start with I started my career as a Java developer. I was trained in Java for three months. Overall, I have four years of experience in Java. With the help of great team, each day was a new learning. So, I quickly understood Java. So, with my curiosity, I dig deeper into the language and I tried understanding how actually the language works rather than how to make things work with the language. With all this learning, I was in a mindset thinking that Java is the solution for all given problems. Then, airline was introduced to me. Airlines was introduced to me by Neeraj Sharma at Redbus. So, for people who do not know what we do at Redbus, Redbus is basically a bus ticket booking platform. I will give more details later on. But, for now, let us consider two systems. One is transaction system and other is accounting and invoicing system. Transaction system is a back end for bus ticket booking, which takes care of the life cycle of entire bus booking. And accounting and invoicing system is a system which automates the accounting and invoicing part of the Redbus transaction. Basically, it calculates how the money should flow for each of the transactions. I used to work in accounting and invoicing system, which was written in Java. It was challenging in its own way, but I had reached a point where all my mind could process was numbers, incomes, expenses, etcetera. I knew I needed a change. And at the same time, in Redbus, the decision was taken to write transaction system in airline. Why this decision was taken is not very sure to me, but I needed a change. That is the reason I felt learning airline was the change required for me to move to transaction system without any knowledge of what this language does or how this language behaves. The sole purpose of me taking up the decision to learn airline was for me to move to the transaction system. That is how I saw it as without any clue of how the language works. That is why I called it as a forced choice airline. Before going on to the technical road of the journey, there is something I need to address regarding a psychological factors which impact the journey. This has a major impact. Let us talk about it. That is resistance to accept change. Why does this happen? It can happen due to multiple reasons. Maybe it depends on person to person, but according to my experience and few of my colleagues experience, all I can say is the major velocity is comfort zone. Coming out of comfort zone is never easy. In my case, knowing airline and being able to, being comfortable working in Java was that comfort zone, which I was not ready to let go of. That was the main hindrance for me. Other reason could be when you are not sure why this change is happening. As I said earlier that I took up a decision to learn airline because I wanted to move to a different system altogether. That was the decision taken by me. But with respect to learning a language, that is not the right kind of motivation to start with. This is the main demotivating factor for me while learning the airline. Few other things are like fear of unknown. Questions like, what if I do not like airline? What if I start forgetting Java? Or is it the right choice for my career? Or will I be an expert in any particular language if I take up this decision? All these kind of questions had popped up in my mind. But with my experience, all I can say is no, you will not forget Java. But learning airline from Java is little difficult because of the object-oriented mindset. But over a period of time, you will accept a language. Actually, you will start liking the language. One thing benefited is for me here is learning airline broadened my perspective of solutioning the problem. That is a good thing about airline. So, while I was researching on this topic, I found this diagram. Actually, it depicts more or less every developer's mindset who tries to learn airline from object-oriented language. So, if not 100 percent, 90 percent of them start in the denial mode. Even today, if you see two developers talking with two developers with the different language background speaking, they will get into argument mode where they will try to prove each other saying that this language is better without even having a mindset to accept that maybe there is something better in the other language. In my case, this conversation happened with the Golang developer. Both being a concurrency language, you can imagine how the argument can go on with it. Next state would be confusion and anger. This confusion and anger is a part and parcel of the denial mode. Denial mode, me being not able to let go of Java, in that mindset, me trying to understand airline leads into lot of confusion, end up being angry on yourself for taking up a decision to learn airline. So, most of them tend to quit in this space where because of confusion and anger for the decision taken to learn airline, you tend to give up. But I survived this phase by asking two questions to myself. Either get a solid reason to quit or try to find a reason why you need to accept the language. Here I am talking about my journey with airline. So, you know what question I found the answer to. So, whoever has, whoever are stuck, please try finding this answer. Either answer you get, you are sorted with the journey. So, give you 100 percent before you quit. To give a overview of both the language, since I have background with both the language, let us start with Java. Java is a object oriented language where its main building is the objects. Where real world entities are represented as a object. For example, let us take this conference as an example. Here conference hall will be an object which encapsulates all you individuals as a set of person object and me being another person object with special property as a presenter. This is how object model will look like for a conference hall. So, once the session is done, I may leave the room. Some of you might leave or somebody can join. All these changes are represented in Java world by modifying the existing conference hall. And this modification of modifying the existing conference hall is called mutability. So, in object oriented language, they serve the, they use mutability as the main construct. And this mutability is achieved by having shared heap memory. Where in Java every objects created are put into the shared heap memory. Where n number of threads with the right axis can modify them. So, JVM, JVM is a Java virtual machine, a runtime execution unit for Java. Coming to Erlang, Erlang is a functional and concurrency oriented language. Its main building blocks are process. Not to confuse with OS process, but it is a lightweight process created and maintained by Beam. I will, I will explain this further later on. But for now it is a process created and maintained by Beam. Beam being the execution unit for Erlang. And this language, taking back the example of conference hall, in Erlang world this would be represented as conference hall will be a supervisor, where each of us will be a process created under that supervisor. And me delivering this page would be like a message passing from me as a person process to each one of you. Since we do not share a memory, this messages are copied onto your brain. That copying is called immutability, where a message is not mutated. Each one gets its own copy, right. So, this language works with immutability as the main construct. Since it is a immutable language, you do not need a shared heap memory. So, each process gets into its own private heap. I think key difference what you need to understand here is in Java concurrency is achieved by thread class, which is a abstraction of native threads. Since there is a shared memory involved, you will have to have synchronization or some locking mechanism to have a safer concurrency. But in Erlang concurrency is handled by Beam, detaching that concurrency model from the OS layer. So, to give a overview of Redbus, Redbus basically is India's largest bus ticket booking platform founded in 2006 sorry as of today we have served 8 million customers. We are not just in India, we are operational in Indonesia, Singapore, Malaysia, Colombia and Peru. As a Redbus platform we even have services like bus hire, hotels, pilgrimages and Aarpool. Aarpool being the newest service of Redbus which enables bike pooling and car pooling for the office commuters. If you see the bottom most part of the slide shows the search bar where you have to enter your source location, destination location and the date of journey when you have to travel and hit a search button you will get all the set of buses in that route for a given day. Even this search result is powered by Erlang different Erlang platform which is rewritten in Erlang a year back. I am not going to talk about this because I have not worked with this but in Redbus we do use Erlang for search result. So, the middle part of the slide where you can see the seat layout where you select a seat enter your boarding point, dropping point and your customer basically your details and try to book the ticket. That is when the transaction system comes into picture. The slide equal to booking the ticket is handled by transaction system. Basically right from creating tentative block payments and confirmation all happens in transaction system. That is where I have worked on and even this has been written in Erlang from last two years. So, next few slides I will be talking about few of the constructs in Erlang which are easy to understand but hard to adapt to. So, one being the syntax. Syntax every language it differs but this syntax is so different than Java it is little hard to understand. So, as learning any language easiest way to learn any language even I started learning with syntax. As a added advantage Erlang provides with the shell where you can start with all the basic operations like summing up number even try printing the hello word or any syntax for that matter which is shown in the right side of this slide can be tried and tested only with the shell not even bothering to create a module compile it and run. So, that is the best part of when Java where you have a shell and try out different things how Erlang behaves. But the question is is this the right way to start learning the language? It is a no answer I agree but for me it is it is an yes. So, it is better to start learning a language with syntax. Is it the only way to start learning the language? Not necessarily actually you can even start understanding the Erlang concepts first before you even try on the syntax. But being from the object oriented mindset it is little hard to understand the concepts in Erlang. So, whenever you try to understand a concept in Erlang you always tend to say that oh this is this can be done in Java as well what is so special in Erlang. But that mindset will have that problem that you will not be able to accept those concepts. So, it is better to start with the syntax this is something which is very simple to understand but very hard to adapt to that is mutability versus immutability. Mutability in simple terms being once created can be modified n number of times and the immutability cannot be modified once created simple enough to understand. But once you start writing a Erlang being that in Java we always try to modify the object once you create you keep modifying that object n number of times. But in Erlang you cannot do that that will create problem while writing the Erlang code even though you understand that Erlang will not support immutability you will tend to write a mutable code in Erlang that is the first thing at least for a month or two months you will end up writing creating a variable modifying it and assigning back it to the variable. First part is it is not even a compile time error. So, you run it and then it fails. So, you will have to get used to it for a while to get to adapt to this feature to adapt to this immutability. As I said it is not easy I mean it is easy to understand this concept but hard to adapt to. So, it will take some time to reach that place of where you can create a variable and if there is a change recreated and used it. One thing I found easier is that in immutable language it is easier to debug why I see this is that any given program behaves differently when there is a corrupt data when there is a corrupt data that corruption of data in immutable language where the data is the only place where anything could go wrong. So, you know the exact place where you need to check and identify the bug but in mutable languages when there are when a single object is mutated from different places you need to check each and every place to find which code might have corrupted the data. So, it is little tedious in mutable but since people are used to doing it in Java world they do not find a problem but once you move it move to a immutable language you will see the benefit of immutability while debugging. Next thing is precaution is better than cure versus let it crash attitude. So, I hope it is understood that precaution is better than cure represents Java and let it crashes for airline. In Java we usually tend to write a defensive code defensive programming where a normal execution of a code is layered with multiple layers of errors. Basically write from the null pointer exception till the IO exception checked unchecked every possible exceptions you try to handle it beforehand correct. Even if if you are using certain external resources like files socket you need to close even if there is an exception all this cleanup has to be done in a Java program. But when Erlang says let it crash does it even make sense in first place for Java developers at least. So, whenever we try to write a program we tend to say that it shouldn't have any problem once gone to production right. So, you tend to say that normal execution also should work and any of the unexpected behavior should also be handled usually we do that in Java. But in that sense it if you say let it crash what do you understand. So, it is little hard to get into that practice of let it crash. Does that even mean write does that even mean saying that write the happy path and do not even bother about doing anything actually it is not that way. You are supposed to handle all the validations data validation null check validation all that you need to do in Erlang as well because you cannot end up just giving some random error to user you need to handle all those questions all those errors it is not actually an error it is a proper validation as per the business correct. But there is something which you need not worry about like like hand cleaning up the resources like if in Erlang each user request are a process. So, this process as I said earlier has a private heap space right. So, any resources needed to run this process is within the process itself. So, if the process dies it will clean up everything by itself. So, if a file is opened or socket is opened it is all part of a process when a process dies it cleans up. So, Erlang you need not worry about cleaning up those kind of resources that is one advantage I see in Erlang. But more than that why does Erlang say let it crash? That is also because of the supervision tree Erlang provides you with the supervision tree what does it mean? Application is started I can you can say the main guy who starts the application is a supervisor. Under the supervisor each of the processes started the supervisor's job is to handle the process state basically if anything happens to a process you define beforehand saying what needs to be done. So, there are multiple strategies to handle these cases like one to one one to many like if there are n number of process under a supervisor and you say one to many if any of the process fail it will restart all of the process. So, all this are part and parcel of the Erlang's beam you do not have to do anything it comes as a package. This is one of the best thing in Erlang it is also you can say it is as a self feeling property of Erlang. Few other concepts are like there are no loops in for in Erlang basically no for no do do while no while this is scary enough because in Java we always need for loop to even simplest of simplest problem you would write a for loop right that is not there in Erlang. So, to change from that mindset of no loop to having a recursion for every every list is little hard as I said it is easy to understand but you cannot adapt to it that is one change required when you move to Erlang no inheritance no overloading but overloading I correlated it with the pattern matching in the functions. So, in that sense it made little sense to me. So, how I started seeing the better side of the language by now actually it is been a year I was struggling with all this and then we were already live in production like one year back I mean two years back we were live but ok sorry thing is we were live. So, I could I have seen the matrixes where we had no major production bugs there were no unexpected bugs in the production so and there is no CPU hike no stop the world GC. So, it all went smooth so I knew there is a benefit in Erlang but I could not sense it because I could not know actually how the Erlang works. So, I could see the benefits but to get that into my sense like I was not able to accept because I was not very sure of how Erlang was used to work that is when I started finding the answers of how and why this language works this way. So, it almost took one year to get here. So, this was the first thing I answered for myself whenever people talk about Erlang Erlang means soft real time. So, finding answer to what is soft real time was very important because when I started the language I had no clue what soft real time meant. So, this is something very important to me because by finding the answer to the soft real time I found my confidence with the language I actually started feeling that I know Erlang rather than saying that I know to work in Erlang I started feeling I actually know Erlang. So, let us try understanding this. What is soft real time? Basically, systems under certain circumstances can tolerate certain amount of delay these delays are acceptable but not desirable. Let me repeat systems under certain circumstances can tolerate certain amount of delay and these delays are acceptable but not desirable. Let us take an example for as a ticket booking experience during the offer season where given a choice is it ok to fail few bookings or is it better we serve everybody with a little delay? Obviously, the answer would be is the little delay is ok rather than failing the few of the bookings right. So, that serving everybody with little delay during the unexpected load is the behavior of soft real time. To coming back to the definition if you say systems under certain circumstances meaning book a ticket booking experience during the offer period where it is an unexpected load there and we serving all of them with little delay is tolerable because people are getting bus tickets at least even with the delay people are happy that they are getting a bus ticket. So, it is acceptable because people are getting the bus ticket it is not desirable because there might be some irritation due to the slowness that is why we call it as that is why we say it is a best effort system it will try its best to serve everybody equally even at a unexpected load. Another way of looking at it is systems with no hard bound SLA meaning service level agreement also you can say timeouts which are not very time bound systems saying that given a request it should be executed within X time if it is hard bound then that cannot be achieved with soft real time where impact of SLA breach is not life threatening. I think it is better we try understanding which systems cannot be considered as soft real time. Taking examples of example of flight I do not know every operation in flight is not eligible for airline or not but at least in general sense you can say that landing of a flight even there is if there is a delay in execution of that command that means a plane can crash leading to death. So, I believe airline cannot be used in that condition at least where it is so time bound that if there is a delay it is a major disaster. Another another example could be a trading system where if there is a delay of any microsecond or millisecond it is equivalent to huge loss which is equivalent to huge debt which is also year to near which is also equivalent to a near death situation. So, even in this case where it is so tightly bound timing is required this is not suitable for a soft real time. So, airline would be is not right choice I believe ok. Why is this important? As I said during unexpected load where traffic spikes correct. Another reason is to tame creative processes. Meaning if there is any buggy code or a process which consumes lot of CPU it cannot bring down the system. Because of the software real time it makes sure that system will work well even when there is a process or something which is you can say consuming lot of CPU. For example, we can take RabbitMQ consumer consuming a faulty event it gets into a loop I will not be able to acknowledge it because it does not have a relevant information. So, it gets into a loop even this process it is a infinite loop if you look at it even this process will not bring down the system but there is some openness in the system because of software real time. I hope it makes sense ok. To understand further let us try understanding how it works in Java. So, if you have a application deployed in Tomcat Tomcat being a application server we need to consider three factors max connection except count and max threads. What is max connection? Max connection is number of maximum request Tomcat can take in basically and except count except count is a OS level backlog queue size basically. Let us understand what it means. If a system is taking in more than max connection request then those requests are piled up at the OS layer in a queue called backlog queue that size is defined by except count. Meaning if a request comes more than max connection plus except count those requests will not be taken inside the system. Means systems will start rejecting those requests. Basically you can say total number of max connection plus except count is the total number of connection a system can take. Max connection is total number of connection a Tomcat can take. Clear? Ok. Coming to max thread what is max thread? Max thread value is that value which defines how many threads java threads you can use to serve all those correct all those requests defining meaning that if you have set max thread as 100 meaning only 100 request can be served simultaneously concurrently basically. So all the remaining request will be piled up in a internal queue. So whenever a thread gets free it takes up the next request and process meaning only it will be in execution. Others will be in waiting state. You can say that but in airline max connection and except count holds good. It is the same meaning even here but there is no concept of max thread. Instead beam creates those many threads as many as number of cores. Meaning if we have two core system that means there are two threads created internally and each request is created as a process airline's process correct. Now if you have n number of process all those I mean basically by meaning only two threads are created at beam does not mean only two requests will be concurrently handled. All the requests are concurrently handled just by using these two threads. How that happens? We will explain further in the beam further but this is how the airline works. It will concurrently process all the request using those two threads. This is this example I think Indians can correlate to panipuri example more than understanding the soft real time. This example will actually make you feel the soft real time. For people who do not know panipuri it is in the nature how it is served we can correlate it to soft real time. So try understanding how it is served. So considering a plate of panipuri which has six puris basically a panipuri guy takes one puri at a time puts some ingredients into it and serves one at a time to a customer. This process will repeat until a customer gets six puris. This is how it is served not per play I mean directly in a plate of six puris instead it is served one by one. This is how panipuri is eaten. We can correlate into this soft real time word. In regular days let us consider two different guys one panipuri guy serves one customer at a time let us call him A. Another panipuri guy serves multiple customers at a time let us call him B. In regular days if there is single customer each for both of them then there is no difference how it is behaving both will serve one customer at a time there is no difference so does not make sense soft real time system and others behave more or less same sorry but when there are lot of customers A will still serve one customer at a time serve him six puris then the next person will come all the others will be lined up behind him waiting to eat panipuri got it B being more efficient I believe at least I feel that he is more efficient what he will do is he will serve every customer by giving each puri one at a time in some fashion maybe in a round robin fashion at least if you see the behavior wise here five people are at least getting something to it in the first guy one guy is happy that he is getting everything but rest all are little anxious that when will I get it type maybe that fifth customer even gets excited and leaves the queue this second behavior is soft real time behavior actually if you consider this as a thread or single core process where B representing a soft real time system a single thread or I mean it is in in airline terms it is scheduler but let us say it as a talk say it as a thread he will serve ok let that those customers if you think it them as a process he will give each puri one by one to a person and that puri represent the time slot given to him to execute certain amount of work in that process this is how airline actually behaves he gives chunk by chunk time slice for execution of the process here that is represented as a each single panipuri to given to each customer but in java like application if there is only single thread that means if customers are considered as a user request only one user request will be served and rest all will be waiting to execute that is the difference between how things work it makes sense ok at least you could relate to ok understanding how I mean more or less that panipuri concept is within the I mean in simple terms is within the beam ok in beam everything works as a process and each request are a airline process right but there should be somebody who is managing those process to run concurrently that is done by scheduler there will be one scheduler per core if there are two core that means two scheduler and each scheduler will have a run queue where all the process which are ready to be run are kept in sorry ok all the process which are ready to run are kept in the run queue and now the scheduler's job is to efficiently schedule them so that it runs I mean concurrently right so what does scheduler does it it takes a process from the run queue gives it certain amount of time to execute and puts it back to the queue and takes the another one and puts it does the execution and puts it back to the queue this happens for all the process within the run queue and that certain amount of time given for each process to process is called reductions in airline actually it's not a time factor rather in simple terms it's considered as a number of function call it executes it's actually in simple terms it may be much more complex than what it is but for everybody to understand you can understand it as a number of function calls executes so by default airline provides with 2000 reduction per process meaning that a process will run 2000 function calls before it is preempted and put back to the queue so this goes on for every process so if you see every process moves one step at a time this is how you serve everybody in airline the migration logic written above that part is to make sure that run queue is optimally used saying that if one run queue has a more process or long running process it makes to balance it out that is how airline works this is in simple terms it to explain this entirely it is session in itself scalable fault tolerant distributed language whenever we hear the terms scalable fault tolerant distributed we tend to understand with respect to systems being scalable system fault tolerant system distributed system we achieve it by having fault tolerance by ELB keeping two nodes all that but actually this is the only language which supports scalability and fault tolerance as the language itself because of the right concurrency model used by airline so distribution is as simple as building a regular system where all you need to do is cluster them and clustering is as simple as running your node with the same all the running all the nodes with same cookie and running either one of the command all will be clustered by itself best part is sending a message to a process in a local node and the remote node is same the syntax if you see it is a PID and the message only difference is a PID could be of a local or a remote this is as simple as it is I believe this is the only language which makes it that simple this is the graph taken from last Thursday when there was the offer running where by booking the red bus ticket you could collect rangoli stamp in Google Pay I believe everybody most of them at least know what that means Google had this concept of collecting five stamps to earn 251 rupees so by booking the red bus ticket you could get a rangoli stamp so when this campaign was run we had certain spike in the bookings even at the forex spike in the bookings our systems worked beautifully there was no downtime we could serve everybody with little delay that is what a soft real time has benefited us with hello system is actually one of our systems only but not written in Erlang where you could see huge numbers of time out it could not serve all those requests that is the main difference we have seen moving to Erlang this is some of our strengths this entire transaction platform is handled by 2 plus 3 people where 3 people are with little less experience on an average their experiences of 3 years and in that 2 are new to Erlang it's been only 4 months they are into Erlang with the background from Ruby and Java so it's it's a mix and match of team even in this small team we could handle I mean maybe that because of Erlang it enables us to work better with the small team compared to Java time to market in Erlang has improved 80% this is the proven fact at Redbus at least even in increase in transactions we had not changed our infra, infra could hold all kind of transaction spikes when moving from Java to Erlang we have reduced our infra from 12 cores to 4 cores with the same amount of traffic the top most layer is from the search engine taken that it used to serve the same traffic with 100 plus cores in Java and now it has been reduced to 32 cores we had 2 major down times we are live from 2 years and 2 months so in that even this year we had no down time no more sleep nights thanks to self-healing capability of Erlang we have not seen any issues so far touch wood at least these are few opinions from my colleagues the first guy he has started his career with Erlang no background of any programming language what he has to say is easy and interesting to learn it is the language for concurrent feature right side he has 2 years of experience in Java 5 months in Erlang what he has to say is migration was difficult due to maddening syntax however it is an effort worthwhile concurrent and distributed this C and D keeps me motivated good thing to hear at least I believe moving from Ruby to Erlang was not difficult initially I was stuck with the fact that you cannot update a variable see this are the simplest I mean I can even say that it is the silliest problem anybody could face but Erlang is awesome but I miss loops these are the comments what we hear at our workplace what I would like to say is learning Erlang from Java is not easy but not because of Erlang's complexity but because of the object oriented mindset and not being able to push yourself out of your comfort zone and for people who have not thought of using Erlang or not knowing started understanding what Erlang is you are missing out on huge benefits with Erlang or Erlang like approach thank you and these are some of the links which you can go through for more details of how we are doing using Erlang at Redbus any questions? nice presentation how is the hiring experience are you finding like hiring experiences as I said people with different background are willing to learn Erlang so it was not very challenging yes finding a hard core Erlang developer is little hard but people willing to learn Erlang that has helped us because Erlang developers they were from different background one was from Java one was from Ruby they are adapting very well with Erlang so yes and we could handle all this work with less small team compared to how we used to work with the Java but answering your question yes finding a hard core Erlang developer is little hard hi I am Pranav Erlang also provides this feature called hot code updates basically you can update the code without having to restart the entire system so do you use that at Redbus no we have not used it but no actually we have not used it I know that it provides but we have not used it thank you great talk my question was like you know there is an observer kind of tool so do you guys use to debugging your system or you have something else to do observer to debug basically so you can like get into the remote servers using that as I said the distribution thing right you can cluster your system by starting your setting starting your application with the same cookie and then use that command to connect to each other then you can do if you know the PID in that system maybe from the log you could find it and you can do whatever you need to do the remote debugging it is that simple we do not use observer as such but observer we use to understand the that supervision hierarchy I have at least I have used it for that purpose but debugging remote system even in two of our instances we have done this where locally we run a node by setting the same cookie and connected to the remote server by doing giving that command and then running and finding we tried even getting the process info to find if there is any problem with the process memory or reductions all that kind of information we have used it thank you