 Hey everyone, this is Arvind, I'm working as a SDF just-pay, just-pay is a new basic start-up which has started here in 2012 and we were mainly focused on payment systems in India. Yeah, sorry for my voice, I had a bad throat today. Yeah, we have been doing payments from last 10 to 12 years and most of our strikes are mainly in functional programming. We use Haskell and PureScript at a very high scale. So over the years, using PureScript and implementing a very huge data code base, we have faced some issues while using compiler. So then we evaluated some of the ways to make the compiler better and I'm going to share some of our thoughts and what are the changes we did and what are the changes we're planning to do and how can we merge this to upstream PureScript compiler. Yeah, if you have any queries, I'm happy to take one the talk or we can meet in the hangout after the session. Yeah, this talk is mainly about PureScript compiler in cases of incremental. So before even going further, I'll just give you some of the metrics. I just based one of the code base, not considering all the code bases and I considered one of the main code base that we use which handles around 5 million transactions per day. So the metrics are like, we have like almost 2.5 lakhs lines of code and every time a developer does ability to take more than 10 minutes and we have on an average 100 devs working on the project on a day, on an average. So this is the agenda all the way. So earlier it used to be like this, like when we say PULP compiler PULP build, it used to take like, you can just go out and grab a lunch and come back. So your PULP build will be ready. So now with the incremental changes we did, now it's like if you do a PULP build, maybe you can get a chance to go and grab snacks. The next thing was like the future that we're planning and some of the ideas that on the implementation were like, you just do PULP build and no more, like you're not going anywhere. So you have to be stay hungry and stay foolish. I guess. So considering main use cases, when we are working on any project, some of the major use cases would be like adding a new functionality that would be considered as adding new functions in functional programming language. And second thing would be changing some values. Maybe you want to try building out with different values in UI or different incremental variables or some constants you want to switch to. And something else would be like, you want to do some debugging. So you want to add some logs, maybe comment out some logs, sorry, comment out some code to make sure that which part of code is taking more time or per analysis or even understanding where the work was introduced. So considering these use cases, like this is one of the main code ways that I was talking about. So ideally, first-time compilation takes around 4,365 seconds. And let's say I made a small change, which is nothing, but I have updated my default version from one to two. And then my compilation time took around almost the same. It's like I'm compiling every time to almost 37 minutes or around that time, 37, 98 seconds. So ideally, the thought was like, why don't we are making these kind of changes? Why does it have to do the compilation like how it does for the first time? From this pain point and from this thought, we have started working on the optimizing compiler. So you can even see how the CPU was behaving. I mean, it's taking all the cores and memory is also high. And that's the reason why I said, like when you do pulp build, you just go and grab a lunch. So it's going to take a while. Another interesting thing that we observed was like, when I even don't make a change, sometimes some of the IDs make this crazy auto save, even though we don't change anything, or you might have commented the line and uncommented the same line, or you might have removed the extra lines and all. That's what I considered as no change in the code. Even then, the compiler took almost the same time, like no more of an intelligent system which says that, hey, man, you didn't really change any code. I'm not going to compile the code versus, hey, I have seen some of the file has been changed, the modified time has been changed. I'll just literally recompile everything. See, okay, now I'm going to talk a little about a principle thought of mine and mostly that we followed Jaspe. So one of the principle thought is like we always try to attack any problem with 80-20. So the main idea itself is that can we find that 20% of effort that can be solving 80% of the use cases. And see, this is one of the main reasons why this compiler was, we kind of found only few places where we really need help or it takes most of the time from developers. So those are the main problems that we focused on and those are the main problems we solved with more or less a 20% effort. To be honest, it's not like we have considered all the scenarios and it's going to be like so good at all the scenarios, but we have considered most of the scenarios that developers use and which were added also in the use cases part and in all those cases we have put 20% of effort, minimal changes that you can do to the compiler to get the maximum of the benefit. And this is one of the way we work at Jaspe. We will always try to find any 80-20 and try to, if you see the metrics here, it's like you can solve it, 99% of the problem with just 24% of your effort. So this is something always we try to find out and solve the problem more or less very fast away. See, this is the first thought that triggered to me when we were actually trying to understand why the compilers are taking time. The very first thought that triggered to us was why do I have to compile when my function signature was not even changing? See, even before coming to this point, we were asking questions like what do we mean by compilation? So what are we really saying when we say compilation? Is it doing a type in friends? Is it generation of whatever the back end we support, JavaScript, Python, PHP, anything? Is it what exactly do we say? Who is taking how much of time? And in that we have noticed that when the function signature was not changing, ideally you don't have to do the type in friends for all the dependencies, right? There's one of our first thoughts and we tried to evaluate that using some of the changes. So if you observe, we read the modules, like we have a module, which have a function called add five, and I'm trying to import that module in another module called y. And when I have made change in module x, saying that I just issued the x plus y with y plus x, and then we have realized that module y shouldn't be recompiled ideally. If you don't want the module y to be recompiled because the type in friends are no more required, right? Because add five is always an end to end and module by using add five won't have any issues. Yeah, I mean, these are some of the metrics that I like after making this one change where when a function signatures was not changed, we don't want to recompile all the dependencies. With that one change, we could able to bring down numbers like on the left, which are very high at the second and all, and we bought them down to 10 seconds. In the same way, when there is no change in the code, maybe you moved the lines, maybe you commented and uncommented the same line, then and all, these numbers are again, the numbers on the right side are more or less using a stop board. So because there is no time come and I could do in a watch mode. But yeah, these are more or less the numbers that we that we were able to achieve now with a very simple change that I explained in the previous slide. I'll just give you some of the implementation details what we actually did. So to solve this problem, what we said was when we are actually parsing and when we're actually trying to build a particular file, we actually build a hash map which will have a function name and the function signature. So every time you make a change and you try to compile, we again build this name and function signature hash map and we make sure that nothing has changed. Let's say for particular functions, function signature has changed. That's when we will say, okay, go and recompile, like I can't give any advantage. But let's say for any particular function, the functions type signature has not changed, then we can simply say that, hey man, see you compile yourself, but don't expect or don't tell your dependencies to compile them. So that's how we majorly solve the problem. See one of the main concern, to be honest, till now is the CPU and the memory which which we started working on. We kind of figured out a couple of ways to make these changes and try to use less CPU or memory, but it's still in our implementation. So I can't really give out numbers as of now, but to be honest, it's right now good enough to heat up the laptop. So this is still a concern from the compiler that we are using. See we did a release. So people can find this release in NPM packages under the name hyperfine pure script. But as of now, we released only for the four versions which are mainly used at just pay. So we have very old code base that even still using 0.11. So we have only these four versions, but we're planning to add for 0.15 or the latest versions also. And we hope like before any new releases, we will try to bring these changes into upstream. If anyone is interested, they can get these old versions, which I mean, I can't help it if someone is using 0.15 as of now. But if you have any code base using these versions, you can try out the compiler using the NPM package. Or if not, you can visit just base GitHub. It's a pure folk of pure script repo and we made changes on top of that. So so ideally you can even observe what kind of changes we did and maybe some more suggestions like like any any suggestions regarding the changes that we did and the way we did. Also any people have any other thoughts regarding the use case versus this 8020s that we can do early and try to come up, try to achieve some higher numbers or higher advantages where we are happy to hear and there was also a post on this post regarding this and we are happy to like interact with folks and we are happy to speak with more folks to get more inputs regarding this compiler and see if there are any more changes that we can or if there are any issues that are being faced with people or more or less in that way. So as I mentioned, until now this is one of the main changes we did and we were able to achieve the numbers that I shown ago. But there are other thoughts which we were evaluating and also started working on it and they seem like giving even more benefit and also helping out in many more use cases. One of them is to build a function level dependency. So if you observe like in the previous explanation what we did was like we said if a particular function signature doesn't change then only you don't recompile. But if a function signature change we will simply say go and recompile the dependencies. But ideally let's say a module imported a function and that function is not being used or it being used but it's really not changing the function signature of the current module. So example could be let's say we have a x-module and a y-module. x-module have an add-fi function and y-module imported that add-fi function and somehow we changed the add-fi function type signature. So ideally we say go and recompile y and we don't reapply the same logic on the y and we simply say recompile all the dependencies. I mean the better way could be you consider y now and since add-fi signature has changed you again look at all the functions of y and see if the functions of y functions of y's add signatures has changed or not. If not I don't recompile all the dependencies but recompile only if y-file function signatures are changed. So this is something we were evaluating and looks like most of our use cases are like this. So even though some of the util function type signatures have changed I don't need to compile till the main mostly because the util functions type signature or the function whose signature has changed is mainly being used by more or less only few modules and we want to compile only them and then we re-evaluate the same logic and on the change in modules evaluate whether those functions the functions in those modules type signatures has changed or not. Yeah I'm taking a question so sort of ask like what about the inline functions? This is one thing that we observed when there are these inline optimizations the ideally we have to recompile and generate the JavaScript code but as we noticed like most of our code is not doing any inline optimizations so that's one of the main reason that we were able to use this compiler but yeah this is this is really a this is really a right question where if any inline optimizations were done by the compiler we can't really avoid recompiling the dependency modules. Yeah see this is something we were evaluating and trying to solve the problem in the more general way not just not just like saying that oh see in just a minute we were not getting any inline function so we'll just use this compiler but we were trying to look at we're actually evaluating the JavaScript compiler we're seeing is there any way we can actually disable the inline while we are doing the while we're using this compiler and maybe the final pull build maybe or the release build right you we can add a flag saying that it's a release build and then we'll just remove the optimizations and we'll simply say hey just go and compile everything and generate the proper output so yeah it's a it's a it's a genuine question and this is something we have evaluated and and and it's a known issue but but as of now and since we're using this compiler for a very time as of now we didn't see any issue or or any bug that that got introduced because of the inline inline concerns yeah yeah back to the functional level dependency graph we just don't want to yeah we don't want to go with the file level but we want to go at the function level and and build the graph and only the function signature change we build the modules which are dependent on the function not not like okay I imported this file so I'll simply go and recompile that file other than this like there is another thought which is for first time compilation but but this is mostly I'm not sure like how well it is useful for a general audience but but considering a company like us where we have around hundreds to 120 days work on average every day and and make new features make new releases we thought we can actually use something like IPFS and maybe share the output folder across us such that our first time compilation also won't take much time because it's always like this right like when the code is ending up in a mid bucket it's being pushed by one of my friends or one of my colleagues so ideally if I could just like I didn't if I could just create a tar file of the output folder and get it into my laptop I can use it so we thought like maybe why not IPFS and the piers to compiler itself can using the git permit hash it can look at the IPFS network and if the output module is fine output module it found it can just bring it down if not it can it can do a simple compilation as it does right now so again as I mentioned I'm not sure like is this really useful for the general audience but but if you're a company and you have like a good enough number of people and you make changes most of the time and you see a lot of new branches and new features that you always do I would say this is something that we are yeah this is something that can be majorly useful because like as I as I shown it on the screens earlier the first time compilation is something is still a problem and it takes around yeah it takes like huge amount of time of devs and which is not like which is which is one of the major hindrance that we're still facing facing but we were seeing if IPFS can be evaluated and and maybe maybe add the commit level based output output bringing into the system by using IPFS network is something we were looking at but we are not completely sure okay yeah see other than this like majorly just maybe we work on functional programming and and we build products of five we build products of payments at the scale of India and most of our stacks are running in Haskell and PeerScript and even Rescript so we were one of the major company who uses types and who who gets the functional programming to the scale of India and into the production directly and as I mentioned about the 80-20 this is one of the things that we always like are trying to figure out any 80-20s in doing the task see as if you observe even the piers to compiler it's not like we have considered all the all the scenarios and we have come up with a complete 100% working compiler but it tends to solve 80% of the problems and it helps a lot of our developers right now in-house and and they could able to make releases faster they could able to push code faster otherwise like it usually takes an hour or two for for any person to like make a change if there is any bug found it takes around two hours for our developers to make a change maybe maybe add more logs go through the code the compilation again you make some more changes compilation again you make some more changes find out about come up with a solution and push it to production it takes around a two to three hours on an average but but it kind of dropped into less than five minutes because because because of the use cases that we usually have but but there are scenarios as I mentioned if we have to change the function signature or adding a new functionality or or updating the dependency graph or is it a first and compilation we are still at we're still at a issue level where we have to decompile from the first and it does take a lot of time and and as the code base increases at the as the code increases we kind of see we kind of seeing this compilation is actually going exponential not just linear because earlier when we have very less number of files or less number of modules it used to be like very linear not increasing or not taking much time it kind of went from five to twenty very very fast way because because we are just like working we are just adding more features more features and suddenly we just went to this 30 minutes 40 minutes compilation time and that's become a high endurance for us uh yeah I mean true I'll kind of have a bad throat so if if if you have like if if something was not clear maybe you can ask questions so I can I'm happy to help but sorry for my voice I I'll just add a back throat today so thank you Arvind