 Okay, welcome to comparing CF serverless environments This talk is Gonna be interesting in the sense that it's different from what you've heard from serverless. It's also Something that started out of conversation in the hallways at pivotal like a lot of things tend to happen with CF I'm max and this is my colleague Neema He actually probably could present everything because he did the most of the work He just happened to To be stuck with me So I'll try to make it short and sweet What we want to try to do is to discuss a series of question around serverless and also At the end kind of discuss also where we're heading As IBM you probably heard of us discussing open whisk and Different serverless environments that that we are essentially trying to sell This is not gonna be one of those talk where we're selling that as a matter of fact That's part of the challenge here So we'll get started and then you'll see at the end where we are trying to encourage you to participate So what is serverless computing? There are different definitions, but at the end of the day It's just a web function, right? It's the ability to execute some kind of a remote function from your Program, hopefully it would be something that reacts to some event. So for instance, you have an IOT like some of the Common demos around serverless. So you have a some kind of an internet of thing that's like telling you for instance the weather and then you want to execute some computation or on that and Typically, it would be lightweight Stateless and hopefully cheap because you don't have to keep the container or the VM wherever that code is executing You can just execute it when you need it. So it's very reactive event-based and Hopefully short and sweet sort of like CGI Few people at Pivotal when we were chatting kind of compared it with that and I'm sure on the web They do the same. I think it's a little bit different, but it's similar So what are some questions around serverless and why did we start this work to compare them? Well, obviously the first question being part of CF should be well, why don't you just use Cloud Foundry already, right? I mean we provide all these things and then people will say well with Cloud Foundry you spin up an app You can even do better because you'll have a log You'll have tons of information about what's going on. Why do you reinvent what we've already done with PaaS? I see some people smiling because that's a common question. I think it's a fair point But the people doing serverless will tell you it's a little bit different too, right? Because you you can build a workflow around some of those web functions. So if you look at our solution OpenWisk and even Amazon's or Azure's You'll see that the UI allows you to sort of set up a workflow around different functions That's a little bit harder to do with CF because then you'll have to have different apps and so on so it's a little bit different It's a different model for programming. I think But there are still questions around that right like so what's the advantage? What are the advantages of course cost performance from the perspective of the person operating a serverless environment as well as the person using Serverless environment is a big question. So can you can you kind of compare those? And then of course, could you just use CF, right? That's a typical question So our goals is not to make another CF proposal Because in some ways we want this to be separate from CF although there's some relationship And we also can't possibly explore all serverless environment even though the solution that we have Once we make it open source and we intend to We will you will be able to try other serverless environments Okay, so that's that's part of it and again, of course, you know the big problem with this talk I can tell you up to right before walking to this door. I had people from IBM needing to want to see our slides Because they want to make sure that we're not trying to favor things and of course since we're IBM We're comparing things. How does it look? So you'll see we had to dumb down things a little bit So you'll have to go run the experiments yourself so that you can compare them if you want We actually look pretty good, but we're not the best in every category. So it's kind of like, okay Yeah, I can't even say that so maybe I'm in trouble already So just heard you just said the best part, you know It's also not a full suite of tests. I think that would take a longer effort But it's a it's a good beginning. So the way I like to say it is, you know, we used to be As a company and we're trying to get there where you know, we not only participated but we led we were the leaders And we would help lead efforts and I think that's where we are with serverless We want to help lead the the movement. So think of this as potentially a contribution in that direction So the approach that we're taking is to essentially try to figure out which is best But by defining some experiments and then running those experiments and sharing the results and repeating so very simple very classic you know in some ways Scientific method of which both Neema and I have background and and You know academia, so we're trying to do the same but not fully with the rigor of a scientific paper So with that, I'll pass it to Neema But the first thing I want to mention is that we've looked at all of these environments So Azure obviously is the Microsoft functions. It's in beta open whisk, which is our solution Which isn't in beta also lambda is the only one that's public right now iron IO I'm not sure they are fully beta, but they're certainly part of it and then this thing in the middle called CF serverless that Couple of people at pivotal one in particular. He doesn't want to be named I work with him on Bosch. So, you know That actually hacked up this version so with that I'll pass it to Neema who will talk about the result But he actually is gonna do a cool demo too. So, all right. Thank you Thank you max so I'm Neema and Caviani and I'm a contributor to Diego runtime full-time and This is what we did on the side to understand so so CF serverless or basically serverless systems as a whole So as max mentioned when we started thinking about understanding these different serverless systems We decided that we wanted to have a way to see what the underlying architecture is like and that's one of the reasons we went about Implementing CF serverless so out of the discussions that we had with folks at people at all We decided that okay Let's implement something on top of cloud foundry that more or less emulates the behavior of a serverless system and then based on the Lessons that we learn we go and understand what is the overall behavior of several serverless systems in general So we implemented CF serverless as an application on top of cloud foundry Which can actually turn off and turn on other applications So if you have a function that you want to run and if you want it to be serverless You can deploy it as a cloud foundry application And then there's a layer on top that manages that application when the request come to that application It goes and it starts the application brings the application up responds to your request and then if the application is idle for some some time it takes the application down and By doing it by just turning on and turning off of the application It saves on resources and essentially saves on money So once we did it then we realized okay So that's a very basic implementation of serverless on top of cloud foundry And now what is the lessons that we learned from it and what experiments can we conduct in order to understand how the other systems behave? So we went and basically thought about the possible experiments and we built a system that would allow us to run those experiments So I'm going to show you a demo of what the system is but basically before that I'm going to tell you what experiments we came up with So the first thing that we decided to do was to understand the behavior of these serverless systems on their high throughput and low throughput loads So basically if you have a function that is constantly receiving calls Then what is the behavior of the system and if it's only frequently receiving calls then what is the behavior system then? Why is this important it's because essentially when you have a function that runs as a serverless function What happens is that the code gets loaded into a container, right? And that container has a lifetime usually See of certain like several systems are short-lived containers because they want to save on resources They kill the container as soon as the function and has done with its job, right? So the question is how long this container stays around? For it to be responsive and at the same time be cheap the way be implemented our approach as I mentioned We kill the container if it sits idle immediately The problem with it is that's the next time you call that function Because you have to bring the container back up and redeploy the code and all of that it's going to take time, right? So the question for high throughput versus low throughput was what is the approach that other? Serverless technologies are using the other things that we did were like typical performance measurements and memory intensive computation And we did it by doing matrix multiplications to 300 by 300 Matrix we multiplied them and we just is a known memory intensive Problem and so we measured the amount of time that it actually takes for different functions to respond for CPU intensive We did finding of prime numbers and that's a very common and CPU intensive function We tried the across all the environments that we mentioned We also looked into container management behavior basically the same thing on I mentioned whether the containers stay around Whether they get killed immediately or whether there's another strategy involved so for all the experiments that we did we actually Deployed because we wanted the the the deployments to be similar and close So we gave every single function in every environment 512 megabytes of memory and we deployed all of them In US East because when you go to these cloud platforms, obviously you can choose where your functions and are Deployed except for open this which is I think only available in Dallas. So with open risk. We didn't have that much option So then we started launching requests to these functions and we did two ways of launching requests sequentially so that every request Hits the endpoint after the previous one is finished and then in parallel So all of the requests hit the endpoint at the same time and we did it with 100 requests And we basically did it three times and we did a ramp up period so that we actually have everything up and running before starting to collect it Okay, so for the demo This is basically what you need to supply to the To the basic code that runs the experiment you define all the different environments you define the endpoints and you you define the type of headers and the type of Bodies that are gonna be passed to these HTML calls because at the end of the day DCF These serverless functions are just HTML like HTTP endpoints that you hit And once you provide everything in the right format for those functions. It's just a simple HTTP call So once you provide all those information that what happens is You can define the number of times that actually a function is called and you can define the time between intervals basically every two consecutive calls And it just runs it against all the environments that you have and collects the data and reports the data back So as you can see it comes up with some numbers on how much time it takes for every one of the environments So with that we started running all these experiments against all the environments that I mentioned And we went to all of these environment open with skazure lambda iron IO and deployed these functions the functions that we implemented we implemented the functions in Python and I believe the code for the experiments and also for the Test suite is going to be released. Hopefully sometimes soon. We are planning to make it open source So let's look into the results. I basically for each of the experiments. I'm only Showing one of the environments the one that I thought was more interesting If you look at it for a high throughput function, this is basically a simple echo function You hit the endpoint and it responds with a simple like hello world You see that the all the 100 requests were successful and the average time it took for every request to come back was around like 180 milliseconds or 200 milliseconds and there are spikes in the In the experiments and those are the ones that actually took longer for the for the container to be set up So the spikes that you see are the ones that the container wasn't there So it took probably like close to seven minutes to create the container put the code on and then respond to the first call But once that first call was through the remaining calls, but actually much faster That's why the average is in is there are 200 milliseconds So that was the sequential one right where every request came after the the previous one Then we did it in parallel and Azure was very interesting because as you see there is like there's steep Basically response time like their response time increases kind of linearly as a number of requests as as we basically push more more requests or basically make more calls and It seems like Azure does not a good job Bringing up new containers in order to be able to equally balance the loop I don't know exactly what is behind it But if they had enough resources to respond to these calls, this should be like a flat line This is what we observed in some of the other the better environments Usually no matter how many how many requests would you launch towards an endpoint? It was kind of a flat line But for Azure it actually spiked up So for low throughput, we actually made it so that the same echo function rather than Calling it and basically one after another exactly We made it so that there was like a five minute delay or one minute delay between every every two consecutive calls The way we implemented CF serverless was that if The endpoint was inactive for 30 seconds. We would kill the container We wanted to see what the other platforms do and we actually played with the time Here I'm talking specifically about the one minute delay But we played with five minute delay and ten minute delay to see whether changing the time would actually affect the time that it takes for the container to respond back and Well, CF serverless, we knew what the behavior was the container was killed So every request took around like 10 seconds to respond But we realized that for other environments, they're actually not killing the container What they do is that they freeze the container So they use the C group freezing and it allows them to free up resources, but keep the container on the disk So the next time it's called it's actually slower slightly slower, but not drastically slower So it's not in the order of like seven like seconds. It's usually in the order in the upper end of You know hundreds of milliseconds for for all the other environments that we observe open mist lambda and Azure for iron IO, it's different. The thing with iron IO is that it's specifically designed for running Tasks that are Supposed to run only once but are like heavy in computation So their implementation was very similar to serverless They would bring up a container launch everything into it do that thing and then kill it and throw it away So it would take a lot longer for every single car from iron IO to come back And that was an interesting comparison or observation compared to the other environments So we did the memory intensive function experiment And in a sequential case see this is the usual behavior that you would expect to see from a serverless function a flat line The average was around, you know, 1.3 seconds and all the requests were successful Now we did it in parallel And the interesting one was again Azure one thing we noticed was that out of the calls all the calls that we made only 15% of them were successful and the other ones basically failed and resources got congested for some reason and They couldn't respond immediately. It was usually a timeout or out of memory error that we received when we hit the M For the other environments, it usually was better. It was again kind of like a flat line The other experiment was CPU intensive function. So calculating prime numbers and what I'm showing you here is a sequential case for CF serverless and as you can see except for the sparks that I explained earlier, it's pretty much a flat line 100% success and Each calculation took around 2.5 seconds And we did it in parallel for open-visc and you see that as requests come in It is it goes up like this this pattern kind of similar in all platforms It goes up but not that much the overall average was like around I think this one is actually 26 seconds So I think I am I have a typo here in the case of sequential It actually took 26 seconds and then in the case of parallel the response was around 8 seconds to 9 seconds For the platforms that we experimented So general observations about the experiments that we ran one thing that we realized was timeout was an issue and a lot of these Environments, even though they claim that they actually have configurable timeouts. They really don't for example in Amazon The default is 30 seconds But if you change it to five minutes, it's still 30 seconds like whatever you do. It's 30 seconds For for Azure, it's not configurable yet I think in iteration 22 that came out like a few days ago They made it so that you can configure it. They claim that the default is five minutes But usually what we observe was that as we hit these endpoints, it would usually time out a lot faster than five minutes So, yeah, it's still a kind of a sketch. The other thing is that there is this notion of money versus Time right and the general idea is that if you spend more money, obviously You're going to be able to do things faster and if you think about serverless Even though there are like subtle differences between serverless and a long-running Long-running program at the end. They're more or less the same So if you keep the container around for longer and if you provide more containers and more resources, obviously things are gonna run fast So if you're willing to to basically spend more money then essentially you can have good number of containers running your functions in a long-running model and you can basically Create boilerplate code along all the orchestration that you need between these functions and you can get something similar The point which the server serverless in general is that it's going to help you significantly in terms of resources that in terms of costs of using resources that you would use So the last thing is that the experiments that we are running We're actually moving towards Creating it something that everybody can use and everybody can run against their environments We're hoping to be able to release both the code for running the experiments and also the actual Benchmarks open source so that in case you're interested you can try it with your with your own environments I Think with that. Yeah The one thing we want to discuss I think we Maybe I should have stress is that a lot of these environment is still in beta. So like what can whisk is in beta as yours and beta I don't know. I'm not so sure We tried to get Google functions and those not only send beta, but you can even use it. So the point is Numbers we're showing you here of beta except for lambda. So keep that in mind but what we want to do in terms of You know, where do we go from here is to release something like a spec server less So to that extent what we did is to we've got a draft It is actually a document. So if you drop down that Google We sent us email we'll send it to you and this is to essentially say there is spec server less is Something we think could be useful and there is gonna be a way to run some experiments around it and obviously Part of the idea is not only to release the experiment runners and the experiments publicly, but to get everybody Tried on their own environments. So that you know, maybe you can optimize it and part of the idea of creating a like a spec is You know how when Intel releases new CPUs then you can run spec ant or different benchmarks to kind of compare the The CPUs and I think serverless environment might have a similar Potential so we'd love to hear your opinion about this If you're interested, we'd love to have you know collaboration. This is very early in the You know in the stage of where a serverless computing is gonna go but this is potentially a way for you to get engaged in and Would use something that would be valuable So with that we thank you you'll find us on Twitter here and then we can take some questions