 I'm working at SAP Cloud Foundry Core Team and I'm responsible for metering of SAP Cloud Platform and Cloud Foundry Part. Hi, my name is Tujan Rachev. I'm also working at SAP as an architect and I've been working on Cloud Foundry related projects in about 2 years now. And today we want to talk about building distributed cloud native applications with Node.js. And in particular we want to touch upon one magical property of distributed systems known as resilience. We define resilience as the ability of systems to withstand, tolerate and recover from failure. And in this talk we would like to look at these issues from several different angles, examine different kinds of failures that we need to tolerate. And also talk a little bit about how we failed in handling them properly but in the beginning but eventually learn to do this better. So this talk is about the problems we faced when trying to work with Cloud Foundry Abacus. And let's dig a bit deeper into what Abacus is. I don't know how many of you were on the session about monetizing your applications from our colleague. But the scenarios we are talking about is that we have a lot of desktop, mobile applications, even IoT devices that work with the cloud. And they work with services, applications, APIs that basically use cloud compute resources. And what we are trying basically to explain is how we do the metering for these resources and how we try to monetize them. Well, we do this with project by IBM, that's Abacus. And this project is sophisticated metering engine that does things like collecting data, aggregation, accumulation across several different dimensions like Cloud Foundry organizations, bases and across different time units. All this data is then collected, aggregated, accumulated as I said and sent to the billing engine. And that allows us to monetize these used resources and they finally end up in the invoice of our clients. So from architectural point of view, this is how Abacus looks like. It's a set of a lot of different microservices deployed as Cloud Foundry applications. The project started as IBM internal project was contributed in 2015 to Cloud Foundry foundation. And we're working on that project for almost a year now. And all these microservices basically constitute a map-reduce pipeline. The first two stages are basically the map part of the pipeline and the accumulated and aggregator are the reduced part of the pipeline. The input is JSON documents that contain the usage sent to Abacus' small JSON documents basically, almost a free form. And we process them through the pipeline and get report back. So collector does storing of these documents into the database, metering this transformation from one unit of measurement to another. Basically the unit of measurements we chose to present to our customers. And this can be regarded as an example bytes to gigabytes. So that's the unit we chose to present to the customers. Accumulator accumulates across different time dimensions like seconds, minutes, hours up to month. So we get breakdown to these different units. And aggregator aggregates across organizations and spaces. So this is how Abacus looks like from architecture point of view. And the first topic we want to touch is networking. The problems we had there, as Tujan said, how we felt to the things and in the end how we intend to fix everything. Well, if you do a search in the internet about distributed computing, you might end up into these fallacies of distributed computing. We knew that we'll have problems, we knew that the network is not reliable, latency will be a problem, bandwidth also will be a problem. The topology of CodeFundry or our application would not stay the same. And we'll have, for example, updates and that would probably, we'll face an issue or two. But knowing and believing in these are different things. And Tujan will present you how we became believers in all these fallacies. Yeah, so what are the issues that we actually faced? So first of all, the network is not reliable, meaning that the connectivity between the different stages of our pipeline may simply be broken at any given moment or unstable. Second, even if the connectivity is there, the bandwidth may be insufficient and therefore the latency may be low. And finally, even if everything with the network is working OK, some of the stages of our application can be down, not responding due to the fact that they have crashed or due to updates. And all this motivates connectivity design that is highly tolerant to all these kinds of failures. How do we do it? It all starts with retry. So if a request doesn't come through, or if we do not get a response to the request, we simply retry it. However, this has to be done carefully because maybe the downstream application is not responding because it's overloaded. And then if you keep retrying, you actually make the situation worse. So we use exponential back off and we increase gradually the amount of time between two consecutive retries in order to avoid this issue. Second, we also use circuit breakers, high-strict type circuit breakers. So I feel familiar with that. Circuit breaker will flip the connection open if retry doesn't help. And will in this way prevent cascading failure from propagating through the system. When the connectivity is back, the circuit breaker will close and retrying and normal request response can continue. We also use two other techniques to increase the throughput. The first one is batching. We batch many different requests into a single one, compressed one. And we can even batch requests that go to different endpoints on the same host into the same request. This allows us to make much more efficient use of the open network connections. And ultimately to achieve higher throughput. Finally, we also use throttling. With throttling, we limit the number of requests that we send up to a certain predefined limit and queue the rest for a certain amount of time. Throttling also allows us to control the amount of data we send over the network much more effectively. And all these different techniques taken together, really, they play very nice together. And they come to a very effective solutions to the network issues I mentioned previously. However, there is another kind of issues that we do not cover with these techniques and they are related to the availability of our application instances themselves. And they may not be available due to various reasons. So, the platform may be updated, the backing services may be updated. The applications may be updated or maybe they have crashed. And here Cloud Foundry helps us because it will restart the applications in this case. And it also allows us to use blue-green deploy and minimize the downtime during updates. However, for data crunching applications like Abacus, we have this issue what happens if the application crashes just before it was able to write the process data to the database. We need to be able to recover from such situations. To do that, we use a technique we call replay. We have an input database and an output database. After an application starts, it will check the input database for any documents that do not have corresponding output database documents. And we will replay the processing of those documents in order to ensure that the issue doesn't happen. Well, even with all these techniques, we still need to have redundancy because in order to guarantee availability, usually that's the way we do it. And while redundancy comes at the cost, one of the things we need to consider is scaling. Thus our applications have the ability to scale horizontally. If they do, then we should consider how much do we need to scale. That usually goes with the cost, so we cannot afford infinite scaling for our applications. And even if we can afford that, we need to discover which of our instances are actually up and which are down and not working. And that goes with discovery. So we need to discover which of our applications basically runs. And to do that, we might face a problem in Cloud Foundry. Why? Because in Cloud Foundry we have Go Router. That's pretty much OK with applications. We have Router. We have a URL for our applications. And if we request that application, we usually end up requesting a single instance. And Go Router does round robin by default. So we cannot tell exactly which instance will be actually accessing. So we cannot basically pull data for monitoring out of our applications. We can of course use techniques like pushing data to some server or service. But that means we need to change our applications, we need to inject some code inside. And we cannot simply pull data out of it. And using some things like Hysterics Eureka is not really an option. So we have a problem there. How we try to solve this? Well, we came up with some idea to basically have index applications or numbered applications. So on the previous picture we had like six collectors in production. And we numbered all of them like 0, 1, 2, 3. So all of them had URLs. So we can easily go and fetch some data out. Of course this means we will have to run them with a single instance. Because otherwise we have still the same problem. So we kind of solve the problem. Now we can monitor them. But we still had a problem. Because in reality we, to access from one application the next one, basically to push data through the pipeline, we either had to use backend service or go through the router. And Abacus currently uses the second approach. So we wanted to minimize the number of use services. And that's really a problem also. Because that means we have all our performance. We always go through the router. Because that's how we can access applications in co-authentic through URL. And router does the lookup. This is also security problem. Because all of our applications need to be exposed to the internet. Why? Because they had to be seen from the internet and from the applications in the pipeline. And not all of our stages actually need to be externally visible. Probably just the first one and the last one, the reporting. How do we solve this? Well, here container networking comes to the rescue. Unfortunately it's not released yet. Or we had the first release, I think. Before the summit. But it's still not productive in the major platforms out there. So let's see how the container networking and in particular container-to-container communication solves our problems. Well, we no longer need to use these index applications nor going through the router. We can use things like Eureka or any service discovery out there that needs direct access between the applications. So we have both monitoring and discovery. What else do we have? Well, we have performance because we basically can access our applications and we have increased security. We can have rules that forbid access from the internet for instance. So container networking is really helping here. Yeah, and our course is written in Node.js Why does this matter? How does this help compared to any other language or platform? And we would like to mention two things. First of all, Abelkos was written in ECMAScript 6 from the very beginning and using a very consistent approach of building upon all the latest features in the JavaScript language available in ECMAScript 6 and especially it uses functional programming concepts such as higher order functions in order to enable very elegant and powerful composition of functions from other functions. The second thing we would like to mention is that since Abelkos, sorry Node.js is asynchronous it makes very easy to use a reactive approach to building modern applications. However, the default callback based style of asynchronous execution used in Node.js If you try to use that in any real life complexity you may very easily find yourself with code that is very hard to maintain due to the so-called callback hell. Fortunately in ECMAScript 6 there is this generator feature that makes it very easy to write asynchronous flow in a way that it actually resembles synchronous flow so without any callbacks just using generators and the yield keyword. But it's still asynchronous, it's still reactive and if you, we can consider for example this line of code in which we compose a function that is actually a request having all the features we mentioned previously so it's retry, it's trotode, it's botched and it uses a circuit breaker and it's also a generator and can be yielded upon to return the result directly into the the response directly without using a callback and it's still asynchronous. Well, we saw that as well of course that's why we wanted to share with you the things in this session and the problem was that we had near terabyte per day in some situation depending on your installation that means a lot of data flowing through Avacus pipeline and since we had really small documents that meant that we had a lot of them as well so how did we manage to basically do that well, we had partitioning that's pretty old technique and if you Google it of course you'll find this characteristic we like it very much because it helps us to manage our data it helps us to distribute the load between several databases to do things like taking data offline and because we use partitioning we can afford basically to have keys generated in our applications on up level we do this because our data is basically time based we receive the data with certain start and end times and we make a key that is basically based on the time we start in Mongo or Couch databases so all of this basically allows us to to delete data we can either take it offline when we no longer need it for example previous month database can be simply taken offline or backed up or whatever you can think of but more importantly we can do things like retention either we can say we still need that data or we can basically say we don't need that data because we have legal requirements like we should not keep data longer than 3 months so we already know that we have partitions based on the month we receive the data so we can simply delete the whole partition and that help us solve these legal requirements you do not pay proper attention to monitoring so how do we do that in Abacus besides using Hysteric style circuit breakers we also use Hysteric streams and these are very convenient because they allow you to stream data about the health of your applications the traffic the state of the circuit breakers and the error percentage and to monitor all this in standard pieces of infrastructure such as for example the Hysteric dashboard if you have used Hysterics in the past you are probably familiar with that so in this dashboard you can monitor a single Hysteric stream coming from a single instance of one of the applications so that's quite good however we have many applications in many instances and we actually want to have an aggregated combined picture and we do that first by registering our applications in Eureka that Christo mentioned already and by using turbine that aggregates the per instance streams into per application streams that are much more useful however there is a drawback to this approach and to actually use it you need direct communication between your applications Eureka the different pieces of the Hysterics infrastructure and that is really you can take full advantage of this if you have this container networking release that Christo mentioned previously and second you need the Eureka client in all of your applications so you need to instrument them properly in order to take advantage of this so what if we wanted to monitor our applications without all this complexity probably we would use for debugging and even monitoring productive systems and what Muzzle does is basically pulling data out of cloud foundry nozzles because they contain not only logs but also metrics and by using it we are able to pull metrics feed them into Riemann and Tim Fuchs database and on top use Grafana to visualize all this and this helps us as I said not only debug when we are developing but also monitoring productive systems impact the metrics are always there so you don't need to instrument your applications we just connect and pull them into database so we can analyze them and if we have all the data we can get something like this we get requests times we get types of requests number of requests that failed we can even pull the data from heistrix and not really using heistrix dashboard but using Grafana we can visualize the heistrix data as well so we can get which circuit breakers flipped open and what this resulted in like error codes restarts whatever we have there so that's really nice we have the metrics and the next thing we want to do is basically to have alerts and how we do this usually we just hook the events to some alerting and that's the reactive approach I believe everyone basically does this the guys from IBM came with pretty nice idea anomaly detection and machine learning we have the first steps in Abacus already and the idea there is to basically build a profile of our resource consumers and resource providers basically the two sites of the pipeline so if you push data to Abacus you probably push it like once a minute, once a second whatever interval you decided and if you stop pushing data that's a change in the pattern so you get instantly alert because something changed probably network is not longer there or simply you don't have usage so you've changed something so probably someone needs to know about that either customers started using kit wrongly whatever happens we need to know it and that's really easy to do because we don't have to do anything basically we don't have already to define some rules or to do something manually when some new customer decides to use our platform and here we wanted to talk about and we are running out of time I will skip actually this part and we come to the takeaways from today's session yeah the first thing for us they pretty hard because we had a lot of problems with networking we had a lot of issues monitoring the whole solution we had to come up with ways to basically overcome the limitations of 12 factor even because we needed to pull data out for applications and container networking helped a lot fortunately we also use Node.js mainly because of the asynchronous nature of Node.js because of the possibility to use really powerful composition techniques from functional programming with the latest language features and last but not least because of generators which allow us to do asynchronous programming and reactive programming and still have code that is highly readable and maintainable about native applications I think everyone a few knows this and what helped us a lot is the container networking that will basically allow us to deploy avacus as a set of processes not really single process applications that will allow us to bring the different pieces together and to increase even more the throughput and will also allow us to make the operations of the whole solution really easier because currently we have around 60 applications and that even for automated system is a bit of a nightmare if something happens you still need to do a lot of retries even on CF push level so we hope with version 3 that would become much easier thank you and I believe it's time for questions we can take a couple in the case of Java we have a lot of sample code basically from big vendor sort of thing and very credible type of code basically either from private or from IBM do you have such services for Node.js for a model or things like that we can get a glimpse of the real working code on which we can build some real GitHub URI or something of that sort the source is already available it's open source project it's part of cloud foundry foundation it's in the incubator at the moment and if you have a look at CF dash avacus you will find it easier we anyway yeah we forgot CF avacus does the name of the project it's in the incubator so there you can see a lot of our modules they are already published in the registry npm so you can simply either download it or contribute whatever you want it's open source ok one last question back here do you have your Grafana dashboards exported are they generic enough that they can assume general CF metrics or are they more so customized to your own app and then be if they are generally available would you be averse to sharing out the dashboards you mean the dashboard we we showed here ok the Grafana is also open source for the bugging and if you want to modify it you can simply have it and play with it it's part of the Mosul project we are doing but we don't have it open sourced thank you very much