 Okay, I guess we are ready to go. Camera seems to be on, it's blinking red, so let's get started. Yeah, welcome to our talk, which is gonna be about ArangoDB running on MISOS. So maybe just as a background information for the history of ArangoDB and MISOS, it's been like a really long relationship and those guys have been really, really helpful in helping us to develop stateful frameworks because they were basically the first framework who were really running on those stateful primitives in MISOS, and you'll also see more about that throughout the presentation, but this is kind of like, we have a really close relationship between MISOSphere, DCOS and MISOS and the Arango guys, and this is actually also why I'm here. I'm Yerk with MISOSphere, you might have seen me this morning already, and I'll give this presentation together with Kunal and Claudius from ArangoDB, and I would actually, do you wanna get started? Sure. Thank you, Yerk. Hello everyone and good afternoon. My name is Kunal Kusurkar, I'm a director of solutions engineering at ArangoDB. I joined pretty recently, but I have a history of working in the NoSQL big data events, processing, messaging, ESBs over my career over the last 12 years. So with that, let me quickly give it to Claudius for his introduction, we can start. Sure. Yes, my name is Claudius Weinberger, I'm the CEO of ArangoDB. I founded the company together with my co-founder four years ago, but we started the project, let's say, five and a half years ago, and we're building databases now for 18 years, so we did a lot of different stuff, customer, customized solutions and so on. In 2012, we sit together and thought about should we build another database, there are already a few in the market, but you will see later that we found something, what we call the native multimodal approach, but from our perspective, adds something new to the market and gives us more possibilities. Thank you, Claudius. So let me start off by asking a question, why distributed data to begin with? Can anybody? Good? Hi, am I soft? Okay, perfect. So why distributed data? That's a question I really want to start this conversation off with. Of course, the topic of today is distributed data and distributed infrastructure, so I'm taking the first part and starting off there. The question really stems from the challenge that what we saw over the last decade or so, if you see in track history, what you would see is we have seen a significant amount of growth in global web, cloud deployments and mobile applications, and that brought with it over a period of next, I would say over the last decade plus, it brought an unprecedented amount of data with it. In terms of volume, in terms of velocity, and in terms of variety, and a combination of those together brought in a certain level of combination pressure that really pressured our existing database systems. In other words, the database systems that we know, the relational databases that we know and have worked well, loyally for us for a long time were just not suited because they were not made for it. The new applications that we were building had a completely different requirements in terms of these three Vs and any and all combinations thereof. So what was the frustration? And then what happened overall was the attempts, as it always happens, is to figure out, how are we going to solve that problem? What do we do to solve the three Vs problems that we have? No matter how much hardware you give or how much you optimize it, the monolithic back ends just wouldn't scale up to that challenge to solve it. They were made for a different era with a different purpose for different systems that were siloed at that time and to work on a limited dataset overall. They were really not made for voluminous data. They were not made for a variety of data. They were not made for data with high throughput and high amount of reads. That is essentially, then at that point, what was realized, well, the solution really lies in a way to think different, to think what is really the missing part to solve that problem? And distribution was that answer and following that distributed data. Fundamentally, what solved that problem or what really attempts to solve that problem now in the modern world is a database, is a native multi-model database that will provide a solution in three ways. One, be able to store and serve a variety of data. That's the word, multi-model, meaning you should be able to store any kind of data, the new applications, the web applications, mobile applications. They're not really just working on relational data. They're working on documents. They're working on key value stores. They're working on graph data. And all of that data needs to be used continuously as the applications are being built and as they evolve. And also between them, which means you, at one given day, your application might want to just use a lot of document stores on a given month or after a few cycles of development where you realize, well, my data is more connected. It's more relationship. I'm trying to analyze relationships more than query entities. Well, I need to switch to a graph data model. The existing databases just don't allow you to do that. You need a fundamentally new database that's at the core designed for it organically. And that's the word. That is what makes it an organic, multi-model, native, multi-model database which will support documents, which will support key value stores, which will support graph data structures to begin with. That is the first way to solve the variety of data problem. Now once we attack that, the second is the ability to scale out across multiple machines to handle the velocity needed, to handle the generally great amount of throughput that's asked of you to solve the number of write operations and number of read operations in a given day. You need to be able to be up for that challenge of solving. Third one is the ability to spread out across machines when you need to handle volumes. Your data set is not going to stay the same when you start your mobile business or a web business and two years down, four years down, you're going to see exponential growth in the amount of data you're handling. And you cannot solve that by scaling up machines, giving more memory to make by bigger machines and absorb a higher cost of ownership. You need to spread that data out over commodity hardware on multiple machines and tackle the volume problem as well. And this database exactly attempts to do that by attacking the very need of distributed data. With that, let me introduce RangoDB. So as I spoke, those were the characteristics, the challenges, the realizations upon frustrations and the solution approach. A database like RangoDB defines it. We start off with that value proposition, attacking that problem, a native multimodal database that will provide a unified declarative query language, a simple word, a single query language to query graph information, to query document information, to query key value information, whatever information you store in the database. Scalable and highly available with configurable consistency. At times you need acidity, you need strong consistency. Other times you're probably okay with asynchronous replication. You're okay with getting a dirty read back because that's what your application needs at that time and you should be able to tune and change it as you need. Extensibility and then lastly, I would say, ability to expose your database operations over a microservice framework. The ability, your mobile applications, web applications, they need, they're essentially built with the idea of microservices. Your data tier over these years have not evolved as much. RangoDB solves that. We expose the database operations using an HTTP REST API and that API is also extensible using the Fox microservice framework and you can write service manifest that will go and push directly the information you write and read from the database exposed as microservices. All together really becomes a modern database that you can use to service your microservices based applications. I'll take a brief pause here before I go a little bit deeper and explain each of these features to you. Any questions I can answer in the meanwhile? Any quick questions? All right, so multi-model, I'll go ahead and explain what I mean by that. As I was talking about different kinds, the V, the variety of data, these are all the different types that we have in the market right now. There are point solutions to solve point problems historically like we have seen the relational world, right? Relational tables that were built to solve order systems, catalog systems, but there were other data types that evolved. There were documents, there were graphs, there were columnar stores, time series, key value databases, all of these different kinds of model. RangoDB is an attempt to solve that problem really well and we have taken the approach to solve be a native multi-model database to do two, three things really well. Absorb graph-like information and serve it back really well, absorb documents, absorb key value stores, all in a single deployment artifact. And what I mean by that is a single engine, a single demon that will encapsulate all this functionality and also provide you the freedom to run on all of this on a scale-out infrastructure which mesos can handle. So what do you get with it? You get a unified query language for all these models, you get a clutter-free, simplified cluster deployment scaling, you get a native multi-model advantage overall and what it gives you from a developer standpoint, what's the advantage? What's the value that you're going to get out of it? A significantly reduced operational moving part overhead, right? There is not much of an operational trouble that you will have to deal with unlike systems that you have seen. Probably there is Hadoop, ZooKeeper, most of these if you're familiar with these technologies bring in a lot of operational clutter, even databases like Cassandra and others, they will bring a lot of operational pain for you to deal with over a period of time. We attempt to solve that with it, with the simplified artifact that you can deploy and deploy across multiple machines and still serve the same workloads without you having to create a new point solution every single time. Faster development cycles. With the unified query language, a single query language, you can query all the information stored in all these different types and even join them with a simple pseudocode-like language. That's a huge advantage from a development cycle, from a time-to-development perspective. And lastly, richer quality applications. With multi-model, with attacking three Vs that we spoke about, you will definitely get an overall advantage in term of building a high-quality mobile application because you have a microservices layer that's really at a data tier. You don't have to deal with it, have to work with it at an application or a front-end side. It's really done at a data tier site for it, scaled accordingly as we will scale the data for you. A significant advantage you'll get out. This is a slightly expanded picture of how things will get deployed from a distribution standpoint. These are the different roles, how we will deploy a RangoDB. The data itself is kept on DB servers, whether primary or secondary, distributed across machines. The microservices can be exposed using the Fox microservice framework, which will run on the coordinator nodes as well. Coordinator nodes can be scaled independently from the DB server nodes, which means you get computational scalability, as well as you get data scalability based on what your application needs are to service your requests. And lastly, the agency is really the consensus mechanism that we have, which will do the job of stepping out of the cluster and overlooking it like a hawk and figuring out how are my coordinators working, how are my machines doing, how are my DB servers doing, and how is the health of everyone? If anybody goes down, it will make sure that it gets automatically started on different machines, or that is a process that you can automate in the way you feel best as well. It is essentially a very hardened way of figuring out what your cluster state is and tackling it as your needs grow. Oh, that is one more second. So, this is actually where Missus is coming in. As you can see, this is like already complex architecture, which I have to deploy and actually also maintain because we're talking distributed systems, there are failures. Each of those instances might fail at any time because the underlying node is crashing, because there is a network partition. So, this is why we actually have to do something about how we're deploying it. And as we all know, we're at MissusCon. Missus is really built for that, but seemingly the HDMI connection isn't really built for it. For, we see already like network partitions happening here, between the laptop and beamer. So, it's gonna happen even more in this big cluster over here. Yeah, so, and this is actually why the Arango guy started out pretty early talking to us in developing like a Missus framework for it. And developing such kind of Missus framework is actually, it's really, really hard, especially we're talking here about a database, developing something, I don't wanna say even like Marathon because Marathon also has like a lot of moving parts by now, but in the beginning it was actually quite simple because it had very little state. But here we're actually talking about a stateful framework which has to be resilient. And so, in the end, you don't have to read all of this, but kind of like the outcome was that we wrote about 5,000 lines of C++ code just to maintain all states. And I think this picture is even symbolizing it better because this actually shows the state diagram we had for all those components. So in Missus itself, they're like low-level primitives for those of you who might have looked at it. It's reservations and then dynamic volumes and so on and so persistent volumes. So it was actually, it was a rather a lot of code we needed to write here and maintain, but now it's actually, it's a pretty good framework. And maybe this is also like one of those key takeaways from writing it. If you really dare to write a framework yourself, I'll tell you on the next slide why you shouldn't, you should actually start out with like state diagram like this, how you want to deal with your persistent volumes next slide. But as said, you actually, you shouldn't be writing your own framework in most cases. Just quick raise of hands, how many of you have either been to the SDK talk or have been to the SDK workshop even? So yeah, you know all of this. So the basic idea is that we don't want people to actually have to write frameworks. We want to generate it for them and this is what the SDK is actually about. So the simple thing we also did, for example, in the workshop, we don't have to do anything. We just have a very simple YAML configuration and we can deploy it. So we don't need any deep knowledge about DCS. We don't need any deep knowledge about the framework we want to run. We simple say run these commands and then the framework will auto-generate such kind of framework scheduler for us. In most cases, also if we're dealing with like a database where we have certain startup constraints or certain recovery operations, it might happen that we actually have to write some code. So with the SDK currently, this is some Java code and I need like a little understanding of DCS. But all of those up here, this is actually, this is something I can probably, the highest uppest layer, the default, I can do that within a week. This is probably somewhere within months and how long did it take us to develop the entire framework? Like nine months or so? Yeah, rough, yeah. So that's actually a whole lot different time scale of developing such kind of code. But thank you again for doing it. It was really, really helpful because it actually helped us build the SDK with all the experience we got from this. Next slide. And now that's actually, we talked about the replicated data part. Now let's just talk about like the replicated infrastructure part. Once the slide is not black anymore, next slide. Yes. So we're talking about data center replication and actually we have multiple options for that. So maybe we should first clarify what we're talking about. So we can either use replication or our goal could be to use replication this is like an off-site backup from where I might be able to restore my data. So this is option number one. Option number two is a little stricter requirements. This is like disaster recovery where I actually want to switch over to my backup. So my backup has to be something which can run and also serve queries in the end. And the latest and probably strictest goal or option could be to offer geolocal services and distribution of data. The Beamer is really fun. So in that case that I can actually serve queries from both data centers in parallel, which is good for example, if I have customers around the world, but it's of course, it's also a lot harder because all of a sudden I'm really dealing with hot master, master replication. So basically like the basic idea here is we have a cluster one, a cluster two and some other zone. And now we want to see how we can actually get that going in which different scenarios up there. Next slide. Yeah, so and for the first iteration, a RangoDB after also discussing with us, we decided to go for just to solve the first two options because for many people that's actually sufficient. I don't necessarily have that much load that I need multiple sites serving requests. So actually just being able to recover my data is something very useful if a data center is going down if an Amazon region isn't available anymore for example. Next slide. And so what we decided as like the goals for this first iteration is to run a database clusters in all data centers involved in this replication procedure and then replicate data automatically. This is nothing what I want to do as an operator. This is something which should happen out of the box and then also be prepared to actually switch over to the other data center. So at time point one, all requests are going into data center one, then a hurricane is coming as unfortunately it's happening too often in recent days. That data center is offline, power is cut and I can actually switch over the request to the second data center and then I can still keep on serving queries from the second data center. So the first implementation we are currently implementing or they are currently implementing is to have this replicated Arango DB cluster which includes basically all user settings. So the user settings there also replicated between both clusters and Arango already has a replication API. So we can, we will utilize that together with a newly written tool called Arango sync and then we actually, we looked into different options and what we can use to ship the data over and we looked at different implementations or how much effort it would be to do it ourselves and our decision was actually to use Kafka or more particular called Kafka mirror maker which allows me to run like a distributed replicated Kafka cluster across two data centers. And that decision is actually it's kind of if you looked at all the other talks and this is also why this talk is actually in the smack stack because of a lot of those architectures are starting to use other tools. As for example, Kafka, here we have if you're talking about the smack stack here we would replace the storage part with Arango DB but it's basically we not necessarily have to write all the tools ourselves because we have now 2017 a really great toolbox available of tools to use. And so we use that and this helps us to write load spikes in one data center because Kafka can also queue them so it can just accumulate events as well and it can then distribute them across data centers and even if there's a network outage we might lose some of the events in there but it's still gonna be running and Kafka can also reconnect. And that was a nice thing is that we can actually or natively implement we don't have to do anything to very little to get that is to implement back pressure meaning if the second database is getting too slow the second data center and just can't handle all the requests Kafka can just buffer that and we have automatic back pressure and we can even then pass it on to the tooling around Arango sync to have further back pressure and not overload the Kafka cluster. The TLS protection, we're shipping that across data centers so probably across some communication line where someone else might be listening so we need to make sure that all of this is encrypted. Next slide. And so this is then basically this is similar to the picture before so just these lines there are basically this is the Kafka connection and so Arango sync is just making sure at which time point we are or how much we have replicated and then basically the Kafka is sending us over the data we are still missing as events. And that allows us if that data center, data center A is failing we might lose some of the events which have not been replicated yet by Kafka so which might be in the DB server but have not been written yet in Kafka slash might have even been written to Kafka but haven't been mirrored yet to the second Kafka instance. For most use cases this isn't a too big issue but it's just something you should be aware of in the first implementation. Next slide please. So and this is actually also the limitation and it's basically that it's asynchronous. So as said in the beginning this is targeting the first two goals of just having a backup and having a data center to which I can switch over but it also has the advantages that we quickly get an implementation which helps to solve those first two use cases but it can help in unplanned outages to losing some of the events in the queue. I say unplanned because actually if we plan our outages if we plan to take one data center offline because we wanna migrate nodes we wanna I don't know create a new cluster then we can actually just wait until the Kafka queue is entirely depleted and then switch over. So in that case we can actually do without data loss and yes it's asynchronous as said in the beginning we don't have any load balancing of query so no master master we can't query both clusters in parallel. Next slide. And yeah that's actually it's a good topic to just talk about what are like the current ways and plans for DCS replication across data centers because the same problem you actually have will have for many clusters that you want to distribute that across different data centers or across different Amazon availability zones, Amazon regions. And you have some options. So the first one could be that you actually start spreading out your masters across those different data centers across those different zones and that actually works rather well if you're staying in the same availability if you're spreading out across different availability zones but stay within the same region. Once you start, so this is like Amazon talk if you're thinking about your own data center or Azure or GCE you just have to translate it to the respective names there. It basically tells you you can't spread them out across low latency links but not across high latency links. And why is this case? This is mainly ZooKeeper. Once you start spreading out a ZooKeeper cluster across high latency links you'll end up in some issues. So you should be very careful with that. And that's actually also one of the reasons why Ben in his keynote yesterday mentioned that we want to get rid of ZooKeeper is just to make that more scalable across high latency links. The easiest thing to spread us is actually agents. So we have a lot of users who are spreading out either MISOs agents or DCS agents across, for example, there were at least one talk today about using spot instances and they don't matter too much whether they have a high latency in between or not. So there are even people running their on-prem data center and then extending that with spot instances for spiky workloads because there isn't any synchronous communication required that's all asynchronous so latency doesn't impact it too severely at least. The plans for the future are also to actually be able to link two clusters together. In the beginning it's just gonna be UI, CLI, so most of that control, it's gonna make it easy to control two clusters in parallel but so there are people working on a linker module it's called which makes it like those clusters aware of each other that it actually makes it very easy to support use cases like that. Next slide which I guess is a demo. Yes, this is a demo. So we can talk a lot about it. I'll spin up the cluster for you and then you can take over, okay? So we can talk a lot but let's actually just see also what ErangoDB can do. We obviously for time reasons we won't spin up two clusters and have the replication in between also as this is not quite working yet. So we'll just show it to you on a single distributed cluster instance but already distributed within one data center. So installing ErangoDB is quite easy. The only thing we have to look out for is that there are actually two instances. So ErangoDB is actually the old one we should add a prefix to that probably but we want a new version ErangoDB 3 so we are installing that and let's just have a look what's happening in the background because for those of you who might not be that familiar we see here is the first thing spinning up. If we look at it, this is actually the framework scheduler. So the framework scheduler is a thing which will deploy afterwards all the other tasks and this is running now so rather soonish we should actually see the other tasks spinning up. This is also why it's still unhealthy because the other tasks are still staging and so these are now the actual task which you're doing the work. The first one is kind of like the controller who's controlling all the other tasks in missiles terms, the framework scheduler and then the other tasks they'll be the DB service and controllers starting up. While they slowly come up we could actually check whether we can already access UI. You're a pessimistic. All right. Service unavailable because the coordinators are not quite there yet. Just wait for one more second. Distributed systems take time but it's still deploying. Let's just wait. Now you need to click on that little icon. I know. Thanks. Just switch back. Do you wanna tell more about the demo you'll be running? No. Now we are coming up. Ah, perfect. Thanks, Jack, for all the stuff you told us about our asynchronous multi-data center application. But what I want to show you a little bit is what would be cluster running on DCS. So first of all, we spin up with two clicks, a cluster on three nodes, DCS cluster. Per default, you're getting two DB servers, two coordinators. And if you just want to add a new one it's really one click. Let's say, and you start another coordinator in the cluster. And what I can already do in the same times maybe start a collection. A collection in a rowing would be the same as a table in a relational database. And you can really define for every collection, first of all, the number of charts. So that's at the end limited scaling. At the end you need one chart at every DB server. And the other thing is the replication factor. And that is also configurable on every collection. So let's start with two for the demo. It's funny. So now we have this collection. This is empty yet. So we can just put in some information. Let's say some by own key in that case. And then just adding some JSON code here. So I'm not used for English to use an English key word. You just want to tell me what I should type? Adjust the name, adjust the shot JSON document. Yes, there's another value, just save. Okay, that's funny. Wow, sorry. So if you go back to the cluster overview then we can also directly have a look at the nodes and also on the charts. So we created this collection. We see now all the potential charts and now I can also move the charts around in the cluster. For example, if I want out of some reason, manually clean up one of the server to maybe do some maintenance on that. I can easily do that on the web interface but also we are a wasteful API. That's all possible against the framework and so on. And what is also possible then if I want to have three nodes of the DB server, let me also show that as the last short thing in the demo. That maybe takes now a little bit more because the DB server is already a little bit more to start than just a coordinator. Maybe also to explain that on the coordinators. The coordinators are completely stateless. They only do the first phase of the query optimization, the query handling in the direction of the application and also do this fox, the Microsoft frameworks. This is something at the end, complete stateless. You can have 10 of them, hundreds of them. It really depends how much you use these features and if you lose one of them, it makes no difference at all. So now we have three DB servers. Let's go also on them in another collection and make a web application factor now three because I have the possibilities for that. Oh, sorry, test is over. Now if you go back to the nodes, now we have the two collections and you see we have a replication factor of two that means we have now spread it over the cluster, the leader and the two followers over all the three nodes. And now it's from, for example, it's not possible to scale down the cluster into one node. You should think about it. Normally it would be not possible but it's something where we say it's up to you if you want to do that or not because it's mean that the end in a situation where you have the leader and the two followers only on two nodes. So in the sense of a high availability, it makes maybe not so much sense but it's something, even if you need that for maybe scaling up the machines or use bigger machines for your cluster, it's still possible. Maybe also give you a short example of the query language. So that is also very simple. At the end it's very similar to SQL. You can do something like a loop in some sense or a collection with documents. Then you filter on some stuff and then you make the projection of the result at the end of the query. So maybe that is also the main difference between SQL and SQL at the end. But what you can also do with SQL is to combine different stuff what we're offering. We can do graph queries and joins and document lookups really in one query. So we have really use cases where people start with a graph database and then the optimization at the end, if you use a runway to be, is to use the combination from joins and graph queries at the same times. So it's a sample of what we're always saying is if you have a fixed length in your path, if you make a query or a graph traversal, it's more efficient to use a join instead of a graph traversal. If you really can combine it in one query, then you get really new possibilities in the direction what is the performance, what you're getting out of the stuff. So once we're offering sub-queries, it's a really complex query language but you can really combine the different stuff. So maybe let's go back to, so we're still trying to bring down the number of charts and to be honest, I missed something to tell you at the beginning. We change that, you cannot scale down the cluster until you have to free a replication factor. So let's abort that, go back to the collections, just delete the collection of the replication factor of three, go back to the nodes, then let's have a look at the charts. We have only the one collection with the replication factor of two and try the same again. And now it's possible to get out one of the DB servers of the cluster. So that is the different of why we choose our own framework in D2S. We really can take care of all the stuff. If you want to take out one server, it's no problem, the framework will move the data if possible to the other nodes. If it's not possible, it will protect you that you don't come in the situation that makes not so much sense. And now let's maybe take a little bit before it's the agency recognize that you have moved the data and so on. It's just what York already mentioned also with the agency that this is a rough based key value store where we really hold the state of the cluster. If you change something, it's more or less like a changing of a plan in the agency. And then we have a supervision code, what really then makes sure that all the things happens. And that's also once on the leader of this roughed based key value cluster. So it's sure that it's only once, once a time. So now you see we have getting out one of the DB server of our cluster and the data are still in the way of how we need it. So it's only on DB server one and three we take out number two. So some questions on that. So that's maybe for now for a short demonstration should be good. Especially as we were running out of time. Especially that, yes. How many of you have used either graph database or document database before? How many of those databases were Orango DB? Okay, half, half, okay. So for me it's actually it's really so my background is actually databases like my PhD was understood with the databases back in Germany. And I really like all those, all databases or all the databases which I allow themselves to call databases. And so I think this is a really interesting space. And just from what I see when talking to customers and users, make sure that you choose the right database, the right data model for your application because it actually it matters a lot how you model your data and how you get your data into those different systems. So just seeing so many customers doing that mistake please think about which database to choose and then also how you to fit your data into that specific database, into this specific data model. Thank you very much for listening.