 Ond wirhwch. Ddwy'n ddwy'n gweithio, ddwy'n ddwy'n ddwy'n gweithio y taeth. Rwy'n gobeithio, ond rwy'n gweithio eich gwerthu, ond rwy'n gweithio eich gwaith a ddwy'n ddwy'n ddwy'n ddwy'n gweithio. Dyna'r newydd o'r ffysgau yn oesbytydd. Rwy'n gweithio a'r gwahfodd mewn iawn. Rwy'n gobeithio i ar gyfer yn Berlin. Rwy'n gobeithio ar y teulu mewn gwahfodd, a i'r museum teulu teulu, So dwi'n ganddo peth o'r awr perthynion Fitcher-Dog wedi amser er Åfr HOur ffwrdd dw'r platform o famlawr a'r tyfnwyr. Mae'r john yma i'r cyfarwyddol yn ddechrau, a'r cyflym o amser. Fe'n olygu, mae'r cyflym o'r cyflym i'r cyflym o amser. Mae'n gweld ein ffwrddwch o'r ddechrau ac mae'n cyflym o ffwrddwch, mae'n dwi'n ganddo ddim yn g épidio. Mae'n ganddo dwi'n ganddo i'n ganddo ganddo i wneud nyf. Ieisioning databases in memory computing with Kubernetes So let's get started So very quickly, what we're going to talk about is deploying distributed databases on Kubernetes We're going to talk about the different ways we can do that We're going to talk about deploying in memory only deployments and with state A wedi gweithio cwpan sicrhau cyrraeth cyfaint a'r tynnu ymwneud ac efallai gan gwybod gwnddoch variants cydweithio. Efallai'n gwybod yn ychydig o'. Y demo gyda'r labdoch. Rwy'n clysig i'r cwp bod du, ac tych yn ymwneud i wedi'u gwnaeth y pwysig i'r cwp yw'r gweith i'r cyflwyff proudi, a ble'n goll! Wel, y cwm dechrau eu gwestiad am ffornig roeddwn i bwyr eich bwysig na gwybion cymunedau? Fe amser yn oedd ffyrdd o'i cyfrwyr deployment scaling and management of containerised applications. What you might not know quite so well is what in memory distributed database is. And while there are a number of different products available to do it, the one I'm going to talk about here is Apache Ignite. So Apache Ignite we define as a memory centric distributed database caching and processing platform for transactional, analytical, and streaming workload delivering in memory speeds at Peterbite scale, and this presents a number of, I wouldn't say difficulties, but differences with how we normally deploy applications in Kubernetes. Kubernetes typically you scale things out in the nodes, the pods sort of don't interact in many ways. You kind of scale them out and each node is different, sorry, is identical and you can add and remove them to scale things without too many consequences. Apache Ignite and other in-memory databases are slightly different, so we're going to take a step back and kind of talk about how they differ from traditional databases and what they look like. As I'm going to talk about Apache Ignite, others are available, they look broadly similar although there could be some differences. Ignite, in-memory centric storage, so the way I tend to start when I talk about this slide is right in the middle, the red bit, the memory centric storage, and although this is like one pretty picture, you can think about this spread across potentially thousands of nodes and terabytes of data, and this is like at the centre of the system. Ignite is memory first, which means that in general everything is in memory, you can process it, you can store it, you can access it really, really quickly, and if we kind of go down on the slide, you can see the other aspect. So being in memory is great, it's fast, you can access it very quickly, but what happens when memory goes, when the node goes down, when you need to restart for some reason, you potentially lose your data. So we've got two different ways of kind of keeping that data. Firstly, we have Ignite native persistence, and we also have third party persistence, so you can kind of save your data into an oracle or a sidebase or a postgres or a MySQL database. And the other advantage of having this external disk base storage is, of course it's slower, but disk base is much cheaper than memory, so you can do things like storing historical data on disk and just the stuff you want to process really quickly, keep it in memory, and you can still access it using exactly same APIs, it just works a little bit more slowly, so it's kind of that hybrid, and as we'll see in a minute, this fact that we store stuff down to disk has an extra complication when we're trying to deal with things in Kubernetes as well. Nothing we can't solve, but it does add like an extra wringle. Then we kind of go up and onto like more kind of client side applications, kind of building on the memory centric infrastructure. So the simple one is key value store, we have a key and a value, it goes to the grid, comes back to the client. Very straightforward, very much like many other key value store databases. And much like them, it's going to spread out across the grid, you don't need to know which node the data is on, it's kind of nice and simple. One thing we have is SQL, so you can access all the data in the grid using SQL, and that's again distributed across the entire thing, you can do joins across nodes if you want, although that's much slower than doing it on one machine. Again, it kind of works from the client side and can do your activation. We do transactions, compute is an important one, rather than most databases, you take the data on your client application, you go to the server, get the data you want, bring it to the client, do the processing, send it back, the results back to the server. In Ignite, you tend to send the compute to the data rather than the way around. So it works much more quickly because the data is already there, it doesn't need to be transferred across the network or all that stuff. We also have services, so again you can define your APIs, how you want to put it, so like microservices within the grid. Streaming data, a lot of the fast moving data, the stuff that you want to kind of deal with in memory is sort of real time or near real time, and Ignite can deal with that, and machine learning, because we have all the data in memory, we can access it really fast, we have lots of compute because we're in a grid with huge numbers of CPUs, we can do machine learning really quite quickly and kind of do it in near real time as well. Obviously this is a horizontal application, there's many many use cases and we have many clients in different areas, so financial services is an obvious one because low latency requirements, high volumes of data, but there's many other use cases, telco, travel logistics, e-commerce, anything that has internet scale, I'm not sure I like that term, but internet scale, kind of huge numbers of records, petabytes of data doing things in a grid. So in terms of Kubernetes we have a few challenges. First thing is we have lots of nodes, Kubernetes can handle that, that's totally not a problem, it knows about pods, it knows about how to distribute things around the actual physical machines, that's kind of great. But when you deal with something like MySQL or Oracle, you tend to have like a bunch, like a data, a volume, and you have one or more kind of compute nodes, kind of pods that talk to the data. And when you scale out you tend to just add more nodes, the nodes don't need to talk to each other because they just talk basically directly to the disk and everyone's happy. That's not how we need to work in a kind of distributed world. In a distributed world the nodes are kind of independent, but they also need to know where each other are because when I as a client go in, I connect to this bottom node and I kind of say I would like the value related to this key, that data might not be on that node, it might be over there or one of the other machines and we need to be able to find out where that data is and go straight to get the data so we can kind of pull it back and do the compute. Or the other way around when we're doing a computation we want to be able to send the computation to the right node and all the nodes need to know what the other nodes are doing in order to kind of link them together and figure out what's going on. So the first kind of complication is the nodes need to talk to each other. Unlike traditional databases again we don't have one big bucket of storage that is accessed by all the nodes. You could do that but you'd lose one of the big advantages which is kind of scaling out. So if you've got one big chunk of data that's your single point of failure, that is your area that's not going to scale to the same degree as the number of nodes. If I add another node I get another disk and that helps me scale out much more quickly and much more seamlessly. So each node has a persistent store as well and every time we add a node we need to add some more storage as well especially when we're dealing with the persistent store as we mentioned before. And of course we also want to be able to dynamically add and remove nodes. That's easy, right? We do that with web servers and things all the time. However, not all the nodes, all the nodes are capable of doing the same thing. They're all ignite nodes, they all kind of store data, they all can do computations. But of course we don't put the same data on all the nodes. That'd be quite wasteful if nothing else. So all the nodes are equal in terms of the code they run, but they're not equal in terms of the data they have. And if you just randomly shut off a node, you tend to, in this case we shut off a node, you might lose a quarter of your data unless you're careful. So we need to be able to decide which nodes we add and remove easily. And Kubernetes has the facilities to do all this, but the way we normally do it with web servers and so forth is not how you do it in this kind of distributed database kind of world. So how do we do it with Kubernetes? So lots of stuff is exactly the same as before, but we need to make some decisions ahead of time before we kind of start writing our YAML or JSON files. So firstly, the database is a set of pods, as you might imagine. We have the code, we have maybe your application, it's kind of built on top of that. We kind of, Kubernetes provides a lot of the information for you. So IPs are signed dynamically, auto discovery is needed. We're going to talk about this soon. So how is the application deployed? Well, is that within Kubernetes? Is that outside Kubernetes? Again, it's not something that really matters to the distributed database, but obviously matters in how you deploy your cluster. We obviously need to expose a certain number of ports and things like that if it's not going to be within Kubernetes. We're not really going to deal with that in this talk because the way it deals with that is very much the same as with other applications. And finally, one of the big deals, things that we need to think about is whether it's going to be stateless or stateful. So obviously we maintain state in a distributed database, but the way Ignite works is we might not be fully stateless. We have the state, but the permanent state might be stored in a database. Remember a couple of slides back, we talked about the underlying persistent storage and one of the modes we can work on is with a third party, like a legacy database, so we could run on top of MySQL or Oracle. And the thing about doing that is if we switch off and restart the database grid, our database grid, then basically the data comes back and we don't need to worry about it. So it's effectively stateless and we don't need to worry about it quite so much when we're building a grid this time. But it's something we need to think about ahead of time. So all the nodes, when we deploy them into the grid, need to be able to find each other and clients need to be able to find the nodes in the grid too. How do we do that? Well, Kubernetes knows where all the nodes are. It has its own DNS service, it knows where all the bits of the components are. So it keeps a track of all the Ignite pods and it also operates as a gateway for remote apps. You can define a service, it can go in and has all the load balancing and all that kind of stuff, you can find a node within there. But the nodes within the grid also need to be able to find each other. And how do we do that? Well, you kind of need a bit of glue between these systems. So Ignite needs to be able to talk to Kubernetes and vice versa to connect the two sides together. So node one can find node two and we can add and remove them on the fly. And luckily Ignite comes with one of those. So firstly, we start up a service, a very simple service, just something you've seen before. We have, in this case, we're going to talk about having a load balancers, it's available outside. We have a number of ports that we open. Again, pretty standard stuff. So in the case of Ignite, we have a REST API that you can access. SQL, so you can connect your JDBC or ODBC clients to it. And we have thin clients, thick clients, there's all kinds of stuff. These are just examples of the kind of thing you might have. And we have, we run the kuba control create and kind of create the service that way. All kinds of things you're very familiar with. And the way we kind of tell Ignite about Kubernetes and kind of vice versa is using this configuration option. You don't need to take pictures of this. It's all in the documentation. It's kind of easy to find. Ignite uses a spring framework to configure stuff. So a lot of this, if you've ever used spring, should look pretty familiar. Basically, what kind of saying, we're configuring Ignite. We're using this TCP discovery SPI. One of the nice things about Ignite, it has huge numbers of SPIs, kind of pluggable architecture so that if any kind of bit you don't, it doesn't work quite the way you like, you can plug in your own implementation. One of the ones we have here is plugging into Kubernetes. So the IP finder, which is how the nodes find each other, we're going to say we're going to use this TCP discovery Kubernetes IP finder. And what we're doing is we need this Nenspace and Ignite. And what that goes to is goes back to the service. And we call that Ignite and the two work together. So each node knows about all the pods. In that kind of configuration. And obviously that way you can have multiple clusters as well if you need just to have different service names connecting to each other. This is distinct from the usual way we configure it. So normally Ignite connects to other nodes. The default out of the box for development purposes is multicast. So it kind of comes up and kind of pings the network. Kind of says anyone else around that looks interesting. The other thing we can do is have a list of nodes in the configuration file. We can have JDBC database access to the list, the list, a bunch of others. Obviously Kubernetes because it knows about the nodes. This is simply the way to do it. But of course all the nodes, not only do you know where the other nodes are, but it also needs to know this configuration. So how do you get the configuration to find the other nodes into the node as you start up? And the short answer to that is use a config map. So we take the quite lengthy Ignite configuration file and we just add it to a config map. Here I'm just taking it from a file and put that as a config map into Kubernetes. Then every node can access that in their file system. Again in the demo I'll kind of show you where this is and what it looks like. But you've probably come across config maps. This is the simplest way of getting the configuration across the entire cluster. You can use the same configuration file for servers and clients. So you can just have the one. Sometimes people use slightly cut down versions for clients. Just a design decision doesn't really matter hugely. So we got a node, we got a configuration. Next, the next decision we need to make is how we're going to persist things. So we have a number of different modes. We've kind of touched on most of these already. So from the top, in memory. So we have a data grid. We're just using the... If you remember the diagram earlier on, we've got the red in the middle. So everything's in memory. You have all the APIs, all the machine learning, all that cool stuff feeding in. You can compute, you can do your key value stores, your processing, feed data in and out, all that kind of stuff. And that's it. Switch off your grid, you lose everything. We use this quite often because anything in real time, often what happens is we feed the data in, do processing, feed it out and it's persisted to an external store or something like that. Source seems pointless to have an end memory database and not store it anywhere. But just because we're not storing it doesn't mean the data is not stored. So this is often used. The other use case we see for this is when it's literally just a cache and the data can be recreated or it's not that important, maybe not the right phrase, but things like web session caching. In order to lose the full memory, the entire cluster needs to go down. So that's pretty rare. So if your cluster goes down and comes back up again and all your customers have to log in again, it's not great, not the end of the world. You're not going to lose everything. The data still comes from your database. So that's a fairly common use case. The next one we used to see quite a lot in the past, maybe less now as Ignite and in memory database in general become more established. So in memory in a third party database. So effectively we have a third party database at the bottom, that sucks data into Ignite and people access the data through Ignite. What this allows is you can scale out. You can get much faster performance because most of the data you need is in memory. It's much more effective than hitting the database directly. It takes some of the load off the database which is kind of useful. Then it goes further down in memory full copy on disk. So this is based at the top one except when you switch off the cluster the data is still there. The last one is the one I mentioned earlier on where you've got terabytes of data on disk and maybe it's today's data in memory and Ignite transparently just brings in the stuff you need. So unlimited data scale beyond RAM capacity. RAM is cheaper than it used to be but it's still kind of pretty expensive. We often see this last one in machine learning because sometimes it's useful to have the full history to do some computations but just today's to like the ongoing work. So when we're dealing with things in memory maybe the simplest way of doing it is just using it again a deployment. You might prefer to be explicit about using a replica set but broadly speaking doing a deployment kind of works because if you shut down or add a node it can deal with things relatively seamlessly. And yeah and that kind of works quite nicely. So again this is unlike MySQL where everything is kind of singleton. Let me start again. This is in memory installation so you can use a deployment for that because every node is sort of equal and if you configure your data to be spread across all the nodes but with a number of replicas backup nodes if you shut down one node at a time the data will be rebalanced on the nodes you do have and you won't lose anything. So it doesn't really matter which node goes down and comes back up again. The problems come later on when we're dealing with a stateful deployment. When we're dealing with a stateful deployment we want to make sure that we don't lose any data because this is our real official data store right. We don't want to shut down a random node and expecting we're going to do this other node later on. We need to kind of understand the order things are going. So we don't when dealing with a persistent store this model doesn't work. The traditional kind of singleton instance where like my traditional database kind of doesn't work. Replica sets which is basically replica set doesn't work. Demon set should be used when you have a single copy of your application must be run on all or a subset of nodes again. That doesn't really work in this case. What we do have and it's been out for about a year in production I think is called stateful sets and stateful sets are nice because they're sort of like replica sets but they define which nodes get affected when you scale out or back. You can also do things like dynamic volume provisioning where when you create a node you can create storage with it. And that's kind of what we would suggest use when we talk about stateful deployments. So here I've got a stateful deployment. We can define the node where it's mounted, how much data we need and the stateful set will kind of decide how we scale out. And the nice thing about it data is persistent to disk ordered restarts. So when you start up a cluster it starts off with node zero then node one then node two until your full set is complete. And of course you can have separate volumes. So the way we persist data is very similar to legacy databases so we have the data and the indexes. We have transaction logs and you can still do snapshots and backups and all that kind of stuff. And again we would typically put them in different partitions so we can kind of manage them differently because they have different usage patterns and it's better to kind of split them out like that. One of the other differences when we're dealing with a cluster that persists data to disk is that it suddenly becomes important that the cluster knows how many nodes are in the cluster. So when you're dealing with a memory you can assume that all the data you have is all the data you have. Because if it's a memory you have it if it's not you don't. If you have data on disk and it could be on any of your nodes if that node is down for some reason does the data you're requesting exist but is on the node that's switched off or does it not exist at all? So I do select star from a table where value equals 10. Is 10 on the node that's not working or does just that value 10 just not exist at all? Without this kind of known set of data this known set of nodes we can't really answer that question we can kind of give you a best guess which is like sure it's not there now but it might be in the future not the degree of certainty we normally require. So one extra wrinkel we have to deal with when dealing with stateful data in Ignite is we have to activate the grid and what this does is it creates what we call a baseline topology and what that means is we started up the grid we have 10 nodes however many and we're kind of saying this is the number of nodes you should expect to see in the future so if next time we bring up nine nodes and someone requests some data we should go back and kind of say sorry I don't know the answer to this and we might not have everything we need and where we do that is we connect to the grid we run this control script and activate the grid and that kind of says bum what you see now is what you should expect to see in the future and we can also do things like add and remove nodes from that grid which goes back to the stateful set where we know which node is going to be added added and removed when we scale up and scale down so I realise we're running slightly over so I'm going to skip over a couple slides here but management and monitoring pretty simple so Kubernetes has a dashboard that you can kind of see what's going on in terms of the cluster at a very high level it's like we've got 10 nodes up and running they're all happy obviously in your configuration script you can do things like monitor whether the server is kind of happy in each pod and you can kind of do things like check is the system up and running is it happy bunny that kind of thing and Kubernetes will display that in the console Ignite and Griggin also has a web console that you can plug in and you can put that into your cluster and monitor the grid at a slightly lower level so in this case you'd be talking about how many compute jobs are running how much data is in there how many rows is in this cache are we running this machine learning job are we doing some compute are the services up and running so it's at a slightly lower level you can run SQL queries in it that kind of thing and the web console kind of doesn't care where it's running doesn't care whether it's running on bare metal in a virtual machine when a Kubernetes cluster so I'm going to do a very quick demo I'm going to do it all from the console on the from the web console just because it comes out bigger on the screen it's a bit easy to see find the mouse pointer okay so as you can see just before our session here I brought up a simple cluster in using minicubar because I wanted to run it on my laptop so don't expect huge amounts of data to be available very simple so I set up as we mentioned before I have my config map this is just a very simple configuration file for ignite so on the top you've got all the boilerplate all the XML stuff in the middle we have a very basic thing so like normally when ignite writes its data it writes it to the ignite directory obviously you want your your pods to be as stainless as possible we want to be able to remove them and bring them back not trying to change them so what we're going to do is you create a work directory ignite work and that will put things like our data files as well as our logs in that directory the next bit here says we're going to enable persistence so persistence is true and then at the bottom we have our Kubernetes discovery thing so each node can discover each other next we have the service in this case very simple I didn't do the load balancing here because we don't need it for this training session it had two nodes in it which are defined using this stateful set so if we look so replicas 2 we called it ignite we're using we have this ignite config map and we're kind of mapping it into the file system we're using the docker image Apache ignite ignite version 260 which is right now is the current one we have a few ports here do interesting things in the grid again I'm not going to go into the full amount of detail the config URI we kind of tell it where the config file is and then the storage options so here we have the config and we have our storage we're both going into this ignite directory at the top level and the other interesting thing is that we have the volume claim templates this is so when a new node starts up it can find some new storage in the new new pod as you can see all the nodes that pods are up and running so everything should be pretty familiar here so what I'm going to do and bring up is that big enough can you see that I'm not sure how I can increase the size so let's look at the logs first most of the top half you can ignore the interesting thing is kind of at the bottom here so firstly ignite cluster is not active remember we have to activate the nodes activate the cluster so it knows how many nodes are in the system but at the bottom we can see here topology snapshot we have two servers so using the Kubernetes IP finder the two nodes have found each other we can't do anything yet because it's not active but they know they're there and we can do and we can start doing things so firstly I'm going to do there we go so now the cluster is active so that means we can do things like it's nothing like doing a live presentation to show how bad your typing is so we can do things like create table there very simple we've created a table with two backups which means when we shut down a node the data is still going to be there regardless and then a select star from and found all the data so just what you kind of expect but we're doing it in a grid and then when we go back to the stateful set we can do the scale out we can add a new node it's just starting up there we go a new node as you can see got zero one and two two has literally just started seven seconds ago so if we go into the first one hopefully we should see where's it going servers equals three so we've added the new node but what we should see so here we see how we have our baseline with the two nodes and we've got a third node that the cluster doesn't really know about it's not really trusting I don't want to use this right now so it might use it for things like compute but it's not going to store data there because it doesn't know how long that's that node is going to be around so what we can do we can add the node to the cluster there you see now we have three nodes in the cluster and you can kind of do the same thing scaling back so you can do the scale down go back to two nodes and because we have the back up equals two when we take down a node the data is still going to be there and we can kind of take that node out of commission so I realize I'm running a bit over so I'm going to leave that for now but you should get the idea it's kind of pretty straightforward and that's kind of it I know that's kind of a quick tour through how you run it but basically the tricks are this Cumares IP finder so all the nodes can find each other and stateful sets is the other one that you might not be familiar with it's a relatively new Cumares feature but it's kind of critical in making something like this work you probably could fake it a bit using replica sets but it's a bit harder to kind of manage because you don't know which node is going to kind of go it's effectively kind of random if you want to find more information about Ignite ignite.patcher.org is the obvious place that's the kind of centre there are a bunch of readme's documentation on the patchignite.readme.io site the full set of documentation including the Cumares deployment instructions and if you're really working to dig down with the details all source code is on github and the interesting place for the Cumares bit is in the modules Cumares directory I got a bunch of information with Cumares open running book which is going to be very helpful and yeah I work for GridGain GridGain is the company that basically supports Ignite commercial versions of it so everything I've talked about today is in the open source version you get it all there but if you play around with it and need kind of professional support where the people to talk to so that's me unless anyone has any questions well I'll be around for most of the rest of the day say hello if you see me other than that thanks thank you very much for your time hope you found this useful