 Hei, rhai gyntaf ydych chi? Yn mwanolwyr murwyd i fanylwyr hwnnw'r gysylltau a'r wych. Yn ystod ddiwydur yn cyfnod i'r ddweud y rhan. Dwi'n twfyn nhw Mythbeth – llawer ei wedi заel effaith'r ddaf yn ymddangos hwnnw. Yn ymddangos hwnnw'r collaidd – mae'r holl o'r ydydd amddangos hefyd yn cael ei gael. Fe feddwl rydych yn wir i gyda'rllai a gael. El wasddai'n hynny, Nigel. It from the bunny and the thousand foot drop on that. I would just give you warnings. There's going to be some audience participation. Nothing too embarrassing, there might be the arm raise he's here in there. And we've got a demo app which I expect some of you will want to get your phone out. There's a bit of a bribe, an awesome prize to give away for one of your village winners. So there's another has quizich we'll need to put on naming. So, if you are British. Or a little bit braver shouldn't you say. If you don't want to put anything rude in, put a number in because the last thing I want to do is say, hey, something such she ever won a prize. It's happened this week. So just remember we're on video. It's been recorded. Right. OK. So I'm David G. I'm part of the solution engineering team for Senedia. This is Tom Hatch-Peartrick. Did I pronounce that right? It's good enough. OK. We both work for Senedia. I'd say we're in the Nats team, but the whole company is kind of the Nats team. So Senedia CEO is Derek Collison. He's the creator of Nats. So back in 2010, he created it and then rewrote it in Go in 2013. So we've got the absolute luxury of working with some of the best brains in this kind of area. And Nats is one of those technologies where if you ask somebody, do you know Nats? They go, yeah, we know Nats. And it's kind of like going to like one of those, or you can eat world buffet restaurants where you've got 40 or 50 different cuisines. And if it's like me, with my chiseled figure, I go straight for the Chinese noodles. I love them. I go, hey, did you go to the restaurant? You go, yeah, I ate the Chinese food. What about the Korean? What about the vegetarian? So you have a main dish and you take small spoons of all the main dishes and you put them on one plate. And Nats is kind of like that. Hey, you know Nats, yeah, we use it for a quest reply. Hey, we use it for pub sub. And it's not that Nats doesn't do any of that poorly. I think it's just that humans have got this kind of mental state to pick something for one job and then tie all the things together to build a full architecture. And what I want to do is leave you with a sense of simple. So when you leave this room today, instead of just thinking about it in one way, you'll think about building things very, very simply with fewer components rather than more. Again, I've asked about using sensible names. So what we've got, if you're interested in doing this, please grab your phone, scan the QR code or go to this app here. It's really simple. We don't capture anything dodgy. The source code, you can read it later if you're really worried about it. Please put a sensible name in again. I don't want to be offending anybody or upsetting anybody because I'm the kind of person that will just call it out. So you've been warned. And don't worry if you don't catch this, the next slide has also got the link as well. What I'll do, I'm just going to do this. This is what I love about conferences. You lose the ability to type. It's amazing. How familiar are you with Nats? Well, let's do this. I'm a complete newbie. I've been with Senedio for four months, so I'm still learning some of this stuff. All right. And what you should see is a graph. And this is actually a WebSocket-driven application using Nats in the background. We're using the demo Nats system to collect all this data, and then I've got a local server running on my machine as well. So I'll collect some of the answers to the questions, and then later on I'll show you some of the things we can do. But this is all live. It's hosted on Vasell. It's very, very lightweight. Right. I'll get back to the talk. Okay. So we've got a saying. There is a Nats for that. And as I opened up with, is it a messaging system? Is it a broker system? Is it a connective data substrate? It's all of those things. So Nats has got this ability to deal with request, reply, semantics, and pubs. So we've got the persistence layer, which we can just turn on, on the same binary. Once you compile it, it's about 17 megabytes give or take. So we can do streaming. We can do key value. We can do object store. And then we can build these really, really complex topologies. But if there's one takeaway, Nats is a phone system for applications and for software and for microservices. The idea is that we can have these named conversations between one or more machines. Robots can pick them up, have a lovely conversation, and without persistence, when the conversation's over, the data's gone. But then we can turn persistence on. And then we can use it for caching. We can use it for deploying objects on the same system. We can replay those conversations. And because it's really, really simple, we get a lot of different ways you can use it. I think this is the big problem I opened up with a weird noodle analogy. It's really, really powerful. I think that's one of its biggest downfalls, and it's well worth explaining. So you think, OK, if Nats can do all these things, where do we go? How do I get started? Well, we've got 11 official client libraries. And that's got very wide to almost full coverage. Yeah, 30 plus community contributed libraries as well. If there is a language, chances are we've got coverage. If there isn't, again, it's open source. So if there's something that's missing, feel free to either raise an issue or do a PR. So if that's how to get started. So when we talk about client, you take your code, you glue a client library to that, and then we can talk Nats. Now, let's talk topologies. I've been to a few demos this week, and I'm sure you will have. And I've not really seen the word simple used very much. Every stand you go onto, it's kind of like, hey, we're going to take this centralized thing, and it's going to deploy this, and this, and this. And look, it's magic, and we get this lovely graph. And most of it's management plane or control plane semantics. So you have to hope that it's interpreting a DSL in the right way and then giving you the right data back. And think, ah, I've been here before. This is reminiscent of a few other problems. So with Nats, we've got this lovely idea of simple. So you can take one, Nats binary, give it some configuration. You could deploy three of them in a region to get a cluster. That's kind of nice. They gossip to each other as well. That means when we get a client and it connects up to Nats, we only have to tell it where to go for one of the servers. And as soon as it's connected all the other servers go, hey, we've got all these other gateways as well. So we get all this lovely kind of simplicity. Now we can turn on TLS so we can encrypt the TCP sessions between the client and Nats. DNS is much simpler. We only need to have a few DNS entries actually for the core servers themselves because clients effectively call home. And then we end up in this position where a client can then start an unsolicited bidirectional communication with something else which perfectly serves this kind of microservices approach. And you might find it a little bit odd on the right-hand side where I've got an ISP called out. How many of you have been roaming and you've gone to a campsite and you think, man, this connectivity is really quite terrible. You're losing these sessions and everything else? I think probably a fair few of you. I spent a long time in networking and the amount of multi-layer Nats systems, not Nats, but Nats, not to confuse everybody in the network address prevention. IP starvation technologies. So you end up in several layers of private addressing before you even get out to the edge. So when we start talking about IoT or the pervasive edge, like a connected floodlight, how are you going to turn that thing on and off from a cloud? The cloud can't call in this thing. It has to call out. So the whole thing, what we used to do is do things like firewall punch through. That's not going to work. The ISPs can only hold so much state. And even in this scenario, Nats absolutely shines. I was going to mock the conference Wi-Fi, but I'll be honest, it's been pretty good. So if we can connect easily, we can build these powerful topologies. We put the clients and the code, the next step is, well, now we're going to operations. This sounds really, really complicated. What do we do? Well, Nats comes with the notion of batteries included. We can pull a tremendous amount of data out of the Nats servers. So we can pull it from a management tier perspective. And I'll just give you a bit of an example here. You know what I've done? I'm going to jump out of here. All these tabs that you can see here, this is live coming off the demo system. So there's no trickery we can go through. I'll just keep refreshing the tabs. We can put all this onto a control panel. We can build an ICUI. You can see what's connected, how it's connected. We absolutely hide nothing. So you can build all of these lovely graphics. Whether you use Prometheus, you can turn Prometheus on for Nats. You can build or use a Grafana dashboard if that's a tool of choice. Or, like the screenshot on the right-hand side, you can build something yourself. Maybe you can take all of that data and give yourself some custom views. So sometimes you wonder how things communicate with each other. How well are they doing? Nats has got this ability to load balance. So you can see how things are clustered together. Well, imagine a cluster between the US and the EU. You might have services deployed in both. But actually what you want is you want the nearest service to respond to a request. Nats can do that out of the box. No load balance required. But you might want to see how that's doing. We provide enough data for you to do that. So we think, okay, this is kind of interesting. We can simply deploy very, very complex communicating systems. We can give them persistency. But experience in operations, every time we get a new problem, what do we do? We go, ah, new problem. I'll throw a new system and new service at that. So the operations team has got something new to learn. And security has got something new to learn. It stacks. And I'm going to be very careful with words here. But we get this effect of compound interest. We can reduce the number of components that Nats can otherwise cater for. And again, this isn't a case of the noodles are great and the chips are crap. This is Nats can do all of these things really, really well. And if you don't trust me, just take it for a test drive. And we've also got documentation we can share with you. It really genuinely costs less to run Nats. So as we progress this story and the current trends, so we're here at KubeCon, microservices are still the kind of burning rage. So we've now built in this thing called a services API into some of the client code. So the idea is you can build some very, very simple services with some callbacks. You tell it what subjects to work with. You give it a schema so you can have a request and a response schema. So the system actually validates the payloads that come back in. And this isn't on the core server. This is right in the client code as well. So we end up with, I think that's readable. Would you agree that's readable? I've not actually stripped out much from this. This is pretty much off a live demo. Self-documenting is always a bit of a sin. I'm not really going to say that. I'm just going to reference it. I'd say we can read this and any developer coming along can go, hey, we know what these things are doing. So this is integrated into the Nats clients. It is an experimental preview, but Jeremy on our team has got videos on this as well. So simplicity meets microservices, ease of connectivity, ease of operations. So then how does this look like in a kind of Kubernetes landscape? Well, I think we have like all good things. It depends with sufficient thrust. You can make a pig fly. Not that you want to see the end result of that. But we've got numerous options for architectures. So we could have pods where the containers call out to a centralized Nats cluster. Or we can take the very same 17 megabyte binary, put that in the pod, and have everything talk east to west through Nats, and have that one Nats server call out with TLS back to a cluster. Ease of networking, dare I say, having been networking for a good decade or so, working for vendors selling complicated network overlays, you don't really need a network overlay. We can get Nats to do all the transport. We can collapse the control plane and the data plane into one system. So yes, you're going to have some ingress points, but the rest of the communication stack can be very, very easy to run. Right, I'm not going to hand over just yet. What I'm going to do, I'm going to flick over to see the kind of data that's come back in. Hopefully you've all given your best. You put a name in, and let's see what's going to happen here. Right, so first thing we're going to do, let's see how many of you have ignored the advice and put a stupid name in. Let's get them. Oh, okay, that's not very many names. Ah, whatever. All right, I'm pretty pleased that there's no weird ASCII phallus symbols, which is always a bit of a relief. I did mention that there is this notion of low-balancing building to Nats as well, so everybody right now that's got your phone open that you're viewing the app, just make sure that you've got a live page running, and this will be your tester, because if your name comes up, this will be great. Right, hopefully what you're going to see, and I just need to make sure this is not me. If it's me, it's not a fix, I absolutely promise. Pascal, where's Pascal? Right, I remember you said I've got a price for you. You're going to take it home. I need some, like, game show. Ooh, thank you. Pascal, brace yourselves. He's going to take home some socks. You see what you missed out on? Yeah, I thought you'd say that. Okay, but the other thing is, because, again, I've gone, we just don't have enough time to do, like, any of this, any kind of justice, like, whatsoever, however, you can use websockets to communicate with Nats. You can go east to west, you can embed Nats in your applications directly, you can stand up really severely interesting architectures like stateless backbone and stateful edges, and it's dynamic in the terms of you don't need a lot of IAC to do these things, you deploy very simple components, you deploy very simple state, and you can build some incredibly powerful things. A lot of this functionality and a lot of these features you can play with yourselves, so we've got a great website which is Natsbyexample.com, and if you now look at your screen, I've just sent you there through the app. So I'll leave you with how to get started, the examples. Hopefully this bit has gone actually way quicker than I thought it would go, so either I'm talking really fast or time is doing this weird kind of relativity and it's bending, I'm not really sure what's going on. Have we got any questions? So I'm here to help, here to answer things. Any questions, any grumbles? You're going to get your socks in a minute? Unless it was a different question. When we are trying to achieve something like an active passive cluster, so we have a customer who has an active cluster and a passive cluster, he has no storage shared, so it's pain. And when I'm trying to achieve this with my Microsoft, so I can send in my actual active cluster stuff into the Nats messaging, and hopefully the passive cluster is actually up and not recreated, is it possible that when the cluster is switching, it's over? So because they have an automatic failover, it doesn't matter. Is it keeping up and connecting to the other cluster even when the local Nats on this other cluster is currently not available? So I have my applications on both clusters and not both Nats clusters are up because I saw it on the slide with a line but with a greater line. I think this is fairly nuanced. Can I rephrase a question? If we take a scenario and you need to confirm this for me, if we have two clusters, so we've got cluster A, cluster B, I can do it way around, does a client connect to both simultaneously or will it try and connect to both? Or do you just connect to one? There's a load balancer in front of it. Interesting. The scenario is where active, passive is in play because what typically we do is everything just stands up and works. In that scenario, maybe we talk about this afterwards. It's a really good question. I'd love to know the psychology behind the active standby, whether it's enterprise design that's kind of like bitters all on the behind. Maybe we talk. I'll deliver your socks and we can have a chat. But in essence, so flicking this round, don't do that. Keep it all active active. I'm proud on the same system unless you're wanting to exercise operations in which case you can literally stand up like a global cluster on your laptop. It's quite impressive and you can play with all the semantics. In your case, you could probably do TCP checks against the endpoint. If the system is up, then balance to it if not, then don't. But I think you're probably causing yourself many headaches by going down that path. But it's a great question. Anybody else? Wow, see you later. OK, in that case then. Oh, over to you. Mosh, enjoy. OK, so now we will go to the part when we will discuss what have changed in more or less last year since the Valencia Qubicon and especially last to the 10 release. As I've seen in the app, many of you, which is also really great to see, are new to NATS. And because of that, most of those features we show here are kind of advanced for advanced use cases. So I'll try to introduce them a little. And we'll start without call outs. And before I go, I think that's one thing to just refer what you said about the noodles that people get in and then just want only this part of only that thing. Because of how NATS is flexible and unopinionated, I think you could say, there are so many ways. I think it's a question. We didn't have a definite answer for that because there are so many ways you can configure and create your topologies, your clusters, your conflicts. That is very hard to give just sometimes more than one answer how you can do it well. And because of that, there are also people who are not coming for noodles or pineapple salad or pizza. Then one day we'll show up somebody at this Italian restaurant for pineapple pizza. And some of the things that are into the 10, I think most of them, are actually about increasing the flexibility, going out to our users and customers with their needs that we didn't even thought about many times, that we can do better and we can extend the flexibility of the system to make it even more adaptable to many use cases. And of call outs is a great example of that. NATS has many ways you can do off into it. In the centralized model, in the centralized model via conflicts, via jobs, via any case, a bunch of them. But still there was a need, especially by the corporate users or big production implementations of NATS, when there was a need to be able to hook into some other out provider, identity provider, or to do some stuff that basically means having custom code that handles the process. And that's what off call outs are. They allow to either hook into some other system or to do a custom logic that is not part of the NATS server flow. And how it works, as we do a lot in NATS, almost all the features in NATS are built on top of the core NATS. So the core pub sub request reply, which request reply is also a pub sub to be fair, and we build the gesture on top of it and on top of that is build key value. And the off call outs is an example of that again. It's not like some separate system or a thing like this, it's part of the NATS and it's using NATS internally to provide, will not get into technical details, but to provide a way to very easily create a separate service that will hook into the NATS system and listen on some subjects and reply on them and do whatever there's need to be done and the server will pass everything it knows about the client. So all the certificates, all the passwords, all that you request will be there. It can be of course encrypted if that's done it. And that allows for various ways that you can extend, change, alter, hook into the like, I don't know, Google out or whatever you want to do. And I think that solves the remaining chunk of the of the scenarios for out that user has. Next one. This one is pretty advanced. So there is a subject mapping. NATS is subject routing based. So you can when you have a subject, which is what normally other messaging system is called topic, you can have wild cards when you subscribe to things, different kind of wild cards. But we then push it further and we added subject mapping. So as you see on the right when you have a subject and you publish to like event one two, you can have a config in the server that will do whatever you need with those tokens. So it can revert them, it can compile them, it can create partitions out of them, deterministic partitions and bunch of other stuff. And that was very useful and powerful but it was server based configuration. And most of the use cases were based on JetStream. So our persistence layer built on top of Coronats. And this is the thing. The thing is that now when you create a stream and stream is the thing on the server that listens on the number of subjects and persists them in order and allows you to fetch them via consumers. And now with this to the 10 release you will be able to configure those mappings per actually stream, not the server. And to give you a concrete example how it could work is imagine that any of you works in the IoT industry, bad example. I think that's anyway a very easy way to grasp it. Imagine you have two, because of NAS topologists which can be very different. We've left no super clusters on all of that. It acts really well with the edge. But there are cases where there might be a hub in the cloud, for example a big Kubernetes cluster that's running huge workloads of analytics and then you would like to ingest source the data from all the factories. And you don't want to have a hassle that each factory has unique subjects and unique names of things. Everyone is a single, the same configuration for simplicity. And thanks to the subject mapping now you can have a factory one factory two, they both have messages and data say events and signal subjects and inside this aggregate when you're sourcing the data from the streams in the factories you now can remap them so despite that factory one has events and factory two has events now on this aggregated thing you can have factory one stream, factory two stream so you can differentiate them really easily. And that's very powerful. There's also another thing you can do is that you can kind of closer to the normal subject mapping is that you can, if you want to you can map the subjects while the stream is ingesting the data you can now do the mapping there too. You can also do the partitioning. As you see here in the configuration there's a partition function used which I will not get into that details maybe this time but if you are interested, reach us out afterwards we are very happy to discuss those. Next thing is multiple filter consumers any of you guys using that to use JETStream? Okay, nice. So in JETStream there are streams so this is what stores the data and our consumers we just serve a site construct which what it does it allows you to fetch the data from the stream and keep tracks where you are on the stream. There was always a thing that consumers can have only one subject it listens to one filter subject so it could ingest a whole stream or it can ingest one subject it can have wildcars and all that stuff but if you have multiple subjects or you want to say only two those in the wildcard not all of them you basically had to either as you see on the left it's the same left as on you you have to ingest everything and then on the client side filter things out which is not very clean and not very efficient or you can have to create multiple consumers which is not very clean code easy to maintain because it multiplicates the instances of things you have in your application so even it's simple as an image as you can see now each consumer can have multiple filters as many as you like to and that allows you to have either this use case when we have again the factories and if you want to sit in not all of the data from all the factories but just let's say A and B you can filter them out only those two so you're saving your throughput you're less trained on server a lot of good things happening there there's also another use case you could have streams could have multiple subjects in itself and in this case stream could contain data from few domains or whatever entities you have and you still would have to if you weren't interested in this example cars planes boats if you wanted to pick only planes and only cars you would have to create two consumers that would independently fetch the data and if it was only out of the needs to get only two of them not all of them you would need them to merge those streams and it was not very clean and easy to use and you were also losing ordering there and now what you can do is just specify more filters on the consumer and seamlessly you just get those planes and cars in this case it's much simpler on the configuration side and it's also much more performant and not maybe much more performant but it's easier and better to use on the client side and it's better for the server in many cases so that also retains the order because if you have two consumers and each of them is fetching on their own manner you still know what's the order each of them has their order but when you merge them you then suddenly have to sort it or do some work with it here it's all done for you the next new thing is on this stream compression I don't think it's worth talking too much about this subject I mean it's very useful, very good but it's pretty technical the point being the value out of it is that you can save some disk space because we see more and more users that have terabytes of data on streams and if they would like have to nest to handle the compression they will be able into 10 and they will be able to zip into two which are different needs a different future set there are also few smaller things the stream and consumer metadata there was something that when you use streams and consumers you had to put all the information you want about it in the name and in the description it was not very clean now you have the metadata which is basically a hash map so a key value map but you can put some metadata information it's useful for Kubernetes situations or for versioning or something some multiple CASTP support, republish edits and UTC based logs and nothing to talk longer about that's basically what we have for 2.10 the 2.10 is not yet released we just released 2016 which is the last release of 2.9 and the next one will be 2.10 which we don't have concrete date yet but it's soon it's probably somewhere next month coming in oh come on so we would like to give you some time to ask all the questions have some open discussions when you're done with time just reach us out, we're here there's also folks from India working on us back in there wave their hands guys so we are pretty distributed and scaled here you can reach any of us and ask questions but here we have time to just have a question that can be answered for everybody no questions you have a question? no, I was just joking I've got a serious question though compression so with it turned on what are we saving on disk with the compression 2.10 we've got some test results in terms of what we're going to save well it depends it depends but it's typical for what you do with compression if you have a lot of the payloads are JSONs with repeated fields a typical compression scenario it will act according to like a stored Gzip compression so it will save you but actually that's a good question meaning because usually the payloads are JSONs it fields over and over and over again and then it's doing good job very not concrete answer but it'll do here we go going back on the active passive question if we have active passive clusters serving just stream is there a way to sing the persistent layer and the state between the active passive yes you can do sourcing and mirrors which will asynchronously source one stream from one side to the other side asynchronous it does not mean like it's a minute or seconds behind it basically it's the latency behind the clusters and you'll make sure that all the data is properly synced and we even use this feature actually to we can make the roundtips shorter because for example you have a globally distributed system all over the world and the stream is like in Europe and you're reading in US you can make a mirror from Europe to US so of course the data has to go through the ocean but when it's there everybody who is fetching the data having very close access to it so the streams and mirrors have many use cases and one of this is this that you can not move the data replicated the data also and the difference is that sources allow you to source many streams into one so this was the example I showed you that you for example source each factory to take this example have set of streams on themselves so they can persist the data without connectivity to the main cluster but when the link is up the stream automatically will automatically synchronize the state and one question thank you one question about the globally distributed clusters I guess we have different clusters with different brains eventually hosted on different Kubernetes clusters and how do we make the link connect to both or you use a load balancer in front or just DNS load balancer there are two patterns to think about it one is a super cluster so you connect many clusters via gateways into one very big super cluster and this is one way and the other way is leaf nodes as you mentioned when you can create some hub and spoke or hub and spoke and hub and spoke because you can extend it how far you like it until you are lost in the topology and this is little different use case it works little differently but it's kind of new once when you want to choose which one activity is very good you are happy with the big super cluster because it is seamless to use and work with when it is leaf nodes you need to consciously say what is available where and what accounts etc what permissions but both are can apply here and fix the problem on the global scale quite often it is like super cluster a central one span across like three data centers for example so it is cloud agnostic and that is often useful because it is pretty good because you can create a super cluster you can be really easy cloud agnostic or span across clusters across many cloud providers but then if there are devices a little more susceptible to the outages, link down etc then you use leaf nodes so they will synchronize when they are up and they are not interrupting any work when they are down so little more new ones I give you like over it can I just add something right on to the end of that really really good answer what I am noticing a lot is we are seeing old enterprise designs meet head on with Kubernetes so Nats is really good at being Nats so if we throw a leaf node at a cluster we can give leaf nodes and clients multiple connections and the clients will try and reconnect they will try and do the right thing so if you can steer the logic either in business code steer the connection logic in business logic you can take the load balancers out you can let Nats distribute things across queue groups as well so now I am saying like the old ways are dead Nats is good at being Nats and the second we put things in the middle you are then tipping the scales in different ways and what is important you don't have to pull all the addresses of all the servers and clusters in because the Nats service is a gossip protocol it will gossip to the clients all available servers and when you try to reconnect to find one it works so we don't have to handle also not this any more questions? there you go hi thank you for your presentation I have a single Nats server running outside of Kubernetes and I have a workload running inside of Kubernetes I want to move Nats inside Kubernetes as well in a clustered way and should I consider currently running server as the cluster and spawn the new cluster as leave nodes or the other way around when I want to keep all the messages and move my workload to the new cluster inside Kubernetes gradually you can do both and again it kind of depends on the because Nats can be very performant and very quick in responses and low latencies and I would say that this would be what should drive your approach because you would lose a lot if you increase the latency being here or there so usually the answer is just see what's more performant and work working better which usually means closer to the client and also when you form a cluster it's close to each other so the rough consensus work quickly and swiftly and based on that I would move forward If latency is not that big of an issue because all the servers are running inside the same VPC AWS I think that's one of the greatest thing is how Nats is flexible though it also sometimes is a challenge that you need a little more input just a short discussion to find the best possible approach Thanks As always it depends I was expecting him to say well it depends and he didn't do it it's such a disappointment if you're considering splitting a cluster and you're going to take it down to Q&A try and think of Nats as a requirement of a normalised environment so if you're going to run a cluster you've got to kind of think same CPU same RAM, same disk to get that normalised experience think of it as a pipeline less surprises less surprises we've seen all sorts of weird things but you can find us it always depends we're always here Thank you Last minute, last question Oh it was a challenge Throw it Hello Hi We are currently playing around a little bit with Nats and especially with Nats Superclusters and we have very dynamic environments where we spin clusters up and down or Kubernetes clusters for the Superclusters would that be that would you rather put a Nats cluster in each Kubernetes cluster that comes up and can connect to other nodes in a Supercluster or are Superclusters more a static thing and you would work around that with leaf nodes or something like that This question I think that the staticness is more around persistence layer because if you have persistence layer the data in Nats is stored on specific nodes and you can of course and HA out of it with replicas etc but this setup it will not automatically move to the other nodes to the other clusters on itself because it can be very expensive like it's one terabyte of data but when it's again it depends but usually the reason if you want to have one big cluster or smaller Supercluster because I guess this is kind of the point is if the again latency because Superclusters is not sensitive to bigger latencies because it does not have to propagate the rough consensus across the whole world and it localises the latency so if your latency around your whole ecosystem is good then you can just use one cluster span across many things if it's not that's good then you probably localise here this region is a good latency so I'll put cluster here and cluster there and just don't allow bigger latencies to be in one cluster because it will slow down the whole system did I answer? thanks she really fansy way saying it depends thank you all we are unfortunately out of time