 Hello, welcome to this talk about opening DCN's journey to connecting OpenShift clusters securely and transparently with SubRainer. I'm Stephen Kitt. I'm a software engineer at Red Hat. And with me is Martang. Yeah, hi. I'm Martang Slamal. I'm working as a cloud engineer for the Dutch government. Yeah, and so we'll start with an overview of what SubRainer is so that you can understand exactly what we're talking about. SubRainer is a project that is designed to connect multiple Kubernetes clusters at the networking level. It does so by exposing new customer resources that are stored in the standard Kubernetes data stores. And I'll go into that in more detail in a little while. It's an open source vendor neutral project that was started by Rancher and is now maintained by Red Hat with help from contributors from a number of different companies. And one of our big goals is to make it easy to set up and use. And part of this is we make it available for deployment using an operator, or Helm charts, or our own tool, which is called SubCasal. So some common use cases for SubRainer are application availability. So this is where you have an application that's available inside one cluster. And you would like to use it in another Kubernetes cluster, but you can't easily deploy it in the other cluster. And you don't necessarily want to make it available using publicly accessible endpoints. SubRainer can help you with that. Another use case is disaster recovery, where you want to basically ensure that your data is replicated in a number of different geographical areas, for example, so that you can recover if you lose one of your clusters. And another big use case is data residency guidelines. So you might have workloads that are available in a specific, or that can be processed in a specific area, or you want to be able to use, for example, cheaper CPU time and one availability zone, but your data has to remain resident in a specific geographical area for legal reasons. So SubRainer can help you with that, too. So what SubRainer does is help you with the networking there. And so to understand that, we need to go over briefly what Kubernetes does. Kubernetes has a very cluster-centric view of the world by default. The cluster boundary is a hard boundary. And so you can see here a fairly typical view of the network stack in Kubernetes. You have a number of pods inside your cluster. Each pod is connected using pod IP networking. So each pod has an IP address and can communicate with other pods in the same cluster on a private network. Each pod also benefits from service discovery and load balancing. Workables can be made available as services. And pods can address those by name and have load balancing among multiple instances of the same service. And finally, you also have network policy, which allows you to define rules where to say which pods have access to which parts of the network. And SubRainer extends this across multiple clusters. So it doesn't touch the pods model itself. Pods are still private to each separate cluster. But it extends the networking data plane across all clusters so that every single pod can have access to any other pods and any other connected cluster transparently. It also extends service discovery and load balancing so services can be made available from one cluster to another and load balanced within each cluster. So typically, for example, a cluster would connect to any remote cluster which makes a service available unless the service is also available in the local cluster, in which case it would provide the local cluster. And it also extends network policy because the networking layer preserves source IP addresses, you can have rules which take into account the problems of traffic. And it does all this, as you can see with the padlock, in a secure fashion using IPsec tunnels by defaults and also supports wire guard. So all the traffic is encrypted between the clusters. What are the benefits then? So as we've seen, Direct East West pods to the pods and pod to service, routing across clusters. It works with any Kubernetes cluster and it can work with a number of different CNIs. So we test with OVN, for example, which is the default now in OpenShift, also Weave. There's some work that's to be done with Calico and a number of other CNIs and we'll give links later to the documentation you can look at that describes all these. Services can be deployed across clusters. So this is the basic level just pod to service IP connectivity but also, as I mentioned, service discovery and network policy. And it provides all this in an encrypted fashion, as I mentioned. Next slide, please. Here's a diagram of the high-level architecture of Subrainer. So the top section is what we call the broker, which is one of the clusters that can be a dedicated cluster or just one of the drawing clusters, which is used to store and share all the data that's required across all the clusters. And each cluster that you want to integrate as part of a larger set of clusters using Subrainer is connected to this broker and uses that to obtain information about all the other clusters. And inside each cluster, each node gets our route agents. This is what takes care of traffic that has to go to another cluster and make sure it gets to the right place. And at least one node in each cluster is designated as our gateway. And so that hosts the gateway engine, which takes care of the tunnel to all the other clusters. And so traffic gets routed from each node to the gateway and then across to the target cluster and back again. Next slide, please. Now we're connectivity with Subrainer. So there's no impact on intra-cluster traffic. This is very important. All the traffic that stays within that cluster uses the standard network setup, whichever that is inside your cluster. So there's no impact from using Subrainer on the local network performance. And only traffic that has to go to our remote cluster goes through one of the gateway nodes. And as I mentioned, source IP is preserved, which means that, well, traffic can go back. The other way, of course, and you can also specify network policies that take the source IP into accounts. And by default, we encrypt cross-cluster traffic. And another big feature in Subrainer is that we can handle clusters that have overlapping-overlapping-siders. So if you set up multiple clusters without unnecessarily envisioning that you were going to want to connect them at some point and so they have conflicting IP addresses, Subrainer can add its own addressing overly to provide unique addresses across all the clusters. And this is called global net. Next slide, please. Service discovery, so this is the layer on top of public-to-service IP, which provides service discovery across clusters. And this is actually an implementation of a spec that's not defined inside Subrainer itself, but it's part of a larger effort in the Kubernetes multi-cluster service, special interest group. And this is called the multi-cluster service API. So that defines a standard or shared rather set of terminology and some shared features. So the first term is cluster sets and this is a group of clusters that are all connected to each other and that share services. And this introduces an important concept in the multi-cluster service base, which is that all namespaces with the same name are considered to be the same across the clusters. This is called namespace sameness. And then the multi-cluster service API defines two CRDs that are used to share services. The first one is the service export CRD. So this is an administrator-controlled CRD which is used to specify services that should be exposed across all clusters. Services aren't exposed by default. To export one, you create a service export CRD. And this makes the service available across all the cluster sets in a new subdomain under dot SVC dot cluster set dot local. So it follows a similar pattern to services inside a single Kubernetes cluster across the cluster set. And then the other CRD is a service import. And so this is the in-cluster representation of a multi-cluster service, but this is really just an implementation detail in practice. That slide, please. So how, now this is done in sub-brainer, service discovery uses a component that we call Lighthouse. So this builds upon the existing sub-brainer architecture with the broker and so on. And it adds a number of components. And so the general idea is that when a pod or a service requests a cluster set wide service, so it uses the dot SVC dot cluster set dot local name, that gets resolved not by core DNS or well the main DNS provider inside the cluster because it isn't aware of all of this. Instead, the DNS resolver is configured to forward these requests to Lighthouse's own DNS server. And so that DNS server is aware of service imports. And so it knows about services that are available in other clusters and that have been exported. And it will use the information from the remote clusters and from the broker to resolve those services to an IP address that sub-brainer can then take care of. And so traffic can get to the other cluster. So that was a brief overview of what sub-brainer does and some of how it goes about it. But the purpose of this talk was to go over our journey with ODC Nord and so Martyn will present what ODC Nord is in just a couple of minutes, but I wanted to do before I conclude my section of this talk to give some of my impressions from working with Martyn and his team. So the purpose of the POC with proof of concept with ODC Nord and sub-brainer was to get from our perspective early feedback on features and usability. So when we're in this of engineering groups, we don't necessarily have much direct feedback from end users. And so this was an opportunity to get that. And it's led to of a big initial drive first on ease and speed of installation and the importance of being able to upgrade because when we were working with Martyn and his team, they were very willing to test new things and iterate frequently, but that meant that we had to make it easy for them to do so and frictionless because obviously we didn't want to lead them into a situation where they'd upgraded to a bad release and they ended up stuck there or we'd end up spending lots of time helping them to fix things. And one of the big gains from our perspective is that we were able to get a good idea of customer, well, features that were actually useful for customers. And so ODC Nord provided us with a number of feature requests that we hadn't necessarily thought about and that were provably useful for end users. And so the first one of these, which is insecure connections, that was actually somewhat surprising for us. I mentioned that cross-clutter traffic is secured by default and this was a big feature in some arena from our perspective, but it turns out that it's not desirable in all cases. So in some cases, end users can have private connections between Kubernetes clusters. And so there's no point in adding extra layers of security on top of them, it just reduces the network performance. So this was a big request from ODC Nord. It's taken a while to implement, but it's actually landing into the next release of SubRainer. Another one that went with this is high bandwidth. So ODC Nord has a high-performance network, physical network, and the desire was to be able to use as much as possible with this in SubRainer. So this led also, our investigations around the high bandwidth and bandwidth issues led us to implement a benchmarking tool because we knew how to run benchmarks and the ODC Nord engineers knew how to do that as well, but repeating that all the time was somewhat painful. And so we integrated all that into a benchmarking tool at their request. And then in addition to that, so network policies, they were an important feature for ODC Nord's own customers. And another big takeaway that I had from the working on the proof concept was that knowledge is power. We benefited greatly from having different people on the team with extensive expertise in all the layers of the stack that ended up being used at ODC Nord, in particular, the OpenStack networking layer. And so if we had just OpenShift or generic network experts in the team, we might have had trouble helping ODC Nord with some of the problems that they ran onto because they have a fairly complicated OpenStack setup. That's that setup from me and I'll hand over to Mark Time now. He'll give you lots more information about ODC Nord. Okay, well, thank you, Steve. Let me introduce myself first. I'm Tain Stabel and I'm a cloud engineer for the Dutch government. And ODC Nord, it's Nord stands for North. We are located mainly in the north of the Netherlands. It's one of the foreign Dutch government data center agencies. And so in the past, we had many data centers inside the Dutch government, but they decided I think eight years ago to consolidate the two four data center agencies. And at the moment, we have two data centers running in the north, the two red dots you see in the picture, the land car of the Netherlands. And the third one is available this year. So a little bit of a perspective and history from our as a resident customers. We are former core web users. So in 2017, we started our container journey with Kubernetes because we saw inside the Dutch government, many government agencies are moving towards microservices and wanted to use container platforms like Kubernetes. So we thought, hey, that's an opportunity to create and get just to create some kind of managed service for our government agencies. And we are until now, we are pretty successful in it. So since Reddit took over Coro as we moved to OpenShift, so I think since January 2019, we're in production with OpenShift 3.11. There are still some 3.11 clusters left, but the rest is all OpenShift 4.6 now. And well, yeah, our main, so I think we have about more than a 3000 cores running on multiple clusters, but we have a main, we are ready for, we want to move to the next step. And the next step is make these clusters more resilient for failures. So our customers can make their workloads more robust for our Dutch people. So we were wondering and looking how can we make these OpenShift clusters more reliable, more high available. You did a lot of reading and proof concepts and well, we reached out to Reddit Netherlands and just please help us, how can we do this? They came up with the Merida project. We did see it when it was at the Enhanced Rancher and we left it first, but then Reddit took over and we got an opportunity like a season explained to work closely together. So yeah, so we're almost ready for the next step. So and what's that exactly is what do we expect to if we just say let's say install OpenShift clusters on all these free data centers, that's what we want to achieve this year. We want to be able to connect clusters so we can send data or we or our customers can send data. We want to have part to part connectivity. As I will explain in my next sheet, we are already able to, we have a highly high available network with high bandwidth between the data centers. It was already present. So we were able to communicate between these clusters in several different data centers, but we were not able to do part to part connectivity. So we got a fairly static setup, which was not really helpful. So that's why we want to have part to part connectivity. We want it to be secure. We want to leverage the available bandwidth, not a complex configure. It had to be cloud slash network agnostic. So Stephen mentioned we are running on OpenStack, but maybe we are using in the future something else. So it's also has to work on that. So it must be open source. Because we believe in open source. Everything is open source what we do in our company. And the last two bullets, low balance traffic, multiple clusters and do dynamic health checks. So, well, we thought maybe that's a little bit too much to do in one year or let's say one and a half years. Let's focus first on the East-West scenario. And now that's what we can do exactly with Samaram. So we have been working with them closely together since June 2020. So I think it was version.o4.4. And yeah, like Stephen explained, we've been testing new features, new releases. We could test upgrades. We have regular feedback sessions, which were very helpful. We learned a lot about our performance. So just how much performance can we get out of the network and out of the clusters. So that also led to several OpenStack optimizations, which were really, really helpful. We didn't figure out ourselves yet. So our entire cloud just benefited from these OpenStack optimizations, also non-open-shift container workloads. So this was a really cool, cool, cool benefit. And yeah, we made our cloud available to the Samaram team. They could just use it to investigate issues or to make things better. And even so important, it was very cool. It's a very cool and professional team to work with. So if you ever get this opportunity, grab it. So it's really, it can be really, really helpful for your own use case. Okay, something about our infrastructure. So, yes, we're reddit customers. So we're using OpenStack. At the moment, we are at train. We're running more than 6,000 VMs. Installing OpenShift on the OpenStack. So we're all also using the nice OpenStack APIs for send and block storage. We also have an encrypted variant. We're using Octavia for server-type load balancing, Manila for read-write-manage storage. And a roadmap thing, we're probably going to look at the ironic bare metal integrations for next year. Chef has a storage layer, more than approximately 20 petabytes. And we have S3-compatible object storage, implement on multiple regions, the red dots. And the data centers in the north, at the moment are connected with dark fibers. We say dedicated fiber channels. She has a lot of bandwidth though. And this network has been implemented as a layer-free network. So this was a really good fit for Summinar. So, some high-level network architecture. At the moment, so we have two DCs running active. And the third one is in the making. I'm probably operational by the end of this year. And what we want to do is to connect, let's say, three clusters together. So we can make a very reliable infrastructure. Yeah, so I think one and a half years ago, we had a power outage on DC1 and everything was gone. It's not good for our business. So we want to make things more robust. So these DC links I've been drawing here. One is for the DC link between cluster A and B. It's the connection Stephen mentioned. We control it ourselves. We already have encrypted that with our network devices. So we didn't have the use case there to have an extra encryption layer on top of it. So that's the requirement we wanted to be able to turn it off. But for the other connection to A to C and maybe to B to C, we do not control the network path. So we can enable the extra IPsec. And maybe just in the future, we are going to connect. We can move as a government public workloads to the public cloud. There is maybe also use case for Sonerno. So I want to conclude my talk with a small demo. It's a pro-pro-gb of two open shift clusters on DC1 and DC2. I will show you in a minute. And I've installed a pro-pro-gb on both clusters and the node form cluster. And I will try to create some kind of database. Well, it's going to be created on the other end. So I will show you first. We have a multiple cluster running. So you see here our first cluster. You see here GN2, that's DC1. We have our second cluster. You see it here is GN3, DC2. And inside these clusters, we have installed Sonerno. So I will show you the gateway, which is installed on a, there's a pod running on a certain node. We do not have dedicated nodes for it yet, but we are planning to. So if I just use the open shift, looking for a gateway. So you see how many instances do we have from a gateway? You have to look up the right namespace. We do have, you see two gateway instances running and one of these is the leader. So I think it's the, this one. So if you want to show you, you can put you, it's a nice feature. You can look up in this gateway, what the connection is between the, the other, between the gate, the gateways on both clusters. So, oh, sorry. You see here, we connected, this is DC2. You see here, we are connected to DC3. And you have all kinds of features. We can see the latency, the average, the last max, minimum, where the status is, it's connected. Active, this is the active one. So this is not the passive one. All kinds of, the backend delivers one. So all kinds of information, this is useful to check if the gateway is alive. What's also, is nice is the, I will switch tabs now, is the able to look at some metrics. So, Submariner's Exposure and Premediate Metrics. If you want to just look the, you can plot these latency, this connection latency seconds in a nice graph. You could use the, of course, the Premediate Workload Monitoring, which is in the side of OpenShift for monitoring and alerting on it. It's a really nice feature to add it. So, yeah, really cool. So, now back to the demo of some proof. This is really working. So, I believe we're installing two, the name, CockroachDB instances. So, this one is running on DC2. Switch the, and this one is DC, yeah, I will look it up. This one is on DC3. So, you see the Pots are running. They go on the container creating thing. So, if I hover over to the CockroachDB UI, you see it's connected. It has nine nodes, three Pots on each side. So, you can see these nodes are connected. On GN2, DC1, and GN3. You also see this on the other side. This is GN3, GN2, GN3. So, this proves, this CockroachDB cluster is able to communicate with each other. You have all kinds of nice latency maps. Also, you're sure to see how fast is the communication between these nodes. We haven't been looking and tweaking it in detail, but we'll do that later. So, yeah, this is really cool thing to show. I think it's working. Just to prove the part of the on-the-potting is to create a database, see what kind of databases are in this instance. We have a demo database. I should have been deleting it. I will delete it now and re-create it. So, I will go inside a CockroachDB pod. Keyboard sleeping, there it is. So, I'm connecting to the Cockroach instance. Oh, sorry. So, I've been connected now to the database instance. Now, let's say I will create a database, create a database following demo two. Well, it's telling me it's been created. So, if I go back to the CockroachDB UI, refresh the instance. I see I have a new database created and now needs to be also on the other end. I see, yeah. You can see it's also been created on GN Free. So, this proves, it's a really simple demo, but yeah, it proves that it's syncing the data between the both CockroachDB instance through Samara. So, oh yeah, this concludes my demo. And I will go back to the presentation. So, if you'd like to really look up on Samara, just visit these links. Thank you for your time.