 Thank you everybody for joining us for another all things data on the OpenShift Commons briefing channel. This is a new briefing series and we're excited for everybody to join us. And if you have any ideas or things that you would like to see in the future, please let me know. Today we have Jakob Schultz from Red Hat as well as Karan Singh from Red Hat and thank you both for joining us. They're here to talk about AMQ using OpenShift container storage. So Jakob, please take it away. Okay, let me share my screen. So in this presentation it will be me presenting together with Karan. So I am principal software engineer in the Red Hat middleware engineering team and Karan is senior architect in the Red Hat storage unit. And today we will give you some introduction to AMQ streams and to the storage requirements which AMQ streams has. And then we will talk a bit more about the importance of OpenShift container storage for AMQ streams. How does it fit together and what are the advantages which you can get from using these two together. So let me first do a quick introduction into what AMQ streams is. And it's Red Hat's distribution of Apache Kafka. It has two versions. One of them is for Red Hat Enterprise Linux. So that's suitable if you want to run Apache Kafka on VMs or bare metal directly without any containers and OpenShift or Kubernetes orchestration. And the second version is for OCP or OpenShift container platform, which obviously for this talk is the relevant one. And on OCP we are using the power of operators from the StreamC project. StreamC is a CNCF project which makes sure that Kafka runs natively on Kubernetes and OpenShift environments. And with the operators, AMQ streams can run basically anywhere where OCP is running. So it can be on VMs, on bare metal, on private clouds or public clouds, wherever OpenShift is supported. You can run AMQ streams as well. And with AMQ streams on OCP we support OCP 3.11 and higher. So if you haven't jumped on the OpenShift 4 train yet, we have you covered as well. But OpenShift 4 is of course great with a lot of improvements. So ideally you should run it there. And for those who don't know what Apache Kafka is, let's give a quick introduction of how it is, what makes it special and how it works. And there are actually many different definitions of Apache Kafka. One of them is streaming data platform. What does that mean? What that means is that Apache Kafka basically provides capabilities for storing, delivering and processing data, which makes it kind of ideal platform for doing streaming data, even driven architecture, stream processing and so on. But it is also a published subscribe messaging system. So it can be used as an alternative to let's say the traditional JMS style messaging system and can be used for the traditional integration use cases. And then the third definition I have here is a distributed horizontally scalable fault or a land commit lock. I love this one because it's full of very nice buzzwords and it sounds very sophisticated. But don't get misled by that. It's not really just a random collection of some fancy words. It's actually pretty accurate definition of how Kafka works because if you look at the Kafka broker in detail, how it is implemented, it really is distributed horizontally scalable fault or land commit lock. So it's pretty accurate. Kafka was originally created at LinkedIn, but later it was open sourced and now it's as the name suggests part of the Apache software foundation. And it was from the beginning designed to be fast scalable, durable and available. And it achieves that by being distributed by nature. So it does basically everything it does is based around data partitioning or sharding if you want. So all the data are split into some smaller shards, which are handled individually. And that's how basically Kafka is able to achieve the high throughput with fairly low latency for delivering the data. And that's what gives it ability to handle a huge number of consumers. What I think is always important to mention is that while in this talk we will mostly focus on the Kafka broker, which is where the storage comes into play. Kafka itself is ecosystem rather than a single project or a single component. So apart from the broker, it has its own stream processing library in Java. It has its own clients for different languages. It has its own integration framework called Kafka Connect. And even beyond Apache Kafka project itself, there are integration with many different third party tools. All the big data tooling stream processing frameworks such as Spark, Storm and so on. All of them have great support for Kafka. So I always like to point out that it's not just the messaging broker itself, but it's really ecosystem of many different components. So I already said that Kafka was designed to be distributed and it does so by partitioning the data. So when sending in or receiving messages, we always send to or receive from a topic. But every topic is always split into one or more partitions. So it can be that topic is just a single partition, but in most cases it's many partitions for each topic. And these partitions act as a shard. So most of the actual work is always done on the partition level. And the topic is really more a virtual object where you send or receive the messages from, but which gets kind of internally pointed to the right partition. And each message is always written only to single partition for a given topic. And Kafka is following this idea of smart clients and dumb brokers. So the broker is really just the commit lock as I mentioned before. And it's the clients which have a lot of the logic of distributing the messages into the partitions. So when you try to send some message, then the client decides into which partition it should be sent based on the message key or based on some round robbing distribution if there is no message key. And then the consumers will similarly decide which consumer should consume from which partition and so on. So that's what allows it to scale so nicely because there's no single point through which all the messages need to flow in the brokers or anything like that. So where the partitions were doing the scaling and the sharding of the data. Replication is a concept which provides availability and durability guarantees. And basically every partition can consist out of one or more replicas. And the replicas have always two roles. One of the replicas will be a leader. That's kind of the main replica which is used by the producers to send the messages to. And then the other replicas will be called followers and they will act as a backup to kind of provide the redundancy for the data. And the followers will really just connect to the leader and will copy all the messages which the leader receives. And they will try to stay as much in sync as possible so that when the leader is lost that they can take over the role and serve the messages without any data loss. And of course that means that the roles can be changed during the Kafka cluster lifecycle. So in this picture you can see that we have three brokers with single topic which has three partitions with three replicas each. And you can see the dark blue partitions or replicas they are the leaders. And you can see how they are feeding the data to the followers so that the followers can keep the copy of the data. And now of course what can happen for example the broker see goes down because of some hardware failure or whatever. And basically immediately the one of the follower replicas on the broker B was able to take over becomes the new leader. And the clients can just reconnect there and keep working without any significant disruptions. Now the replication is really important for Kafka and cannot be fully replaced by mirroring on the storage level. Because having multiple active replicas available with the same data means that the availability is really great and they can very quickly take over if the leader fails. But it's also important for the reliability that the data are written to multiple different disks on a multiple different brokers when they get put into the disk buffers and then later written to the disks. So the replication is not something what you can fully replace with storage mirroring or to the storage mirroring can help with other things. Now because we want to talk about how OpenShift container storage fits with the MQStreams let's look a bit more about what kind of storage MQStream supports and what are the requirements for storage which we have. So on OpenShift we currently support three types of storage. The first one is the FMRO storage which is using the empty dear volumes. So if you don't know what the empty dear volume is on Kubernetes or OpenShift you can imagine it as a temp dear somewhere on the host. So it works quite well for things like development or CI pipelines but it doesn't really provide any data reliability so it's not supported for production. Then the second storage type which we support is persistent volume claims storage. So what we do here is we just use the storage classes and persistent volume claims as a Kubernetes based mechanism to dynamically provision the storage and that's really great because it means we do not need to kind of write a separate support for each kind of storage type which Kubernetes or OpenShift support but once the support exists in OpenShift or Kubernetes we can automatically take it and use it. And then the third type is something called J-Bot storage, just a bunch of disks which really what it does is it allows you to specify that the broker should use multiple volumes of one of the two types I talked about before and that's useful to increase the capacity and performance of the brokers in some cases. Now Kafka was basically written to not require any sophisticated storage features or appliances so it doesn't really need storage mirroring, it doesn't really need rate arrays to make bigger disks instead it doesn't need read write many disks so you can easily use even local storage with it and the actual requirement which we have for the storage is that it needs to be block storage because file storage things such as NFS does not really work well with Kafka and is not supported so what is really the main requirement is to have block storage and ideally with XFS or X4 file systems. Now the OCS block storage is of course one of the supported storage types and while it might have some features which I said before are not required the fact that they are not the minimum requirement for Kafka doesn't mean that they are not useful and in the second part of this presentation Karan will talk a bit more about where OCS can add the value over some other storage types. And so I'm really from the engineering not really from the field but I get involved in talking with our field people and with some of the customers and OpenShift Container Storage actually comes up quite regularly in the AMQ Streams discussion and while in the next slides we will show some of the numbers and examples mainly from Amazon AWS because that's easy to reproduce for everyone. It actually comes up quite often in discussions with the customers or users who want to run AMQ Streams on-premise because they often start up with some questions like does Kafka support NFS can we use that and the answer is no but quite often the users don't have anything else on-premise and that's why how kind of the discussion quickly turns to OpenShift Container Storage because that's an interesting alternative for them. And just remember that with OCS you always need to use the block storage with AMQ Streams and not the file storage. And with this I will hand over to Karan who will talk a bit more about the specific details. So Jakub have gave you a pretty nice introduction to Kafka and he also painted like what storage requirements Kafka imposes on you, right? So in the second half of the presentation we're going to transition to how OpenShift Container Storage adds value to your AMQ Stream or Kafka deployments. So if you're running AMQ Streams on top of OpenShift then by introducing OCS for your persistence storage layer for Kafka it gives you additional performance and additional resiliency. We're going to talk in detail about these two parameters. So as Jakub mentioned Kafka is natively designed with fault tolerance and you know it's inherently distributed and provides better. It's pretty good in doing that but OCS can add additional resiliency features and could provide you additional improved service availabilities during infrastructure failures like a node goes away or a storage volume destroyed itself. So these are the benefits that you can get from by using OCS. So let's quickly cover performance as the first thing. So in our lab we have tested various different combinations of storage vacants. So we have tested this on AWS. So standard OCS OpenShift Container deployment with three masters and three workers for Kafka. We have tested different versions of OCS just to make sure whatever we are testing is consistent and just to have various different data points. So we have tested OCS 4.2 the current release and we have also tested the future release of OCS 4.3 which is coming in a few months I would say. We have also tested different storage classes which Amazon offers. So EBS, GP2 and provision IOPS storage classes. We have done a comparison just to see how different storage vacants performs once you fly Kafka alone on top of it. The architecture looks like this. Everything is running inside OpenShift and storage is provisioned using OpenShift Container storage. Each of the Kafka broker have their own independent block volumes because Kafka works nicely on block and the supporting component which is ZooCuber has also got some tiny bit of block storage so that it can store its metadata information. On the load side we have tested, we have used Open Messaging Framework which is again a CNCF project. We applied Kafka, so a Kafka producer app, eight of them were writing to the topics of A&Q and then on the other side another eight consumer apps were reading the same topics from the Kafka topic. So this is some pretty standard Kafka architecture. So let's look at the performance. There are a lot of information on this graph. So this is performance characterization across different storage options. So before we go into the numbers let's talk about the config real quick. So three Kafka broker pods and with each pod having three... So three storage nodes for OpenShift Container storage because storage has to run on a node so which is why we have to use a physical or a virtual instance on AWS for OCS. Standard small message size and one PV for every broker pod in Kafka. On the y-axis we have tested produced messages per second, the metric that we were looking. In the next slide we have messages consumed per second. So on the column side here we have tested OCS on OCS consuming EBS for its storage needs which is the current design. So for OCS you need to use EBS for storage layer. We have also tested the upcoming new feature in OCS which makes use of local instance storage just to make sure just to compare how performance looks like. And then the third one is using AMQ on top of EBS without using any OCS layer. The last one is ephemeral storage which is not something you want to do in production so I'll just keep it side right now. It's mostly for Dev and QA testing purpose. So as you can see this from red and golden lines, OCS on local storage delivers the best performance across the workloads that you've tested here. And OCS when running on EBS imposes some performance stacks. The reason is that we are running a distributed storage OCS on top of EBS which is also again another distributed storage. So which is why we have to pay a performance tax that you can compare between golden and blue. So which is why there's a performance tax. So some ways that OCS on I3 instances delivers you the best performance that you are looking for. Same story for the second metric which is the messages consumed per second. Again OCS on top of I3 instances delivers the best performance. One thing we don't have in this graph is we have not tested Kafka on direct I3 instances which Yako mentioned that Kafka, usually Kafka people deploy Kafka on physical machines and they just use JBords or local storage. So we don't have that metric so we accept that. So this one was standard performance on a steady state Kafka cluster. We thought let's induce some failure in the Kafka system. So we have triggered, we have destroyed a part and tried to inject some artificial failure scenarios into the situation and tried to measure how much does it take for Kafka running on OCS versus Kafka running on EBS. Hit on the recovery. So you can see this, the red line is Kafka on OCS and the blue one is Kafka on EBS. So the smaller the line, this line is, the better it performs. So you can see this, the red line quickly goes back and the partial recovery is pretty much fast on OCS-backed Kafka clusters compared to EBS because Kafka has to do, you know, some regeneration of data because EBS does not provide, you know, storage level redundancy, which Kafka requires or not requires but Kafka would enjoy if storage by default provides that. So yeah, OCS helps in faster recovery of the ANQ cluster in case of infrafilers like node goes down or storage goes down. We're going to talk about that in a few slides. So this is the additional resiliency part which OCS brings to the table for Kafka. Kafka is no doubt inherently resilient but adding OCS gives it resiliency plus plus. You can think of this. So this is the performance metrics from a different cluster that we have tested because we have been getting some questions from the feed like, I need both performance and resiliency which is multi-AZHA. I'm going to talk about this word in the next few slides. For Kafka, so which storage should I use? Should I choose OCS or should I use the more expensive storage on Amazon which is provisioned IOPS? So we did a comparison between these two, one of these two storage classes. So the first one is AMQ on OCS and the last one is AMQ on EBS, provisioned IOPS. We found that because eventually at the end both these storage classes are basically using NVMEs underneath but we found that OCS performed slightly better compared to EBS provisioned IOPS which is kind of nice. And for both messages produced per second as well as messages consumed per second. So yeah, if you're an architect, if you're designing clusters of Kafka for a customer or even for your companies then probably this makes sense to like OCS or IT instances provides you the highest performance and the resiliency that you're looking for multi-AZHA. And because EBS does not provide multi-availability zone, high availability kind of, okay? Because it's a zonal service. I'm going to again, I'm going to talk about this in a few slides but you will understand that. The second question that we have got from the field is like, hey, I love to run, let's say I love to run AMQ streams on OpenShift container storage but you have like two storage classes over there. You have block storage, you also have a file system storage, FFS. So which one should I use for my Kafka clusters? So Jakub has already pointed out that NFS or ReadWrite many volumes are not, I mean Kafka works on that, you could run Kafka clusters using that but in the longer run there are stabilities problem if you're using file systems underneath Kafka cluster because of the locking and all these things which are there in file systems. So hence, if you are designing this on a Kubernetes native environment you should be using a block storage provided by OCS or even in general you should use block storage for the Kafka needs and file system causes stabilities and performance issues. So yeah, so this is what we have on the performance side. The second part of the equation is resiliency like how could OCS helps in improved resiliency for Kafka clusters. So for that, we first of all need to understand the problem. What is the resiliency problem do we have? Let's suppose I'm running my Kubernetes and Kafka environment on public clouds because this holds true for most of the public clouds, every public cloud has their own az. So in this diagram we have like we have OpenShift cluster running on, let's say running on AWS and you have your Kafka clusters spread across multiple availability zones. Your Kafka cluster is consuming PVCs which are being backed by EBS volumes, all good standard architecture. You're using your system, Kafka takes care of the data application and but if anything goes down, let's suppose any of the EC2 instance in any of your az goes down or let's say your EBS of that az goes down or the entire availability zone goes down because it's all silicon, things can go wrong, right? So if a mishap happens in this environment, Kafka by its nature, given that you have enough nodes in your architecture, it will spawn up another part because it's all Kubernetes and it will request for a PVC and EBS will give it another block storage however this new block storage would be empty because it's a fresh PV which EBS to this class has given it. So Kafka has to reconstruct all the data in the failed EBS volumes and it could take minutes or even days depending on the size of the data that Kafka needs to rebalance. So during this time you're taking your resiliency of the system. So think another mishap on the same az or a different az happens, you are in a problem. You're losing your availability of the service. So that's the problem. Let's now understand how this problem could be solved using OpenShift Container Storage. So exact same architecture but this time we have an OpenShift Container Storage layer running on top of EBS which is OpenShift Container for storage 4.2 thing. You can create an RBD storage class out of EBS storage class and this time your storage would be across availability zones. So again standard data application taken care by Kafka, everything is good and storage level replication is taken care by OCS. So all the data is tripled replicated across the availability zones. In this case let's suppose same thing happens if the az goes down, EBS goes down or the entire availability goes down. Kafka will spawn up another pod and it will request for storage but since it is backed by now it's backed by OpenShift Container Storage. OCS will give Kafka the same PV or PVC which was originally assigned to your first pod and Kafka just need to sync up between the changes. It doesn't need to replicate or regenerate all the failed data because the data is still there because the application was taking care by OCS. So this is the additional resiliency you can get out of OCS in your clusters. It can handle az failures in case of public clouds. But wait a minute, you can definitely debate on here like hey, okay, so we are going through the performance text problem which I've explained in my performance testing. We are running a distributed storage on top of a distributed storage. So whether I'm going to lose some performance? So the answer is yes, you're going to lose some performance because it's storage on storage. So how can I solve this at the same time having the same resiliency? So you can deploy an OCS on top of AWS or VMware or any public cloud of IT type instances which are equivalent of Azure and Google. So these physical instances provide NVMe volumes exposed to the OS and OCS can use local storage operator which is a new feature it's going to come with in coming releases which means there is no performance tax layer of EBS. So we have completely isolated or removed the storage layer provided by the public clouds and you can simply have an OCS simply have Kafka requesting storage directly to OCS and if anything goes down, anything goes bad in your environment Kafka will take care of the data because it doesn't need to regenerate all the data the data is already there and it's fast because it's packed by all flash storage. So this is how OCS adds performance and resiliency benefits to existing resilient architecture of Kafka. Real quick going through deployment. So it's a standard ML file which you could just use. You can go and take a look at streamz.io. It has a fantastic documentation. Yakuban esteem is maintaining that. It will give you enough guidance how to deploy your first AQ clusters on top of Kubernetes or OpenShift. With respect to storage thing, you need to make sure that you have an OpenShift storage cluster on top of it, top of your OpenShift platform and then you just need to add these like four lines in the respective sections of Kafka and ZooKeeper so that your Kafka and ZooKeeper clusters can request PVs from OCS. So just six lines of code and you will get additional resiliency and performance from the system. If you want to go give it a try, you can have this URL and which I've created like you can have a step-by-step instructions to deploy AMQ streams backed by OpenShift container storage on top of OpenShift platform. So this guide will give you step-by-step instructions how to do that. All right. So this is all we have today for you guys. If you have any questions, you can ask in here. Drop into the chat. We can answer otherwise. Yeah. So back to you, Karina. Thank you so much, Pran. And Jakob, that was excellent. Thank you, everybody, for joining us today for all things data and our special guests, Jakob Schultz and Pran Singh from Red Hat. And join us next time. Again, same time next Tuesday at 8 a.m. Pacific, 9 a.m. Mountain, and we will all see you soon. Thank you again for joining us.