 Well, hello and welcome to another Dev Nation Live. I am super excited to be here today because we're going to talk about asynchronous messaging. Hopefully, you folks have actually tried some messaging technology before, but today you're going to see something really interesting from a messaging standpoint because messaging is obviously incredibly important to an enterprise application workload, whether it be Java or Node.js or Python or .NET. It doesn't matter what your language is, but the ability to distribute transactions across multiple computing paradigms and platforms is huge and we're going to show you how we do it across the hyper cloud. So with us today is a very special guest, Ted Ross, who's actually one of our lead engineers, our top guy to talk about how to do hyper cloud software development and distributing transactions through AMQP. And he is the man who's going to show us how we do that today. He's got a super awesome demo that I think you guys are going to be super excited about as well. Remember to put your questions into chat tab. I will try to answer as many in real time as possible, but then we'll also save some questions for Ted at the end. So this point, let's turn it over to Ted. Ted, you ready to get started? I'm ready, Burr. Thanks for the introduction. My name is Ted Ross. I'm with Red Hat and I'm based out of Westford, Massachusetts office. And we're going to be talking about hybrid cloud communications today. So I'm going to share my screen and get started. Hopefully you can see my slides. So I've added a little bit of contact information here. So if you need to find me online, I can be found. I should note that if you go to my blog, you'll see that it's actually quite a sorry excuse for a blog. There's an article on that. It's about three years old, but I'm putting it here to incent myself to get more content up there. So I'm going to go through a couple of slides just to talk about what we're doing, but I'm going to spend the bulk of my time on a live demo. But I just want to make sure that it's understood what it is that you're looking at. So today we're going to do a demonstration. The basis is the Verdex Messaging Work Queue Booster, which is a Java-based distributed application using the Verdex reactive environment. And it consists of two components, or three really, as a front end, which is an interactive web front end. There is a worker that can be deployed that is going to do the work that's requested by the front end. And then there's a messaging server in the middle. And the way that the booster is set up, it just uses a very vanilla active MQ Artemis messaging broker, which I'm not going to be showing you today. And in order to talk about distributing load and going hybrid cloud and going big, I needed to add a workload generator. So the workload generator is an additional component that has been added that can go much faster than an interactive front end. So a couple of changes that have been made to the booster, first of all, that that message broker has been replaced. There's not a message broker here. It is replaced with the active Apache, I'm sorry, Apache Kube dispatch router, a network of routers to be specific. And I'll show you more about this in the demonstration. So there are no queues. There are no brokers, all the communication here, even though it's using the AMQP messaging protocol is all direct, not through queues. And one more thing I've made. Now this is a slightly different view of the picture. This is the logical view. If you look at that booster, you'll see that the queues are all named queue something. And I've removed the word queue from the addresses that are in use here because I want to avoid confusion, since there are no queues. So what is being shown here is the patterns that are being used for communication. The worker takes requests and provides replies. And the worker also provides periodic worker updates where it gives statistics about how much work it's done, how many failures it's seen, et cetera, who it is. And those worker updates are consumed by the front end for display. So there are actually two patterns of communication that you'll see here. The front end and the workload generator are both sending work requests to be picked up by one or more workers and the replies are coming back. So this is a remote procedure call or a request reply pattern. And the worker updates is more of an event flow where there's no response, but the worker updates that can be handled and displayed by the front end. And we'll see all of this as we go. So how are we going to scale this up? This is interesting but not very interesting until we start scaling up. So this is the depiction of the various data centers that we have in play here. So the one on the right, RGU actually stands for Raleigh Durham. That is an internal bare metal server that I've got here within Red Hat that I'm using. And that's where I've got the worker, the front end and the workload generator deployed. They're deployed within OpenShift. OpenShift is a container that you can find much more about this. I'm not going to talk about it in detail, but OpenShift is based on the upstream Kubernetes project and it's an orchestrator and runtime environment for containers that can be used to deploy software easily in different environments. So each one of these data centers is running OpenShift. And on the left, I've got two public cloud locations. One on Amazon Web Services and one at Google Cloud Platform. And those two are identical. They have workers associated with them and they have a workload generator as well because I'm going to show what happens when we generate workload from different parts of our deployment. I should note here that there's a project and I've got a link at the end of the slides, which I use. It's called OpenShift Hybridizer. And it's a set of playbooks and scripts that are used to install and deploy OpenShift on public clouds. It was very helpful in getting everything set up. So what does the messaging network look like? So in each of these locations, I have an instance of the Apache Cupid dispatch router. And Cupid dispatch router is a messaging server. It's not a broker. It doesn't have queues, but it's a router for messages that runs at layer seven for ANQP. And I've got one in each of the locations and I've made connections outbound from my private cloud to the public clouds. And the reason those connections go outbound is because the firewalls prevent connections to come inbound. Obviously the public can't make a connection into my private network. So the connections are made outbound. And you'll see also here that there's a cost associated with each one, which I created arbitrarily, where I associate a cost with the connection where I want to favor AWS with a lower cost and use GCP under more extreme situations. So you'll see that in the demonstration. So without further ado, let's go over to the live side. So what I have here is I'm going to have a number of tabs I'm going to go through. Hopefully we'll keep track. But I've got OpenShift web console running inside my private cloud. And as I mentioned before, you'll see a number of things here. This container called ANQ Interconnect is the keep a dispatch router. And there's one in each one of them. This is the Verdex messaging front end, the interactive front end and the worker. And there's also the workload generator. So just like I showed you in the pictures before, right now I've got zero workload generators running, but they're sitting there at the ready. And I've got a messaging worker. I've got a front end all running. If I go to AWS, you'll see a similar thing where I've got the interconnect router, and I've got the worker and load generator in the waiting as well. And over at GCP, well, I've got nothing running here, because I wanted to actually show you how that gets set up. So what I'm going to do is I'm going to copy the login command. So I can get here from command line. I'm going to go to my command line. I'm going to paste that command in. So that now logs me in to that particular public cloud. And I can now do a setup of OCP all. And GCP helps if I use the right term. And this is going to now cause some scripts to run that are going to. As you can see already, there's a project called demo and queue. And you can see that it's now spinning up. So I've got the interconnect router spinning up, keep a dispatch router. I've got the workload generator ready to go and the Verdex messaging worker. Now this is actually being built. This is a deployment that it's going to build locally. So if I wish to, I could actually make changes to the code. Push them up to the Git repo and then cause them to be rebuilt here again. So that might be something that you might wish. If you're playing with this demo on your own, that might be an interesting thing to do. I'm going to go to the console now for our router network. And what you can see in this topology tab is the router is depicted. So I've got the RGU, my private one, connected out to AWS. And also connected now to GCP because we just started that one up. If you see these yellow, the depictions here, these are the endpoints. And this actually is the worker that is deployed there. And you'll see that there isn't actually yet a worker on GCP because it's still building and it's still spinning up. Okay, the build is complete. We're doing a rolling deployment. And if all goes according to plan, it should appear momentarily. And there it is. So now we have our complete network. We have workers deployed at our remote sites and also on our local site. So let me talk a little bit about what's going on traffic-wise. And I'm going to talk about the worker updates first. And so I can actually depict this traffic in this console. What you see is that every five seconds the worker reports in. So it sends an event saying, here I am, here are my statistics. And if I go to the front end, you'll see here that this is the front end of the Verdex application. And it's got three workers that it knows about. And it hasn't processed any requests. So if I do some requests like one, four, five, six, seven, eight, seven, eight, what the service is actually doing is now it's uppercasing the text that I sent to it. And you'll see that this bottom worker, Verdex 423C, has processed for requests and seen zero errors. So I will take you over to the links overview. And again, I'm looking at the messaging console. So this is from the perspective of this messaging network. And so I want to talk a little bit about those worker updates. And you'll see there's actually four listed here. And they are listed as either in or out. So in the case of worker updates, there are three in and one out. And if you were to count, you would see that the number of in deliveries is added up. The sum of those is the out. So what this means is that out is an output from our network out to a consumer. That consumer is the front end. It's the only component in our network that is subscribed to worker updates. It wants to receive worker updates. But the three in's, there's one at AWS, one at GCP and one at RDU. These represent all of the actual workers that are running. And they have inlinks into the network where they're supplying these updates. So these events that are flowing are flowing in from AWS, GCP and RDU, and they're being routed by this network out to RDU, which is the only consumer. And this is all done automatically. And if you look at work requests, this is where the requests for actual service are being done. You'll see that there are four because I typed in four. And they came in at RDU and they went out at RDU. So these are all served locally. And the workers that are deployed on the public cloud haven't seen any action at all. And the reason for that is because it hasn't been busy enough. There's been no reason to offload any of that work anywhere else. Let me show you one more view here that we're going to see a little bit of. And here's where I'm going to look at the worker updates first. This is a chord chart. And it is showing us the relationship for a particular API or address to the worker updates API. This is telling us how many are going, in this case, from RDU to RDU, originated in RDU, serviced in RDU, originated in Amazon Web Services and serviced in RDU. And then there's much smaller number here that are going to have been originated in GCP and serviced in RDU. And the reason there's such a small number here is because I started it much later. And this is the number of requests. And if you sit there and watch it, you'll see that every five seconds it goes up by one because that's the way these messages are flowing. And you can see that also in the topology. So let's bring on some load and make it more interesting. So I'm going to go to the workload generator inside my private cloud and I'm going to scale it up. I'm going to go back to my front end here and we'll take a moment to scale up. It is coming online. Oh, I'm pulling. So it's pulling it down from Docker. Now it's starting up and now it's running. And now we're going to see the work requests. Well, we're not going to see anything flowing here because they're all happening inside RDU. So now the rates are much higher. We've dealt with 40,000 some odd requests. But again, they're all being handled by the Verdex worker inside RDU. And if I go back to my message flow and now I'm going to look at the work requests and I'm going to just look at the rates here and you'll see that RDU to RDU is now handling about somewhere in the vicinity of 3,000 requests per second. So the way that this load generator works is it emulates about 10 consecutive requesters going as fast as they can. So we need to increase the load. So I'm just going to deploy more workload generators. So I'll scale up to three. So now I've got 30 simultaneous users being simulated. And what I'm going to see is that we're going to start to have a little bit of spillover. So the work rates now for the work request, we started out doing most of it, RDU to RDU, but now we're starting to see some spillover to AWS. And if you remember before, I said that I had a lower cost for AWS. So it's going to be favored. So I'm spilling over some work there. And if I bring the load up even further, we will see this balance change around a little bit as those workers come online. And I'm sorry, as those requesters come online and use the workers. And now you see that GCP has been thrown into the mix. So still the majority are handled locally because the local one is very quick. And it's local and there's a very low latency. But we're starting to spillover some to other clouds. So we're actually utilizing now the clouds as a backup. And the way that this load balancing works, it's not round robin. And it's not any algorithm. But what it's actually doing is because we're using advanced transport protocol, in this case, AMQP, AMQP and the routers know about the disposition of every delivery message. So when messages are delivered, it knows what the backlog of every server or every worker is. So it's going to route deliveries of messages to the workers that are going the fastest or have the shortest backlogs. And so this is actually latency oriented. And it's really about once, you know, when the latency starts getting long on one particular area, it starts spilling over to other areas in order to keep the latency in check. And that's what you're seeing. And if you look at it closely, you'll see there's some fluctuations. It's pretty steady, but there will be some fluctuation in the balance here. And that's because of there being lots of network components between RGU and Northern California. And, you know, where the place is where these public clouds are actually located. There's a lot of variation in network latency and throughput that occur just because of, you know, other ambient traffic. Okay, having showed that, I'm going to, I have time to show you another interesting aspect of this. And what I'm going to do is I'm going to go into each of our cloud locations and I'm going to turn on the low generators there. Because there's something that's kind of interesting that I want to show you. So I'm spinning up a low generator now and I'll take the one that I had in my private cloud. I'll spin that back down to one. And we're going to look at this. Now we're going to see, it's going to go through some fluctuation here as things come and go as we scale down these seven. But what we're going to see more and more is there's going to be a locality. So there's a lot of work requests. Now RGU is handling its own work. And GCP is handling its own work. And AWS is handling its own work. So I've got now clients and servers co-located and I'm seeing now locality of traffic. So each site is handling its own traffic because that's the optimum location of a server for a client. And if I go to my topology and look for traffic flow for my work request, I don't actually see anything going on. And the reason for that is because all traffic is now local. So we have aggregate capability. We've aggregated the capacity of our workers in such a way that there's no shared resources here. There's no shared work queue. There's no brokers that are trying to rebalance traffic. Everything's happening locally. Now if I take away our workload, RGU and from Google, now the only one that's actually generating any work right now should be AWS. And I'll wait for that to come in. Again, it takes a moment for OpenShift to actually shut down its services. So you can see that it's still scaling down to zero in both cases. RGU's gone and there goes GCP. It's going to go out. So now all the work is happening in AWS. But what's going to happen if I go to AWS and I take away its worker? So I'm going to take away AWS's worker. So AWS now has lots of workload, but it worker to serve its requests. So what should happen here now is that the network will realize that it doesn't have any local resources to do the work. So it's going to find the next best deal, which is going to be over here in RGU. So when the, so we're still running AWS, AWS, because we're waiting for that worker to go down, right, I jumped ahead. I shut down the wrong one. And what occurred then, I actually shut down the worker in RGU. So now what's happening is the work that's being generated is the clients are all in AWS, but the only server available is in GCP. So what's actually occurring here now is that those requests are now being routed through RGU to GCP, because that's the shortest path to get there. So we're basically handling failover here. And now if I were to go back to RGU and turn his worker back on, then we will no longer be more cost effective to handle the work locally at RGU. And there's where you see it happening right there. So we're no longer forwarding over to GCP. We're doing it at RGU. And if I look at the message flow again, I'll see that the majority of the traffic here is going to be from AWS to RGU. And at that point, I think, Burr, that I'm going to open this up for questions. Does that seem like a good stopping point for this? All right. We do have questions for you, Ted. And so let's kind of bring Ted back into the scene here so we can hear your responses. And so Fernando had a great question right up front. It's like, Hey, and it's a good question, Ted, you may or may not have the answer. But is this capability part of AMQ71 or AMQ72? You know, from a product standpoint, right? Well, yes. Now, AMQ7 is actually that that version is referring to the version of the Artemis Broker, our productized version of the Artemis Broker. But when we shipped AMQ71, we also shipped this new component, which we call AMQ interconnect. And that is what this is. And so are any of the capabilities from what you showed today different than what's in that release? The only thing that's different is in that original release is that the console is a little bit more capable in the newer release. Okay. Yeah, that's what I was thinking. I did mention on the chat that it was a part of the upstream. Now, there's actually a couple, there's actually great conversation in chat we had about, and your demos touched on this, but I want to make sure we drill down on it. So, Banked and RAGT, I think is, if I pronounce the name correctly, they're very interested in understanding redundancy and what happens when some component fails or another. And you showed part of that in your demo, like what happens when some workers go down and what happens maybe when something else goes awry inside the network. Could you talk more about that, though? What happens if, like, where are the points of possible failure? If the router goes down, what happens if a worker goes down, if a producer or consumer goes down, and what might happen with the reload balancing, if you will, bringing those things back to life? Yeah, that's actually a very good question, and it's worth pointing out, of course, we probably need a couple of hours to cover all these things, but what I'll say is that the router is different from a broker. In a broker, you would do high availability using some form of clustering because it's stateful. The router is stateless, and it's much more like what you expect to see in a Cisco IP router or such. So you get your reliability and your high availability not through clustering, but through redundancy. So the network I showed you was very simple, three routers, but there's no reason why you can't have a much more complex topology where there are redundant pads. And it doesn't mind redundant pads. In fact, I would almost recommend that in any case of a realistic deployment, you would set these routers up with redundant pads. That way, if a router fails, then it will recompute the locations and the best paths for things to get to other places. So that's one aspect of it, which is how to heal the network if there are failures in the topology. The other aspect that you touched on, Burr, was what happens when connections to endpoints fail. So now in a broker, you would have a transactional exchange of either a message from a producer to the queue, and then a separate transactional change from the queue to a consumer. And while it's in the queue, it's owned by the broker, and it must be stored in some persistent file store in order to keep it safe. The router works similarly, but that transactional exchange is from sender to receiver directly. So it goes from the sender to the receiver, uses the same protocols and the same levels of reliability that you would get with a broker, but there's no storage in between. So you're actually into, in our case, the client sends a message to the server, that is a transactional exchange. So if that message were to have been lost or if the server were to have failed before it finished the processing, then that message will be in doubt and could be reset. So in any situation where there is no consumer, for some reason, there are no consumers out there in the network, your message producer would not be able to produce. There would be no one to send to. In fact, in fact, the way the router philosophically is about not having a lot of messages in flight. It wants to have enough messages in flight so that it can go very fast, but that's all. So if there are no consumers, then it's going to apply back pressure to the senders. The senders won't get credit and they won't be able to send at all. Likewise, if we're in the process of sending and messages are flowing and the last consumer disappears, then the messages in flight will be released. They'll be sent back to the senders saying, sorry, I couldn't get there. You're going to have to retry or do whatever you want to do with these deliveries, but they didn't get there. So they're guaranteed. So the back pressure is interesting. So is that just part of the client library that is provided? Yes. The client library and the MQP protocol have flow control mechanisms that you can use. Okay. I was unaware of that. That's actually very interesting to me in particular. So I'm going to have to check that one out, please. Yeah. Okay. Well, we are out of time or we're definitely coming to the top of the hour. Let me double check to see if there's any other questions. Which MQ service do you use? Sorry, which MQ service do you use? I'm not sure how to answer that question. The other question from Raymond here is the Cupid Dispatch River console. As mentioned, the one you saw in the demo is more upstream, something Ted and Tim have been working on, not yet part of the AMQ71 or 72 binary. Was that right, Ted? It's actually very close to the most recently shipped AMQ interconnect to 1.2. So I think there may be some of those little animations that aren't in the productized version yet, but they'll be coming out next. Okay. But always feel free to work with us, reach out to us. If you're more on the supported side of this and you want to see that showing up in AMQ, let us know if there's something that's missing or not quite working for you. At the same time, the demo, do you have some links in URLs, Ted? Do you want to pass on to folks where things are located? There are. There's the last slide, which I didn't show you. I hope, let me, should I share it and put it up? Yeah, if you could real quick. It'll be very quick. I'll produce yours there. All right. I think you have access, but so what we have, the demo scripts, the first one is basically the entire demo I just showed you, which you can download and run for yourself. You'll be responsible to create your own OpenShift clusters to run it on. But once you have them, you can run this demo. Exactly as I just showed it to you, the OpenShift hybridizer, which I use to create those clusters, is linked there. And just for completeness, even though this is copied into the demo, I did also include the Verdex MessageWork Key Booster as a standalone, which you can also access. All right. Well, that is very cool. Well, thank you so much. And I do love this demonstration. And for those people who've been kind of watching a number of our donations, you can hopefully see some more of this cool messaging technology coming out. We actually have a couple of sessions this fall talking about Kafka as well. So the messaging area is an area we want to focus on. And this is how we kick it off. I love that Cupid Dispatcher Outer Ted. It's really cool stuff. Thanks, Per. Okay. All right. And Fernando is going to try to run all this on three mini shifts on his own laptop. You may not have enough memory on that laptop, Fernando. But definitely check out running how to run across the multiple clouds. We work really hard on the hybridizer, so you could run this literally across Google, Amazon, and AWS, and your mini shift, if you wanted to. And that's something Ted and I, and Commission, the guys who've been working on it, will talk about more in the future. Again, thank you guys so much for today's attendance. Ted, awesome job for that demo. Thank you so much. And we will see you all again in the future.