 I think these guys need the applause. I'm nothing yet. All I've done is turn up. So great. Thanks very much. I'm Paul Bredner on the open source technology evangelist. I work for Instacluster in Canberra. We've recently been acquired by NetApp by their cloud part, which is called Spot, which does stuff with Spot instances on cloud providers, so I believe. So the inspiration for this talk came from where I live, which is Canberra. It's called the Bush Capital in Australia. It's very unlike Singapore. It's very low density. And I live right next to the bush. And about a year and a half ago, one of Google's subsidiaries started doing some drone delivery trials over the top of our house, which I thought, well, initially I thought this is cool. We can get drone deliveries like our neighbors. We're getting, I don't know, donuts, coffee, and stuff at all hours of the day and night with these drones buzzing over our property. So I downloaded their app, put in our address, and they've got too many power lines and trees and things, which is really annoying. So we had to put up with the buzzing drones while our neighbors got the free, or not free, but got the exciting drone deliveries. So that point I thought, OK, I can come up with a more interesting story around drone deliveries for a demo application. So this is more or less the origin of this story. I don't think it would work in Singapore. The density of living in Singapore might be a challenge for this. Instaclust provides a managed platform for big data open source technologies. We've got a whole bunch of technologies around storage streaming analysis search. And the latest one that I'm talking about particularly today is orchestration. It's a workflow orchestration technology, Uber's open source cadence. So it fits in really nicely with the previous talk around workflows as well. This is something that's focused a bit more at the enterprise level, high scalability workflows. So workflow orchestration is basically pretty simple. It's just about task ordering. You want to start something. You want to perform a task. You want to perform another task. And then you want to end. So it's relatively straightforward at that level. So doing that with cadence, pedaling with high cadence in the bicycle world is called spinning. Mashing or grinding is slow and bad. So you want to spin your workflows basically at a high cadence. Cadence is fault tolerant. So workflows can and will fail. And at that point, you want to be able to recover from them in a sensible way. Cadence is horizontally scalable. Essentially, the number of workflows that you can be running at the same time is unlimited. Cadence workflows are just code. So that's one of the nice things about cadence. Here's an example of a couple of tasks being performed in sequence. This is a workflow implementation of two tasks executed sequentially. In terms of the architecture, the architecture is a bit complicated, which is why we provide it as a managed service. At the back end, there's actually some databases. Cassandra is the one that we prefer, Postgres works as well. If you want to be able to do what they call enhanced visibility, so you can see what's going on. You also need a Kafka and OpenSearch, but they're optional, which I'm going to talk about in this talk as well. So Instacluster actually provides the managed cadence part of the system and the databases as well. You as a developer actually have to write the workflow code and run some workers as well. That's run on the client side, so you need some like some EC2 instances running on Amazon or something to actually be able to get that to work as well. And that just connects via APIs to the managed cadence. So yeah, the workers run the actual workflow logic in Java and Go Clients. I currently need the two supported client types. So just looking under the hood a bit, how does cadence fault tolerance work? Well, it's essentially using event sourcing, which has sort of been around for a while, but it's a continued hot topic. This basically means there's a history mechanism and replaying of the history. So essentially as the workflow is executed, the history is written to the database, which the database side of the story comes in. That's the persistence mechanism. And then you get workflow state recovery by replaying the history. So if there's a failure at some point between task one and task two in this example, the failure actually causes the complete history to replay back again. Yeah, and the replay basically recreates the workflow state and the restart point to continue the workflow from. So it all continues to work as if nothing bad has happened. Cadence activities, cadence activities are quite central to cadence. Activities, including things like cycling backwards, can fail. Remote calls can fail. So one thing cadence is used heavily for is calling external systems and that call can fail. So what you do is you wrap them in activities. They can contain any arbitrary code you like and activities are executed at most once and can be automatically read on failure. There's a couple of restrictions around activities. Replaying must not produce new history events. Activities are executed at most once and this ensures that once they succeed and the result is basically fixed or immutable. The other restriction is around workflows, the actual sort of top level of the workflow itself. Replaying again, as I said, must not produce new history events. So due to replaying workflow code is executed multiple times and that's by design so it must be deterministic. So you actually have to use some special built-in methods for things like time and sleep, random number generation and then nothing that has side effects otherwise something will blow up at that point. This example here actually shows the use of the inbuilt time function. So what are some good use cases for cadence? Well, essentially when you've got a very large number of running workflows, things that are long running processes of a sleep and scheduling are also a good candidate for cadence. Anything that requires any stable, fault tolerant application execution, complex workflows are a good example, integration with unreliable external systems, integration with streaming and event-based systems, for example Kafka and the other constraint I guess is that really it works best for short workflows with hundreds of steps rather than with very long workflows with very large numbers of steps. This can have a side effect in that when the history is being replayed on failure it can take a longer period of time. So that's perhaps the one case where you don't want to be using cadence. So what are some good examples for finance, retail and as I mentioned at the start, delivery. So hence, spinning your drones, a drone delivery demo application is what I came up with. So drone delivery is sort of complicated. It's got a lot of steps in order to be able to have some drones starting at a base ready to go. Then an order is put into the system by a customer. They want some donuts delivered or something. So the drone has to get that request, it has to go and actually physically pick up the order. It's got to then fly to the actual location where the delivery is to be made, deliver the order and then fly back to base and get recharged and repeat and get ready for the next sequence of steps. So it's reasonably complicated. The actual flight for the drone is quite complicated. So here's an example of a drone delivery flight on an imagined desert island or something. Basically there's a drone base that's got to fly to the order location, fly to the delivery location and then fly back to the base again. That all requires fairly complex calculations. Remember this is all the simulated. I'm not actually flying drones, but I've done everything that would be required for real drone flight basically. So the drone flight path is computed inside an activity. That's where all the complex numerical calculation happens because it can fail and you want to know where your drone is at all times. It uses location distance, bearing speed and remaining charge and it does this every 10 seconds. So it moves more or less in real time what's going on. If one of the activities fails, the drone won't just suddenly crash and fall out of the sky. It'll continue flying from its last known location. So just having a bit more of a look at the actual, the overall system, I'm using some swim lane diagrams here to explain and the different parts of the systems involved and the steps involved as well. So here's the first workflow. There's actually two cadence workflows. This is the first one, the actual drone workflow and this basically does the stateful sequencing of the drone starting, being ready, waiting for an order, doing all the movement activities, recharging and repeating again. So that's relatively straightforward. There's actually a second workflow as well. Orders themselves are actually stateful. So I've turned those into a cadence workflow as well. So for the orders, the order starts, generates the location where it needs to be picked up from. It gets ready for the drone to come and pick it up. It updates the locations continuously based on the drone location. And then essentially if it's delivered correctly, then it ends successfully at the end. So because we've got two workflows, we actually need some sort of coordination between them. And this is an example using one of the primary mechanisms in cadence which is signaling. So it provides a built-in mechanism for asynchronous signaling between workflows which works really well when you've got multiple workflows that need coordination like this. Okay, so just to make things a bit more fun and perhaps realistic as well. And because I like Kafka, I introduced Kafka into the architecture. So cadence meets Kafka. Kafka for those who may not have come across it, it's pretty common now. It's a distributed pub subsystem for stream processing. It allows producers to send messages to distributed consumers via a Kafka cluster. And it uses topics to loosely couple the producers and consumers. I integrated that cadence architecture with Kafka and this adds basically a cadence cluster with three Kafka topics, three Kafka producers and two Kafka consumers. And what you can do essentially is start a workflow from Kafka. That's the second star up there. So we're actually using Kafka to initiate the cadence order workflow. And we're also using essentially a Kafka microservice to coordinate drone and order workflows. So this is where you can actually make sure that the drone picks up only one order and that all orders are actually picked up. So that was quite a good use case for Kafka as well. So just showing more or less how the system works in its entirety. I'm just gonna go through a few steps here. See what's going on. So first of all, the customer places an order with some sort of an app or something. The order is sent to Kafka for the new orders topic. Step two, the Kafka consumer gets the order it starts the new order workflow using a cadence client. Step three, the order is ready for the drone pickup. This is basically an activity where it sends the order ready message to the orders ready topic. Step four is again, back in the drone workflow. So it's an activity, it sends the drone ready message to the drone ready topic. The Kafka producer gets an order ID and sends a signal back to the drone and starts the order pickup. Step five is in the drone workflow, it flies to the order. So that's again, in an activity, it flies to the order location and picks the order up. Step six in drone workflow, it flies to the delivery location and that's in an activity as well. It flies to the delivery location and hopefully does something else. Yes, okay, so it flies to the delivery location, drops the order, continuously sends location updates to the order workflow which in seconds so that the order itself knows what's going on. People can, so the customers potentially can get an ETA for the delivery and check to see where on the map their delivery is and also so that the shop that's supplied the order knows what's going on as well. Step seven in the drone workflow, it flies back to the base again. So once it's flown to the delivery location, it flies back, it recharges and back at the base and is ready for starting a new drone workflow. So in fact, the drone workflows are never ending, they actually use a particular mechanism and cadence which essentially allow them just to repeat. So they actually are more or less permanent workflows at that point, whereas the order workflows do actually end. Step eight, back on the order workflow side, it receives the update location so that it receives the state and location signals from the drone workflow updates the state and finally if successfully delivered it actually ends the workflow. And that's it, as far as the architecture goes. So one thing I'm really interested in is performance and scalability and that's one reason that you'd use cadence is because you'd need lots of workflows. So I didn't experiment to see how many drones, simulated drones again, we can actually fly at once. Using this particular cluster, it's not a particularly big one, you can scale it straight out horizontally in any direction you like. What have we got? We've got six virtual CPU cores for the cadence part of the system, 18 to Cassandra. So you can see Cassandra is actually the part of the system that needs the most resourcing and on the client code side, running on some EC2 instances that was eight virtual CPUs, giving a total of 32 virtual cores for that system. And with some load testing, I did manage to get 2000 drones running currently. Which also means you've got 2000 orders concurrently as well, which gives you a total of 4,000 concurrent workflows running with that system. That's not a particularly big system. You can easily scale up to hundreds of thousands. And if you want to find out a bit more about cadence and other open source technologies, there's my URL, my blogs. I've written about 100 blogs in the last six years on open source technologies. This is just the latest in a series of interesting technologies I've had to learn and demonstrate and with a bit of luck, I think I've got an example here. So this is where I live in Canberra. I've assumed my backyard is the drone delivery headquarters, the base where all the drones start from. And I'm in luck, I'll be able to set it going. And essentially the orange icons there are where the orders are being picked up from. The moving darker colors there are the drones. So they go and pick up the orders from the shops. And once they've done that, they then head off in a different direction to the red icons, which is where the delivery locations are. And this is only, I think I've only used 20 drones in this simulation. And this was done actually using JavaScript on in the browser, subscribing to a Kafka topic that actually had all the information about the drone locations and other information that needed to generate this interactive map. So as the drones pick up the orders, the orange colors disappear. They're not really interesting anymore and they head off so that one's about to deliver in order to a particular location. Once it's done that, the color of the icon changes and it heads back to the base again. So that'll run for about another couple of minutes. I'll take any questions perhaps at this point while it's still just going on in the background there. Thanks a lot for the talk. I'd just like to remind the next speaker, I think Jiang Xu, Jiang Xu. If you could start getting your laptop hooked up for the next talk, because there was a little bit of delay this time. Yeah, please go ahead. Any questions at all? That must have explained everything you need to know about cadence. That's good. Cadence is a lot more, it's a great tool for developers. So if you're interested in building complex workflows that are scalable, have a look at it. Then it's all open source and all very developer focused. I have a question. I know she have Java and Go support. Do you have any plans for Python support? Yeah, look, I'm one of the actual developers of it, but we do work with the development team quite closely. And from memory, I think Python is one of these supported languages that's not really officially supported. I'm not sure if there's any plans for any other languages at this stage. This is sped up, by the way. It actually takes about half an average delivery. They go at about 100 kilometers an hour of the drone, so they're really annoying. They go, psssh! Did you add your own address? I did. Well, no, no, because, no. But I am the sort of the drone lord at this point. I've got the base in my backyard, so yeah. All right, I think that's all from me.