 I'm making a huge mess These who will be covering workflow is a service. Okay. Thank you Good. Hello everyone. And thanks for coming. My name is Lin Xian Yes, it's a Chinese name and I heard many times that people would call me Link-Sien because you know the pronunciation of the letter X is different between Chinese and English so for people's convenience I Picked an English name for me. So you can also call me Larry easy enough, right? and I just came to New Zealand half a year ago from China and It's my first time to live in an English speaking country. There's a lot of things I need to learn especially language and Currently I'm working at Catalyst in Wellington and in our company. We have a lot of talented people specialized in designing Implementing and supporting enterprise-grade systems using open source technologies and at the same time, I'm also an active contributor in OpenStack, which is Another open source software for creating private and public cloud and of course in Catalyst we We are running a public cloud based on OpenStack and in fact the OpenStack consists of several projects that are available freely under Apache license and Most of them are written primarily in Python language So today what I'm going to talk about is one of those projects called Mistro Which is a workflow as a service in OpenStack And I will have a brief introduction about what's in workflow and what's Mistro and how Mistro can solve real program problems in our daily life and I will introduce some interesting but useful features in workflow in Mistro and in the end I will do a demo to give you a basic impression about how Mistro works So because Mistro is a workflow as service, so maybe the first question we need to figure out is what is workflow and Basically, no matter you are developer or operator or you are a marketing or a sales guy almost every job you do every day can be described as workflow, right and Actually, I searched with the workflow definition in my English dictionary. Yeah, I use dictionary to learn vocabulary, but I found There the information there is not very helpful. So I refer to Wikipedia. I got this So it says a workflow consists of an Oxpirated and repeatable pattern of business activity Enabled by systematic organizational resources into processes that blah blah blah. Yeah Every time, you know, believe it or not every time I read it. I found it's Still difficult for me to understand so let's just think about a real example so as I mentioned in the beginning in catalyst we are running a public cloud and our cloud Contains a range of services for example cloud computing service and blog storage service Object storage service and so on and everything in our cloud is metered and Follows pay-as-you-go model and So we have a billing service You know to take care of the resource usage and Charge our customer so we can get money So the first thing of our billing service do is to get a resource usage of customer Because we run the service every month. So there may be a tens of thousands samples of the outputs so the first thing we do is Data transformation because we need to get rid of the invalid data for some reason or We need to convert the measurement unit from bite to gigabyte and something else and after that We need to get product price definition from somewhere in internally and for every type of resource and Before we calculate the final cost We need to apply discount to some of our customer who have Vultures or cloud trials or any other different type of Credits because in Catholic's cloud we have different discount scheme for different customer and After that we can calculate the final cost and apply add sales order in our internal ERP system For review and if everything looks fine, and we will generate the final invoice and Send the invoice to our customers So the whole process here is a typical workflow, right? Our billing service is just a do every job step by step But we have a option here. We assume the task in each step Works fine, right? But in we all know that in reality, you know fillers can happen here and there and We need to take care of the everything you know to make them works fine So think about what if the host our billing service is running on just a crash during we add in sales order So in this case, maybe we need to rerun the whole workflow from scratch or We can you know to store information somewhere to avoid duplicates sales order and Another possibility is as a developer. Maybe I want to debug a particular task without modifying the code That sounds impossible, right? But later on we will show you how I can do that with MISCHO So with all those in mind that we can define a workflow So basically workflow consists of one or more tasks and the connections between them so in MISCHO we call the connection flow and in that flow you can define how to proceed the whole workflow based on some conditions and workflow and the task Should have state so you can know what's going on for that workflow and you can know which task is the running and which task just finishes and also The purpose we run workflow. Maybe just need to get some result, right? And each task may also have a result that can be used by subsequent tasks and data in workflow, maybe the workflow expect some input from our user or from some external systems and Also the task can also have data associated So when consider Implementing a workflow management to software there may be many other factors we need to consider for example, we need consider, you know, we want the workflow can be executed in parallel and we need to to take care of the failures, we need the recovery and sometimes some tasks can be can be run for a long time and In this case, we need to support us a synchronization, right? And other factors such as monitoring and scalability so What if we Implement such software in Python so I can imagine there will be a lot of try accept finally blocks in your code and In order to support a running task parallel Maybe you can use multiprocessing or a threading library, right? And we can also need the database technology for data persistency And maybe we need another software or program just for Monitoring because you want to know what's going on with that workflow and what status of the tasks in that And also there will be more and more workflow running at the same time. So in this case, maybe they will Assume all the resources of the on the host so you may consider buying more servers and If you are doing things like those I have to say you are lucky because You are here today, and you will know Mistro, which will see you a lot of time and as I Said in the beginning Mistro is one of OpenStack official project created in 2013 which means it's stable enough and it's also Implemented in Python. It provides the rest for API and the very convenient command line tools For a workflow definition, we use YAML because I think most people are familiar with that and we also use Yaku it's a query language and It's typically used for a simple conditional evaluation and transformation in Mistro workflows and Because there are many cases You know you you need to decide whether to continue workflow based on task results Or there are needs, you know, you need to convert the task result to another data or structure so Yaku is very powerful for that and in OpenStack community Yaku becomes more and more popular and Mistro currently support more than 3,000 action definitions out of the box But I have to say most of the actions are Aiming at interacting with OpenStack, but we still have several generic action You can use for example, you can use our action definition to sending email or SSH to remote host and something like that If you found Mistro doesn't support the action definition you need you can even define your own action and Just plug it into Mistro so you can use it and so here is a Simple example of workflow definition, so you don't need to understand the exact meaning of it I just want to show the basic structure of workflow definition so you can see here we in a workflow definition there are workflow type and You can define input and output And there are three tasks here and in each task Just to run some action and you can also define the task result by publish and You can also define what to do after a task finishes and actually Mistro have a lot of feature to let you you know design the powerful workflows and Today I just want to introduce some of them So the first is conditional task transition. It's a basic feature that you need when defining workflow definition so here is another example and in this workflow, we have input, you know, we have a receiver and We also got three tasks here and the task one just to get print the receiver name and we use Uncomplete here to decide which task needs to be run according to the receiver name So this example is very similar to the old ways, you know, if you want to call someone so for example, you have two friends called Tom and Jerry and There is a telephone operator You just need to tell the operator and say hey, I want to call Tom and the operator will refer you to Tom according to your command very easy, right? and fork and join sometimes some task Can be run only if all the preceding task finishes and it's successful So this is a more complex example, but to this example you can use this example, you know to use catalyst cloud and We have three tasks here the first task is to create a virtual machine in the cloud and The next is create a volume in our blocks garbage is service The third one is attach the volume to that virtual machine. So apparently the third one can only Can be run if only the VM and the volume are Both created successfully, right? So you can use join all you know to wait all the preceding tasks loops sometimes you want to do some iteration of a list so in this case we can use with item cloud here Just like this example you can define an input and You expect to get a list of links and in the task it just Get each link and use HTTP action to get the web content So if you are a patent developer is very similar to the code on the right side You just define a function and use the for loop to do the things you want task policies and Personally I think Task policy is is a thing to make workflow definition powerful In Mistro We support several task policies like retry Wait before wait after and pause before timeout So you can guess the meaning of the task policies from their names, right? And for example, you have a task that can be run for a long time and you don't Want to wait forever So in this case you can just simply use timeout policy and say hey, I I can wait you're running for For example for maximum of half an hour if not just feel the whole workflow and There's another two policy very similar one is Wait before and the other is pause before so the difference is a wait before The task with wait before policy just wait for some seconds you defined But the whole workflow can also be run after that that that that time you defined but pause before means When and when the task is running maybe it will pause the whole workflow and The workflow can only be run if you you know to rerun the workflow manually So later on we will show you an example to use pause before and Contrigger and Actually, Krone trigger is not for workflow definition Krone trigger is a separate feature in Mistro Krone trigger is the Object that you can run the workflow on schedule so you can define which workflow can be run with what to input and you can also define how often the workflow can be run So if you are familiar with the Krone tab in Linux, it's very Same thing, you know, it's a cloud alternative to Krone tab But the Krone trigger support high availability, which means because in in Mistro deployment we may have more than one Mistro engine processes and Even if there are some processes just die and the Krone trigger can pick that the healthy processes to run workflow, so it's very helpful and Here is an example for creating a Krone trigger Using the command line so you just need to specify the Krone trigger name and Specify the workflow name and give it a pattern and say I want to run the workflow for every five hour so If you create the Krone trigger, you could say you could see there will be a workflow execution for every five hours very powerful so demo time so on my laptop I have a Fresh deployment of Mistro. Can you see that? Okay After installation By default Mistro will create some sample workflow. We can see So by default there are two workflow defined one is delete instance and there are these created instance because you know as I said Mistro is just for Interacting with OpenStack most of the action definitions, so we can also see the action definition here a bunch of them and most of the action definition is for other OpenStack services, but also we got some standard action So you can see we support HTTP and JavaScript and then some other SSH Something like that So as an example, I have our workflow defined here So this is our example in this workflow We expected to get some input as my name Variable right and we define the output and the workflow contains six or six tasks and the task one Just simply print hello world and Goes goes to task four and the task two will wait for ten seconds After that it will print hello Mistro and go for a task three and Task three have a policy defined the pause before so when the Task three is touched the whole workflow will be paused and if we manually Start that work execution the task three will print hello KB Picon and then go to task four and Task four have a clause defined here join all which means the task four will We will be waiting for the task one and the task three and then It just do iteration from a global environment with I defined here and just to print the names and According to According to the name here the task four will decide which task To run according to my name, right? So just for For your convenience, I just draw a simple workflow chart here so this is a Description of that workflow definition. We have six tasks here and the task one Simply print something and go to task four and we have different policies defined in task two three four and The talk of task four will Go to task five or task six according to the input name So the first thing we need to do is register that workflow So in mistro Sorry We just create our workflow. So we have a workflow here. Oh, sorry because just now we use We use a global environment here remember so Before that I just registered a global environment in mistro so it can be used in workflow definition so we can see the Environment here So in the environment definition, I just to define a dictionary The key is names and the value is all my name Lin Xiang Kong and Larry Kong, right? So it can be used in workflow So with workflow defined and what we're going to do now is To execute that workflow. So we just create an execution Based on that workflow and Copy and paste for three time and also because in workflow definition, we expect some input. So we're gonna Put the input here So my name I gave my name a value Lin Xiang Kong, right? And I should use the global environment I defined before in this case. It's my ENV so we just Run that workflow So as we expect the whole workflow will pause So we can see the task of the execution So you can see there are already two tasks here and the task one is success task two is success and remember task three just wait wait for a Sorry task three is pause before So task three status is idle because it didn't run actually and Task four need to wait for task one and the task three Right. So task four status is waiting So in this case I Just need to Continue the execution. So the execution The status of execution is running and for a while We can see the execution status again It's a success and because we defined a result of the workflow so we can check the output Execution get output So you can see the result is found my name because the input given here is Lin Xiang Kong. It's my name, right? so the second time What I'm gonna do is I Gave it a running so let's say James It's not my name, right? So this time We can also check How the workflow can be run Now wait for a moment and the mist the execution could be pause as We expect Yeah, it's paused and we also continue that execution. So this time The output should be not found my name because I don't know who James Kohn is So we just get the output to check You see it's not found my name and the third time I Still gave it a running and run the workflow and After the workflow goes to pause the status We can check Yeah, it's paused. So this time I also continue the workflow, but the difference here This time is I gave another extra parameter so So this time I still we run a continue the execution and to make it's the status from pause to running but this time I just gave an Param here and This param here is just a temporary environment Dictionary so I just put James Kohn here, but without modifying the global environment right and Oh, sorry the idea is wrong and we can check the global environment is still Lin Xian Kong and Larry kong but this time if we check the output You see it found my name. I didn't Modify the workflow definition and I didn't modify the global environment and I can you know to make the wrong input and get the right output so this is I think mistrust very powerful and another example I want to Show is how to use Krone trigger and For the Krone trigger example, I want to create another workflow this time Very simple, right? It only contains one task and just print. Hello world and We just create a Krone trigger Using the workflow. Oh, sorry. I should register the workflow first and Then I just create a Krone trigger using that workflow. Oh, sorry the first parameter I should give is the Krone trigger name so Let's see a Krone and We use the workflow name and we give it a pattern and We say please run the workflow for every one minute So after that you will see a new execution for every one minute So you can check the execution Yeah, so there are still Execution that is created just now and the new execution could be created Yeah, you can see this one In the description it says workflow execution is created by Krone trigger and In another minute, there will be another work execution created. So That's the demo I showed today and maybe later on we can check the execution list So actually there are more useful use cases that workflow technology like a Mistro can be used and For example, you can use Mistro for integration or arbitration of system and services you can also use Mistro to develop your application and For other long running processes and Even for deployment If you find what you are doing now is very similar to the use cases here Maybe you should consider use Mistro And there are also some Useful links. It's our github and developer guide and some simple examples So let's just check the work execution You'll see there are one more because maybe a minute passed, right? Okay, that's all I want to show you today and hope it will help for thank you So any questions? You know great talk. Thank you very much. I was wondering what happens when actually one of the tasks fails You get an extra route does it stop the intervene? What are the possibilities? Yeah, you can define what you want to do if a task fail because Except for a besides for the on success clause. We also support on failure So you can define what to do if the task you want to run is failed It's no up to you You can feel the whole workflow or you can do some recovery You can do some backup You can define that Yeah, yeah, yeah, right All right, then I suppose we'll wrap up that Our next talk we starting in about 15 minutes Mishan, yeah, thank you