 Welcome to my talk. I'm happy to see a few people here to be interested in workload automation Before I start a short question just raise your hand Who did work with the IT workload automation or enterprise job scheduling before? Okay, too. That's good. So there's Few people here to learn something. I hope okay I want to talk up to speak about workload automation because thinking that right at the moment the demand is rising on that point and the intent of being here and talk to you about this topic is to motivate you to reflect on the way you are solving automation problems in your Linux environments and And maybe to find out that there are better ways to do that Okay first we will I will just do a short introduction upon me and my company my company. It's called independent integrative technologies maybe I find some time to Say something about the integrative in there because it's very important and And Yeah, this is a quite small company. There's a me and my Co-founder of the company. It's Ronald Jenninger. He's a mathematician. I'm a computer scientist so I'm not a business marketing guy and You will also see this talk will not be a very technical talk because it's not not possible to explain our system in technical way in 20 minutes but it's more a Talk a general talk about the topic but I'm into nearly every line of code. So if you're interested to And have question on how we did things just come to our booth and then I will answer every question Just a little bit of history our company was founded But in 1997 as a consulting company for database projects, so we are did consulting jobs in data warehouse projects and telecom and banking area and during our work we found that Doing automation of processes especially in data warehouses where it gets complicated is a big issue and To have a solid automation is important and that brought us to the idea to start developing and Scheduling system to automate all those processes necessary to build up those systems and Yeah, so that was about 20 years ago and then a German linux available having the list Came to us and said what about bringing an open source release of your system because there's no open source Enterprise job scheduling which is usable And I said, okay, let's do that. And so we Released schedule leaks. We will talk about that later as an open source enterprise scheduling system and I personally think it's Yeah, it's the best one available Maybe the only one which is really capable to do a good automation Okay, so What is workload automation basically it's the Coordinated and automated execution of processes in an heterogeneous network. So simple speaking just started starting of programs running programs monitoring programs and recording What's the result of those programs or whether they were successful or something else and Around that there's a software class. It's called enterprise job scheduling systems or workload automation systems to support those automation tasks and Processes so programs have to be started on demand or just time like using Kron or something Or another events using dependencies or resource changes or other events in your IT which will just trigger the execution of programs or whole groups of programs or a whole batch chain or something like that and That's the first point where we will see that Kron doesn't fulfill the needs because you need dependencies you cannot Process or load the data before your FTP transfer of the source of the data has finished transferring the data to your system so you have change of chains of processing which have to be managed and Maybe some of you already know that This takes 15 minutes. So let's get you the next one just half an hour later than we all we will be good and it will work Most of the time but first time it doesn't work It will maybe produce a catastrophic result and you lose a lot of time Um, you have to have a look at resource usage. You cannot just run a hundred processes of Something at once you cannot run on your database system 20 real huge queries producing loads of temp space usage and so on so you have to Manage that to control the resource usage So you need to control the load and you maybe you need some load balancing to distribute The work to more than one machine To be to scale your system so a simple example if if you if you are Rendering a CGI movie and you have to render a thousand frames You will need a system which is Disputing the rendering process to your render form and do a workload balancing for each each frame and You need something which is which synchronizes your Processes so things which are not really directly connected so You have some programs which creating reports and you have other programs or Batches or scripts which are Updating your tables so you shouldn't run a report During the updating of the table because your report might deliver inconsistent data, so you have to synchronize that so Let's look a little bit to the historical background This should be a image of a mainframe, but it's blue genius a supercomputer. Sorry for that But I'll talk a little bit about mainframes because as other the mainframes and mid-range system where the root of IT before Linux and Windows and all those things and That was mainly batch processing systems and Those systems job scheduling was just standard because they were just batch processing systems. So everyone had this so people which Coming from those mainframe side of computing you don't have to explain then why you need a job scheduling system There are adaptions of those systems for Linux and Windows and Unix environments, but they are not widely accepted and that's for Too easy reasons those systems were adapted from the mainframe Philosophy, so they are quite inflexible. They're bureaucratic. They are mostly Railway scheduled based systems which work fine if you have every day the same load and You're quite safe that your processes will succeed So it's if it's all working smoothly You can live with that and you have a railway schedule and it will not break but in Today's complex environments. We have a lot of failures because of snapshot to old in Oracle or out of 10 space or File system full of what else so if you have a huge environment You will get very very quickly into different problems every day And you know you need a really flexible system which can deal with that and railway based schedule Systems which have a daily plan Planning everything at Timing level are not able to do that on the other side we as a Linux users We are coming from a world where we have a very narrow view a more interactive view Development systems are just interactive systems and maybe standalone server systems and in this systems Scheduling is Not really a topic because you don't have much batch processing and it's even if you and that's also a problem I don't want to go into that because then I talk a half an hour just about that is the thing that Today's developers. I think they're just looking at their thing that they're building a software deployment system So they're just look at that the other ones are building database system They just look at that and they forget that if we want to make Linux great in corporate environments we have to integrate them all and to integrate them they have to play together and to play together you have to schedule them To make sure that the backup system doesn't start back up When the data the heavy data processing of your it here loads is still running you shouldn't Because it would waste a lot of resources and might cause problems Okay, and that for that most of the Linux users are not Know anything about this kind of software and they also know Don't know what this kind of software can do So why what has changed that we think that this is more important today than a few years ago? The first thing is Linux systems are no longer islands in corporate IT systems the success of Linux came when Yeah, let's exchange those Microsoft file server with the next file server or just use a Linux system for as a mail server or use a Linux system as a web Server, but it's always this island thinking so just have one system. It has one function and it will do the job but if you think upon workloads from Mainframe systems They will will get more and more transferred to our decentralized Linux computers more and more core processing of banks stay and Processing of financial systems and big data warehousing And so it's more and more moving away from mainframes to Linux systems And so you have get more and more interconnections you've got more and more processes to automate That's more and more virtual or real machines to handle it's more and more processes to automate in all those systems and We've got new technologies like yeah data warehousing big data artificial artificial intelligence intelligence, which have to move loads of data around and do professing change to to Get and result Those systems can work on at the end So there's a raising demand for a solid IT workload automation we think in the Linux world, but there is none so What do you do? Yeah you use the scripted automation approach, so you use cron and The huge number of scheduling mechanism built in Software systems a backup system have a known scheduling system your ETL system have an own automation system Your software into DevOps integration system has an own Property automation system to do that. So you end up with many different Automation systems and each one is just built to do this task. It was built for and No one has built a scheduling system with the idea to be able to Control anything Yeah Okay, so I have to really speed up. Sorry This all will be glued together by a lot of scripting and This has consequences and drawbacks High development costs and maintenance efforts lack of documentation was already what we were talking about before and We have a complexity which is growing exponentially Because the systems will rise in which every new process to integrate it gets huge and there are more interconnections to take care of and So scripted automation just doesn't scale it if it reaches a certain point of Complexity you're in a quite big danger that it will just break down and We have a small story about that our mister or missus nickel. So Looks like a mister. Okay. He has to manage his IT and he has a workload So our blue circles are just processes which have to be automated to run every day to transfer data to create reports to do data transformations to consolidation data warehousing and loads of things and You're quite easily in a in a you have in your company about a hundred thousand of those blue dots to have to be or much More to be executed every day. So what is he doing? Yeah, he uses drone and built-in schedulers and a lot of scripting some meter data temporary date control data so you get a scripted automation which is woven into every production system and it's constantly growing and For the same problem you end up with ten different solutions and It gets more and more complicated a big of alert workload will become so What is every day problems? Um Report is out of date. So he has to dig into all his automation things in log files to find out what's going wrong What was the cause of the problem and at the end if at the point where he realized what the problem was He has to fix a problem, but then he has to fix everything which was Done after that. So he has to repair his system and he has to do it by hand changing groups and Commenting outlines of scripts and rerunning scripts and Real trouble and in addition he has to integrate some ETL process which is newly developed here again he has to to find remember and and review how the data warehouse is automated and again Editing scripts and doing programming to to get everything right and Because it's not his best day. He can have a crash database system causing problems and again. He has to find out Which programs failed and how to restart that and what can I restart without causing troubles and on other points in my my company so He just got a lot of problems because he is in a time stress. He has to fix things and To do that, he needs all those deep knowledge upon this I call this red things automation hat he to not that to hold it that it doesn't fall off and That's a better way to do it So if you use a enterprise job scheduling system with Schedulix, you've got a central server you model your Jobs your processes to execute in the repository of the server you connect them you build up all dependencies all Resource usage things and you have a central system controlling everything you get completely rid of those Scripted automation had you don't have local scripts Interconnecting processes which is other and controlling the correct order of execution. That's done by a central server There is an agent, but that's a standard software installed at the remote systems, which is executing the things as the central server is Thinking to execute, but what do you get? You have a central monitoring operations you go into your office look at starting a browser and Check of the process state of your system in minutes. So you open it everything is green You know it's fine if something failed you see oh, there's something red line You go there look at the log file from the from the web interface and say, okay It was a snapshot to all just rerun and everything just continues You don't have to remote log in and edit scripts and doing scripting and programming to solve the problem So you've got easy failure tracking You've got a very efficient restarting and continuing of batches and backlog handling after some problems I'm changing and adding workloads. It's just a few clicks away if you think you have a script Who is executing three? Subscripts so three commands in or in row and someone tells you please paralyze it Let's do a concurrent execution. So you have to do start them in background into the signal handling and and traps in in shallot so Taking if you're really good, it takes you an hour, but you have to be a very Specialist do that to have a production proof Tested thing of that in our and schedule. It takes minutes to do that and So you've got drastically removed development costs because you don't have to script automation and you don't have to Deploy all your automation scripts all around your system So at the end your system, it's much more stable and it will be also more reliable because for example if if you find out that every now and then a job fails because of A file system full in 10 space and you find out that oh there's another process running at the same time and they Clash it because they both need too much 10 space if you have a scripted automation it may take you hours to Solve that problem Within schedule leaks will just create a resource and say To those things that they should synchronize on this resource And the system will just make sure that they will not both run to the at the same time Think like that. Um and Many more I could talk of ours about that okay, so Scheduling is open source. It's free to use and it can be used from the beginning with minimal costs It's always better Than scripting any automation hut to say it's easy. It's just let's do it quick and dirty It will always will lead to problems later. So it's a short step of installation of that system and a few days of Training and reading but then you have a system which will guide you even if your system grows to many hundred thousands of processes to control and If if your project grows schedule leaks growth with it We have done benchmarking Telling us on the smallest IBM power 8 machine which exists without big tuning. We simulated a thousand agents and Controlled more than one million processes In fact, we had a high mark of three or four million But I'm the defensive I'm saying more than one million But with a tuned system with because it's the the limit is just this guy over the underlying database system It can be stressed much further Real load is our we have a large customer in Germany which does this data warehouse automation Executing about 100,000 process programs every day to get to stay daily data warehouse processing done and if if you need higher functionality and If you're interested I can explain where the differences are scheduling is full-fledged in doing scheduling In doing scheduling. It's not like a teaser where you start with it And then one day later you find out you have to go to a professional. It's not that way So we'll just take a minute and then I'm done So it's open source and it's free it's 20 years technology proven and Reliability and a customer satisfaction. We haven't lost one customer in those 20 years who decided to use it It's always on the same patch level than our commercial system There's a Google forum where you can get help and our mission is that We should implement Schedulics as a standard thing like Apache for web services. Scheduling should be the platform for workload automation so Last thing just click it through here With it our booth get information. Have a look at the website. Watch a few tutorial videos There are white papers with a lot of information on that you can contact us you can install it easily without cost Register to our Google group to get help and just join the community and come to me at my booth If you have any questions that thank you very much Whoo Hope it wasn't too fast at the end but landing right on the 25 minutes It's really impressive so questions See fee Hello. Hello. Maybe not. So a is Is Schedulix available on any Linux distribution and do you know of any interesting? Free software projects which are using Schedulix. I'm thinking maybe Projects like GitLab it would be interesting to use this kind of okay. We have RPM packages for redhead and centers We hadn't got the time to build other packaging, but there is a let's say raw installation guide So you can download the software and compile it on any Linux system and follow this installation guide to Install it maybe if you know what you're doing takes you an hour or so But that's not so so much automated thing yet. Yeah, we're still working on that Let's take take that one offline. Just keep the questions quick. Please. There was another one up there somewhere. I think no Okay, sir, we can talk that Go ahead. Yep. Can you follow the modern dev ops process and configure using code or does it configure using some kind of web interface I'm not really in understand what you Can expand on that later. Yeah, all right. I think we'll cut it there. Thank you very much Thank you