 Are we good to go? Okay Yeah, so hello and welcome to my second talk today With my previous talk I had the slot before the lunch so I was Preventing everyone from going to lunch now. I have the problem that everyone of you had lunch I suppose and Heading into the digestion coma now To prevent that I've decided to do my talk completely in all caps No, I won't Just a short blurb about myself my name is Jochen Liddich. I'm from Germany and I've been a sysadmin All my complete all my professional life discovered Linux in 1993 Started doing sysadmin work then went into it training mainly open-source training training for system administrators Which brought me to Web DE one of the biggest web portals in Germany I first did a few months of it training there and then was offered a position as it trainer Where I started Building a my own team for the first two months I was the team and Then I started hiring additional people and we are we're Running the billing system for example and other things we built the data warehouse system and Eventually were acquired by one-on-one Where I became head of it core services a department of Three teams with about ten admins each and we were providing the complete it department with central services like the backup system for 50,000 servers we were doing central tools like software deployment and Running some security intensive subsystems in In 2009 I decided to start another business and I founded fry steel IT Which is a IT shop basically doing ops for devs. We are targeting developers and Taking care of all things IT so developers can focus on their projects and It's our task to get up at night and replace hard disks or restart some Apache servers We are at fry steel on Twitter and Our main product is triple concept, which is a platform as a service. That's completely optimized for Drupal We started with Drupal concept in 2010 at the Drupal developer days in Munich and Since then have been growing our hosting infrastructure and growing our customer base all over Europe Drupal concept follows the same concept We provide and manage the complete IT infrastructure and our customers can focus on building awesome websites in all those years I Discovered that doing DevOps is hard and if you look for the problems there are many First DevOps is a huge topic It's a broad horizon Doing everything from the idea of an application of a product Getting it into production and keep it up is not only a huge time Space but also Includes many many tasks You have developers and you have admins and then you can go into more detail and say, okay There are database admins. There are storage admins. There are system admins and so on so all these people have their own problems their own advantages and weaknesses and You'll see that when you're doing a project that then composes all of them There are many principles. There are many internal or external people you do work for and Of course Each project they are doing is the most important and for each project you're expected to give it the highest priority Each of these projects is isolated. So it's competing For resources you have in your DevOps team And of course, you never have enough people You never have the right people with enough time to do all those tasks within those projects and Especially in the ops part of DevOps. You also have a lot of interruptions. There's always an incident There's always an outage that prevents you from doing the work you Planned to do or you were supposed to do and With project deadlines Looming you get always into trouble if there is an outage that Takes you more than a few minutes to resolve so In consequence, there's a lot of Load from time to time Most of the times more than you'd like to have and there are also times where you don't have much load for example between projects or During those times in a project when for example, there's more development to do than system administration It's hard to keep an overview over the different tasks that are to be to be done in a DevOps team Who does what? Who does what as a next step is a question that is most of the times hard to answer that causes your Planning to be quite poor It's hard to estimate how long it will take you to complete a certain tasks If you always have to take into account that there could be an outage around the corner so if you say okay, it'll take me two days to finish that and A database cluster goes down or another catastrophe is happening Then you won't be able to reach this this goal If you say okay, it probably will take me two days But I'll take into account that there may be outages and so you say a little take me three or four days You may be able to reach that goal before your estimated time But as soon as you tell the project manager he'll learn a lesson from that and simply Take half of the numbers you give him as a project estimation In turn that generates dissatisfaction on all sides and mistrust He told me he'll be able to finish that in four days. Well Did he take into account that there will be interruptions So I plan three days and get to my project goal earlier or should I plan additional days to to have space for the unplanned things and there's Growing mistrust into all that's Set inside the project context The more mistrust you you have the more external influence there will be by example from higher management and There will be People quite Unhappy with that job there are methods to deal with those challenges and One method that's gaining ground in the IT Space and especially in DevOps is Kanban. So That's what this talk will be about Kanban is a term that was coined at the half of the last century about 1947 by Taiji Ono who worked at Toyota in Organizing the production He took Goals of the lean manufacturing movement Especially the goals to shorten heat time to increase productivity and to create satisfaction and trust at all levels he chose Kanban as a method and Kanban is a Built from two Japanese syllables with can meaning signal and ban mean as meaning card because basically he took signal cards to Mark stations in the production line that were ready to receive another part of a product What you don't want into in a production line is for example a worker dropping her skew driver and Products start piling up before he's or her station So they used signal cards that signaled to the previous station Okay, I'm ready to receive another part and to work on that and until that Signal card wasn't there. There weren't no products coming in So Kanban relies on Visualization you can tell at all times which station is prepared to do another step of the work and It's based on a pull principle. It's not that we are pushing work It's that we are pulling work into as soon as we have the capacity to work on it. There's an interesting Use of a pull principle in real world and That's the Imperial gardens in Tokyo in Japan It's a beautiful place and You can enter it free of charge But at the entrance you will be handed an admission ticket All it contains is that it asks is a text asking you to return that ticket when you leave the gardens You don't pay for admission. So what's the use of this card? well The thing is People can only enter the gardens as long as there are still cards to hand out if all cards are on The way in the gardens you won't be allowed to enter not until Cards are left at the exits and brought back to the entrances There will be the possibility of entering the garden and that way They keep population in the gardens on a level that's Pleasing to all the visitors That's a pull principle take more in as Long as you have the capacity to do so and stop Taking more work in our context in if there's not anymore to spread the Kanban Method was Built as an adaptation of that principle in IT It's mostly David Anderson who Started using Kanban as a as an project management method in IT in about 2007 and these are the core attributes the Kanban method uses to Have more control over the processes in our workflows First visualize the order flow make visible where work goes on Hopefully as planned and where work stops So you can see where to focus on to prevent problems from getting worse The second one. I think is the most important limit work in progress stop piling up tasks have a limited list of tasks that are worked on per person or per team and Focus on what on these tasks get them out of the way as soon as possible and then get more pull in more work This makes it possible to measure and control how work is flowing through our teams and to our people You can measure how long it takes a task from getting into the team being worked on and getting finished and being passed on Define the rules explicitly So define for example what it means that the task is finished How can we see that the task is finished? How can a individual person determine if the task is finished? write that down distribute that rule and Have people understand How work flows from one station to another Building feedback loops so you see early when problems arise and deal with them and Then use models use science for improvement another Word that's Connected to Kanban is kaizen which means continuous improvement so Kanban is a Kind of agile way of dealing with workflows but it's very simple to implement and it's Creating results very quickly How does that work in practice? First desk visualization. I apologize for the non-translated title here So Visualization is done with a Kanban board This can be a physical board like a pin board or a white board and of course There are also digital equivalents you can use these boards are divided into columns and those columns Symbolize the different steps a product or a task or an order goes through You start with a backlog where you decide Which are the next steps? Which are the next tasks we have to work on and then you have columns? You determine yourself what these columns contain for example, it could be the first real stage could be a planning stage So you take a task out of the backlog into the planning stage do your planning and then you go into development Finally you go into delivery and a Task now is in production Every tasks goes from left to right over that board so for example There could be a task C in the backlog still not being worked on a Task B in the planning phase a task a could be In the development phase here you could now go on to work on task a and Move it on to delivery sometimes it's a good step to Divide a column for example if a task like a is finished with developing but not quite ready to Be delivered you can half that and say okay, there's a Task a is being worked on or task a is finished and ready for delivery and waiting for the delivery team for example or for the person Doing the delivery to pull it into their column and now it's important to add Another attribute to the columns, which is the work in progress for example, we could say At all times there shouldn't be more than two tasks in the planning stage If you already have two tasks in the planning column, we won't pull another one from the backlog development could be two as well and Delivery for example, if it's only one person doing that They should focus on one task. So we limit the work in progress in the delivery stage to one That's basically the way you do visualization of Tasks and their stages they are in It's easy to set up such a Kanban board You can use a physical port as I already said you can use an Excel table. They are web-based Kanban tools on the internet It's your choice If you go into more detail, we could differentiate different order types Such an order could be a requirement a feature that has to be developed and go productive a user story a Change request on for for the production environment a production defect maintenance work everything you are working on should go on to the Kanban board and By the for example by using cards of different colors you could Mark the different order types tasks also differ in regards of time so We could say okay. There's some kind of standard tasks. That's simply being worked on and it's finished when it's finished That could be also a deadline connected to the task That tells us okay. We are two days away from the deadline. We should Start working on that or there's Still a few weeks until that thing has to be finished so we can put it on the back burner I can be Wake deadline connected with it meaning Well do it as soon as you have time There are for example some maintenance things that aren't connected to any deadline so you can say okay, these are low priority tasks and On the other side of the scale that could be urgent tasks that have to be worked on immediately and these expedited tasks could be even As urgent as to Break the work in progress limit If there is a real outage for example, you can say okay tasks marked as urgent Don't care about work in progress limits. Everything has to step back and The the task the outage has to be solved first These should be total exceptions though Because as soon as your project managers See that work in progress limits can be broken they try and do that and they shouldn't do that it'll break the whole method and All the visualization will be useless without adhering to those work in progress limits Okay, that's the basics Now how do we get tasks into the backlog you should Take into account different Attributes of the tasks to decide to decide which tasks to take into the backlog In other agile method methods such as a scrum there's a There's put much effort into sizing a Task into determining how much effort is connected with the task. There are methods like planning planning poker and other things Kenban doesn't use such elaborate methods because In DevOps most of the time it's very very hard to determine real effort Since many tasks Have to be worked on by very different people you'd have to get to the table the DBA the system administrator some kind of storage administrator and networking expert and other people and the cost of getting all those people into meetings Which they do very reluctantly I guess is is very very high and To be honest not worth the effort There are far more efficient methods for example the one Okay dear project managers. We have two open slots in the backlog which one of you has the biggest budget and Do a weighted round robin by budget size? That's quite easy doesn't involve many people and You get the tasks onto the board quite quickly and what the your project managers will see is that tasks get off the board quite quickly so It's it's not very reasonable to put much effort into effort sizing It's far more efficient to get those tasks being worked on the backlog should be filled regularly and A frequency of once a week Is is most of the times a very good frequency? Then tasks have to be worked on and finally delivered Delivery should be regularly as well because it creates trust if you can Show Finish tasks for example also every week your project managers will gain trust in your estimations gain trust into the method method and So your support will be getting stronger by Support by management support by project management and other sites What's important and an important difference to scrum there is no time boxing a Task is finished when it's finished if you're finishing it early good get to the next task if it takes longer than you estimate it well get it done and Especially in more mature organizations. You can do a talk delivery. So it's really finished when it's finished As soon as a task reaches the done column to the right You can signal to your project managers. Okay, there's another task done so we can Get on with our work The highest form of that is called continuous deployment. So delivery Takes place all the time Equally to for example scrum. There's also a stand-up meeting where the people involved with the tasks That's the DevOps team or teams and the project managers convene daily and do short exchanges how things are and The goal of these stand-up meetings is process improvement. So everyone is invited to Show and tell where are we getting on well and where are the problems? Most of the time these problems can be seen on the Kanban board as well and I'll show you an example shortly There's also an operations review in Some bigger distances That's also involves management to get improvements all over the place. I've done a few comparisons already. So Let's look at these a bit more detailed Kanban as every agile method is far superior to the old waterfall Method of doing projects where there's first a Big planning phase then a big development phase and then the big bang release and a party Afterwards I think I don't have to get more into that and how that is a really inefficient way of doing projects of course and comparison with another agile method is more interesting and Kanban stands quite well there There are no fixed roles for example, so you don't have as someone like a project owner and Which is quite hard to pinpoint who is the product owner of your post fix cluster for example and Everyone is involved, but they don't get fixed role names An important difference also is there are no Limits to specialization. So your database administrator can be a database administrator and stay a database administrator Iterations are only optional. So you gain quite a lot of Flexibility here. There's also no time boxing as I already said and planning and delivery are independent from each other Which even increases the flexibility of the Kanban method Your central metric is lead time. So how long does it take you to finish a task? New orders can come in at every time. You don't need special Meetings to do that. For example, if you're using a digital Kanban board tasks can be added at every time The order size is not limited and as I already told you Estimations are not mandatory I've talked a lot about visualization. So I'm going to visualize how the Kanban method works in practice And I'd like to take you with me on a day in Kanban That's our Kanban board and that's the people who are involved The product the project manager in blue we have the developers and in green of course the operations guys and They start with using Kanban as a method to organize You also see the work in progress limits to planning tasks At all times let's start with the project manager Deciding what tasks are the first ones to get out of the backlog jail He chooses a and B which Reaches where the planned column now reaches its work in progress limit. Otherwise, I think he would be He would have added C and D as well now the pull principle Starts and there are two teams of developers One working on project a one on task B With a and B out of the planning column Our project manager is quite happy that he now finally can add C and D to the planning column work goes on and our happy project manager Things he'll need also JK and L In the backlog So yeah, that's these Now a is finished and goes into the delivery column and The team that has finished a can now pull C from the planning column into the development phase B already gets finished But they can't get it out of the development column Because I'm full with the ops guys working on a so B just goes to the waiting part of the development column That B is Waiting for delivery doesn't mean there's now place for a third project in development The development column still is full and now the shit hits the fan There's a problem in delivery Development Column and the work in progress limit there is to The project manager still doesn't know about the problems on the right side So he decides that K is urgent and it's gonna be next not a problem now the developers realize that they are blocked and You could ask. Well, okay, so Kanban is a is a method of Preventing people from doing that work Notism It just focuses all people on the problems in the workflow So the developers instead of being able to pull K or D They go over to their colleagues And say, okay, how can we help you? We have free capacity How can we use that capacity to get things moving again? There's a problem with the slide Can you work on K or D please and the developers have to say no We'll we start working on that as soon as we have solved the problem with a because still the work in progress limit In development and delivery prevents us from going on. There's a problem and we first have to solve their problem As I said stop piling up Go and come to the rescue while the project manager, of course things of his own things now also the capacity of the second development team can be put to gain more results because Obviously It's good to invest some time into doing some Test development to prevent those problems in the first place now even the project manager comes and offers help and somewhat later we have progress a That is planning and Our project manager does what the project manager does That's how Kanban works in practice It limits the work that is being done at a time Focuses capacity onto those tasks and it visualizes especially where we have problems If a task doesn't move from one column to another, there's a reason and we should do everything to resolve that reason and That's all you need to visualize how work flows through your teams how work is distributed in your teams and All you have to do. There's one thing you need to do Get management improve approval to do work in progress limits These WIP's are the key That the Kanban method works and if someone I've seen that if someone is allowed to Break those limits the whole method Breaks to the brakes as a whole That's all you need and then roll in a Whiteboard or use a digital tool and start visualizing your workflow You had a question. Ah, yes. Yes. Yes. I stepped over that. Yeah, people realize That's of course a possibility that's a part of the kaizen of the continuous improvement to realize, okay Things are moving more quickly in the development phase than we anticipated and so let's raise the work in progress maybe Sometimes you'll have to do the opposite to reduce work in progress because you see that Tasks always get stuck There are always problems with four tasks being worked on concurrently and so you maybe should lower your work in progress limit Maybe for some time or permanently so as they did You can of course alter your alter the rules under which you work, but I Can't stress that enough Don't break the web WIP's. I have a few links before we go into the questions There are of course books you can read the first thing To suggest is David Anderson's book Kanban successful ever evolution a change for your technology business in which he Explains how he took the production version the manufacturing version of Kanban and made an IT management method out of it there's also websites for example a mini book that compares Kanban and Scrum People using the Scrum method in practice have formed the limited WIP society which is a community of Kanban practitioners And on their website you'll find a lot of resources on that and For the German readers. There's also websites in German or I'm sure in other languages And if you're on the road as much as I am there's also a podcast That's a deal with Kanban named the ITK podcast and That's a short introduction into the Kanban method We've been practicing that in my business for quite some time now in a very small team, but it Has enabled us to get an overview over our internal task as well as our client orders and it's so lightweight that There's not much overhead to the method So especially if you are a small team, it's very very easy and very very efficient to use the Kanban method And with that I'm open for your questions The question was if there's a real serious problem getting a task further through the right Can we put it on the back burner basically and back into the backlog? To work on it at some other time you could you're free to do that, but it's a bit of cheating because that's again Hiding the the problems and Procrastinating on working on their causes. I think what's most valuable is that you see okay This task has been in that column already last week. Why didn't it move forward and Let's rally together and solve the problem. That's preventing it from moving to the next column of course, if there's a reason to to De-prioritize it you could of course put it back into the backlog, but don't make it a habit Yeah What I've experienced is also Most there are oftentimes internal tasks some maintenance things Well, they are expressed that someone should do that at some time and I'm for example quite Good in into thinking of many of such tasks and filling our columns with that But if you have many of those tasks that get stuck because they're they don't have enough priority it's good to Talk about them in the stand-up meeting for example and say okay that task has been stuck for the fourth time now Is it really a task that we should do or? Should we just get rid of that? so I Think that's another way of getting rid of tasks that should be done some time there are many tasks you can lose all together without any Consequences could you give an example dependencies, which we haven't anticipated before? Mm-hmm. If you see that there's a task that's dependent on another So it has to wait until the other task is finished or at least has moved has moved on You could for example do some kind of prioritization Using for example paper cuts on a whiteboard you could classify them into different priorities and First work on the ones with higher priority if you see that a task Contains a part that is has to be done first Maybe you should split that task into two and first and do one first and the other afterwards That's that's a thing you should discuss in your stand-up meeting if you realize okay, it's a bit more complicated than we thought Then discuss it with your team what what to do at that place? Yes. Yeah, that's a very good question if you have tasks that are Specialized for example, is it good to is it a good idea to move them off onto another board? Yes, it is for example if you have Persons that have a very special shop description for example an IT architect. Yeah, you don't have many of these usually Or a DBA your only DBA It could be efficient that these people have they their own boards so they can visualize Which stage is which pro project? I am participating in most of the times these people are in different projects at the same time and they need to visualize that to Keep control on their tasks. So their tasks will show up on different boards then there are Also a project that that big that it's a good idea to split them into hierarchical boards For example, the project manager could have a rough outline of the project on a board and Then Specialized boards with specialized teams for example if you have a huge IT organization You could be able to to split this also hierarchical Moses I come back to you mm-hmm I Think it's it's a good idea to Put all tasks your team is working on onto the board So if you are working with support tickets for example, they could have a place there, too to visualize For example, if there are urgent support tickets that require immediate action and So As I said, there is the possibility of having even urgent Tasks that break the work in progress limits because there's for example, if this there's a real Disaster on the customer side and you need to deal with that first Of course because it's the customer You could visualize that as well by putting it on the board making it a red card for example and prioritizing it over all other tasks Yeah, that's also a possibility to introduce lanes and onto your board which are divided vertically and You may have a fast lane for special projects Mm-hmm. I think that that depends on your organization for example as I Remember it with one-on-one. There are many many project managers doing many many projects and that's where the Tasks originate and instead of flooding the IT department with tasks that have to be dealt with in weeks Project managers can start with breaking up their projects into different tasks They keep for themselves for the time being and then start taking those tasks One by one and putting them into the backlog The question was how to deal with people that are not present that work remotely for example How do they get access to the Kanban board? There are of course different solutions and the David Anderson in his book or actually talks about having a physical board with cards and For example Having someone who is responsible to transfer The physical board which is easily seen walking by Into a digital system that he developed the Kanban method mostly at his time at Microsoft and they were using Microsoft I don't remember the product name some kind of team organization product of Microsoft to Visualize their work, but they all also had a physical Kanban board and someone was responsible to synchronize these I don't know how efficient that can be so We for example prefer to do it all digitally. We are using Trello comm which is a free tool you can get At Trello comm It's basically got these columns you can add as many columns as you want and By dragon and dropping you moving the cards over this board that you create and you can create as many boards as you want So for example coming back to your question You could do a board for each project where you Differentiate your tasks for example you have one column with your planning tasks one column with the development tasks and things like that and Then take tasks from these project boards onto a team board where they go from left to right And I think you I've got quite good experience doing that in a completely digital form Which makes it very easy to integrate people that are not present at your office space There are proven solutions to do that for example Google hangouts Skype chats things like that depends on your way of working I think That's not a thing that that Kanban tries to solve That's a more general topic you have to discuss with your team mm-hmm it is that's why we Got to the place it's got to a point where we decided to only use one trailer board for the work in progress tasks that we are working on divided into columns like like I presented here and they are boards we use for collection Mainly divided by topic or by project Where we don't we use them only as a place to store tasks not to organize them As soon as a task is going to be worked on It moves to the work in progress board where it moves from left to right in the usual manner No, we don't know most of the time it's It's a good idea to measure the the speed of the Tasks moving over your board you can for example Document when that tasks got on to the board and Maybe do a log of when it moved from column to column tools like Trello will do this automatically and then you could do a Measurement how quickly tasks move which will also point you to sources of problems when you see that The there's a column where tasks stay for a lot longer time than in other columns You could focus on the reasons why that is the case but I think if you structure your tasks small enough there's a Really a real speed gain where tasks won't stay on the board very much very very long and That's that's a thing that makes all people quite happy. It makes the the DevOps guys and girls happy because tasks vanish as soon as they come as quickly as they come and Of course, it makes the project managers happy as well You are free to to structure your columns for example, and you are of course free to as we talked already about To introduce additional boards for example a special board for the DBA So you could have a placeholder card in the delivery column and Signifying that this task is being delivered especially by the DBA at the moment for example is doing setting up table spaces or things like that and so Mm-hmm. You just divide the delivery column for example into different sections Yeah Let's start at the you could for example And you could have Two DBAs, but only one network admin of course They the the WIP of the network admin would be lower than that of the DBAs for example, but So so you're visualizing two dimensions Basically you see all the tasks that are in Delivery, but you also see where they are in delivery Who delivers them and that By the way makes it very easy to answer the question who is doing what and who is doing what next? Okay, if you have further questions, I'm here all day. Feel free to