 Welcome to those who just showed up, please. We're trying to get about three at least three people per table Except this table Table with sticky what we should be more specific with our requirements We didn't have enough acceptance criteria. We had a past feedback cycle. That's good. Yeah. Yeah, you were the customer, right? All right, shall we begin? Two minutes two minutes It's a game it's a game We can't promise you'll have fun, but you will learn something one more minute tick show half-price tickets You don't want to miss it come on in Oh almost One my watch says 159 What time does this session end? 330 okay one and a half hours, okay, so we'll begin so my name is Shawn Dunn I'm a agile coach with IHS and this is my colleague Chris Edwards. Hello and We're going to play today the value uncertainty game which was invented by my boss Todd little who's a VP of product development Today with us. We also go back go back. We also we have some helpers. We have some assistants Who will be able to answer questions throughout the game? So we've got Lashman our agile coach here at the Bangalore office with IHS and we also have Vincy thank you very much and Annie is also here to help us give a round of applause. Thank you for your help So as we always like to do when you're Todd or when your boss gives you something You change it massively and you don't tell them about it. So We have made some changes of our an empowered self-organizing team Well, eventually tell them about our changes But the purpose of this game is to explore how risk and uncertainty impacts our decisions in software development This is intended to be fun. It's a nice fun collaborative, but there is a learning objective at the end So there are a few rules and instructions and for everybody to enjoy the game We need to make sure that every hears the instructions and understands the game properly There will be times where we need to go and do activities and you work together as your team and then there will times we need to stop and Listen to some new instructions. So what we will do is Chris is demonstrating. He's raising his hand at the front I see Chris raising his hand. I will stop what I am doing I will look towards the front and I will raise my hand as well and stop talking Everybody else do this now, please Great I love how well that works and we will keep our hands up until everybody in the room is silent Looking at the front and ready to hear the next instructions. Perfect. Thank you very much So we will take one minute to introduce yourself to your new team Go! We hang out together way too much So the goal of the game what you will all be doing together today with your teams is to build a software release So you will have three iterations in this release and You need to you will build your software by Reprioritizing and prioritizing your backlog to optimize value delivery Before that fixed release date and you are competing against each other this team is competing against this team competing against this team You will win the game if you are the team that delivers the most points and there's two components to the points one is Surprise! Okay, yes, great. Okay, so you will be the team that wins if you deliver the most value by the number of Story points you deliver the value plus commitment points you get Merits the company thinks you're a good team if you meet your commitments You can get bonus points for meeting your commitments You will see in front of you an envelope With the team name on it I please open that now and we will introduce you to the artifacts of the game You will need these things to play the game number one. There will be four dice in the bag If you don't have four dice, please let us know it doesn't matter what color they are it does not matter what size they are You will also have 14 user stories printed on little cards like that Is everyone found their dice? As anyone let me rephrase that has anyone not found their dice Excellent, okay You will also find a scorecard and it should have your team's number written on the top and You should see on your table what we will call the value uncertainty matrix It is on your table with sticky notes With t-shirt sizes on the left and t-shirt sizes on the bottom If you don't like where your matrix is feel free to move stickies and that is the wonderful power of stickies There is also a scrum board You will also see to the right-hand side of your matrix We have a scrum board for iteration plan in progress and accepted You can also move that if you wish All right. Is anyone is anyone not have one of these things? Good. Okay. I think we can move forward Contrary to what the slide says there are three roles in the game The first is the scorekeeper So will the the scorekeeper is the takes the scorecard and make sure that the scorecard is up to date The sticky minder will move the stories so they are the ones to make sure the scrum board is up to date and We also have a dice roller and they're the ones who rolls the dice You can move and shift roles around your cross-functional team as often as you like But someone needs to be wearing one of these roll hats at any moment in time All right, so we're going to show you what the stories look like. This is an example story You will see at the top there is a story number. It's got an ID and Each story has a test on it. What does the test mean? Well, we work on user stories until the user story can be accepted But we're not exactly sure how long a user story is going to take we might have an idea We might have an estimate, but we don't know in reality how long it's going to take the user story is done You keep rolling dice. You keep rolling dice. The user story will be complete When the test has been met for that user story So in this case for each day of the iteration you roll three dice and when two or more dice are the same The iterator the story is complete. So we will roll three dice on day one. We roll three dice Are we able to accept the user story? No, we are not because two or more dice are not the same. It does not meet the test Okay, so we cannot accept the user story on day one So we move on to day two we roll again and we meet the test. We've now completed the user story We can accept it and now we put it out into the market Do we get any value from it? We don't know So we don't know what kind of value we get from this user story until we get it out there And we learn from our customers and so in this case the value we receive is going to be the sum of all dice So three plus one plus seven the the value the sum on the last on that last roll question so far We're going to do a couple more examples. Okay Okay, go ahead Good. All right. Let's We'll have lots of examples. It would be good so Relative sizing we're all familiar with relative sizing in in agile and comparing user stories to figure out how big they are So we're going to do relative sizing with these So on the left We've got this this user story number one with the value of the sum of all dice Right on the right. We've got user story where the value is the product of all dice you multiply them together now Tell me which probably has a higher value For because product not always but more often will have a higher number than if you just add them together Okay, so yes for has the potential higher potential value great now. What story has the higher potential cost? I heard for People agree. Okay, so That's a little bit less clear, but you know our gut instinct or our in our Educated guest tells us that's probably harder to get to get three Three consecutive integers. Okay, great One clarification. This is something we weren't aware of is is if you will see if you see cards that say no ones and no twos that means that none of the dice are None of the three dice are ones and none of the three dice are twos There's other ones like no fives and no sixes are no ones and no twos and no threes It's the same idea. It doesn't mean number one. It means that there are none We've tested this a few times and this has been a common confusion. So if you have any questions, we've got volunteers to help clarify All right So you roll three dice the user story is complete when none of those three dice are a one and None of those three dice are a two If it was five five six, you'd be good. It was five five one It would fail because one of the dice is a one Alright, we're going to do some release planning. Okay, so do me a favor and find user story number one Which has the big black border around it? Okay, so this is actually the one we just looked at on one of the previous slides So I'll move forward take that and put it smack dab in the middle of your value uncertainty matrix Right in the middle and we're doing relative sizing, right? So we just need a reference We just need something to compare to we don't know how big this is we don't know how small it is doesn't have to be exact doesn't have to be exact and Now look, we just compared a few minutes ago We compared user story number one to user story number four right find user story number four You you'll see on the matrix on the bottom It's cost on the y-axis. It's value now when we compare this a few minutes ago Which user story had the higher potential value? Four so you would place it above user story number one which one had the higher potential cost There's also four right so you would place that somewhere to the right right higher value higher cost Okay, how far that's up to you. That's that's you that's your team's decision You can figure that out because this is just estimating so what we are going to do now is you with your team are going to Relatively estimate all of the user stories Okay, and here's the thing you can you can try to calculate all the probabilities Unless your ramanusion you probably won't have time to do it. This is like estimating and software. You got limited time Okay, so work together with your team and estimate all the other user stories right now And you've got ten minutes ten minutes to do that Okay, you like right on right there your estimates are gonna be perfect That's Okay, that's how that works Okay, done done all right down One Thank awesome. Thank you All right, we're going to do a little quick mini retrospective here So we just went through a relative sizing as exercise and I've got a question How is this like estimating and software what similarities do you see? complexity Uncertainty so you weren't it wasn't it wasn't exactly clear Right, so you weren't it wasn't you weren't sure which one might be bigger. Okay, that's what anyone else This was easier. Why how is it easier? Okay, much more complexity in real life. Oh We might question you we're gonna hold you accountable to this You better meet your commitments The uncertainties higher here or in software in software. Okay, okay, okay? Right, yeah, there's often dependencies between our stories in real life. Yep. No very good anything. We can learn from this so far any takeaways You're just all waiting to see what comes next Let's get to the dice Sorry, what did you say? Was it we could your team your people you just met work better than the team you have back at Is this what I'm hearing Did Did working as a team you think you you got better you got better results like better estimates than if you had just done it alone Yeah, yeah, okay good any of her team members here right now. No, okay That's okay the videos on the internet This is live stream to YouTube right now Okay, so now we're going to start the game we'll start actually playing the game So the goal will be to prioritize your backlog and Reprioritize your backlog to maximize your team score by the end of the release and we will do a quick example of this So first thing we need to do is iteration planning. We've got three iterations. We need to at the start of each We need to plan what work we're going to do in that iteration So again just a review of the rules the release is three iterations There are ten days per iteration and you get one roll of the dice per day You can only work on one story at a time, okay? And you earn five extra commitment points for every committed story you deliver Okay, but you forfeit all your commitment points for that iteration if Any of the committed stories aren't met and I know this is confusing. We'll have lots of examples Yeah, lots of examples. Okay, you must commit to at least three stories per iteration You can commit to more But you must commit to at least three because that's been your velocity your teams of velocity for the past, you know, like You know little while so of course you have to keep your velocity up, right? So you have to complete at least three stories per iteration next Yeah, you may commit more you may choose to commit four more stories You may choose to commit five more stories or ten more stories Up to you. However, you lose all your commitment points if any of those Are missed so you have the opportunity to earn more commitment points But it's also if you miss any of those you get nothing And then finally we calculate the score is delivered value plus your commitment points And we will get do an example very briefly here. So we're gonna actually we're gonna first We're gonna plan the first iteration. So now that I've explained this Doesn't show up. Well, but what you will do is you will decide what stories you are going to plan into the first iteration So go through your stories choose at least three possibly more and move them into your iteration plan column for the very first Imagine there are three stories there. Yeah, so I think we need two minutes for this Two minutes. Okay, so two minutes, please do this now go Planning is over Thank you So you will see your this is just the top row of your score sheet Thank you. So top row top row of your score sheet now However, many stories you just committed to whether it's three or more You will write that number in that first column number of stories committed to the iteration. Okay Now what we'll do is we'll start working on the story and you cannot see this on the screen But okay, we will move that story into the in progress column Okay, and then this is what the story you're starting to work on you've decided to work on this story first and so here's the test and the value so we'll start rolling the dice and Did not meet the test. So we write a X and day one to indicate we did not Deliver any value that day Now we keep on rolling we roll again on day two did not deliver So we write an X again and a three. Oh We met the test right so yeah two more more dice are the same So now we write in the value which is three plus one plus three which is seven. Okay We move that story into the accepted column and now we begin work on the next story Okay There we go. All right, so here's the next story We roll Did not meet we roll Did not meet And we finally on the third day of that user story. We delivered 36 yeah Yeah, 36 points. Okay. Okay, and then once again, it will move into you can't see it The the card moves into the accepted column after it's been completed Don't move on to the next story. Oh And look at this we actually completed our three stories for the iteration, but there's two days left So what would we do in real life? We sit on our hands You're right you're right we do we do take this story from the backlog so you can choose a story from the backlog and pull it into the iteration and Work on that and oh look on the very last day We got that user story done if you don't get a user story done. You're still working on it Well, what do you do in real life? It just goes back into iteration planning for the next time now We're going to add things up. So Okay, what is the total accepted value can anyone Any guesses? 53 Okay, we count up the total yet 15 right we do we only get commitment points for the stories we committed to so three times five is 15 You don't commit to them then you don't get any extra points All right, so in the case we'll show you another quick a couple quick examples So in the next iteration they decided to be bold and risky and commit to four user stories However, they only completed three So they delivered 16 so they've been delivered 16 story points or value But they only met three they did not meet their commitments. So they get no commitment points You committed to four You you completed three you get the value for those three you get the value for those three So ten plus three plus three is sixteen meeting your commitment is a binary thing. You either meet your commitment or you don't They're a bad team Yes, that's true. You get those points you still get the value points You still get those value points, but you don't get the bonus points for scrum master is is looking at you Disprovingly and you don't get the commitment points. You see saying your iteration fails So in the last case they committed to four stories again They met those four stories and so now they get 20 points four times five is 20 So any questions because we're about to begin Exactly. Yeah, so if you if you committed three So that's fine. So for this if you if you if we didn't get the six points You're saying in this case we we committed to three We didn't finish this fourth one that we started working on you still met your commitment One more question When we are working on a story for example if you're spending three or four days on a particular story still if you don't get a value You can always change the story and you can pick a new story some special things we can do Before we get to that at the very end you can see your total score This is a score will use to determine who wins you add up the accepted value and you add up your commitment points So 98 plus 35 is this team's total score is a hundred and thirty three So special actions like Lashman said it is okay to abandon a story You can come back to it you can come back to it later You can come back to it later that iteration or an end of future duration or not at all So you can abandon a story in progress and you cannot redo a story once it has been completed Okay so Begin now start the release and we have how many minutes 15 minutes 20 20 minutes 20 minutes You missed your you missed your commitment guys Each team what strategy they use so we'll start. What are you guys your team eight? So team eight scored two hundred and forty nine total points Coming in second place anyone want to talk about what what strategy do you did you use to choose which stories you worked on? Thank you all right you guys are team six 141 point oh 277 oh I'm reading the number on the left I apologize. Okay. You guys are the model team you scored the most points Who would like to talk about how how you delivered so much? We have started with the very low things and We mix it with Productive complex things and you showed you with the low variety at low complex which which actually bring We are end up with even taking the risk few risk like how how actually we can Take a combination of complex value and Actually, we end up with taking all those three we are actually end up with taking only three stories So we are trying our luck also and that You took you took some gambles So the more risk you took the more variability you had in how much you yep, okay and team Four which is that's the one that's 141. Oh Okay, who would like to account for for your guys performance? I'm not the scum master, but I can take the blame We tried the simple ones first As we met the commitment we picked up the the complex ones They didn't click We took a less less risk. We have not picked up the higher Oh, you did you guys met all your commitments. Oh, that's important because your scrum master is happy See actually we committed what we could at that time and then after that We would still be left with the two or three days at that point of time We tried the complex ones in each of the iterations and in each of the iterations did not work So basically the difference is that even if one of them are clicked you've done a big number So you guys try you started something hard, but then it wasn't working So you switch to the easy stuff to make sure you got something done Okay, but you guys just stuck with it. Let's just keep grinding into this. We'll figure it out Okay, how about you guys? You guys are 168 so I guess the first round we stuck to the rules and we took the most value We actually did very well for ourselves that we took One of the complex one over our commitment, right? And I think we were you know for the second round. We were completely overconfident. We forgot the rules Right, and we just picked up something. We screwed up miserably And third we came back to the rules And I think we were successful So I think there was a mix of luck as well the first time And there was somebody from Vegas here on the floor We imported somebody All right. Thanks everyone. Thanks for sharing On you okay, so Okay, so we went through how to everyone picked the strategies It seemed like there was a good variety of strategies, you know some different risk takers Maybe let's really you know focus on getting our commitments I've got the clicker Okay, so I think I think someone answered this question But did anyone abandon a story in the middle of an iteration when it wasn't going? Yeah, you guys did too. How about you guys? Yeah. Yeah, okay Anyone changed their strategy you started with one strategy and then told this isn't working and let's try something else All right, so you you were inspecting and adapting as you went. That's good Um Okay, so what did you guys learn? What what are some takeaways from this experience? Say that again Yeah, yeah, you were allowed to abandon a story The rules on that are ambiguous um Okay, anyone else what what what what takeaways? Right, you know, you have to take risk, you know, probably Sometime luck, right, but you know, we were playing it safe. I think uh playing it safe doesn't kind of you know help To be on the top Definitely, you know, you may meet commitment, but you know only to that extent it helps Thank you Another one another question or comment behind you. I think in this team. We were more of commitment driven than value driven So that's why you know, we got four four points for all the pre-iteration But in reality, it's uh, it's not as Crazy as this because you know, there's a lot of probability and there's a lot of that So, uh, I think the main thing is uh To inspect and adapt every sprint and learn from it The interesting thing is when we did the retro about estimating people were saying estimating software was even more complex than Anyone else let's get one over here and then one over there. I think um in the real world We kind of missed that point that it's okay to swap. Um, so I think once we started doing that we did really see some uh value in it Because most of the time you pick something you are committed to it and you try to achieve the same thing It's not working. You're you don't kind of look if I can do something else So, uh, one thing I found common between the game and the reality is Teams are more driven towards, you know Satisfying the management and that the commitment points is nothing, but you know keep the management happy, right? Right, so it was a safe game like looking at okay points is values are fine But you know at least I should get those in an easy commitment points So that is one thing which I saw common Yeah, we we tried to actually influence your decisions a little bit by walking around being like oh you didn't get your commitments done, you know Okay, one more question back there or comment back there So we kind of assume that for each iteration the luck will be the same So we'll have a constant good velocity But you know sometimes it just turns due to some external circumstances in this case guys Uh, so we cannot really predict how it will go Investment that we get is very low. That's one of the stories Are you okay? Are you okay? 180 points, uh, there was a high risk Oh, but they're all ones. Yeah. Yeah, I mean roll the dice and we get one. Okay, so so what is there Is there an analogy to real life there? I mean that's very uncertain To the market and the client doesn't even use it. Absolutely. Yeah, there's there is definitely uncertainty In what we deliver to the market. We believe there's value there, but in reality it might not be the case Yeah, and I think I agree to uh, I'm not sure about the name of the gentleman Uh, yeah at that point that you know, you get in reality You don't think of swapping whatever it's not if it's not working take a break and start from the scratch, right? I don't think we do that we stick to one thing that takes a lot of time and uh High risk is high rewards all the time. I think that that a bit work You said that um, I I had a couple of days left. So you guys decided to do the complex stuff In in real life would be I mean I'm Would we do that? I mean if we have two days left in the sprint, will we go and take a very difficult use of story and Yeah, I'm yeah, okay So so one of the things I'm hearing is that commitments commitments were influencing your guys decision making In this case when you'd met your commitment Your your behavior changed The way you made your decisions started changing. So the way I make my decisions The way I decide on what I work when I need to make a commitment is different than when I don't So Okay, I think I think we've addressed this sufficiently Okay, so In reality there could be a lot large spread here What we really want to do is focus, you know in an ideal world Let's grab the the high value low cost things work on those first and Do our prioritization this way A lot of the times though, that's not really the case If something was high value and easy to do somebody else probably would have done it already So the the I think the comment was there's the high value things also has high risk associated with it so statistics time This game is mathematically solvable if you have enough time and energy Which we did we put a computer against this and we've run this game through thousands of iterations so This is the real data. So if we created this same value uncertainty matrix using Roles of the dice. This is this is solved So there were many Low value low cost items and there were three. I think you probably all saw this three high value high risk items on the board Okay, so this graph shows expected value expected value is value divided by cost So this means if I were to take this story and I would play it for a thousand days What is the average number of story points? I would deliver per day once again the The higher the cost the higher the expected value so We ran a Monte Carlo simulation and what we did was we created two separate strategies We had two ai's that ran two separate strategies the first one Prioritized low cost first. We want to meet our commitments. We want to make sure that we get everything like prioritize Getting stories done over delivering value and the second strategy was completely ignores your commitments and goes after the value first Yeah, so strategy one goes after It starts on the left and moves to the right And strategy two goes after this guy first then this guy then this guy Then this guy You have a microphone Strategy two would continue rolling on that guy for the entire Release. Yeah, so so Mathematically speaking it makes no difference if I try at day one or day two or day three The probability is the same so I should still go after the thing that's going to have the highest expected value So we actually ran this um ran a Monte Carlo simulation with these two strategies and we did 100 runs each And this is the result. Okay, so this is a histogram This is number of stories completed and and on the y-axis here. This is the number of runs through out of 100 Uh that delivered that so easiest first delivers way more stories On average that makes sense right if you're going after easy stuff, you're going to deliver more stories Okay, the second one shows accepted points If you so on average if you deliver if you go after the highest value stuff You're going to deliver higher value in the long run, but there is a risk associated with this There were eight runs with this strategy that delivered zero value over the course of through iteration So there's a very real risk associated with this But this strategy works in the long run And here's just some some averages Easiest first delivers 100 on average highest first delivered 200 on average So you're on on average. You're delivering twice as much value using this strategy, but your commitment points You're only gaining 16 more so Those commitment points are like a little bit of a bait, but they don't really give you anything And does the market care about your commitment points? It's just an artificial thing we've generated for the game So I think I heard someone mention this example So what are examples of things we do in the workplace that encourages decisions to be made certain way? And one example I can think of is you know, we need you know, we need to make sure that we Get things done within the iteration, right? So let's say there's when we're doing iteration planning. How often does this happen? They think someone mentioned it where oh, it looks like we've got maybe like a little bit of capacity in the left in the rest of the iteration but Having stories that aren't finished by the end. That's bad, right? So what do we do instead? We try and fill it with sand Right, we go down to the bottom of the user the backlog We find these little low priority user stories that we're pretty confident about and so that we can fit them in But why would you do that? Why would you not just work on the next highest priority user story regardless of whether it rolls over to the next iteration? Just like commitment points things we made up iterations are things we made up And so how do we let them influence our decisions when you could have potentially a one or two day head start on the next highest priority thing So Commitment points are this this artificial thing we've created but what are some things that we do That are influencing the same behaviors. What what's kind of the real life equivalent of these commitment points? How do we incentivize people's behavior in this way? iteration like an iteration burn down chart Yep, exactly. Yeah, so the one of the things one of the subtle things we do and I did this walking around is I just ask questions When are you going to be done? Are you going to meet your commitment? The questions that we ask people the the charts we produce the metrics we gather Communicate what we value as an organization and what we want people to be thinking about when they make decisions Okay, so in an ideal world we'd have all these high value low risk things But that's really not the case So what are some things we can do if we want to go after these high value things to Lower their risk so that it makes it a little bit easier more palatable to work on any ideas Okay, you could get some spikes you could gather you could buy some information Okay, we could split stories There's ways there's ways of just reducing the size of things. Let's take a big thing and make it a small thing And then get it out. Let's get some feedback from our customers early Let's not let's not wait until we're totally done before we find out that this is really a one and I shouldn't be working on it in the first place so We're trying to show in this game is and I think you guys have come to this conclusion through the retro I heard, you know, there's uncertainty in our work There's variability things take longer. Sometimes they we don't get anything done in iteration Sometimes we get a lot done And and there's this idea that the commitments and and other things that we do questions We ask metrics we gather influence people's decisions Potentially in an on optimal way in a way that will actually drive our decisions away from going after high value things And I forget who the quote was from but someone said, uh, you know without risk there cannot be value We are developing new things that have never been done before every time we go down to code something We are creating something that hasn't been built before and there's a lot and otherwise you wouldn't be doing it Someone else would have already done it and why would we be doing it again? So there is just an inherent part of risk in our in our software development, but that's where we get our value from So, um, it's how do we go back to organizations and and remember as part of our day to day work that Yeah, actually we these things aren't certain. We can do things to help improve predictability But ultimately there's going to be this element of risk, but that's actually where we derive the value from So thanks for joining us everyone. So, um, top name on here is Todd little. That's, uh, one of our vp's at IHS He's actually the one that invented this game that we've now heavily modded. So Which we should tell them about so we'll tell them about it when we get back to Canada Um, so you can you can find us on twitter or by email. This is actually Todd's book stand back and deliver Yeah, he's my boss. So if you buy his book, then I get a better performance evaluation Is that the metric you're being evaluated on is number of book resales? Is that why you're at a conference? I should propose that idea too And then okay the and this the simulator we've actually put it on github So if you guys want to download it and look at the source code run your own simulation Uh, if you want to code review it and find the bugs that are in there that completely invalidate the premise that would also be good Um, we have 15 minutes and there is another scorecard in your folders So if you have something else to do, please feel free to leave if you had so much fun and you want to try a new strategy with your team because you had a retrospective and you want to continuously improve and learn Feel free to take the scorecard out and use the last 15 minutes to run another release And we will happily take the scores because we're going to collect the data So thank you everyone. It was a pleasure Lashman's reminding me to uh, we should practice our ending Oh, wait, wait, wait, wait, thank you to our volunteers first. I can't believe I forgot about that Yeah, thank you Lashman. Thank you Annie and thank you, um, sorry Sukria, Sukria, Sukria, Sukria, Sukria, I'm learning and it's not late enough uh Shubaratri, but that's the other word I know. Sorry it's not even yet Shubaratri Shubaratri, I'll get this down. Shubaratri. We're here for two more weeks. We'll figure this out I'm sorry. How do I say I'm sorry