 Thank you very much. Welcome. Thank you for coming back after lunch. I hope everybody is not too exhausted and full. This is going to be quite a hands-on, active workshop. Hopefully you're not going to be sitting falling asleep listening to me too much. I work for tech systems actually. Who came to my session yesterday? Most of you, not everybody. I'll do a quick introduction to myself. I work for tech systems global services and I lead our agile transformation services practice. Based in the UK, so I live down in Brighton, which is south of London. Once you've kind of come out of London, head south till you hit the sea, and that's kind of roughly where Brighton is. Although I think comparing the size of the UK to the size of India, exactly where we are within UK really doesn't make much difference. When the rush was doing the exact opening exercise yesterday I'm about how far people have travelled. I was desperate on Google trying to figure out how many kilometres it was, and it came out just under 8,000 kilometres. So yeah, we have a team in the UK. We kind of work with organisations on modernising businesses, private services and modernised businesses, and a lot of that is modernising their ways of working, hence we kind of go in and coach and train and consult on agile practices, lean and agile. We're a US-based organisation, so I get to work with my coaches, team of coaches in the US, and we also have an office in India, so one of my colleagues is around here somewhere. I don't think he's made it into the room yet. So yeah, I should start with just kind of explaining the title. There's a kind of a cultural reference here that probably isn't that obvious to most people. Does anybody recognise who this is or this picture? Sorry? Sorry, I can't hear. No. This is Phil Collins. Anybody know who Phil Collins is? Okay, a couple of people. Phil Collins was the singer of Progrock band Genesis, and in the late 80s he produced a solo album. That solo album was called No Jacket Required, and this is the album cover. So what we've done here is we've just kind of put No Lab Jacket Required. So yeah, that's kind of the explanation of what the title is. What we're going to talk about is experiments and learning. Before we do that, kind of another. Is wordle a thing over here? So you have to guess the five letter word. So wordle is a nice, simple example of the sort of things we're doing, because basically what you're trying to do here is learn what the word is, and each attempt is a bit like an experiment. So in this example, I always start with the word adieu, because it's got four of the five vowels in. So I kind of like to get the vowels early. So in this example, I had the I, then I went with groin, and now I knew the O was in the middle. I didn't know where the I was. And then ivory. So I know kind of know the I isn't in those three. There's two other possibilities where the I is, the O, but I still haven't got any letters. So each guess is an experiment, and you're learning from that experiment, and using that information to hopefully get the word. I'll come back to the end of it. I'm not going to tell you what the answer is now. I'll come back and I'll show you kind of the final solution at the end of the workshop. So experiments. Who's talking about experiments? Well, actually two of the keynotes this morning explicitly talked about experiments. Dave Farley and Fred George both talked about it. Canban method, one of the Canban method practices, improve collaboratively and evolve experimentally. So Canban talks about running experiments. Modern Agile Joshua Kerieski's approach talks about experiment and learn rapidly. Lean startup, maybe not experiments, but certainly talks about having hypothesis, testing and your hypothesis to learn and pivoting based on what you learn. Skeld Agile framework, each feature should include a benefit hypothesis. So hypothesis, if you've got a hypothesis, your experiment is there to test that hypothesis. Cannevin, I've heard about Cannevin in a few sessions. They're in complex environments. You can't follow recipes or conduct detailed analysis to understand the situation. Rather, you must experiment or probe. I can maybe talk about the subtle difference between an experiment and a probe. Essentially, we're trying things out to learn. DevOps, culture that fosters continual experimentation, taking risks and learning from failure. And then lastly, last example here, Deming and Schuwer who defined or designed originally the Schuwer Cycle Deming popularised it, so it's often managed the Deming Cycle PDSA, plan do study act, or sometimes PDCA, plan do check, adjust. Swap some of those words around. So lots of talk about experiment going way back to the early days of Lean, through to modern agile, canvan and everything in between. But I think sometimes we use the word experiment just another buzzword that we pick up and we talk about experiments, but are we really running experiments? So this is a quote by Carl Popper talking about Albert Einstein. What impressed me most was Einstein's own clear statement that he would regard his theory as untenable if it should fail in certain tests. It was an attitude utterly different from the dogmatism of Marx and Freud. Einstein was looking for crucial experiments whose agreement with his prediction would by no means establish his theory while a disagreement, as he was first to stress, would show his theory to be untenable. This is the true scientific method. But what this is saying is just kind of a big long quote, Einstein wasn't just trying to prove his theories. He was trying to find ways to disprove his theories because he recognised that just proving his theories and just kind of getting successful experiments didn't really mean anything. But if you try and disprove your hypothesis and you can't disprove it, then that's just kind of giving you reinforcing that your hypothesis might be correct. But if all you try and do is prove your hypothesis, the risk is you fall into confirmation bias. So you're just doing things which look and help you feel confident but you're not really getting anything. So what we're going to do is run through a couple of exercises, games, just to introduce this idea of what we really mean by experiments and we'll talk about how you might apply that in your own day-to-day and some of the things we can do. So two exercises. We're going to do this in table groups and we probably want to have at least four or five people with table groups. I'll kind of give you a bit of warning that you might want to kind of move tables so that you've got enough people on a table. I'll run through the instructions of the first exercise first and then I'll give you a little bit of time to move around. So this first exercise, this first game, I came up with actually both of these. I came up with a friend called Matt Phillip who's a US coach. Back probably four or five years ago I did a talk at Lean Agile Scotland up in Edinburgh entitled Failure is not an option when I was talking about the need for failure and what I meant by failure is not an option was failure is not optional. So another bit of a play on words there. Actually we have to fail if we're going to learn. And Matt came and talked to me after that talk and he was like, we've got any... What games are exercises? Can we run? Do you run with teams to kind of help them experience this? And I didn't really have a good answer so we brainstormed some ideas and came up with this as one of the exercises. So the background to it is actually an original game called by Robert Abbott who created a game called Eleusas. It's a card game and he designed it to try and teach scientific thinking to his students. It's actually quite a complicated game. There's lots of really complicated rules to it. Somebody else had already kind of thought this is a bit complicated and come up with a game called Eleusas Express. And so this is the kind of real way you can go and find out about Eleusas Express. I still, we kind of still thought actually that's still a little bit complicated. How can we make that even simpler? And by simpler, you know, quicker to run. So we call it Eleusas Expeditious. What's quicker than Express? Well, you expedite something. So that's where it's come from. Just the name Eleusas, I had to go and look this up. So I expect everybody's thinking the same thing. What does that mean? Where does that come from? It's named after a place in Greek where there was something called the Eleusas Mysteries and there was a rite of passage to discover the Eleusas Mysteries. So there were these secrets. I think it's called that because what you're going to be doing is trying to discover a secret and you're going to be running some tests and coming up some hypothesis to discover a secret rule that's somebody who only one of you in your team will know. So the objective, I'm going to do this over a number of rounds. So you're going to get a number of goals to do this and experience that number of times. One of you in your team is going to be what's known as the Oracle. The Oracle is the all-seeing, all-knowing person that knows a rule. So the inaugural knows the secret rule and they have a special role. So they're not going to be playing the cards. They're just going to be giving feedback and telling the rest of the team how they're doing in terms of discovering the rule. We discover the rule by playing cards along a layout. So as the round progresses, you pick a card and you play it according to whether you think this new card conforms to a rule. So there's two types of layout. One is the main line. So this is the first card that gets played. And then all the cards along here conform to the rule. And then the sideline is where we discover a card that doesn't conform to the rule. So we play it below the card at which kind of the rule is broken. So I don't know. Looking at this, can anybody guess what the rule is? So the rule here is that every card must be odd and then even and then odd. So an odd card must be followed by an even number card and an even number card must be followed by an odd card. So we can see here is we have got a two. That's even. The next card that gets played is a nine. That's odd. The next card that was played, or that was picked, was a jack. Now jack is 11. The rule says it should be even. So we play it down here, put it in the mind then, so we're still looking for an even card. The next card is three, not even, so we play it down here. The next card is four. That does conform to the rule, so we put it back on the main line. After an even is to be odd and then we get another odd one, another odd one, another odd one until we get another even one. Even, even, odd. Does that make sense? So the oracle is the person that knows odd card, even card, odd card, even card. The rest of the team, you're going to be taking it in turns to pick a card and have a look at the layout. Have a guess. What's my hypothesis? What do I think the rule is? Based on what I think the rule is, where do I think the card should go? Does this card conform to the rule or does it break the rule? Are you going to put it where you think it's going to go? On the oracle who knows the rule will tell you whether you're right or wrong. And if you're right, you get to guess the rule. You kind of go, if you're confident enough. Sometimes you have no idea and it's just 50-50. It's a guess. But if you think you know what the rule is, you can say, I think I know what the rule is. I think the rule is odd than even and the oracle could go, well done, you've guessed the rule, you win. And that's the end of the round. You say, I think it's, let's say you think, I think it's red card, it's not showing up red here because it's black and white, but red, then black, then red, then black. Because you've only got here, the oracle kind of go, no, that's not the rule. And you keep on going. So I've probably just explained a whole load of rules more than I should have explained there compared to the side. So you've got the horizontal timeline where you've kind of put the cards that don't follow the rule and where in the sideline is kind of tells you where it broke the rule. So the oracle knows the rule. So I'm going to tell you what the rule is. So you don't have to make up your own rule. So I've got a stack of index cards with a number of rules on it. So first thing you'll do is decide who's going to be the oracle in your table. That oracle come to me, what the rule is. Handy if you bring a phone with you and take a picture of the rule because some of them, they get a bit complicated and they're difficult to remember. So the easiest way usually is just come up, I'll put it on the floor somewhere, you can take a picture with it and then you can just keep referring back to it in order to remember. So you don't have to kind of keep it up here because I've only got one copy of them. So then you come up, get the rule, take it back to your team, shuffle the cards. So you kind of start off with a fresh pack of cards. So actually can we keep one of these packs of cards here and maybe just give each table. So you're going to need four or five people so I don't know. So yeah I think that's good. Woody if you want to join in you might want to go in. Okay that's fine. If you've got any jokers we don't need the jokers. Before you shuffle them. So give the deck a good shuffle to start off with just so they're kind of reasonably mixed up and then you just put the deck of cards face down on the table. So that's going to be the start and you probably just turn over the first card. So we're going to start with this pack of cards, we're going to start with the king of diamonds. So that's the start of your mainline and then as you go round everybody then turns over a new card. Has a look at the card has a look at what's been laid out has a think about what they think the rule is. Decide whether they think this is going to conform to the rule or whether they think it's going to break the rule and position it and then the oracle will kind of go yep you've got that right or no you've got it wrong. If you've got it wrong then move it to where it should be. So the layout of the card should always give you a visual indication of what the rule is. So that's the oracle. Any questions about the oracle roll? Does that make sense? Yeah. Sorry? So if the second card is wrong then it would go under here. And if you put it here and the oracle says no that doesn't conform to the rule then you move it. Does that make sense? Yeah. Everybody else then are the players on the team? You're going to take it in turns just to turn the top card over the deck, lay it on the table, decide whether you think it conforms to the rule or not and place it and kind of wait for the oracle to confirm or reject your positioning. If they reject it then move it to the right place. So if it conforms you put it to the right if not it goes below on the below the last correct card. If there's no sideline you just add it to the sideline if there's no sideline you start a new sideline. Any question on this got a bit of instructions? We're going to do multiple rounds so like the first round there's usually a bit of a learning round. You learn from doing this the first time. When you make a correct assertion so if you put it down and you put it in the right place and the oracle says yep that's right you make a guess and then if you guess right and you've guessed the rule you're done we'll just do all one round together and then after that once you've done with one round just come up and find me I'll give you the next rule so come up with a camera take a picture of the next rule take it back swap the oracle each round so everybody we might not all get chance to be the oracle but hopefully most of you get chance to the oracle and then the reason we just take it in terms of picking cards is just so everybody gets a chance to to have a go at having a guess and get a feel for that so we don't just want it to be a spectator sport where one person is doing all the guesses so I just put in this there sometimes I go to places and do this and people are not entirely sure about what I mean so spades, hearts, diamonds, clubs is that all familiar to people so the rules are going to talk about the colour of the card wedge cards, black cards the suits, spades, hearts, diamonds, clubs and then when we're talking about numerical rules aces 1 jack is 11, queen is 12 king is 13 and j q k so everybody kind of we don't have a problem with that sometimes it gets a little bit lost in translation so I'm going to leave that there are we doing it for time perfect let's do one round the first rule is hopefully fairly simple because I don't curse myself there so who's going to be your first oracle that first oracle just come up and get your first rule so one oracle per table one person on each table will come up and be that first oracle so I'm just going to put that down there if you've got any questions about the rule let me know and I'll try and remember to turn the microphone off before I give the answers if I just put it down there I'll leave this one up so every table has one oracle and that oracle has the rule and understands the rule so we're going to spend maybe 20 odd minutes on this we'll see how it goes we'll see how quickly you get through the rules and give you a good chance to experience this so start so oracle place the cards down turn over the face card and then start going and I'm just going to wander around and I'll just see how things are going and answer any questions okay I think everybody's got that nice easy one to start off with they're not all going to be that easy they're going to get progressively harder so we're just going to keep going now so choose another person choose another oracle next rule, bring a camera take a picture of it take it back and just start playing and then as you guess the rules just keep going because you're not going to be doing it in sync I just wanted to make sure everybody's had one go same rule for every table in the same order yes yeah yeah and let me know if you have any questions about what the rule means that's the next exercise so just carry on playing I'll wander around I'm born and brought up in Grand Kail oh steel my father was working I'm born and brought up my two words were I did my graduation graduation also I completed there I didn't do engineering I did master in computer application in T best it's a maths so so we're going to give this about five more minutes and then we'll do a debrief and move on to another exercise so I'll just give it another couple of minutes just kind of try and finish the one you're on yeah yeah I mean I'm sure some of these packs are cards I've got missing cards okay so just kind of finish the current conversation oracles so if they've not guessed the current order around just oracle let them know what it was just so you know people aren't staying awake at night trying to guess what it was yes yeah we're finished with the cards now okay so let's do a kind of a bit of a debrief on that hopefully it kind of sounds like people are having fun lots of conversations but what was the point why did we run that exercise so I'm going to give you just five minutes on your groups just have a conversation on your tables thinking back to what you just did how did it go about discovering the rules what were the thinking process you were using what techniques did you use and was there anything that was really useful things that you did that found really helpful or anything that happened that was helpful and similarly anything that you did that didn't happen that didn't help you that was unhelpful so just kind of think about those three questions how did you go about discovering the rules what was useful or helpful and what wasn't useful or one helpful because these five minutes had that conversation and then we'll just kind of see what people talked about okay so we've got some microphones there's one other table that's kind of just every table but does somebody on a table want to talk about things you talked about what answers you came up with anybody want to go first great thank you Dan one was like through the learning process I think with every action there was a moment of learning whether we are on the right track or not and again like it was a team accomplishing the course rather than someone telling this is how it needs to be done oracle playing an important role in guiding the team but letting the team discover on its own learning I would say and reinforcement when things were falling in place when we are in the right direction I think that's the first one and probably the second one was also that it was useful oracle coordinating and the team working together and sharing the thoughts and discussions and whatever what was not useful I think is we've seen the same thoughts coming up coming up with the incorrect conclusions at every step but eventually figuring out that was not a few tables you just get a long sideline where you're just constantly getting cars to their match what's happening there is you're not getting any new information maybe occasionally but it's only then when you suddenly get in the mainline that's really powerful then why is this special, why is this different to all of those and that generates lots of new information and lots of new thinking another table want to anything different not answering each question but overall two things which emerged out of this table one was the wrong answers the wrong guesses were the right guesses that was one and the second one what we stopped in between just guessing what could be the pattern spending a lot of time thinking about what the pattern is rather than playing quickly and trying to figure out the so maybe we should just play and it's faster two quick points there one is yes not just the having the wrong guessing wrong and those failures it's actually having it visible so imagine if you played this and when you had a wrong card the oracle kind of went oh that's wrong I'm just going to hide that under the carpet out of the way because we don't like failure so we're going to pretend it didn't happen then you only had the mainline you can make it so much harder it's being able to see not just the right but also the wrong another thing yet sometimes you just kind of get in really slowly and sometimes those early rounds those early guesses you're just getting information and you're not really having hypothesis so I talked at the beginning I kind of mentioned that distinction between an experiment and a probe I would say those early rounds when you're just playing cards you're probing you don't really have a hypothesis that you're testing you're just kind of a talk of probing does it give you a slap does it laugh so yeah you're just kind of getting you're just getting information and then at some point you start forming a hypothesis and now you can be a little bit more intentional so that's kind of one way of thinking about what's the difference between a probe and an experiment both are valid but really we will probably want to most cases want to be getting to a point where we're experimenting but sometimes if you kind of go back to Cenevin when you're in chaos you don't have time to make a hypothesis so you're just doing stuff until things start stabilizing and now you've got a bit more information and you can have hypothesis does one more table want to add anything I think one thing that was not useful was overthinking we were under the assumption that the rules get complex as we proceed and then even like the simple ones which were obvious we couldn't guess it because we thought it is more complex yeah yeah so so yeah those rules are designed to kind of ease you into it a little bit but sometimes even a simple rule if you get the wrong run of cards because of that point where you're not getting any information you know even if it's red black red black and you play a red and then you're just getting constant blacks it's not obvious until you suddenly get a red and then suddenly it's blindingly obvious isn't it yeah and that's that kind of that information so what that's happening actually so this is generally what goes on here and this is kind of tying this back to the scientific method and tying a scientific approach you usually start off with a you know a question something you want to answer in this cases we want to learn what the rule is in order to do that we're kind of gathering information so we're observing the pattern of cards and you know sometimes in the beginning we don't have a pattern of cards so we're just playing cards and we've got something to observe from that we start forming a hypothesis that might behave with I think the hypothesis is it only number cards let's say that's what hypothesis then we test the hypothesis so effectively when we're playing a card and saying whether we think it conforms to the rule or doesn't that's testing the hypothesis so our hypothesis that you know if the rule is only numbers or hypothesis then the numbers 10 follow the rule and paste cards don't run the experiment and analyse the data so maybe what we find out when we start looking at the data now is that yes 2 and 6 follow the rule 5 doesn't okay so we've just disproved our hypothesis but equally we've got a lot more information there so we can now form a new hypothesis so we interpret that data form a new hypothesis and now we think well maybe it's just even rules and we repeat that and then we can share that learning with everybody else so that's the scientific process and trying to be intentional about what is our hypothesis how are we going to test that hypothesis on an experiment let's look at the data, analyse the data let's see whether the data disproves our hypothesis and then we kind of compete and we've got enough confidence in our hypothesis the key point is setting up an experiment such that you can disprove it so many times we call them experiments but really we're just reinforcing confirmation bias and there's actually no way of disproving it and it's really hard I kind of showed this slide yesterday but hopefully you've kind of experienced this now and kind of talked about this a little bit information theory where the probability of failure is 0% so when you know the rule and you're just playing cards and you're getting it right all the time you're not getting any more information because you know the rule equally those times when the probability of failure is 100% and this is those times when you're constantly getting cards which don't pass the rule as you just experienced sometimes you're not really getting any information there so the sweet spot is round about the middle where you're failing 50% of the time when we want to generate new information I've never actually done this but one of the things I want to try and do this is I'd like to somehow figure out a way of capturing some data around that where I think the quicker you get it is where you've kind of got a closer mix or a closer number of main 9 cards and side line cards and it becomes harder when you have more side line cards or sometimes you might I don't think I saw it and it's rarer but sometimes if you get all the main line cards and you don't get any failures equally you could be getting no information and you still don't know the rule so there's this idea of we have to fail generally when we want to generate new information I always caveat that because an example I used yesterday at home I don't want to be generating any new information about how the plane works and how the plane flies I'm hoping everybody's figured that one out already so I don't want my plane to fail but when we're doing product development an organisational transformation usually there's lots of stuff we don't know so we want to generate information the other idea that I've thrown in here is what we're doing there for is buying information when you think of these experiments as buying information so the smaller the experiment we can run the cheaper it becomes and the safer it becomes to buy a new piece of information so this is an example of Don Rynas and uses the principles of product development flow let's imagine you're just doing a lottery and the lottery is three digits so there's a thousand possible combinations of 0 to 999 0 0 0 to 999 you can pay three dollars to select and this is how lotteries work you pay an amount you guess all the numbers you pay three dollars you've got a one in one thousand chance of getting it right what we're trying to do by running experiments is saying you know what I'm only going to pay one dollar just to guess the first number so now I've reduced the odds from one in a thousand to one in ten and if I guess five great I can carry on but if I guess six I know I've not got the three digit number so I've just saved myself two dollars that makes sense so by buying smaller pieces of information instead of doing a one big change you know developing a product putting it to market it's effectively doing this your chances of failure and chances of losing money are much higher and by information we can kind of pivot and kind of go actually let's pay another dollar to guess the first number again until we get it right and you'll probably get in all three digits much quicker than just paying lots and lots of three digit guesses yeah I'd say generally it's when you're trying to learn something that you don't know so I don't think I'd apply to you say sport yeah when you want to when you want to generate information so maybe not manufacturing because manufacturing hopefully you've already learned you've got any information you know how to build it and you just want to use that information so in manufacturing you don't want any failures but we're not doing manufacturing we're doing software development and we're trying to learn so that's that kind of caveat when you want to learn new information but if you know what you're doing then you don't want a failure that makes sense okay let's do another exercise so let's build on that so this might be another kind of western cultural reference I'm not sure do you have this game called mastermind yeah this is a child where you've kind of got coloured dots, coloured markers and you kind of create and you're creating your own secret code here and then everybody else has to guess it and the white and black pegs are giving you feedback on how accurate your guess is so we're going to do a version of that now so yeah if we could put a couple of copies on each table that should be enough we're going to do it slightly differently because we're going to do it with numeric codes rather than coloured codes because that's easier so one of you so we might only have chance to do to one round of this one of you is going to be the code maker so you're going to take this piece of paper so there's one of your sheets there's an area at the bottom for you to write the code so whoever's going to be the code maker hide the sheet from everybody else and you're going to write a code at kind of a digit so any of these boxes can be 0 or 9 so you're basically going to make a four digit number and then you can just kind of fold it over there so you're now hiding the code from everybody else on your table so come up with the code the code is four digits so it could be one, two, three, four it could be one, one, one, one so you might repeat a digit if you want to, if you're being particularly evil and then everybody else is going to try and guess the pattern so you write your guesses along the top here and then you're going to get feedback in this area here based on so you're going to get two bytes of feedback so a full dot or kind of what I call a black dot means that one of your numbers is a correct number and in the correct position if you get an empty dot or a white dot that means you've got a correct number but in the wrong position so you're going to guess the code maker will you know, they might just kind of want to turn a quick double check remind themselves what the code is they'll fill in the feedback and get to do another guess so an example here so let's say we've come up and as a code maker I've written the code one, two, three, four on the bottom and turned it over and hidden it our first guess is four, five, three, seven I'm going to fill in a black dot because the three is the right number in the right place and I'm going to fill in a white dot or an empty dot because I've got a four right number in the wrong place now this important thing there's no correlation between the position of these dots and where, which number is there because that's going to make it far too easy for everybody okay so basically what it is to do the black dots first top left top right, bottom left, bottom right so black dots then white dots so all this is telling you is the number of digits in the right place number of correct digits in the correct place and the number of correct digits in the wrong place that make sense okay, hopefully people are familiar enough with the original game for that to make sense as well so that's basically that's just mastermind yep, question yes so the black dot the three is correct and it's in the correct position but you're not going to tell them which one no no no so where that black dot is so it's not giving you any indication of which of those numbers I've just drawn the arrow in there just to kind of have an explanation if there's two if we had four, two, three sorry, let me kind of one yeah then the seven would be in the right place but you wouldn't but the four is not correct, one, two, six, seven you'd put two black dots and no white dots yeah there's there are four here because there are four here so when you've got the code correctly you would end up with four black dots if I had if I'd guessed the code as four, three, two, one I'd have four white dots because I've got all the right numbers but none of them are in the right place yeah and is there any other questions on kind of the basic mastermind rules okay, the next bit that's the easy bit why do we call it lean mastermind we're going to add a lean element which is we're going to kind of do like the idea of a mini A3 so when you're doing an A3 an experiment A3 you're writing down in advance of the experiment what your hypothesis is how you're going to test it and then you write the results done as well so the other bit of paper you have is this one so the people that are doing making the guesses don't just write the numbers down write any numbers on this bit of paper write down your hypothesis so what do you hope to learn what assumptions are you making why are you making that guess that's your hypothesis then when the the code maker gives you some feedback do you consider that hypothesis do you consider that experiment to be a success or a failure and then what did you actually learn so it's like really mini kind of A3 thinking here where we're just using a row so what we're trying to do here is it's really reinforced that discipline of thinking through your experiment what is it you're trying to learn and what do you actually learn if you can write it down in a relatively small piece of paper relatively small space on a piece of paper then probably you've got a good coherent argument so this is just starting to exercise and starting to think about how do we write down and how do we sort of articulate what our hypothesis and experiments are yep any random number any four digit random number anywhere between 0 0 0 0 and 9 9 9 9 so any questions on that bit any questions so one last instruction we just have kind of just silence I know everybody's desperate to get into it code makers you can be kind of the judge and referee here as well don't give them any feedback until they can show you they're written working of what their hypothesis is so really make sure they do write something down okay let's just kind of give this 10 minutes so we might only have time if you get it really quickly then we might have some space sheets and you can have another go just let us know but we might only have time to do one round any questions otherwise I can just let you go all good so one tip with this is to try and make your learning as small and as specific as possible make your hypothesis as small and specific as possible okay we'll just give this another minute okay let's wrap that up so can we let's all kind of come back together if you just need to kind of let people know what the code was again just so you're not kind of sitting there worrying about it and then let's just spend a few minutes again just in your table groups three questions again and think about how did you go about discovering the code what sort of techniques were you were using techniques or strategies any kind of particular strategies were you were using anything you did to try and maximise the learning you were getting and then how did you judge looking out your hypothesis and whether there were success or failures how did you kind of think about how did you think about and judge success failure so just a few minutes talking about those three questions and then we'll just kind of get some comments and answers back can we do the same thing with the microphone okay so let's do the same thing was before just kind of wait for the hand held mics is one of the tables maybe share the things you talked about the answers you came up with advantage was that the first code that we built sorry can we just have a little quiet while we do just kind of talk about the debrief thank you the first attempt we built the code all the four digits were incorrect so it made it easier for us to build the next code or to guess the next code because we came up with a new set of numbers eliminated the ones which were not correct so you were trying to eliminate numbers and then once you knew what the numbers were thought you knew what the numbers were then you could work out which position we got the numbers we just had to guess the right so that was your kind of strategy did that work did you guess the code yes we did okay fantastic okay anybody else have any kind of different strategies or thoughts so we experimented by first trying odd numbers and then the even numbers and then we started removing each to figure out what the code is we didn't get the code but we learned that in the previous exercise the value of experimenting quicker and here using the hypothesis it helps us think about applying some logic and in how we experiment and the piece that we didn't manage to do is applying speed in applying the logic of experimenting as well okay right okay did anybody the last question in terms of success failure any kind of thoughts on that one so basically out of six iterations he was able to get the answers right so it's really intriguing that you know how to correlate because you only have a choice of nine numbers so I think in the first three steps only we will get to know what are the code so it's easy to crack the code maximum seven or eight or like six okay great thank you so the success or failure question in there is kind of because there's two ways of thinking about success or failure one is does the experiment succeed or fail so does our hypothesis prove correct or do we disprove it but the other way of thinking about success or failure is a successful experiment and we get learning from it and it's not a success if we want to kind of go non the wiser don't have any more information so generally when we're talking about failure we're talking about are we you know failure is disproving the hypothesis but actually you could argue that that's a success as well I wanted to give you a bit more of the back story between how we kind of decided to use this experiment and then I'll give you the secret or how I do this what my strategy is so probably three or four years ago it was Christmas family Christmas get together we'd all had a big Christmas dinner a few drinks it was the afternoon most of the kind of the adults grown-ups were kind of stunk and nod off and fall asleep my daughter who would have been you know 15-ish at the time was getting bored so she went to the you know my mom's games box of all the old stuff and she found my own version of mastermind you know the original version from when I was a child like daddy can we play this anyway I just started playing it I suddenly kind of made this connection between this is scientific thinking this is how you solve this so I started to teach her scientific thinking through the game of mastermind and in the process took all the joy out of the game which wasn't really my intent but you know I think she learned something in the process and then the other thing that benefited me was once she took the joy out of the game she didn't want to play it anymore and I could go back and have a little snooze but what I did was and the approach I ended up taking was and this is my tip was to create a small focus on learning a small bit of possible I would start this with a code one one one one the only bit of information I want to learn that is is there a one in the code if there is I'm going to get a black dot so basically and if there's two of them I'm going to get two black dots so I know however many black dots I get tells me how many ones there are now I don't know where it is but I can I can figure that out in future experiments so if there's say one one in there my next guess might be one two two two two now I'm kind of figuring out where's the one and how many twos are there in there so if I now get just a white dot it means I know there's no twos in there and the one isn't in the first position so really really specific experiments to generate very specific learning and actually you can guess the code quite quickly which is why it becomes a very boring game because I would always win and my daughter kind of because you know she was just making random not random guesses but you know the more numbers you put in there if you do one two three four okay you found out is there one in there is there two in there is there three in there and four in there but that's quite a lot of information you still have to run more experiments to figure out well is it the one, is it the two, is it the three and the four so I started really small so that's that kind of tip with this one and that's what you want to be trying to do with your experiments the more specific you can make it the more you're going to learn and then kind of to the point about running them quickly you actually can do that really really quickly because you've kind of got this very clear strategy so sorry I apologize if I've just taken all the joy out of this game for if you haven't played with your children in the future was there was there yes one variable at a time yeah yeah the more numbers you're guessing the more variables there are yeah exactly okay so that leaves us in the last bit and I think you should have a copy of this we're not going to have a chance to fill this in but this is kind of the next stage that you could do is the experiment A3 so what you're doing here you've got a few more fields in there is that idea of intentionally writing down your hypothesis, your experiment you know your context is is kind of well what's the current situation in the game what information do you currently have or what's the current problem you're trying to solve why is it you're running the experiment what's your hypothesis so what do you think what do you think might solve this and then what's your rationale so if you can kind of go what problem are we trying to solve how do we think we're going to solve this and why do we think this is the right solution being able to articulate those things in an A3 piece of paper means that you can now communicate that with other people you can get feedback on it because you can show that to somebody and they can make your rationales wrong or I don't believe this hypothesis in which case where people don't believe the hypothesis what do you do just run the experiment and find out it's not about whether the hypothesis is correct but it's about the learning you get from it then you start coming up with some actions and the key point is there are actions that might prove it but also what actions might disprove it so explicitly thinking about proving and disproving which means we can you know getting that idea of success failure what are you going to look for that's going to tell you whether your hypothesis is correct but also what are you going to tell you what are you going to look for that hypothesis is not correct and it's a failure so being explicit about thinking about these things upfront before you run the experiment and then similarly because we want these experiments to run really quickly it's not the end of it so if our hypothesis is correct there's probably going to be further work that we can do so what are our next steps going to be so you know if our hypothesis is my 1111 if there's a 1 in there my hypothesis is correct my next steps are going to be figuring out where the 1 is and then figuring out what the next number is so what are you going to do next or if it's a failure what are you going to do so if there's no 1 in there well I'm just going to move on and start looking at 2s and 3s etc so thinking about this articulating it writing it down in advance it's just good discipline to help you kind of think this through and then you can do this as a collaborative exercise it's a learning exercise it's a knowledge sharing exercise this is a kind of a mentoring exercise so A3s in Toyota were used as kind of a learning and a mentoring exercise so a manager would get one of their employees to fill in the A3 and come back and they would mentor them through it so they're not giving them the answer they're mentoring people on figuring out the answer so you get a lot of learning and then once you've run the experiment you've now got documentation about what you did, why you did it and what the results were so this then becomes information that you can share making it all explicit making it all visible ok, just to wrap up finish off my wordle so this example again my daughter does wordle and this is one where I kind of got really really stuck and I was saying to my daughter I basically said to her if you get this and I can't I'm going to be really really cross with you and of course you got it but I did get it in the end but at this point here I was stuck and I didn't know I couldn't think of any words that had an O in the middle and an I either there or there which is my logical next step so what I actually guessed was piano I know piano is wrong because I've put an A here and I know there's an O there but I made a deliberately wrong guess to get information because what I was trying to find out is well is there a P in there is the I there specifically and then I'm kind of figuring it in the end there so ok it's wrong but I now know where the I is so I've got some really valuable information there again ok so now what's a word that's something I O something something couldn't think of anything again racking my brain getting really frustrated so again I made a deliberately wrong breast and ironically we're in India it's balty well I've not even used the O and I've put the I in the wrong place I'm trying to find out and I've used the A again I'm kind of just going I'm just thinking of words that fare that use letters that I've not used so is there a B, is there an L, is there a T so at least the information I've got there is more letters so I'm now down to wordily I kind of have a keyboard below here and it kind of grays out the letters you've used I've kind of got enough information now that I know it's something I O something something and the number of possible letters that it could be it's small enough that it gives me kind of enough information to guess and is anybody anybody going to want to quit guess and you want to know what the word is it's kiosk it's got two blinking K's in there I mean how would I supposed to know that but anyway I got it eventually but only because I made these two deliberately long guesses to one figure out where the I was and rule out enough letters that was basically at that point the one possible word it could be now luckily wordly lets you in between here doesn't show wordly lets you guess words that are not words and they don't count against you so in between certainly here and now I was just randomly typing words in just in case they existed that I didn't know about them and eventually the brain makes a connection and you kind of go oh yeah kiosk so yeah sometimes you do things which you know are going to be wrong because you get information so that's actually deliberately failing to get information okay I'm not sure we've got time for questions but hey I'm going to stick around if you want to ask me questions come up and say hello I'll upload the slides to the website and you'll have noticed on here that both of those games and if you go to my blog of eligibility.co.uk both of those games you can download the slides the card game the LEC6 expeditious has a slide which has all the rules in there as well so you can go off and they're all open source you can go off and do these things yourself and the templates are there for download so thank you very much hope that was fun hope that was useful