 followed by both give them give you a quick positioning statement from themselves. Then i'n gaće gwaith i gwybod nance i ac sefydlu. So i'm a microphone. I'll put your hand up for you want to ask a question. I'll bring the microphone to you, so you can a question could hear it. And we'll let the panel answer the question and we'll keep going so we'll run out of time, hopefully. Is Abdul doing the wrong thing writer? The phrase doing the wrong thing writer is a I think I picked it up from a guy called Russell Ackoff, who's a systems thinker. What he meant was if you have a system, you have something which is in itself flawed, trying to improve that flawed thing isn't really going to help you. So that's the idea of if you're doing the wrong thing, trying to improve it, trying to do that wrong thing writer isn't going to help. So the idea of the question is agile doing the wrong thing writer. If agile is just helping us build software, and that software isn't actually solving our customer problems, is agile helping us? So is agile doing the wrong thing writer? Let's start at this end, maybe Craig, just introduce yourself and say a few words about what you think on the subject. So back in 1998 I wrote an article called Doing the Thing Right, doing the... I can't remember what it says. Essentially it was doing the right thing, doing the thing right. That's something I've been thinking about for a long time. And then in a book I wrote called Agile and Iterative, I wrote a whole chapter on Tom Gilb's Evo method. Because if you're familiar, if you're a student with Tom Gilb, who's really one of the founders of the idea of iterative and evolutionary development, his system of thought, which is called Evo and Planguage, has been focused on this question for a long time, which is something I've been very interested in. And in his famous 1988 book, called Principles of Software Engineering Management in Tech, I urge you to read. He drives home this point, which is the idea of having some kind of a quantified and measurable goal of what we're really trying to achieve in the business, rather than creating a laundry list of features. And Tom Gilb emphasized the following idea, which is actually an idea that used to be found in Scrum of the earlier days of the kind of laws. And one of Tom Gilb's dominant ideas, which is related to innovation and self-organizing teams, was do not give the team a laundry list of features to implement, but rather have the product management and business leaders clearly quantify what they're trying to aim for, and then give the team the freedom to innovate and how to get there. And if you're familiar with Scrum, there's this old consequence called the sprint goal. And one of the ideas behind the idea of the sprint goal was that instead of giving the team a laundry list of features to do, explain to them what we're trying to aim for, and allow them to innovate within the sprint in order to get there. Now it's been picked up again in the impact mapping movement, which I similarly support. And I urge you to take a closer look at impact mapping to try to address this problem. Thank you, Greg. So I'm Mary Popendick, and I tend to be the person that tells all the folks in Agil that you're kind of behind in time. Let's head that direction. And so my latest thought is that it's been decades, and I've been doing so for, involved in so for decades, that it's always been known that the biggest failure in software development isn't the software. It's building the wrong thing. It's putting in stuff you don't need, putting in stuff customers don't want. And as typically implemented, Agil has no protection against that, because the concept is that the team does the technical stuff and somebody else tells the team what to do. The person, quote-unquote, telling the team what to do is the one responsible for building the right thing. And as far back as Kent Picks explained, it was, well, we really don't want the team to be responsible for that stuff, because then they would get blamed. And so we want some outside organization responsible for figuring out the details of what the team does, and then it was a blame-placing thing. And so far, as our current Agil methods maintain that concept, somebody else figures out what the team is supposed to do. Teams can have technical success, but they cannot be engaged in having business success. So in my experience, the fun part about software development is figuring out how to solve customers' problems and create full-blown real success. Forget just the technical part. So we need to move Agil towards the concept of having the team be involved, again, one more time, the way it used to be, in figuring out what it is that should be done and what it is more importantly, that shouldn't be done so that you get the right results. And as Craig said, there's some movements in that direction, but nowhere is near enough and way too much, quote-unquote, by the book implementation of a micromanagement concept of Agil. All right. Hi. I'm Henrik. This is my voice today. But I'm feeling fine, no worry. So I live in Sweden. I'm an Agil lean coach. I help companies get these principles to work in practice. I've worked with companies that are very, very successful and I've worked with companies that have a lot of trouble. And it's interesting to compare how these are. I'm currently working with Spotify, which I consider to be one of the successful type companies. And when I compare these different companies, I notice one very interesting trend. That is the companies that are having a lot of trouble. They've fallen in love with that part of Agil that deals with predictability, estimates, velocity, and that part, which is useful. But the problem is they've kind of become obsessed with it. So it's all about, oh, we've got to get a stable velocity. And the problem is with 100% predictability, you get 0% innovation. So that's a problem. And once we can get out of that notion that predictability is, that's not the key thing, then we can start focusing on other things. The other part of Agil, which is like, hey, talk to customers and try to solve the real problem, et cetera. So yeah, I've seen that trend quite a lot. That people just tend to fall in love with that micromanagement aspect. We can get out of that. Hi, I'm Linda Rising. I'm the oldest person on the panel. In fact, I've been on this panel before. How many times? I've been on this panel many times. And I'm still here. And I wonder why is it that we are still talking about this problem? Why are we not moving forward? I liked what you said about Tom Gilb. I'm definitely a fan of Tom's. And in fact, I know that going out and finding resources can be difficult for some people. And I'm here to tell you that you can get a lot of stuff that Tom Gilb has written for free. You just go to his website, Google on Tom Gilb, G-I-L-B. And there's a lot of free information that he makes available. So that's certainly one thing you can take out of here today, even if we don't come to any kind of useful conclusion up here. So I will just tell you a story. I've changed everything about it. So I'm even trying to remember what exactly happened. But I was working with the team and we had the customer there. And there was a little tiny problem. And so let's say that it had to do with the color of a little corner of the webpage, a little bit of screen real estate. And somebody said to the customer, well, what color would you like there for the background? No, that's pretty straightforward, isn't it? The customer says, black. Just make it black. Okay, you made a note of that. We'll move on to other problems. The customer went away, came back a short time later, and now we have everything ready to go. And the customer immediately says, what's that over in the corner? We said, well, that's a little piece of real estate where we decided we were going to implement that. It's black. That's what you said. Customer says, and I quote, oh, when I said black, I didn't mean black. Did we get it right? This is a hard group to follow. In way of self-introduction, my name is Jeff Patton. I will tell this group what I told the group I spoke to yesterday that I've hated agile development since the very first day I got there. Throughout the 90s, I spent time building successful products. I wrote code. I worked with customers. In 2000, I got the opportunity to work with a new start-up trying this new process called Extreme Programming. They had hired this guy named Kent Beck to come in and help this group out. The first choice I was given was to be someone in the customer role or to be someone on the development team. I said, well, I can do both. I'd like to do both, please. They said, no, that's not the way agile works. I said, well then, if I can't write code, I like making things, I'll be a developer. They said, well, actually you can't because you're not good enough compared to our other developers and you have too much domain knowledge so you need to be a customer. I learned early on that if I'm going to build successful products, I enjoy, I do this for purely selfish reasons. I like making things for people. How many people in the room have set with a customer or user while they've used a product you've made? How did that go the first time you did it? I heard bad, I heard scary, and it usually goes bad. There's usually this shock of realization that what we chose to build didn't work the way we thought. But I found over time that I could actually deal with that, fix that, and I could get to products that people really loved and valued. And it's a point of pride for me that I've got products out there that people have used for a long time. So I'll see if I can draw this to a conclusion here. I learned that by spending time with people and understanding them deeply, I can build products they like. On my first XP project, I learned that the way that this process worked, we hired smart people and smart people for what to build. And those smart people already were smart, they didn't need to actually talk with customers and users, and that's the way that process worked. The truth is I fell in love with process discipline of thinking about process more deeply, and there's so much I love about Agile, but so much I think that needs to be fixed. If we want to really build things we're proud of, there's something missing that I've been kind of ranting about for the last 10 years. So I'll stop there. So Linda was telling the black is not black story, and so I think it was to paraphrase Lord Kelvin, we cannot help but understand something if it's expressed in numbers. And so it seems to me one of the most powerful tools that we have as people building stuff together is to, insofar as it's possible, move away from fuzzy language to being able to express what we're aiming for numerically or quantified, and that's my personal bias. And similarly, not only be able to quantify it numerically, but be clear on how we're going to measure that. And I think that this is a view on defining our goals and separating means from ends that is very little talked about in product development, except in some very edge case situations. And I'm not really sure why, but one of the things that I've observed over working with different groups over the years is that typically the people that are interested in serious improvement, or for example this very discussion that we're having here, is that most of the people that are having this conversation are on the engineering side of the fence rather than the business side of the fence. And for a variety of reasons, some of which I speculate, I think I understand, and some of which I don't, there's not a lot of people on the product management side or the business side that are actively engaged in the kind of question that we're having here. Henrik, if you just have a last comment and then I'll take a question. I want to compliment the tendency of whininess up here on stage, complaining that things are so bad and Agile is messing things up. I tend to be the optimist, that's my role. So let me compliment this with a positive story about how Agile actually helps sometimes. And what I sometimes see is this little template you've seen as a blah, I want blah, so that blah, and I know that everybody doesn't love that one, but I've seen concrete examples of what that does sometimes. It makes developers ask this magic question, why? And for whom is this needed? And by just asking that question, wonderful things sometimes happen. So that's an example of how Agile has sometimes helped people build the right thing. Great, that helps because I was going to say, yep, I have another mic. You have your own. So up until that point it seemed like we had an answer to the question, is Agile doing the right thing? Wronger, wrong doing the wrong thing writer, and we might have finished the panel. So thanks for bringing a nice positive spin on that. So what questions, does anybody have any questions? This is the first hand I saw. So actually this is a question that I got from a manager. So don't ask me to clarify the question or anything like that. So I'm just going to ask it to open-ended. But the question you asked me to ask is actually how do you quantify value of iterations? So the question was how do you quantify the value of iteration? This then immediately leads to the question of what do you mean by value? And from Tom Gilb's point of view and the impact mapping world's point of view, value is a meaningless phrase. And so for example there's this movement called impact mapping or impact management. And that's one way I like to frame it better. Instead of this more amorphous thing called value, can we speak more about impacts that are being made on something, whether it's the number, I work in investment banking a lot, whether it's the number of open breaks at end of day or the number of reworks on a trade, that kind of thing. And if you quantify this, let's just take as a concrete example the number of open breaks at the end of day in an investment banking process. If you quantify that clearly, define what number you're aiming for and have a way to measure that, then the way that you measure value is actually by creating what traditionally has been called a double loop system, which is that you have an inner loop, which we could for example call the scrum loop, where we actually implement something towards the numeric goal that's trying to make an impact, and then an outer loop where you actually measure by installing that solution into a solution space, you measure whether that business impact was made, number of breaks open at end of day or whatever, and that's how you define and measure. That's the gilb system of thought and the impact mapping system. So a complaint I've had for quite a long time about our entire industry is that it's not scientific. We like to think that we are, or maybe an engineering discipline of some sort, but that if we incorporated the idea of a controlled experiment, which would involve some kind of measure, I'm just building on what Craig said. So we've even lost, I think, the meaning of the word experiment, because when I talk about that, most people say, well of course, we experiment all the time, and by that they mean try. So the definition of experiment has become just be a risk taker, try something. But I mean experiment in terms of a scientific experiment. So there should be an hypothesis. What is it that you want? Or what are you testing? And then there is the running of the experiment and you should know at the end whether or not your hypothesis is true. And I think that's what agile development is. It's a series of small experiments. And I don't mean experiment in the sense of try. I mean that in every iteration you should be attempting to measure something that we could call an hypothesis. And you would know when you got to the end whether or not you had really done a good job testing that hypothesis or not. Just add one more thing. With every iteration or with everything we're doing, I'll focus on learning. What did we learn? At minimum if we built a part of something, while that part of something isn't necessarily valuable by itself, we can at least learn how fast we're moving. If we predicted that we could build five parts and we built four parts, well that's useful information. But ideally we could build something that we could then put in front of people and learn from. I have this weird twist in my head just when I think of solid metrics. And so I'm thinking of one of the older iterative processes that I'm aware of. And it's the process that doctors use to cure patients. And it's one used by veterinarians and used by doctors that work with critically ill patients in the intensive care unit. They do what's called a daily soap note or hourly soap note in some cases. S stands for subjective. We gather subjective data or information. I look at patients. I observe what happened. O means objective. I gather the results of tests that I've run and measurements I've made. I combine those two things and do an assessment. And then I create a next plan. And we do this at a daily cycle, a weekly cycle, an hourly cycle. But it's considered medical malpractice to just use objective measurements or just use subjective measurements. You've got to use your head and numbers and your judgment and both go together. OK. Next question. We're discussing is agile building the wrong thing, right? So are we saying that we are analyzing whether the input that the development team has, the requirements, they are essentially wrong and they are not the feature that customers are looking for. Is that the only thing we are discussing here or is it a little more than that? I think we're kind of discussing that, but there's probably more. I want to hear what Mary has to say. We're discussing how to create something that the people who use it, let's call it a product, really like, really love. Let's just take the classic iPhone. I swear that was not developed from a long list of requirements. I know it wasn't. I know it was developed through a classic industrial engineering design process and the software was subject in the motions to the same kind of process. And it was the right thing because it was very successful. Now, how do we do that? It's not a list of requirements. It's a how do we build stuff that people love? What is the answer to how do we build stuff that people love? Development. Separate process is the problem. OK, I'll take that back. I'll say as a process that helps improve development process. We are not talking about agile as solving what comes in. That's a different thought process altogether. How do you make people think about the right things to get it in there? Oh, there's plenty of people in the world. They're called designers that do that for a living. I think of agile as a tool or as a thinking tool and there's almost never anything wrong with a tool, whether it's a hammer or whatever. You can do great stuff with a tool. You can also slam yourself on the toe with it. And to repeat this idea from Tom Gilb, connecting it back to not separate processes, if we give the team more opportunity for innovation, instead of giving them a laundry list of features, explain to them where we're trying to go or what we're trying to aim for, ideally through some kind of quantification. I should mention, actually, there's two broad areas that seems to me that we're talking about. One is products for the marketplace and the other is internal solutions like in an investment bank. The dynamics of those are quite different. When you're creating solutions which often are related to reducing cost or reducing risk inside of companies, quantification of goal is actually more strongly applicable and very clearly applicable. When you're trying to create something like the iPhone, although there are aspects where quantification related to power consumption and so forth are perfectly clear, there's somewhat more intangible elements that are difficult to do that way. Okay, sure. But there are elements, for example, from the design thinking movement that I think are applicable within the team. But the broadest comment I wanted to make was that Tom Gilb's idea is not that a separate group defines what the agile team does but that they're given more opportunity to innovate within the context of a communication of a goal. So I come from a background of product development. And to me, measuring the success of a product is like piece of cake. It's easy. You put it in the market and your company is either going for revenue or margins or something like that. You have clear idea of success or failure of a product. The work of the people working on it is directly tied to whether or not the product does the right thing for the company or doesn't. Whereas if you have indirect goals of cost reduction, how do you know the cost reduction is really the thing that's going to make the company successful? So products are much more closely tied to company success and much easier, in my experience, to latch into and say, I'm going to make a successful product. They're more interesting. You can get more enthusiastic about it but you have to think about it as a product that a group of people, a product team, is delivering to the market, not as a set of processes with independent teams doing pieces. I'm still getting twisted up sometimes about setting concrete goals and numbers and an example popped into my head. There's a U.S. company called Hotwire.com. They sell hotels. They're a division of Expedia. I work with product teams there. One of the teams is led by a guy whose name I wish I remembered. He manages hotel supply. He doesn't even put any of the UI that customers see, but he had the notion that we just don't have enough supply and we work with large chains like Marriott's and Hilton's and other Western hotels, but if we could get a lot of smaller independent hotels, I think people would have more choice and that would make a difference. He didn't set a quantifiable goal. He didn't know. He knew how to measure. He figured out what the least he could do to put some more supply in a small geographical area. I'll be damned if putting a lot more supply and didn't start to double the results. There was so much benefit that no one could have predicted it was going to be that good. If they had set a goal for an increase of 10% or 20%, they would have been super off because they doubled things. They proceeded with a notion. They built as little as they could to confirm a hypothesis and then they figured out how to expand it to the rest of product and the whole team understood that goal and focused on it. The whole team worked together to figure out what's the least we can possibly do to test this assumption. I see successes like that all the time when teams are focused on the goal. Certainly we need to be able to measure, but I don't know that we need to quantify sometimes. I've been waiting for a chance to read this. I brought a copy of a little snippet from President Obama's inauguration speech because when the press covered it, I thought they missed something that was very significant and I thought indicated something that maybe meant that the government could begin to be agile. So I'm quoting, for now decisions are upon us and we cannot afford delay. We cannot mistake absolutism for principle or substitute spectacle for politics or treat name calling as reason debate. So here it is, we must act knowing that our work will be imperfect. We must act knowing that today's victories will only be partial and that it will be up to those who stand here in four years and 40 years hence to advance the timeless spirit once conferred to us in a spare Philadelphia hall. You move forward even when you know we're probably not going to get it right. But we're going to take a step and what we think is the right direction and we're going to learn that's agile to me. Cool, thanks. So I'm going to compliment that with a very, very, very concrete recommendation for something you might want to try. Disclaimer, I don't know if it works. But I know that other, I've seen people trying it. I've seen people trying it, but the data isn't in yet. So you can help participate in this. Now this is the merge of the agile and lean start-up movement. So suppose you have a team and your feeling is maybe it's your team you're like we're cranking out features and widgets and we're focusing too much on output and too little impact. Here's the thing you might try. Take your backlog full of features such as we need to add chat functionality inside our application. You take away those features and you replace them with hypotheses. Our hypothesis is that if we had a chat feature that will change the way people use the product in this way. It'll increase our net promoter score or it'll have people stay in the application longer or whatever. It's a hypothesis. That's what's in your backlog. It's when your hypothesis is validated to be true or false. So how you do that, well that's up to the team. They'll probably need to do A-B testing or whatever. I don't know. Try it. Tell me if it works. Great. I'm going to try that. All right. I saw one here, one here and then one here and they're all over the place now. I'll try and keep up. A couple of conferences ago, Mary and Kiko, manager at Ubuntu, presented a session on the difference between agile and open source. And the differences were many. But Kiko summarized the differences by saying open source works because we care. Now, the question that I'd like to pose is does agile really permit participants to care or does it get in the way of their caring? Good question. Yes. I don't think anything about agile prevents people from caring. I think, as Craig laid out yesterday morning, processes are built sort of defensively. Sometimes it's difficult to care when you're, and as Mary said also, when you're financially rewarded for not caring but hitting delivery targets, you're not rewarded to care. I believe people do care, but I believe it's the way organizational culture is, the way they apply agile practices, we get results that allows people not to care. We say that that story meets the acceptance criteria, even though we know it's bad, and even though we know nobody wants it. We treat stories as requirements without caring if they're going to actually benefit people. And there's nothing that causes that and nothing about the agile manifesto that causes that. I think it goes back more to the organizational culture, more to what Mary and Craig have been saying. People use, you can use a hammer for good, or you can become a claw hammer murderer. Anyway, and I think people do bad things with it. There's research to support this, but also just to say it more anecdotally, if my job title is Tester and I work in a separate department, and we have an organizational design where there's six steps of different groups from requirements to architecture to programming in module number three to testing module number three. I've never seen the customer. I don't see the whole. I only see the part, and I measured and rewarded for my task. And there's lots of disconnection from my work to seeing the whole and connecting to feedback from the customer experience. It's my experience and my observation that people will tend to care less. And insofar as we can create organizational designs and cultures in which people and systems are set up to see the whole, get feedback from the customer and be part of the entire sort of cycle of success or failure, I think that agile can support that. I think actually, to not exactly agree with what Jeff said, I think that structurally within systems like Scrum there are the roots of mechanisms for people to care more because it's set up potentially to see the whole more. Is there a question? There was one round here somewhere who was next. OK, I'm going to go over here. I'm trying to keep some sort of order. I just had one quick question. I'm agile coach and I meet a lot of people who are on the ground working and a lot of them believe in these concepts and they really would like to follow a lot of these beliefs. However, the organization structures, the way the organizations are structured, the HR practices and those kinds of stuff, those are really big impediments. Now, Craig had mentioned that bottom-up is extremely difficult, really. For those people, what would be your suggestion? My response is that it's rather off-topic. I'm trying to get to his question. Is agile doing the wrong thing right? Is it valid to say that we are trying to identify the wrong thing early so we can take incremental steps in correcting and doing the right things? It's not only about requirements. It could be about our engineering process. What are the wrong things that we are doing in-depth? What are the wrong things that we are doing in-test? What are the wrong things that we are doing in QA automation? I'll give you an example where on the product we had 80% automation of our test cases, but they were the wrong test cases. They were not executed frequently, so they were used only 30% of the time in the year. You're losing a lot of productive work because you're doing the wrong thing. So, just not to say what's wrong, but I want to paint you a picture of my experience in product development. A product development team was not a team of six to eight people, a product development team in 3M, which, by the way, put new products on the market as its core competence. It was 20 or 30 people led by a champion. The champion was, let's just say, if the champion ever tried to tell the team what to do or what priorities were, they'd lose the team. The team made decisions, the champion helped the team be successful. The team included people who were going to design the product, people who were going to get the technology there, people who were going to do marketing. But there was no concept that the marketing function told the team what to do. They had the same status as technology, same status as quality, so you have a bunch of functions that have to have arguments. They have to disagree with each other because they're naturally not going to see things in the same way, but they have a customer out there and they want to give something to that customer that delights them. As a group, they go and talk to the customer, at least senior leaders of all of the different functions, and they come up with an idea, they try it out as a sort of group of co-equals. There's no one person, whatever their background, saying this is what we'll do, here's your requirements, here's your priorities. There is a group of people with different backgrounds with a leader who's helping them to be successful, understand the customer's issues, the job they need done, and they constantly come up with ideas that they test on what's going to make that work. We had this concept of make a little, sell a little. We could even do that with adhesive products. We could actually put a test product on the market on the slide for no money, kind of big brown steel, people's time. So we did an awful lot of experiments, but the teams that we had were always not just cross functional, but there was nobody that really got to dictate to the other functions. They had to work together and they had a leader that made the groups work together towards a successful product. We need to structure that kind of structure when we're trying to make something happen. We need to have varying people with different backgrounds and a leader that helps make them successful put them in touch with customers so they can come up with ideas of how to make that customer happier. It's maybe not as methodical a process as you might be looking for, but it works a lot better than any methodical process I've run into. In cases where agile does go wrong is when we are too bogged on of thinking that as a process. To me, being agile is more of having that agile mindset. So, yeah, as long as you have the agile mindset, don't think about it as a process. Now, the question is, does having that mindset work better with teams who are in product companies versus services companies? Yes. Because the development team is closer to the customers. The real customers. We are trying to have that agile mindset going. How do you get that same mindset? I didn't hear the first part of that. I said in Indian IT industry, 80% of the companies are services oriented companies. That's a fact. How do you get that agile mindset going as a community if that's the fact? I want to make sure I understand the first question clearly, but a service is a product. Banks may have software products, but banks provide a service and the software supports that service. So, you've got to, when you're in a service company, and if IT is your service, for some people I'll draw this continuum, where on one side I'll put the word doctor, and on the other side I'll put the word waiter. When performing our service, we need to act a lot more like a doctor and a lot less like a waiter. It's not our job to take orders, it's our job to understand problems and honestly help. Sometimes understanding problems and honestly helping the person who's asking you to build software is asking them tough questions like, who is this product for? How do you know it helps them? How will you know it's successful? The questions my doctor asked me about my weight and how much I eat at McDonald's, those are uncomfortable questions to answer at times. But they help. So, what Jeff said, Spurt, a thought of the connection of your question actually back to the panel topic, which is doing the right thing, which is the intersection with outsourcing, and especially in countries like India, where there's often a lack of a sense of personal safety for the people to ask questions and probe, and to say no, and to investigate amongst, like for example, I work with my colleagues here at Baltic India, so I have a sense of what it's like here over the years. And it's very common for the Indian folks to feel uncomfortable to ask probing questions, and that then exacerbates the likelihood that we end up creating the wrong thing. And it's not exactly a wicked problem, but if you're going to succeed in helping to create the right thing, the conclusion is that we have to create a sense of personal safety, as Alistair Coburn would say in an agile environment, where people feel safe to challenge, ask questions, ask no, and ask the right questions, speaking to the right people instead of just receiving orders. If you just receive orders, it's especially easy to create the wrong thing. The question is a follow-up question of the iPhone example you gave, and in iPhone, okay, agile, if I understand right, it's more customer-driven, and say iPhone is design-driven, so basically it comes from somebody's vision and who believes that it's going to make a trend. So how does collaboration work here? And how do we keep moving forward? What are our measurements? And how do we do the impact analysis when we don't even know when it comes in market how it is going to be? We had a theory at 3M that if you put 100 products on the market, you were lucky if 40% of them were going to actually be profitable and make it over any period of time. And if you were really lucky, you got 10% of those being really good and an occasional every year or two blockbuster, because you did a bunch of experiments. And believe me, there was no guaranteed process when you were doing product development that for sure what you were doing was going to be successful. And if you need, what did you say? If you need guarantee of success while you're doing something, then you'll have zero innovation. 100% predictability is zero innovation. So companies that experiment and allow failure and have ways for people to... I mean, we were always told if you don't fail, you're not trying. And if you're in that environment, then you can create stuff. If you have to be quote-unquote successful all of the time, then you don't have the freedom to do stuff that's really going to be innovative. And that's true at all levels. That's also true for individuals. And that's what the Agile Mindset talk was about, is that the individual level, if you have to always be successful, if you cannot fail, then... Actually my question was, here the customer inputs is missing. So... The customer inputs is missing in a design-driven case. So... That's not true. No, I would violently disagree. I'm sorry. Maybe in certain cases, if you happen to have a benevolent dictator like Steve Jobs, who seems to have an amazing... seems to have an amazing knack for understanding what people want, if you have that kind of person, congratulations. But I wouldn't count on it. It's not having customer input. Design is taking customer input as one of multiple input streams, using your sense, using your ability to try many things. Designers don't just go down one path. It's just not what they're trained to do. And just because... I mean, it's not like I tell something, this person tells me, it's like, I've got 100 people out here, and I have to understand across this whole group, and it's my job to figure out across this whole group, what is going to be the most interesting and have the most impact? And you guys, you know, you don't actually know what you want. So I have to try a combination of input, understanding job, empathy, walking in your shoes, to get what is the most interesting thing that's going to really, really be interesting, exciting. Here's an old quote you've heard probably before, but it's worth mentioning, and that's the old Henry Ford quote. If I asked my customers what they wanted, they would have said, faster horses. Design thinking means, no, no, no, I'm going to build a faster horse, so I'm going to get them faster from A to B. Two things, just to add to that. Correct design means lots and lots of customer input, and it means asking customers what they want, and it means getting behind that. People say, I want a faster horse, and they say, that sounds great. What would you do if you had a faster horse? Well, I'd get to places faster, so it's important for you to get to places faster. Okay, now we've uncovered a need. Now, there's another weird thing that happens where I see a lot of successful product companies that built what I call a FUBU product, a For Us By Us product, and when you're building a consumer product or when, you know, if you're 37 signals building base camp and you are your primary user, possibly you don't need to look so closely at the outside world. You can trust your own judgment, but the reason I know about soap notes, for instance, is because I spent time with people in a newborn intensive care unit watching them fill out these darn things. I don't know what their life is like, and I can't do a FUBU product for them. I've got to pay close attention, but I'm acting as a designer, and my first step is learning a lot. And they keep asking for faster horses or faster software, and my next step is to understand why. I have to thank you for that template. That's beautiful. I need eggs. What would you do if you had eggs? I would do why. Aha. Thank you. Great. All right. I think we have time for one more question. Basically, I have two questions, but I will start with the second question because my friend already asked this question. Okay, so we have this product company, and they outsource to vendor, and they say we are agile, okay? They say we are agile, but also we would like to implement agile. Fine. The vendor says yes, fine, because he's a vendor. He has to say yes to his customer, okay? And this product company says to a vendor, okay, so these are my backlog, these are my requirements. Bring the estimation. Being a vendor, he says this is my estimation. The product company says no, no, no, it's too long. You have to build the product before the deadline. This date. Because of that pressure, this vendor tweaked his, he changed his mindset. Okay, we can do it. They work overtime. There is no agile in the service company. The vendor, there is no agile. They put a pressure on developers. Okay, this is the overtime work. Where exactly is agile working here? It's only in product company or in vendor? I don't want to walk over that territory again. Kent Beck, one of the founders of the agile movement, encouraged the value of courage. And one of the things that he encouraged, I know this is kind of like a tough thing to say, is have the courage to stop working at crappy companies with crappy managers, and see if you can find organizations that are going to do good things. Another thing you might do is try to find places in India where you can use your great talents inside to put in products for this country that make India a better place to live and a more exciting place. You have one of the most rapidly growing economies in the world. You have all kinds of customers here that are close. So if you get tired of doing what other people from some other world tell you, focus on what you could do as a product company here in India and all of the opportunities there are out there to do all kinds of new and interesting exciting things here instead of just having people dump requirements on you. You said something very interesting. As a vendor we have to say yes. One of the things that happened a bunch of years ago when I started working with Mary and Tom, they came to Sweden and I was telling them about my company, and I was pretty proud of my company, and then Tom asked a very annoying question. The question was, so which clients do you say no to? And I was like, damn, he got me, damn he got me. And that changed quite a lot, because we had to start thinking about which clients do we say no to. Until you can answer that question, you're pretty much gone. You don't know what the heck you're doing. You don't know who you're going to say no to. So yeah, think about that. You don't have to say yes to everything. In fact, you shouldn't. Thanks, Tom. Okay, so with the continuation to that question. So how do we, I mean, we talk about product development always, so when it comes to agile, product, product, product, product. So how about operations? How about marketing? How about service? So where, do you have any read, I mean any books or any article which says how do we maintain backlog for operations and service and all the stuff? The panel discussion. That's okay. No problem. And then the last question. Sorry for too many questions. So what is the role of project manager in this whole agile context? None. All right, four minutes left. Sorry, Geoff, you're just going to, you're going to jump in quickly? I have another little piece of, I want a faster horse, tell me what would you do that. If you're focused on really benefiting people and you said the estimate word, I have another little piece of verbal jiu jitsu that I use a lot. First off, Mary, you use the example of industrial designers a lot, and anyone who designs a something usually starts with a budget for how much that something is going to cost. And if you talk to anybody selling anything, one of their first questions is, well, how much you've got to spend. If you go to a car dealership, they're not going to ask you for your requirements for a car and then tell you how much it's going to cost and then wait for you to say, I don't want to pay that. Well, it may be a little bit different in India. Usually when I'm working with people, I'll say, well look, what do you have for a budget? What problems are you trying to solve? And let's let us come up with the requirements or come up with the details, the thing we can build that satisfies that. All the way down to planning meetings instead of product owners showing up with stories that describe what they want. I'll have product owners describe what they're trying to accomplish and set a budget and let the team talk about what they could do to meet it. They've worked wonders. They take ownership of the goal. If you can convince your customers to let you know how much they want to pay and then let you guys work with them to figure out what you can do to spend their money wisely, that works for me and clients in a vendor situation. In any company with project managers, you probably have a very good collection of really good team leads. Wonderful people that are really good at leading teams and those are very valuable people that you want. But the concept that somebody takes requirements from outside and gets a team to implement them is the rolling concept. So you don't want project management. You want those wonderful team leads to play other leadership roles on your team. All right, thank you very much. We're out of time. So apologies if we didn't get to your question. Thank you, panel. I certainly picked up some good tips there. Good ideas.