 Pleasure to see you all back and if you're coming this is the first day then welcome to the conference We are already late by one minute. So I don't want to take too much time Really happy to have Dave Thomas here. We've been trying many years to get him here And finally we've been able to get him. So Thank You Nuresh. It's an honor to be here and I Just want to say something about Nuresh because I Think the reason that many of the speakers are here, and I'm sure many of you Because Nuresh and a few other individuals in your wonderful country invested their personal time to try and make sure that the Job of building software could be better more interesting and you could be continuously learning So I don't have a chance to thank him So I wonder if you would join me in thanking Nuresh and his team for this wonderful conference He's truly an outstanding software engineer and recognized around the world and I Think many of us are happy to try and arrange our fairly busy travel schedules Because we know how important this event is and events like this in our respective countries Community matters and community makes things much different. It overcomes The blockages that exist between companies and creates new opportunities. So Thanks to the team here So I'm going to be talking about value-driven development if you abbreviate that to VDD it's probably not a good thing to do I'm not it's not a new brand and It's a certified certification free I'm a in a previous life and still a university professor. So I'm used to provoking students and irritating them So I may say may may say some things for the point of view of making you think You know, you don't need to be confused that I'm actually correct in anything. I'm saying But hopefully they'll make you think about things a little differently Since that kind of the job for a keynote speaker As you've found out from Josh and his very progressive work and modern agile The outline for the talk since I'm at the beginning of the scaling session and in a previous life I was a did a lot a lot of very large-scale agile transitions Some with thousands of employees globally But by many companies that specialize by doing something stupid like putting half the team in India and keeping the people who Held all the knowledge in other countries like Europe and America. I Was pleased to hear last night at dinner talking to people that some people are smart enough to realize that lean says whole team whole team So Clearly we're making some progress, but some of you may work in organizations that need to learn that I Can assure you those of us that are working with your Colleagues in America are telling them how stupid it is to split teams across geographies, but we have many good speakers to talk about that later Please do make America comm So I'm going to talk first about a little bit about scaling And I'll talk about processor in development. I'll talk about value-driven development because that's really the focus So the question is can you scale agile and the answer is we know we can It's pretty simple to do provide you have a few few ingredients. The first is strong consistent and respected leadership We have a hard time doing that for our countries let alone for our businesses I'm Canadian so there may be a bit more humor in this talk than American speaking The business needs to go from talking about projects to products lean is fundamentally about how you build products So if you still talk about projects I'm sorry. You're unlikely to be very successful in your journey other than having many many scrum teams and not much difference if you don't begin by putting in continuous integration and delivery and Assisting that can measure everything then you can't improve it again the basic lean principle if you can't measure it you can't improve it This is why scrum has almost no impact if you don't have a measurement system And the other thing that many people don't realize that if you want to sustain High quality and engineering you have to give it you have to have people connect as leaders and mentors and Communities of practice. These are found in the best companies in the world And when I was at IBM we put in something called the distinguished engineer role where a senior technical person can work without staff or with a small staff or managing a large product team and Be paid at the level of a vice president or an executive These people are key to the successful development of software in the parts of the world where I work Just a disclaimer. I'm a product person. I Have a lot of experience working in large IT companies all painful and unsuccessful or less successful than I would hope There's a lot of rhetoric about emergent leadership and so on in the Agile Bible Unfortunately it doesn't either contain the Old Testament or the rest of the New Testament It just contains the simple part the Ten Commandments Or maybe it's well for XP But it's very simple and success in any organization that I've worked in depends on leadership Because leadership is the foundation of trust and respect and Leaders have these qualities and realize that slides are probably too small for you and I'll save you reading them But you get the deck But the most important thing is leaders have time to listen. They're not always busy. They don't spend their time going to meetings Now the performance metric that I like is one over meetings If you go to them more than one a year your bonus starts going down Death by meetings is a wonderful book I have a quote from a very good friend Nigel Dalton Who have you've ever seen the wonderful work at Lonely Planet or the lawyers and the accountants? Everyone in the business worked with cards and put them on the wall It was absolutely fantastic and I was there in February one year and I said Nigel He just succeeded in getting the UX people to put up because the UX people never wanted to put their stuff up until it was done You know they didn't really want anyone to comment. They would they would user test it in their own silo But there's a silo so and the big thing that he came across and Nigel what happens if you're not here And he said Dave Agile is fragile six months later the BBC changed the management And Lonely Planet went from being the best exemplar of Agile in the world to Like the rest of people mediocre Agile The people sustained it and so on but the leadership which so critical the person who had those values who had the trust was gone Agile is really hard. It's really hard on leaders They have to deal with problems every day to try and move things out of the way And it's very difficult to sustain this leadership and He's now CIO real estate comm Where there's 800 people doing amazing things with Agile But I fear that if Nigel would leave there the same thing would happen What this tells me is That scaling and systematic transformation is a problem You want to talk talk more about your individual problems on scaling I'm filling in for a speaker who couldn't make it this afternoon And you can bring your questions about scaling in your organization or your friends organization because you wouldn't want to ask Question about your organization, but you know someone that's scaling and they need some help If you bring those along, I'll talk a little bit about Agile in large But I don't want to talk. I really want to talk about your question. So that's not my talk today So I'll move along. The other question is that scaling in my view should not be a goal Systematic transformation is a good way to waste a lot of money and energy Across a large organization and get very little results Often it doesn't I mean you you're going to scale hundreds of thousands I worked in organizations with you know 15,000 employees building software, right? And you might as well get a hose and spray them with Agile holy water, right? Here you go You're blessed you become Agile, right? You're smoking dope if you think this is going to work You know, I understand there are amazing. There are a few amazing people who can do this Evan's amazing. You work in there are people who can do this, but they're outstanding leaders and As soon as you lose that you lose the whole thing and It's very very hard the Huck is an example of a person who transformed a company that that Joshua mentioned yesterday, but there aren't thousands of them. There aren't hundreds of them You know, there's probably 10 20 in the world at any point in time The other problem is that you know when you go to processor of in development You basically end up with a small improvement But the you see the CEO when you told them Agile what they heard was agility and you were so Enthusiastic to tell him to get going with Agile the CEO said well The rest is this going to give me you know productivity and cost-saving You know because if not, I don't think we should go with this agile stuff And then that moment of weakness because you thought it would be so good for the company you went Yes, and you lied Because the agile contract other than it violated by Jeff Sutherland and unsubstantiated by any proof Does not give you productivity That's not the contract. The contract is a better working environment predictable delivery and quality That's not what executives care about as a first-order bit They want to have that But only if they get the other stuff they want which is productivity or cost-saving so fundamental the agile contract doesn't deliver and That's the expense on systematic change in my view is poorly played So this is just an example of the kind of CIOs I meet and basically they have a CEO all hands meeting Everybody comes in here and basically the message here without going into the specific words is Okay, we've given you everything We gave you objects Maybe we gave you Scala, whatever it is your favorite language your favorite tools. We gave you your favorite process We've done only said we even changed your space If people renovate their whole buildings to get this amazing space, right and does the output change? Usually no It's better quality probably 30% which is significant for people to have below 30% in quality Which is the average In IT shops and so basically his message is guys. We've done everything we can for you But we need to be faster at executing high-value Projects or have a high-value products. We need to turn up the dials on productivity while maintaining quality and features and At that point the tape self-destructs as in mission impossible because this is your job How can we make a change where we deliver? substantially more value faster Because in the end what people really want is better cheaper faster and Good enough look we use the web You know 404 is good enough for everything. We just throw code at it We finally found a business solution where we can ship lousy software continuously and The badge of honor is how much junk can you ship today? That's called continuous delivery in many cases and the measure becomes a metric of how many releases do I do not? Did I actually deliver value to the customer? This is a serious serious problem and its companies decide that they want to look more like the California web companies They take themselves on a very dangerous journey Because the California web companies are in a very different kind of situation with regard to the people they have the small business problems they're solving In this you know Netflix is an amazing business I'm sure you've heard Adrian talk about his experience in CTO, but the business is actually fairly simple Right you select the movie They charge you for it and they deliver it And they do delivery through a separate channel So they don't use the cloud for any of that So it's an amazing story, but the business is actually dead simple Unfortunately, you know boring things like insurance and health care and things like that Which actually matter are much harder and much more complicated And we get this model that you can take whatever happens in California and just do it in your shop And this is a very very naive and a very dangerous thing So basically what the CEO is saying here is we need to focus on value. We need to change our practices How many of you are our steel programming with object technology? And how many of you are still programming pre-object technology and the rest of you don't want to say Who in here uses property-based testing who in here knows what it is folks Agile is 20 years old objects are almost 50 years old. We need to move on You're not using best practices you're using yesterday's practices You have to be responsible for picking up the new practice. You can't expect your corporation to do everything yourself For them you have to do things for yourself. We need to look at these new practices And we need to get rid of everything that's low Because those things block us that may mean your entire production department Which is a challenge in a large organization You need to be able to fail fast That means you need an environment where you can experiment and deploy quickly but also pull it out when it doesn't work so value-derived development Basically, I've been building software for a long time. You can tell by how old I am and And this is kind of my secret for how I do I never did it the way I said I did it But that's what you're supposed to say because otherwise all your competitors figure out how you do it But basically I never actually believed in building software the way people said because it was just too hard I'm not that smart. They're always really smart people in California and they're very scary smart So you can't possibly be off in the boondocks in Canada and build successful software It's just not possible Unless you cheat So you have to find some hack Which will cause so I'm really going to share with you my hacks for how to get things done and The reason that people let us use any programming language or tool or practice They want is not because they actually believe the tool or practice is any good But because we ship We guarantee the software we build will deliver on time or they don't pay us for it It was a clever thing. We did when he started doing business. We didn't actually know how hard it was But it's you know like the great story about the basketball team, you know cutting down the net We just started saying look, you know, if it doesn't arrive on time, you don't pay for it We did have a few painful experiences Where we didn't get paid But we learned how to do it in contrary to no estimates We actually teach people how to be really good at estimating because when you deliver things on time for fixed price You have to be really good at estimating So we invested a lot of time in continuous estimation So what we really want to do is find a quick way to figure out where the value is in the system Validated at scale because everyone knows you can prototype a wonderful new technology solution easily But when you go to lean on it Oh, it doesn't quite hold up. I can remember a client. They're a whole business was based on a website And a wonderful new young man came in with his team showed them how they could use this wonderful new JavaScript framework and he built the best demo and they let him go to the board and Enthusiastic he told the board how he could have this done in six months and this would take the company of this huge hole and He did have it done in six months except you couldn't lean on it And the system crashed under load and the company lost hundreds of millions of dollars So it's great to have a new idea, but you have to validate it at scale. This is a critical part of the process The other thing is we don't want to work on anything that we can't deliver in less than three or four months But as we're looking for the shortest time to value because that's the way the corporation thinks If you can do something for a corporation in a quarter, you're a hero if you can't you're a schmuck So you might as well drive to what the real goal is which is getting something significant done every quarter If you can't do that go someplace else find another project you can do where you can So to get management buy-in and mitigate risk You have to have strong senior sponsors And that's a longer talk about how you do that You have to have a clear client tangible and measurable goal Ideally in dollars or throughput or something like that you can measure so you can actually Guarantee that you can test the business value. You can show it. You have to have a show a model It shows that you're going to produce significant value in value business value Ideally something that were 25 or 30 percent, but nothing less than 15 Have to be a limited quickly and you have to be minimally disruptive on existing operations Because most large corporations have a lot of systems that are making the money Those are the ones written in the language is you don't like to program in but they're the ones that are working while your new software isn't Right legacies are are great because they make the money The rewrite of a legacy is where we waste a lot of money So the challenge is how can we insert this stuff in the right place because we're not going to rewrite anything significant in three months So there's a set of activities that Can do this. There's a set of technical challenges. We need a small team with the right skills. We need a whole team We're gonna work. You know, we're gonna be working with SAP BW. Somebody knows how that has to know how that mother works Right because there's gonna be a billion tables in SAP BW And you have to know how Hanna works or the previous version works So you really need a trained killer in SAP BW So you'd have to have the right thing. So sometimes contrary to the agile religion specialist do matter You know, if you're gonna really do something where you have to go and insert small amount of explosive in the middle of SAP BW To get what you need You need to go in there with someone who knows how to do it And so the two of you can die together when it blows up in the wrong place And the odds of that are unfortunately fairly hot So you have to you have to invest in live backup technologies, which are not available If you ever tried developing for SAP BW the whole process of you know, I mean configuration management This is an absolute nightmare. You know the old prod the law of law We have one client That has 13 systems you have to release through to get they have 13 prods I'm in prod. Well, you're not in the real one. Then you're in the next one. Then you're in the next one Terrible. So you know just trying to get the thing released could take you three months So you need a straightforward deployment minimum dependency So some of the techniques we use for doing this And this is not all them. I just want to talk about some of these things and For some of you that are post technical, which is something I hope I never am You know, this may go by you But please listen to your troops if they start talking about some of these things and say we want to try some of these Because these are to me the kind of secret weapons for attacking some of these problems. Many of them are quite old Everyone knows about microservices, right and Process I so what you have is you have all these little processes and they connect through data flows And they all run concurrently. Isn't that really cool? You ever talked to your dad or your granddad and he did something called structured analysis and design Dataflow diagram Take a look at your microservice architecture But Fred George does and sell people Dataflow diagram Execute the only people in the world who made sense of the Japanese banks who figured out when they were doing cobalt Not to throw away the Dataflow, but to actually build an execution infrastructure, which where the process is actually run So for 30 years, they've been running microservices in a main fair Sometimes the old ideas come back And then they become really cool Until they complicate them with all the extra stuff that the new smart people have to bring So envisioning There's one thing That's been the most significant thing for me and for the teams that I've worked in in different companies It's basically figuring out how not to build software How can we write less code? And so the most important thing and this this all comes from product design One of the unfortunate things about agile is it took some of the simplest thing from lean the workflow things From a small part of lean But left out a whole lot of things in product design So if you happen to accidentally have done anything in product engineering You would actually know about all these things market analysis technology evaluation delta competitive analysis Qfd health of quality and so on but unfortunately, they're hardly talked about in the agile community or the lean software community Product engineering has been doing these things integrated product teams have existed For many many years and Toyota in the early 80s with the the key model for these things and much of the work there comes so The key thing here is that you find a way to fail fast So envisioning and envisioning what we do is try and build the product as quickly as we can out of the Simplest thing we can use paper The competitors product by marking up the competitors product That's often one of the best ways right there that one of the best ways to write product requirements You want to write the story? That's really easy to take the three products that you want to compete against Delta the features from those it's called delta analysis and then you basically go and say okay now We got our spec. We want to be able to take these three guys out of the market in that order This is one's the one that's just above us will take him out Then we'll go and take the next one out then we'll take the next one out That is a story of IBM Java Goodbye semantic goodbye Until we're only two players left in the market. So this is sort of an example I mean the classic problem when you want to build a calculator is that you if we all know about the waterfall pitfall and Analysts and you get this huge huge calculator that you can't carry So the alternative of course is you get the Bob Martin calculator Right where you sit there and you write a unit test and you look at this. Hey, I got this thing done and it does plus Isn't that cool? Yeah, well it took that time and I have to go to problem Oh, yeah, I have to build plastics for it and everything else. Oh, well, it's a part of a product Well, that's not gonna be too good Well, what what but I want time. Oh, no problem We'll go back into another cycle and we'll build you another wrong thing because I'll be missing divide You need a sense of the whole This is just too expensive Writing unit tests is a huge waste of time relative to writing acceptance tests Unit tests are for programmers that can't program class tests or component tests or acceptance tests are for customers and programmers that want to build in a great systems and Not to say that unit unit tests like story points our training wheel The problem is how long are you going to be on that stupid training bicycle? Someday we'll actually teach you how to really estimate in units that people understand like dollars in time the way the business work Not going around talking about points and velocity, which is a measure of nothing Throw your velocity diagrams out and put in cumulative flow diagrams at least read flow That's probably the only good lean book on software. That's deep lots of good inspirational books So what we want is something that lets us quickly build something that looks like the calculator Very quickly so we can find it. It's wrong and so we can show it to the customer interact in them How does this work in practice? This is a very large system. It happens to be an IBM product now it basically looks at websites globally and Essentially audits the website for security 508 compliance to checks all sorts of things for corporate websites It's a wonderful piece of Canadian engineering and has a UI that was built by a Canadian engineer. It really sucks Yeah, I mean you would not this is the UI from hell But the back end is amazing You wouldn't have any products like that, but we do in town and So the challenge was rebuild this Now of course while they were fighting over which UI framework to work and there was no one to talk to And which technology to build in We had one person Who was in our vision team and basically they went out and built a low fidelity prototype those little red dots So things you can click on and we use a very sophisticated technology building that which I'll tell you later And so he built a simple model three structures and people played with those who liked them then he said they said let's go ahead So he built another iteration now. It's gone from eight pages to 48 pages. So it's getting a lot more fidelity to it and He accidentally placed it on the external web page and told a customer about it So we could get a real customer feedback Got spanked and promised never do that again But got some really great feedback and then did the detailed version which is a hundred and forty eight screens That's a lot of web space and code and The investment was less than two percent of the overall development less than two percent We believe that IBM actually bought the company having seen the mock-up. How has this done? interactive PowerPoint No fancy tool How do we make sure it's authentic because you know basically when you're looking at websites around the world? It takes time to sweep all those websites So if you go click and the page comes back full instantly, you're going to have a complete phony experience So it has good engineering behind it about the performance of all the page accesses and so on even some engineering code They built the prototype to see how long it would take to fill that page to know whether even the UI model was reasonable for doing that two percent of the overall effort How many story cards is that come on folks? Let's put this on story cards. You were you know good stuff You know get serious about story cards And then you throw them all away. That's really useful. Come on back. You know, here's my project a dump of story cards How do you maintain this stuff? Come on Agilis get serious We cannot scale or build products if we can't have documentation. I see that's allowed I was so excited to see a sketching talk here. We're making progress in modern Agil. You're now allowed to sketch What a cool idea sketching the key part of the vision This to me is the biggest things coming but what this does is save you from building the wrong thing So you can fail fast very quickly using various because there's no code here refactoring is really easy Refactoring is of course a joke for large software Refactoring is a great hype and the dance and for people that are really good like Joshua and the people he trains They can actually the but when you get to these large large corporate code bases There's no chance that you're going to do these serious refactoring. They're very high-risk and very difficult So for individual teams the refactoring is key and if they do it all along You won't get those big ugly code bases But when you do have one do not be confused to say how many of you worked on a project said we told them It's all right. We're going to refactor it because you knew refactoring is a good word But what you really said we're going to tear the whole thing apart Rewrite it and restructure it and you were lying when you said this is not an equivalence preserving refactoring It's a rewrite Those are very dangerous So envisioning can save you So the key thing When building software of high value is to find a really fast and simple way to do it because you need to cheat Because if you build most software the way you're supposed to you're never going to get it done At least not going to get it done in the timescale. I have to get it done And if I can't show short time to value I'm not going to get the business and I'm going to be out there competing with all these other smart people who can Probably program better than I So what I have to use is to use my head to try and found is there a way I can cheat So the fastest way to make things better is to change the way the business works Many people never bother to ask Like why do we have this process? Good question, but most of the time people don't walk upstairs and talk I've been in gone and talked to lawyers and said this doesn't make any sense to me But I'm told that few people reject it. I mean like that they go make perfect sense to me I think you give me a letter saying we don't need to do this. Sure Again, no code required Very easy change the business The other big thing of course is the programming has been saved by years by the hardware engineers have built faster hardware and Today hardware is amazing So with today's hardware you can simplify all sorts of problems. I'll come back to these Improve software. There are better ways to do things You know database technology's changed a lot We have machines lots of machines that can have lots of processes We don't all have to live in the same memory space anymore In the old days, we had to link the whole ugly thing together Now we can make it into independent small processes be careful of course of the microservice trap which is to take your large tar ball and put it into a nicely set of dependency preserving microservices So you no longer can you release either because they've all got version signatures and they all depend on each other Which is something I see congratulations in microservices bad news has got all the same problems you had before So you have to understand the so new language is hell and I guess that's my kind of problem It's I'm kind of a language addict Because I found that the amount of code I have to write is the amount of code I have to maintain So if I can find a language which will let me express myself then you know, I have less code I mean if I if I get in one of the reasons I can't deal with Java and God forbid I did build a Java VM and an eclipse ID. I my apologies for that You know, I also was a founder of the Agile Alliance. So I owe you You know, I've lot of sin so you don't want to hear what I'm doing now I won't tell you You don't want to go there and Improve practices and one I'll mention today is property-based testing So testing is still 30 to 50 percent of development And I subscribe to the test soon principle that is the time between when the test and the code is written should be short I don't want to argue whether you have to write it first Or it's likely after or during or whatever, but it's soon But how our code inspections are really good you can find hundreds of bugs using code inspector. Remember the principle from lean is collective ownership Which is you know, I call four eyes on the code What do you call that you know 16 and mob programming is a wonderful example a Prairie appropriate at certain points of time for having increased sharing or for achieving collective So the principle from lean is collective ownership pair programming is one way to do that so our code inspection so our design reviews So as mob programming, you know, don't get hung up on one Now who would like to write all the code in their system twice Who wants to write everything twice? Well, that's why you use cucumber isn't it so you can write all the tests once Have them all break when you make any change and do this other. Why do we do these tools? I mean BDD is a thinking process is great But come on you can automate almost all the code that comes in You can don't need to write all those silly tests that break and fail big problem unit testing not a big problem for Module testing or component testing or acceptance One of the best techniques I know for high-risk systems is side-by-side comparison of two independent implementations This is my favorite really strong technique Independent programming teams basically build the system twice and they test the results or they build the test part independently without access to the other code Been used for years in things like NASA and so on it's actually not that hard to do You consider how hard it is to write all these tests sometimes It's easy to write the system twice because the programmers like writing code. They don't like writing tests So this is my favorite technique for really serious code For UI, I love selenium. I love it, but when you're doing something complicated I tend to build things that look like IDEs. Sorry They're very hard to test with something like selenium and we keep changing them, you know making them worse each release Adding features we don't need that's what big companies do Supposedly because it's hard to talk to our users, but we can't measure the problem is It's very hard to actually keep up with the changing UI So the tests are almost always fragile if you're fortunate and can do functional testing and you've got a business sort of thing That's a wonderful thing. I learned a product here called Sachi, which I think would be very useful for a lot of the IT companies But for doing the kinds of UIs that I'm involved, which are very complicated We have a very simple approach which we've been using for 20 years and it's very simple as soon as Any significant amount of code goes into the system? Into the UI we freeze the code base Everyone uses the component. They're not been working on They bang on it for two or three days We get hundreds of bugs off by four pixels, you know the 15th item in the menu doesn't work Why do we have 15 items in a menu? I don't know That's a first problem, but you know we try and identify those things and get rid of them Within three days after all those bugs are closed none of those bugs are ever going to come back if I wrote tests for those bugs They would be useless. They would just load my test This is the best practice and it works and it produces very very stable code Because there's lots of just accidental stupid bugs in UIs and I don't have any faith than any other technique But by far the best testing tool today is something called property-based testing And if you go away from nothing else today, and you care about the quality of your software encourage you to look at property-based testing It's available for every programming language on the planet Come something called quick check which was written by John Hughes for Haskell It generates you essentially write a specification or a model of your program and given the model of the program It generates tests that conform to that model randomly So it's really good at shaking all the edge cases essentially by writing one spec You get a thousand unit tests or 10,000 unit tests or a million unit tests Really really useful Of course the problem with random testing is you get a zillion of you get you know millions and millions of tests that fail And you go oh, thanks, right? That's like fine bugs, right? Fine bugs is great except it finds way too many bugs So you don't know which are the good ones Which ones which ones do I need to fix because most code has way more bugs than you should you believe So in the end it's only about fixing the ones that matter Spensive ones ones that impact customers So the big thing that John did is he put in something called shrink And what it does is it finds the smalling the smallest failing test So it finds the sequence of operations that causes it to fail and isolates it down to that test This is just fantastic. I don't have time to go into talking about How it works or some but it's been used on some very very very react is a distributed database It has like 90 95% code coverage written by a team of PhDs Within two weeks John's team found eight critical bugs in react That should also tell you something if you try and use distributed databases, but I don't have time to comment on that If God had believed in distributed databases, he would not have concentrated all the brain in the head and Connected the rest of the body with a slow bit serial interface quote IP shark So this two things I think dog booting and property-based testing are the essence for us Our success the other thing is simplicity, right? We need to find a simple way to do things So fast software equals fast hardware plus simplicity and simplicity to use Martin Thompson's wonderful term is typically About essentially mechanical thin but sympathy or software sympathy Understanding how the system the code and the architect should go together and flow So today's hardware is very fast You can go to a website I think this is Dell for $15,000 you can buy a computer with a terabyte of RAM 100 terabytes of disk and 64 cores There are many business in the world that can run their entire business on this computer all Universities of India could run on this computer no problem. They won't ever do it, but they could You just immediately means you can do automated build and test You could also go to Amazon and spend a lot of money But more than this using Amazon instances, which is the fashion, but I don't time to talk about that Hardware is very cheap. That's why Amazon's making so much money charging you for using their cheap hardware all The interesting data is in memory. We have customers with machines with 10 terabytes of RAM next year you're going to be at 10 terabytes in your PC of Non-volatile memory Think about think about it My I am convinced that within five years ten years programming will be essentially reduced to queries You see there's now people doing very sophisticated things like compiler analysis and so on points to analysis Optimizing by very sophisticated algorithm do it now through queries Datalog Extended SQL dialects and so on Unfortunately the one thing we don't teach people in computer science or software engineering is how to do query But that's what we're all doing today Very important you want to understand how to do query and what the meaning of the joining data and working together These machines are very very so the nice thing is now when you've got a machine. That's really fast It's got lots of memory all those contortions Algorithms, you know goofy libraries, you know, oh, I need to implement my own red black trees because I learned in CS So that was really cool Or I want to use this buggy open-source framework because my friends are using it You can actually do things very simple you can do linear search Why is the kind of technology I use fast because it's in harmony or sympathy with the hardware Hardware is not round. It's not objects. It's rectangular the registers are rectangular the memories rectangular caches rectangular This is rectangular and so as long as you traffic and what the machine does you're going to be able to execute very very fast And it's going to be very simple. You don't have all this serialization garbage and everything like that You maybe use things like Google protocol buffers or God forbid Avro or parquet or anything like that You do what you don't want to Well, it's just horribly slow and inefficient You're really working on a machine and when I use fast what you want to do is just stick the data together with a counter for fun That's called a vector And so that's how you get the other thing is now we can do things with end-user computing So we can actually compute over huge amounts of data and end-users can do that We don't need to get a Java programmer or a C sharp programmer or a C plus programmer Because the machines are plenty fast enough for doing that. We run spreadsheets on trillions of rows at blazing speed Easily doable with today's technology would not believe how fast computers are and they're way cheaper than program Even Indian program. I don't know what you can get done for $15,000 in India But I know I can get zero done in America for $15,000 By a programmer versus I can give this to an analyst and they can be flying and living in their data I'm serious. Many people could run their entire country on that one box Because the interesting stuff is in there Look around things to change now the real serious problem in software is software We have too much of it. So the biggest waste is bloat the biggest contributor to bloat is object-oriented programming and the associated languages and Frameworks basically bringing a whole lot of stuff You didn't care about into your program and give you all sorts of dependencies so that when you try and separate things out You by the time you loaded five frameworks are in this tangled web that you can't remove things. That's because We shouldn't ever let frameworks come out. We should have they're supposed to be sealed components So you really want to get rid of objects when you can because objects by their very nature and Pointers or references create tangled tangled webs I'm sorry. I did pimp object very oriented programming But I did it with a light and programming language called small talk that even normal people to do But then of course the industry brought you the very best in things like Java and C sharp. I Would gladly take your hundred ten million lines of cobalt and give you my ten million line my one million lines of Java I would make a lot of money Java legacies are very very scary and They're not possible to refactor unless you've been doing it all along So there are some very old techniques when I graduated which was a long time ago The badge of honor was to learn something called table-driven programming. I Have to call it cheap TDP because TD is already taken But these are actually really understood by BA's devs and so on they're very modular and they're all data driven So you can change them without you know rebuilding the system. You can even live change them on the fly This is at the core of many very successful system Yeah, if you're in a bank, you may have heard people go. Oh, we use this old piece of garbage called Hogan Terrible old banking system. I don't know why they use this stupid old system because it's all table driven You can change it. You can introduce a new product new banking very easily. No code required Data modularity is much more important Moving data is much harder than moving code Unless you're trying to change the code. You want to change the code then it's much harder So they're used in all sorts of applications and these are some of the common decision tables is a fantastic technique We're working with users To do this and I'll describe some case studies reducing it, you know, basically Integration the whole notion of integrated came up with this notion in order to put things together what you need is API's Here's all my API's and to make it easy for you from you to use them. I've sorted them alphabetically thank you and You know because I knew you'd want you'd want to be flexible I gave you ten different ways to do certain same thing Without telling you which one would be better in what circumstance So my first answer is if you're doing data integration if you're doing integration think data integration just forget API's Just leverage the web or leverage the physical page format. It is easier to talk to SAP BW or Oracle financials by reading the physical disk pages that is to use the API's or To put an atom feed on top of it. So it just comes out as a pop now. You've got you can use use web tools or If you have an old technology and it really is hard to use you can put an ODBC interface on it Now anybody can talk to a ODBC and they can get their report out and crystal reports whatever or simple mashup tools So just by putting a standard interface on which is a data interface as opposed to a programming interface You can get rid of thousands of complicated API's and of course the API's never really work anyway, right? Because they only give you half the data they want because they were added later, right? Somebody went along says we've got this big ugly mess We're gonna put some API's in it so you go around randomly drilling holes in it and pounding API's in which give you part of the stuff So they're always low-performing and low functionality Loose coupling of course is the rage And the key thing here is the API's are value-based. We never want to pass references We only so that's what component programming is really about is API's that take values So we're going to a world of stateless things or immutability And you want to use the more fancy program we're going back to a world of data flow pipes and filters Occasionally disconnected many system. It's easier just to replicate the data That is to try and keep it up in real time. There's most query systems actually travel behind the real data anyway and You know you can things like Erlang implement things very efficiently just using co-routine These are the modern ways to do things and they work very well It's a very old architecture Called batching This is the notion that instant you know all sorts of people get try and get their website tuned in their database tuned They process every transaction in real time and they get spin up all sorts of scale out servers And look at this we can and then of course you can't query them because they partitioned the data across all these servers And you can't query across them But we'll solve that problem by putting another layer in there and build a whole lot more infrastructure on top of that But in practice if you actually batch those queries say run a thousand at a time the users won't know the difference You'll run all the queries in the same time. It took you to run one Batching is the secret to high-performance systems And if you've never heard of Martin Thompson I please encourage you to take a look at his mechanical sympathy blog and his most recent writing on batching in the world that I come from Currently there's a very old technique of Basically having all the data the world available This is what people have been doing in the stock market for years What they have is all the data that happened today all the ticks Every trade in the market and they have all the historical database of all the ticks that happened since 1992 or whatever Using these two files You can do very efficient and scalable processing And this has now come back as something called the Lambda architecture Again once it goes to California it has to have a new name and a more or less efficient implementation in Java called Hadoop and Then you basically have a real-time database and a batch database this architecture scales You can scale it out. It's very simple to use and it's because it's immutable You don't have any kind of problems in recovery or things like that Event sourcing is a closely related technology that's becoming quite hot It's very simple relative to the previous sort of data technology. So if you're looking data architecture, that's it and please please Do not ever let Do not ever output strings Or if you do output strings do not let them into your data store data using Jason or XML or CSV Guarantees the your your processing time will be 10x longer and your storage cost will be 10x and Of course if you're like most companies you don't even put a version identifier in your Jason file So when some goofball decides to change the Jason string you can no longer look at data over time Welcome to the job of data scientists. They're hacking pipeline Python scripts to remember that Oh, yes, it was December 2nd when we added that other attribute to the UI testing So we have to actually run the different program and we've got all these different Python programs strung together in our data pipeline Because we can't version our schema Or we don't have a schema No SQL is one of the biggest ripoffs of all time It's makes it really easy for programmers who don't want to understand data to write it and it makes it absolutely hell to store and process So if you are a person who writes a program that writes no SQL You should have to write every report that I have to get take that data because I have to parse that data It's very expensive Process it and so this is just dumb dumb dumb. Please do not work with text It'll never be fast Maybe there's a message here so I'm not going to give you the functional programming talk because I'll have to talk about a monad and Everyone will feel like they don't know math anymore and they feel bad But the simplest way to program is to basically work with collections. What are most mobile applications do? Why is mobile programming so hard in the end all they do is they put a collection on the screen from which you pick Most programs just work with collections That's one of the problems with object or any program We got carried away and every time we had a collection We made it into a class and gave it a name and made things very complicated in practice all we're really working with data structures and functions So today The fashion is to use filter reduce and other idioms that come from functional programming You do not have to buy into all the type theory and everything else associated with functional programming Unfortunately the Haskell community which is a beautiful language But where everyone is a math head makes it difficult because every time they talk to people they make you feel bad that you didn't do well in math Or if you did math, you didn't do the kind that you definitely didn't do category theory But more and more the world is able to do things with data affect a big data and functions The reason that functional programming is finding is not parallel computing It's working with data because most data is very simple. I take it from the disk I filter it I chop it and I put it on the screen Someone decides something about it and they do the transformation back It's straightforward ETL and there are lots of language. You're doing this. I very strongly recommend you look at some of these I Personally I didn't put my own ideas. There's a strange language. I use Q there I'm not recommending you do that every other language I sold I I pimped to you was obviously the wrong thing to do So, you know, why should you follow me again? So pick one of these other ones But try and pin something similar and of course Python and these other languages JavaScript. They're all very much headed that way So That's some technology Let's talk about the sort of business side of it How do I find out where to put my three months of effort? What I look for for inner innervation opportunities are insertion points And the places I look for them Most often I was where the data rests or flows The last thing I want to do is touch code Whereas most people when they go at something what they want to do is change the code change the codes really hard I Worked a lot with Michael feathers. I know that Michael feathers walks away from big big refactoring projects As many others because they're just too hard But most interesting places the other thing is data flowing and value flows value streams They're often very closely related in computer systems. So there are lots of places. This is just a partial list Sometimes I'll actually go to the code if it's a real compute bottleneck small piece of code There's a highly structured place where there's a lot of rules There's a place where the code has to change all the time and I'll change those into data driven or table driven pieces typically Performance stop. I'll end up basically doing the kinds of tricks that Martin Thompson does Or I would hope to be able to do that Once I find one of those points then I can have associates. I can typically connect the flow points very lazy to the value chain And given the value chain then you can apply a transformation. So what I want to do In the last few minutes, I'm probably over slightly, but I'll make it quick It's just give you some examples and I'm using some legacy systems Because I actually believe there are some people in the audience that talked to me yesterday who actually work with legacy code I know all of you probably write new code all the time and I'm working in really cool new things But most of my clients have code that makes the money and programmers that want to write new things that don't work So this is a large insurance company and they had a new CIO who Basically lost 200 million dollars rewriting the insurance application. It was one of those large consulting group, you know, I'm I They're in the room. So I can't use their name But you know, they begin they begin with I and a and C and things like that And basically big companies just rotate which one after they get burned and they get them spin the wheel and get the next one I and they give them another 10 or a hundred million dollar project which becomes a 300 million dollar I wish I knew how to sell those but I don't So they had a whole lot of legacy systems They had a bunch of vendors supplied things like rating engines and so on printing systems for insurance forms a bunch of enterprise applications SAP people soft Whole bunch of integration services a bunch of commercial infrastructure and since The first big vendor starting with I was on this particular client They had they had figured out that what they needed was a new object store and a high speed They said it was high speed Bus enterprise bus this would be a SOA and so And of course the CIO said no, we're gonna do this agile So we were brought in to sort of train the people the agile the first thing we found out is they didn't actually have any Java developers But they found some C++ developers down the street by the C sharp developers they sprayed them all green and then send them to of course with Bob costs and Bob Martin who I worked with at the time And it was the situation was pretty hopeless. It said well, there's no chance. They don't have any programmers They've got all this stuff from vendors the vendors are just fighting with each other Not that your vendors would do that our vendors in America do that but yours are all get along nicely I know and of course they had a enterprise software bus and I just said well, we're done. We have to leave we can't be successful and Jeff Eastman who was working on A floor above came running down to no no Dave we have to stay so that these people are amazing Turns out to be they were in a part of America Which is very stable so these people have been working in the business analysts and so on the the domain experts They were very rich in domain experts. They actually understood insurance Intimately all different kinds of insurance commercial personal and They actually had everything You know for years because they've been supposed to write the specs But of course everyone's busy building all the technology So each time the project had canceled is kept working on another kind of insurance and they built these huge tables They had Excel sheets that used all the columns going across an Excel So the first thing we did is we taught them how to cut their except you know modular Excel how to cut the Excel sheets up So you didn't have to scroll them or try and figure out how to clip all the paper together And then we basically said you know These things can all be basically described by tape decision tables validation tables and they've got it all done They've got this back And of course the agile guys. Oh great. Let's go. We'll have some story cards. Let's start doing this. Wait We can just take the specs and generate all the code because remember I'm very late And I want to get out of there quickly and get my charge. So we generate all the code had a small team that built the interfaces to the Enterprise bus and then basically delivered it to the acceptance test The other thing is we forced all the vendors that were delivering to the system to run acceptance tests in the contract You cannot pass your process across the contract But you can pass the fact that if you deliver with acceptance tests I will give you the check right away if you don't deliver with the acceptance test I will give you the check when I have a chance to try it That is IBM's policy for an a-class vendor versus other vendors who have to wait to get their check So your contract can deal with vendors that don't use a process compatible with you. This is the system HR ball that can't backwork done this one was a company in Chicago They do they do complicated HR policies when they negotiate with the union the union They have to implement within so many days very complicated the analysts do everything in Excel They had a hundred cobalt devs implementing the code They came in and they did it in small talk, which is an amazing and they got 15% trained everybody in small talk Look, we're using the new tools. We're using a new process. What did I say about systemic chain? Not much different but again Can't look at this and said oh Everything's already coded It's just coded in Excel and As you know Excel is not always the most disciplined programming language So what they did is they wrote a rule checker to basically put constraints on the Excel sheet So that they could take the Excel directly from the analyst and they wrote a spreadsheet first in small talk It runs today in Java that runs in the main frame talking to the cobalt program and They were able to redeploy the cobalt program is to work on other things that need to be done in the business And now they can make changes relatively instantly because the whole thing is data driven You see a pattern to this a simple engine That actually does the work that automates a key part of the development process that reduces the time to do this It works over and over and over again This is a large data migration This is a factory automation system. You know that factory automation is very expensive You've got all this equipment heavy equipment and it's all proprietary and you can't change it You got three hundred million dollars worth of Alan Bradley or Rockwell or whatever You're talking to all your factory equipment and it's all proprietary and they basically change the standard So they can patent something that puts the pins in the wrong place. It's so nasty. This is What we did is we put in a TCPP TCP IP card because they're proprietary bus And then we just turned everything into HTTP and atom feeds Now all the UI all the fancy control system stuff could be done by people that knew how to use the web Trivial change no massive UI development Last thing and I'll finish here last project. I worked on I was at system for doing cyber cyber analytics basically trying to identify a bad guys in cyberspace and What we had to do is support people that are using fast data So we're working interactively in tens of hundreds of terabytes interactive response and 10 to hundreds of terabytes What you can do on modern hardware And we wanted to make it possible to work with both the live data in the past it because if you're looking for something like a bot The way many things work in securities They have a window and they look there and they see if they see you know Someone's doing port scan or doing something bad in that window then they shut them down or try and do that The problem is if you have sophisticated cyber attacks when they want to set up a botnet or something like that They actually hide in the regular stream and so they set up over months To find those you have to look at not only the real-time data, but you have to look at all the historical data So in general what you have to do is use queries that go against both that data so that's kind of the problem in a in a high level and and We basically build a system that has both analysts and estimated programmers model because in this case the people who are using this This is like an IDE, but it's designed for normal people as opposed to programmers and Those people have very different backgrounds So essentially we have a set of tools on top that makes it easy like spreadsheets except the spreadsheets can work on a trillion rows of data And visual tools with a high-speed rendering and then we do all this stuff underneath To do this the only technology we could use that was fast enough and compact enough because we actually run in the network as well As in the cloud Was a vector functional programming language? Which to me is the most powerful technology I've ever used I can write programs that are very small and very fast So I love it and I'm not telling it to you because it's too hard And you just be the wrong thing if you start using it many people do then it'll obviously go bad like small talk And so there's a very fast and simple infrastructure running at the bottom the entire execution engine is is 80 k bytes 80 k bytes. That's the database the programming language the tcb stack odbc everything in it So it's very compact and very fast again close to the metal and so we were able to build this with a team of six people Front-end JavaScript back and it's easily the complexity of a clip Thank you. I'm sorry if I went over time and took away from your teeth Right. Thanks a lot Dave One of the reasons I really really wanted Dave to come here is as you guys would have seen I Mean he's obviously done a lot of cool stuff, you know Much before a lot of other folks, but he also continues to lead a lot of interesting stuff right from whatever is happening right now terms of bleeding at stuff and I think having someone kind of open the talk For the day helps people to think about scaling in a different way You don't necessarily have to think of scaling Processes but scaling other kinds of things and I think you've done a wonderful job to help people understand Some of those thought processes and really appreciate that. Thanks very much