 So, my name is Natalie Holia. And I'm Joe McLean. And we work at ThoughtWorks New York. We're product and experience design consultants. ThoughtWorks is a custom software consultancy. And we help our clients with ambitious missions to make great products. For the last year and a half, we've been working with a company called Amplify Access. Amplify Access is an education technology startup in New York City. And they're using lean product development to build a tablet for K to 12 education. During today's presentation, we'll spend the first half sharing some of our experiences and some of the things we've learned. And in the second half, we'll take your questions and have a discussion. So before we start, what do we mean by lean product design? Lean principles came out of Toyota manufacturing as a way to eliminate waste. And it's an idea that's become really popular in a lot of areas. So when we're talking about developing a product, it means moving away from upfront design to continuously changing and evolving the product as it's being developed. We do this through a process of prototyping, testing and feedback and take somewhat of a scientific approach. We come up with a hypothesis of what might provide value to the customer. Then we do user research and testing to validate if it actually does or not. And only if something's valuable do we continue to spend time and effort in developing those features. So it's a really flexible approach. It allows you to pivot your product quickly based on learnings and arrive at a more valuable product faster. Lean is a really exciting way to build a product, but it definitely comes with its challenges. One of the things that we've seen is that the challenges to lean product development change depending on your team's size and the maturity of your product. When you're starting out, if you have a small team or a relatively undeveloped product, your challenges are largely around finding the right starting point, finding ways to move really quickly and have an efficient process and make decisions effectively in an environment where there's a lot of uncertainty and a lot of unknowns. You may not be totally sure what the endpoint of your product looks like. However, as your team becomes larger and you have more of an investment in your product, there's more pressure to plan, especially if you have a large development team that you have to resist. There's a lot of potential features and a lot of different directions you could branch into and so you really have to be careful about where you invest. Because you have an investment in your existing product, it can also make changing more difficult. It becomes harder to pivot. And also you have many more decision makers. So making decisions really quickly and efficiently can become more challenging. In our time at Amplify, we have seen them grow from a very small organization to a much larger organization. And so we've experienced a lot of these challenges firsthand. We'd like to talk about some of the different tools and processes that we used at different points to address these challenges. So Amplify's first approach thought works with an ambitious mission to change education in schools across America. They had a huge vision for their product, this tablet for education. And we knew that we couldn't build it all at once. It was going to take quite some time. However, Amplify was really committed to building a product that real teachers and real students would use in real classrooms. So in order to do that, they wanted to get the tablet out early into classrooms and pilot it and get feedback from exactly those people to ensure that it would be successful. So our challenge was we needed to find a starting point and move fast to get the tablets into classrooms early well before commercial launch so we could start getting feedback and learning and improving. So when we were a small team and we were just getting started, we really had to focus on finding the right starting point. So we looked across all the things that we hoped the product would do, all of the areas that we hoped it would provide value and tried to identify a solid foundation for moving forward. So we really decided at the beginning to focus on just facilitating the communication between teachers and students. We thought by focusing on this value we would give ourselves a foundation both from a product perspective and from a technical perspective that we could build forward from and expand into the other areas that we hoped our product would provide value. So once we had our starting point, it really became about finding ways to move as quickly and efficiently as possible to provide value. So a lot of our practices that we enabled were about getting that speed and efficiency. We did a lot of sketch to code and to design dev pairing. So rather than building wire frames we just sketched things out and in many cases had the designers sitting right next to the developers as we built functionality. This accomplished a few things. First of all it helped us resist the temptation to plan too much up front and do a lot of design. But it also meant that when we encountered things that were maybe more difficult to develop than we anticipated we could make changes on the fly and not have developers wasting time trying to implement something that was maybe harder than we realized when we designed it to actually implement. A big part of that was really tight collaboration between all the different roles on our team. So this is a picture of a developer talking with a QA and a product person and a BA and the designers all working together to try to figure out how we were going to approach building a particular functionality. This really helped us avoid waiting for decisions to come down the pipe from different groups. We did a lot of this stuff in flow as we went. Finally we had a design wall so we took all the things that were going on in the design team all of our different wire frames our sketches and even our visual design and printed everything out and put it up on the wall. This was really helpful not just for facilitating design critique but also for just showing the entire team what was going on in the design process. This was really important especially early on because things were changing really rapidly. So having that visibility for the entire team was really helpful and it allowed us to see kind of the entire scope of what we were working on. So all of those things were designed to work together and they really helped us move really quickly and kind of the net result of all that stuff was being able to get a product out to market in less than six months. So we had a product in schools and we were getting feedback and learning. So Amplify began as a startup with this very small single team that were able to work closely together. But obviously as we grew we reached a certain point where that was no longer possible. We began to have all these different moving parts which made communication and staying in sync really hard. However our product was still young. We still wanted to make a lot of changes. We were getting a lot of feedback. We were pivoting and so to keep the product consistent and coherent it became a worthwhile investment in our time to make some of our tools more efficient to allow us to continue to make changes quickly. So one of the things that we did straight away was change to a flat UI. Instead of having designers slicing assets so we could have rounded corners and shaded backgrounds we changed our visual design so that it looked similar to this so that developers could change the size, the color, the shape, all the UI elements they could make them directly in the code. Flat UIs become really popular recently and this is exactly why. It's not just a fad it actually makes designing and developing products much faster. The other thing that we did was use style tiles of our flat UI. So instead of creating a visual comp for every single wireframe for a new feature that was being built instead the designers pulled together these boards which showed all the common UI elements and developers could refer to the wireframes refer to this style board and start working in the code straight from there. It allowed us for a very light touch to have a consistent level of visual design. Another thing that we did was actually make our own custom icon font. So again moving away from designers creating images which had to be changed every time we wanted to make a change in the software they would instead once off create a scalable vector and then we pulled all those together into an icon font which developers were then able to style those icons again the size, the color, the shape everything in the code which made it much faster to make those changes. We started using a live style guide as well. Because things were changing so rapidly and because things were new if we had a new UI feature that we wanted to build it may or may not be in the code already. So developers could refer to this live style guide and see has it already has it the visual design already been done. If so they could reuse that and keep that consistency and if not they would add it. And it really just saved us from creating that same orange button six times in six different places. Finally we pulled that together into a style gem. So creating a library where we could use those styles across products. We started to instead of having one product we had two and we wanted to share that visual design across the products again very easy to stay in sync make changes. So all of these things of making our tools more lean it was an effort to keep making changes really easy so that we could keep moving at that rapid pace keep learning and stay lean. As the team grew though it became very large we became a team of 50 people 60 people and even beyond now and we were distributed across the United States across Brazil and India it became more difficult to make decisions like we showed you before with everyone kind of huddled around a desk and collaborating together. It really became unrealistic when we reached this scale. So we needed to rethink the way that our organization was actually structured. So we moved away from having larger product or feature group oriented teams and split up into smaller empowered goal oriented teams. The teams were still united by a common purpose the kind of the company vision for education but they were able to set their own objectives and frame their own hypotheses as to how we could provide value. This allowed them to make decisions independently. We were still unified by our common purpose but the day-to-day decision making at the team level was able to be much more flexible. Also to that end we took our design group which as we had grown has started to kind of work as one coherent group and we split it up. We took a visual designer and a user experience designer and we embedded one each and all of these different goal oriented teams. We still brought everybody together frequently to talk about kind of the decisions we were making across the product and what our interaction patterns were going to be and also whenever we needed to do visioning for new feature areas. But for the day-to-day decision making the designers were embedded in the teams and were really able to help us get back to kind of that collaborative model of everybody working together that we had had at the beginning of the project. Also at this point we really started focusing a lot more on problem definition. So before we would do a design workshop we would have a problem definition workshop and so we would go through a process of really trying to understand the problem that we were trying to solve for our users and then we would use that to kind of frame up and validate all the different solutions we were creating. In a way this acted as a filter so we could understand which of our concepts were more effectively addressing the problem statement that we had set out to accomplish. All of these things kind of worked together to start to build in a lean mindset into our organization to move us away from just thinking about efficient processes even though that was still important to really focusing on value and to measuring everything that we did against the value that we anticipated to provide and validating that with metrics. So we've really been on a journey over the past year and a half with this team from doing lean to being lean. It's much easier to start doing lean processes using lean tools on the ground than it is to change the organizational approach to product strategy. However, as an organization grows and becomes more complex it's ever harder challenge to stay lean and avoid this upfront planning and design. And so we were continuously challenged to evolve our thinking and move this lean thinking earlier up in the decision making process to make us more effective at eliminating waste and creating value. It's not to say obviously that we stopped using these processes and tools but rather we built on top of this and the way that we did lean evolved as our practices matured. We moved from trying to build features efficiently to trying to build the right features. Lean product design is a great approach to developing a product but it won't work unless you kind of absorb some key organizational values. Even if you're an expert in lean and you're really excited about bringing it into your organization it's not always possible to jump straight into using lean to define your product. You may need to work slowly and incrementally to build trust in this way. It's scary not knowing up front what the endpoint of your product is necessarily going to be and you have to have some trust in the process to bring you to a good place. You have to have humility you have to accept that you're not going to have all the answers right away. It's not about being right or wrong it's about taking a scientific approach and always be willing to validate or invalidate an idea. And you can't do that if you're too attached to any specific concept or decision so you have to have humility. Finally you have to have courage you have to be willing to start anywhere you have to not be afraid of making mistakes you have to be willing to make a mistake if it's in service of doing the right thing. So we started off by telling you about amplifiers ambitious vision. They were thinking big about a new kind of product to transform the way that we teach and learn in schools. We had to take a step back from that vision and find a way to start small and move fast. And lean product design was a technique that enabled us to do that with great success. Taking a lean approach to product design is really well suited to situations when you're trying to develop a new kind of product or enter a new market and you have a lot of unknowns. It helps you to learn quickly about all of those things that you couldn't possibly have all the answers for up front. It helps you encounter all of those unexpected external factors that were always going to come into play earlier rather than later. And so from this vital feedback you can make changes early on in the product development and for a new product or a new market that can be really key to its success. So we hope that this has given you an entry point to using lean product design in your organization in some shape or form. Whatever phase of maturity it's at and whether it's big or small. Thanks. We're sure you have a lot of questions so as you can see we've left a lot of time. We'd like to use the second half to have kind of an open discussion and I open it up to you guys. Do we have the portable mic? I think there's a... Is there a mic on it? Oh, you have it. I guess if you could just give it out to whoever has a question that'd be great. How did the culture in the system that you're trying to build affect the hiring decisions as you're building the team? Do you know anything about how that process went in picking the right people that could rapidly adapt to the way you were doing things? Yeah. It's a big part of thought works the way that we hire people. We spend a lot of time and effort trying to find people with the right cultural fit and who are kind of open to working in these ways. So we definitely help try to encourage those principles and philosophies with amplifiers. They were hiring as well. There's also an aspect of, especially as a design team, I think we really tried to focus on people who could think in terms of simple and elegant solutions. That's something we really very specifically targeted when we were looking for people in interaction design especially. When we were evaluating portfolios that was definitely something that we had in mind. Is there effectiveness of reducing a complicated problem down to a simple solution? When you were changing the culture, what were some of the hurdles ... Sorry. When you were changing the culture, what were some of the hurdles and challenges that you had to overcome to get buy-in with the culture and flow changes? Yeah. That honestly was a huge part of what we've been doing. Probably over the last six months. It does take a long time. A lot of it was around being clear on spending very hard to talk about a product or any use of value that you're trying to ... One of the biggest shifts I saw is when we really started focusing on the problem definition workshops. One of our colleagues, Paul Sullivan, has really spent a lot of time thinking about how to frame up a problem workshop. Actually, if you're interested in getting more details, I'd be happy to talk to you about it afterwards. We really discovered that when we were really diligent about defining the problem that we expected to solve before we even went into the design process at all, it really changed the entire tone of the conversation, and it became less about people's opinions and more about how we were going to get more insight into the value area that we were trying to address. I know it sounds like a really simple thing, but even just going through the discipline as a team of starting by framing the problem and writing it out and putting it up on the wall and using that to define and kind of filter through the rest of the design workshop was extremely effective for us and changing the tone of the conversation around what we were talking about. Yeah, and also just that shift in thinking about the product owner not having to always have all the answers to have that pressure of always being right. That shift in mindset really allowed the rest of the team to contribute and for us to collaboratively try to arrive at the best solution. It also, I think, provided a path for the developers to be a lot more involved, when you're having the conversation on the level of addressing problems, it kind of opens up the floor for people to present different kinds of ideas. Like Natalie said, taking the pressure off the product owner or the design team to be the generators of these experiments. Two questions. One is that the level of six and the other one is the level of 60. So when you're at six, you mentioned scientific approach. In lieu of any statistically significant data, what kind of scientific approach would you use, right? And then at the level of 60, making very quick decisions is very important. And how do you manage that with so many strong-willed and knowledgeable people on the team? Who does this decision-making? What process do you use, if any? But at the small level? Sorry. Here, we'll take that mic. So your first question was about the jinx. When we're small, how do we validate without? Yeah, so I'd say the approach is scientific, but the conclusions are maybe not, right? Because it's about human interaction with software. Everyone's different. One of the things that we tried to do early on was build a better understanding of teachers and students. So we did a lot of user research, going out and talking to them, before we even had a product. So not testing out our product with them, but actually finding out, what are some of their day-to-day pain points and challenges and issues that they have? What are the ways that they think about teaching or learning? What gets kids excited by school, or the teachers or not? So really trying to gain that understanding so that we'd be able to better make decisions from that. And again, taking our product into the schools early and observing people using it, observing teachers or students struggling with particular things. So that's how we were able to get feedback quickly, even when we're small. So yeah, it wasn't, you know, about 50% of these people in this A-B test, it was very much, let's go and talk to the real people that are using it. The second question. Yeah, just from, I guess, like a kind of nuts and bolts practical approach, one of the things that kind of guided our research at that phase is, we would just talk to people until we started hearing the same things again and again. And that to us, I think was the indication that we had talked to enough people a lot of the time. We were at that, especially at that point in the product development, we weren't looking for statistically significant information. We were looking for a lot more qualitative data. We didn't really talk about research very much, because that's a whole other story and how that has changed and grown over the organization. I think we're more data driven now. It was important at the beginning to be more qualitative because as you identified, we didn't have enough people to have that kind of statistical insight at that point in the product. Oh yeah, decision-making is really a big second part of the question. Yeah, I'll repeat it. Yeah. So I believe the second part of your question was with a lot of strong-willed people who have well-informed opinions, as the organization gets larger, how do you manage decision-making? Roughly, is that good? How would you start? Again, I think one of the techniques that really helped us with that was the problem definition statement and having that focus on, it's not about individuals asserting their opinions and their ideas were better than other people, but it was about looking at what was the feedback that we were getting from our researchers at the schools and from the teachers that we were bringing in and seeing how people were using the software and trying to take that approach of moving away, detaching your individuals and their association and their fondness of their own ideas, I guess. That exact problem, though, was also part of what drove us to break into much smaller teams. I mean, it was really a direct reaction to that issue. We had started a sense that, especially in conversations about architecture and technical decisions, there were just so many decision-makers that it was hard to get conclusive results, but this is exactly what you would expect. So having teams have more specific focuses within the product and kind of be able to make decisions a little bit more independently really helped us with that. We definitely needed to make an investment in infrastructure to enable that. Obviously, like, there need to be certain systems in place for teams to be able to work independently. So a lot of that, like, infrastructure and plumbing between groups was important to figure out at that phase, which is, again, kind of a larger story. But that was a big part of why we made that decision at that point in time. And again, I guess, the goal-oriented teams was also an attempt at that moving back to the situation where a team can work really tightly and have, you know, very fast real-time feedback with each other and be empowered to know that the decisions that they're making and what they're choosing to do are right because they can measure them against these known goals. Whereas if it's building out a product backlog, you know, it can take a long time for kind of product vision to come down through the organization. I'm curious to hear a little bit about the makeup of the small teams that you created, who, you know, how many engineers, who's a part of the team, what's the balance that you found that works well? Sure. So it's a little bit different depending on the team. We have some kind of back-end infrastructure teams that, as you would expect, are a little more engineer-focused. But generally speaking, I would say that, like, for instance, the two teams that I work on are both six to seven developers, one to two QAs, a dedicated product owner for the team, and then, as I mentioned, a user experience designer and a visual designer. So I guess, like, we've been hovering around a three-to-one developer-to-designer ratio for a while, and I think we've found that that works pretty well for us. Anybody else? I'd be curious to hear how do you deal with the earlier phase of design being account for things that might happen later, and obviously, Leen, you don't want to do that, but then, as new patterns emerge and new needs emerge on common components, how do you manage moving those forward so that you don't have a product that has dissimilar behaviors across what should be common components? The question was around how to keep the product coherent, or it was around... One of the things that, yeah, we noticed early on was, as we were very first beginning to build this product, we had no UI patterns, right? So as time went on, we really felt a need to kind of create these patterns and make sure that we were using something, we were using it elsewhere. Initially, what we went with was the things that were kind of faster and easier to develop, right? Because we didn't know if we were going to be throwing out that feature later anyway. So we just wanted to kind of get something there that worked, that was usable but maybe not the greatest thing ever, kind of. And so once we kind of got the product to a certain point where we had examples of UI patterns, we were able to see, you know, through observation in the field which ones worked well or not and ones that worked really well, we then took those and we made those UI patterns and so our design wall was a big part of that, was having, you know, our designers kind of huddle around the wall and talk about, you know, I noticed that you designed this feature but it's really similar to this and this one here. How about we use the same pattern? So I think that the design wall sort of facilitated that cross-team collaboration and consistency. Another thing we did is, you know, there's a lot of conversation around tech debt, the idea that over just the natural time of building a product, you wind up with a lot of technical tasks that kind of need to be done to have a solid infrastructure that is good for the long-term. We found that we were building up UX debt as well and one of the strategies we employed is having a dedicated designer, developer pair every once in a while just to kind of go through the entire platform and make sure that everything was consistent again. And one of the strategies we employed is just kind of like, even though we would still write stories for that and track the work, we would kind of track that outside of the progress we were measuring and the rest of our teams and consider it just like you'd consider it to be almost like a tech task to pay down that tech debt, we would pay down our UX debt from time to time as well. Yeah, I'll just quickly add on that as well that that actually worked much better to do it afterwards as a UX debt because when we tried to bring it up front and bring that consistency beforehand, it just like suddenly, you know, slowed down the speed of the team greatly because one thing would change and it would change a hundred other things. So it really was much better than before, you know, particular launches that we went through and did that quick cleanup. There was a great comment in the keynote about engineers planning for reuse before they plan for use. And I think like the design equivalent of that I think is planning for consistency before you have even figured out what you need. So we, well, you definitely need to do a certain amount of forward planning. It's a lot easier to figure out how consistency should work in retrospect than trying to anticipate. At least that's certainly been our experience. I think we're out of time. I'm afraid you are out of time, but thank you so much, Joe and Natalie.