 All right, looks like it is time and I've got the first item Hey, somebody else did add a second item. That's good. I was gonna be only speaking today So it's a little too bad that you know is not gonna make it but based on Well, this was an item that rolled over from last week But it came up a couple of times and it's actually been coming up a lot lately the idea of product discovery, you know versus Whatever it is we call what we do today I've been thinking about how to actually sort of approach this. I was even tempted to Sort of just organize a workshop or something where I can just sort of describe what it is that I mean by product discovery at least I've also been tempted to Just sort of do an experiment with a team or two to do a product discovery sprint or more than one or whatever And just sort of slowly introduce it But what I'll do today is just talk about it meta a little bit what it is Why I think it's important for us to understand what product discovery means and why we should consider doing an experiment and See what you all think About about it and whether we want to go forward or not So I did include a link to a Bad blog post I wrote at some point It's not very long or detailed But it's the extent of my notes on the subject unfortunately This is actually derived very much from work I did at Heroku just to be up front and how we did product discovery there and it's a process that we Discovered so to speak sort of accidentally from Actually originally it was derived from an interview at Heroku. We did week-long starter projects and I basically wanted to give a candidate a Yes, we discovered discovery The reason I say that is because it also looks very similar to lean startup Just product discovery, but we didn't know that until after I read the book later, you know months later But to be fair a lot of it was also probably inspired by one of the founders of Heroku Who had been mentoring me on stuff and he had gone through YC? So I don't know there's probably some relationship between YC and start up and so it's all related But the point is it sounds a lot like lean startup, but I didn't actually read the book yet I have since and it sounds a lot like an actor in retrospect. Anyway The funny thing is that originally the reason I came over this really compressed kind of schedule is because I wanted to Come up with something for a candidate where the work was like nearly guaranteed to succeed And I was just interviewing the candidate like on what it was like to work with them And so I came up with this really aggressive thing like let's make a tiny tiny little sliver of a feature That we can guarantee we will ship within a week in fact to make sure we guarantee we ship it Let's guarantee we ship it in two days and we got the rest of the week in case we screw up razzle radically We can still fix it in three days left It turned out though that we actually shipped an amazing amount of stuff in a week And it was awesome and it was an eye-opening experience for me Which is one of the reasons I want to share it with you guys and girls. I should stop saying guys I know you say you don't but I really should stop it anyway Anyway, so so the idea is The process we stumbled on and I should also just be clear too So we stumbled on it during this one sort of project and then slowly it sort of calcified Into this process that I'm going to describe and it turned out that like anytime we ended up challenging the process We we basically said no this process works exactly as is like if we if there was a challenge It was an interpretation of how we were supposed to do things But it was just like the process actually really worked Which is also especially funny because we started with this whole thing because our old process wasn't working and we Throughout process entirely and so all of this was generated from scratch from a couple of people who basically said Scrum sucks all this like Planning poker and all these other things are just a total waste of time Let's throw out all process all together and let's just talk And then we ended up coming up with this very rigid process Which is just ironic if nothing else But it really really works because it it was based on better principles is the net of it I'm not saying scrum wasn't based on better principles, but it wasn't looking for us so also to give a little bit of just more Whatever emphasis on this it was So I was literally working with a team of I don't remember how many people it was Just say five people or something like that Maybe more eight people and we were doing scrum and every Sprint, you know, we do a weekly sprint every sprint We try to squeeze out like a few percentages improvements in our process. The things were still just horribly bad We spent a whole bunch of time like okay Let's iterate on making sure that we've got really good high fidelity mock-ups so the engineers have something better to work with than the low fidelity mock-ups and then We'd still have problems where the engineers would implement something wrong whatever so then we got even higher fidelity mock-ups Then we got like let's annotate the mock-ups with like exact dimensions of things or let's talk about the things We care about because back then we were using Photoshop too And so like you have gradients that didn't exactly translate from Photoshop to CSS And so some people would always interpret things a little bit differently And one of the struggles was the designer was like I want to just say like I don't really care about this I wanted to look like this don't follow my Photoshop mock-up exactly because it's not CSS But this one I do care about like this really has to be five pixels. Don't make it three or seven It's gonna be five and so we ended up like literally mocking this up like an engineering architecture diagram And we have this really really rigid thing so product and design would work really really closely And then we'd still throw it over the fence to engineering And so the best we got to was we would do that we'd spend a week Mocking up something and then we'd give engineering two weeks before they'd pick it up And that was based on a recommendation from Marty Kagan's book And then engineering would pick it up and then they'd work on it for a week And then they'd show it at the end of the week on demo day And then we'd see it and then we like okay well that looks pretty good And we come up with some immediate things but then after we go to use it We come up with more so we could come up with all the bug fixes of the next week We'd feature assurance it as product and then the following week engineering would pick up all those bugs and then deliver it So you know net we're talking about like I don't five weeks of time calendar time with with a pretty large team When we ended up doing this product discovery process we basically delivered that same kind of thing in one week with half the people and So I roughly speaking say that this is like a 10x improvement not in terms of like code lines of code But in terms of wall clock time of idea of a feature to delivery of the feature One week time also the quality was much better because we didn't have these stupid misinterpretations between engineering and design And worse is the feature assurance part like a lot of times again You you design a static mock-up with like the ideal lore mipsum lines of you know text and you say okay Let's put three lines of text and it turns out that in fact people don't use three lines of lore mipsum They use weird bullets and they use 20 lines of text and they use all these other things and like Designing with real data is really really hard. So a key thing about this is like within 24 hours We're actually working with real data We can see whether it really works or not and we can iterate the next day or the next four hours In fact to make changes based on that. Also, I banned lore mipsum. We would never use lore mipsum again Always used at least realistic data, but still that wasn't enough. You had to have actual real data So yeah, all sorts of things learned. I'm gonna try and speed this up just a little bit I know we've got an hour, but I don't want to use up all of it So anyway the idea here key things are Product did not fully spec things out before passing off to the team Things were just spec enough to know that we could deliver in a week that it was viable to deliver in a week at least and That we had at least some belief that it was an important feature and we had a hypothesis You know that if we do x then this will improve something Um, what we will then do is take that sort of nub of a topic and we do this think big session and The think big session would literally be a couple of hours long and we really open it up to brainstorming but also There's some subtleties here that it's really hard to express but One of the big things but think big and in the brainstorming in general is not the actual brainstorming per se But it's but getting the old team on the same page so that all the ideas are now shared instead of having like Three different people have different parts of the stuff in their head and talking about it on an issue where they only talk about like some small Intersection but not really actually being on the same page and you do the think big session and Invariably we found just people were on the same page afterwards product and engineering design and everybody Knew basically the entire scope of what we could possibly be teaching or talking about here Also an important thing about the think big was totally disregard Limitations like whether something was feasible whether you could do it in a week or not Even though of course the ultimate goal is to deliver in a week in the think big session It was like let's radically rethink everything we could possibly rethink on this topic I know that's kind of broad and scary, but again, it's only two hours. So, you know But it helped you then open up like ideas that you had never thought about before like I don't like What if we didn't just do this, but what if we threw that whole thing out and did something else instead? And sometimes you ended up being like, yeah, actually that sounds like a really good point This idea that we thought was really worthwhile actually isn't let's do something else It doesn't happen very often. Hopefully that you know it sometimes does but more often It's just it gives you a bigger perspective of what's going on again being on the same page Anyway, then a really crucial part was the think small and this was again The accidental thing because I was just trying to make sure the starter project worked But we really said like what can you ship in 24 hours and practice we've hardly ever shipped in 24 hours So it's really more like 48 hours Although at some points we got really good and got it down to 24 hours But the net idea is like, you know, you starting on a Monday Let's say by Wednesday, you've got the first iteration ready by lunch, you know Wednesday to lunch time Not rent Lindsay end of day and You've got the first iteration and it's gonna be ugly and it's gonna be Disappointing and it's gonna be whatever but it's gonna functionally be there. I may as well just go along with the story now This first time we did this it was we had the hiraku status site told us told the customers whatever Whether or a hiraku was experiencing an issue or not And it was all manually entry done kind of stuff. We had completely revved that site. It's now dead So if you look at it, it's not that site, but it lasted for a good several years But in the top right-hand corner was a little place where we wanted you back to While was there was a place to subscribe to stuff But before that existed there was no place to subscribe the idea was like we said, okay if A hiraku is down like it's great that I can discover that there's a problem and then go and check the status site But why don't you just tell me that the damn things down, you know Or even better, you know, like during the command line tell me if things are out or whatever But just tell me proactively that something's down instead of morning me Especially since you don't necessarily discover things until your customers are complaining about stuff And so this was back before any of this stuff was automated So we said let's let people subscribe to notifications on the status site and that was it And I'll actually to flesh out the story a little bit more. I went in as product management thinking a Couple of things were key. I thought well people are gonna want to be able to Subscribe by email, but they're probably also really gonna want to get text messages because we're not sitting by the computer all the time and I want to be alerted in the middle of the night if my Critically, you know, awesome important website is about to go to the VCs and her who's down I don't need to know about it. So be able to text me Maybe even phone me. I don't know but at least somehow communicate on my device And then I thought as a user you already have my login So don't make me create a new account don't make me log in again Or maybe make you log in but then get my email address from my login I thought that was a critical piece. Those are the only things that I really thought about Beyond that I said, okay. I don't know what else there is Awesome thing is during the think small session one of the developers was like Well, you know, make me log in first and everything else. That's actually really hard What if we just make you type in your email address again? And I was like, well, that makes no sense. You've already know my email But it's like but if I make you log in you're gonna have to type in your email address Anyway to log in this way where at least not asking you to type in a password Let's just type in your email address and then it's like, well, how do you unsubscribe? How do I go to my project? You know my settings to unsubscribe? It's like, well, you get the email click on the link to unsubscribe You don't need an account management system. You just need an email and an unsubscribe mechanism and that simplification was awesome came from a developer Scoped it down, which I will honestly say in the entire history of heroku We never tied the login systems together this thing that I as product management thought was critical was not and in fact made it Awesome to not do it Because of course there's this obvious downside of like if heroku is down How the hell do you log in to go and do anything on the status site? That's kind of dumb like decoupling these things makes a lot of sense So anyway, just a good little lesson learned there about like this is why this stuff is important and why it's important Not just that product involved because product can be done sometimes and think that something's important when it's not So anyway, so think big think small you get it down to what can you ship in the next 24 hours? and then you actually go and work on it and And again the first version we had showed it to a bunch of people including one of the founders of heroku They just you know kind of like that whatever fine Not the experience we were looking for we're hoping to be like oh, yeah, this is cool I can see that it's ugly, but I can see the potentials like no whatever just didn't care But then we iterated on it and at some point, you know, we started off with roughly speaking rails scaffolding I think it was actually real scaffolding. I'm sure it used our own C2S But it's still roughly speaking scaffolding and then we said, okay This you know, it's got to be integrated a little bit better And so we ended up putting it in the toolbar not in the tool, but there was no problem but we put it in the top right hand corner and so there's a little subscribe link and We made it all a single form pop up whatever so a bunch of GUI optimizations, whatever And we did a whole bunch of other stuff too So like we actually implemented an API on the back end and we also implemented like, you know We had a trigger a Twitter feed with this kind of thing And so we said like when you go to subscribe, we gave you all these mechanisms We gave you you could scribe by subscribe email. You can subscribe by text message. You could subscribe by You know, here's the API so you can do it or how you want with it Here's the Twitter feed and then there's some there's five different things you could do So it was a little bit of scope creep, but it was awesome because we got to it We said what we actually can do these things we scrambled and made sure that all these things actually work and then So we we did that and basically iterated by the next day by Thursday We had this thing that was pretty complete and pretty polished and then by the time I showed the founders then They're like, holy crap. This is awesome like this is pretty much everything we believe in and the reason he said that was because we made a bunch of interesting other choices like When you subscribe by email You would get all the updates on a message So you'd get an alert and then you'd also get when you know the alert was updated and when it was closed anything else But if you got a text message, we only alerted you when the thing started because again, like, you know If you're woken up in the middle of the night, it's awesome that you now know who goes down But do you really need to know that? Oh, we're still working on it like every 10 minutes You stay awake for that We figured if you've got the text message if you really wanted to you would then go to your computer Then follow along by email But but part of the thing is that we have these huge debates about Do we need to let people choose because like maybe I do want to know I want to know when it's closed I want to know all this. Do we need to give you options and choices and we said screw that no options and choices Let's just think really hard But what the right answer is and we will give you that she had an option of course of email or text or but that's it And then we made the rest of the choices for you It's one of the reasons that the founders really loved it because it was like we made it really really simple So anyway, we iterated then by you know day four again Thursday We had this we actually launched it to our public or our beta list not a public or private by little beta list With metrics because we said okay What a success look like and that's thing I didn't really mention but in the very first iteration We even said what a success looked like before we even started coding and and we came up with what were metrics were And we said well if 10% of the beta lists Subscribes and keeps on there and doesn't opt out after they get the first a few alerts then that success So anyway Thursday afternoon we had launched it and so we actually had numbers within four hours We knew how many people subscribed and we got a bunch of feedback and everything else So by Friday by the time, you know the starter project was basically done at noon We already had measured response rates. We knew that 20% of the people had actually signed up and stayed there That this was actually valuable We had the Antic data sort of responses of people saying they loved it But we had the numbers to back it up and we had we're fully done Now there were a few little bugs we fixed over the next week or two, but basically a shift entire feature From start to end in a week The reason I really love this and the reason I think we should start caring about this is that This is really valuable for high uncertainty projects We didn't know for sure what You know subscriptions should look like we didn't know what many of our features at or should look like And we really wanted to get feedback from the beta list early on too And so part of that was a you know a two-way conversation with the beta folks Whereas currently a lot of things we do is you know, we we work on something for a month and then we ship it And then actually in fact we should feature freeze and then wait several weeks before we get any feedback about it But by then we're already working on the next iteration of it And so we better be right in some level I've always said this before though So speed does sort of solve all our problems And the thing that really I was going for it is that we ship quickly And so yeah, we ship it a month and even if we're totally wrong Who cares it's just the month of effort we throw away More realistically most of the time we're right and we just need to tweak it In which case then we'll tweak it really quickly It still does mean though that calendar time it can take three months before we actually have a polished version of something and And boy, I'd really love to get that cycle time down, right? This is part of what we pitch right we pitch our cycle time is meaning something And it's awesome. We ship in three months And it's awesome that we ship the future quantity of things we ship So the bandwidth is high, but the latency so to speak is is high as well, which is not a good thing And product discovery can really get that latency down It can mean that we ship things quicker and with higher uncertainty we can discover That's the whole reason we call product discovery we can discover what the feature is supposed to be The reason that's all came up is because we were talking about sort of like do we design things ahead of time Do we have things fully specced out before we hand it off to engineering and that works for some types of problems And it's really bad for some other types of problems If you just sit there and design a static Mock up and and say great go and ship it, but then it turns out to just not really satisfy anybody's needs Didn't really done anything or at least it's two months before you Realize how you've made a mistake and then you can go forward um Bringing a little bit of lean sort of thought on this and I'll try and wrap this up quickly is um The difference for me between sort of the way product discovery works and the way that scrum works Is like scrum can tolerate uh being wrong You know the idea is you move quickly And you can react to your customer telling you how you made mistakes, etc Whereas lean assumes you are wrong. You just don't know where you're wrong And so you want to get it out as quickly as possible so you can learn what you make your mistakes on And that that's not a problem that you made mistakes. It's you don't know this is a high uncertainty situation so get something out there in front of Target users beta users whatever as quickly as possible so that you can find out where you are wrong And I think that's In a lot of ways different from our approach with get lab 2 is we we're tolerant of being wrong We have this issue tracker. We have public, you know contributions. We have all this kind of stuff But still we're basically marching forward assuming we are right We're we're doing an issue and we have the next issue and we have the next issue planned out And we're assuming that less unless I mean we used to assign five issues worth and we know that that's not right But still we're basically like cross-project pipelines are doing right now We're basically planned three releases worth And we're not listening to any feedback in the media in the in the middle of that or any part of it really We're still just assuming we're right and we're delivering it and we're tolerant of You know making, you know, whatever mistakes And I think the big difference again with product discovery is assuming we are wrong We want to get there as quickly as possible We want to learn also does tend to mean that we have a polished a more polished product more quickly So anyway, that is um, I think enough of my time You must see do you have something to say? Me? Oh, sorry Sorry, it's just my uh, my knees entering here, sorry Okay, I did it. So mark just a question for you, uh about your story Were you and your team just focusing on that specific Uh feature during that week or so of work or are you also doing something? Let's see else during that That is an excellent question And that is probably the biggest downside of this process. Actually, there's two big downsides One is that it is basically 100 focus like the product manager I would be doing other things but the developers the designers They're working 100 on this one thing Um, which is great But means you're doing no bug fixes. You're doing no maintenance. You're doing nothing else, whatever um The so marty kagan who a lot of stuff was inspired by and by the way, so he's got this really great book Um, but he's got a better blog because his book is old and basically he doesn't Believe in everything he wrote in his book anymore Um, but he's actually really a smart guy and his blog post is right on, you know, right on target But if you read the book like I did I'm thinking well, this makes no sense. This is dumb It is dumb. His new stuff's awesome. Uh, anyway, um, he in his book though proposed Basically having two teams like you've got your product discovery team and that's mostly about product and and design And you'd have like one or two engineers allocated to doing the discovery And then when that discovery is done, you'd pass it off To the delivery team and so you'd have a discovery team and a delivery team And then of course the delivery team can be working on anything else in the meantime um I personally never did a discovery team and a delivery team separately Um, so I'm biased here, but I don't I'm not even sure I think that's a good idea His arguments were things like well if you do discovery, it's still just proof of concept You'll move faster if you're not trying to write production code, but then pass it off to a delivery team that writes good production code Um, but from my perspective, I just had one team that wrote good enough code Right away, and we would ship it and maybe we refactor it later on um, but uh But I don't know I didn't do that But the other aspect is that potentially if you had that delivery team then that delivery team can be working on bugs Instead what we would do is every once in a while We would not do a discovery sprint and we would do a bunch of bug fixing sprints or whatever Or what you'd do is you'd wrap up a bunch of bug fixing into some other topic that then you would do a discovery sprint thought or something but also just say in practice, um, we just shipped a lot and this team and um And it didn't in practice be a problem, but I know that we Would do things differently here and and we might really need more maintenance and whatever else The other big drownside to this is the synchronous communication Um, we had at least four hours of overlap with every member of the team In fact, in the early days it was you know, seven or eight hours We had everybody was in the same building same physical whatever we would discuss something in the morning You know at our sort of stand-upy kind of thing We would sync back up four hours later and then we sync up again four hours later And um, and there was a lot of communication as a product manager Uh, it took at least half of my time just working with the discovery stuff So massive high bandwidth requirements from the product The the the positive side of course is that and this is where a lot of the stuff was um inspired by Was that like, you know engineering would be like, oh, I tried to implement this thing and it's like It's a problem. What do we want to do here or this spec was unclear or whatever And so instead of just getting stuck putting an issue, you know Comment on an issue and then going off and doing something else you would say, okay Get the product manager and the designer together right then and we would make a decision within five minutes That was actually before we had the product discovery sprint. That was the thing we Precursor to all this was if you've got a problem basically pull the red line on the Japanese Kanban kind of thing Production stops you solve the problem Then move on And that actually was incredibly powerful. Frankly. It meant that our calendar time gets really reduced I really don't know how the hell that's going to work in an asynchronous global distributed world Which is the big reason that I haven't been pushing this more I think it can and it might mean that you do something like you've got your delivery team That does the normal stuff asynchronous and then you've got a different team that at least has four hours of overlap It might mean that you can't have a global distributed, but you can have Continentally distributed. I don't know throwing in this out there Other comments mic. I see you've got some comments. Go ahead. Yeah, so you actually touched on some of them whilst you're talking now Um, like a like I was saying yes, please I'm kind of used to working in an environment like this where you have much faster feedback loops. Um, and this is Sort of proving to be somewhat painful for me um However, I think If this is like Everest Then we're almost in like the bottom of the kimberley hall from a Like cultural perspectives to get there and I think there's like some very big cultural things that we that were not There yet, you know number one is just This ability to ship something in days not effectively six weeks, right? Um, and that goes right the way through to production engineering and you know, we actually do ship things in days when it comes to security releases and like all of these things so we do do it But it's it seems to be the exception of the norm and so Like number one, we need to figure out how to change that Um, two, you've already but I would push that just to add to that Sorry, we should also be thinking more in terms of like shipping in hours or even minutes Like the fact that our ci takes two hours is just ridiculous. It kills a lot of this stuff So a great functioning team would be able to ship something within five minutes. Yeah, I I I agree But like I'll come to some sort of points and like thoughts and action items Maybe afterwards. Um, number one that number two, you already pointed out like less async more sink um, so we just had this example where we've put all of these pieces around licensing for like gitlab.com and EES defaults and like for the first time we actually brought Like a lot of people together in one call and actually did a synchronous call and went through like I actually took all of the flows out of the issues and put them into a doc It was like here's one doc that basically says everything and we and everyone's like, okay cool How far we will understand what's going on and it was great And I really just in general want to try and be a bit more synchronous But again, we'll come back to that one. The third one. You also pointed out was to some extent like people's bandwidth um, and You know, I think I'm sure the ux team and sarah will You know, this this would be unachievable right now to be able to dedicate to design it to this with the ratio that we have Of developers to designers, which I don't even know if anyone knows what it is, but I'm sure it's horrendous um I'm not sure that's as bad as you'd think. I mean we I've been able to you know one designer for a project is enough So the question is more like how many of these teams do you have and if you had 20 then yes You need 20 designers. Yeah. Yeah, we only have three of these We've got three designers you can throw on this. Okay. I understand that um But but I still think even now to to dedicate a designer to something like that would like the rest of the design pipeline Would just fall through the floor basically um Again, we spoke like about the ability to measure and loans a like shipping something but be Like knowing tomorrow if you ship this as a b-test or you know, if you have metrics where you can say Okay, cool. We turned this on and this happened immediately, which again, we currently don't have right now um But what I was trying to figure out is How can we try and move towards this like how can we discover? product discovery like actually discover the journey to us like, you know, are there things that we can do where guess what You know, maybe the other point I was trying to make is Are there two solves here that this thing solves like number one is the process of designing something and just Playing with it yourself with real data or like semi real data or like actually just using it interactively It is one thing versus like shipping it and learning that you were wrong or whatever and like are there ways that you know using review apps or You know something that you know, you could do something as a team and you know what get like Maybe even everyone in the team is running that release branch And they're all running it on their machine and like the designers and the developers and the product people Are able to like run that release branch on their machine And like the next day you play with it yourself and you go Like even internally because I think sometimes even internally you realize that you got you got things wrong really quickly and you can say hey like let's Let's not do this. Let's change this or whatever And so I'm trying to figure out like are there sort of two or three action items that we can maybe try and take in an experiment In like a sprint or something to try and prove out some of these things and get the benefit of some of these things And then rally people behind them because getting to this world Seems to be kind of insurmountable and scary But I think if we break it down and say, okay, let's let's practice something and like how would we do that? Yeah, and that's one of the things I've been struggling with Um, I mean so frankly, you know when I joined a year ago, whatever this was still very top of mind um, but I consciously wanted to you know, learn the get lab way of doing things and I wanted to embrace this stuff and not break stuff um and I I don't know how to introduce this slowly um, and I'm not even sure that I should because We have a different culture and we have a different way of doing things and Just forcing this on in fact could very well break things. So I've been here a year now I feel like I've got you know a pretty good handle on velocity and whatever and I can always still improve but um, I one of the things I will say front is that The biggest thing that the product discovery sprint solved for me and my team before was velocity But that's not actually the challenger. We have velocity We're shipping great stuff So I like it's like, okay. Well, if that's not what's for then why am I even considering this and for me it is stuff like wall clock time um polish and uh like You know to some extent the embarrassment of shipping RC1 or RC5 and then being like what the hell this just doesn't even work Because no product manager got a chance to test it until it got up on getlab.com Or you know, most you tested it with some fake data or something like that But even that we just don't even really I don't get to see it until it's on.com um and uh, and then just the whole customer validation cycle like how many times do we ship something and Then it's a total waste of time it's not often but it does happen occasionally and You know, I'm just gonna throw up the cycle analytics like I'm not aware of anybody using it It's got big bugs actually that I'm pretty sure stop anybody from using portions of it That we've just never even fixed but because nobody's using it like it nobody's even submitting issues to fix the issues because nobody cares Why did we ship it? Well, we shipped it because we had an idea that we wanted to fulfill and we just blindly went through and that's exactly the scenario where like We could have validated that earlier or or devaluated or whatever. You know, whatever disproved the hypothesis earlier Or we could have said well once we hit the customers Maybe we could have just twisted a little bit and it would actually been valuable at this point I genuinely don't know what it would take to make cycle analytics valuable. I believe there's some value in it But we're not actually like the execution is not it didn't deliver the value that we it didn't deliver on the hypothesis But again, most of the stuff we ship it's pretty awesome um Most of the stuff is great and most of the stuff we do get good feedback on if it's a used feature We will get feedback and we will eventually ship it and it just takes a little longer And again, if I look at wall clock time over the last year, I'm incredibly proud of what my teams have shipped I would I don't want to do anything to slow that down But I'm just imagining like okay. This is what I could do at haruka where we actually at that point in time I had a really low velocity And this resulted in a 10x improvement So what can we learn from that and like what you know? Could we squeeze out another two or five x from our teams today? Like can we raise the bar even more and deliver even faster with better results? I think we can um You get on this i'll just add because I wrote it down there as well but like a couple of the tactical things that make this viable are A cd, you know deployment structure to calm again I don't mind necessarily if we never get to ee or ce customers We've got enough representation in calm that I think we can learn that way um But I think we need feature flags and we need a beta group We need to be able to then feature flag in the beta group Which is primarily thing But also as you pointed out even just using it ourselves. Hell, we're all we all but we're all developers um And we do learn a lot by testing internally and that was one of the other things like I do believe the rhetoric about Like you've got to get customers using it. But the reality is like If you use it yourself first you can knock out 80 of the problems with it and then you know Frankly again at hiroku. We we expect like people would say My mentor would tell me like if you ship 10 things and two of them are great successes and the rest you delete That's okay. That's a good ratio one at five But in reality we would ship nine out of 10 things like only one 10th of the time did we ship something? We actually regretted shipping or that we'd want to change um We just didn't have that many failures and I'm not trying to be arrogant or whatever It's just that maybe there's a whole bunch of low hanging fruit or because we were a developer platform and we were developers We knew what people would like or I don't know whatever Maybe we just picked things that we were pretty confident in which is again one of the reasons I think that lab is doing fine like nine out of ten things or maybe even higher are in fact valuable So validating early just to see whether the idea is valuable or not. It's not actually the most important thing um And again with our feedback loop like sometimes we'd be frustrated because we'd ship things and all we basically get a thumbs up And it's like okay, then you know, why did I bother with all this just to get thumbs up? But not always sometimes you'd get some much better clarity You know from the from the group so anyway, so shipping better products get raising the quality shipping faster I think is important for me. It's also sort of about momentum Um Shipping things within a week feels awesome, frankly And it means that then the next week you can continue to deliver on that or you won't want something else But you you deliver everything really really closely whereas shipping something over this period of three months Um Feel slower, but it also means Like I don't know you just keeping that momentum going for three months is harder It's far worse if you're trying to do six months So if you let's got a whole bunch of other people beat But anyway, so feature flags beta group and uh in a cd process I think are some of the requirements And those things i'm pretty positive would be worthwhile no matter what so One proposal then would be like all right. Well, let's work towards that. Let's get cd working Let's get com working so that we don't have to have an rc to get com changed That we can have something behind a feature flag that we have confidence in We will have no impact on anybody except for internal users and beta users And everybody else is using, you know, what they think it'll be the last, you know, published release But dot com is deployed daily or hourly or whatever And um, and again, that's valuable for other reasons, but if we do that then Maybe we can open it up to doing more proven concept stuff another thing Is uh, I don't know how to phrase this but proven concept with production data um, I know we've had this discussion about from an api perspective, but um One of the things that did happen to improve our velocity was the visual design The visual interface of heroku completely used the api the public api Which meant that I could spin up review apps of the ui that touch production data um Because it didn't want just the production day. It was the production api Which meant that at some point once we had review apps as a product That was actually how we did beta tests of radical design changes We didn't have to even worry about feature flags We just had to review app and we would literally send the review link to our beta list and say try this out Um, it turned out to be awesome. And in fact, if we had that from the beginning We may never have even invested in feature flags now that I think about it Um review apps alone were that But also before then we would just do proven concepts We would stand up a whole new interface that looked different Um, but touched again production data so I could go in and do real things but on a totally Shell new app that we would throw away Ended up actually doing that out of necessity Because we would do these starter projects with people and we didn't we couldn't legally necessarily use the starter project code In a shipped product We didn't pay them and there's this weird legal stuff But if we did proof of concepts that we'd throw away then it would be fine I would at least the theory Um, but it turned out that we did a few great things. I like review apps itself was actually A proof of concept throw away app for a while and then several months later We finally picked it up and then we wrote the whole thing of course But that proof of concepts might be another tool that make this viable so we have so I think also, maybe we should move on quickly to give Sarah some time because They could not need it. Um, but uh So just to sort of big up my Uh, brethren, uh, Andrew and Danielle So we had all the stuff at Gitter and I actually found out recently secretly that Andrew basically Implemented feature flags, um In gitlab. So we already have it. Um, it's just it's super primitive at the moment All you can do is funnel a percentage of traffic through it. Um, and in The work that we're doing with Sarah on the navigation We're actually going to be being able to opt in to it So and also the the stuff that we use We will be able to add like hey turn this whole group on so you can take The gitlab anyone in this group and turn it off of that group or turn it on for this user or whatever So a we're moving towards that and we'll be able to do something better with that and then b daniel Um has implemented, uh canary deployments on gitlab with like full data basically so we can basically Do some of that so in this world where we have some of these things Like maybe we can start making some progress there But for me the big big one is the one that you just last day, which is cd where like hey Can we make a canary server like point to a feature branch? Like effectively like review apps or make it point like me. I want to get into this world where gitlab.com is Like develop or some branch or whatever and then actually we only choose to cut it Every six weeks or four weeks into a release or we don't or we don't pick things into it And that's like how we get there, but I'd love to Like pick up on how how we can go about like taking a project through this um and getting significant production um on board and just like doing a proof of concept of this and and winning um by proving to people and Sharing it in the team meeting or in something where we say hey We guess what we did this and this is the metric that we moved from a to b as a result of this And we shipped it in one week and if you prove success like that, I think you'll be able to Bring people on board and change some of the cultural Um sacred goats if you like cows or something And I think the path forward here could be to focus on what you just talked about like the feature flags The cd everything else which we all I think are on board with anyway Without attacking say the synchronous part. We could still do asynchronous whatever and maybe that means we're not Can be able to do five iterations in a week but if we can still do One a week and get it out there into customers or for that matter like even just the simplest thing of like having a beta tester use something Mid-cycle so that by the time we launch it in a release We've had the beta testers use it for two weeks like that alone would be a huge milestone All right, and then if we can shorten that up and be like, okay But then we actually iterate three times before the release like yeah now we're just you know getting crazy At some point sure maybe we'll be iterating 20 times before release But you know, but we could still probably tackle it and maybe we'll hit some threshold where Asynchronous and synchronous becomes a problem But let's defer that and focus on just the cd part is important Which by the way since we're selling cd to people like the fact that we're not using it is a problem already So like that's a good idea and frankly if we can come up with a great way to do cd plus a release cycle I think that's an awesome topic for our customers because You know, whatever people some of the folks are struggling with the same kind of thing But anyway having some cd and review apps and everything else Yeah, if I could have a review app for think alone if I could have a review app for our features the use production data That would be fantastic Because I'd be able to see stuff earlier that and that's a precursor to then be able to iterate five times in a week So again just in such a time. I'll try to go fast, but I think um I obviously feature flags and review apps and all stuff is amazing But the one thing that I think for me in particular for like new features, which would be really helpful is I think sort of Along the lines of less async morcing to try and maybe In the world of gitlab have like more focus because like one thing I've seen is that you'll have like the back end team working on the back end api the front end team working the front end api the back end team is Working on tests the front end team is using mock data and then like like a week before you're ready to actually like You know sort of getting to reviews Maybe is when like the mirrors which in the front and back end happens and like it's super scary stressful and like really risky um, so I wonder if we can I don't know how you structure this but try and maybe have like a focused area like okay So this is a marquee feature or this release for this team for me. This is obviously small So it's easy to do But maybe you can be like look, you know, we're gonna try and focus on this feature for two weeks See how far we get and try and have like maybe like a common branch. We both work against that way like, you know We both can see we're using real data Like the product and design can click on it can like check out the branch and have both functioning parts Like and then maybe you can try and solve it that way and just be like look, you know Don't work on bugs don't work on other stuff Just like drive towards this and and really try and have and as early as possible Like a complete end-to-end function anything even if it breaks anything if it's fragile I think that'd be helpful as well even you know without of course again The feature flies and better groups also super helpful But that I think would also be interesting to try and have some discipline there and see if we can do that and try and solve this sort of hairy risky back and front end thing which Doesn't seem to have a great answer for you Yeah, I think that's another symptom of you know the same kind of problem That we're having debates about whether we should work in the same branch or not. It's just kind of crazy to me Um, we've also got yeah, we've had lots of disconnects at the end We've had lots of things where only one side like it's worse like oh front end is ready to merge But back end isn't and we push overly like what or whatever. Um Sorry, I'm going to try and respect time here But a couple of other really interesting things that we learned about too in this process is wire Whereas um, you know when you go into stuff usually it's like, okay, well product They're responsible for this and designers responsible for that and engineers responsible for this and and everybody has these rigid roles One of the things about scrum that we sort of did take forward was that we're all just on a team And we're all responsible for shipping something and in fact we get zero credit for like doing our part But not shipping like it's irrelevant if we've got the design done or we got the product's backed out Or we got the back end, but if we don't ship we don't ship when we get zero What that means though in practice was that I would hire for but people would be like designers and developers would talk together and collaborate on things But it also meant that it was a back and forth and If the developer said well, what you've come up with is really hard This other thing would be much easier and you'd be like, oh, well, yeah Do that then because it's Then 90% is good or maybe it's even better But just an alternate that we hadn't considered and I feel like I'm having that debate on one of the issues right now With Dmitri not to tell him under the bus But like we're coming up with this elaborate design and he believes it's better I'm like, but if we could just kill that it'd be simpler and we could probably ship it this month because I'm really worried We're not going to ship And how much do you need to stick to this is the better way versus let ship and we all get zero if we don't ship And I just feel like when you do throw everybody and it helps again If it's synchronous but the team part is the most important part The team's all pitching together and the goal is for us to ship as quickly as possible And in fact The other flip side is you know that you've got five iterations in a week So the first generation doesn't have to be great ship for the tiniest kernel of the thing and then improve as you go And trust that you'll get to a beautiful design in the end But if you always start with this beautiful design and you start that from day one It just puts this high bar It might be more efficient But in practice, it's not because a lot of times it means you don't ship that release and you skip it You go to the next release and then that means you've delayed learning and you've delayed customers getting value And it probably means you've delayed every other feature from there and it all ships down by one Whereas if you made a few sacrifices in the complexity of your design And got it out, we'd all be better off. Um, I firmly believe that anyway Um, anyway, really exciting topic. I haven't heard from victor or fabio or sarah on this Do you have anything you want to add? I'll add something really quickly just uh more observation No solution yet because I think all the solutions have been said. Um, there's a lot of challenges So I won't comment on those One thing I don't think that was explicitly touched on but something that I sort of thought about more in the past week working with Related issues, which will you will see in the retro because it missed shipping for two releases Which is terrible. But one thing that I've been thinking about is that every new feature we create There's a huge Cost of maintaining it forever. So that's what I've been telling people Like sarah was on the column when we when we talked about another feature with chris and I told them that Adding is easy and taking away is impossible because a bazillion people will descend on an issue and complain to you If you take away something, but if you never add it, they can't complain about you taking it away So every time we add a feature, whether it's the co complexity the product complexity support team security There's a bazillion things that The cost is super high. So that just all goes into everything that we've said But something that I've been you know meditating on a little bit more Um from sort of the other side of the coin where It's a cost and it's not Everything you've everybody's been saying is correct But in addition to that there's a cost of maintaining it So when I looked at related issues when I looked at a native group milestones adding that all those extra features I have to jam my brain with all that extra context every time I think of a new feature Because it's now part of our product um, and so Uh, I I hope that you know, obviously product managers everybody understands that but I hope everybody at gilab also understands that and I've been sharing yobes A little short blog post about everybody's a product owner and not a and there's no product owners So I want everybody to think like that as well So that when they're designing or when they're coding It's not just for the scope of that issue But every every decision you make from a design or code perspective. It has ramifications, you know forever on gilab I'm done and I added something in the in the handbook to that effect Actually like in two sentences not not it didn't take two minutes to read It'll take you 30 second or 10 seconds to read so you can read it in the mbc section of the handbook if you care Yeah, thank you. I want just to give you my feedback on this but very shortly I really love to try this product discovery and but I'm at the same time I'm scared about the fact that we have also we have a discussion about the as That we want to fix a lot of bugs Existing regressions and so on so I think that if we put in this mind in this mindset Every time we will see that there is something more It's important. So fixing bugs and so on and so we are not able to move forward with the product discovery So what just came up to my mind my baby is just something stupid but we can try maybe as as a test to Pick up just one little thing that could be the feature that we really want to the to deliver in in in a release or just the most Simple or what whatever and maybe focus we have a four weeks Release cycle just focus one week all the team just for that without the Sorry without any other interruption. So it's clear to every to everyone that We have to focus just on that maybe not all the team because maybe you just need the one front end one back end One designer and the the product person and after this week that feature will be Let's say left alone. So if ready, okay, if not ready was an experiment. So it's fine And we can have other three weeks for let's say them Old way of working. So we are sure that you can Deliver something we can release without any problems at most we are spending one week for this experiment Maybe this can be the way to test it without impacting too much and without having the Let's say that the the blocker of saying if we fail with this experiment the next release will have Nothing in it. No, because I really want to try it. It's really interesting But I it's to figure out which is the best way to find it And to try it without bragging all the release of the month. This is my idea Um, this is your guys's meeting. So don't ever worry about time I'm just an interloper Um, I've worked this way as well. Uh mark. I'm very familiar with this. I love this um I am used to doing a lot more designing in code than in sketch. I find it painful and slow, uh, and While I love the freedom of async. I do Find it excruciating to spend a very very long time on one issue We've only been here a month and it already feels like three There's some issues that you know, it just seems like it's moving Mountains to try to get feedback and try to get it going along and to get people to At least come to some kind of If not a consensus some decision Right. Um, so whatever We can do on ux to help with this. Um, I can I can say that we're all in it's something that we talk about a lot Design is struggling for collaboration Struggling to find ways to work together more because it's something that We're used to and that we want to do so I'm in for sure Uh, and mark you had a question as far as I know. Yes, I know Tori is I am Uh Dimitri and chris. I think Pedro I'm pretty sure I know he was talking about taking care of little issues on his own Hazel is the only one I'm not sure I'd have to find out. Um, and my my thing on that is, um, I never push designers to do that I think that it's good if they can But I know plenty of designers that just don't want to and I don't want them to feel like they're not valuable because they can't So, um, I just say that, you know, if they can great if not if that's not something they want to do Then there are other areas they can help out in Well, that's good to hear because um Just frankly I tried this at a different company and it didn't work so well because the designers weren't hired for that and so zero of the designers could work in code and um It meant a lot less collaboration In practice, one of the things is like by the third iteration The design is going to say, hey, can you just shove this over a pixel or whatever? And it sucks to make a mock up to show somebody to do something It's like I'm just gonna edit the css for you and just do that. I refuse. I won't do that Yeah, and a lot of times what I'll do when when in the teams that I've been working on where I'm doing design And I'm doing development because typically I would do both Then the initials were in sketch so that we could kind of talk through and go from wireframes But as soon as it left as soon as it lived in code somewhere I worked in the code because it's it doesn't make any sense to be Doing all these crazy mock-ups and exporting out and then you just have more crap to keep track of So, uh, yeah, I'm in for sure I just wanted to add An fyi on our new ux research cycle Sarah just came back from vacation. She was gone for the first like three weeks that I was here So it was a little tough to to get moving on on What's being done for research and how we've been doing things? I did find and so my big thing has been What are we using for statistics metrics? How are we measuring things? How are we getting baselines? What are we measuring against? I've been looking at okr's and when I'm used to doing any type of okr's I'm always listening my success metrics like what is the goal and how are we measuring that goal? So it's been a little weird for me Trying to quantify that or get some kind of uh information on that. I did find an issue Where that allison had created a while ago about Pwik or pickwick or some something you guys have been using And that that isn't really very helpful Sarah filled me in said that I can check it out, but she doesn't think it's been kept up to date There's some talk about using prometheus, but that's not something that can happen until the end of the summer So all I have left right now is to continue working with Sarah and have her do traditional research methods without Real-time metrics And that's what this UX research cycle is about. She's planning on starting on the 21st I invite everyone to comment on the issue and add feedback What I'm trying what we did or what I did was I went through all the current UX research issues Kind of we did a quick spreadsheet just to say this is You know priority wise where it's at and some of them weren't even UX research at this point And then I went through I actually went through all the uh issues With the UX on them in CE they're like 900 and something One by one and I looked at each which is really good to to understand like what's going on out there And I created a couple of boards to better track and what I'm looking to do is really get baseline stats on areas That have a lot of issues that seem to have a lot of problems that people are talking about If you disagree with that, please chime in and let me know if you have an area that you think really needs research That we're not calling out. Let me know And that's pretty much it. That's all I wanted to say about that I figure most of that could be done inside of that issue. You could take a look there I don't have to talk at you about it Unlike mark. Oh, just kidding Can I um can I comment um around this era? um Yeah, I was thinking while I was uh in on vacations That maybe uh A a a problem sober we had uh, we could have with uh, a little bit of UX front and back end is Uh, if we could find a way to link the documentation on each, you know part of kitlab from the UI To the documentation for example a two tip that you define what that thing is and from there you Link there's like a learn more link or something like that that links directly to the documentation By doing so, I think we should we would improve the UX by Um making it easier to find out what that stuff is what it does and where do I find Documentation around it. Uh, does that make sense to you guys? You know, you know something like that Some probably something that we could enable and disable because um who is already used to get lab wouldn't you know Want to see you know a two tip for every hover, you know that you do would be like Crazy, but if we you know have some kind of enable disable would be like, you know What do you guys think about it? I I agree. We I think we actually have maybe two or three issues out there They're all kind of tangentially related one is about onboarding for new features So anytime we do a release there is a dismissible This is what's new We haven't really fleshed that out into how what that would look like and what the experience would be But that's one of them and then another is adding tooltips links to documentation And there's I think a third issue and then there was another one that we closed because it was too much like the others so I will try to find those and link you to them so so that you could definitely give feedback on those But that's something that I agree We definitely should be doing more out to help people discover those features Thank you. Yeah, I think the first one I know, uh, but the other ones I I don't Uh, any inputs from you know, you guys uh pms about it One thing I'll add is my gut reaction to uh documentation is that if a ux depends on reading documentation to understand it Then we've made a mistake in the ux That's not entirely true, of course, but it's a blood generalization that you know, like things should be relatively obvious A good ux shouldn't need documentation Let's say in general There are some really complicated subjects like how the hell do you do ci cd and the github ci yaml is not going to be Self-explanatory, but that actually alone has its own things that it should be self-explanatory We should have never visualized issue for whatever but till we do that point of the documentation is all we can do but Hopefully in general my point is we should be the ui should be self-explanatory I agree, but I think exploring that might point us to places that aren't so like if we're sitting there saying Hey, I learned more would be good here. That's an area. We're like, whoa, wait a minute Why do they need to learn more like it shouldn't be that hard so I see it both ways I totally agree and I think that we could probably eliminate a lot of really bad ux decisions Pretty quickly by going through those issues All right. Well, there's Nothing else on that. I think that wraps it up. We're a couple of minutes late. So we should probably close it up anyway All right, thanks everyone. Um, I think as far as action items go we should probably just you know put things Think about this a little bit product discovery, whatever and then maybe talk about specific proposals Next time or any other time But I at least wanted to plant the seed and have people thinking about it All right, ciao