 So yeah, my name is Alex and we've been doing agile development a rally for about seven years And so it's been interesting because one of the things I noticed when I first started with rallies that the stakeholders were always sort of waxing You know gloriously about about how wonderful agile was and how it's kind of like a dream for them And and so you know hearing this over and over again and and I was thinking wow There's just this weird overwhelmingly positive thing going on Over time was that a lot of what was happening for them is that many of them have been on traditional projects for years And they've been struggling with you know the sort of long blaze the long development cycles And so for them to be working with a team that's shipping software every eight weeks or so and patches every week And you know they're getting lots of features very quickly. They're you know not spending as much money They're they're getting feedback. They're being able to respond to the market and improve So there's a lot of just positive things going on for them But what was interesting to me is that that the sort of there's an undercurrent of Kind of concern and what was interesting is although as product managers We were very hard to sort of take into account You know everything that people were working on and promote our roadmaps and make all this visible Getting a lot of questions and a lot of people were asking things like you know Where are we headed and why are we working on this feature and not that feature? It just seemed like we were doing all the right things That's right Give me a chair I'll try I'll try like this see how I do so anyway We're getting a lot of sort of feedback a lot of stakeholders asking well Why are we building feature X and what about feature Y? So even though we're putting a lot of time and energy into getting feedback and building the right things You know people were still kind of confused about where we were going and and so I started to think well Maybe the agile is like a dream kind of thing is not not really an app description So when I was writing this talk, I tried to come up with it a better metaphor something I felt was more appropriate for what was going on because on the product side We were struggling to we found we were kind of you know shifting direction a lot and that kind of thing and so the basically the thought I came up with is that really agile is more like beer and The reason the reason is a little bit complicated You know if you think about you know, you know going out for beer you can easily you know have a have a drink here And there you have one or two no big deal you start having six or seven drinks And sometimes there are issues and so you know with with agile development you start out your ship a first increment and you know You you basically you know build only the most important features first you kind of wait Until you get feedback to go out back and round things out So you leave out the window dressing and then your sales team comes and says you know if we can you know Only have this one You know if you can build this one little feature for us very quickly We can close this next big deal and you know Then you try to build a couple more features to sort of experiment with the emerging market and you know Then you sort of test the waters with an integration or maybe you start building features in support of a partnership And basically what what's happening is something I like to call getting drunk on agile and What what getting drunk on agile is is basically that the business is getting intoxicated to the You know on the sort of flexibility the responsiveness to change and it's it's very hard to resist And what you very quickly get is is you've gone in lots of different directions at one time You're sort of you're not taking time to round out the corners And you've got lots of partially done ideas in the market Which is in a great way to sell lots of different things But not a good route to long-term success with your customers You also tend to end up with lots of customers with contradictory needs So what I'm here to talk about today is sort of why I think you need a product council Product council is sort of the solution that we came up with that that enabled us to to basically get consistent buy-in Get alignment from the stakeholders and and basically follow a More consistent path towards our vision and then roadmap and ultimately this makes the product better over time so We were thinking well we work on these eight week releases So we initially started out sort of doing a what I call a release-oriented product council and with the release-oriented product council We we we basically said let's do a meeting every two weeks Let's get all our stakeholders in the room every couple weeks and And and with that we'll we'll sort of get all get everybody aligned so that each time we go into release planning We kind of know We're all in agreement about what we're doing So the first meeting we started out with in this cycle was the idea of sort of the bring your ideas meeting and we Did it sort of in true agile fashion with a ton of sticky notes and you know up on the wall And basically we said everybody it's a blank slate every release you can come in with all the ideas you want for everything You think our product should do we'll we'll talk about it. We'll we'll You know get these up on the wall, you know We'll find out the commonalities and and and we'll generally discuss it and then so typically these would be ethics These would be you know larger features that are going to take multiple iterations or maybe even multiple releases to implement And then after the bring your ideas meeting the product owners would go off and and work on Trying to understand these epics trying to break them down into smaller pieces Get get finer-grained backlogs because some of that you can't just take a giant epic into release planning and expect to To come out with a you know with a with a plan that's you know in line with what you expected as a product And or without breaking them down without talking to to your developers And so we'd spend the next couple of weeks You know sort of fleshing these out and then we come back and hold another meeting Which is really a sort of about the details in voting we present the detailed backlogs for these and then you know We we'd sort of answer questions the stakeholders had about what are the implications of each of these these features and The the stakeholders would then start to kind of Would start to vote And the voting was an interesting process because on my product council I've got a group of all kinds of different people I get sales reps marketing people so the support team and Typically the things that the support team are voting for are not Remotely connected to the things that sales team are voting for so what we found is that that coming out of the voting We'd have completely inconclusive results, you know where there would almost never be a plurality and and so as product owners We kind of go off and say okay, what do we do? And because the following meeting is sort of the presentation of our final backlog so going into release planning, you know What what are the backlogs that the development teams are actually going to be able to look at and so we'd spend a couple weeks you know going through those backlogs and And try to figure out how it how it fit together best based on the voting as best as possible And ultimately come back present the final backlogs and then and then you know our stakeholders would question Why why we did things and it'd be a very contentious meeting is because you know We've got you know the VP of services and the VP of support are kind of in a buddy heads about you know Well, we should do this to close deals No, we should do this to make the agile customers better And you know it's just it's a very very tricky situation, but it did work for for quite a while One of the other things we added on because we're kind of a you know trying to be good Agilist as we added the idea of a retrospective so at the end at sort of the fourth meeting in the eight-week cycle was always a Sort of let's talk about what's working and what's not working And they never leave there's a lot of frustration that came out in the retrospective and and so ultimately we We tried we decided to kind of explore that in a little more depth and figure out You know what was going wrong with this product council process so The first thing at least from my perspective was that fundamentally was too much work as a product owner I had to spend a lot of time just just preparing all these backlogs getting getting together all these ideas that that that were You know really good ideas, but ultimately things that we might not be implementing right away I have a database of about 2,000 feature requests. Actually, it's probably closer to 2,500 at this point of You know great ideas, you know really 2,200 of those are just wonderful ideas and things We should definitely do with the product and the reality is that that you know out of those epics, you know We're going to do 10 or 20 in a year so There's there's just such a huge number of requests and and ultimately it's a ton of work to be going through all those requests and and Just sort of processing them even though you're not going to be dealing with with them The other thing that we found is that that there's a fair amount of stakeholder stress And I kind of liken this to the experience you get in a diner So if you go into a diner there's a sort of a you know often a huge menu and you know with you know hundreds of items to choose from and Frankly a lot of them are not very good I mean so and whenever I go into a diner, you know, I basically know what I'm going to order It's probably going to be a bacon cheeseburger and fries and maybe a chocolate mall And you know because those are the things you can rely on to be good in a diner and the rest of the menu You kind of learn to ignore because it's just there's just too much there What we realized we were doing to our stakeholders is that we were giving them a diner menu basically every single Release, you know look at all the possible options and and and pick all over again And that's just leads to a level of cognitive overload that none of them really wanted to deal with Ultimately the the thing that we realized it was happening as a result of this was each stakeholder was you know Looking at that dining menu diner menu and finding their their favorite thing and you know So we ultimately realized that stakeholders were coming back with the sort of the same thing over and over again Regardless of whether it was really the most important thing for us for the sales VP to increase sales They had sort of their pet feature that they would keep bringing up and they go into our meetings And they would look for you know, where's my pet feature, you know I just need to make sure that it's here, you know that I that I have this pet that that I have this pet feature and As soon as it disappears The stakeholders would become anxious because they couldn't track down the one thing that they were holding on to to keep them oriented throughout the Process so ultimately we were having a lot of churn just because stakeholders were trying to track down that one Item on the diner menu that they weren't able to keep a hold of and finally, you know eventually after you know You know an hour or so debate in the meeting eventually they'd be able to get that thing back up there on the board and it'd be there and they'd be happy and You know and and and we could then move on But it was just exhausting go through this process and really stressful for them I saw you know all these stakeholders would come into the product council and they were they were worried and Nervous, you know, they would they would rush in kind of, you know coming off a sales call or something And they would say well, you know, I You know, I really feel like I need to be here because I need to be advocating on on behalf on behalf of the little puppy, you know, so So that was a you know a challenge that we had with it Another thing that we found was that that basically doing this sort of once a release cycle thing Led to kind of a bias toward churn We found that it really meant that by default we were going to rethink everything everything single release So that this was really the biggest problem was that we found we weren't making progress on longer-term initiatives because we kept sort of Revoting and rethinking everything on a tactical level based on what's going on over the next eight weeks You know, is there a feature we can ship in eight weeks that will increase sales? Is there you know a feature that we can ship in in eight weeks that will make a you know big difference for our services team and And that was really really tough so another thing that happened is that that we basically We had we tended to to really you know go through this sort of very large list of potential features every every release and You know, I said that there's a database of twenty twenty five hundred You know requests and twenty two hundred of them are you know really good ideas The other three hundred are really bad ideas and this this this process kept sort of you know Causing those items to pop up and we we talk about things we talk about building Gantt charts into our product Because that's what customers kept asking for and when we'd work and we'd figure out You know how you know how to how to teach the sales team that no you know we don't need to give all these customers Gantt charts just because that's what they think they want and We'd we'd kill the Gantt chart thing and then eight weeks later it pop up again Well, I've got five customers asking for Gantt charts and so ultimately it kind of felt like a game of whack-a-mole You kind of you know get rid of this this this problematic item, and it just comes that comes up back again in eight weeks Sort of more fundamentally there's there's a bigger problem one is and they really that's it Everybody's got an opinion about how you should build your product You know the support team knows what their cases are the sales team knows why they're losing deals the sales engineers Knows what know what they're hard to explain the support the services team knows what would help our customers get better at Agile of course the engineers know what they would build if they were in charge and So ultimately we have all these different opinions and the challenge is for product management is that they're basically most of them are wrong And and this is for a fairly simple reason You know if you're familiar with sort of the story of the blind men and the elephant You know if you have a bunch of blind men around an elephant one of them grabs the leg and thinks it's a pillar and the Other grabs the tail and thinks it's a rope the other grabs the the the truck and thinks it's a tree branch You know fundamentally Each of these stakeholders is in a position to ask for the things that they need But ultimately that's not what's for the best of the product So the idea of having a completely democratic product council really kind of fell on its face Because none of these people are really in a position to be looking seeing the whole ultimately after the challenges You know how as product owners do we basically work and do the job of sort of seeing the whole while allowing feedback and Supporting participation listening to all these opinions. You know, that's that was a kind of the challenge for us. So Basically, you know, I started to think about well What what really was the need for the you know for the product council and and fundamentally what what a product council is about is building alignment It's really about helping you manage opinions and make sure that each stakeholder has a has a chance to have their opinion heard by the rest of the group here what other people want and And and ultimately, you know what they wanted more than anything else anything else is a system that they can trust you know because The situation where they're they're basically going into the room and they're the other kind of not you know Not seeing that one item they want and then looking out there It is you know kind of thing. It's just not a pleasant experience for anybody involved. So basically I went back to the drawing board and and And and sort of tried to rethink sort of how You know, how can I redesign this process to to be a little more aligned with what the stakeholders actually wanted? So basically, you know, so last fall I happened to join the the Kanban development mailing list and basically what we decided to do was a Kanban based approach to managing the the product council and so the really quick summary of Kanban for anybody who is not familiar with it The Kanban is a really simple series of Essentially bins that potentially have limits on them So I work out of move from one bin to the other and and we know we just used a wiki to store ours But basically we had a list of you know the work in progress The development team is working on for the current release a list of the items that are ready for development A list of the items active that we're actively exploring and then then sort of lower priority items that we either explored in the past or that That were were basically On hold so ultimately we basically changed the way the meeting works to focus more on you know On this Kanban and kind of going through it and so the discussion became less about sort of why is my puppy not in the upcoming release? But more about you know our things in the right order Are we researching the right the best things we can be researching given that we we're only going to be actively exploring three things at a time So it had a drastic reduction in the amount of churn that we had to do as You know the amount of sort of rethinking that we were constantly doing as as product owners And the bigger thing that that I noticed is it you know just gave us a much stronger bias or its stability So so items tended to move Upward when it made sense and items are now tending to get sort of rethought when it makes sense And there's there's less of this you know bias towards churn that we had before in the in this sort of reinvent everything every eight weeks So that's not that they say things don't actually change and items do frequently sort of you know that we're exploring We'll get you know we'll get far enough in the exploring. We'll decide it's not important anymore And we'll stop working on it and sometimes items will jump the queue if they're really highly urgent But but ultimately there is a lot more sort of stability in this process so so we've got that bias towards stability and Ultimately the other big benefit is it's much easier to maintain because as a product owner I sort of spend You know a small amount of time every every couple of weeks You know five minutes or so sort of updating it so that it's consistently up to date based on what we're actually doing and And so that makes a big difference for me At a broader level one of the things that we'd always struggled with was sort of the focus on the upcoming release What do you know that in the neck? What are the features we're doing in the in the next eight weeks? It's a very tactical kind of conversation and and the development team of course is focused on their iterations and releases Which is also you know tends to be focused at a fairly tactical level and one of the things that I found that as we move Towards this con bond of the style of development we Well, there are a style of planning is that we were able to do more work around Thinking about the bigger picture because going through the con bond in a lot of cases would take takes you know It can take anywhere between 10 and 30 minutes You know just to talk about what's going on update people on the upcoming features that kind of thing and that leaves a fair amount of Time to sort of think about the bigger picture So it's sort of the clouds lift and and and we can actually have meaningful conversations with all the stakeholders You know about you know, where are things going longer term where? I mean, what are we going to you know be doing you know, wow, it's getting dark in here So, you know, where are we going longer term? Well, you know, what are what's our longer-term focus because because they can all sort of feel comfortable that their issues are taken care of They always have a place they can go back to to see you know Where exactly does my pet issue stand even if it's stuck in the idea has been the lowest possible been in a lot of Cases that's a very satisfying thing for a stakeholder to know that their idea it may be a good idea It's not on the radar for the next release or even the next six months But it's still here It's still something we're thinking about and that kind of reassurance that that that these good ideas are not going to be lost is Actually, you know quite valuable and allowing us to to have these longer-term discussions about you know Are we prioritizing correctly and are there changing market conditions that really are causing fundamental shifts beyond the The two or three small features in the next release So the big consequence of all this is that that you know We've kind of moved away from these stories that you know, we're very you know very granular and too focused for VP's You know as a user with a big monitor I want to increase the number of columns I see so I can fit more content on my screen Which is a great story and it's actually one that you know, we were working on last week And it's but it's not something that has anything to do with strategy You know, it really doesn't have to do with with market conditions It doesn't have to do with partners and it's the wrong level of granularity for for that group of people to deal with so So moving up one level, you know is enabled our stakeholders really to relax and and that's you know Leading to much better results, you know within the group. So so you know, basically, that's you know That's been largely my experience you know, I think you know really within a product company having a product council makes a huge difference in in Getting alignment and I think you know getting a visible Kanban that's that's actually very distinct from the thing you use with your development team makes a huge difference because You know you take taking your stakeholders or to see your development wall is lots of fun But as soon as a VP is talking about your you know, your stories that are finished in a day or two You're you're at the wrong level of detail So get a visible Kanban that's useful for the stakeholders and and that makes a big difference Now that's not to say you shouldn't spend time with your stakeholders You know one of the things I found really valuable throughout this sort of process of Developing our process here is that you know sitting with each individual stakeholder is really key and taking the time to spend an hour So with you know with each of the VP's you know every every month Not every month But every every every couple of months to talk about you know What are the key things on your mind outside of the context of the product council makes them You know to go through the Kanban with them to ask what they think about where things are is you know Really important to making the the collective meetings go smoothly because they can ask questions about the things that they're uncomfortable with and they they get time to You know Have their have their opinions discussed in a little bit more detail So they don't feel as much of a need to sort of you know project that out during the larger meeting So ultimately you know I think you know that you spend some time with the stakeholders And you're going to get more stable stakeholders support in the longer term and and the hope in the end is that Ultimately, you know, you can kind of get to this point where agile is more like a dream again. So Any questions from the audience Right here curious about a couple of things one which is I didn't see a limit on the number of ideas that you have And that seems like a place where the blood pressure could get fairly high And the second part of the question is If it's not democratic use of the democratic thing is problematic and I understand that how then do you how then do you handle the convom to keep it somewhat what, you know, benevolently Authoritarian It is what it is it's kind of benevolent authoritarian because it is my job to synthesize all these all these opinions Oh, so let me answer the first question the there's no limit on the ideas just because I haven't had a need to limit it Also that I have a separate database that stores customer feature requests is not going to convon So for something to get promoted from from that database into ideas Somebody's got to bring it up during the meeting and there's just not that I think a lot of people recognize There are a lot of feature requests that are sort of lower have lower business value And so those don't get it I we would put it a limit if it became a problem that just isn't the one yet And that's one of the interesting things about convon is that the limits really are dynamic So we change them, you know, if we find they're not working if we find we've got too much in in one statement It's causing a problem will change the limit and we'll move stuff out The other thing that we that we often get in in the meeting is we do get people saying well What about you know idea x that's way down here and and we say well, that's yeah, that's a great idea Is there something up here in the work in progress in the stuff or researching that we should take out and? They're almost They're almost never is you know, that's that's that's the interesting thing Is it is it as long as you're actually asking people to make a decision between two items, you know It generally seems to work out on some So all right. I've got a question there Jim How many people do you have dedicated to this sort of on-site So basically out with my product line, I have there are two product owners and There's a user experience designer for this product line for basically about a 15 engineers It's a continuous process. I'd say that's you know, that's something I just I do on an ongoing basis You know, I would estimate I spend about half my time Doing that kind of work and sitting with the team working on the current iteration You know working on breaking down stories about half my work is sort of the technical side of being a product owner The other half of it is all the interface with customers and stakeholders. So What are the questions one right here? Two questions That's an interesting question So we've had it had it be various sizes We had it up to like 20 20 plus people at one point and that was when we were doing more of the the sort of fully collaborative style and and that ended up being quite tricky to manage I think I Like I'm like with the news with this convoin approach I'm enjoying having the the VP's there because we have time to get up to a high-level discussion. I find that that We have some basically some Like sales managers and they're doing okay I think one of the things that I found is that is that if we include a lot of actual sales reps who are Spending the bulk of their time dealing with customers We end up focusing a lot on the on the very detail-oriented items that they want with sales in particular It's been really important to have somebody who's really focused on sort of consolidating those opinions And so individual sales reps are usually focused on meeting their quotas and that's that's hard for them. So We've got you know, we have a couple people from marketing You know a couple people from support a couple people from services So, you know, it's and it does vary from time to time based on how comfortable people are with what we're doing And releases so at the moment, you know, you know one of the things that that my VP of marketing has said to me Recently is he really feels like we have things under control. We're really aligned with him with that And so he doesn't come every every two weeks. He'll come once a month or once every two months now So it you know, it kind of evolves over time and I would say it's sort of as a product owner You know, right now I have an intuition for example I know that the services team isn't showing it up enough And so I know their voice isn't getting heard as much as it should be within this group So I'm currently working with them to try to get some of them to show it more often Road coaching all the time, so it's hard. So you kind of have to sort of listen to what are you hearing? And do you feel like it's balanced or not? And yeah, so Should have a long-term Picture And I've rarely seen product owners and product managers have that long-term vision And I'll give you an example from automotive There was a 57 Teebert. There was a real sweep on the roadster It over in the 60s and 70s and 80s The Teebert turned into a monster And they seemed like Ford had lost that vision In the in the late 90s and early 2000s There was a retro version of the Teebert that came out They recaptured the initial vision of the 2c roadster and it was streamlined it was a gorgeous car and somebody recaptured the vision of what that product was and Recreated it and I I've seen software products over the years evolve into monsters because everyone is adding whatever happens to Pop into their pill-praised heads yeah, and there's no coherence unity essence to the thing yeah, and I Do you think product managers or product owners should have this vision? How do they how do they maintain that when they're always competing demand to what should go into the software system you Yeah, you absolutely have to have a vision I think the default for software is entropy basically if you don't have a clear vision You're going to get software that becomes more complex and less cohesive over time so It's it without a vision you're going to get that that's why it happens to so much software It's it's tough You know it was hard for me because I'm you know you can see how old I am I'm not the oldest person in the company and so for me to come in and say I have a clear vision for where this product should go and The rest of you should listen listen to me and put the rest of your your ideas on hold is a challenging thing and it really took a couple of years of Kind of you know working through this process to get to the point where where you know we on the product team really felt comfortable saying this is what our vision is and you know sort of it's You have to sort of define that vision on the product team You've got to define sort of a roadmap for how you think you're going to get there And then you have to do a lot of manual selling so I actually you know I was doing this because we sort of embarked on a new project at the beginning of this year then you know and and What I ended up doing a lot in November and December was going around each individual stakeholder and saying you know Tell me about what you know what are the key things that you that are on your mind and then and sort of brought those back and then a Couple weeks later said well, this is where we're headed and this is how this is how our vision is aligned with your needs And that was a fairly, you know time-consuming process But it was really necessary to sort of keep these sort of averaging out that happens in the democratic process from from destroying the Long-term focus, so I don't see any alternative if you're trying to have a cohesive project to having You know a person or a small group of people who have a strong vision and a clear vision and sort of and and who were able To stick with that the other thing I found that was interesting is that most people want you to have a vision and You know even if the even as they're asking for all these little features, you know They're really happy if you can give them a long-term vision and and and they want something to hold on to you know The sale, you know We have an awesome sales team and they'll sell whatever we build for them as long as they feel like it makes sense And they can tell a good story around it and and and in a lot of ways if we give them the if we help them Understand the vision and where we're going they're happy to work with that and and that's true of a lot of different people You know as long as they know what the parameters are so question over here I Having a vision implies that you're saying no to things are aren't you and I'm not here I actually haven't realized I haven't heard anybody use the word no about product ownership product development product management But that that's totally essential. Yeah, and I think one probably one of the causes of a lot of Lack of vision that people complain about is the fact that you know whoever is responsible for this vision Not saying no to things. They're outside. Yeah, they're not to site And it's causing pain for this person and this is one of you know This is a very tiny proportion of your population You've got to say no at the same time that person's got to live with it And the support teams got to deal with it and it was really taking a lot out of me personally to be having to say no and The nice thing about the convo approaches it kind of allowed me to reframe things and say look I'm not saying no, you know, they're there certainly, you know, it makes sense to do the Cyrillic stuff It's just it's not the thing. We're going to do next release and it's a good It's an idea. It's something we should explore. We should see how big the market is It's just not the thing we're doing tomorrow and that's a little bit different than a no And for me as a product owner help me sort of get past that just psychic pain of how because you don't want to say no all The time as a person, especially when people are coming to you with very legitimate needs I mean, you know, there's you know, there are customers who have very specific things They need to do for for particular reasons And you know, I would love to give exactly the software that that was all things all people to all of them But it's just not practical kind of service The product backlog for me is is a finer level of granularity So really I'm you know, I try to keep the product backlog relatively empty So that you know and then relatively short and sort of separate out the things that are close to being ready for development That are small and story-sized versus the bigger picture ideas like you know, I don't know We built a recycle bin this year. So that's a pretty large thing So yeah, it is a lot like a backlog But I think it's that that concept is sort of staging pieces of the backlog so they don't pollute each other that that it's really important I think I do have time for one one or one more question. Let's try it back there When you're working through your canvass and queues, are you looking at business cases actual ROI numbers? In many cases we are and that's actually something we try to requiring people to have lightweight business cases Especially when they want something to jump the queue and and then move ahead, you know You really have to say, you know, this is worth X amount of dollars for you it's really hard to quantify dollar value of features and I you know and often The problem is in a lot of ways the less significant features are often easier to quantify in dollar value and the things that are worth a lot more are hard to measure so So we've we've we definitely try that whenever there's a case of something that wants to jump the queue It's like well, how much is this really worth, you know There's a good good we were trying to do something recently working in support of a partnership And we actually did the car that we figured out the cost of it was about one to one and a half million dollars We tried to figure out the potential value of the partnership It didn't match and so we we basically said There has to be a better justification for this and the value you're currently presenting and that you know took You know took a couple of developers and a couple product owners a few hours to do and it was definitely worth One more right here in the middle You Talked to a lot of people like I said I spend about half my time talking to customers talking to Talking to support reps, you know, I sit yeah, I sit with the development team, but I'm walking around the office all the time I'm spending time with sales sales reps on calls with customers, you know I'm going out to visit customers. So I think it's really just immersion I mean my job is to be immersed in hopefully the right proportion of different types of Activities the right proportion of sales activities and that kind of thing. It's just So You know it's interesting debate about whether product management is an intuitive process or a data during process It's really it's some of both. I think you know the data is very important You know as to as far as how many customers are requesting this how you know how big are the markets that kind of thing So you have to look at that data But ultimately a lot of the a lot of the decisions are kind of intuitive It's it's you know the the question about when you build a feature is really complex It has to do with what other features, you know combined with it make a real synergy What parts of the market can we go after if we have this now versus later? I don't see a good alternative to that being an intuitive process and I think you know It's it's really hard to train a group to have that level of intuition effectively You know, we're we're kind of working on it within our product management team The other product owner and I and the user experience designer all sit together and try to have that intuition as a group But it's very it's it's a result of lots of face-to-face interactions lots of you know close discussions and And and you know how to do that with a with the broader group is you know It's tricky to involve them in that intuitive process It's easier to basically come to the conclusions test them out see how they fit and bring them up to speed about how you got there So that's kind of what I try to do And one more question here How do you manage the relationship between the cool screen things on your campaign list and you'll find a great news stories Which comes first? Have you made sure they sort of correspond well enough with each other? Ultimately, you know, you know if you look at Flip back to the convo here You know if you look at you know all these are really coarse-grain things recycle bin subscription merge report printing customizable charts Those are all kind of big things. So typically we start with the big ones I also have a smaller queue of small customer requests and defects That's not managed as part of this process that gets blended in sort of in real-time iteration by iteration with everything else so so that the two both feed into the the backlogs All right, thanks everybody