 Hi everybody, am I close enough to the mic? Can you guys hear me? It's okay. All right, so welcome. Thanks for coming. We're gonna get started. So this talk is called How Changing Our Estimation Process Took Our Project Endgame From What The F to For The Win. So if anybody here was not planning on listening to a talk that goes into estimation process and how we have improved that, I'll talk about a past project failure, do a quick case study on that, and also talk about some of the pitfalls of RFP and kind of fixed bids. If that's not what you were expecting to hear about, then now would be the time to go find another session. And so that's what I will be talking to you about. So yeah, now is the opportunity. So since you're still here, I will get going. So a little bit about myself just really quickly. My name is Ashley Tavenay. I am project manager at Blue Spark. I also do a lot of operational things for the company and kind of have my hands in a bit of everything. As project manager with Blue Spark, I've managed small projects, big projects, lots of projects simultaneously. We've had some tight budget fixed bid situations. We've also had more ongoing development type situations where it's kind of like just build this and whatever it costs, that's cool. So lots of different project experiences. Before working for Blue Spark, I was actually e-commerce manager for a luxury jewelry brand based in Paris. So I have experience on both the client side and the agency side. A little bit about myself personally, if you caught up, caught on to, like I just said, Paris. I did live there for 14 years and you probably picked up on the accent as well. I am American, but I did obtain French citizenship at the end of my 14 year stay. So I am dual citizen now. My husband and I moved back to the States a couple years ago. We're now based in Chicago with our two young children. So that's a little bit about me. And now we'll get to the part of this discussion, which is the ideal project budget. What is that? It's generally something that's pretty elusive. I'm sure most of you would agree that it kind of looks like this. Just endless piles of money so you can build everything that your client wants. So what's unfortunate is that that's generally not the case. Most clients are not coming to you saying, here, we want this, build it. We don't care how much it costs. That's not really very realistic. So getting back to reality, a lot of clients are going to come to you and they're going to have a budget that they have defined internally. And so that actual budget has usually gone through some type of approval process internally. It may be somebody who's in the purchasing department or maybe it's their CFO or the marketing manager. Whoever is kind of in charge of the website, in charge of the funding, is going to have approved the budget for this project that your client is bringing to you. And so the problem is that person oftentimes is a bit far removed from the actual web development and doesn't necessarily know how much it's going to cost. So they've just kind of based the decision of what this budget is going to be on maybe a past project that they did that may or may not be related to the current project. Or it may be that they have a cousin's daughter who did a web development course in college and thinks that maybe it's about this level of effort to build this site and they're going with that. Or maybe their budget says we have $300,000 for what the web this year and that's it, no matter what projects they want to fit into that. So it's not always a good fit with what they're actually trying to build and that can be a problem. So oftentimes clients that have a fixed budget are also going to have some form of an RFP that they come to you with. So an RFP or a request for a proposal, so a list of features that they're looking to get built with this project. So oftentimes, as I was talking about kind of the financial processes that they may have to go through internally, they also probably have some type of process that they have to go through in order to get this RFP written up and approved. So that's where the different stakeholders can come in. If you may have the product manager, the person who's going to be your main contact and who is really responsible for the success of this project, maybe they wrote up the list of things that in their experience they know are actually needed, but then they had to bring that around with their request for funds to different people internally. And one guy said he wants this other thing and this other person said no, but it'd be really great if we also had that. And so this list of features just kind of gets longer and longer as different stakeholders have their say. And it becomes essentially an RFP of not necessarily what they need for a successful project, but what a lot of different people want or would like to see. So where that can get kind of sticky is that they're sending you an RFP, but they're also generally asking you for a relatively detailed estimate in order to win the bid. And that can essentially be a ticking time bomb because at this point you are really responding to just a bullet point list of features. You don't really know how the different features are going to be built, what they actually include, how they're going to interact. So there's a lot of details that are missing at this point in the process. But of course, you'd like to win the project. So when you receive the RFP, that of course includes all the things. It's a really long list and has a very small budget. You go to the client, you ask them some of your basic questions. This is kind of your pre-sales process. You're trying to ascertain what they need. What are these features going to kind of look like? What is going to be the level of complexity? You're trying to figure that out in a couple of quick conversations with the client. Oftentimes at this point in the process, they're kind of talking to a lot of potential vendors and don't necessarily have hours upon hours to sit and talk with you about all of their dreams for their website. So you get the information you can. And you talk internally, try and figure out, okay, we think that it'll take this much time for UX, maybe this much time for design. And so you put together your proposal and you attach a budget. This is just a sample budget. The numbers don't mean anything there. And so the thing with this is, you spend a lot of time internally, kind of talking this out, thinking it over, trying to put together this great budget, this great proposal because again, you want to win the bid. And the problem is you've worked really hard on it, but at this point, it's really just based on assumptions. And so it might as well look like this. It could be just a bunch of question marks on a piece of paper, because you don't know. You really don't know how long it's going to take to build any of the different features that this potential client is asking for. But at this point, that really is kind of okay because the client just basically wants to see that you can meet those stated objectives that they have in their RFP within their defined budget. So the actual numbers of how you're going to get there are not that important. I mean, I've been on the client side and I've seen how this process kind of works. In general, you receive your proposals, you receive your bids, you skim over the first page, you open it up, okay, where's the budget? Are they within the number that we told them? And then maybe you go back and read the rest of the proposal. So the number is important to potential clients, but the detail that's in the estimate generally is not. So essentially, as a project evolves and as it does gain in definition, you need to revise your budget. It's really important that as you continue defining the process, working with this client, getting to better understand their needs, that you take the time to revise that budget and do not necessarily stick to the initial budget that was, again, based on assumptions. It was your best guess at that time when all you knew was an RFP and a couple of phone calls. So unfortunately, I kind of learned this the hard way. And this is where we are going to get into the quick case study that I mentioned at the very beginning. So here's how not to manage a fixed bid, brief case study of that one time and there was really only one, I promise, that I messed up. So in this situation, we had a client come to us who had a really very long RFP. It was like I was describing earlier, bullet points, lots of items, lots of features. And so that was all these things that they wanted. They also had a very tight timeline. They had some very specific milestones that they needed to meet and at which point specific features needed to be ready. They didn't need the entire site ready for phase one, just phase one needed this stuff, phase two needed other stuff. So their wants were very, very long list. Their actual needs of what they needed to get phase one out the door and phase two and a much later phase three were different. It was definitely not the same thing as that RFP list. So in this specific case, the RFP included user login, event management, an application process, juried selection of user submissions. They needed image galleries, a blog, user profiles and event ticketing. So there's a lot of stuff going on here. It was relatively complex when you put it all together, all the different things that they wanted to happen with this site. And so we did, what I'm sure you all do, looking at this long feature list and their type budget, we worked with our client and we tried to define what were their actual needs. Let's take this RFP and now we're going to figure out for phase one for that first deadline, what do you need for phase two, what do you need and then what can we maybe wait on because the list is pretty long. So we want to make sure that they get what they want, what they need when they need it, not necessarily all their wants. That's the whole problem. So yeah, we kind of in talking with our client and discovery, we moved some of the items in that RFP to layer phases. They had their phase one deadline, the things that they needed for that that had to be out for going to get done. Phase two was going to get done with the other things that they absolutely needed for that deadline. And the rest of the things that didn't really fit in those two phases that weren't immediate needs were put in phase three. And that was kind of a common accord. We discussed it with them and kind of discussed what that would mean in terms of their budget, that their budget was going to be for phase one and phase two. And then with next year's funding, we would get to phase three and those other items. So after at the end of this discovery process with them, we started development and maybe two weeks in, we were far enough along that I was ready to do a demo for them. So we set up this demo and I share with them the demo and a budget update in which it was apparent that a lot of the project budget had been used in the discovery phase that was to be expected that was kind of what we had planned and we were so far along in development so that much budget had already been used up and basically what we had left was enough budget to complete phase one as planned and finish phase two. And so at that point, we're looking to be on time, on budget and on scope as we had defined with them. So I went into this meeting very confident thinking they're going to be so happy. They weren't, they were not happy because in the end, I hadn't really managed their expectations and so essentially they in this meeting said to me, but where's the other stuff in our RFP? Why aren't we getting that in this budget? We had this list of all this stuff and we're not going to get that. So despite the fact that I had communicated these things to them in our meetings, I obviously had not communicated it effectively enough because when we were having these meetings and they were making decisions as to their priorities and moving things to phase three, they weren't really cognizant, I don't know if it was intentional or not, of the fact that moving things to phase three meant moving them out of the current budget. So it ended up to be a bit of a failed project. It was on time, it was on scope, it was on budget, but my client wasn't happy so that is not a success. So what we did after that is we kind of stopped and took stock of our process and we said, how can we avoid this type of situation in the future? What can we do better? And essentially we came up with two questions for ourselves for improving our process and the first question was, how can we do a better job of being accurate with our estimates? And our solution there was to integrate two estimate revisions in our projects at the discovery and design phase and these are now included in our process as project milestones. So we essentially stopped work at two key points in the project and we re-estimate and we present that to our client and we say, this is how much what we are now scoping out in discovery is going to cost. So we'll get into more detail on that in a second. The other question that we asked ourselves as to how can we improve this process was how could we do a better job explaining the effect that decisions that the client is making in the design process have on the overall project scope and budget? And so there, our solution was transparency. We have started to really just explain before, during, after exactly what our process is and where is this money going? So we tried to make it abundantly clear that when a client is prioritizing things, it is not, oh, I'd like this first and this and this and this and I'm still gonna get it all. They are making hard decisions wherein by prioritizing A, B and C they may not be getting D, E and F. So we explain this whole process that I'm going to go into detail about in just a second. In pre-sales, it's in our proposal. It is explained to them in the kickoff meeting and while we are in discovery and while we're in development, we keep going back to, this is the process. It works, the end result is probably going to be pretty accurate. You will most likely not spend more than what we have told you it's going to cost because we've refined our budget as we went. So the other thing that we also do in terms of transparency is we have basically all of our project information is available to our clients. We use JIRA and Confluence. There are lots of other tools. The tool doesn't really matter but in any case our clients have open access to it. They can see all of our tickets. They can see all of our time logs. They can see when things are moved to different statuses. They can comment on all of the wireframes of the project documentation that is in Confluence. So all of that is completely open and transparent to them. We really just try and give them as much project information as possible. I'm going to take questions at the end. So basically we redefined our entire estimation process and we decided we're going to incorporate these two estimate revisions at two points and we're going to increase transparency with our clients. And to now, how does this whole process work? Here is the detail part. So the process begins, as I'm sure many of yours do, with a kickoff meeting. So in the kickoff meeting, important things to do are make sure that you're asking the right questions. You want to kind of get from your client's mouth what their needs are. It may or may not be what's actually in the RFP. Sometimes, like I said, different stakeholders added a whole bunch of stuff to that list that your key project people may not actually care about. It's not really a need. So it's important to ask them to kind of redefine that in this kickoff meeting. So the other thing that we do is make sure that we review the process. Again, we like to be very transparent and just repeat it over and over and over. This is how the project is going to work. We're going to go through these different steps. This is going to happen at this time just to make it really clear. And so that they feel kind of confident in how things are moving along because they know what's happening. So we also then will discuss the different project milestones, scope definition, and at what point we'll be determining the MVP for their project and when they will receive different budget and estimate updates. So that's kind of what happens in our kickoff meetings. After that, we would move into discovery where, well, I guess the kickoff meeting would technically be considered the beginning of discovery, but we move into UX. So in these meetings, we generally have our UX designer, project manager, and then the different client stakeholders. So what we'll do is there's generally a first meeting where the UX designer kind of goes over general strategy, tries to define the different parts of the site. What are the main features? What are the main sections that they need? And then tries to kind of prioritize that he knows which areas he wants to spend time on when in the subsequent meetings and in the sketches that he does. So he does then, after that first meeting, he'll set up another meeting with the client as soon as possible afterwards, in which he will present to them his first round of sketches. So whatever those one or two prioritized features were that came out of that first strategy meeting, he's gonna show them sketches for those in the first meeting. And he does then what's called rapid iterative design. So basically the idea there is you iterate on your sketches rapidly, trying to get feedback from your client very quickly and while it's still fresh. So essentially he'll get his sketches ready, show them to the client in a meeting, present them to him, walk them through, like this is this, this is that, this is how these are going to work together and get their immediate feedback. At no point is he saying, here, here are these sketches, why don't you go talk about them internally for three days and come back to me with the opinion of 500 people in your company. No, he's getting their immediate feedback in the meeting and applying that to a revision of the sketch. So he'll revise it, maybe the next day then he'll have an updated version of his sketch that he'll share with the clients again, get more feedback, iterate again. So it continues that way until we have sketches that are approved and at that point we stop and we do our first estimate revision. So we call this early tech planning, we also sometimes internally call it tier one. So if you hear me say tier one, it means early tech planning. So early tech planning, essentially at this point we have sketches, they are not wireframes, they aren't full interaction requirements, we have sketches that we're working off of. So we have more definition than when we just had an RFP but we definitely don't have full site requirements and some of the details are still a little fuzzy. So in these meetings we include three people, we include the UX designer, the technical lead and the project manager. And basically the UX designer will do a sketch review so he just walks us through the sketches, the different site sections, what he has worked on with the client. And then the project manager will open up very broad tickets in our ticketing system and the tech lead will kind of think over things and provide rough estimates. So the tickets at this point are things like build the homepage, build the events page. It's definitely not very detailed, just big chunks of work that we're trying to define at this point. And so consequently the estimates that are put on those are very broad as well. At this point what we're aiming for is plus or minus 40% of how much time we think it will actually take to build this chunk of work. So they're very rough numbers, it can be like oh maybe that'll take a hundred hours, oh maybe that'll take 50. We don't know, there are really kind of ballpark estimates at this point. We're not trying to be super accurate. The point is to get kind of an itemized list of the different features that we're looking to develop with a rough idea of how much time I spent on these because that is then going to allow our client to begin to prioritize. So once we have all of our tickets together, all of our estimates, I'll prepare a little summary of all of that and I add it all up and tell them, okay look at this point we think that this is how much your project is going to cost. Plus or minus 40%. We share with them any other assumptions that we have. Oh you're going to be taking care of the migration, so that's not included in this and any of those type of assumptions that are important to a project would be included and listed in a document that we share with the client in a meeting. So the advantage though to sharing this with our client is that it gives them an overview of kind of the complexity of these different features that they've been designing and so they can see the total number of development hours for everything that's in their sketches and it allows them to prioritize early. So they can see, okay this thing that I thought would be cool but I don't really need it, I just thought hey this other site has it, I like it, maybe I'd like to have it. Well I don't really want to spend 100 hours on that. So no, we don't need to discuss that any further, it was a great idea, maybe some other time we'll pick it up, let's focus on the other things. So what's nice is that it allows them to do that important prioritization before you've even moved into wireframing and interaction requirements and long before anything is built. So after that step we then have our list of features that they want to kind of move forward with. These are their priorities. And then we'll go into more meetings between the UX designer and the client to define the sketches and turn those into full wireframes. Essentially we want full wireframes, full interaction requirements, we want to know how are all the pages of this site supposed to look? How are they supposed to interact with each other? How are users interacting with them? And then we also want to have comps so we know what is the theme going to look like? How are all these different parts going to fit together? Essentially what are we trying to build? We want all of those details in these documents that we would then get approval for from our client and they would all be shared in compluence again. So it's very important though to have full approved wireframes, interaction requirements and comps before you move on to the next step in the process. So the next step is the final technical planning or internally tier two. So this time we include a lot more people because we're trying to be much more accurate. We're not talking ballpark figures here. This is several 1.5 to two hour meetings that we spread out over a week, two weeks depending on the complexity of the project, depending on how many things we have to get to. And basically we include the UX designer, the technical lead, I'm sorry, two developers, not just two, two developers, one themeer and the project manager in these meetings. So there's quite a lot of us and we spend a lot of time in these meetings. They can be pretty intense and pretty long. And so what happens is the UX designer goes over all of the wireframes, the requirements, section by section with the development team. The only person at this point who usually has seen much of if not any of these designs is the technical lead. And the only person who is looking at the tickets that we defined in our early technical planning and at the estimates is the project manager. Everybody else is just looking at the wireframes and our UX designer shared screen. Generally when I'm in these calls I don't share my screen, I'm just working with the tickets and trying to update them as we go but nobody's looking at the estimates. And if we're lucky the tech lead has forgotten the estimates that he put on those tickets in our first meeting. So the idea is the UX designer walks everybody through everything. This is how it's gonna work. This is what we're trying to build. And the dev team talks a lot. They'll go through these different features and say okay, how are we gonna architect this? How should we build it? What module should we use? What fields do we need? Is it gonna be panels? They discuss all of the details. All of the stuff that's kind of over my head. Totally boring. I'm zoning out and trying to update the tickets. We also take implementation notes while they are discussing all of this. So they'll kind of discuss, discuss, discuss. They come to their final conclusion. This is the best way that we can build this feature and that goes into the implementation notes with kind of full details so that anybody who comes in could just pick up that ticket and work on it because it's all been defined. They know how they want to build it. So then once that level of definition is there, we'll say okay guys, so how long do you think it'll take to build this? And we kind of do like poker estimation. It's not like they don't write down their numbers and share them at the same time but we just go around the room and each of our developers says how long he thinks it would take to build whatever feature we're talking about. We don't necessarily go with an average or the highest number or the lowest number. We kind of come to an agreement. If one guy says 10 hours and another says 30, well then maybe somebody missed something, maybe we need to keep talking a little bit. We try to come to an agreement and then that is the number that I would put in a ticket. So the goal here with the level of detail that we have and the time that we are spending reviewing all of this and trying to make sure that we're choosing the right architecture, our goal is to be plus or minus 10% of the actual project hours. So then the results of that technical planning are shared with the client in a meeting and the results are also deliverables that we give to our clients at this point. If they wanted to take this fully defined project and go work with another company, they could, they have all the details there. It's totally fine. So these are deliverables for your client. So we have the tickets that I was saying where, yeah, I know that it's a little small, the actual detail of what's in that ticket, but essentially what it is, is links to the wire frames to any other relevant documents. If they decided that they're gonna use some module, maybe we'll put the link to the module there. If there's some research that's required into some integration or something, we may put a couple of links there as well. Just the point is to make it really, really easy for whoever picks up this ticket to just get to work and not have to spend time thinking it over, they should be able to start. And then we'll include those implementation notes that we defined together in our meeting, meetings, and issue mapping with any blockers. So any dependencies, anything that needs to be done first would be then linked in the ticket. We also then have a full feature list with estimates. So what you're seeing here is essentially the different epics, so we'll take the work and define it as epics, so essentially categories of work. There you have homepage, there's search and research, support and services, so those are just different site sections. And in each of those you have the tickets that comprise the work to be done on that section with the ticket title and then the estimate is on the side in the numbers. So that full feature list, I then kind of copy and paste essentially into a confluence document. So we have the full list, all the features, all the estimates, and then I prepare a budget breakdown. So I also prepare the budget breakdown in dollars, but we'll just talk about the hours here because the dollar aspect of that depends really on your rates. So I'm taking the hours multiplying by our average rate. That's how you get the total in dollars or whatever your currency is. So here with the budget breakdown, I'm showing the client what their total project estimate is. So after our final technical planning, we think that it's going to take 256 hours to build this project. The total numbers in this case, in the original project budget were 185. To date, so to get up into the point where we are sharing this with our client, we have spent 82.25 hours. So that means that the remaining hours in their project budget, in that original project budget is 102. So if you then subtract the total hours spent to date from the total project estimate, you'd get the hours needed to complete the project, which would be 174. And so then you can define what your budget deficit is. So in this case, looking at remaining hours in the project budget, subtracting that from the hours needed to complete the project. And that gives us a deficit here of 71 hours. So essentially what we're saying to our client is, look, we've done all of this definition with you. You've told us exactly what you want in scope, how it should work, how it should look. We've taken a lot of time to think it over exactly how we will build it. And we are saying plus or minus 10% degree of accuracy that we're gonna need 71 additional hours in order to complete the project as scoped. So when we present this revised estimate to our client, we always prepare some recommendations. So if there, it's not always the case that they are over budget. There's not always a budget deficit. But when there is, we like to come with some options for them. So with our knowledge of the project, we can generally say, look, maybe this or this feature can be desculpt. Maybe that can be pushed back to phase two. I think you can launch the site for now with your MVP. Those can maybe come when you have some additional funding. Another option and one that we've done before successfully with a couple of clients is dividing the work. If your client has an internal development team, maybe they can take on some of the work to cover those 70 hours of budget deficit. When we've done that in the past, we essentially kind of incorporated their team into ours. And we just did scrum meetings with them and assigned them tickets in Jira and they worked on their stuff and reported back to us. And for the most part, it worked perfectly fine. They're just kind of integrated in our team. And then another option, if everything is a priority, they really have to build all this stuff and it has to happen now, then maybe they have more budget. We've actually found that in a lot of cases, they do. Because they now know exactly what they're trying to build. So it's a lot easier to go back and justify getting additional budget. It's not just asking for a whole bunch of money and we don't really know what we're gonna get from it. They have that all very clearly defined. So that can make it easier for them to go back and get more funds. And also, sometimes since the beginning of the project, maybe other things that were budgeted for, they spent less on or that project is not going to happen. It's been pushed to a later date. So their financial situation may have changed. So you may be asking, you get clients to pay for all these hours of estimating? And yeah, we do, we do because it's not just estimating. It's technical planning and these are really valuable deliverables that we are giving to our clients. So it's time that's wisely spent upfront so that we can avoid rabbit holes and rebuilding once development begins. Our dev team isn't sitting down with very vague tickets and spending hours trying to figure out, okay, how do I wanna build this? What should I do? Those decisions have been made together with the project team before we ever started building. So that makes them all the more efficient when it gets time to develop. So then just a couple of things with the rest of the project process. Once you begin building, your client has signed off and they decided what route they wanna go. Then the dev team needs to kind of be mindful of any serious overages. Small overages are okay. It's okay if you're five hours over here because those type of things are essentially going to even out. But when it gets to be 20, 30% more than the ticket was estimated for, then it's time to maybe stop and discuss internally, discuss with the client and just let them know, look, there's this or whatever it is that's causing you to go over. Maybe the module you thought work doesn't. Maybe there's just a much greater level of complexity than you first realized. That kind of stuff can happen, right? We're not infallible. So in those cases you need to discuss with the client. Again, we need to be transparent and let them make the decisions. It's important to empower the client and make them kind of own their project. It's their site. It's their money that they're spending on it. And they're the ones who are going to have to live with it. So they are the best person to tell you, yes, spend the extra time to figure this out. This is really important. Or no, I don't really care about that. If it's that complicated and you're not quite sure how long it's going to take to figure this one out, then let's table that. Let's keep working on the other stuff. They're the best people to make that decision well. So then, yeah, anything, with every project, as you're showing your client what's happening and as they're starting to look at the site, they're going to come up with new things that they want. Those things are essentially out of scope. If they are improvements or new future requests, those are not already defined. They're not part of the scope that you agreed to when you presented that final technical planning estimate. And so those need to get added as new tickets. You need to estimate them and make your client prioritize. It's up to them, again, to make these decisions. It's their site, they own it, it's their responsibility. So they can say, yes, this new thing that I just thought of, really, really want that. Don't want this other thing. Great, that is perfectly fine. It's just not on you as the agency to make those type of decisions for them or to build a bunch of stuff for free. So the other thing that we want to do when we're building the project is make sure that we are providing weekly status and budget check-ins. So just kind of, you know, quick meeting with our client to say, okay, here's where we are in the build. Here's where we are in terms of budget. These are our next steps. Do we have any blockers? Is there anything that we're waiting on from our client? And then we want to demo often. So as soon as we have something that we can show to them, we want to share that in these status update meetings, show them our progress. They love to see it and generally, you know, they're very excited to see, like, yes, this whole thing that has been very conceptual up to this point, it's getting built. I can use it and look at it and play with it and create content. So you want to make sure that you're demoing often. That's also going to allow you to get feedback from the client. If something is not working as they expected and is not what they wanted, you can find out earlier if you demo early on instead of waiting until your project is completely built and then saying, here, here it is. So when you're wrapping the project, you should find that if the process was followed, you should be more or less 10% based on the actual time spent versus your technical planning estimate. So we found that this holds pretty true. In terms of overall project hours, we are generally pretty accurate in these estimates and it really is a process that is working for us. So essentially, you end up with, hopefully, a happy client because they got their MVP. They were informed throughout the process. They made the important decisions about what they wanted to include in their budget so they should be happy. They have a site that is exactly what they needed to be at this point. Plus, if at any time during the process, things were descoped or pushed back to another phase, you kind of have your next phase then already lined up. You have the tickets, you have the details, you have the wires, the comps. If they were descoped after your second technical planning meeting, if it's after the first one, you may just have sketches. But in any case, you have a lot of what you need to maybe just get going on phase two as soon as they do have additional budget and are ready to move forward with that. So when doesn't this process work? There are a couple of situations where it doesn't really apply. So if the client provides UX, which we've had a couple of clients do, in that case, you don't really have the full discovery process and what we would do is kind of review their UX, make sure that the implementation notes that the interaction requirements are sufficient for us to write up some implementation notes, ask them any questions that we need to clarify, get the information that we need to have the level of details that we can basically just sit down and jump right into that tier two. So then we would go in and try and add all the tickets, figure out the implementation notes and estimate the project and go from there. But we wouldn't need the whole earlier part of our process. Another situation, and we have some clients like this, if it's just a time and materials project, then you don't really need to go through all of this. If your client is just kind of having you work on chunks of work as they need them and they don't necessarily have a limited budget that they're working towards, then you can just kind of estimate each feature as they bring it to you and start working on it. Just do your sprints and develop it. Some of the disadvantages to this process, it definitely requires client buy-in. It doesn't work if your client is not going to be accepting of the fact that you are revising the estimate. If they don't want to understand that, then they're probably not a good client for you. They may just not be a good fit if there's somebody who's still gonna come to you and say, all right, your process sounds great, but I want all the things in my RFP and I want them in my budget. Well, then that's probably not gonna be a good fit. The other thing is technical planning meetings because they require a lot of people and they can be pretty long and it kind of takes up a good chunk of a week, can be pretty hard to schedule. That does mean that a lot of our team is together in a meeting for like five days in a row and it can be hard to get other things done. The meetings can be, they're really long and tedious and it's really, it's kind of exhausting by the end of the week of technical planning. You're just kind of white, but there's also a good sense of accomplishment because here you're all ready to go now. You have all the information that you need to move forward with the project and so yeah, it's exhausting, but in the end it's still good to get through those meetings and be able to move on with your project. So we are trying to kind of improve our efficiency there with how long those meetings can be. I don't really have all the answers there. It's kind of a work in progress. And that's kind of everything that I wanted to share with you today. I'm right on time, so we have 15 minutes or so for questions. Thank you. So the only thing, there is a mic that we're supposed to use for questions so they get recorded. Hi, thanks for that. That's kind of what we do once we get the customer to buy into our service. But my question which may or may not be relevant is how do you get the client to buy in? Because often time we'll get an RFP and that same RFP would have been sent to maybe a dozen other agencies in our area and the customer doesn't even want to go through this process. They just kind of want to see a bottom line before they even shortlist you for meetings. Okay, so the question is how do you get client buy-in to the process? So I don't actually deal with the sales process personally but I know that it's kind of, we require the client buy-in. So it's kind of like if they are not going to be cool with our process and have a different way of working or they're going to require, like I said, that full RFP, the full list of items from their RFP within their fixed bid budget, then we probably decide not to pursue. It's just not a good fit then. Yeah, I mean it's really important to us to follow this process and I guess in sales that really means selling the process and explaining to the client what they will get from it. The fact that they will get their MVP, they're gonna get the most important things that they need for their site and they will get it within whatever budget they have to find but it's gonna require some sometimes difficult decision-making on their end. Some people aren't willing to do that though, so. Okay, great, thanks. Hi, thank you for your presentation. As a developer, this process seems really exciting compared to what we currently use. My question is around documentation and product ownership. It kind of seems from your process that your UX designer would be the product owner in that they're creating both the sketches, wireframes and helping define the interaction but how do you kind of collate all that for your developers once they start building? Is it a Google Doc? How do those resources kind of come together? Okay, so the question is kind of how do we document the different interaction requirements, the wireframes, like where does all of that live? Yeah. So the implement, when we're in our meetings, we do have a Google Doc that we work on with the implementation notes that our team can continue, the developers can continue to refer back to at any time during the build. However, that document I will then put into the tickets. So the document will be ticket 127 and then you'll have all the implementation notes for it and I will put that exactly verbatim in the ticket. So if they want a full overview, they can look at the implementation notes that we have written up in the Google Doc. Otherwise, ticket by ticket, they'll have the information. One thing that we are trying to work out is how can we get all of the information kind of in one place so that they aren't looking at their wireframes over in compliance and then the implementation notes over in JIRA. We haven't found a perfect solution yet for how to get everything in one place because the developers essentially use the implementation notes. It's important for them to review the wires and see the interaction requirements, of course. But to actually do their work, they're really looking at the implementation notes whereas the wires and interaction requirements are also more kind of client facing so they're gonna be looking at those and approving them and that's not to say that they can't look at the JIRA tickets. As I said, we're perfectly transparent. They can go look at those implementation notes if they want to but they don't, it may be kind of overkill in terms of information if we were to include all the implementation notes in our interaction requirements. So that's how we do it now, implementation notes in JIRA and the wires and comps and all of that in compliance and we kind of link them all together and it's not perfect, hopefully we'll find a way to get it all in one place at some point. Thank you, yeah. Hi. Hi. I had a question about the discovery phase. You said that your process starts with the UX designer sitting together with the customer and doing the wire frames. It seems like your UX designer is in a role of business and system analyst in a sense that he has to discover what the customer needs in an event that it's a fairly complex system and with a fairly complicated business requirements. Does this process work in your opinion? Because in our process, for example, we approach is our approach is similar but in addition to the UX designer, we also have an analyst in the initial meetings that also creates written specification in addition to the UX wire frames. Okay, so the question is when you're in those initial meetings and kind of defining strategy really with your client is just a UX designer and not for should you also have kind of somebody in a more strategic role as well as doing more business development with your and consulting with the client. So our UX designer is a bit of a special flower because he can do kind of everything. He's pretty amazing. So he does do the strategy. He has pretty good knowledge of Drupal and what it can do for clients. He's really focused on like the users and how they can get the interaction that they need out of the site. He's good at kind of getting these different stakeholders that he has in these initial meetings to kind of maybe turn the focus away from themselves and what they want and turn it on to their users. So he has a lot of that like strategic role and that ability. He also was able to do sketches and wire frames. It's not so we're lucky to have him. Not all UX designers would be able to wear those different hats and sometimes it sure you could have more than just your UX designer in those initial meetings. Sometimes on pretty complex technical projects we would have a technical leader or a strategic technical person in those first meetings as well. So yeah, it doesn't just have to be your UX designer the person who's doing the wire frames in those first meetings for sure. And does the UX designer now have an overview of the budget now if he sits together with a client and sees that the things the client is asking for is going over what was initially budgeted? Is he the person that starts to communicate that back to the customer that this may be out of scope? Is this his or her responsibility? Okay, so the process is kind of, the whole process is kind of developed in a way so that we're never telling the client from the onset that okay, what you're talking about is out of scope. No, they're defining their own scope. They're prioritizing these things as we go. That's why we're stopping and re-estimating. So we're gonna explain to the client early on. Yes, if they're asking for things that are gonna be huge, sure. Our UX designer in those first meetings can say, okay, that's gonna be a pretty serious level of effort, but let's talk about it. Let's get some numbers around it and then you can prioritize. But the main thing is giving them ownership and making them make those decisions. We're not just saying it's out of scope. They're defining their scope. It's their project. Hi. Hi. I work for a Belgium utility company and we are bound to the European law of public procurement. Basically, how we implemented this law is that we sent out an RFP to X suppliers and then they have X weeks to come up with an answer to that proposal and with questions. And then we share the questions with all suppliers that have gotten the request. So then there's negotiation around and one of them wins the bid and can do the project. I was wondering, I have two questions. I was wondering where in your process is the part that you have won the bid? When you get paid to do all this work? That's the first question. And the second one is a little bit harder. It's, this is a great agile process and how do you combine this with fixed contracts? So, okay, so actually winning the bid would come, that's actually before anything that I described for our process. I started our process with kickoff. So that assumes that we've won the bid and we are going to be moving forward with this client. But in order to win, you still have to estimate? Yes, so that would be kind of what I was referring to before I got into the details of the process where you're gonna get the, you get the RFP, you think about it, you try and ask the right questions and you're gonna come up with numbers but they're just best guesses. We definitely provide estimates to our clients in our proposals, but in our proposal we have a lot of language that explains this is the process we're going to go through. This is our best guess. We are timing materials. We charge clients by the hour. It's not just fixed bid. And that's some of the language that is in our proposal and that we make very clear about ourselves when we're presenting on a bid. Okay. So the question of how to make it work with a fixed bid, we don't necessarily have a lot of experience with government projects. I know that they can be their own beast. There's a lot of rules and regulations that go around it. But I mean, essentially what this is, what the process is also kind of designed to do is essentially kind of get away from a fixed bid in a sense. Like there's a fixed budget, but it's kind of, if that budget really is fixed and there's not a penny more, then we need to find a way to prioritize so that the MVP can be delivered within that budget and then maybe we need to talk about future phases of things. But if we're in a situation where we're completely bound to that budget and there's absolutely no way that things can be prioritized. That's not the case. You always have change requests, but they also tend to have a long procedure to get those done so you want as less as possible of those. Sure. I mean, we have been in situations too where once we get to that tier two, everything is defined, then yeah, we have put forth requests for an additional PO to cover those additional funds and then it kind of, yes, becomes still a fixed bid, but now for the new amount. Okay. Okay, thank you. Hi, you mentioned in your case study at the start that often you feel like you've made things quite clear to people, but actually it turns out they're not quite, they haven't kind of, it hasn't sunk in. I was wondering how that, most specifically how that impacts on things like the contractual phase. So if you have a fixed scope, then you can put in the contract, you know that this work is covered by the scope, which is in appendix A or whatever, but obviously if you're not working to a fixed scope, do you know about how the wording goes then at, contractual level, but also generally when you're explaining this to the clients, I guess. I'm not 100% sure if there is, if there is anything in our standard contract about how the process works, I know it's in the proposal and we would reference the proposal where we're saying this is a process that we're going to follow. We also, you know, in our contract it says we are timing materials. Right, so you actually, you stated that, because I think as the previous questioner said, like a lot of things like governments, they won't take time materials, so that's, it's quite tricky to, often they want to do time materials and a lot of them do actually sort of sneakily want to do our job, but they've got this big process above them that they just can't commit to that. So I guess that's just problematic for this kind of process. Yeah, it may just be that that's then a type of client that's not a really good fit for the process. So then it becomes a decision of, do you want to not follow the process and pursue the client, or do you want to, you know, work with other clients who can follow your process? Just maybe a clear idea of the MVP up front might help that you can kind of refer to that in the contract, maybe. For sure. Thanks. Yeah? Hi, good morning. Hi. My question is about these feature estimates that you talked about where your team gets together and discusses how much time it will take to develop a feature. How do you factor in things like quality assurance, project management that might go around feature development? Sure. That's the first part. And the second part is 10% is really good estimate. Plus or minus 10%. That's like, wow. But how do you factor in something like post-deployment support if things go wrong? Okay. 10% is too less to work with. So, all right. So to answer the first question, we have separate tickets for that. Pretty much when we set up a new project in JIRA, we add a project management ticket, a client means ticket, a technical consultation and coordination ticket, a QA ticket, a deployment ticket. And those kind of have standard numbers that we'll put on them. And that we will, depending on how long the project is on the level of complexity, we will add to those or subtract from them. We kind of guesstimate at those. And also based on past experience, obviously we can look at a past project that was similar and look at how much time we logged on that ticket and make sure that the estimate for this new project covers that. So that's how we account for QA, project management, client meetings, all of the kind of project overhead in our process. As for things that come up post-deployment, generally what we try to do is have kind of a maintenance agreement with our client where we're saying, okay, post-deployment, let's agree to 10 hours per month or whatever to work on whatever new improvements you have coming up, any updates that you need. And hopefully they'll agree to kind of sign on for a year so we can just make sure that everything is working and make any minor improvements that they need in that time period. Okay, so that's part of the initial agreement itself. Yeah, thanks. Yeah, of course. So we are at our time. So thank you everybody again for coming. It was great. Thanks.