 Good morning. I hope everybody can hear me. Sounds like this mic is pretty strong. But I'm going to stand over here so you can see me because I'm short and the podium is tall. So, first, before I get started talking about myself and Blue Spark and what we do in Estimation and this talk, I just wanted to know a little bit about who you are and who's in the room today. So, kind of by show of hands, who here is a project manager? Alright, lots of project managers, probably over half the room. I need developers. Alright, also good number. Anybody who's like an account manager, more on like the sales end of things. Alright, so kind of scoping out projects maybe at first. And anybody who just walked into this room hoping to hide in the back and check email. No, one person. You're honest. Alright, so enough about you. I'll get into the quick bio part. So my name is Ashley Tavenay. I am Chief Operating Officer at Blue Spark. I live in Chicago. I have two young kids. They're four and two years old. They're super cute but I did not put a photo up so you'll just have to take my word for it. So I began my career in advertising. I worked at first doing production of commercials and kind of in a sense project managing that process and getting shoots going and things like that. I had a bit of a change of heart and realized advertising wasn't for me. And I found a job working at a French luxury jeweler called Boucheron. At the time I lived in France. I lived in Paris for 14 years. And so that is kind of where I got my start in the whole web world. I was their e-commerce manager for a couple of years. I was dealing with the back end and dealing with all the orders and some of the customer service things, updating all of our products. So kind of more of like the admin client side that a lot of us probably deal with. That site was actually built on Drupal 6 with a flash front end. So it was really great. I couldn't update anything myself. And so that's also how I got my start in the Drupal world. So after a couple of years there I had an opportunity to join BlueSpark as a project manager. And so I parted ways with Boucheron and went to BlueSpark. I was project manager with BlueSpark for I guess five or six years before kind of transitioning into an operations role. So I've managed a lot of different projects. Big, small, really cool clients, really difficult ones. As you manage projects you do with a whole lot of different situations and kind of learn as you go. I'm sure a lot of you know that. But now mostly my job is kind of keeping the agency running and dealing with resourcing and projection and planning out kind of our future work and keeping the team busy. So BlueSpark is a full service digital agency. We focus in Drupal. We also do some WordPress and symphony work. But we kind of do full project life cycles. So you know design, UX and through development. We also offer support to some of our clients and things like that. We are a remote company and we have been since our inception. So our team kind of spans the globe. We have people from California to Hungary. So we deal with a lot of different time zones, a lot of Google Hangouts and things like that. So this talk is how changing our estimation process took our project end game from what the aft to for the win. And we're going to talk about estimation. It's pretty important to most projects. We're going to talk about how that process evolved in our agency, how it came about, how to do it, if you would like to implement this great process yourselves and the outcome of that change. So before I actually talk about estimation though, let's talk about its precursor which is budget. So this is obviously one of the three sides to our project management triangle of scope, budget and timeline. And it's often, not always but often, one of the most inflexible. It would be so great if all projects just had the ideal budget. Every client who came to you was like, look, I just have endless piles of cash. This is what I want, this is my timeline. You can spend as much as you need to, just get it done. But that's not really very realistic. Most clients do need to stick to a budget. So coming back to reality, the truth is that a lot of organizations and companies who are engaging in a large or small web development project, they've had, there's kind of been the whole process they went through before they came to you, where they have to obtain budget approval. So probably somebody within the organization who is involved with their website, they felt a need, noticed a need for this project. They spent some time probably thinking about it, putting together their requirements, having maybe some internal meetings and all of that. They put together a rough idea of the budget. Maybe they did that just by pulling a random number out of thin air. Maybe they did that by thinking back to previous projects. Maybe they talked to other people in similar roles. But they have a number that they've already taken internally and they've gotten that approved. So whatever that budget is that they set internally, it's been budgeted, the funds have been set aside and their project has then been given the green light to go out and look for vendors. And that's generally where most of us come into play. So the money is already allocated and normally that actual budget is not endless piles of cash. But that's okay. There's plenty that can be done within any budget. You just need to be managing your client's expectations and scope accordingly. So where you can get into trouble though is when there's an RFP involved with a new project. So RFPs, or Request for Proposal, they usually do involve a pretty detailed list of requirements where they're sending out this list of things that they want and they're telling you this is how much money we have and more or less a set timeline. So your project management triangle in an RFP, there's nothing that seems very flexible at that point. And we all know that something's got to get. You can't have it all with your project. So that makes sense though because from the client's perspective, their project plan needed to be mapped out in order to gain that internal approval that I was talking about and in order to begin shopping it to potential vendors. They can't just go around and get internal approval saying it's going to cost about this much. I think this is our timeline, but I don't really know what we're doing. They're going to want to see all three sides of this project and what's going into it. But from a vendor's perspective, an RFP can kind of be a bit of a ticking time bomb because when you respond to an RFP, if you are providing a really detailed estimate, it can get you in hot water once your project starts. It is a great way to eat into your margins and even lose money on a project. And the reason for that is when you respond to an RFP, you spend this time, you put together a proposal. Some of you who raised hands earlier are probably involved in this process. I never was as a project manager. That was something that was also always done by our tech lead, UX, and sales, and I kind of stepped in afterwards once the proposal had been accepted. So you spend some time, put together the numbers, put it into a proposal, you attach a budget, send it out. And so you spend some time, you've thought about this all internally, but the real problem is it's pretty much just a bunch of questions at this point. You don't actually know very much about this client. You've maybe had one or two meetings with them. They sent you an RFP that kind of explains their situation, kind of explains who they are, but you don't really know who you're dealing with. You haven't had the time to get to know these people, to know how they work. Do they tend to take two weeks to respond to every single email that you send them? Do they give you really quick responses? Do they respond but then change their mind the next day? Those are all things that you learn about your client and that could, while will, have an impact on your budget as you go. So your estimate, when you're responding to an RFP, it's just based on assumptions. And it could just be a bunch of question marks. I don't know. I really don't know how much time it will take to complete these things because I don't know you. So at this point, you haven't had time to really assess their needs and put together any detailed specs and gain that understanding of this client and what they need and what you're trying to build for them. So it's kind of like, you know, if you're trying to build, if you're trying to build a house, right, and you just tell the contractor, well, you know, I want a kitchen and a living room and three bedrooms and two bathrooms. Well, that could cost, you know, there's a great range of pricing for a house like that. It's definitely not the same as having detailed blueprints that you're bringing to a contractor in the size of these rooms. And it's even not the same as having detailed comps where they know what kind of tile area you're putting in and the light fixtures and things like that. So, you know, when you just have an RFP with a bullet list of items, but you don't have wireframes, you don't have comps, you don't have interaction requirements, you don't know how much it's going to cost. So as the project evolves and it becomes more defined and you get all of those project details, at that point you have to revise your budget. And that's kind of what the process that we evolved is about. So, unfortunately, I kind of learned that the hard way. So here's how not to manage a fixed bid project. I only did this once, I swear. I never messed up on any other project besides this one. So we had this case where a client came to us with an RFP. And they had so many things that they wanted in this RFP. And so it included, like, really long list. They had user login, event management, application process, event ticketing, blah, blah, blah, blah. It was very long. They wanted all the things. And they had a tight budget. And, of course, they had a tight timeline. So, you know, we met with this client, we explained to them, you know, this is great, but within your budget and with this timeline we're probably going to have to take this feature list and we're going to have to define some phases. And we'll work on phase one now. You can get that for your first timeline. These are the features that you really need. This site was around an event. So they needed specific things to be able to do their first year's event, but then for the next year's event they could have additional features. So phase one will be what you need for this year, phase two for next year, work that all out with them. And so, you know, we moved some of those items to later phases and they were all on board with this, happy with this until we got about to the end of phase one. We were on time, on budget, I was like super happy telling my client, look, look at this great site that we have and it's ready to launch and we're on time on budget and they're like, well, but you spent our entire budget. Well, yes, that is what we explained when we phase things out was that we were going to work on phase one with this budget and when more funds were available next year we'd work on phase two. So we had a bit of a rude awakening when they became upset and there had obviously been some miscommunication there, but their expectations really hadn't been managed. So in the end, that project was, it was a failure. We didn't continue working with that client after phase one and their expectations weren't well managed and that's not a successful project. When your client's unhappy at the end, even though you've delivered what you thought was your approved scope on time and on budget then something went wrong. So after that, we looked at our, at that situation we did kind of a retrospective and we were like, we can do better. Let's, you know, we're smart people, let's sit down, let's put our heads together, let's fix our process. So we, you know, we're trying to figure out how can we better manage client expectations and budget so that this type of situation doesn't happen again. And so we just basically asked ourselves how could we do a better job of being accurate with our estimates? And there, the solution that we came up with was to integrate estimate revisions at key points in our process. So we do one estimate revision right at the end of project discovery. So as we've known, as we've gotten to know some about this client, but not quite everything yet. And then we do a second estimate revision when we get to the end of the UX and design phase so we kind of define those two milestones as the times when it needs to happen. And I'll go into more detail about this next. But the other question that we asked ourselves was how could we do a better job of explaining the effect that decisions made in the design process have on the overall project budget and scope? It seems that a lot of clients, when you're in, you know, discovery and design, they want all the things because at this point they're just kind of on paper. There's not actual development hours going towards them so it seems like their budget can stretch much further than it actually can. And so our solution there was, it was pretty clear to us that we just, we needed to be more transparent. So we changed our process to explain to clients before, during and after projects where their money is going. So we're very transparent with all of this. In our proposals, our entire process is explained. It's very clear that our estimates, you know, in the proposal are essentially guesses and that it's going to be revised. We explain at what point it will be revised and what goes into those revisions. We also explain all of, we go through that process like in our kickoff and kind of continually when we're leading, when we're getting into each milestone we'll explain here's what's going to happen next, here's we're going to revise it again, here's why, here's what will happen. So trying to really manage their expectations and get them to understand our process and what each step is and why it's important for them and for their budget. We also go in back to the transparency part. All of our time logs are visible to our clients in JIRA so they can see at any point like where time is being logged on which ticket, by whom, what they did. And then we also do, once we're into development we're doing budget updates of course on like a weekly basis and sharing that with them so they know what's happening. Are we hitting our estimates? Are we off? It's all very open. So anyway, we try to really take every opportunity we can to explain the process and really involve the client in it so they always know what's happening and how the decisions that they make impact their overall project. So we ended up then redefining our estimation process and we incorporated the two estimate revisions and we worked to increase our transparency. So then how does this whole process work? How could you implement the same revisions, the same estimate revisions if you were interested? Essentially we start off as many of you probably do with a kickoff meeting for our projects. So in our client kickoffs we always include the entire project team. Sometimes I may not know exactly which developers will be used on the project at that time but I try to include everybody who I'm sure will be on the project. Definitely the tech lead, UX, design, the project manager, whoever was involved in sales on the project as well we would all be in that meeting with the client. We prepare that meeting in advance so reviewing the RFP, reviewing the proposal, going over any questions that we have on things that are in that. So in our kickoff meeting we review our process with them. So as I said, we explain this process to them. It's a continual thing throughout the course of the project. We discuss the different project milestones. We discuss scope definition and how their decisions will play into that. We kind of work with them to try and determine an MVP. So you know what really the minimum features are that they need to get to launch so we understand really where their priorities lie. And so we discuss with them at what point the budget and estimates will be updated. And then we kind of get into our discovery questions. We have a bunch of questions that we go through with clients where we aim to understand what their pain points are and what they're trying to accomplish with this new project. So we're trying to kind of really get their business needs out of them and not understand what they want but what they need because it isn't always the same thing. Sometimes they think that they need certain features that they really don't. So after our kickoff we then get into our discovery into our discovery process with UX sketches. So we do kind of a series of client discovery meetings with any of the key client stakeholders our UX designer and the project manager. So essentially in these we kind of do rapid iterative design. So it's kind of fast paced and the point is to get very rough sketches in front of the client as quickly as we can in these initial discussions. So we're trying to just get things laid out very roughly and see if this is really what they're looking for is this the right direction or does this need to be over here or maybe on another page or maybe that feature just doesn't make sense at all. I was trying to discuss these things but get a visual representation of what we're talking about as we go through discovery. Sketch things out and as I said it's rapid and iterative. So we at times will be iterating on a design on a sketch kind of in real time in a meeting with them as a client to say oh no no I would like it more like this well okay let's just move that over there how is this. So we're trying to get feedback really quickly so that we can drive towards what the their real needs are and their final proof sketches will be. We also review those sketches internally with the tech lead to make sure that we aren't designing anything too crazy that's going to cost way way more than we actually have in the budget and just make sure that it's doable, that it makes sense so we'll generally do that before sharing them with the client before the next meeting. So then once we get approval of all those sketches we'll move into our next phase. So this is the first of those estimate revisions. So what we do we kind of stop everything and we tell our client we're going to do our early tech planning now. So we have a series of meetings that include our UX designer our tech lead and our PM. We review all of the sketches so the UX designer kind of walks the tech lead through them we look at the different features of how things are supposed to work together at this point they are just sketches so all that isn't entirely defined the UX designer probably hasn't had time to speak to the client about all the details yet but we're working with what we have at this point. So we start to just kind of break it down into really broad tickets so things like build the homepage nobody, no developer can pick up a ticket that says build the homepage and know what to do, that's way too broad but at this point we're not trying to we're not trying to have tickets that are well enough defined that somebody can actually work on them we're just trying to have tickets that are well enough defined so we can put a very rough estimate on it so build the homepage would be an example of kind of a feature or just a chunk of work that we can kind of conceptualize and say okay maybe that's going to take us 20 hours in this case. We are not trying to be very detailed with these estimates we are not trying to be very accurate we are just opening broad tickets and providing rough estimates with a goal of being plus or minus 40% the actual time it will take to implement whatever's in that ticket. So we then so we go through the entire you know all of the sketches and we list out all these tickets with our big features and then I take those back to the client in a meeting so we'll have a meeting where we will review this list of features versus the estimates that we've come up with and we explain to them like this is plus or minus 40% it's not very accurate so that's not the point the point is to get some type of value assigned to each feature so that the client then has a better way of assessing like is this really something that I want because sometimes this really high priority thing is only maybe $5,000 to implement and they're like that's great that's like the most important thing that I needed that's a really small part of my budget and maybe this really trivial feature that they just really added because somebody in their organization said that they would be nice if maybe that feature is like $40,000 to implement and they're like whoa that is really not that important let's get rid of that now so we're just trying to give them some information something that they can use to gauge the priority here and and say okay I really need this I really need this I'm willing to spend about this much on this and essentially de-scope things at this point that that are not necessary so we can fit into their budget so we're trying to help them prioritize by getting rough numbers in front of them so some of the advantages is that it does kind of give an overview of the feature complexity the client gets a better idea of what's complex and what isn't they don't know for them it's probably just as easy to build an entire events management system as it is like a slide show I don't know so we're trying to show them feature complexity dev hours and cost together in one place and then it allows them to prioritize early when we've only spent couple hours in meetings and on sketch design we haven't done wireframes and interaction requirements in comps and we haven't put together all these tickets and a developer hasn't worked for 20 hours on this thing yet so it's really easy to just de-prioritize now before they've eaten up too much of their budget on something that was maybe non-essential so after that initial revision meeting with them we have our prioritized list of features that we take and we're going to turn into wires so we go back into a series of meetings with our UX designer client PM where we're taking those sketches and fleshing them out into complete wireframes with interaction requirements that are explaining exactly how all of the features are going to work and how do they work with one another and how do users interact with them so basically all of the details that are going to help us figure out how to implement it once we get breaking these things down with the development team once those wires once we have the complete set of wires this is also the point in the project where we would start looking at mobile we generally don't do that in the sketching process unless it's mobile specific or there's like a really specific page that they want to look at that point just to do the mobile versions but then once all of our wires are ready to go and approved then we would hand them over to our designer and turn them into comps so he's going to add the look and feeling of the design to the design of the site to the wireframes so that we really have a full set of PSDs and we know what each page is going to look like once we have client approval of all of that that's when we do our second revision so our second revision is our we call it the final tech planning so we kind of stop the design process I mean at that point we've designed we've defined we're ready to break this down so we can get ready to get into development work so we do a series of meetings it generally happens over about a week and a half to two weeks we include the UX designer, the project manager the tech lead and two back end developers one front end developer sometimes it might just be one back end and the front end depending on the size of the project what we're looking at but so we include all those people in a series of meetings generally we'll do about two hours per day because it can be kind of long and tedious so that is why it's spread out over like a week and a half to two weeks because if we were to sit there and estimate together for like eight hours a day we would probably all hate each other at the end of the day so not a good idea so what we do is the UX designer then is going to take all of the wires and all the wires and comps and walk through them with the development team as we did before at this point the developers, the back end developers the front end developer they haven't seen these but only the tech lead knows what's in our wires and comps so they're kind of fresh on the project at that point they also do not know the estimates that were assigned by the tech lead in our first estimate revision I erase all of those from JIRA before we get in and start doing this work so that it doesn't influence them so we go over everything in detail really lots and lots of detail it would be very boring for me the PM when they get into discussing exactly how they're going to implement things and which fields do they need all of the modules and how are they going to connect this to that, they get into all of the details of how it's going to happen sometimes those discussions can take quite a while because different people have different ideas about what's the best way to implement something but we always find that it's really fruitful conversation because in the end what we're getting at is the best way to build whatever feature we're talking about we kind of come to an agreement as to how we're going to implement this all the while taking notes, we're taking very detailed notes that we can then take and pull into all of our tickets so that any developer on our team when he or she goes to work on that feature they know exactly what was decided like we're going to implement it this way this is what you need to do, here's all the steps so once we've come to a conclusion as to how to implement then we kind of do a round of estimation poker so we go around the room or around the Google Hangout because we're remote and everybody gives their number, maybe one person says two and one person is like ten and one's like twenty we need to discuss this a little bit more because there's quite a difference there maybe the person who said twenty really forgot that this other part was going to be done on this other ticket so we don't really need to estimate that here but the person who said two wasn't thinking about this part of the ticket that we have to implement so anyway we come to an agreement just kind of discussing the estimates that are given in our estimation poker it's not always the number in the middle sometimes the lower number the high number is the right one but once we've talked it out that's the number that goes in our ticket so what we're trying to do here is get to like our accuracy goal is plus or minus ten percent of the actual project hours so we are trying to be pretty accurate here and really that's why we spend so much time discussing the details how are we implementing this exactly which modules are we using how is it going to work so we found that the more detail that we have in our implementation notes the easier it is to have an accurate estimate because you've taken the time to sit there and figure it out like this is what I need to do so I know how much time it will take because I know what I have to do so I'm not just guessing so at the end of this then we kind of have our full project architecture laid out like we have all of our tickets we have all of our implementation notes we have dependencies defined and tickets are tied together in JIRA so we kind of everything's there and ready to go we can just kind of get running then on development so there's also the results then of this whole process are actually they are deliverables for our client so we have, as I was saying full tickets with links to the wires and any other relevant docs with a full description of what we're trying to do and exactly how to implement it what do we need to do and then we also have issue mapping with blockers, with dependencies things that are related and we generally also define everything in epics we kind of use epics just to categorize our work I don't think we use it exactly as you're supposed to but that's how we use our epics so you can look then at the end we have a full feature list with, okay, for search and research that section of this particular site it's going to take us 12 hours to build it these are the four tickets that are involved here's the estimates on them and if you were to click on any of those tickets you would then get the full implementation details these are all the steps that we need to do here's how it links up oh here's the wire to that it's all just laid out and ready to go so for the client they kind of have logs in it, they also can see all the tickets and they interact with us in JIRA so they can see all of this kind of project architecture laid out and then the other thing that this gives us is a budget breakdown so what I do at the end of this process is I take all of the tickets that we have spent so much time painstakingly discussing and putting estimates on and I add everything up so we look at what the total project estimate is after our second revision that's the top number and then you look at the total number of hours that were available in the original project budget the total number of hours that have been spent to date so all of the time that we spent doing our sketches and our wire framing and in client meetings and doing our estimation then the difference would be the hours needed to complete the project your total project estimate minus the total hours spent to date in this case you need 174 hours to complete the project so then you can look at your budget deficit sometimes it's also a surplus kind of just depends on how things go so that would be the hours needed to complete the project minus the total remaining hours in the original project budget so I take all of these things and I then present it to the clients in a meeting so we go through we go through all these estimate revisions together where we'll explain to them okay here's our full set of features essentially that is the scope that we are agreeing to we're going to build these tickets and then I review all of those numbers with them so we can look at maybe there's a concern about one area that's costing more than they thought to the ticket and look at why they still have time at this point to deprioritize any of those tickets if there's issues with them being over budget but I spend the time in this meeting to explain everything to them how did we get to these numbers exactly what's going into them what are the tickets and what is the impact on their total project budget so then before we go into this client meeting we do actually talk internally about our recommendations so if they're over budget we try to come to them with some options so maybe if they de-scope items X, Y and Z they can just do those in a further phase it's really not that big of an impact on the site they were nice to have anyway not such a big deal maybe we divide development work between the client maybe the client has an internal team and they can take on a little bit of the development work and we take on most of it we've done that before actually quite successfully and the client ended up being really happy to kind of get a chance to learn from our developers in a sense and get some hands on training and contribute to their own project or there's also many cases where we need all these things, they're all a priority and so the client actually has some more budget that they're able to allocate from maybe a different line item in their budget maybe they're able to move some funds from next year maybe they always had more budget and they never told you there's different ways that sometimes more budget can be found if they really do need all these items in the scope but essentially what we're trying to do is empower them and allow the client to be making these decisions and not come to them when we're a couple of weeks from launch and there's a couple of key features that aren't yet built and tell them we're out of budget sorry we're allowing them to make these decisions and to be aware of the actual cost to build this project before we start building it so you may be thinking what you get clients to pay for all this estimating that's crazy you may think it's totally surprising but this process is essentially all technical planning it's time that we spend upfront defining the project so that once we get into development things generally go very smoothly everything is set up we kind of have figured out how to implement these things we've talked about all of our different options so we don't have these really like nebulous huge tickets that a developer takes and then he's like oh I don't know how to build this and he starts here and maybe gets down some rabbit hole and then it goes on to this other option we've already talked about all the options and decided what the best way to go was so we reduce some of that time that may be logged on the ticket if you weren't following our process and doing it here's these big tickets and they're not really defined and go figure it out the time to figure out how to build something is going to get spent whether it's by a lone developer and doesn't know what to do with it or by an entire team trying to figure it out together in discussions so what it is it's not just estimating it's technical planning it's defining the project and kind of getting everything ready so that we can just get going and get working so it does we found that it saves a lot of time on actual development and we also don't get into situations where we need to rebuild it because we didn't plan out the entire architecture think about how it related to this other feature we don't have to scrap our work and start over again so once we get into then development after that second revision meeting with the client we come to them with their options and they're going to tell us okay I approve this final scope I approve this budget we're ready to get going and at that point we just get into more of an agile development process I know you're probably thinking but what this sounds so much like waterfall you guys plan everything up front it's not agile I totally admit that it's not agile at first but we kind of try to make it agile once we get into into development so at that point we have to find everything we have estimated everything but we're going to be defining sprints based on what they need to get out the door first and working in those two weeks so we know what we can actually take on in any given sprint and we are not we generally run our projects as time of materials so we are not we don't come back to the client and say oh well you want this new feature well that wasn't in our scope we didn't talk about that that wasn't in our wires like we'll just open the ticket discuss it, put an estimate on it and add it in and either it increases the budget because it's a new request or they remove something so we are flexible it's not as as rigid as like an actual waterfall process would be we're flexible once we get into development we can have change to our scope but we just have to be and are very upfront with the client about what that means what does that mean to your budget what does that mean to your timeline if you're adding this new thing sorry but I probably can't get it done within the same timeline and within the same budget if we don't remove something else so there's just trade-offs and they have to be aware of those and we try to empower them to make those decisions on their own we cannot decide for them because it's not our money and it's not our project and we don't have to deal with this to live with whatever the final product is once the project is done it's their baby so they get to tell us what they want to do with it so one thing that we do though is I said this much earlier that we do do weekly meetings with our clients once we're in development where we're going to be giving budget updates and going over progress and status and we demo we try to do it weekly sometimes it's just at the end of each sprint so bi-weekly just kind of depends on how much we got done clients are generally excited to see your progress so if we can do more demos we found that that's better and also that gives you earlier quicker feedback so that you can make any changes that are needed and then as we wrap the project we've generally found that our estimates are pretty accurate we do end up more within our plus or minus 10% that we defined I've never gone and done like a detailed analysis like ticket by ticket because in the end it kind of evens out plus or minus 10% means maybe this one was 10 over maybe it was 15 over but maybe this other one was 15 under and it ends up kind of evening out we're just worried about kind of the total project budget at this point and what we found is you know our clients are a lot happier they get their MVP they're within their budget and they were informed throughout the process it wasn't just like you know this really dark process where you know we were kind of hiding behind the curtain and doing our thing and then we like showed them this finished site they were involved they got to prioritize they got to you know know what we were doing at all points throughout the process and they could change things if they needed to and you know we're empowered to make decisions about their budget and about their scope on their own and then the other cool thing is if we did wires or sketches for some features that they were thinking were pretty nice to have but ended up descoping well we kind of have phase two then lined up because we already have some of what we need to get started on that next round of features we already know what the next things are that they want to do so you can just kind of you know once they have more budget available and are ready to do a phase two you're ready to get started on it so there are some instances where this process doesn't really work or the full process in any case so sometimes clients come to you with UX they've already done it they didn't use it to another firm or whatever but in that case we go through the full discovery process in a case like that what we would probably do is do a couple of initial meetings kind of review all of this UX with them like you know what are these wires how are they working because that can be a little bit difficult to fully understand when you yourselves did not do them and then we would kind of just jump into our second estimation round where okay like if this is all good and we understand how everything is supposed to work you know we don't have any recommendations for changes then let's just start estimating this we'll break it down how are we going to implement it and just hop right into that part of our process and then if it's really a time and materials project which I did say that we are time and materials that is true it's generally in our client agreements but most of our clients still do have a budget in mind that they're working towards so it's you're kind of working with both if you're in a situation where it's pure time and materials they don't really have a budget doesn't really matter to them it's more maybe a retainer engagement or something like that then you don't really need to bother with this full process you can kind of do a condensed version of it where you know as a new feature is defined you maybe have just like a quick meeting with your tech lead and you know in a dev and your UX person to break down this feature and estimate just that one feature and then you can just talk to the client about okay well this is about how many hours this is about how long it will take us to implement it do we move forward you don't need to do the full process that I described here some of the disadvantages this process requires client buy-in if your client doesn't get this doesn't understand why their budget may need to be revised why you don't why you can't possibly accurately estimate you know a $200,000 project based off of a bullet listed RFP then it's probably not a good fit for this process and you shouldn't try to run this process with that client um you know as I said this is something that we kind of sell from the very beginning this is part of our sales process we're explaining like this is what we do this is how we run our projects these are the advantages to you but if it's not a fit it's not a fit you know some agencies and some clients just are not good cultural fits and they shouldn't work together but so definitely need the client to buy into it um some of the other disadvantages so the technical planning meetings can be difficult to schedule it requires a lot of people and it's like you know one and a half to two weeks of two hours every day that can be hard to fit into a schedule I try to do it as you know early as I can uh that doesn't always mean that things don't get moved you know things come up but that can be a little bit of a difficult part in this process and then the meetings are really very long and tedious I think even for the developers I think they're definitely way more excited about talking about these things and figuring out how to implement it um but it it's draining on them too right it's a lot of working things out and discussing it and so um we're trying to improve our efficiency there and it's kind of a work in progress I'm just telling you what we do now and hopefully in the future we'll find a way to condense the process a little bit more maybe make it a little bit faster so that we like just instantly have you know all of our implementation details we'll see I will I'll tell you if we make that happen um so yeah that's kind of all that I was going to say to you today um thank you very much for coming and listening to my talk about our estimation process I'm open for questions if you have a question though can I please ask you to come to the microphone because uh this is recorded and they won't hear you otherwise yeah uh they are not yet but I will put them up my big question is who's paying for all of the time that you're spending in these estimate phases is that you guys or are you billing the client to re-estimate your projects within it yes so um that was something that I touched on earlier we do bill our time for the estimations because it's not just estimating we're doing technical planning these are you know it's important uh part of planning out their projects that it runs smoothly so you mentioned uh there can be difficulty with client buy-in but when you kind of unveiled this new process to uh the employees did you have any any struggle getting staff buy-in and how did you overcome that you know I don't think that we did because it was something that we kind of discussed together like we were coming from coming off of a project that you know the end result wasn't great we were all kind of frustrated by it so we discussed it and then there was the idea to do these two breakdowns came from one person in the team it wasn't really like this joint conclusion that we came to but when he brought that to us and explained it we were like okay like let's try that let's see if this works and when when we did and we saw the results that it brought to the subsequent project we were like well this is cool let's keep doing that and we tried to refine it a little bit over time but uh yeah it wasn't it is the people who were part of the team at the time when we started this process all immediately bought into it there have been some new additions to our team we've been like what is this process what are you guys doing we don't really get it until they've done it and then they like it so it could be a little bit more difficult to get the buy-in from new hires I guess who are coming into the team but essentially once they've done it they get it and if they don't then we can talk about that and figure it out together you know maybe they have ideas for how to improve it we're always open to that so thank you yeah uh first great great talk learned a lot I don't think as an audience we've applauded you yet so thank you uh so my question is with your early tech planning and your late tech planning you know you have your tickets you show them to the client and you'd be like you know homepage is going to take 12 hours if the client isn't happy with that how do you deal with when they say oh just take out this one item but it's maybe a core piece that you really need in there like if they just cherry pick what they want to get rid of to reduce hours so when we're doing our the first revision we try to steer them away from that type of cherry picking because the estimates are very rough at that point so it really is cherry picking isn't going to get them to the result that they want you know removing like a banner or something like the slider on this page that's not going to reduce their budget by $50,000 if that's how over they are they need to get a little bit more serious about like maybe we don't need this section of our site maybe we don't need and and so those are conversations that you just have to have with them just be really up front and say hey like we can do this if you want to it's not going to be very fun and it's really not going to get you the result that you want so maybe it would be better to take a hard look at some of these you know bigger features where the money is actually being spent and then when we get into tier two tier two sorry that's our second one that's kind of our internal word for it when we get into the final revision at that point it's generally okay if they do want to cherry pick because the estimates are pretty accurate so if they cherry pick enough they might get to the end result that they need for their budget but if it's something where we're like that's really a key feature that you should keep then we'll discuss that with them and maybe try and find something else that can be deprioritized and at that point too if they are just cherry picking since they're much smaller tickets they're things that you can you can always say look once we get this launched if you then have some budget available for like maintenance or whatever maybe you have you know maybe we're going to we'll talk about like 40 hour per month maintenance rider or something like that well those are small small improvements that we can be making after that point you know the site isn't static it's not going to go live and then never have any changes to it so maybe some of those things that were cherry picked can get worked on later make sense thank you again great job that was awesome I would assume though that before you get to the point we can kick off tier one tier two estimates you had to give the client a number to win the work to get to that point how much effort are you putting into that stage of it where it is all non-billable time it's not non-billable it's not yeah before you win before you win oh before like the proposal part yeah because at some point you have a number that was close to get to that point the work that we do to get to the initial proposal yes that is non-billable um well I mean that's kind of why you have margins so that you know you can absorb some of your operating costs like that um but the non-billable time spent there it's generally the sales team who we wouldn't expect to be billable anyway and then maybe maybe like five hours of time from you know tech lead and a UX person maybe they've had a meeting or two with the client just that they get to know them um maybe you know they've been discussing with the sales person trying to pull together the numbers I'm five hours is an absolute guess I don't know it also depends on how big this project is what we're trying you know to respond to as far as an RFP how big the proposal is um but yeah it would essentially be that's just going to be absorbed in our margins okay um but once once it is signed once we get into kickoff everything from that point forward is billable okay makes sense thanks so I've got a question about getting the client buy-in because when you guys first brought this on board you probably had a different model in which you were working with your clients so how did you you know you should probably already had outstanding contracts and whatnot how did you deal with that like getting these clients to come on board to a whole new process that probably to them look like it would cost them way more money so the first our contracts were always like time and materials so that part was always kind of clear that it's kind of going to cost what it costs for time and materials we're not saying it will only cost you whatever this dollar amount is so that part was kind of already taking care of contractually the first client that we did this with was actually UCLA library and they are just awesome people to work with and they got it so we explained the process to them we explained like we're going to try this out this is going to allow you to really get the features that you want because we're going to prioritize them you'll be able to prioritize them you'll see real value for each feature and they got it they were actually one of the clients who when they ended up being over budget after the initial after our second revision we ended up working with one of their developers for a little while and we were able to get them all of their features within their budget because of that and we just made it work together so I don't know if we just lucked out with that situation but it definitely worked they got it maybe if we had a ton of resistance from like our first you know guinea pig client on it we maybe we would have waited for the next one but we didn't know because they were cool about it so thank you and yeah I agree with that previous person this was a really good talk thank you very much thank you hi so my question is when and how do you estimate your project manager time I saw your budget breakdown which is really cool are you doing just a percentage of the overall work or because it sounds like you're just adding up whatever is a ticket so I would add a percentage of like total dev time but when I'm you have to keep in mind that when I'm adding up those final numbers there's a certain amount of time that's already been spent on design and definition and the project manager has been in most of those meetings has been coordinating that whole time so what I would do with those tickets is I would essentially so we have like a general meetings ticket we have a general PM ticket so I would look at how much time had already been spent and logged and then the estimate was that plus how much time I needed to get to the end of the project because obviously if we've already spent 20 hours in meetings if I have a 30 hour meeting ticket to get me through the end of the project I'm going to go over so I would take into account what was already spent and then estimate kind of based on that and what was needed to finish we've actually stopped filling our project management time on an hourly basis and have just started applying a flat 20% okay thanks hi I was really curious about product ownership and where that lies do you drive your client to sort of hold the vision of what their product is going to look like or do you help guide them in that way we drive the client to be their product owner because it's I said earlier it's their baby they're the ones who have to live with this final product it's not a project manager on our end it's their site so we try to find a stakeholder from their organization who can be that product owner sometimes they need some help and the project manager can be there to help guide them our project managers kind of also work as account managers so we're really trying to build relationships with these people and we want to work with them on a long term basis so if they need a little bit of coaching a little bit of okay so here's how product ownership works like these are some of the things that you're going to have to do internally we'll help get them through that process we want it to be a successful and enjoyable as enjoyable as possible process for them so thank you hi do you build in any more budget check meetings throughout or do you do it at like every sprint review or how do you how do you keep them apprised of the budget throughout the entire course of the project so we would do those weekly like status updates with our client and we would always go through the budget at that point do you track your time in Jira or where they can see it live or do you do it okay yeah all of our time is in Jira besides now that PM time since as I said we've started just adding that as a 20% percentage cool thank you I have found that in some projects when things are super complicated the developers or the tech leads need just extra time to figure out what modules they're going to use so they don't really know what modules they're going to use to figure out something at what point then in your process are they doing that figuring do they take time up front during the estimation or is it a wider estimate and they're doing that in depth yes so they would take time when we're doing our second revision that's when we're going to have a couple of developers together discussing each feature and really figuring out what module and that's why the meetings are so long because they may be maybe they're looking on Drupal.org like okay what exactly is in this module who's used it is it maintained looking at their different options so yeah they'll really be going through all those things in our estimation process and getting all those details in the ticket that doesn't mean that there's never a case when it doesn't change once we get into it maybe we do get into it we're like oh that wasn't the best option we can revise that it's okay well we may need to change the estimate as a result discuss it with the client that's the type of thing where the transparency comes into play where maybe we'll say look we thought we could use this module but as it turns out like they're not going to be maintaining it anymore or it doesn't do this thing that we thought it did so we're going to have to change it it has this effect then you know just let them let them know let them also decide you know where they want to go with it what the impact is yeah great talk thank you how do you deal or at what point you may not really know yet like demos for UX testing for example before we move into the full development process or design rounds on the front end so well the design and discovery part we I guess when we get into our tier one normally what I would do at that first revision is I ask my UX designer you know these been working with the client at that point for a little while so I ask okay how long do you think it's going to take you to get through the wires and so we do we do the kind of the same thing I was talking about for the project management or the meeting tickets where I look at how much time was spent and then I'm going to add to that how much time he thinks is needed to complete so I kind of get the designer the UX designer and the UI designer to give me some numbers around that stuff I also generally will have a ticket where I add a couple of hours for design reviews so that once things are are themed the UX designer can come back in and take a look and review everything and I also will have like QA tickets and things like that where I put a percentage so we can be going through so we can be going through QA testing and also that would include some time for revisions when like bugs come out of things okay sorry I can I can take more time but just I can't we have to get packing up