 I think I'm just going to keep going. Keep going. OK. I'm Zach Chandler. I work at Stanford University. My job title is Web Strategist. And I'm lucky enough to work on our own in-house Drupal team. And I consult with projects that go out to vendors. Produced many RFPs, hopefully not quite so terrible as some of the ones we might hear about today. But we've also started to look at alternatives to the process, so that's. Hi, everybody. Is this better? All right, I'll talk like this. So my name is Brian Scower, and I'm with Lullabot. I'm actually our sales manager. So I'm actually interfacing with most of the RFPs that come through our door. I'm kind of the first point of contact with them. And so I see a lot of RFPs, both good and bad. And I've been kind of exposed to the good ways of going about it as well as the bad ways. So we want to take just a moment here to recount some of our favorite RFP memories, because we've all been there. So who would like to go first? Brian? So my worst RFP memory was actually my first RFP. It was before my life in Drupal. It was the first RFP I'd ever done. I was a green salesperson. And it was a hosting RFP. And it was a really big name nonprofit that everybody probably would know. And it was probably like a 30 to 50 page RFP. And it came on my desk, and I was stoked. So I went through the whole process, ins and outs, answering every question possible. Hey, that's nice. And even got to the point where they invited us to come and present, flew down to the city where this big nonprofit is. And basically did several presentations in big boardrooms filled with lots and lots of people, worked out migration plans, spent countless hours of work on this. And ultimately, when the decision time came, the individual leading the project actually said to me, we're staying with our current vendor. Because if I do that, I'm probably not going to get fired. And yeah, that was a learning experience. So at Four Kitchens, we got an RFP from a very well-known Ivy League University in the Northeast. And we were very excited to do this work. And we got on our first phone call with them. And on this call, the two stakeholders who had put together the RFP said to each other, oh, it's finally nice to meet you. Good to meet you. Meeting each other for the first time. And then they proceeded to disagree on the scope of the project and what the project was about in front of us on this call. And the call became us just sitting there and listening while the two of them hashed out the scope of the project. And then we kind of ended the call with, all right, well thanks, everybody, for that hour. We'll follow up with you later. And we walked away. Because if these people are just meeting, and they're the stakeholders, and they're the ones evaluating the criteria for the proposal, and they're the ones that are supposed to understand the project, and they're hashing it out on their call with a potential vendor, bad sign they need to go back to the drawing board and rework what this project is all about. So how many people here are typically on the vendor side of RFPs, meaning you receive them and you respond to them? All right, how many people here are on the client side? You write them, you issue them, you evaluate proposals. That's about what I expected. There seem to be more vendors than there are clients at times. So let's talk about why these things exist to begin with. There's the good reason. And this is the honest, wholesome, everybody can get behind this kind of reason of there's work to be done and somebody is looking for somebody to help them do it. That's what we all think as vendors going into an RFP that we think that that's the process that we're involved in. Are we in agreement? Excellent. Now all the bad reasons. Can I add one good reason before you continue? Please do. I think it also announces that it's going to be competitive. And it's a well understood format. So if you've had any assumptions, I think it presumes some of those. OK. So the good reasons and now the bad reasons. Sometimes a client may feel as though, well, this is how we've always done this. They may think that they are supposed to write an RFP because isn't that just how these things work? We heard in one case where we received this RFP that just seemed to be very formal. And there were a lot of bullet points. And some of them weren't really filled in. There were a lot of these sections. And they seemed to repeat themselves. And it turned out that this client had just googled for how to get proposals and downloaded a template and then filled it out and just thought that, well, this is how all RFPs look. And I'm just going to fill out all of these empty spaces. And then, of course, it's a standard requirement argument. And this usually comes from big organizations and governments and things like that. How many times have you come across something like this? Yeah, I think, well, there's sort of the government nonprofit thing where it's a requirement. And then within big organizations, purchasing departments sort of exercise an authority over the process. And a lot of times you can be working with the stakeholders and then separately have to go through their procurement purchasing process, which involves an RFP, which nobody likes. On our end, the RFP process was actually an improvement over what we had before, which was a referral by a water cooler. And so you trust the person that you're talking with, oh, who did you use? And it was always the same two names. And so we kind of ossified this process. And those companies got very, very comfortable. So I think as a waypoint to this RFP-less future, the RFP process was actually a helpful step for us. And in a best case scenario, it can actually be that solidifying of the project requirements and internal buy-in and that really necessary research pre-step to going out and getting a vendor for things. But oftentimes it's also just kind of a necessary evil to projects of a certain size, which is part of what we're saying maybe it doesn't have to be. But for current reality, for a lot of clients, that's still just the case. It's a requirement, and they have to do it. So related to the requirement is in some organizations, there are rules that say if a project exceeds or is likely to exceed a certain amount of money, then it has to go out to competitive bid. And you have to gather three or four or five or something numbers of bids, right? Now this is under the batteries in category because that means that in many cases, some of the vendors who are replying don't stand a chance. And they're doing this just to fulfill some kind of bureaucratic requirement. And it's really not entirely fair to the vendor. And they usually don't know. They think that they are doing this, that this is all above board, that they have a real honest shot at it. But sometimes a vendor has already really been chosen. But because this project is $50,000 or $100,000, they say, well, it's got to go out to competitive bid. We have to get two more people. They're just going to go through the process. We say no to them automatically. And you get it no matter what. So just to provide yet another counterpoint. So I work for an organization that does have such a rule. And even when it's fair and we carefully choose excellent vendors that all stand a chance of winning, this does create a bit of a problem that I'm still trying to figure out. And that is, what happens when the vendor that you admire and want to work with, you know you have a future with, what if they lose several times in a row just due to the circumstances of the way on the merits? So I really don't have an answer for that yet. So something to think about. And now the really bad reasons. And these are the cynical ones. Sometimes a vendor exists. And a client issues an RFP just to price this project to gain leverage over that vendor to make them lower their price. So in other words, you're working on a project. You have a good relationship with this client. And then somebody comes along within the organization and says, hey, are we sure that this is actually a good deal? Let's put it out to bid and see what we get. And then they get some much lower estimates usually because they'll pick the lowest ones and say, hey, this company X said they could do it for half the cost. And company Y said that they could do it for 75% of the cost. Can you come down on this a little bit? And it's creating this sort of false competition and placing pressure on the existing vendor to lower their price. This is. We can all share that pain. Yeah. And chances are we've all been a part of this and we didn't even necessarily know it. Sometimes, clients want free consulting. So they issue an RFP. They get a bunch of really great ideas. And then they say, hey, thanks, everybody, for all these great ideas. And then they take it in-house. And they use those ideas as blueprints. So there are some ways that we as vendors can maybe get around this. There's the idea of no spec work, right? Never, ever do spec work. And also push back pretty hard on that RFP process to say, well, here's generally what we're going to do. Here are generally the technologies we're going to use. We're confident that they're going to work. But we don't explain how exactly we're going to put the pieces together. And it's unfortunate because we really want to show off our knowledge, right? We want to show that we have a really good plan. We've thought about the project. We understand it. But having to hide some of this information because we're afraid of something like this happening to us is really unfortunate. How often does this happen to you and Zach? Have you ever been on the other end of that? So I think this is awful. And we have never done this, ever. But something I just picked up on what you mentioned with not giving a full blueprint and holding back, I think that there is a bit of a tension there with how do we then judge the way you would execute? Because especially if we're comparing three excellent choices, one might be more in tune with where we think the technology is going and our API design, for example. Something that's outside of Drupal. So if we withhold too many details, then we may not have the right grounds for choosing a winner. So again, it's very true. I have actually gotten a phone call that said, congrats. You won the RFP bid. And we're going to do it in-house. So yeah, completely that, Frank. Hey, you're a top vendor. And we're not going to go with a vendor. Then we have risk aversion. So let's say you're a client and your boss or your organization comes to you and says, hey, we have this project. The stakes are really high. Your job's kind of riding on it. We need to make sure that this stuff goes off without a hitch. Go find a good vendor because we can't do it in-house. That person then is facing a tremendous amount of pressure. This is sort of an existential threat for them. So they put together some very onerous RFP or require a ton of detail and a ton of work from the vendor because it displaces that responsibility to the vendor. That person at the organization who feels that their job is threatened can always say, hey, they said they were going to do this and this. And as you can tell from these documents that I got from them, they put a lot of thought into it. And they did a lot of work. And it creates more of this defensive position that both the client and the vendor have to be in. Have you ever run across anything like this? I don't know if we have specifically because it's hard to tell what happens after what happens leading up to an RFP and then what happens after you win or lose it. Yeah, I think that, I mean, in general, everybody to a certain extent wants to CYA, right? And when I'm kind of evaluating RFPs when they're coming in, a lot of the ways that clues to that is if the questions are incredibly detailed to the point that they're not really related to the project at hand or the work at hand, then it sort of becomes, okay, maybe they're asking this level of detail in these sorts of questions about our financials and our background so that should anything go awry and the onus comes back on them as to, man, well, why did you guys end up with this vendor? Well, look, I mean, they have solid financials. They do this, this. And it's, you know, I think that a lot of times the way an RFP is structured, you can kind of pick up on some of that stuff, but I really don't see it quite so much, you know, I don't really see that as much on a regular basis or when I do, it's generally an RFP that we're not responding to. I think there's another way to read this slide as well and one that I've actually seen more often and it's not that the RFP is overly detailed, it's that the RFP is underly detailed and that can be a really good indication that you have a client that doesn't know where to start and we'll talk more about this later around, you know, clients who really don't have internal technical resources, but basically using an RFP process as a way to get their hands around a project, just get their mind around it in the beginning instead of doing the homework on their side and that can be another big red flag in the process of do they have accurate reasonable expectations around this thing? And finally, we have qualifying new vendors. So what I mean by this is clients, of course, want to develop a network of vendors. They want to have a whole list of people that they can contact when they have new work. They want to know what those people are about, the kinds of work they like to do, what their team makeup is, the types of projects they typically take on. This is a very valuable thing for a vendor to have, especially for a client to have, especially if they have a whole bunch of work stacked up. But sometimes they go about this in the wrong way and the wrong way is issuing an RFP just to basically gather resumes and to start to have this conversation without really intending to give this work to most of those vendors. And there's a much better way around this and this is something that Zach is doing at Stanford that we're gonna talk about in a little bit. But this is what we mean by qualifying new vendors. And I put this under the really bad cynical category because it's generating a ton of work for all these vendors who are responding to these RFPs that then are basically completely ignored because what the client will be reading is likely the boilerplate at the beginning and the team makeup and the types of work they'd like to do in their portfolio and things like that. So why should we even respond? As vendors, why should we respond? So we wanted to lead off with some really positive reasons and a best case scenario of how an RFP can actually be very good. So I may be kind of a mole on this panel. I don't completely hate RFPs because I've had some great experiences with them. I've been parts of teams that have gone out and won Sony.com, which and the version of it that won the Webby for Best homepage and a couple of other awards a couple of years ago, none such records, New York Stock Exchange. So I've actually gone out there and been parts of teams that have won the crazy big RFPs and there's a thrill of the hunt aspect to it and it can be a blast. It is possible, it's rare, but it is possible for a very well qualified team who knows what they're getting into and has really done their homework around this to kind of cinderella themselves into the big leagues to completely mix my metaphors. You can actually, you can get out there and that was in a couple of those projects where we're with a team of six. So very, very small companies going out winning a very big, big project that really ended up being a defining thing for the company and really a breakout project and it can go well. I will say that those RFPs were, those were not 40 to 60 hour RFP responses. Those you're really talking about upwards of 200 hours of research and 50 to 60 pages of writing and in some of those we went and delivered to the client things that they didn't know about their own business or their own competitive market and it was a completely different process than going through and wrote answering a form. But in some of these cases you, if you're an excellent fit and you have an internal advocate there, you really know what this is that you're getting into. It can work out. I would highly advise people against applying to the giant RFPs. And I, if you don't have that kind of prior relationship experience going after some of those big projects or a strong internal advocate. So I refer to this as practicing safe RFP. There are things that you can do to minimize your risks on this but we should also qualify a little bit here. There's a big difference between say a $50,000 RFP and a $4 million RFP. And both of those exist out there in the marketplace. And I've responded to both types and it's a completely different approach. So some of the things that you can do to evaluate whether or not you should be responding to this. I mean first off, in 99.9% of cases you shouldn't be cold responding to an RFP that's just posted out there or somewhere in the wild. It's just like bait looking for somebody to go, somebody has just gone fishing in that case and you can blow weeks of your firm's resources chasing after something that's just out there to get bids. So take a look at the RFP. As I mentioned before, taking a look is it doesn't have an appropriate complexity to the complexity of the project. Over under either way is a bad sign. If it's a $50,000 project and they want you to go spend 200 hours responding to this RFP, that's not a good sign. And likewise, if it's potentially a couple of million dollar projects and they just want a two page off the cuff response, that's also probably not a good sign that the client knows what they're getting into and can be a good client for you. It's a prior relationship or internal advocate. There are a lot of cases where even existing vendors have to, for these sort of internal requirements, have to keep resubmitting RFPs for every project. Oftentimes you get asked to write the RFP and then respond to it. So if you're a firm that doesn't have that prior relationship, you need to qualify if somebody else does. But if you do, then it can be a good thing for you. And the last point, and this was made by the founder of a great design firm I used to work for. And she referred, she made the excellent point of there's a lot of projects out there you can't afford to win. Especially the really big ones. The take into account your own ramp costs and lead times of payments, there are certain sizes of projects that can kill a small shop, even if you win them. Because the number of people that you have to pay before you're going to get paid can really, really affect your cash flows. So one of the points that she made was basically if you can't afford to pitch this well and lose it, you probably can't afford to win this project either. So it's something to keep in mind when you're evaluating what to go after. So here's another RFP positive story from Brian about the adjacent benefits of responding to RFPs. So yeah, I'm exposed to a lot of different RFPs and by and large, most of the time Lullabot doesn't choose to participate in RFPs unless there's a really, really good fit and a good match and some sort of benefit that we see into it. I totally agree with your message that unless, a lot of times, unless you have an advocate or you have some sort of in-road outside of the RFP itself, you're kind of the sucker in that game. What's the poker saying? If you don't know who the sucker is, you are the sucker. And I think a huge percentage of RFPs are structured such that one individual company is designed to win and it takes a lot to overcome that. There are great organizations like Stanford that do it the right way, but there are a lot of organizations that do it the bad way. So when RFPs come to us at Lullabot, really what I'm looking at is a lot of different factors and we kind of have a whole big matrix of different things that we're looking at to see if the project is the right fit for us. But one of the things that we consider that actually comes into play a lot of times with RFPs is are there any adjacent benefits to doing this RFP in the worst case scenario that, hey, we're gonna lose it. If we recognize that as a possibility and come to a point of comfort around that fact, then we can look at RFPs and say, should we still do it if we know that there's a good chance that we can lose? And this is very much an edge case, but there was a recent RFP that we did where we knew we had a really solid chance of not getting the business. But at the same time, we knew that it was with an organization that had a lot of projects and that we had a pretty good relationship with. And by making a good showing and understanding that, hey, we might not get this, but while we're making a good showing, we're essentially showcasing what Lullabot can do and showcasing the type of work that we want to get. In a way, it sort of serves to educate the client as to what we might want down the road. And so in that sort of circumstance, I see that as an adjacent benefit, really rare type of case, but that would be a case where I would want to make a showing on it and get the business, obviously, but if we don't, it's not the end of the world. So like I said, if you assume you're going to lose, and I think that's something that everybody should do in any sort of circumstance if you're doing an RFP, even if you do have the relationship and you know that your odds are really good, you know, just the mere fact that it's a bidding process and there's a selection criteria around it, assume that you're going to lose and then ask yourself, you know, is the investment and the time that I'm putting into going to be worth it in the circumstance? And another thing that we did, so, you know, another, this is the real, real edge case of RFPs, why they can be positive is you can use RFPs, they can be a good litmus test for new services and new products that you're bringing to market. You know, one of the things that Lullabot is putting some muscle behind, recently is our video service, which if you guys haven't checked it out, you should totally go to videola.tv and take a look. It's pretty awesome, video monetization. But, you know, interestingly enough, when we were doing that, we ran into a case where we had an opportunity to do an RFP and we knew that, you know, we had done market analysis and business analysis, but one thing, for all the negatives of RFPs, one of the things that they can do is they can give you a lot of questions that you've never thought of before and, you know, give you a little bit of insight into what your target client base is actually thinking and what's important to them. So in the case of, you know, this RFP for videola, it was a really, really helpful exercise and it kind of forced us in a non-threatening, low pressure type of scenario to, you know, encounter questions that are probably gonna come up as, you know, that service rolls out and reaches a critical mass. And for that, it was, you know, RFPs are, can be incredibly positive, but, you know, in general that's kind of the, that doesn't happen very often, so, yeah. So I guess the summary of that would be that sometimes RFPs can teach you about your strengths and weaknesses relative to some new market. Sure. Could you, well actually, let's do the questions at the end because we'll have to do them through the mic. So maybe you should not respond to RFPs. Here's a case study from Four Kitchens. We sat down and did the numbers and we took all of the projects that we won or lost that were RFP or not and we put them on a big board and we marked them. We were spending on average 30 to 40 hours per RFP response. That's an opportunity cost for us of about $5,000 to $7,000 per RFP that we write. We won only 22% of the RFP-driven proposals that we submitted, so that's our hit rate of 22%. But only 16% of the work that we do results from RFPs in general. So that means on the flip side, 84% of our work involves no RFPs whatsoever, just personal connections and word of mouth. So the things that we concluded were that RFPs have driven a small but not entirely insignificant amount of our business and it costs less for us to send a team of people to a potential client to meet with them in person rather than just immediately responding to an RFP. $5,000 to $7,000 opportunity cost against two plane tickets for two people, if you're sending two people, hotel expenses, some meals if you're covering that and opportunity cost of having people unavailable for those two to three days, less than responding to an RFP in paper format. So we started talking about this idea of not responding to RFPs in relatively closed circles here and there. We'd done a couple of small presentations on the subject where we explored some of these similar ideas about our numbers and how they compared to other companies and somehow the word got out that four kitchens doesn't respond to RFPs. And that word traveled very, very fast and we know of at least one case where a client simply ruled us out immediately because they had heard that four kitchens just doesn't respond to RFPs at all. So if you're going to adopt this kind of idea or if you're going to start talking about it publicly like we're doing right now, be aware that some people, unless you have a conversation with them about what you really mean when you say you don't like RFPs or your RFP averse or whatever language you want to use, make sure that you can communicate that thoroughly because there is that collateral damage of, oh, they just don't play this game. They're not involved in the game. Goodbye, we'll go with people who do. But you can try this at home, right? You should be able to calculate your hit rate. So list all of the RFPs you've ever replied to, one, and lost. That's your hit rate, right? How many people here have actually done this? Not bad, congratulations. Okay, how many people have not? You should do this, this is important. Then figure out how reliant you are on RFPs. So list all the projects you've ever won and then mark off the ones that actually went through the RFP and proposal process. How many people have done this? Not as many, how many people have not? Okay, a lot of people not answering, I understand. It's early in the morning. So here's how you do it. Calculating the cost of responding to an RFP. So we're gonna talk opportunity cost first. You take the time spent writing the proposal, communicating with the client, all of that stuff, time that you're spending on it as opposed to anything else in the world. Multiply that by your blended rate. That's your opportunity cost. That's how much you are sort of spending on writing this proposal. If you're doing in-person visits and other expenses, you need to make sure to add that in. And now you have the cost of responding to an RFP. Take this number and weigh it against the project's revenue. If that number is higher than the amount of money you may be making on this project, doesn't seem to really do you a lot of good, right? Also, compare the likelihood of your RFP win. And that's not only your historical hit rate, but also all the other factors that you need to figure out. And, Crystal, you have quite a few. So I'd like to turn it over to you, actually. Other things that you need to keep in mind that aren't just bottom line. So Crystal actually disagrees with me on this point. Okay, so it's not just, it's somewhat about the numbers. Obviously, that's important. And obviously, calculating what your numbers really are, that it's not against the total price that checks to the writing, but really what your profit is on those and having a good idea of that. But there are some big, I spoke to this a little bit before, but what is the likelihood of actually landing this thing? And that really gets back to it. We are kind of driving this home. Like finding out what their response market is for this. Like, were you asked to respond to this? Is there, do you have a good indication that you are viable for this project? Is it out to a big market? Is it a cattle call? Is it as, are you, as my former boss called it, cannon fodder for this RFP? You really want to avoid being cannon fodder. But other things, like, does this relate to a specific skill set, a specific value add that you guys have as a company? Maybe you specialize in video. Maybe you specialize in mapping. Maybe you prefer social action projects. Does, do any of those factor, are any of those factors involved in this, in this project? And, you know, looking at that, looking at it in a holistic way instead of just possibly getting, it's easy to look at, you know, possible budgets of these and get kind of dollar signs in your eyes. But, you know, these things can be very expensive, not only to bid on, but again, to actually execute. And it's really good to know what you're getting into before deciding to make that leap or not. Meanwhile, on the client side, Zack. I guess this is my section. So, I would say, like I did earlier, that RFPs have historically been useful for us. They did help us get to a slightly more rigorous process than we had before. And I think that is one of the key reasons why clients like them. It creates a sense of rigor around this process. And it's familiar to them. But there are other ways of creating a sense of rigor or real, genuine rigor around the selection process. It's just a lot more work intensive on the client side. But I think one of the things that you'll find is that the time that you spend in here ends up paying you back later. So first of all, do your homework, understand the shops that you're talking to, know who they are, know who's on their team, what are they good at, and what do they like to do? And then contact them personally and invite them to bid. So we don't do, or at least anything that I'm connected to, we don't broadcast RFPs anymore. And if my name is attached to anything that you get from Stanford, you know that you're already a finalist, essentially. So there's this question about the document. Now, the thing about an RFP that's problematic is that it starts with a document. It's okay to have a document, you just don't wanna start with one. So if it's not a specification, then what is it? So I've been working on this genre of document that I'm calling an expository sketch. And what an expository sketch does is it states the parameters of the project, it shows you that we understand it well, that we know what we want, and it also provides one plausible way of building it. It's not the only way. We're not telling you exactly what to do. It's not a specification where it's use these modules and this is exactly how we want it done. Now go make that. Because we know that we wanna be working with creative design professionals and we want your best ideas and we're not always gonna have the best ideas, maybe not even most of the time. So once the expository sketch goes out, we'll call for presentations from each vendor. Now in many cases, I'm already familiar with the vendor, but it's, so I have a client on the Stanford side and they don't know very much about Drupal or they certainly don't know who you are. So this is largely for their benefit to get a sense of your company to find out what it would be like to work with you and to see what kind of questions you have and how well you understand our project and how motivated are you to work with us. And so for those we offer to do those remotely or in person and I will say that the more personalized and more present you are, the more successful those tend to be for the vendor. So if you actually, like Todd was suggesting, you fly out, then we're never gonna say this but I guess I am saying it now. I have noticed the trend that those who show up in person tend to do better. And if you phone it in, it's quite a handicap that you have to make up for there. Then we'll have a Q and A period where we will aggregate questions that come in from the vendors about the project because inevitably because the expository sketch was written to be open-ended and interpreted, it's not a specification. There will be questions and refinements. So the process we're following is that any vendor who happens to contribute questions gets to see all the answers to all the questions. Once we anonymize them so you can't see who asked them. We want everybody to benefit from that in a very open source kind of way. Then, so a lot of work goes into the expository sketch that's kind of hard to make and it actually gets more laborious as we go down here. You wanna have a documented vendor review process. So we have a scorecard system where as a group, usually there's a group of stakeholders that is reviewing these vendors, not a committee necessarily. Lots of times it's a small team. And we make a graph of vendors across the top and we decide what are the selection criteria that we're gonna judge the worth of these proposals on and then we weight them. So there's a point system. And then line by line, by category, we'll say project management methodology and we'll give that 20 points. Then you score each vendor across and then you go down the line and sometimes we have seven or 10 or 15 criteria and the great thing about this is it allows you to give some weight to things that matter but not too much. Some intangibles, like what do we think it's gonna feel like to work with them? I mean, that might be worth five points whereas technical proficiency might be 50. So lots of times these are somewhere between 100 and 150 points and then what we do is we then break up, we tally our numbers and the great thing about this exercise is it can surprise you. It prevents you, any one person in this group that's deciding the outcome to have some arbitrary impression about one vendor in particular, they can actually score a lot higher than you thought they were going to and maybe higher than maybe the one you thought was gonna be in second actually has a higher total than the other one and these are not, it's not deterministic, these then start a conversation. We all come back in with our score sheets and we go around and we talk about them. It was essentially just a conversation starter but it allows us to have something to work from and then that goes into the next step which is writing up an executive summary. Now on the client side, this is something that the vendor doesn't ever really know about but usually at organizations like Stanford there is a stakeholder above the level of the group that's deciding on how to execute this project and they're gonna need to know how you decided this. Especially if you're maybe unseating an incumbent or something like that. If there's a vendor who has worked with your organization for a very long time and they might know the stakeholder that's above the level of people you're working with, it's really helpful to have to show a rigorous process so you can clearly state this is how we got there. So in an executive summary we'll, basically I document the entire process. This is who is on the decision making team, this is when we met, this is what the review process was like, these were the selection criteria, this is who we're recommending for this project and why. And those types of documents though they are, they require a lot of work to produce. I cannot tell you how valuable they are and how much they travel once you've done them. They pay you back tenfold. And then I guess my final point would be that the statement of work should be collaborative and iterative with your vendor. You don't want to hand them the statement of work or this goes to contract and this gets stapled on because like I said, if we're really doing this in an expository way, there should be some back and forth and some good ideas should be exchanged. So I also wouldn't just accept whatever statement of work they give you. So review it carefully, make sure it's what you really want because if that does get stable to the contract then that's what you're gonna get. And finally, all of this stuff is really great but it results in a ton more work for the client. But it's gonna yield a better project in the long run. So I want to make sure that we leave some time for questions and answers. So for the advice portion, I'm gonna kind of go through this rapid fire and if you have anything you wanna add, if you could just pop up your hand, cool? So we sat down, we've been talking about this for a few days, we sat down and came up with some advice to each groups of people. So our first piece of advice to the client would be require speaking to the people who will actually be working on the project. Sales people are there as facilitators but they shouldn't be the mouthpieces of the vendor, meaning they should not be speaking for what the developers and the designers and everybody will be doing to the exclusion of their input. So always make sure when you are talking to clients that you actually get some of the people who are likely to be working on your project in that conversation. Because after all, those are the people you're gonna be working with, not the sales people, right? Please beware of low bids. They're almost always too good to be true. These are companies that maybe aren't really paying a lot of attention to the RFP or there are a lot of reasons good and bad why they might be charging a lot less but please beware of them. And in fact, it might just be a good idea to take the highest and the lowest bids and maybe set them slightly aside and talk to them about why it's so low or why it's so high because chances are they're simply misunderstanding something or it turns out that they engage in a lot of really weird like subcontracting practices and this is how they can get away with it so cheap and you need to know about that up front. If you're comfortable with that, great. If you're not, you should know. Please pre-select research and vet vendors. That's something that Zach at Stanford does all the time. He's constantly engaging in conversations with potential vendors to see if there's a good fit in the future. And don't do open bids, just don't do them. You will have to read a lot of proposals as a result. This is going to save you time, a client. You will get, here's an example. I did a version of this presentation with somebody from Happy Cog a few weeks ago and his horror story was he asked a client how many people had received this proposal and the client said 364. And his response to that question was how many did not? Do not create an exact specification in advance because the vendors will likely have a lot of input on there and they may think that what you're doing may not really be that great and that there's a better way to do it or there's a brand new emerging technology that they may not know about that they want to tell you about. So don't try to create a very detailed exact specification. Instead, talk about what you're trying to do not how you want it done necessarily. Be sure that you absolutely check references because word of mouth is great, but it's not always right. So do ask for references, do follow up with them. Please to the extent that you can be open about your budgets because the vendors will let you know whether or not this is feasible. So this is something that I know Zach is going to want to talk about. So let me quickly say something from the vendor perspective. From the vendor perspective, we want some idea of how much you think you're going to spend on this project not because we're going to say, oh, $200,000? This is going to cost you $200,000, right? That's not what we're doing. We want to make sure that you're not coming to us and saying, hey, we want to build a brand new search engine algorithm to compete with Google. We have $20,000, go, right? Not feasible. It's very valuable for the client to know that maybe their expectations are way off and the only way they're going to find out unless they have a conversation at the very beginning is to receive a ton of proposals and bids that are suddenly way off the mark and then they go, oh, we screwed this up. This is actually way bigger than we thought. Sorry, everybody. We'll have to reissue this completely or maybe not do it at all now that we know that it's going to cost more. They may cancel the project. You've wasted absolutely everybody's time at that point. So, Zach? So from our perspective, so I, from my personal perspective, I completely understand and trust the vendors that we work with that it's really a sourcing question. They can give you a $200,000 version of what you're asking for or they can give you a $80,000 version of what you're asking for. So, knowing what they have to work with has everything to do with what they pitch to you. Now, from the people that I work with that aren't as familiar with these companies and the work they've done, it can sound a lot like, how much does this cost? Well, how much do you have? So, I'm interested in this notion of how do we establish trust? Of course, if you've worked with them before, that's a heck of a lot easier. But one of the things that I like to ask firms that I'm interested in working with is what's the minimum bid that you'd be interested in? What's worth your time? And then I will do my best to make sure that you don't ever hear from us on something that's lower than that. Because from my perspective, one of the risks is having groups from my organization because I can't funnel all of this. It's more of a, you know, I have a little spigot and there's a fire hose over here. Stanford is seven schools, it's not one school. And hundreds of centers with their own budgets. So, one of the dangers that I'm trying to avoid is if they don't have an appropriate budget, but they don't know that, the work sounds really interesting. And there's a lot of back and forth, there's a lot of time burned on it. And in the end, we could have skipped all that had they just known that it wasn't appropriate in the first place. So I get to funnel some of the traffic from Stanford that way, but in terms of an industry-wide best practice, I don't really know how you go about fixing that. Yeah, a lot of this just comes down to trust, which is something that we harp on quite a bit here. Right, I was just gonna jump in and make a quick point also that a lot of times, the reason why it's good to be open about budget is because a lot of companies use RFPs as a way to set their budget. And that just ends up wasting so many people's times on both sides and if that's the case, being open about your budget in the near term allows every vendor that's participating to understand that it's a fully thought out project and it's not just a wasted budgetary exercise. So vendors, so there are more vendors in the room. Advice to you, ask the clients how many other vendors they're inviting to bid, right? If it's 10, okay, if it's 364, maybe not. Be open with them, the clients, and get them to be open with you about these details because this is a trust thing and when you actually start to work together, you're gonna have to trust each other then, so why not try the trust exercises now, right? So this is what I mean. You need to start asking some very blunt questions to the clients. Why were we invited to bid in particular? The answer to that question is going to say a lot. They might say, oh, we saw you on a list of vendors on Drupal.org. Okay, or we heard about you. I have colleagues in another space who did some work for them and this project is very similar and we think you might be a good fit. Oh, okay, well that sounds really good, actually. You should also ask how many vendors with existing relationships are under consideration. We have 20 vendors that we all really love and we're all pitching it to them as well. Okay, well, may not sound very good for you and if they're ever cagey about any of these questions, like what's your budget, why us, things like that, you can always ask, how did you find out about us? That's a very typical conversational kind of question. Hey, thanks for reaching out to us. How did you find out about us? We'd like to know, we're curious. That will help answer a lot of these questions. That will elicit all kinds of interesting feedback. Ask if another vendor has been hired to write an RFP or evaluation prior to distributing this because there's a very good chance that that vendor is going to get the job, not you and you should just know what you're up against. Ask about their selection process. What are the selection criteria? If it turns out that one of the things that they're heavily weighing is we need a local shop and you're virtual or you are somewhere else, you need to weigh that in whether or not you want to respond. And just like the client, require a phone, video, or in-person meeting and pay attention to how these stakeholders talk to each other. So in the example I gave earlier, we did this phone meeting and our evaluation was they haven't actually figured out the project yet because all they did was talk to each other and not us. So pay attention to what they're saying. Do the stakeholders know each other? Did they get along or are they fighting? Because otherwise, you're gonna have to deal with that fight every single day that you're working on the project. This is a really big one that a lot of us forget to do. Ask if the client has ever worked on a project like this before. If this is their first website project ever, they will have zero expectations in terms of time and complexity and cost and all of these very, very valuable factors. I said this before a couple years ago in a project management talk, but good projects do not happen in spite of bad clients. And there are bad clients out there, people who are, and usually not because they're evil people, but they're inexperienced or they just have unreasonable expectations or it just may be a company culture that's very counter to producing the kind of work that we do in this industry. But make sure that you like the client. You're going to be spending a lot of time with them if you get this job, but make sure that they're prepared to be a stakeholder on a project of this size before embarking on this. I can't stress that one enough. Here's another interesting one that we normally don't think about. Does the client actually respect what we do or we just bunch of code monkeys or techies or whatever? Hey, you just do whatever, you internet people, right? That means that they're not going to respect the work that you provide them and they're not going to respect the product. So are they actually approaching you with some degree of respect about the complexity and the creativity behind what we do? And you can always offer alternatives to the RFP proposal process. Everything's a negotiation, right? So you can offer up things as opposed to just delivering a proposal to them, which if you're just basing everything off of comparing proposals, it becomes an essay writing contest, right? It's a numbers game and essay writing. And what we do for a living is not crunch numbers and write essays. It's make websites. So you can offer to write an evaluation rather than a proposal. And there's a really great article about this approach in Smashing Magazine from I think beginning of this month, very recent. But there's a short little link and these slides are available for download on the session page as well. Suggest an RFI, a request for information as opposed to an RFP process. And the request for information is low commitment and allows you to exchange just a couple of small pieces of information or a phone call or a document that says, hey, we the vendor, this is what we're about, this is what we do, these are the kinds of things that interest us. And then the client says, this is vaguely the project, this is what we want the goals to be, here's who the players are. And if you both agree to continue forward, then you can go through with the RFP. And that's gonna save everybody a ton of time. This, I joke that this was the speed dating of RFPs, but it really allows you to establish if you are a cultural fit with that company, which is another often overlooked but really huge component of this process is, are you really gonna be able to work together from a philosophy standpoint of what both sides are really out to achieve with this? And so even just a quick meeting, phone call, in-person meeting before you go through and base this all on numbers, can really save everybody a lot of time. And if it goes really well, then you've got a leg up when you actually are submitting numbers. If it goes really well, then you've got, you've established that relationship, you've got a bit more of an advocate in that organization and it's gonna go better for you. So where do we go from here? Crystal, do you have any final thoughts you wanna add? I feel like we've been through a lot of things that we hashed out here, but just it's a process and there's a wide range of things that an RFP can be from the different sizes, but a lot of these alternatives and being open to suggesting these other alternatives of even charging for evaluations, answering with a request for information, the four kitchens model of, hey, can we just come out and spend some time with you guys and get before we go through this process? These kind of things can really get around a lot of the pain points of this as well as just the overall. Just as we, the design community has done a pretty good job of saying just say no to spec work. The development community really needs to get strong about just saying no to open bids, to open RFPs, open things out there, because this is a big investment, especially if you're a small shop. This is a very big investment of your time and put it out there wisely. The payoffs can be huge if this is done well, but the costs can also be very prohibitive. So just be careful out there. Brian? So I would say my closing thoughts is anyone who's making an RFP or responding to an RFP, well, I guess making an RFP. I would say RFPs are kind of a, they serve a very specific function, and I think that you should try to be as open as you're asking the respondents to be with your own processes, your own selection criteria, your own budget, your own stakeholders, and really get to know the people that you're asking to respond to those bids. Because at the end of the day, the RFP is kind of the criteria and selection process around how a decision is made, but you're gonna be working with somebody. And at Lullabot, I know for sure a huge, huge portion of what we evaluate when we're evaluating projects and clients to work with is if they're a culture fit and if we want to work with them, if they're good people that we enjoy being around and that challenge us in new and interesting ways, and those sorts of things don't come through an RFP unless you open yourself to that. And when you do that, I think you get a lot better bids in return and you end up with a really good result as opposed to just putting it out there for someone to bid. Zach? For clients, I would say do your homework, understand your own organization and its institutional preferences, and think of vendors as partners. And I would like to suggest that if we all stop responding to RFPs, they will stop writing them. But what we really want to do is maybe eliminate RFPs, but of course you can't eliminate the bid process. We want to create a responsible bid process. So we want to leave some time for Q&A, but I realize that we're pretty short of time, so if you need to leave, that's fine. We have some resources listed on the slides, on the website, so I'm just gonna skip past those. If you could leave us some feedback, that would be really great. Thank you very much and we'll have some questions immediately afterwards. So if you do have questions, just feel free to go up to that mic there so they can record it for the video. Hi. I think it is. Yeah. Hello, okay. I know you kind of mentioned that the RFP process, traditionally speaking, is more or less dead. Not necessarily in those words, but is it appropriate for clients to ask vendors for their recommended RFP format or their document template if one exists? How, is it appropriate to ask for, how do you want to receive an RFP? Yeah, I'm not sure I understood that. I didn't quite hear. Yeah, could it be possible to turn up the audience mic? I believe that the question was whether it's ever appropriate for a client to ask a vendor for their preferred response format. From my own experience, that would be a huge red flag to me as a vendor that would say this client hasn't done their homework. And so I'd love to hear what the other panelists have to say on that, but it would come off to me as really wishy washy and we don't know where to start with this. As a follow-up real quick, is there an approved, approved, loosely speaking, an approved format or a recommended format to create your RFP? Because then you have some that are 30 pages long and they have all kinds of project points or are there a selection of key points that this is what we need to know right now. Everything else is fluff. I've seen some long ones that are just awful. I can't imagine taking up 30 pages. And I think that just in the spirit of the panel I would say that there is no approved format that we have and we have used them in the past but we're very open to moving away from them so long as we can establish rigor around the process. Thanks everybody, this is very helpful. I run a small Drupal shop in Boston. We work for a lot of non-profits and many of them have really no in-house technology resources at all. So Zach, your idea about the expository sketch sounds terrific, but it seems to assume some technology knowledge on the part of the people writing it. Do you have anything that you would recommend for a small non-profit or like a small, even a small university's plan for writing something like this to get around the problem of not having very much technology knowledge? No. So in cases like that, yes we did, we actually talked about this a little bit. We talked about this a lot yesterday. The idea is if you don't have in-house technical resources to help guide your RFP or whatever the equivalent is process, you can always hire a company to do this for you, to do the discovery for you to write an evaluation. Yes, you can do that and a lot of companies do that for flat fees. So they won't necessarily be doing it time and materials by the hour or something like that. So you can always ask vendors like, hey, you guys are great. We have a project, can we pay you to help us write our RFP or evaluation? Awesome, thank you. Sure. Very informative. I came from a corporate background, so I've been the client. I became a vendor as a freelancer and now I'm kind of growing my business. I track all my time religiously and I have a couple observations. One strategy you can use is to track your time that you're putting into an RFP and work that into the quote. Sometimes it happens that it takes a long time if you are tracking your time. That can happen. The other thing that you have to watch out for if you're a small studio is you have a really sexy bid come in and you're like, I gotta drop everything for this. But sometimes you're dropping everything in a really critical moment with another project and that is another cost that you have to consider when you're doing this. Absolutely, good point. Thank you. A question for mainly Zach, I think. Most Drupal firms have now adopted some sort of agile development methodology, yet RFPs encourage more of a waterfall approach around a defined budget and scope. So I'm curious if you've looked back at the historicals around your projects around how much of the cost that the vendor originally bid came in, how accurate was it? And how much do you account for iterations and agile and things like that? Well, first I would say that we're totally open to agile processes. We have seen some budgets go way over that were a fixed bid. And recently I've gotten some access to our procurement records and I've seen a history where a project will start out and then go through some phase and there's this string of change orders. So I don't know if we're really even, I mean, going back years, I don't know if we were even really paying that much attention to the total cost of some of these. So that is with a more waterfall approach, I think that in a fixed bid, that is cost overruns is something that we have experienced recently. And with a more agile approach, I think one of the interesting options that we have at our disposal is time and materials based billing with a not to exceed amount. Because if you're doing agile right and you have a project backlog, you're gonna see these kinds of problems coming way before you get to that point. Agile takes a lot of trust and I've pitched in one actually several RFPs that started off very waterfall written and we've ended up actually winning them on an agile approach. It's a difficult thing to balance as far as how much you promise and how much you try to give them a roadmap for what the agile process is going to be like and setting expectation around those fees. It's possible, but more than anything, and I know Todd has a lot of experience with this as well, it's that client has to have a lot of faith in your team that they're going to be getting a good amount of value for every sprint, when it's not laid out explicitly with deliverables and an SOW of exactly what and when, that they're going to be getting very high value from very fast thinking, good, solid people to give you that kind of free reign that an agile approach really requires. Right, yeah, the agile and how it fits into this process is sort of a discussion unto itself and there's all kinds of ways that we get around it. Unfortunately, there's not nearly enough time to go into our process right now. So about five minutes over at the moment, if we could just do like last three questions real quick, that would be great. Sure, great panel by the way. All of you, Zach, I appreciate your transparency with your organization. My name's Darren, by the way, but I'm just curious to get your thoughts on building more trust in the RFP process at the time of which you're asking about budget. We encounter so much resistance at that point and without revealing too many trade secrets, I can imagine, just Brian, Crystal, Todd, any thoughts on that, I'd appreciate it. So I kind of take the approach of asking for rough orders of magnitude, understanding that there's a really natural and valid reason that people will not want to say, my budget is exactly this, but by doing rough orders of magnitude, 10,000, 100,000, you sort of can get a feel for what they're thinking without pushing them too hard on that topic. That's usually what I do. For my part, I get them to tell me and then without disclosing a number, I talk to the vendor and I let them know if this is appropriate or not, because I have a sense of what, you know, about the project itself and the vendor's requirements. Sure. I just try to get as much open honesty out there as soon as possible and if that counts against you, that's probably not a good sign for the client, but if you're willing to put that out there early on and be honest about it and not be trying to play games or, you know, get more than, you know, if you're not trying to play games with it, just be honest about where you're comfortable taking on that project. You can get things off to a good start that way with just some really upfront honesty. Thanks guys. Thank you. Yeah, hi, my name's Ian, it was a great presentation. I just started a new job. This is week three. We have no process. We're a client company dealing with vendors. So I have a specific question to the gentleman from Stanford. You said that when vendors ask questions, you share the answers with everybody. Does that give an unfair advantage to, you know, somebody who's working, somebody who's not working hard, but is reaping the benefit of somebody who is. They're doing all the research, asking all the good questions, but everybody's getting the benefit. Well, I think the fairness component, thank you. I think the fairness component comes in where if you didn't think to ask any questions, you don't get the benefit of that process. But I mean, to be honest, we haven't done this too many times. We're still trying it out. And we happen to find that a lot of the questions were similar or were nuanced around the same sorts of ambiguities that might have been in our sketch, but were qualitatively similar so we didn't feel like we were, you know, giving an unfair advantage. One final comment. I'm going to be developing my own process. If anybody answers, isn't there a similar vote and will be interesting in collaborating on developing a client process? I'd love to talk to them. Great. So to speak very quickly about the Q&A thing, Zach and I did this and I found that process very useful because I don't think there was a vendor that sort of asked like one softball question and then anticipated getting a whole bunch of really great feedback from everybody else. People who have questions have a bunch of questions. There's no such thing as, hey, I read all your RP. I've got just one tiny little question. No, you have a ton of questions. So a lot of the questions did overlap. A lot of the questions that other vendors asked were very good and there were things that we didn't necessarily catch. So the idea of if you participate in the Q&A, you benefit from the Q&A is actually, in my experience, has been tremendously fair. Hi guys, my name is Terrence and great presentation. Thank you. My business partner Dustin and I have a small shop in San Diego that we launched last fall and I believe that our view is that the RFP process is a good way to market ourselves for the time being. And so I'm curious, Todd, from your perspective, do you believe it would have been good for four kitchens to adopt the no RFP model early on or did you actually benefit it from it in the early days? Our first big jobs didn't involve RFPs. People saw work that we had done that was not client related work but kind of stuff that we did for fun. We find that marketing ourselves in terms of outside projects and open source involvement gets us the most business, not the RFPs process. But I do recognize that it's unfair to say to people who are getting started in the business, oh, RFPs are terrible, don't do them. How else are you supposed to get business, right? It really comes down to even though as a new company that perhaps isn't very well known or you're just getting started or something, even when you engage in the RFP process, somebody has to take a leap of faith on you because you don't have the portfolio, you don't have the experience, you may not have the reputation. So whether you do RFPs or not, it comes down to somebody taking a chance on you. You're gonna get a better ROI, especially in this community by getting out there and contributing back to the community and blogging about that, contributing modules, contributing themes, getting active in the camps and here, and that's a similar time, it can be a similar time and cost outlay and I think you've got a much better expected return from getting that stuff out there and you're helping the community at the same time. Thank you very much, everybody. Thanks, guys.