 So it's a 545. So that means that we're gonna start right on time, right? This is the custom of the land, I understand. Everything runs like a German train system on time. Actually, I'm gonna kind of hang out and talk and ask some questions, and then we're gonna dive into it if some people come. So my name's Andy Kouharski. I'm gonna say all this stuff again. I'm gonna talk, this is a business talk. I'm gonna try to slow down. So hand, do we have some folks who are running businesses? Okay, so you guys are running businesses, and where is everybody from? Texas, whoa, Texas, that's Denmark, Denmark? Okay, is this, where are you from? Okay, so this is the, okay, if anybody's, everybody from South America, you cannot sit on that side. Welcome, welcome. I was just in Korea, a couple, who do you work for in Korea? Which company? Open, OpenF? Great, and where, Brazil? US, you gotta go on that side, Ron. Mm-hmm. Oh, okay, okay, you're, Peru, Peru. Colombia. Colombia. California. California, come on, you guys are getting it wrong. Brazil. Okay, we're gonna start in one minute. Okay, so, I guess I might be being recorded right now. I am, okay. So my talk is going to be about measuring everything, and it's a personal journey of many, many mistakes that we have made in my company, and so I'll share those with you, and it's an evolving journey, so I am here, of course, to learn from you guys, and hopefully I can share some things that will help you avoid and make some of the mistakes that we've made. All right, so I am, my name's Andrew Kuharski. I am the founder and president of Chromat Source. Those are my kids, and I think they are making a snowman, wondering where their dad is, because I'm always on the road, so I do travel, and I do Drupal stuff, and I run Chromat Source. I've been doing this, well, I've been in consulting and manager and running business for about 20 years now, and as you can probably tell from the previous picture, I am from Chicago, and in Chicago we are Chromat Source. We are a Drupal development shop. Founded in 2003, we've been doing Drupal for about since 2008, so that's like seven years now. We offer 24-7 support, and we focus on custom development, and this is important for the talk, so we do custom development. We also provide DevOps services, so we have DevOps philosophy, so we have support services, system administrators that work with developers and our support team, and we provide support to our clients, and that is a separate service line. So I like measuring things, and that's really why this was a fun talk for me. So I was just in my hotel, and I realized that up to today I took 4,878 steps. I use Fitbit, and I'm happy to report that right now I am up to 6,767 steps. If I keep on walking, it's gonna help me. So I like this stuff, I like statistics, I look at numbers, so I see how much I sleep, and just because I look at it, I know how well rested I am and how much I more sleep I need. Not really, but I also like competing with some friends, so everybody has some Fitbit right now. I'm pretty good doing pretty good right now, but maybe if you have a Fitbit friend me, and we'll compete. Another measure, I have this scale at home, and in the last 2.8 years, I've measured, I've weighed myself 864 times. Luckily, as you see, this trend is going down. It's good. I hit a milestone right after Christmas, the heaviest I've ever been, but it's trending down, right? That's important. Obviously I like numbers and I like measurement, but why is that, what does that have to do with business? Well, I put something in the beginning of this talk about numbers, which inspired I think the DA to accept this and vote for it, but I think it's true. Data is important, I like this saying, and God, we trust all those bring data, and the second one is really important, and I think that what gets measured gets attention and gets results. I truly believe that. So some more words about metrics, right? So besides the personal inspiration we get from these metrics that help us motivate us to lose weight or walk more in business, they tell us about what we're doing, right? So that could be used for marketing. Are we committing enough? Are we delivering on time? Do we have less bugs? You know, we have what, 56 critical defects in Drupal? Well, that tells us something, right? We still don't have an update, but it's less than 149, right? So that's good. I mean, it's a little bit of marketing, but it's good information to share with your shareholders that gives them a sense that it's not just your opinion, but you're backing your information by some fact. Okay, well, 56 critical tickets, that means that it's going down, that's good, we're on the right path. And it's a good way of communicate, but I think the most important part of it is, and I've noticed this around my teams, and I think others, is that what gets measured gets results and it gets attention from your team. So if you're running a business or a team and you're looking at measuring some instance that you wanna improve, for example, your team will notice that you're doing it and you're doing it on a regular basis, and it'll hopefully get the results that you're looking for by measuring that. So, before we start looking at more numbers, I think your CEO from Thayer, Thayer, Thayer, Thayer, did a really nice job on the earlier presentation. I encourage you to see it online about setting vision and starting with why and having a passion for what they're doing. And I think that measuring all things also starts with this thing. You start a business or you start a venture and you think about why, why are we doing this, why is this important? And that drives to the mission and that starts, Dries talked about this at the Acquia Summit yesterday and talked about, well, at Acquia, we started writing down what are we about? So he's got it down to five sentences, I think he said. And then from that, you get goals and from goals you get, okay, you break that down to objectives, what are we going to do next, what's next? And from that, you get KPIs, right? So we're the key performance indicators and then you break that down further into metrics. So that's the kind of exercise that we like to go through that I think is important. So for example, my management team on a weekly basis, we have a set of numbers and it's not too many, it's just few numbers and we thought about it really hard at the beginning of the year. What do we want to report on? And those numbers, they may be responsive before the developers or the other teams that generate those, but it helps us drive the business and it starts with the top. So what do I get evaluated on? I'm a CEO of a small company, we're about 30, 40 people depending on how you measure, maybe up to 50. My primary way that I get evaluated is financial performance of the company. So that is revenue and net profit. Okay, well what about all these other things? What about giving back to the community? What about client satisfaction, which is really important? What about your retention? Are your folks happy? I believe that it's my job to make sure that we have good financial performance so that we can achieve all these other things and anybody else doesn't have to worry about those things. Once we have that footing, we can do things to increase client satisfaction and we can do things. We have the time for developers to go to camps and we can do all of those things. So my primary job is very simple, right? I get measured in hard numbers. Did we hit our profit objective? Did we hit our revenue objective? Yes or no, right? Luckily, I can't get fired, maybe not yet, but so it makes it a little bit easier, but it's key, right? There's also a concept of a balanced scorecard. So that's a concept of measuring a lot of other things besides that one number and that is also important and we look back and then we take a look at it from an annual performance basis. So we look at those few hard numbers for our team and for myself and then we take a look at a balanced scorecard. Okay, well, now that you've hit this number or maybe now that you're having a problem hitting this number, let's take a look at these other activities and other parts of what we're doing to see whether we're hitting that. So we look at financial perspective and we look at financial perspective for all of our key members. We look at customer perspective. So this is very, very important to us and the second part of my presentation is really mostly about customer perspective, internal perspective. So are we creating people that are growing themselves? Are they enjoying what they're doing? Are we doing good things? Are we growing professionally? That's one of my goals as well, right? So I get measured on financial performance. My personal goal is to grow as a CEO and become a better leader. So those other things are important but they're not as clean cut as did you hit your number? So just a little bit of balanced perspective here. Oh, Karen, I didn't know you were gonna be here. So this is Karen. Just a question. How am I doing in terms of not this side? Am I doing good in terms of speaking too fast or good? You guys are okay? Perfect. Uh-oh. Okay, so I'll let you know. Yeah, I walk around, I'm pretty good with that. All right, so further. So this is Karen. She's one of our project managers. She just so happens to be sitting right here. Last time, last time I gave this talk, she was not, she was another continent, so sorry, Karen. So Karen has some measurable objectives in terms of what we're looking for and that is project profitability, right? So we share project profitability with all of our project managers. So we have certain goals. At the end of the year we review where we're hitting, whether we're hitting those goals. Client satisfaction. So there's the financial objective and there's client satisfaction. And we would almost put that, I should have put that one as number one. Our net promoter score, we're gonna explain what that is, is number one, our clients happy. Team growth, revenue contribution. So is Karen managing big projects, small projects? How are we doing? And is she supporting us in terms of marketing? And by the way, so here's a plug. Karen has a session tomorrow on women and technology. Really highly recommended, right? So we try to measure these things in real time when we have real numbers associated with that. So COBS is one of our solutions architects. He's a senior development on a way to essay. And so what are our essays get measured on? Estimation of projects. Are those projects accurately estimated, right? We do a lot of fixed bid or RFP responses. But even during sprints, right? Are you able to achieve the things that you're planning during the agile process? Those are important for us. So are we, are those estimates realistic and are they being hit? Correct, architectural solutions. That's a KPI. So how do we measure that? We measure that by defect counts. We take a look at QA duration, QA process. Are those things bouncing around a lot? And then we have a number of quality of code delivered. We were looking at number of defects in warranty period. We take a look at the feedback that our developers are providing. We have a, it's a, before you commit, you have to get reviewed, right, Dougie? So we take a look at all these things and try to get numbers around that. So again, Doug, you weren't Dougie, not a developer, didn't expect you to be here, but just using as an example, don't take it personally. Uh, so what are your developers get estimate? Yes, what are your, what are your key measures? So I'm gonna ask for the internet audience that's going to happen afterwards. I'm going to pass my, what are your developers, get what are your measures that you use to evaluate developers? Okay, so right now we are using the, how can I say this in English? Let me check. It's like we are measuring our performance through the pipeline of our trial process. So we are trying to measure quality against bugs and the code that is delivered. We also measure the estimation like you guys and also the time of, we have to get the result of the value of our delivery. And so it's something like if we understand our expectations of our clients. So I think it's like a really major indicator of the quality we are trying to deliver. I wanna try some Q and A. I'm actually really curious on how you achieve those, you know, key measures, right? Cause that's sometimes hard to measure. How do we measure? Well, that's a lot of things we measure. It depends on the company, the maturity level the company gives for the project. So because we work with CMMI maturity levels, so I've worked in projects of five maturity level and three and it carries a lot of weight with that maturity level. So we do code reviews depending. It's almost like a standard already. So we agree with the client the number of bugs that we go with releases. So for us, it's like first internal standards and then agreements with the client. Yeah, CMMI is a really interesting model. It's been around for years and years. It's really cool. Anybody else here to? So it seems like we're pretty aligned with what we're looking at. And the one thing that we also keep track, like administratively, that's an administrative part of us is are we submitting the things that our developers hate to do, time sheets. Keep track of whether those are complete on a weekly basis from everybody in the entire company. So we also do support, right? So we measure a lot of things around support. Some of this stuff that you see on this slide in the next one is borrowed from a call center. We take a look at, we have SLA, so service level agreements with our clients around SLA. So we talked about whether we're hitting those. So response, right? So in a typical call center when you submit a ticket and measure it against whether you hit that response, you have a call, a massive call center, so abandon rate, people, they get through the issue. So the same thing here, do people actually stick with asking that question, issue, how long does it, do they wait until they get to somebody or whether they can submit it? So another call center thing is average speed to answer. We also take a look at, do we answer that response to the particular problem? Response time, and again, another call back, those are really call center issues, but they do apply to a support the way we do support. Of course, resolution, right? So just because we have an SLA that says within this critical of an issue, you have to answer it with this many hours, there's also a resolution, right? That's even more important. Just, I mean, it's nice to have somebody say, yeah, I got it, I know your problem, but it's nicer to say, okay, I understand your problem and here's your solution, right? So we take a look at that, whether we are getting those issues closed within the time we promise our clients. And then, of course, it's really important to see whether we have tickets that are reopened and that also gets tracked to the person who's closing those tickets and the people managing those. So there's lots and lots of other support metrics so we want to provide high client satisfaction around support, that's a really big part of our business. So we take a look at things like planned or unplanned time, we take a look at whether we, so we trend that over time, we try to see differences. So we know, and this is what was very interesting, this topic of queue, wait queue and how much time is, or how busy your team is. So can we answer, do we know, we try to predict how many, how much unplanned work we're gonna get on a weekly basis and then go back to some of this. We don't always, we don't keep track of all of these things but just to give you an idea of all the different measures that can be taken in a call center. So of course in project planning, we use JIRA so we're starting to employ more and more type of measurements in terms of our projects. So, and we do both agile and fixed bid. So we take a look at burn down rate, whether we have our budget being used versus the number of features that are being completed. We take a look at open bugs, so all of these things and we just recently adopted JIRA so it's making it pretty easy for us to do. Take a look at, you know, we're in sprints, are we ahead of budget, under budget. And again, going back to time reporting is pretty key. Actually, it's not only important to measure those on a micro scale, on a sprint or an issue or an estimation, but then to kind of pull up and take a look at it. I don't know if you guys had this experience before, we came to a project where somebody just wanted, a client of ours just wanted a little bit of help and this is a large project they've been working on for over a year. And we said, sure, we'll take a look at it and we took a look at it and we sent one of our project managers to kind of do a deep dive and they used JIRA and they said, we just need a little bit of help because we want to finish it in a month. And we said, all right, let's take a look at this. So we looked at all their take hits and we said, all right, so are these estimates correct? Yes, all, you know, these hundred or thousand issues there or whatever number of issues they're outstanding, all these estimates are correct. And we said, well, that's interesting. Are you increasing your team? No, no, no, we just want a little bit of help. So we said, all right, and you're communicating that you're gonna get done in a month to your stakeholders. Yes, we are. Well, if you're not increasing your team and your estimates are correct, that means that you have six months that you can complete those estimates with your current team. And it was like, whoa, they're stakeholders totally blindsided by this, right? So it's good to roll that stuff up. I'll just look at it at a micro level. Yes, your question. Are you using JIRA with sprints for SERVS desk for support teams? So that's a very good question. It is a little bit outside of what I'm talking about here, but yes, we are using JIRA service desk and we are actually, we just transitioned there. So I don't have a full result, set of results there. Yeah, so this is a great discussion because there's a lot of measurement around support. So for example, JIRA has a service desk. JIRA is a tool, if you guys haven't heard of it, it's a great tool. It's really a bit like Atlation. It has a lot of plugins, including Get Bucket and all those other things. And we've adopted it recently. One of the nice premises around it is that it keeps track of the SLA. So if you establish SLA, service level agreements with your clients, you can say, okay, a high level priority issue, I'm gonna address it in, let's say, five hours or 10 hours. Whatever you agree to do, whatever your capacity is. And then you can see how quickly you need to add on, react to that issue before it crosses the SLA, right? So the question is, how do we use it? And so on, I can have that conversation with you a little bit later. Actually, my support director is running this and I'm not that familiar with it, but I can tell you what I know afterwards. So in terms of project reporting, of course we take a look at big projects. We analyze issues, how many are high, medium, low, and this is one of those metrics that's really key in making that happen. And you can report back to your stakeholders what a likelihood is something that's gonna succeed or not. So these are all great and we implement them in our daily lives and we also do a lot of retrospectives so we can learn from those things, how much time we spend on projects, how much, you know, where do we spend our projects? So we look at like this, we did this a while ago, but this is a pretty highly focused development project. We actually developed for like 57%, the rest of it was spent in other activities, right? And this was like a development project. So retrospectives are important for future ability to estimate and lessons learned and be able to do a better job on future projects. Also take a look at capacity planning, looking at how much burn down you have per developer. This is another small project as you can see this is pretty old, but retrospective estimation. So keeping track of time and what your time is spent on is important for us. So we were doing all of these things and we felt like there's something that's still missing, an important part that we wanted to capture. And this was around, you know, quality is really hard to measure and client satisfaction we thought was pretty hard to measure. So we thought about this and we started doing some research and we really, really found some great things. And I'm gonna talk to you about this for the rest of this session and why we made this a key part of what we do on an everyday basis. So we made a decision late last year to make focus on less, focus on key numbers but less numbers and make a company goal focus on one number. So we started doing some more research around client satisfaction and how we can measure that and how we can do better. And some really interesting things came up when I was looking at this. So I just wanna share those with you in terms of for those of you who are running your businesses and companies, right? So I always, while I didn't love it because customer complaints usually come at a time like right before I'm giving a presentation at DrupalCon or right when I traveled for 20 hours and I just wanna hit the pillow but I need to hit that. But those are really valuable. So I did not like them, right? Why? So just like a bug, I was at a MySQL conference many, many years ago and a keynote said the most valuable thing all the developers can give me is a reproducible bug, right? A client complaint has by some pretty decent research here many other clients would never bother to say that. So the client raises their hand and says, hey, I have a problem with X, Y and Z especially if it's tied to a process we try to stop everything as you guys stop the line and try to figure out what that is because behind that there's many silent voices and then that means that we need to address it immediately. Another interesting thing, 2% increase in customer retention is the same as decreasing costs by 10%. And I wanna talk a little bit more about that but that's great, right? So if you're running a business or you're running a team or you're responsible for a P&L making your clients happier is basically like cutting costs. So how many of you guys had problems flying here or had some issues? Yeah, right? You talk to, did you tell people about that? Yeah, hell yeah, right? Like everybody, I complain about travel all the time. Talk to Dries, he's like, oh, it's airport. So unhappy clients will tell many, many, many more people about a bad experience. But happy clients may not. So another goal that we have is not to just make happy clients, make really happy clients, make help them, make them feel like they need to tell your story. The other thing that we also are starting to focus on is that, and this can be like a tough thing in engineering, right? Because we deliver a project on time, we deliver it on budget, and we deliver everything that the client asked for and they're still not happy. What? Well, there's an emotional part of interacting with another human being that needs to be satisfied as well. And for us, a lot of the feedback that we receive was really important, didn't really have to do around the areas of improvement. Not many of them had to do around code quality. It was perceived very high, the value was perceived high. It's really about that emotional, it's about that emotional connection. And so it may be difficult for engineering and a software team that's kind of, and especially me who loves numbers, how do you make that emotional connection? So it's key, but it really does matter. Oh, and by the way, notice this one, right? So you can charge more if you provide better service. Like they will not walk away if you charge more. They'll be happier because they want better quality service. And I don't know how much money or time you guys spend on marketing, but it costs and we haven't done this, we should do this, but I wonder if it's six or seven times, it costs a lot of time and money to acquire a new customer. If you really break it down, right? The amount of marketing, blog writing, your website, we have a marketing person, basically all they're in charge of requiring new customers, right? So how many new customers do you get divided by the salary? Then how much time do you spend on the phone, negotiating a contract, all of those things, a lot of money. Keeping that customer just gotta do a good job, right? So it really does make a lot of sense. The last thing, another fun fact I found during my research is that, and I think Seth Godin wrote this cool book and that says all marketers are liars. Well, they're not liars, they should be telling a good story, but people tend to believe people they know. This is the blue line here is the percent that folks trust completely. That number one here is recommendations from the people that you know, right? So if you tell me that Lufthansa or Austrian air or whatever it is that you flew here is terrible, I'll probably avoid it, right? But if I read it that they're great on a flyer, I'll probably not believe it as much as I believe you, but I don't know you. So anyway, word of mouth, right? Your customers keep them happy, they'll give you more business. So I hope I've convinced you that this is really important. So as a business, we really made a decision to focus on this, but this talk is about measurement, not client satisfaction. So how do you measure client satisfaction? What are we doing to do that? So who's here heard of NPS, Net Promoter Score? So from me, I've told you I weigh myself five times a day, I measure my sleep, walk, whatever. I like to have a number around that. So somebody found a way to basically measure client satisfaction. And this is, so this is Frederick Raicheld, I think that's how you pronounce his name. And he wrote this up in Harvard Business Review more than 10 years ago. He basically did research on surveys in which surveys are most effective. And he found that one company in particular was doing a one-question survey, and they were using that survey to apply their practices, they were more successful than these other companies. How many times do you get a survey that goes on for pages and pages and pages, right? First of all, you never complete those. And second of all, you just kind of, how do you do the analysis on it? I'm just gonna do the time check. Okay, so he found that asking this one question provided so much value to this company that they were ahead of the competitors. So he then took this question at the time and applied to the companies who were growing and companies who were staying steady and companies that were losing market share. And based on this one question of their clients, he asked for permission, he noticed that the companies who have a higher score in this one question were in a growth category, right? So clients were getting high points for that one question, we're also growing. So there's a correlation between having great customer service and through answering this question and business growth. So that's basically an NPS. So the question is how likely is it that you would recommend our company a product or service to a friend or a colleague? That's essentially that question. And you include one more text box for some input. Anyway, so, but that has to tie to a number, right? So we take a look at the number of the people who answer nine or 10. That means they're extremely happy, not just like an eight or a seven. How likely are you to recommend somebody else to see this talk on a scale of one to 10? An eight, right? A nine or a 10, that means I did a great job. Five or six, like you're just being kind to me. It sucks, basically, right? You're a detractor. That's basically what it means. So what it does is the net promoter score basically throws out the seven and eight. The seven and eight don't count. It takes the nines and the tens and subtracts all the people who gave you a zero through a six and it gives you a net promoter score. So that one score, it's actually a really widely adopted measure. There are many companies now that are building businesses, online businesses that help you measure this stuff. And it gives you an idea of how well you do from a customer client satisfaction perspective. So basically we started adopting this and we really started thinking about our philosophy of, yes, we measure a lot of things in terms of building our projects. We're developers or project managers. We provide value, we try to partner up with our clients. We have to provide value to our employees. But in the end, we wanted to have a number around how satisfied are our clients? And it was, we gave a similar talk to an all-hands team meeting. And very quickly, everybody kind of saw the value of this and we started talking about how happy are our clients? How are they responding? And we're tying the NPS. None of you guys remember back. The net promoter score is tied to our project manager's perspective or evaluation. So most of the field folks have an NPS score. Of course, I have an NPS score as the entire company reflects our ability to deliver client satisfaction, but it drives down to everyone who interacts with our clients. And we're looking at taking this, we can't overpole our clients, but we're looking at having interactive measures and a reset. And we're looking to obviously continually improve on that number. We want to generate more promoters. We want to generate fans. So we're going to tell other potential clients or other potential employees or other folks how great we do. And that's started to become part of our culture. What's our NPS? What are we doing to provide better customer satisfaction? So we started looking. Yes, question. I'm not a manager or anything. I'm a developer, but my question is simply this statistic that you take, it seems like something that you can only ask once to a client. Or is it something you can repeat every year? How do you manage existing clients? If your client base is small, but you have a lot of existing projects and continual projects, how do you measure this? If it's something that seems like you can only ask once. Yeah, that's a great question. So there's a small industry around best practices for not promoter score polling. Yes, you're right. First of all, statistically speaking, your sample size has to be really big to get a true, true NPS. We don't have that luxury. We have 50 cell projects, let's say going on, or a hundred projects going on every time. And then you multiply that by the number of people that we interact with. You don't get that grade of a statistical number. And the other question, a very, very good question, is how often do you poll? So we're looking at resetting that. According to best practices for NPS, you're not supposed to poll more than six times every six months. So if you keep on polling your clients, they'll just get tired of it. It's like, yes, you still suck, or yes, you're still amazing. Thank you very much, right? Or I don't care. So every six months, the other thing that we're doing, so because we tied this number to our team members' evaluations, right? So if Karen gets a really high number, we're gonna try to tap into that system, that secret and see what she's doing better than some of the other folks that may not have that high NPS. So we have a concept of continual polling. So as we bring new projects on, or we have new folks that are coming on board, or we have new experiences, we incrementally poll. Now we understand that could, again, a statistical variance doesn't give us that much, but it's better than nothing. So we started looking at understanding better on how we can deliver on that quality, on a high score. So we take a look at four areas of our service, and that is, what is that perfect service? We've been told perfection, nobody's perfect. Thanks break, we were redundancy, nothing's really perfect. So we're trying to basically, we understand that, yet we try to provide a service that is reasonable within a foreseeable framework of instances. So we try to set better expectations during the sales process. We have a re-onboarding during the project manager introduction, if you have a new project, and then we do certain toll gates during, or we try to do certain toll gates during projects to see whether we're still delivering. So delivering in a caring, friendly manner. So maybe as engineers, we may not be so great at this always, and I guess I'm speaking to my former self as a former developer many, many, many years ago. But first of all, there's another, there's an emotional intelligence and business course on Coursera that's really, really good. It basically talks about your attitude and how you interact with people, and people can really tell, basically the biggest takeaway from this course that I took is that people can really tell when you're bullshitting them. When you're not really happy to deal with that client, they know they don't like them, right? They really do know, and one of the things that's interesting about me is I really love my clients. I love all of our clients. I'm humbled by the fact they choose to spend thousands of thousands of dollars in time and bet on the fact that we're gonna get them a promotion, get them what they want, help them achieve their business goals, whatever it is. So we talk about partnering up with our clients and seeing what they're trying to achieve, and hopefully also buying into that, like really getting in what their needs are, whether they're a fact they're totally disorganized and maybe, well, all of our clients are super organized, perfect communication, always show up on time, they know what they want always, right? So we're just here to help them get this but in a really caring matter. And so we say that we wanna partner up with our, we wanna become partners with our clients. We really do. And it's really important that we also emotionally get involved and care about that. And if we show that, that's really a big part of the way. So in a timely fashion, this is also another interesting thing, right? In a timely fashion, may not necessarily mean what? To a client, what it means to you. So obviously there's some expectations to be setting, right? There are no projects that can be completed tomorrow. So you have to set those expectations but you also have to take a look at what we're doing and how we're doing it and see whether that's timely. So that's another key part and backed by an effective problem resolution. So are we empowering our teams to enable them to solve that problem? If something happens, do we have a communications process and in a process where we can mobilize and solve that problem? So now we said there's no, the perfect service is within normal operating circumstances. When something bad happens, how do you react? And from our experience, some of our best clients that are most that have great retention and we always wanna make our client life first, we call them life first, we want them to be with us forever and ever, we had major problems. I can't even tell you how many screw ups we had. Like major, major issues and we basically stopped what we're doing and said, yeah, we totally screwed up and here's what we're gonna do to fix it. And we mobilize and we get things done whether we delete files, whether we lose a backup, whatever it may be, and we communicate and fix it and I see that an effective problem resolution problem, problem resolution process is in place and they can trust us not to abandon them when we own a problem and they've been great. So we try to deliver value. We take a look at the timelines and what we're trying to do, what the client's trying to achieve, not just what they're telling us. We try to break down different types of project by teams and different types of project by project managers so that we can deliver to what the client's actually expecting. Establishing a relationship is really important, not just the project managers, but the technical level, we establish relationship between those folks from account managers and myself. I try to have a relationship with most of our stakeholders. We do focus on automation and testing, continuous deployment. That helps us in terms of setting expectations and it decreases the time in the deployment. So in terms of that timeline, we try to destroy silos so have our system engineering team work with our developers, be able to quickly respond and power teams so whether it's giving credit or whether it's spending more time, our teams are empowered to make those connections with their clients and get that client satisfaction higher. Culture, so this is tough, right? Because Drupal developers are really hard to find. And some Drupal developers think that, or some developers, some folks, some people just think that they're always right and if they can embrace that culture that says, no, we're really here for the client because that's really what drives our business and what we're trying to important, I think it's better to say goodbye. If they don't have a good culture fit, it may hurt you in the long run to have that person than not. And I borrowed this from Scott Meze. He's at Pantheon. He runs their support department. We also try to do this. So we always say that a project risk, there's a greater project risk that we don't have clarity. So complexity is not the risk. It's a lack of clarity, whether we're accurately responding to whatever we're trying to do, executing correctly and of course having empathy for the client. So those are some of the things that we're doing. We measure a lot of different things throughout our different types of project and support and our personnel and time and where time goes. But ultimately one of the things that we're focusing and it's starting to work really well is that client satisfaction over net promoter score. So having that mindset, are we doing things that are gonna increase that? And we have a cheesy theme this year and that is put yourself in your client's shoes. So shoes is our theme, first part of the year. And what are our clients thinking? Because we're gonna ask them to see how likely it is they're gonna recommend us. So that concludes my talk on measurement. I hope that was enlightening and helpful. I have already posted these slides on SlideShare. So I think that's my Twitter handle. I'm gonna tweet that as soon as we're done. And I wanted to open up the floor to questions. I'd love to hear how you guys are doing and somebody give me a time check by the way. I'm sure I have, I had 45 minutes I understand. So I've 45 minutes talking in 15 minute question Q&A, right? I'm actually, so in terms of time, I would give my, I'm just two minutes over. That's pretty accurate. Okay, so Q&A and I think we can go, have they been letting? Okay, so we have like 15 minutes for Q&A. I have a question. This is regarding some figures that you presented but I didn't want to interrupt you. I noticed that, for instance, you presented that you measure the time that, I mean, the meeting time that you have, for instance. So also I noticed that you show that you work under sprints. So I am assuming that you work under an ashy approach. So how do you tie those kind of metrics if you are working under an ashy approach? I mean, it seems to me that it's pretty tough, you know. I don't know if I'm explaining myself. Yeah, I totally understand. So we have basically three types of projects, right? And it's pretty, it's important for us to recognize that they're different. And so we have support and that's a completely different department just around support. And we do measure time there because we have a different, we don't have an all-you-can-eat type model or basically a gym membership model where you said sell 2,000 memberships, especially after Christmas and New Year's, right? Like everybody's like, yeah, I'm gonna go to a gym in January. They sell 2,000 memberships and it only holds 200 people. So if everybody showed up, there would be screwed. So we don't do all you can eat. We actually, we set time aside and we measure and we charge our clients for everything because we provide value, we believe we provide value in every meeting. So we keep track of time there. Just so, and then yes, on development project, which is a different department. Agile is a little bit different. We take a look at it. We try to check once in a while and make sure those projects don't exceed kind of what we expect. But when the first time we did, when we first time looked at a fixed bid project and we took a look at the time of the meetings, it was like, wow, that was huge. And we try to tell our clients, like come prepared, come on time, let's try to make this efficient because it is costing you. So if you show that to a client and Agile we have a problem, for example, and we say, look, this is how much time you're spending in times and this is costing you, it's like, okay, we have some clients that just really only wanna have 15 minute meetings with us. We're like, great. Did I answer your question? Talking about the measurement and about the amounts of project management, for example, you need or an amount of meetings you need. Do you show these numbers to the clients only afterwards after the project or do you have checks in between where you say, okay, we are not on track on budget and this is why? So Karen can tell you about this for hours and hours and hours. She had an interesting project just recently. We depends. So first of all, our philosophy is to be completely transparent. So all of our clients have full access to our time tracking and Jira. Every one of our team members is required to put in their time at the end of the day. We have a call out when it's not done at the end of the week. So they get called out and said, hey, everybody else did it but you. And yeah, if we share, they have visibility into that. I don't have metrics on how many clients actually do look at that. Like I said before, when we pointed it out to this one client or like when we have overages on support they wanna get a detailed list, like everything is in there. That has to be communicated internally also, right? Because we share comments and so I make sure that those comments are respectful and so did I answer your question? Barley, so when we have showed it to, so our clients are aware of that. Obviously you're working in this business, you understand that this time is there. We haven't made it a practice of saying this is how much time you're spending in meetings. Like we don't break it down in those areas but maybe we'll try it. Sure. Anybody else? You had some questions before? In your opinion, what is the importance to have one account manager and technical account manager? Because in my company we just have account manager and sometimes we have problems because you don't have technical account manager to serve this customer in this time, right? Yeah, that's a really good question, right? So we have to take a look at it in context in terms of context for what kind of project it is and the size of the overall client. So for example, there are clients that I have a monthly check-in with them because they are a very key strategic client. We have recently introduced a concept of account managers and project managers. So we have account managers who check in on the emotional health of the client, the project progress, and have an escape valve. Because if you're the account manager or if you're the project manager and you're basically in the trenches, right? Sometimes if you have a conversation with the stakeholders or with the clients, project manager, they may tell you something that they may not tell your project manager it's just a different relationship. I think the other part of the question was you have your project managers who may not be so technical and then do you have solutions architects or developers sit in meetings where sometimes they're completely non-relevant, right? But sometimes you really, really want that technical resource on that and I don't have a great answer for you on that. I know this problem exists. I know that there's a need. Obviously we need to solve that. On one hand, we don't want to waste people's times and our client's money. On one hand, sometimes it's really valuable and it really gets that, when we talked about time to solution and time to resolution, especially on a support side, it really helps that. So we've seen a number of things. We have a 24-7 support team so we have teams like kind of around the world and we noticed that on certain client interactions it does help to have that technical team on these in the same time zone so that was really helpful and we're trying to basically listen to our clients and when we sense like there's a technical person needed, do it. Well, we don't have like a hard rule that says, every two weeks you have a half an hour of the technical solution architect, not yet, not yet. Sure, my pleasure. Any other questions? Yeah, sure. I love questions. So you said you do both like new projects and support projects and you have two different departments for that. So at some point there's a break. So at some point the project is launched and when does it go actually to support? And like, is it a matter of size? Can it go back to new project at a certain size of a new project coming up, like of a new development? How do you feel like that as you said it's about building a personal relationship so do these relationships break? Do you build up new ones? Do you leave them the same? Many questions, sorry. No, I mean these are really, really good questions, right? These are very, very important questions that we have asked ourselves and are still asking ourselves. You know, we try to have a hard definition around what goes to development and what goes to support. So we actually have very complimentary skills in terms of the technical capability on our development team and our support team. We have great developers and solutions architect in development and we have great developers and solutions architect in our support team. The approach is a little bit different and we have really big what you could consider continuous development projects and a support team. The way we internally define it right now is when we have a project that has a start and an end date is in development. When we have a project that has maybe a monthly budget or has a backlog of features and security updates and some other things, then we move them to support. That's how we differentiate it. At first we thought it was just gonna be the size but then we noticed that our support team is really well equipped to have those ongoing meetings without an end date and also we managed the security stuff and sometimes internal infrastructure and DevOps and we build it up for our clients and train them under support and they have other developers internally and we have other dev shops that are working with them and our support team is really kind of well trained and equipped to deal with that. Whereas our developers are just like we're gonna hammer this away and we're gonna do a great job and we're gonna crush this project and deliver on time on budget and it's gonna be great. So maybe that's kind of the same difference. Then the other question was how do we hand that off in a relationship? That's really key, that's a really big concern because we talked about we try to build great relationships so sometimes the account managers help a little bit so they help bridge that kind of loss of a friend of a relationship, don't worry it's gonna be okay, I'll introduce you to your new project manager that just as great as dealing. So the account manager helps a little bit. We do have a process where we hand off but really we do have to break and we announce this and say look, the project's gonna be delivered here, there's a warranty period and of course we're gonna be available for some questions and we're gonna do our best to do a transition knowledge transfer but this is going over to support. Hi, I'm a developer. So I started my company as a product manager and you said earlier that there is this emotional connection and I think I'm having that problem so how do you develop that emotional connection with the customer? I mean, I think it's very interesting. Well I think emotional connection is most related to how much you want to do for your client so yes, it has to do by putting yourself in the client's shoes and trying to think about how to make him happy but mostly it's based on communication because when communications won't go wrong then things escalate wrong very quickly. Like if you didn't tell him that his budget is almost gone or you didn't tell him that maintenance is going to be at a certain hour and his site is going to be down or things like that when you know that but sometimes you don't tell it at the right time then it becomes a big deal and that affects your relationship a lot. So first thing to work on, communication and really try to keep like a right outlook of life when you talk to them really. Yeah, so to me most of the time it's really easy like I really love our clients and I care about them and I want them to succeed. I think about the fact that whether it's an owner who's paying you out of their own pocket and it's their money so they're gonna be really careful or whether it's somebody who works in the multi-million dollar company like what do they want? Well they want to raise, they want their projects to succeed. So I really try to put myself in their shoes and see how and I think about it all the time and that's kind of how I interact with them. I talk to my clients all the time. I had an hour and a half conversation with a client of mine about some very personal things Saturday night while being totally late for dinner that I planned for months in advance. Like I just, you know, we interact very openly and honestly and I feel that's my approach. I don't know whether that works for everybody else but I think it's really important and I think in the end it shows that you care and also when you make mistakes and when we make mistakes and we really work on it like, you know, we dumped some AWS servers that were unrecoverable and I got a call from a client. I said, look, I was on vacation, like here's the deal. I'm on vacation, the door's about to close, I'm with my family, client, baby. I'm gonna land in two and a half hours and I'm gonna call you in two and a half hours and sure enough, like we landed, I handed the baby over to my wife and I'm like, what's going on? How can we help you? That helped, like that was a really big investment for what seemed to me not a big price and they're a lifer, I hope. There's no guarantees in life but I think so. Okay, I actually think I'm out of time. Gracias para invitarme, gracias para tus preguntas. And hasta luego.