 Let's go Being agile we're very timely and We're back. All right Hi, everybody So my name is Adam Ash That's James Kutz. That's Chris Herring. We're from NBC Universal and we'll talk a little bit about what we each do but The topic that we're speaking about today is really on What we've learned over the last Year and a half two years of trying to move our organization Onto of the Drupal platform essentialized Drupal platform for everybody Before we started doing this Different parts of our company and we'll talk about that a little bit. We're each on their own either platform or instance of Drupal or custom platform Yeah, and the truth of the matter is they were on lots of different That's true and as an organization the decision was made to really make a push for going to one centralized platform that we could take advantage of as an organization and Being an extremely large organization, which we'll talk about momentarily We've accounted many many challenges that we have come up with ways to address starting with the beginning of our projects and What we very often see and I think that a lot of folks have this same experience is when we start Projects and we had started projects in the past our partners and the people are going to work with right away feel like their thing is the most unique and interesting thing in the world and One centralized platform couldn't really address their needs that Most times when we start any project with any client or any partner right away They were thinking that that is something that isn't possible a lot of folks come into our first initial meetings with a picture or they go out to get a design before Even starting the project and they hand it to us and say here just build that how hard could it be right and Then of course we all get the question is is it done? Pretty much from the initial meeting right okay now that you know we want a website is it done and so And so these are kind of the things that we have learned to address on a very high level and we'll go into a little more detail so who are we and What do we do I think most people in the room have heard of NBC Universal But I think this slide illustrates our challenge pretty pretty succinctly, which is unlike you know One of these brands building a platform or content management system for one site We have the unique challenge of building not only a platform But a process that can scale for all of these sites and we're not talking about We can talk about this in a little bit more detail later But I don't necessarily mean just like a hosting challenge or performance challenge I mean the challenge of sharing modules or or function in my functionality between sites That's something that's very difficult to do in a large organization like ours because each of these brands Think that they are unique in one way or another and they are of course And we love them and we love their uniqueness, but they're always listening But I think you know what we have learned over the years and not just the last two years But we the three of us in our entire team have many many Dozens of decades of years collectively of experience building websites is that if you strip away the chrome and the design The functionality is very much the same across a lot of sites on the internet and almost all of our sites within the company So this goes into a little bit about what each of us do and how we actually address this problem. So Starting up a prop the project is really a critical part And so what we found and kind of the theme for today is really we the way we've addressed the most critical places to make our project successful and This kind of leads into a little bit about what each of us do so in the organization. I'm responsible for the process we follow Chris is responsible for the platform we build on and James is responsible for the implementations on the platform executing this so We've addressed each of these things within our process, but each of us really own a piece of this So when we're talking about the process we're talking about the roles and responsibilities of the people that are on the teams we put together and The control mechanisms we put in place for actually following our process We're talking about building a foundational core platform and we call it the core team and the core platform And we actually call it publisher seven. It's our version of Drupal seven copyright Chris Yes, and and James is responsible for the execution of this so James really is thought feeling the brunt of our partners Needs and desires all of the time because he's really at the forefront of actually making these things happen for them That's right, but actually with a clear process a good platform. My job essentially is pretty easy Which he'll tell you after the he's lying So the first thing we did was look at actually what our core platform will be and Chris is responsible for that platform Yeah, so the team that I manage for the company is responsible for building this product, which we call publisher core And the thing that's interesting that you're gonna hear about today is we use the same process that we're talking about today Not only just for building the sites that we mentioned earlier for all of those brands But we use that same process for building this product and the products The products built on Drupal 7 obviously, but what we do is we focus on core functionality that is common across the company common things like Editorial workflow that's very common for scheduling previewing publishing and syndicating content within the company or externally common functionality integration with third parties for analytics and content distribution and Video management for example, so we want to build those common areas of functionality in the form of typically Drupal modules by the way And make them available for all brands So when publisher is installed for the first time and made available for a brand on our hosting platform All of that functionality It just needs to be configured and usually it's just a simple configuration screen where you're putting in your keys That enable that functionality for your brand But that's the job of the platform and of this team So as you can see when we start doing a project already out of the box We have some standardizations that we're starting to work with So the first thing we did when we were applying this as Chris alluded to was we had to come up with a process that works within the organization but then works across our partner platform Implementations as well. And so we started actually applying agile methodologies into the organization a few years ago And what we've done over time we started basically with kind of a scrum framework and then over time We actually applied more and more lean and con button other agile methodologies that help us actually manage projects Within the organization. It's a complex organization. One methodology doesn't necessarily Scale across everything we need to do. So what we've done is we've created an agile framework an agile process that we apply To doing this so on the bottom row here is our agile project SDLC or life cycle Basically, those are the phases and we're not going to go into those in detail today But just to get an idea that we go through multiple phases that address specifically how to successfully start a project through How do we successfully deliver and measure what it is that we've delivered? We'll talk about some of those things today. Those are the critical things to actually what we're doing today but at a high level what we're really always addressing is what you see at the top which is Critically doing a great job of defining what we're doing Delivering the value and the benefit of what we're going to do delivering Benefit and value and then measuring benefit and value and so throughout the life cycle of our Projects we're always looking to do one of those things and those are the three things We're basically focusing on to make sure that we are successful each time the goal of all of our processes really to Reduce the risk of failure. That doesn't mean we won't but it means where we've learned at least in our organization how to try to address that So Basically when we're getting into the defining phase You're gonna see a theme here where we basically start at a really high level and then we start breaking it down So one of the first things we like to do when we're starting to work with our black brands is really start to identify first of all What are they gonna leverage from our core publisher seven platform? What's gonna be custom work and in some cases what are we gonna develop as part of the project? That's gonna go back and become part of our core product because other brands are interested in it Or we think it might scale out to those brands. So these are some things that we're gonna identify pretty much You know right in the beginning of the project We're also gonna look at on both sides once we sort of define Whether something is going into the backlog of our core team or whether it's going into our implementation team at both ends We want to start to get an idea of you know, how long this thing is gonna take What is it? How big is it? How complex is it and we want to start getting high-level estimates the nice thing about Working in a common platform and having some experiences with some similar brands is that you know a lot of this is Repeatable so and we're also we'll get into this a little bit But we're also working with teams and those teams have a little bit of history But even if we're spinning up a team we can start to talk about this At a high level and get a feel for you know What is the complexity just given our own experiences and the experiences of other teams within our group? Yeah, I think the important part here just to add is that Just because we're doing an agile Development process doesn't mean that we don't estimate and try to project how long a project is going to take I think a lot of our brands think agile. I don't have to plan and it's really quite the opposite We do we do more planning than we ever thought we were going to absolutely Yeah, and that's one of the critical issues that we address all the time one of the key misconceptions of doing agile first of all but also that Good project management is good project management regardless of the methodology you're following And it's always really important to understand that Any customer any partner we have really what's important to them is when are they going to get what they're going to be getting number one but number two I think it's a misconception that agile doesn't work well with deadlines and We work in an industry where you could imagine has very very strict Unmovable deadlines when a TV show is going to premiere on the network It's they're not going to wait a day because we don't have the website up when a movie is coming out They're not going to delay the premiere because something's not right So the Olympics and we can't move the Olympics believe me So, you know, those are the kind of things that we address all the time that yes So what we've actually done and we'll talk about that throughout is managing by scope But also determining what the value of things is So I talked about our approach where we start really high level and we start breaking it down One of the first things we want to do when we're talking with the brand is get their vision So in the past we might and many of you might experience this, you know, our brands come to us And they say, you know, build me a website and that's sort of their vision I want a website at the end of the day Well, there's a lot more to that and so we'll run an exercise We'll actually dig into this is actually what you're seeing here is actually the outcome of one of our exercises Which is we actually want them as a brand, you know Sort of the stakeholders to get together and and agree upon what the vision is for the project You know and that vision will carry out throughout the project and really aligns everybody At the beginning of the project and through the end and helps us really determine the priority of what we're doing Yeah, and I've seen the teams just say well that doesn't align with the vision They actually remember it, you know, developers scrums in four or five sprints in they actually remember that, you know A new feature request doesn't align with the original vision. So that's healthy Absolutely So we also Then take, you know that after that vision statement will gather high-level feature sets and this looks like It can look like a variety of things depending on who we're working with This is just an example of And we're giving you a little bit of preview of how we actually then further break that out But sometimes it's kind of a one-liner with a little bit of a description Sometimes we have a little bit more context there, but the idea is let's get the whole feature set of the project It may not be every single thing that we're going to do, but let's let's bound this thing Let's figure out what it is and what it isn't so that we can start having the conversation about scope and you know Given that we work with deadlines will often start with the date and then get into the project And so the really the best way then to manage that is either through Resources that's you know, that's one option that we don't have. Yeah and Don't always have and the other way is through scope and it's not eliminating large chunks of scope It's eliminating scope of of pieces along the way And a great example of that I just like this example because it happens quite often is what one of the first requirements most of our partners asked for is a photo gallery and one of the first First of course we got when we're doing and by the way, this is most of these Images are coming from our actual one of our actual projects NBC.com if there's a redesign of NBC.com One of the first things they said before we really got started with them was we need a photo gallery And they thought that was the requirement They had they were perfectly happy and satisfied with we gave you a requirement just build it and they had a comp They did have an image they had a picture of it But but what they've learned really soon thereafter when we said that's not really a requirement What do you mean? You know what what do we actually mean by that? Well, how many images go into the photo gallery? What sizes are they where do they come from what happens when we don't have a license for something what happens when? Something's not available any longer what happens when you click on it You can imagine all of the things that can happen when you have photos in the gallery and what can come up they were actually happy that we actually did that because Previously people would take that I want a photo gallery and build them something that more likely than I didn't satisfy their need But someone took the assumption though Well, I think this is what you mean and we actually say wait we don't want to think what you mean We want to know what you mean even if it's not perfect We can interact with it and then we could talk about changing it So that's a good example of a high-level feature might start out with photo gallery, but then we'll decompose very quickly into more detail Once again, what's critical for us is that we are estimating and planning all throughout the lifecycle of this project Starting from the very beginning We're starting with very high-level estimates what we call t-shirt sizing so small medium and large We have a very good idea based on some team performance and velocity and the types of work We're doing so remember where once we're on a centralized platform We're going to be doing similar types of work So it makes it a little easier for us to then project what certain other things will be like So NBC comm took X to do this thing another partner will take a similar amount of time to do a similar thing So we can project in a high level out early and then as we move into the project We're decomposing into more granular Estimates so we go from t-shirt sizing to Fibonacci sequence story point sizing to get our velocities and then when we do the work We get into tasks And so we're at the actual level of effort when we're working during iteration. We're doing ours So we get go along the whole way doing that And the way this sort of translates to the team is that we're doing what I like to call just in time planning So at a high level we're really true our goal there is really to try and get an understanding of how big this project is You know largely is it going to fit in the time frame that we have how many resources? Do we need where do we need to look to sort of leverage more of what another team did etc? When we get into story sizing now we actually have a consumable Digestible requirement that we're actually going to share with the team and the idea there is to get them to Ask questions and talk about it when we get into the task level Everybody's clear on what we're doing and we're really just sort of laying out a plan for our iteration So it's just in time planning. It's not doing that task level at the beginning of the project It's doing it just when we're ready to start executing on that. Yeah, and at the task level I like that the team agrees to the commitment. That's right. That's right not any one person So what example of this is really it's I know this hard to see but I'll show you another slide in a moment But this is once again, we do these exercises with our teams to actually elaborate and get this information So earlier saw one of the vision and we get often get very long vision statements that Basically have a lot more information in them. That's part of the one we saw this although You really can't see it is actually are a very early in a project planning Kind of elaborating on how long some of these big sections will take from the t-shirt size point of view Basically what this is is then put into something like this. So we've translated it into This yeah, so you know this again still a little bit high level This is not a Microsoft project plan that has every task written out, but this is enough This is enough for us to sort of set a date to understand, you know the how much time and This ultimately leads into what I call sort of giving each feature set an allowance You know if we're restricted by a date and resources We're going to control scope right so we want to really Focus on what the priority is of something and sort of give more and allow more of an allowance to that feature set And then things that are lower priority, you know, it doesn't mean we're not going to get them done But we're just not going to spend a lot of time on and this is sort of the beginning of Starting to you know put that together very high levels still We also have to remember that we have executive level sponsorship and support Very often regardless of the methodology we're following people want to have an idea that they're comfortable that we're going to hit dates That we have things laid out that things are planned for once again. We're still agile We're being very flexible about what we're doing when we're doing it managing by scope, but that doesn't preclude us from needing to actually Plan things out a little more elaborately than you often see in some agile projects And again, you know a lot of the things that we're doing because we're using a common platform Some of these things are repeatable. So we can in some sense Start putting this together and really getting giving the the brand kind of a good idea of what the the project's gonna look like So we're really as we move forward from those high level. I just want a photo gallery We move into more detailed requirements Detailed stories that actually start talking about not only what we're doing, but the acceptance criteria for delivering them And we just came from a session talking about Excuse me BDD behavior-driven development, which we very much support and our acceptance criteria Although we aren't doing a formal BDD yet. We write our acceptance criteria so that we can actually number one and very Well defined when we're done with delivering something, but also we start with those as a platform for our testing So it gets into a lot of decomposition as we go along with the more detailed stories At this point in our project. We're also getting wireframes and the difference between a wireframe and a Comp is that this is not a definite thing that we're going to deliver We don't want to ever have our partners confuse a finished looking picture with what they're actually going to get online I mean we haven't we've just started doing In browser design and I think that's really a great step forward to what we're doing when we're working in a certain platform We believe that it's best to move towards showing people what things are going to look on that platform as opposed to a piece of paper Number one, it's interactive. But number two What's on paper doesn't always translate very well into what people are going to get at the end And we don't want to them to have a poor poorly conceived misconception of what they're going to actually get So our rule is that the requirement is what comes first The wireframe is actually a visualization of the words of the requirement We know because we work with teams actually all over the world And we work with teams that speak many languages that a picture is very important for us to be able to translate our words correctly So it's critical that not that we have them But we don't want to confuse the picture with the written requirement We also want to make sure that it's easily updated because depending on whether we have a ux resource on the team or not And often we don't we might have to update a requirement We want to be able to update the picture as quickly because that will be the default that people work from So, uh, again this theme of breaking things down. So Once we get into actually writing stories now, we can actually start to lay these stories out Maybe even epics, which are sort of high level themes Across iterations and what this again helps enforce is How does this how does this go? How does this Align to our timeline The other thing that we like to do is in this actual example We're actually using two teams to execute on the project But we also have things prioritized for our core team And we also have things that our support team is is executing on it as well So again, yes, we're using an agile process But you know, there is a lot of planning that needs to go on when you're working across multiple teams And really the key is to define this stuff start breaking it down in such a way and again that just in time planning Which is getting at the level of detail that you need at the right time So just to reiterate this goes from this which is a very high level plan to this which is a more detailed plan I think that we've all seen and since we're doing this for a while now And we're doing but we're doing with a lot of folks for the first time That they're a little bit surprised with the amount of work that has to go into just setting this up Just getting good requirements just really getting to the level of planning that they really want that's accurate It's very easy to give someone a plan that looks like this And not really map to it not actually deliver to it It's really not that easy to deliver to this and we people expect that we will So that's the big one of the big challenges for us Developments developing something is expensive Working out some of these planning issues working out better requirements is a lot less expensive than trying to figure out What we want what through building it so that's one of the things and the other thing I just wanted to add is that my team's publisher roadmap is influenced by the various implementation planning that happens Right, so if I see common threads where three or four brands need a certain type of functionality I'll prioritize that for for our team so that we can deliver that and build it once and then make it available For the implementations and then then it's less work on the implementation team as well So there's this constant realignment of roadmaps that's happening Continuously and you know one thing we didn't really mention But even though we do sort of plan this out the plans change Right, and so we need to be flexible and understand that but having a plan and then changing it now You know what the impact is right? So if you didn't have a plan and you were just sort of iterating through this and hoping you would hit the date You really wouldn't understand the impact of a change. We certainly welcome changes into our process But at the same time we have to understand what the impact is and it's really our job to really Educate the brands about what those what those impacts are and in many cases it's multiple brands that are affected by that So it's a it's a big challenge, but Through collaboration communication some of the groups that we've set up Across the organization. We're able to communicate that And you know to some level some degree get a little bit of buy-in on that Right. So we're going from the defining process, which is the beginning of our project life cycle Where we're setting up making sure we have really good requirements making sure we decompose to have enough information to start measuring And be able to deliver and now we're going into our delivery phase Which encompasses our setting up our environments making sure our teams are fully in place Doing our release planning as we were just doing and now we're really doing development And once again, we're always thinking about delivering benefit and value And managing by scope To make sure that we fit in the timelines that we're looking for I think one of the you know besides requirements I think one of the most important things that we are doing is really setting up the right team structure And this is hopefully It's a very sort of complicated diagram, but it really explains sort of how we How we have teams different teams doing different activities And again, this is across a large organization. So in some cases, we're working with multiple teams are the output of One particular team's work. So it's really important that we have teams of people and not individuals So, you know, first we have sort of a large team of people who are working on requirements. We have You know stakeholders. We have designers. We have Requirements engineers, which is kind of our our a little bit more than a business analyst All of these people are contributing with our product owner and with our brands to produce requirements. We then have And this is typically for a large project we then have Tech leads sometimes it's usually multiple tech leads on a project So that they can help sort of set the approach, right as all of you know with Drupal There's a million ways to do one thing We're trying to create not only Use a standard platform use a standard process, but really down into how we're implementing the approach that we're taking on certain things We want that to be somewhat consistent and we do that with Our tech leads and our tech leads talking to one another and working closely together and then that can sort of That approach really spans out to the teams So once again, it's it looks complex and maybe a little complex, but Uh, it is complex a little bit. It's not so simple just to say we have a product. I think it's complex because it is complex Well, it is I guess I guess you could say it but once again people don't realize sometimes and this illustrates it There is some complexity going on here. We're being very diligent to manage it in a way that it's Is opaque to our end users. We don't want people to have to worry about this We manage this so that once again our our partners and our clients and our end users They don't have to deal with this so we do Yeah, and so we can answer those original questions like when will it be done? Right, we can accurately answer and we've organized our teams and james alluded to this in a way that we've Actually created a position for people just to help get requirements done Because we found that one of our gaps in our organization and frankly I found on many projects through the years Is that we often work faster than the amount of consumable requirements we have ready to go And so we created a group of people that just helped build the backlog so that we have consumable requirements For the teams all of the time. It's it's something that they just that they only focus on that So this is an actual sample of one of the iteration our iterations are two weeks This is really our day to day schedule and it's consistent. So every every day everybody knows What meetings are happening? What where we are in the process? What everybody should be focusing on and the interesting thing is that we actually in that two week process We're actually starting at the end of the iteration We're starting to plan for the next iteration and that really is Can be challenging And it actually it turns out that developers and testers actually like sort of a consistent schedule Imagine that so by having a rigorous schedule and really sort of moving Have it in earth to get everything in by the that time and keeping the meetings is really important to us because that way We create consistency And really all of unpredictability and all of this is really around that predictability You know, we find that once we start delivering while we still have that end date Sometimes we have brands that have create sort of artificial deadlines within that project And we find that once we start delivering every two weeks and there is transparency and they really know to expect Oh at the demo, I'm going to see These 20 things that the team is working on and I know I'm going to see exactly that A lot of those other deadlines sort of vanish and we start having different conversations And then we can start talking about, you know versus are you going to meet that date? When am I going to get it? We can start relaxing a little bit and start talking about What's the best thing that we can be doing and really start moving on around and you know, that's that's really You know, I think all of us here in the One of the presentations we saw this morning somebody mentioned that You know, we don't just do what we do just to sometimes we do it for a paycheck or Keep ourselves occupied to keep ourselves out of trouble during the day, right? But I think all of us are here focused on quality, right? And so if we can build quality into our process And really allow for You know changes and and the sort of the best possible Approach to things to come forward, you know, then we're doing our job Yeah, the only thing I would add to this slide is that I think predictability breeds trust with our brands And that's something we've struggled with for a number of years is getting the brands to just really Trust that when we say something's going to be delivered every two weeks It actually is and by having that predictability it actually helps with that trust factor So continuing we're decomposing further Into our yeah, so this is just another look at You know that last step we said so we have our you know our high level Sizes we get into stories now We're down at the task level and you know task level is really we have it broken down understood And we're really just trying to map out How our how our teams are going to execute across that two weeks print two weeks is not a lot of time Especially when we're talking about developing and testing In that time frame and being completely Production ready doesn't mean we're always going to production, but it does mean production ready So that means we're breaking things down. Maybe smaller than some some some of you are used to that's okay Maybe we're You know having more people involved in developing one feature, right? That's not so bad Maybe we're trying to build, you know better test cases early on and leverage test cases that other teams have used And also other functionality that other teams have used these are all good things and you know, that's really we really drive it through This what I call aggressive iteration schedule And of course this is also where we're applying during our delivery phase our visual design We haven't forgotten them But when wireframes were enough to start in the beginning to really get started and working and make sure we're delivering The overall framework of what we're building We have in parallel the designers working on the actual design. We're going to apply So we haven't forgotten it We just want to make sure that that the design isn't driving the features that's not the requirement Right and you know, we we can build without this. We don't need this right we can start building Back end functionality even front end sort of how we're going to use that right? This is all theming and styling right at this level and we can do that later in the project and that's sort of where You know that that's that's great because now our designers can really see what it sort of looks like what that functionality looks like And what our interactions are going to be and start doing that work Once they sort of deliver us pictures and Yeah Also, well, this you know, you can start you want to talk talk about this one Start The most one of the other most important things although There's all pretty important and one of the things that we we really feel are really important the metrics We're getting out of this so from the process perspective. I'm looking for certain things out of these I'm looking to make sure that we're delivering consistent velocities that we are addressing the numbers that we're seeing And that I being responsible for process make sure that I'm adjusting the process where I see anomalies in the metrics Or some of the metrics tell me something about where the process might not be working very well Each of us and James and Chris will elaborate use these same metrics for different things So we're actually, you know creating a lot of numbers, but we're actually getting something out of it So, yeah, and you know, I would say, you know, we definitely use like a team's velocity to understand You know predictability, right if we know a team can consistently deliver x amount of points per iteration That helps us with our our obviously even our high level plan, but also our detailed planning And then defects as well helps us really sort of control the the quality that that's happening in the iteration But you know, the other thing I want to mention on this is that this is kind of all that we track You know, we don't get into a lot of really fancy Spreadsheets and you know diagrams and whatever Keep it simple is sort of our motto, you know, just enough These are the metrics we need now. These are the metrics we need today. And so that's really what we're tracking I totally should put in that keep it simple slide. Yeah. Well, just make sure you spell it So ultimately we're delivering what we said we were going to deliver at the end of the project Which actually we're not quite at the end of nbc.com But this is pretty close to what you know something close to what it might look like in the future But the fact is delivered doesn't mean done And so this is part of the process that we want to make sure so, you know, once again We're in a large company. We're delivering value not just a product at the end So once we've delivered this the next part of the process actually begins Which is measuring our benefit and support and we often forget that that even though the development phase might be over And the team might be rolling off of this The business is still obviously very much involved. We're now delivering something Once this goes in front of our audience once this people start using this We're going to have to start measuring whether or not we're achieving the goals we set for this So this goes back to the vision our vision Usually includes some of the goals we've set for this this redesign or this project And so basically what we're saying is once we move to a new platform We want people to do this more or do this less or go hear more or experience this Now we get to start to measure that so that's really important for us We're measuring the benefits. So once again the business is looking to see if they're getting out of this What they expected once again if we're already in production if we're delivering every two weeks We can track this all of the time tracking functionality is something inherent in support making sure things are working We start gathering feedback again because now people are using it so we're getting something out of it and Creating user stories. So we're back to The backlog creation and each one of the Projects that we work on for each one of the brands have their own backlog My core team has a backlog as well and they're all influenced by each other And the point is and the reason why we had that slide titled not done Is that at the end of the project we want to re-groom our backlog and take another look at it And sort of start again and make sure that we're in a good place to deliver more value to each of the brands that are using the product And more about value to those brands themselves, you know, it's interesting because you know I'm in charge of the projects and actually implementing them But that's really sort of the beginning right at the end of a project is really the beginning, right? So now we have a community, right? We have a community of Of internal developers, you know, some of the brands have their own smaller teams that they use We have an internal community of end users and now we have a community of You know of visitors out there that are all using the same platform. So really at the end of the day This is the beginning and now we're really all part of a larger group Which is exciting Yeah, so to summarize I think what's what's the message we're trying to get across to you today is really that this process Although it is complex. It's really designed to Deliver on these five areas, which is a scalable platform in in the form of our publisher product That delivers common functionality to all of our brands So that these things are built once and reused and that's the point of the second bullet Which is reusable parts that come out of the publisher core team as well as functionality that's built during an implementation that could be used for For an input an implementation that's built for brand a could be used for brand b, right? So on mbc.com if they're building something that's Oxygen could use we want to be able to reuse that defining your requirements Obviously, I think we all know how important that is we put a particular emphasis on requirements in our organization Mostly because we've been doing this a long time and we know that without requirements We really can't build anything and we can't project with any certainty when something is going to be done Which is all that you know our brand leaders ultimately care about The relationships and collaboration obviously we're here at Drupal con and we've been coming for several years We want to participate not only and collaborate not only internally within within each team But between teams and with our brand leaders as well as with editorial personnel throughout the company That's a lot of collaboration and a lot of communication that has to happen But one of the things that I've really become fond of is the fact that talk is a lot cheaper than development. So We we put a heavy emphasis on that as well And then the metrics that adam mentioned just help us refine and improve our process And make sure it's sort of like a gut check to make sure that we are delivering value to each one of our brands And we are delivering value to the company if we build something once we want to measure the effectiveness of that Functionality by seeing how many times it's used how long it took to build and what that value really is at the end of the day So I think two things are important. Well one most important thing is when we started this I don't think uh, which was a while ago a couple of years ago. We started implementing this Not everyone was buying into it. Not everyone was sure this was going to work and one of the gratifying parts of this is now that we've been doing it for a while Especially with new folks that have never worked this way before we're getting a lot of satisfied customers They're telling us that they appreciate the way we're doing things that it's a little more painful in the beginning Usually we're getting a little pushed back at the beginning But um We're really pushing them a little bit more than they may have been pushed in the past They're getting good results out of it and really ultimately it's getting the result We're looking for you know a lot of this revolves around estimating and planning and really having a good process In place to do that. So actually just to mention that james and I tomorrow total plug Tomorrow james and I are holding a buff Just to talk about estimating and planning Uh, we could talk about what we're doing But we'd also like to hear what other people do because that's really one of the biggest challenges that we have is really Refining and getting better estimating and planning. So please come to that And uh That's it for today. We'll take questions So yeah, go ahead. Go ahead. So uh, we we try to use Simple tools we're using rally mostly for managing our iterations and managing our requirements We're trying to Use it to We also use it for our metrics as well We're trying to use it for some of our sort of pre-planning But it doesn't really do a good job. So we're really using homemade custom spreadsheets at this point They're just aren't tools out there. But uh, you know once we develop them I think rally is a little bit better on the product side for release planning, but not on the implementation side Yeah, can everyone here because we could I'll I'll repeat the question. You know actually that mic we could pass around Yeah, the question was how big is the core team and how long did it take to release the the initial version Um, the core team now is about 20 people and that's all in that's requirements dev qa product people project management as well And it took really before the the product was ready for the first site to build on top of It took close to a year And that was uh There was a little bit of faith that this company certainly gave us to build that core product But by using this process that that we've been talking about today to build the product itself Um, there was a lot of trust that was gained through predictability. We didn't start that large either It was a little bit smaller. It was probably about 10 or 12 in the beginning Um, but then because we had some good metrics We were able to project that we probably needed a little bit more help in order to Leverage it for all brands, which was the goal all along other questions sure Okay Oh, please We'll get to you next Excellent question So the question was, you know, if we're measuring velocity and we have so much pressure from the brand leaders, how do we Handle that communication and that pressure You know to me, I feel like we handle it through transparency. We don't push back We let them make those decisions. We're just very transparent about What the velocity is of the team what we think they can deliver in a certain period of time And let those brand leaders make those decisions themselves We don't want to force that force their hand in any way. I think that's the best method quite frankly You know when it works well The the brands our partners make those decisions so we have before they even come up actually So we had on this very project mbc.com and we started with the scope of the project We knew that there that there were things that we knew they were crazy They were just too big for us to have well, I don't know if we thought they were crazy But we knew that they probably bigger than what we can handle even at the beginning We had an idea But the partner to their credit said no, we want to keep them in we we understand we hear you we may not believe you right Which is fine. They're really running their product. They're the business But over time when we were delivering consistently they saw that they were getting good quality They saw that we were legitimately working as hard as we could we were delivering maximum amount of stuff They started to realize it for themselves that we weren't going to get all of these things And that it was true right and what we were saying and they literally took out Everything we had said probably should have been taken out in the beginning But they at that point realized it for themselves. It's very helpful to us when they do that They also knew that we could add some another team which we to add capacity. That's always the possibility But the fact is you're always constrained by something And so the constraint is either the amount of things we can do or add more people to do it Which adds to complexity which doesn't mean you're going to get everything still so Ultimately, we're just being very transparent about what's reality is is is the killer here because Traditionally people are are led to believe they're going to get everything or they're been yes to death Oh, yeah, no problem. We'll get it for you. Oh, no, that's not going to be a problem when they don't It's quite a disappointment and rightly so but we're saying look We're going to give you everything we can give you and we want to make sure we do the important things first Which is by the way also a different way of doing things people aren't used to having to make choices Even though they they should be right So there's always a finite amount of time or capacity or something yet in the past people have just for some reason felt Well, we're just going to get everything at some point. We're not going to get everything at some point And so we're being realistic about that up front It's it's uncomfortable in the beginning for some of these folks that haven't worked with us in the past or haven't worked this way ever But ultimately over time. I feel like that uncomfortability goes away Yeah, it's also, you know about breaking things down even further nine times at a ten something that is big The team will come back and say well, that's pretty complex as we break it down We can have additional discussions and that goes really a long way So if things seem too big where they want to get more work in a lot of times We break things down a little bit further and we find out that they're a little bit more digestible too The the velocity is the Is the result of the team? So it's not something that we control. It's a metric that tells us About predictability right and so we can actually if if you know in that slide You'll see sort of some dips and things like that. I can tell you you know almost You know there's there's reasons behind everything the 131 we had some vacations Dennis, I think you were out on vacation I'm 27 and we were still had people on vacation So there's reasons why the velocity dips but predicting that throughout your project Gives you the ability to have that conversation about prioritization scope And you know, uh that that goes a long way Yeah, that number is just a reflection of the point sizes that they were able to deliver not not hours Yeah, and then we have conversations like what would a second team how would that help us? What if we move these feature sets into core? What if we leverage something that somebody else used, you know to start really these are all conversations that we can have rather than Hey, can't you just have the team work faster? You know I think I heard that one the other day or yeah more hours there. Oh, we had a question back here We call it James. Yeah Now we need the picture of me on the four phones. Yeah Any anything and everything I think is the answer. I mean Primarily we do a lot of meetings, right? But for a lot of the planning meetings for these projects They have to be people have to either be in person or online So WebEx and teleconference calls we do a lot of that for some of our larger projects where We're fortunate to be able to use things like video conferencing. We do that as well So anything we can it's a we really want to foster that collaboration as much as possible And we're pretty religious base camp users. Yeah, and IRC as well. Yep In your point of view Which are the like the skills that you were looking at project manager? If you were hiring someone new for your organization that fits You know your your methodology and and what really matters for you because what project manager might have so many skills Yeah, that's a good question. What three are the the ones that really matter I think first of all, it's somebody who's done this before so done some level of agile Hopefully it's somebody who's sort of done it Obviously successfully but really with kind of a team that was co-located because we're not co-located We have teams all around the world But having that as a reference point for a project manager And what they're driving towards is really important having that vision of like You know, this is what the team should look like even though. I may never get there This is how we should be operating and constantly driving so experience with With agile And but flexible enough to realize that we actually stopped calling our process agile because everybody has their own idea of agile So we just call it the process the process, but it's you know, it's based on sort of scrum And other agile principles. I think also we look for technical project managers So somebody who we don't need to really dive into code But somebody who really understands when a developer says I have this block I need it removed Like you need to understand the severity of that, right? And you need to be able to talk to other teams in some cases and go to you know Talk to the architects on other teams and say, hey, listen, you know, we've got this problem, you know So you need you need those skills and really somebody who is You know incredibly good at communication That's really those those would I would say be the top three upfront yes well we Our philosophy is it's still based on the principles of agile if you if you've done scrum, you'll recognize what we're doing It's not a completely different thing. We use Much of the language of scrum Lean and kanban a lot of basically it's scrum lean, you know very lean thinking in kanban So it's it's not like there's no lexicon for what we're doing But in reality what we felt and james and I have been doing actual agile implementations for a while is that We we wanted to make a process that fit in the organization So our organization This process would work pretty much everywhere, but it might be tweaked again for a different type of organization So what we've done is fit things to work Agile within an organization that isn't necessarily all agile and nor really understands agility at all levels So just now we're starting to interface with upper management and understanding Maybe we can actually help them project and do budgeting with some of the numbers We're having because we're getting that consistent things where most years and most places will just you know Do their usually yearly guests for the next year's budget So we're we're just applying things but what we found over the years just from our agile experience And I've been doing agile before scrum right so before even scrum was scrum that One methodology doesn't necessarily scale or fit all of our needs And so we don't want to be not agile, but we want to find what's maybe a best of methodology that fits in What I like to tell people is I know when it's not agile Right and that's what we really want to do and part of what we're responsible for also is training people on the So we actually do a three-day training course at the beginning that so everyone's Starting from the same place and the reality is that scrum and agile don't do a good job Describing how you work with global teams. They just say you don't do it, right? But the reality is we do And I think we're successful in that and so we I wouldn't say that we've changed it We really take that and it's it's more of a broader Sort of process that goes into more detail into things like this is how you write a requirement This is the type of meetings that you have. This is how you work with global teams, etc This is how you work across two teams. These are things that really the agile community hasn't Defined and things that people are struggling with and really if you think about scrum and most people are familiar with scrum Depending on who you speak with and some of the founders of scrum and they've changed their point of view over time as well They some of them say well, we didn't say these things But we didn't mean that you ignore them like architecture, for instance And some people say if it's not in the book scrum doesn't do it project management isn't part of scrum But project management is necessary to in our organization to run a successful project So what is how do we make that an agile thing? Requirements have a standard form which we've learned from scrum as a user which we use But we we extrapolate we actually take the benefit statement and make that a separate line So that we can actually force people to think through what we're going to measure later So it's not mixed in with something it's separate. So we've just tweaked it a little bit to kind of highlight What we think is important in the process Do you want to try that one? I don't know if I So I I think that what we're trying so we're just starting basically to implement this idea of business value now So we haven't been doing it for a very long time. We're just at the point So what's great about our process is it's evolved since we've been there and the focus on business value is actually something That's organically happening within the organization So we're actually now starting to focus more about what it really means to have business value What it is we're going to measure so we actually use the semantics of a benefit is something That you can actually measure a trend or see so if I have I'll start with the value a value is a definitive number So if I have the goal of increasing x by 10 percent or views or advertising revenue Decreasing something by a number 10 million 100 million. That's a value that we can measure So every iteration we can measure to see if what we're doing is actually working within the organization A benefit is is less measurable. It's more like I want people to watch more videos I want people to go to this page more So the first thing we're doing is introducing the difference between the value and a benefit to the organization And now we're actually just starting to look at how our partners are Measuring things so they can actually take advantage of this So these are numbers that we're just starting to introduce because not that they weren't important before but people weren't using them And so this is something that's actually new in our organization Especially since we're able now people are buying into the process So they're buying into some of the results of the process And I think the value is a little bit easier to Measure on the product side to a certain degree because you could say, you know Did this save the editor time in publishing this type of content or did it, you know Was it reused on a number of different sites without any added? Development time right so that's a value that can be measured But we don't have a lot of control over the performance of a website in terms of monetary performance or traffic Right, that's all a very complex thing that we don't control But we can start to measure against that and we want our our partners who do control it or have more control over it To really start thinking about it So when we say something's not done from our organizational point of view once our thing is delivered We are kind of done But we want them to know that there's something more that these metrics are important So we're building that into the culture of the company more than we so it's a great question I hope next year I have an actual definitive Answer with results for you And a formula I think we got time for one more sure So at the at the task level where it's really at this point we have the stories the requirements broken down And it's really, you know, how much time am I going to spend on that? And yeah, it's it's a little bit of a group effort in some cases We'll have sort of a lead Work with some of the other people that are more junior to sort of, you know Share their experiences and say yeah, I think this will probably take you this amount of time but the But the important thing is that they're having a conversation about it and you know, you can't just sit down and say Oh, I'm going to build this module I think I need this task and that task and it's really about the conversation that you're having Over and over and over again first with your product and requirements team And then with your peers on the on the development side to really understand what that is and at the end What we get is that on that first day of the iteration people are sitting down and writing code Instead of like sort of, you know, spending a lot of time thinking about and dreaming about what this module could be Because we've already done that we've done that ahead of time So yeah, and the whole team has to agree on that on that number right that size including the testers who have to test it So So that's actually interesting that so the leads are kind of uh, you know, some of them certainly participate more in the iteration than others Um, it really depends on the team size Some of them are literally coding and in the team and some of them are more mentors dealing with upgrades to core Um dealing with um, you know, planning and architecting and figure out what modules Um, we're going to use um, and so for their time. We're not really interested. We don't really track that necessarily in the iteration when we're tracking our tasks in the iteration What we're really trying to figure out is how are we going to get this work done? Given, you know, all the tasks that we have to complete and what's our path to do that including testing So it really helps us do more like plan out two weeks. So and then the leads we really sort of uh, Just overwork them to death, you know Well, thank you everybody