 Let's see, we're going to start with an intro here and we'll introduce the people on the panel. We're going to talk about the MSNBC.com project and sort of how it came together and kind of a top level overview and maybe get into some of the technical details of it, but mostly we're kind of talking about the sort of business end of things and the decision-making and sort of the high level sort of how stuff came together, but we're happy to talk about whatever aspects of it you'd like and we'll have a big question and answer period at the end, but I suppose if people have questions during, you could raise your hand and we'll try and accommodate them, but we are mostly sort of focusing on. Questions have to go to the mic, this is all being recorded and posted on YouTube and so if you don't speak into the microphone, you don't exist. People will not know who you are if you don't talk into the microphone. So I'm Jeff Robbins. I'm the co-founder and CEO of Lollabot. Lollabot sort of took a tech lead position on this project and mostly I'm here sort of as the Rube representing the everyman and going to ask questions as if I don't know the answers because I don't know the answers to most of the questions anyways, but I'll let people here introduce themselves on the panel. I'm Richard Wolff. I'm executive editor of MSNBC.com. So my role was to put impossible demands on all of the developers and run for cover. My name is Karen Stevenson. I'm with Lollabot, I'm the architect for the site and my role was to run for cover. Hi, I'm John Keegan. My title is VP of technology strategy at NBC News Technology Group and I herded the cats of technology for MSNBC.com and continue to manage the technology for that site in our highly matrixed organization. So Karen, do you want to explain sort of how the title of this presentation came together? Yeah, we decided we wanted to do a presentation about the MSNBC project. It's a great project and we had a lot of fun doing it and that was really exciting and we're trying to figure out what to title it and one of the things that came up as we were going through the project, we had all these kind of impossible things to do. We had a kind of a crazy deadline. We had a huge team of people. We had a lot of things that were going to be very difficult and that were quote impossible and Richard always had this quote. We would talk to him. He talked about the fact, well, he'd say, well, you know, we do live TV. He said, we do the impossible every day and so that everybody just sort of picked up on that and every time we had something come up that we had to do, everybody say, oh yes, we do the impossible every day and there just sort of became a mantra for the project. So that's where the impossible came from for the title of the program. So we created a sort of a case study video kind of thing that I'm going to show now. MSNBC. Sorry about that, folks. MSNBC started life about 17 years ago and it was in the mid-90s a vision where TV and digital would all converge and it would all be great. MSNBC on TV had come to mean something very different, a place where, as we like to say, we lean forward. On digital, this thing called MSNBC.com had continued to trundle along with the original intent, which was to be a sort of 24-hour version of NBC News. And there was this disparity between the two. So we unwound what was left of the joint venture that created MSNBC.com. We renamed that NBCNews.com. And then we were left as MSNBC to create something wholly new. Here at NBC News, we talk a lot about the newsroom of the future. The concept of a story has really involved. We all knew what a story was. It was how to headline, how to lead. Maybe there was a related image. Today, we're creating new content types that allow us to tell these stories that we weren't able to do five years ago, let's say. We know that our audience is consuming us in multiple spaces at a time and place of their choosing on any device that they own. We know what MSNBC represents. We have a clear sense of mission. We know how to create the content. As a publisher, I think you have to not say, how do we do what we do? It's how do we translate what we do into this new medium? The scope of this project was very large. There's this company out there that you might have heard of called Facebook. We tried to build Facebook and a brand new news site all within the space of nine months. We had a lot of discussion with you guys at Lullabot and with our designers right from the beginning about responsive design, about mobile first. We are creating all this content and it should be in a form that can be distributed anywhere. I think what we ended up with, it's immersive. It feels great in all of these different spaces. And it allows this organic path for you, whether you want to stay just on the content side, whether you want to move into this more of a community space. We felt that it was important to make the social network part of our news. That's not just sticking a button on a piece of content and saying tweet from here. It was really embedding every piece of the community. Users had profile pages, group functionality. It's not the journalist organization that matters. It's how the user organizes all of the stuff that we're producing and they do it on their own. We have a very large editorial team. We have a separate team for video. We also have a design team. There's a kind of level of complexity and increasing sophistication that's required on the back end to allow editorial teams to build these new types of content in a way that won't slow them down. And that works. They're happy with it? Oh, they love it. When you really get to the crunch, you want a team of developers that say, set us the highest challenge we have. We want to reach for that extra piece of what we can do rather than just repackaging what we've done before. Very early on in the project, I could see that the Lullabot team had said, you know what? I think we're working on something unique. They really opened their mind to the fact that we were building something brand new. It was a monumental undertaking and to do that in nine months on the kind of budget we had was really extraordinary. We knew that there might be multiple points of failure across the board, but Lullabot would not be one of them. The most amazing thing about this launch was that it was so anticlimactic. The very first day when we went live, there was some excitement and some clapping, but it worked because really there's only one thing that's supposed to happen. It's just supposed to work and people are supposed to go and create the news now. I'm not just saying this. It's been the most fulfilling. It's not just my comment. My senior team has told me it's been the most fulfilling professional experience of their entire careers. It's not often you can say that. Richard, why don't you talk to us about the history of MSNBC and the past 10 to 12 years that led up to the business decisions that came about to making this site? Sure. You had a little brief version of the story just on that video, but back in 1996, 1997, Microsoft and NBC, that's the MS and the NBC for those of you who don't know, they got together and they did a... You'd have to take my word for it. They did an extraordinary thing, which was to sign a 99-year deal. A 99-year deal because that's how committed they were to this idea of what they called convergence back in the 90s and it was a 50-50 joint venture. So in addition to having this very, very long-term marriage, they also decided that there would be no casting vote, which was all wonderful in the 90s bubble, but convergence took a little bit longer than they thought. And then finally, when we got to the point of convergence, the two sides didn't exactly get on. We had bought out, we, NBC, had bought out Microsoft on the TV side about eight years ago now, and the digital side was still trumbling along in this 50-50 joint venture. And as MS and NBC grew to have a bigger profile and a bigger voice, everyone realized on all sides within the joint venture at Microsoft and certainly NBC that the website had become detached from the bigger brand, the TV brand. There was a dispute about which was the bigger brand and not, but we ended up saying this isn't working. Let's go our separate ways on the digital side. So we reshuffled things around, took the old portal site, handed that over to NBC News so that they could carry on with that mission, that sort of 24-hour news mission, and we could build this thing from scratch. And that really was the mission impossible, which was to create something out of nothing in a very, very short space of time. You all are familiar with obviously creating things from scratch, but when you've got a big media brand that has a tremendously high profile, you've got very, very impatient journalists who are suffering from ADD every single second of the day. They don't understand why you can't just flip a switch and create this spectacular thing that does everything and pleases every stakeholder and there are a lot of stakeholders in a big media company. And so everyone wanted it immediately. We had nothing, we had no website, we had no digital presence, and the digital presence that existed didn't really reflect us. So it was wildly ambitious, still is, on a ridiculously short time frame. We had people who, senior people who to this day don't understand why you can't just create something like this in three weeks, four weeks. So there was a long educational journey we all had to take. Does that answer the question? I think so, yeah. And for those of you who don't know MSNBC, for a start I don't know what you're doing, but we say we lean forward. We obviously cover a lot of politics, we cover it from a broadly progressive perspective. Our audience, super engaged, are very passionate about the brand. They identify themselves as MSNBC people. I mean, that's a, we wanted to make a statement with us, not just that we could build this thing from nothing, but that it would look different, it would operate differently. It would really embed with a community because the community already existed. We weren't trying to create that out of nothing. It existed on lots of different platforms and lots of different spaces. So I will stop there. So let's talk a little bit about MSNBC.com and sort of the decisions that were made about that. John, do you wanna sort of chime in and talk about things like the decision to use Drupal and sort of how that sort of trajectory started for MSNBC? Sure, sure. I'll start with that one. That's a great question. Our friends here at a group at NBCU called ONTS are here. The great thing about our company about NBCU as a whole is that there's a lot of support behind Drupal across the organization. We have a strong team, a central kind of technology team that is dedicated to Drupal and has a couple years, three years maybe behind the program. And so there is kind of an infrastructure and a support system around Drupal, both in terms of development as well as the hosting of it. So it was kind of a no-brainer as far as choosing Drupal. The company already had the support behind that. So really at that point we had a content management system but the site as you can see really is not just about the content management system. We kind of thought of the three legs of our platform were the content management system in terms of Drupal. Also our video and our video experience and of course the technology required to create that. And then the technology required to create the community and social services which I guess as a technology architect I was very lucky that we happened to have this group that was part of the original MSNBC.com and that was a company called Newsvine. And I don't know how many Newsvine users are in here but they were kind of pursuing this kind of community news space since around the mid 2000s I think they were acquired by the MSNBC.com joint venture in I wanna say 2007-ish and a group of really talented developers that we chose to partner with in building the community and social tools. So those were the three kind of technology stacks that we had to integrate to create this platform. What was your plan around mobile and what ended up being responsive and stuff? Was that your trajectory originally or how did that end up coming up? Well we had a lot of discussions about it. There's obviously the whole concept of responsive design was preeminent in everyone's minds. There's also a school of thought that I think for sophisticated publishers CNBC is an example that uses an approach where they build very customized templates and serve content based on user agents. So their philosophy was that if you can build these separate experiences, you should. And we chose that for I guess really for our not only for the technology and the user experience but also for the timeline. It would be a smarter decision to go with a responsive approach. Part of that for us as well was starting from absolutely zero. We had no apps. We had no presence anywhere and we just couldn't roll out everything together. So responsive was a neat solution. I will say, and we touched on this in the video, there were some interesting and long and passionate discussions with the Lullabot team about where our designers really mobile first. Was it just, was it mobile optimized? What was this thing? And I'll be honest with you, we fudged it. And I actually think fudging it was the right choice to make in terms of the design because I see sites today, new sites that are really perfectly mobile first and in the desktop experience which still accounts for the bulk of our audience, it's unreadable. I mean, they are hideous, hideous things. So, yeah, you can break things down to have great mobile experiences and that's really, really important. So I'm not trying to minimize the importance of the mobile audience at all. That's clearly where all the growth is and where all of our audiences migrating to at a fairly rapid pace. But right now, whatever it is, middle of the day East Coast, about 80% of our traffic is coming through desktop laptop. And if we had taken the approach that some other publishers had had with this, you know, strictly ideologically pure mobile first approach, I think we'd have really been in trouble because there's no hierarchy. The hierarchy is all resolved by the break points. So I think from a design point of view, we flipped it around. Some pages we went desktop first, some of them we went mobile first and I'd prefer to think of our design being mobile optimized rather than mobile first. I think this was really interesting because this was kind of one of the early big areas of discussion in this project was how are we going to handle this? What does this really mean? And I think what was important was that we kept bringing back to everybody's mind that what we cared about is whatever we do, it has to work across everything. And that is the end of the day was whatever we do has to work across everything. So let's get into the stack. We have a slide here that, funny enough, people on the panel won't be able to see. So I'll describe it for people on the panel while people in the audience are able to see it and those of you on YouTube are cozying up with it. So let's talk sort of about the stack. I'll just sort of list off kind of what's part of it here. Maybe I'll sort of turn it towards, no, John can't see it. He's got a slide. You're playing along at home, great. So do you want to run through it, John, maybe for us then if you can see it there? Yes, sure. I think we just talked briefly about it. We host via Acquia on their cloud platform. We, again, we're partners of this internal NBCU team and they actually manage that relationship for us. So physical infrastructure is there. And I have to say I came from a previous role at another media company where we managed our own infrastructure and got very used to, on an engineering level, being able to get in, make certain kind of changes to your stack. When you move to kind of having this managed infrastructure, it can be kind of jarring, right? You have to kind of give up some responsibility, some, I would say you have to give up some customizability, but you give that up and you gain scalability, you gain, basically you get the ability to walk away from issues like running out of CPU and RAM and whatever and it just becomes someone else's problem. And that's really nice. You get used to that pretty quick. Then obviously we talked about the choice of Drupal. The really neat thing about NBCU is that we have this Publisher 7 team here. Shout out to the Publisher 7 team. What they're doing is really they're taking Drupal and building essentially a kind of core distribution that all the brands at NBCU can then take advantage of. So they've got a development team, they produce this Drupal distribution and we get to kind of gain the advantage of, kind of getting to offload some of the development that we otherwise would have to do ourselves. So that's really neat and then we expect that kind of partnership to grow over time as all these NBCU brands start sharing more code and sharing development cycles with each other. So that's neat. We layer our own customizations on top of that. Again, we talked about Newsvine as a technology stack. We do some neat integrations where we're sharing data back and forth between the Drupal stack and the Newsvine stack. If you look at the site and see how it's managed, you'll see groups pages and user pages, they reside on the MSNBC.com subdomains and the experience is very uniform because we share the same, we share the headers and footers out to their system but it's all hosted on a completely separate stack. And then we have our video which we use the platform for video which is a Seattle-based company that a lot of our NBCU partner brands use and we have our own kind of user experience built around the platform's player development kit. So UGC here stands for User Generated Content for anybody that's curious about that. And just to sort of expand on that a bit, so Newsvine is being used for all of the user-contributed content, comments, anything along those lines. So user registration, all that is handled by Newsvine. So although Drupal has that functionality, we're not using that for MSNBC.com. Just wanted to be clear about that with everyone. Karen, you wanna talk a little bit about Publisher 7 and how it was used? Yeah, so again, as John said, we, the ONTS team, which stands for Operations and Technical Services, I knew that. But the ONTS team had already started on this Publisher 7 which is a distribution. So it's a package of modules and features and functionality that they had already put together for NBCU, they were in the process of building out the Drupal 7 version of that when we started on this project and it wasn't completely ready. So we started with what they had and built on that. And so we kind of built on, we built first on Drupal and then on top of Drupal we've got the Publisher 7 functionality and then on top of that we have our own customizations. And in some cases we have things that can be contributed back. In some cases we've got things that are just pretty specific to MSNBC that don't make any sense, but we've got all that flexibility. And as the project, as we got into the project we could identify that we had a need that was likely to be a need that other units at NBC would also have. And that would be something where maybe this is something that needs to go into the Publisher 7 stack or an improvement for that or whatever. And then we had other needs that were, again, very specific to the particular way that we were rolling this particular project out and it didn't make any sense. So we tried to work with the Publisher 7 team a lot back and forth on that and they actually got involved in some kind of sub projects because they were things that had applicability across the rest of the network. All right, I'm aware that we wanted to leave a good amount of time for questions. So I'm gonna kind of keep buzzing through. So this slide has the sort of explanation of the different teams that worked on things and stuff like that. This is not even beginning to be complete because there were so many players in this project and in this process. I just tried to sort of call out some of the key ones. Obviously we had MSNBC and their internal resources. We had ONTS, a company called Sapient Nitro that was working on the design aspect, building out the designs for the project. We were doing development and some of the design work. We had Newsvine for the user-generated content. We had the platform for video content and then we also had the video team that was available on top of the platform. We had several other Drupal shops were involved. Phase two got involved. We had a group from ThinkDrop that was on the project for a while. We had at least one team member from Chromatic. So we had a number of Drupal shops that were involved in that process. And actually what I'd like to do, we've got a lot of these people in the audience. Could everybody who had anything to do with the MSNBC project that's sitting in the audience right now stand? Because I know we've got questions. Yeah, if you're listed on this slide or you're feeling like you should have been listed on this slide, stand up. And this is not everybody, obviously. This is just the ones that made it to DrupalCon. So it was a big group. All right, so one of the things that was interesting about this project as is the case with a lot of these large scale projects that we work on, is sort of the communication mechanisms, especially when you're coordinating so many different teams and groups and stuff like that. So we've got a slide here that sort of talks a little bit about the different technologies that we used. I guess that there's sort of a larger discussion to be had sort of in the, not so much the technology, but sort of the methodology and sort of techniques. But let me just sort of list off here. We used a lot of teleconference phone lines. That's sort of a Lullabot thing that we do. We like telephone lines a lot. IRC, Internet Relay Chat, Online Chat System. If you're at DrupalCon and not familiar with IRC, by the time you leave here, you will be. Email, good old email, nice sort of fallback. Good place for asynchronous discussions and stuff like that. Join me as a screen sharing system. Google Hangouts is popular, but more difficult when you have larger teams or people that are on the road, stuff like that. And then calendaring stuff just to make sure everybody's in the same place at the same time, despite different time zones and stuff like that. Did any of you have anything to sort of add to this communication stuff? Richard's shaking his head here. Yeah, look, the only way we pulled this off was kick-ass project managers. And I mean that. And nobody likes project managers. I'm not even sure project managers like project managers. You know, from a business point of view, they're the ones telling you you're not gonna get what you want. Obviously the developers are all being whipped. And so they're always the people that get zeroed out of any discussion budgets and everything else. We could not have done any of this, even with all the tools and all the talent and all the smart ideas without the best project managers on each team, but also centrally pulling the whole thing together. So I had no idea what they did before. Before I joined this whole thing and now I'll set up a tabernacle to praise them because I don't know how you get through a complex project like this or anything like it without having you're actually the best people doing project managers as opposed to someone who was an intern six months ago. Especially at this rate, the feeling I had on this project and you guys can chime in, but in sort of watching it progress was that basically at any given time there were project managers talking to each other, monitoring what was going on and just a constant discussion of how the development was gonna happen, how the process was going to happen that was happening always, constantly, planning out even what to filter down to the developers, when to filter it down to the developers, and that really helped the speed and efficiency and ultimately sort of the success and happiness of the project. Yeah, there were, and I thought of even more things after I put this slide together that weren't on there, but I mean we did Google Docs and Hackpads, share documentation back and forth and email lists. We ended up doing a really simple thing that just helped tremendously. Lullabot had one email list that would be copied to everybody at Lullabot that was working on the project and MSNBC had an email list that would be copied to everybody at MSNBC that was working on the project and so you didn't have to constantly try to figure out who do I send an email to to hit everybody? You could just hit that list and everybody got copied but that was a huge issue. But yeah, all this communication was like you have to bend over backwards, you have to over communicate, you have to communicate until you're sick of communicating and that's still not enough communication. It's just so important. Let's run through this next slide pretty quick but some of the tools that we used in particularly in the development process of this project, do you wanna hit those quick Karen? Yeah, the original idea was we were gonna kind of do discussions in Basecamp and figure out what we were gonna do and then once we figured out what we were gonna do we were gonna document it in confluence as that was a wiki that was gonna be where all the documentation would go and then GitHub is what we were using for the code repository and then Jira for the ticketing system. For the most part this worked out okay, I think using Basecamp for discussions was a fail. We had a million discussions and things kept getting lost in there and all that kind of thing but the rest of it, it worked out pretty well and Jira was enormously flexible and which was really important because we had thousands and thousands of tickets and we needed a good, really complex, really sophisticated ticketing system. Cool. This slide just says goals. Again, let's try to sort of hurry along here but let's talk about sort of the general goals for the project. I mean, we're gonna talk a little bit about the timeline but kind of what were the business requirements and ultimately sort of creative goals of the project that needed to be hit? Well, apart from the obvious one which was to launch on time and on budget and that wasn't a small one at all. The goal was to translate this spirit of MSNBC into the digital space and to a large degree reinvent MSNBC. There was no template, although obviously we were working with templates but there was no path, there was no map, there was nothing we could copy and that's why the integrations were really important. It wasn't just that we wanted to have some video there. We were already streaming a lot of video and we currently do. We stream pretty close to about a million videos, streams a day and people stay with our video for a very long time. So it had to be robust. It had to, these integrations, we couldn't just pull out video and say, well that's the MSNBC experience. We had to have this connective tissue between the video, the text, graphics, photos. We wanted a very strong visual identity because we were growing out of a TV brand so you'll see that right from the homepage and then we wanted to give this room to the community space. So, and not just as an add-on but as the infrastructure all the way through. So you would see community at all points and one of the best ways to understand that is to go to an individual article page where all of those elements come together. The community aspect, the strong visuals, embedded video, it's all there and something that actually is shareable. About 20, 30% of our traffic comes from social media, third-party social media platforms, obviously mostly Facebook but we know that people are sharing, we're going to share the individual stories and pieces of video. So business goals were super ambitious. That's all just on the editorial side. We didn't even talk about the integration on the pesky ad side which is not a small thing as well. Our business taskmasters want a path to viability. The days, although we were sort of in a startup mode for a big media brand which was a bit like the 1990s, it certainly wasn't a business mentality from the 1990s where we could say, you know what, we'll just get eyeballs and figure out the money later. That there's a strict business plan. The great thing is we actually beat our traffic goals for 2017 in January, three months after launch. So this slide shows the original timeline which encompasses four months. What happened? Oh, is that one for me? I thought you were going to explain how... So what happened? We needed time to get our shit together. That's the technical way to explain it. So we didn't really have the project defined, designs done, completed, not even substantially never mind actually completed until the May timeframe. And part of that was, which was I think initially when we were supposed to launch. So part of that was, although we had a clear sense of our identity and the various elements that we would develop within this digital space, it took us an inordinate amount of time to figure out the decision-making structure. Getting to a system of decision-making at the editorial brand business level, the real, the key stakeholders was a challenge. And what we ended up having to do was have a very extensive consultation system. So each layer of UX went to, I'm gonna say three dozen stakeholders. In the first round, every single one of those three dozen had about 30 comments. I'm happy to say that about the seventh or eighth round, there were only six people who were still in the conversation. So there was a certain winnowing out but there was all that expectation built up or there was no website, there was we were creating it from scratch. It had to be ideal, it had to be the best thing ever. And what we had to do at the sort of senior level was obviously involve people, get the requirements, the key stakeholders involved, but try to narrow down the decision-making, the actual driving force behind the project down to something much more manageable, which really ends up being two, three, four people. So that really, that's the sort of high level of why we blew through those initial deadlines. They weren't feasible. Again, this was a group of people who said, don't have a website, how can it take so long? What is a website anyway? They seem to be, I can click on these things, so how hard can it be? I'm really not exaggerating a great deal. Even today, there's still members of my team who think that the system we use is gerbil, not Drupal and know that it's actually true. And we just had to do a bunch of presentations to the broader MSNBC audience and explain what CMS is and all of these various points of integration, how video gets played and how it's integrated and what this community space is. So the education thing is really still ongoing and that's among people who are sophisticated users of the technology, but they have no clue. I guess it's like driving a car. They have no clue what's going on to the hood and no real interest either. So we talked about sort of what it took to keep the project on track. I mean, we're talking a little bit about the timelines here, but a big part of it was around the decision-making process and sort of the weighing constantly, this process that was going on with the project managers of sort of weighing the level of effort of a given task versus the importance. On Lullabot side, we've found that we've had the greatest success in hiring technical project managers, people who are familiar with the inner workings of Drupal and are able to make estimates based on experience. And I think that having, being able to have those discussions with the more, well, less technical, they have other talents, project managers on the, MSNBC side with more focus on the understanding the business needs of the project and stuff like that was really helpful and important and ultimately pretty efficient. Yeah, this whole process, this was a long process. The process of going through the designs, you remember Richard talked about what was going on in the business side and then once these things would filter down to us, then we had to go through the iterative process of trying to figure out, okay, what does that mean for us? How can we do that? Can we do these things? Do we need to suggest that they change some of this stuff to make it into something that's more feasibly done within a tight deadline? But it was a constant process of balancing those things, of trying to say, okay, this is a really important feature. Yes, it's hard, but this one's really, really important to the business. We gotta figure something out, or not, or this is really hard. We're gonna have to figure out a way to de-scope it because it's just not gonna happen. Yeah. And that really was the critical point of management. These very fluid discussions, very intensive decision-making, and the creative thinking among the developer team about, well, you want this, but if we tweaked it this way, we could get you to where you wanna be much quicker. And I would classify that whole thing as a sort of de-scoping or re-scoping discussion, and that really gained strength as we got into the final month or two, where we were just throwing things over the side of the air balloon to keep it afloat. I think we're going to the side of the road really fast in that last couple of months. But that was a really critical point of the conversation that had to go on between the business, the brand, and the developer team, really through the project management team. So the timeline got modified, a little bit more realistic, but still being able to meet most of the business goals of MSNBC. We decided a fair amount of sort of key dates for things. Did anybody wanna sort of chime in on these? One thing I wanted to mention that I think is really important is we had an original timeline that wasn't realistic, and everybody recognized that it wasn't realistic. And then we had a lot of discussions about how to make a realistic deadline. What could be realistic? And the business went back and they came back and there was a decision made, okay, we're gonna move that deadline from May to October. So I think it's really critically important to say, what didn't happen is we didn't just keep missing the deadline, missing the deadline, missing the deadline. That's not what happened. We reset the deadline once. Everybody understood that wasn't gonna happen again. That was a one-time thing. We were all committed to make sure that whatever we did, something was gonna get delivered in October and it was gonna be good. Yeah, and our calculation on that, our calculation on that was to say, if we extended the timeline, how much more would we achieve? At a certain point, you gain momentum and you're heading towards a fixed point and an extra month might not actually achieve a whole lot more. There's a quality argument, so you don't wanna push things to breaking point, but having an impossible deadline can encourage people to do really creative, amazing, productive work. So it's a really fine balance there and the one thing that we did, which was really important, was not to share the ultimate launch date and not even internally. We had an idea, a window, but we didn't say internally, nevermind externally, we will go public with this thing at this point because that would have led to a really distorted view of what was a very fine judgment call about how hard could we push the team, how fast could we do this. We knew we wanted to hit this calendar year of 13 and ideally the joint venture deal had been unwound in July, so we were trying to be closer to July than to December and so it wasn't that much more sophisticated than finding a midpoint between the two, but keeping it out of the public eye and keeping internal expectations to a relatively low level was really important. And it launched. Ta-da. I'm gonna switch over to questions. If anybody has any, there's a microphone up here. If anybody has any questions, we're happy to answer them or we'll just continue to talk amongst ourselves. I skipped over a whole bunch of slides so I can go back and cover various things but I'm sure we've left you with some questions. I have a lot. Go right ahead. There's no one behind you. I'll just do two. Thank you so much for your presentation but how much user testing do you guys do with your editorial team? I'm sorry, how much testing? User testing. Oh, user testing. With the editorial team. With the editorial team. Not enough. How much user testing did we do? There was extensive discussion about what they wanted, what they needed, Karen in particular was leading that so she can talk through some of that. We didn't have a whole lot of time built in to do the kind of testing everyone wanted to do because it was so rushed. This was final designs in May and launch in October. That gives you a sense of, we tried to get some feedback going but I'll be honest with you, that all got compressed. That's user testing in terms of actual users as well as editorial. Yeah, the whole process of dealing with the editorial process, obviously you can't create an editorial process until everything's in place. Everything's built. You can't say how are the editors going to create these things until the things are created. So by definition that has to come really late in the process and that means we don't have a lot of time, there's nothing you can do about it. We had a lot of conversations with the editorial team about what was good enough for launch. Okay, what was really horrible about the experience? We tried to fix the really horrible things. We tried to get as good as we could get on the rest of it. And that led to my second question. How much of the editorial workflow is actually integrated with CMS? Do they start with a lead and it's all in the CMS or is it taken from Word or some other platform? So writers don't trust any CMS, they just don't. So they will still cut and paste things and then have all sorts of horrible formatting questions, at least for the copying homepage editors. Once it's in the editor's hands, it all stays there. So there are a couple of brave writers who do trust the CMS to not erase all their work. Video is a whole different workflow, so that's vastly more complicated. I will say that we, just to pick up on the last question as well, it's been very iterative. We keep improving the editorial tools. Even as we speak, there are people working on improvements to the editorial tools. The editorial team really wants to be able to track content much better than we currently do. That's still a challenge for us and that's something we're working on, but there's pretty much every week discussions about how we can simplify workflow, streamline it, improve it. Photos in particular, we are a heavy consumer of photos and that's a challenge with all the breakpoints and the re-cropping and everything else. Maybe it's a different discussion or an offline thing, but for photos, how much control is that given to the editor or the person who posts the stories online? Is it separate? We have photo editor team. We don't let the writers do anything but write. And that's hard enough, believe me. So the photo editors really control this and they're very precise about what crops they want and where as photo editors should be. From a technology standpoint, that kind of division of labor presents some issues which we're trying to figure out. How do we solve a photo team that has the rights to do certain things and make changes, let's say, to a homepage independent of the story editor and how do you kind of keep that, create a good workflow where those editors aren't colliding with each other. So that's something we're trying to figure out and I see our developer team over there going. Yeah, we're gonna get it. It's a really interesting workflow. I mean, for most of us, we're familiar with a blog where it's just you. This is a whole different kind of workflow where you have multiple people on different teams working on it. We've got a few people lined up for questions now, so let's... Thank you. You're welcome. Hi, two quick questions. First one is what discussions if any took place identify functionality differences between your native app and your mobile website and then the second one is I saw a date up there for design freeze. Did you approach the project methodology here using waterfall as opposed to agile and why? Thanks. The project manager is actually sitting right there, so she could talk about the philosophy that we used. First question was... Oh, app, the app question, yes. So the app we have currently, the MSNBC app, which is the first ever MSNBC app, was really courtesy of the NBC Universal Partnership. It's a sort of placeholder for us. It's a really elegant placeholder, but it's really driven by the video experience. We'll be launching our own native app in iOS and Android later this year, fingers crossed. So what was the discussion around that? Part of the idea of having a really responsive site was so that you could have the full experience in the mobile browser. That was absolutely central to the strategy. I think that we, looking back, believe that the mobile browsing consumption patterns would grow more strongly across the board for all publishers than they have. In fact, all the data I've seen shows app consumption growing, so that really does push us into having a much more robust app strategy in our... Our guiding principle at this early stage is really that you can do everything in the browser, so actually, apps shouldn't be that. You shouldn't be doing everything in an app. It needs to have a different prioritization structure and different features and functions. And I think the mistake of a lot of publishers in the app space is to just repeat everything they can do in the browser, in which case, what's the point? Jen, did you want to deal with the waterfall question? So I actually came on to the project in June after the design freeze. So I was kind of in an interesting position because we had designs and then a tech team that didn't know quite what to do with the design and a lot of folks were kind of frustrated that they felt that a waterfall approach had been taken. And actually, my mentor was the one who came in and helped define the decision-making structure and I don't like putting labels on it, but the first thing, kind of the methodology we went with is first define the who, define the team. That decision-making structure had been set up when I came in, so that was fantastic. And then there's really just the three, what we call cogs of project management, right? Scope, schedule, budget. And that's it. Whatever way works, we had really, really great technical PMs who were familiar with the Agile methodology and they knew how to run their technical teams, so I relied on them for their technical expertise to run their teams. The problem is we had 10 development teams. So it really was about kind of, I guess you could call it Agile because it was self-managing teams, but that's really what we had, but it was, I'm not gonna say we used Waterfall or Agile, but what we did was set up a roadmap from June to October and every two weeks Sprint had a theme and it was, we're gonna get as much done as we can on this specific area and if it doesn't get done, then it needs to be good enough and we're gonna move on because if we don't move on, we won't get to launch. So it was very, I think what led to the success from a PM perspective was also that we had complete support from our senior management team in terms of that de-scoping, re-scoping, letting the teams kind of take full ownership of what they had to deliver and not tying them to a very specific design, but does that answer the question? Sorry, okay. I think we're only gonna have time for one more question so that we'll, you guys can fight it out. Oh, look at that, wow. That's an interesting move there, right on. Did you pay him money to get a cut in line? I like it. I don't know. I mean, I heard you earlier, John, talk about Drupal being a solution where you could customize the content types for kind of the future of news and the way stories evolve. That being said, there've been a lot of new kind of new sites like Vox Media, Buzzfeed, very social heavy as you talked about the social integrations. So it's interesting to hear you guys talk about Drupal being able to meet that when we see sites like Buzzfeed and Vox Media saying we have to build our own system to handle this. So the question is, what sites are in your, that you would find SMEC comparative to? And do you feel like you're with Drupal abreast of kind of the trends that these other companies are trying to tackle? Do you see yourself in that kind of echelon of those sites? That's a great one to finish with. I mean, that's a super broad question. I think Richard could probably better talk about what sites he thinks we compete with. I guess if in a general sense you think, and that's a great question about, there was a whole series of discussions happening on the internet recently about CMS as competitive advantage that some of these sites were talking about, oh well the reason Business Insider or the reason that Buzzfeed is successful is that they developed their own technology. And I think we could have a session about that question entirely. I think I fall on the side of that, if we were a VC backed, funded organization that had the kind of money that Buzzfeed was backed with then creating a custom built from scratch, CMS might be an option. But you know, honestly for our organization and budget wise, it's an impossibility. It's unrealistic to think that we could have gone out and created a custom CMS from scratch. I would say that Drupal is probably the next best thing to that, right? It's a framework for building a relatively custom application and I think, in our case, really kind of a no-brainer. The comparison set is huge. It's everything from CNN.com, a giant portal era news site through to a small progressive site like Talking Points Mamo. I mean, it's a giant area and pretty much every place you can watch video, including YouTube. So the competitive set is enormous. I have to say, we've looked very closely at all of the claims people are making about their custom built CMSs. We actually just met with Vox Media. I'm a huge fan of what they and Buzzfeed do. I do not believe, however, in spite of their press releases and their press coverage, that their CMSs are, in fact, the secret source, no matter what they believe. I think the culture of development and very close alignment of editorial and the developer teams, I think that is the secret source. So they may express that culture by saying they've got this kick-ass CMS. At what point is your CMS the advantage or is it your listicle of cat videos? I mean, you know, it could be that the cat videos are actually really popular and the CMS is a marginal influence at best. So I do think the culture of innovation, of design that really has a purpose of knowing your audience and knowing your brand and having technology that can serve that, that is what matters. And I, like I said, we sat down with the Vox folks, they made those claims, we pressed them up. Like, what exactly is it that makes you so happy with this thing? And it turns out it's having the editorial leader sitting at the same desk with a project manager who can then pass on the editorial needs and desires directly to a small, fast-moving, design development team that can get things done in a very compressed timeframe. That could happen for us if we were set up like that. Sometimes it does, sometimes our projects are more ambitious. It doesn't always happen like that for Vox. I just, I still don't believe that it's really their custom-built CMS. I think it's a philosophy. Great. So just the tag on here, Lullabot is hiring. We work on awesome projects like this and have others and are looking for great people. MSNBC is also hiring. Lullabot.com slash jobs, or find John. Find me, we're always looking for great developers, especially if you're a front-end architect and you're looking for, please speak to us immediately. Also, I put about 30 pounds of weight in my suitcase that has. Oh, we've got swag. MSNBC.com bags, I have Rachel Maddow, cocktail shakers, which are really, really enjoy yourself at home. If you ask the question, be sure to come up and then we'll have stuff left over after we give stuff to those people. Also, Lullabot has a booth in the Expo, the exhibit hall. Stop by if you've got more questions about the project or would like to find out more about what we're hiring for. Thanks, everybody. Enjoy your DrupalCon.