 All right. Good afternoon and welcome. We are with the University of Colorado Boulder, which is just about 30 miles northwest of Denver here, and we're very excited to share with you today. CU Boulder's journey into Drupal. You can't hear? Did you hear any of that? So I'm Joe Bertrand. I'm the web manager in our university communications department, and my co-presenters today are Mike Carter, our integrations manager in our office of information technology, and Matt Tucker, our lead web developer in university communications. And in the spirit of our open source community, we really wanted to share with the community, particularly our fellow universities, our experience as a large university in developing and implementing an enterprise level Drupal environment to be used as a campus-wide service. In particular, what we're going to go over today is we're going to give you a brief overview of some of the challenges that we're faced on our campus, the solutions we identified, as well as some of the research and planning that we put into the project. Most of the presentation is going to be Mike and Matt going into some of the technical details of how we built and architect our web infrastructure, how we configured our Drupal environment, as well as our deployment process. We will also kind of wrap up with some, like, lesson learns and expected outcomes of our project. We would like to take questions after our presentation, so we strongly encourage you, if you do have any questions, if you want to tweet them, to use the following hashtag. We've had some staff members up here who are going to be monitoring for us, as well as if you want to just, like, jot them down. We have plenty of time for Q&A afterwards. In addition, we have also scheduled a buff called Drupal in higher education. It's at five o'clock this evening, and that's going to be in room 206. Just out of curiosity, how many higher ed institutes are in the room? All right! Well, we're very excited that you're here, and we're hoping that you can join us at five o'clock. We'd love to see what some of the things that you guys are doing as well. So this is the University of Colorado Boulder. Our university is located at the foothills of the Rocky Mountains, and we have a population of about 29,884,000 degree-seeking students, with approximately 24,000 of those are undergraduate students, and around 5,100 are graduate students. We also have approximately 5,600 faculty and staff on our campus as well. We are a tier one research university, and what that means is that we have very strong research partnerships with federal institutes and labs such as NASA and NOAA and NREL to name a few. So in relation to that, we have about nine research institutes and approximately 80 research centers on our campus. We also have eight colleges and schools that are overarching with architecture, arts and sciences, business, education, engineering, law and music. I think I got them all. And within those, we have over 85 departments within our colleges and schools. So as you can see, we have a fairly large internal audience base that we are serving, and I think that helps to kind of put in perspective for you some of the things that we as a team really needed to keep into consideration as we were looking to roll out a new solution for our campus. Some of the challenges that we were facing on our campus was that we were sitting on an extremely outdated server environment that would not allow us to use the most latest technologies such as a CMS. It was really just a static HTML environment where we can use Perl or CGI scripts to do some of the more dynamic stuff, but it was extremely limiting. And our departments were finding it very challenging in trying to build and maintain and update their websites in that environment. In turn, a lot of our departments were starting to jump ship. They're starting to jump off of a free campus-wide server and going out and purchasing and supporting their own server so they can take care of the latest technologies on their own. We did a survey about two years ago and found that there were over 600 surveys across campus sitting under people's desks in their closets being used as coasters that our departments were using for various websites and web application needs that our main campus server could not support. It was really quickly becoming an IT nightmare for us in a very decentralized environment and just not the best resources for our campus. And many, many of our departments were struggling with this. On top of that, we were getting a lot of complaints that our university web presence, our website, was completely laggy. It was out and out pretty lame. I was involved with this, so I can say that. Because we were stuck on this static HML environment, we really just did not have the capabilities to really showcase some of the amazing research that was going on across campus and some of the great things that our faculty and staff were doing. One other key challenge that we also identified is that because our current server environment had been around for at least, correct me if I'm wrong, 15 years or so, that we had a ton of legacy data on those servers. Mike did a quick tally and he found that we had about 700 websites on our campus environment. Some were probably had been abandoned and some were still active. And so we needed to make sure that whatever we came up with with our new solution that we had to figure out what we were going to do with all of this content and all this data on our current server environment. And then on top of that, our university rolled out a new brand in the spring of 2011. And we were charged with a two-year time frame to conform all of our department websites into the new branding standard. So we quickly realized that we needed an entirely new environment to help us to manage this brand and to roll it out for all of our different departments. So we identified three main solutions to help us with these challenges. First and foremost, we completely needed to modernize and revamp our entire campus web infrastructure and really build something that was enterprise class. It was highly available, highly scalable, and something that will meet all the particular server needs that we were seeing across campus. We also identified that we really needed to convert to a CMS environment and really leverage the power that a CMS can provide us in really streamlining our web development process for not only the university website, but how we can roll that out and create a service for our departments for their websites as well. We also identified that we really needed to create a completely new service model for our campus around our web development process as well as web application hosting. Looking at a service that would allow our departments to very easily build and maintain their own website without having to go out and hire their own web staff or technical staff, as well as a service that will allow our departments to take advantage for all their different web application or database needs. So as we started to plan out this project, we really realized that to do this correctly, we needed to do, first and foremost, we needed to invest in equipment and invest in our staff. The equipment part was pretty much a no-brainer. We were sitting on some pretty old hardware that we really needed to invest and bring in some new hardware to really build a more robust technical infrastructure. But we also realized that if we were going to support this whole new environment, as well as this whole new service, we wanted to provide our campus. We really needed to invest in our staff so that we made sure that we had the bandwidth to be able to support the 36,000-plus people that were on our campus. We also did a... We took a deeper look into the type of service model that we really wanted to provide the campus around our new CMS environment that we were going to build. And we identified two key areas that will meet a lot of the different needs that we were seeing. The first is what we've been calling for now is a self-service template environment. And the idea behind that is really to have an environment that our departments can go to, log in, give us some credentials, and we provide them with a suite of packages of pre-developed designs that are within brand standards, of course. And as well as pre-configured functionality that the departments can really kind of turn on and turn off based on their specific... their particular needs for their website. And the idea is that once they kind of give us all this information, they pick their package that we would have this environment ready for them within a 24-hour or less time period so that they can then start to build their website on their own. We identified that this will probably meet, you know, a good 50%, 60% of our campus needs. We also identified that around our CMS environment we wanted to provide kind of more of a professional service around building custom websites so that for those departments such as our larger ones like admissions or financial aid that have very specific business requirements for their website that we could build a custom theme. We can build custom content type, a really a custom Drupal environment just for them and all of this supported by our team as well as on the new campus infrastructure. We also identified that another service we wanted to provide was related to web application hosting, something that will help us departments get rid of all of the 600 servers that we saw across campus and really kind of come back in and use the campus-wide service model so that we can kind of get away from this decentralized model that we were in and really create something that's more of a consolidated and streamlined service for our departments to take advantage for their various server needs. It could be for a non-Drupal website. Oh, no. We won't mention those others. It could be for, you know, file storage. It can be for backups or database needs. So the idea behind it is something that's really flexible for the various needs that we are seeing across campus. We also started the plan for how we can create a process to phase the migration of our legacy content into either our new CMS environment that we'll be building or into this hosting environment that we'll be building. We realized that there's just no way that we are going to be able to convert the 700 sites that we have identified within our timeframe into a Drupal website. And so we really need to make sure that we could provide a way to have both our new shiny infrastructure as well as our legacy infrastructure, both under the Colorado.edu domain name. Some of the research that we put into this project is first we created a list of requirements for our CMS so that we can use that to evaluate all the different tools that are out there. I think we looked at about 15 or 20 different CMSs and Drupal by far rose to the top very quickly. It not only met all of our requirements on our sheet but really just kind of went beyond that. And the other thing that fit very well with us is the whole Drupal community. The idea that there's this amazing community that is really dedicated to advancing and growing this environment just really was a great match for our campus. We also did some research into kind of an internal audit of our staff. As I mentioned, we identified that we're really going to need a strong staff support to run this new environment and new service. And so we looked at what we currently, who we had on staff, what their roles were, and looked to see where we can maybe reallocate some staff, redefine some job descriptions. And once we finished with that, we could see where the holes were. And so we identified a couple new positions that we needed to hire to make sure that we can provide a nice, robust service for our campus. The other thing is we reached out to our community external expertise. We talked to about four different universities that have implemented Drupal in some form or fashion on their campus and really just asked them, what did they do with Drupal? How did they do it and was it successful? What were some of the challenges that they faced and what were some of the advantages on their campus for rolling out Drupal? And really just, really figure out if they had any kind of words of wisdom for us. And this was very, very valuable information for us and we really strongly encourage anyone who's about to embark on a very similar journey as us to really reach out to your peer institutes and find out what they're doing and connect with them because they're more than willing to help you out. We also realized that we engaged some professional experts to really bring us up to speed. Two years ago, none of us knew how to work in Drupal, let alone spell it correctly. Yeah, we spelled it wrong a couple of times. And so we realized that in order for us to do this correctly, we needed to bring in some experts to not only help us with the Drupal part and how we wanted to configure our Drupal environment but also how we were going to build this infrastructure to support that, how we were going to create a deployment process that will make it very easy for us to manage the many sites that we were anticipating that would be under this new environment. And then once we had identified Drupal as our CMS for our campus, we reached out to the Drupal community and that has been just an amazing experience for us and I think really solidified for us that we made the right choice with moving forward with Drupal. After we finished kind of all of our research and some of the planning, we decided that there's no way we can do this all at once. So we created a phased approach for this project and we're just finishing up with our first phase, which is really just building out the infrastructure, building out the foundation for this environment and for the service that's going to be on top of it. And on top of that, we decided to start out with the university website and we launched that on January 4th. So we did not have a holiday. We really wanted to start with the university website. One, so it kind of gave a face to this really amazing robust infrastructure that we just built and also to generate some buzz and excitement on campus to show off that we have now invested in this new technology for the entire campus to take advantage of. This is just an example of a site that is built in Drupal and letting them know that we are now in the process of building a service for them to take advantage of. And so phase two is where we're going to build out that CMS service environment that I was talking about, so that self-service template environment and that custom development environment and we're just about to start on that phase. And then phase three is when we'll start to roll out the web application hosting for all the other various service needs that we have on campus. And with that, I will pass it off to Mike Carter who's going to go into more of the technical details of how we built out our infrastructure. So I'm Mike Carter. I've been at CU pretty much since before there was even a web. First thing we had is some really core decisions to make about how we were going to try and roll up this service out. And obviously the first one that you probably have been hearing about for the last two days is do you do this in the cloud? Do you do it yourself? And there were some key points that made us take the road of doing it ourselves rather than hosting it in the cloud. We're a big university. We have data centers. We've been running data centers since the 70s. So the facilities were in place. The other thing that really drove that is that, as Joe pointed out, we had 700 top level domain sites that needed to be maintained and trying to figure out the networking for that in the cloud with what we were rolling out new. It was beyond our ability and comprehension to be able to do that in the cloud. And so we did decide to do it locally. The other thing that we needed to decide is how we were going to do this load balancing and web traffic management. We picked F5. We'll go into that in a little bit more detail later. VMware, we already had VMware clusters running. We have the people that understand them, the people that manage them day to day. So that was a pretty easy choice. And the last thing we did is we built a sandbox. We wanted to see if we could do this and how all these different layers that we didn't understand would talk to each other. So the first thing I'm going to talk about a little bit is the web traffic manager. And the big thing that that gave us is that we didn't have to move everything at once. As Joe pointed out, we had lots and lots of sites. We were going to bring one site, the university homepage up first, and then additional sites that had already been brought into Drupal, but were running either in the cloud or on other hosted services on campus. And the big part of that was keeping legacy alive. We've got a half a terabyte of spinning, static, and CGI content out there, much of which is extremely important to some department or some Nobel laureate that if we disturbed would be not very happy with us. And the last thing that I want to point out about that is that it does bring up a little bit of maintenance and a little bit of overhead that our university communications group wasn't used to in that they can't just spin up a new site and have it magically appear. They have to work with our networking group. We're developing a process. It's fairly manual today. We've got a tool that we're prototyping to make all that much easier. So what we're doing with the web traffic manager is directing really five different kinds of traffic to different places. So the first and most obvious one is the stuff that's going straight to Drupal, non-authenticated Drupal via HTTP. And we inserted a varnish layer in there. We'll talk a little bit more about that in a minute. And then there's the authenticated Drupal, which we bypass varnish. So we're doing the web traffic manager sitting there in the middle making all these decisions for us, data groups, all sorts of stuff in there that make that happen. We can also keeping just that one single domain route traffic to somebody else, our housing department. If they want to live under the www top level domain, we can route their traffic. And then of course the thing that we've been harping on here, which is our 700 legacy sites that sit out there on static HTML and CGI. So the really important thing about this slide from my perspective is that we have a lot of layers here and there are lots of VMs at each layer. So obviously the web traffic manager is sitting there at the top and that's what we'll talk about. Well, I just talked about that. Underneath that we have a web proxy cache for speed and we'll go into each one of those in a little bit more detail as we talk. So I guess the thing that I really want to point out here is everything here is in the open, all this stack is all open source from how we're providing our database files. I mean, you guys all know that the core of Drupal is all open source, but everything else that we're doing here, other than the web traffic manager, the F5s themselves, everything else is open source. So that web proxy cache layer is, for those of you that aren't familiar with it, that's just speed. Being able to cache pages, et cetera, in Miram, we're currently in our production environment, we have four. As at least one of the experts in our audience will tell us, he'll tell you, we probably don't need four yet, but we've got four. We're sort of in that more as better, but we may scale some of those down. After the varnish layer, we've got the web Drupal layer obviously. I think probably everybody here understands what makes that up. The one thing that's added to that is our APC cache, which again is just one more thing to try and cache your PHP byte code, speed things up. We've got an object cache layer here. We're using memcache. Again, we've got four. We may not need four, but that's where we started. Our MySQL is in a dual master configuration with the non-master node being a read-only node, and we're using Red Hat 5 actually because the pacemaker cluster that we chose to implement that on didn't support six yet, but that's obviously highly available. That's actually the only layer here from the Unix-y side of the world that's not virtualized, and we did that again for speed. The last layer, for what Drupal needs in terms of files, is an existing highly available NFS cluster that we use as actually the NFS cluster for our legacy sites. We've added a little bit more storage to that, and that's what's providing the site's default files for the UDrupal heads in the room. Specifics on hardware. F5s, we've got two for dev test, two for production. There are 3,900s for those of you who are shopping for things with the LTM package, that's the local traffic manager package. When we were looking at all these layers, we figured out pretty quickly that there was a lot of intra communication between these layers, and so we were using pairs of 10 gigabit switches to let all the layers, whether it's memcache talking to Drupal or varnish talking to Drupal or whatever needs to talk within these stacks on dedicated 10 gig switches, that solved two things for us. When it simplified our overhead wiring and it also made our racks more easily portable, which we'll talk about in a minute. Their Dell R710s is the base technology we picked, 96 gigs of RAM, 6 cores, fairly standard 2U units. The storage for all of this, whether it's the VMs or the highly available NFS, is what we call at the University of Colorado our enterprise storage, that's HP left hand ice guzzy sort of stuff, fairly standard. The one downside to doing something locally is obviously you don't get the geographic separation that you would in the cloud, or that if you do things right in the cloud, you get in the cloud. And so we planned for that, we haven't actually implemented it yet because our second data centers networking isn't ready to support the F5s or the highly available NFS and the enterprise storage at the moment, but I'll bet we're there within a year. And so before I let Matt here talk about where we are today and show you some pretty cool stuff, just a real couple of quick slides about where we came from. So this slide is actually our website in 1996 as seen in Netscape 3.0. And the most interesting thing to me about this is that Netscape 3.0 right there is actually still running on my desktop on an old Solaris box. So I was able to recreate that web image pretty easily. And the last one is just to sort of show where we came in terms of servers. In 1996, the server that was serving this page was a single Sun Enterprise 450. It wasn't really sitting in a hallway like that outside my office. But we've come a long way from one 450 to 12 Dell R710s, four dedicated networks, which is in four F5s. So Matt Tucker. Hi there. So I am the lead developer at the University Communications and Joe wasn't completely right in saying that nobody could spell Drupal two years ago because I've been using Drupal for about five years. But just joined the team about a year ago. And today, the University Communications maintains approximately 60 different Drupal sites. And most of these are still currently living on external servers that are not using this new production stack. But we are slowly moving these over as we have the time. But again, the most visible change was the Colorado.edu homepage. And what I'm going to kind of just quickly run through is some of the modules that we use to build this. Our search in the top right is using the Google Appliance. We have a Google Appliance sitting in our data center. And what just happened to my screen? My screen is gone. Tech help. Interesting. Interesting. Yeah. Well, we just might kind of have to make this up. Of course. Well, I'm kind of stuck. 25 seconds, cool. So yeah, our search is done with the Google Appliance. One of the big things that we really, really like about Drupal is its robust taxonomy system. So the primary menu that's at the top of the screen are all top level terms in a vocabulary that we're actually planning on sharing across multiple sites. So we're using, we've prototyped a combination of the services module in combination with feeds and there we go. Cool. I think this is just going to pan through again. So yeah, we're planning a combination of the services module and the feeds module to actually share this vocabulary across multiple sites on campus so that we all maintain the same general taxonomy system. There's two main node types on the site that are, of course we have about 15 or 20 different node types on the site, but there's really two main ones and that is news releases and feature articles. Feature articles are a little bit richer content. It allows our news editors to add in photo galleries and videos and slide shows, whereas news releases are a little more just basic text. The, all of the different displays are of course done using the views module with heavy caching on all fronts. The site itself is built on the 960 grid framework and it's built using a responsive design and so if you pull up the site on your iPhone or iPads you can see the site responding to the size of the screen. Responsive design is an interesting topic that Luke will go into a lot more during tomorrow's keynote, but it's really difficult, not necessarily technically. The media queries in JavaScript is actually pretty simple, but it's really hard to in a way philosophically determine what is the most important content for mobile users, what's the most important content for somebody on an iPad and how much of the site do you need to change for the various devices. And so we actually just launched this responsive design on Monday and so it's definitely still evolving. We're going to continue to change it over time. As you scroll down the site, there's quite a bit of content, but if you keep scrolling all the way down you get to a beautiful campus image. We're able to quickly swap this out for major events. It's snow day, we put the beautiful snow picture and when CU was recently in the NCAA basketball tournament we were able to switch out the image in support of that team as well. This particular functionality is provided by a module that I've contributed back called the Backstretch module and it's simply using a jQuery library that already existed but it just integrates it into your Drupal environment and allows you to very easily select which images get displayed as the background image. So kind of moving off of the homepage and on to the bigger scope of this project in terms of maintaining lots and lots of Drupal websites. A single Drupal website really isn't that hard. It's just a couple files that talks to a single database but when you start working with 20 and 30 and 50 and 100 sites it becomes extremely difficult and there's a lot of challenges that you run into in terms of keeping modules all up to date across all the sites and keeping Drupal core up to date and so we developed a module called the Drupal SimLink Manager or DSLM which its goal is to try to make this slightly easier. Basically what it does in order to use DSLM