 Red is good. All right, awesome. Hi, everyone. We will get started. So thank you for joining us today. So today, we will be looking at a business case study, therefore not a deep dive technical exploration into WebPAC, which is creating and maintaining 30-plus faculty of medicine website with a Drupal distribution. This is also not the Google WebPAC package manager thing. So just to clarify, in case you're in here, for those topics. So today, we will talk about four things, the background, the requirements, and the background, sort of the story of how we ended up with this distribution. Talk a little bit about the implementation and an overview of that. Then we'll talk about how we supported the growth of this distribution in the organization. So it's mostly a people process. And then talk a little bit about the technology infrastructure that we use to support it. And so while your mileage may vary for the talk today, what we hope to accomplish in presenting this talk is that maybe you can gain some insights from the challenges that we have faced and bring them back to your own projects. OK, so good morning. I'm Roberta Brown. So I'm the digital communication specialist at the Faculty of Medicine at the University of Toronto. I'm not used to adding that at University of Toronto part. I work as part of the communications office. And we're obviously dedicated to managing internal and external communications. And we produce a number of publications over the course of the year. Our award-winning magazine, I have the covers up right here. But we also do, obviously, dean's reports and those types of things. In my capacity as a digital comms person, I manage 30-plus Drupal websites and about, I don't know, 15 or so ever-growing WordPress sites. So I'm a little bit in two camps. Perfect. My name's Aiden Foster. I'm a front-end architect and UX designer. My company is Foster Interactive. And so we have been in business 12 years and we do a lot of work with U of T. So we've made the alumni website, Woodsworth College website, and most recently launched the University College website for them. And yeah, so if you're interested in getting in touch, please reach out to me on LinkedIn. So before we get into anything else, we really need to answer the question of what the heck is webpack? So I think depending on who you are, there's different descriptions of what this is. From a technical perspective, it's a custom Drupal distribution and it allows us to run 30-plus medicine sites that have a common branding and structure. We have a set group of features that people can choose to use on their sites. And our guiding rule for this product is really configuration, not customization. So we try to veer as far away from having any sort of really custom stuff done because it gets in the way of our maintenance. Yes, and this is sort of a walkthrough of some of the different installations of webpack. It's next. Okay, so how did we get to webpack? When I started in the faculty, the first thing that they wanted done was a revamp of our home faculty site. It was looking quite old and it needed a lot of love. And as we were going through that process, we realized that other departments and units needed to update their websites as well. And so we kind of had two problems. One was look and feel, which you can see over here, was varied and not identifiably always a U of T website, not brand compliant, but also units really didn't have the capacity in their own departments to be able to do a website rebuild. So they didn't have the knowledge and they didn't have the time. I worked with a lot of people that are admins that do comms off the side of their desk. So they needed help. And when we had done the faculty site, we did a lot of heavy lifting in terms of making sure that the website was accessible and responsive and met brand standards. So the idea was that the faculty takes the brunt of the workload and the cost and then they benefit by being able to use this product. So for the first year, it was all about just getting people onto the template. And as we got more and more sites on there, we realized that we sort of had some capacity problems propping up, hosting and support. So how do we deal with all these sites being hosted and supported? We were hosting on the U of T web servers and we were finding that that wasn't working out so well. So we made the move to Pantheon in 2015. And Pantheon we chose because they had the very specific set of developer tools but also security features that we really liked and it would really help us streamline the process of getting our sites sort of out the door. The second was maintenance. So we had our in-house IT team that was really meant to be handling maintenance. And then when we had bigger projects, we'd go out to foster to have that work done. And we're finding the capacity for the IT department was becoming a real problem. They didn't have the time to dedicate to the product and the product was very stable. So that's when we started a maintenance agreement with Aden's group and the IT department just stepped away from it. So now it's actually just managed between the communications office and an external vendor. So Aden's gonna talk about it. Yeah, so a part of the implementation question that we started with was, you know, we started with one website and then we needed another and then we needed another. And the sort of, it was planned but the demand that ended up happening seemed to be far greater than what we had originally expected at the beginning. So the first sort of solution that you can, you know, I guess perhaps the most obvious solution is scaling a website like this. Simply you copy the code base, replicate the database and change what you need for the second website and off you go. And the problem with this is as you increase the number of sites you have based from a common origin, basically you end up multiplying the cost of maintenance for those websites because they're all kind of unique, slightly different and if you have to update the modules on all of them separately. So that option was not a great solution in this case and probably in any case, basically. So another option that's available in Drupal which is really cool is Drupal Multisite. And what Multisite is is you have one common code base that runs a bunch of different databases and user files for those multiple separate websites. And while it is a powerful and flexible tool, it has problems of its own. Mainly it's a bit of a house of cards circumstance. If by some problem, something's not caught in a QA review and you push that code up to your main website, you end up with like 15 websites that all go down or have the same problem at the same time. Not to say that there aren't DevOps infrastructure solutions that can avoid these problems, but you really need a super strong internal IT and process put in place for that to happen. And we also find they're difficult to develop on in the sense that we had a number of sites that needed their own unique customizations and those become harder to manage in a multisite code base. So we ended up going with the Drupal distro and at a very simple level, when facing scaling, Drupal to me feels like it is a large box of Lego that comes with contrib and core modules and you can custom build exactly what you want for a website. And as I mentioned, that's not great from a scaling point of view. So the metaphor is like a distro is like a pre-assembled bundle of stuff that you can get out of the box very easily through a bunch of configuration, a bunch of modules and theme elements sort of bundled together in a particular pre-assembled package. So Webpack really has sort of two groups of feature sets, the core features like the landing page builders, editor tools and the theme. Every instance of Webpack has all of these pieces enabled by default. And then there's sets of optional features, news, events, faculty directory policies and FAQs. The individual instances of each Webpack website can turn on and off these particular packages of features so or have none of them at all. So the advantage of bundling these features is it gives flexibility and options to the site owners and it also has an advantage from a documentation point of view where if someone has a set of features, it will have the same kind of documentation requirements regardless of the website. So what these actually look like, we have the medicine website on the left which is kind of the Royale website that was the ancestor of all of them so basically it implements almost all the features and the comparative medicine website on the right with just a few features enabled. So you can see there's an example of the news feature and the events feature that appear on the medicine website and they simply don't appear on the comparative medicine website. But the landing page builder, which is a core feature is on both and it allows you to configure and implement some, you know, a nice robust layout in both circumstances but from a content perspective, the team, the comparative medicine team didn't wanna deal with the effort of producing news and events on their website or didn't want to. So the story of this evolution, it's like a long timeframe we've got here. We started off in 2014 and created the Faculty of Medicine website. Then internally, there's a group, the IT group at U of T cloned it twice and did that obvious solution, right? Like just copy it and spin off the two websites. That quickly, you know, the times three maintenance requirement that came out of that was a lesson that we're like, hmm, maybe we shouldn't continue on this path because we need even more websites. So in 2015, we created the Drupal distribution and got all three websites back onto that common Drupal distribution. We scaled up to about 10 websites in that year. And then in 2016, we actually had a number of users using these websites and they generated a whole bunch of features and ideas and things that they wanted to accomplish with the website that they weren't able to accomplish with the features that we launched with. So we ended up producing a number of different add-on components. So the faculty directory got a big enhancement and those kinds of things. And some of those websites, what's nice about it is some of those websites that are already live could add those on to their pre-running existing website and other ones just launched with different sets of things. Scaled to about 20 websites in 2016. And then in 2017, that was the time where even the maintenance of these websites on a common distribution started becoming a challenge to the IT team. And instead of, you know, instead of working on Drupal maintenance and support, they wanted to work on new product development, which is really their core mandate to make cool new things for the faculty. And so at that time, we took over maintenance and established it. Did an effort to set up some maintenance protocols, procedures and tools to make that easier for us as a team to manage. Continuing to scale the website. 2018, we did some minor feature updates. You're scaling up to about 30 websites. In 2019, we actually started doing, we came back and did an audit of the accessibility of the websites that we had on the code base because we had been, we had made sure that each website was passing accessibility when we launched it, but we actually kind of noted some problems that started to creep up because we weren't doing accessibility maintenance. And also keep in mind that accessibility tools themselves became smarter. There's greater clarity about how to interpret the rules that have just evolved over time. And so while they were accessible when we launched them, we were discovering they weren't accessible anymore and we needed to go back. And that's both from a code and a content point of view. These content writers got their training on how to do accessible writing sometimes five years before, right? And so there might be new team members, all of that. So just drift over time. So we did some audits, cleaned up some pieces of the code and also work with the site owners themselves to say, hey, here are some problems, let's start. Let's just, you know, not all at once, but just over a period of time to go through and improve that. Now in 2020, we have the conversation of, this is a Drupal 7 based distribution. We have to have the conversation of what are we gonna do moving into the future. So really from a maintenance point of view, it's clear what the advantage of a distro is instead of a cost multiplier for a website, you have kind of like a high upfront setting up a distribution, testing it and setting up your infrastructure. And then you get a small incremental cost increase per additional website. So, and you know, and just as another note that came through with this sort of evolution, as new features were evolved, initially we got a number of requests for people like I want this cool new feature for my website, just mine. And we started to try and steer everyone away from that to saying, how can we make that cool new feature something that's not just for your website, but just a feature that we can add into the bundle of features and make it available to everyone because it's much, much better to deal with. It has more value to the entire product suite than it does to the individual. And from a maintenance point of view, it is better not to have the unique customizations that are out there. All right, so supporting growth from a user management perspective. So I'm gonna talk about creation, communication and collaboration. So basically onboarding new users, user management and project management elements. So in terms of onboarding, we have a very set process for the onboarding. So whenever somebody asks for a site, if we take them through this process, we give them both communication support as well as technical support. So it's not merely, here's your package of website, there you go, go off. We work with them on their content to make sure that they're outputting a site that represents the brand in an ice way. Allowing us to really identify this process means that we can actually get people moving through this much faster and making sure everybody's on the same page. So all of the work that Aiden went through just now allows me to actually set up a site in a very short amount of time. That is not playing. Thank you. So this is actually me spinning up a site on Pantheon. It's actually, it means that I get to do this work. So I'm not a developer in any way, shape or form. But I can go in, set up the site, go in, turn on the features, do the initial layout work and get it ready to be passed off to the client. So I'm not dependent on Aiden for setting up a site. And likewise, the client, our internal client, doesn't have to wait a long time for their site because I'm not taking that at the door. Is it worth actually watching all the way through? I'm not sure. I feel like you guys probably know what it looks like to set up Drupal. This is actually real time. I did not speed this up any. This is just how fast I work. Yeah, this gives you an idea of what the... Yeah, so the, and the configuration that's happening here is a number of features being enabled and depending on what that onboarding process would be, it'd be like, okay, let's go in and activate the default ones first and then activate the optional ones that would be requested for that particular site. Yeah, so I make sure that our custom features aren't in override before I start turning things on. But we've got webpacked core and then we've got webpack optional. So those are things like FAQs or faculty directories that people might or might not use on their site. Okay, communication and user management. So this really takes two, two... Sorry, I couldn't think of the word. Training, so we do monthly trainings with people to, in terms of basic training to get people onboarding, onboarded and ready to work on a website. If they need to do something more elaborate, then that's sort of core level. We usually do customized training so they come into my office, I sit down with them and go through it. The other side is help desk. So of course they call me whenever something is broken. I get two calls. This thing is broken and how do I do this? Most of the time I can triage those things. There is a very small amount of those where I'm like, oh yeah, I don't know what that... And then I call that guy and he fixes it. And I'm like, oh, thank God, it's not my problem. Collaboration, so when it comes to maintenance updates, it's sort of a three-way collaboration between Aiden, myself and the end users. So I let the end users know when the maintenance testing is being done and so they, and I ask them to go in and actually do the testing on the sites. So they are the content owners. They know their content the best. So we actually ask them to participate in the maintenance updates. And because there's a big enough volume, that actually helps us really get to the bottom of problems most of the time, we find them pretty fast. In the first five test sites, it'll tell us. Aiden was nice enough to schedule a maintenance update this week. So I have to keep watch on my phone to find out if my users have a problem. It's like we scheduled the maintenance windows before the Drupal North event, so. Yeah, yeah, yeah, yeah. The other thing that we do is hot fixes. So obviously, I think you guys probably know about that. So if a security update rolls out, it is critical to our systems that's patched within 48 hours. Faster. 24, yeah. And then much less frequently, we do new feature work. And especially because we're heading towards a migration to Drupal 8, we really would push back on any kind of a feature update at this point. Perfect, so now I'm gonna speak a little bit about the hosting infrastructure that helps make this all possible. And you need a couple things to make this like to manage more than one website. To keep yourself not up late at night. So the first thing is version control for code deployments. So in the ancient days, we would push websites up via FTP and Drupal has 70,000 files or something like that. So if there was a connection error on an upload, a random file amongst your 70,000 might be a problem. And so version control based deployments, basically what they do is they set an exact known snapshot of your website to somewhere. And either if there's a problem with your connection, none of the files go. It's sort of an all or nothing commit, which is very useful. The next piece of the puzzle that you have to have is a replicatable live and test environment. So some way so that you can take, you are 100% certain that the environments are identical from a hosting infrastructure point of view and that you can identically copy the site's files and content between those environments. The third component that we need is automated testing tools because we've got enough websites in place here that a human being really can't be trusted to manage all of those tasks to do quality assurance on them. And then the final thing is we need to include command line tools or maybe not command line. Some set of automated tools where just imagine each site launch to deploy a security update is like a 10 step process. Well, if you have a checklist of a person doing those 10 steps, and then you have to do it to 30 websites, the odds of someone making a small error on 300 tasks are pretty high. And so scripting and automation is a necessary part of the puzzle. And so we use Pantheon provides three out of the four of these things on the environment. So that's why initially we had launched on the U of T servers and you kind of have to do all this yourself and implement it all yourself. So for us, it just made sense to use that infrastructure. And what Pantheon sort of gives you by default is sort of three identical environments per website, dev test and live. And so that's very, that's part of that puzzle where we can have the ability to easily replicate our live files to either of these other environments and work with them. And so, and then from the infrastructure point of view, we would have say one account with Pantheon for the medicine website and another account for the comparative medicine, each of which would have three environments per account. And then we just imagine 28 more of them. So all of these accounts have these multiple different environments. And what we also have enabled with that infrastructure is what's called an upstream distribution. So that is the webpack distro, which you basically configure in the Pantheon environment. And if we push a change to the webpack distro and say we push an updated suite of security updates to the modules, the trip modules or Drupal core on the webpack upstream distro, what it does is each individual website becomes aware. Hey, there's an upstream change. And then you can just from inside of that infrastructure say, okay, let's try that out, that upstream change on just my test website. And so you can do that individually to each site. So our deployment plans look a little bit like this. We basically have the live website and we duplicate it out to an identical copy on the test server. And then we apply our upstream websites to that copy on the test server, trigger a number of automated tests to make sure that nothing has gone wrong with those updates. And what those automated tests are actually doing is it basically takes screenshots of a bunch of core pages on the live server. And then it takes a bunch of screenshots on the identical pages on the test server and runs an algorithm against them to see if there is any changes. And we would expect with a security update that the screenshots before and after should be exactly the same. Nothing should look different on the website once we just upgrade a Drupal security update. And so we run those test suites and the vast majority of the time it's fine and nothing exciting occurs. And then, so now we've got the updated test site. We're confident that nothing has happened to it. We get in touch with the actual site owner. So I reach out to Roberto and say, we're ready to go for them. We kind of do these things in batches. So batch one of the websites is ready to go. And she will reach out to the site owners and we will say, you have two days to take a look at your website or four days or whatever it is. And it's important part of that workflow that no answer is approval. So we have implicit approval and you can stop the deployment if you find a problem. But if we don't hear from you, that is you saying yes, it's fine to deploy. Then we get to the last step where we apply the updates from the Webpack upstream but we do not apply it, we apply it directly to the live website. And what we're not doing is taking that updated version of the test site and pushing it live because this process could have taken many days. So people are continuing to work on content on their live website. So we would have to say, no, no, no, you're not allowed to publish any new content on your website for this entire window of time. And so that's really not a feasible option given the workflows of all the site owners just for a maintenance release. And so we push those site updates to the live environment and we have a high degree of confidence that those are gonna go smoothly because we just did this very thorough test on an identical copy of it that's a couple days old. So that's why we can trust that deployment on the live environment. So what the automated testing tool actually looks like, it's called Backstop.js. These are sort of the snapshots of the sites in the before and after mode at a number of different breakpoints. It tests at like different sizes because you're CSS, you know, CSS regressions and basically these pink blobs. So this isn't a webpack website, but it gives the idea the pink blobs are diffs where it's saying something has changed. So this test has failed and actually do this test, all I did was run the site a couple days apart. So they published some new news and that's what you're seeing as the change happening before and after. And it's got this cool scrubber so you can like see exactly what happened. I mean, if you accidentally break, if your web font account expires, you know, and then the font suddenly revert from to aerial or something, your website, you might not see that like because you know, if you haven't, if you're not looking at that website day in, day out. So that's an example of a kind of thing where like automated testing is better than people because it can be so much more precise than an individual site or an individual person looking for problems. That being said, we still do manual testing. So they're like, it's not, you can't just ignore that step entirely but you do the automated tester just more thorough for catching typical kinds of bugs that we'll find. So looking forward to Drupal eight and nine and what are some of the lessons that I think we learned working with the distribution over this period of time is we ended up with like six optional features. And I think that's too many. What we ended up having to do was test every possible combination of every optional feature, which is fair amount of testing across your different components. So I think when we look at this on the refresh for the Drupal eight site, we'll try and reduce the number of features, optional features available, and then maybe implement some approach where we can hide the visibility of the features under the in the interface rather than sort of disable them from a code point of view, which should make it easier to test. Another sort of lesson that we're thinking about or a lesson we learned that we didn't see coming really was that the unique site customization. So individuals who are saying my site must have this unique feature. We knew that we'd have to build some time to implement that feature, that just makes sense. But what we didn't realize at the time was that those unique sites actually will increase the cost of the maintenance of everything overall because we might need to set up additional like additional automated tests for them and make sure that we run extra, that whole basically extra set of tests on top of what's already run there. So there's just something to consider with those each unique site, it increases your maintenance as well. So as much as possible in the future, we will say no, we will not, you know, instead of if that unique feature it could actually be turned into maybe a configuration option instead of a unique feature that would be better. And then the final lesson is really tied to accessibility where accessibility that we made sure we audit everything and it looks good when we launched it, but that is really not a one-time task. That is something you have to take care of forever. You have to make sure your content editors are aware of it. You need to pay attention and audit what's on the sites and have an effective communications channel so that we can speak with our audience and communicate what they need to do so they understand how to fix the accessibility problems in their own site. And that's it. Thank you very much, everyone. Does anyone have any questions? Sort of tangential, but I'm curious. How do you handle billing, budgeting for this kind of scenario, right? Like, you've got all these different clients, presumably Roberta, you're sort of the single-point contact for anything in this, folks, but you showed that you have separate hands-on accounts for each downstream client. Are you just, then is it just a pass-through or is there something more complicated in terms of how you, if I want a feature do I have to pay for it as my department or how does that all work? So there's two prices that go with Webpack. So the first is an initial set-up cost and we keep that very low. So it's a little bit over $3,000 to have your Webpack site set up. And that's meant to cover the cost of my time. Then they have the hosting cost. And the hosting cost is pretty much a pass-through from Pantheon. However, also with the hosting, they get the maintenance updates and the security updates and then support from me. So it's kind of, it's there, but it's not really accounted for. Really how we cover the maintenance cost is that it's just funded through the dean's office. And it just makes the most sense for us to do that because the brand improvement is enough to make a really big impact. So we just kind of bite the bullet on that and that's just the cost. So that wasn't a hard sell in terms of that branding improvement instead? Or was it? I mean, it was a lot of discussion, but our chief administrator was a really big fan of the project. So he made the space for it. So we've had good luck with it financially for sure. And sorry, you said new features. If somebody wants a new feature that not everybody else wants, they have to fund that feature themselves, but we don't allow for somebody to build a feature that is exclusive to their site. If they build it and it's implemented in Webpack, then everybody gets to have it. Because we can't have a site with one thing and not, like it has to be equal across the board. So that's our first in-page response to the development. Yeah, and if they can find other departments to split that cost with them, then we're all for it. Yeah, yeah. What are the criteria you have between choosing a Google and WordPress? Has he mentioned there's a bunch of Google? Yeah, our magazine site is actually, and our Dean's report site is hosted on WordPress. And that has a lot to do with not wanting to invest the same amount of budget in those properties. And those properties are much more simplistic. The complexities that we need for managing the faculty websites is just, Drupal makes more sense. For WordPress stuff, I can handle myself in-house. I can deal with the security updates and all that kind of stuff. So it's really managed by me and the cost is good. Wow. We don't use Pantheon for hosting, so the cost is really low. So, can you back? Do you keep the themes in our college style? We don't let them use anything other than the one theme that is available. So, yeah. And there are minor variations that have crept up because of, I think, more of the early customizations before the sensible rules of, if you make something, you have to pay for it for everyone. That was a great rule, but I don't think we started quite there. Yeah, I'm the only one that's figured out how to break it. There are, yeah, we have minor configuration and overrides per website. Secretly is something we can do, but definitely don't make that known as a general. She can say for that. Yes. Oh, the recording. The recording will go, shh. At that time. Because we sit in the communication's office and we guard the brand as well, when they come to me asking for, I just want a green box on the front of my homepage. My answer is, nope. Which makes me not so popular sometimes. Yes? Is there like a set of a guideline or like a guide for whatever someone wants to, if you're like close to a website for medicine, like those are the things that you should do, those are the things that you're not allowed to do, is you want to, if you want to add an event or use extra to your website, it's going to cost you more or something like that, or do you just give them everything and then they can. Yeah, like it. Go add it. They can use every feature that we have for no extra cost, it's a flat rate. It's only if there's a new feature that they want to add that in that case. Yeah. Is there like a limitation on what feature they want to add if they ask for a feature and they're ready to pay for it, you're all good to give them that feature? So, I mean, whatever feature development we do is going to impact over 30 sites. So, at a very basic level, I have to evaluate how that is going to impact the other three sites. If it is a feature that is going to be of use to many, like even a quarter of the sites, that makes a lot of sense. If it's going to be a feature that is going to make life more difficult for everybody else, then it's a no. And it doesn't matter if they want to pay for it or what, like it's just no. I mean, that's, that is the shackles and the benefit of this system. With such a lockdown way of doing things, which is needed, do you have either, you know, site owners department that go roll and just go do their own thing or try to do? Yeah, I, it's funnily enough, I had a department that went out and built almost an exact replica of our template because they wanted these color blocks on the homepage and we couldn't do that. So, like they, it's, the font is just wrong. But. So, did they come to you first and said no and then they went and. They hired an external vendor. Can you stop? They keep, oh. If I could, I would. U of T is very siloed. And up to now, the branding police have been very laid back. So, you know, we have a new AVP of branding so I expect that that's something that's going to see a bit of a turnaround. But on the other hand, it's not exactly the problem. But if they go and do that and then it breaks, you're like, well, I guess you're going to have to fix your own problem now and I don't have to fix it. It's always my problem. No. Because, like, I'm always the one they come to and they're like, my site is loading, it's broken, it's been hacked and I'm like, please fix that. And I'm like, dudes. Do security updates. What's a security update? Yeah, we've had websites that have gone out on their own and have ended up with real disasters. And then they've come over and they've gotten their webpack site and they live happily ever after. And some units go out and they build a completely separate site and they're totally fine and everything's good. You must be approached by other departments outside of the faculty of medicine who are also interested in using, is this model available across a bigger number of departments or have you thought about how to make that happen? So we feel that the template is part of medicine's branding. And we have found that other schools and other departments have built websites that look very similar to ours. So we feel like they've been inspired by us. Which is nice, but we've never, we don't want it to go past our faculty because we feel this is part of the brand identity. I am, when I presented in Toronto last year, there were a lot of other departments because it's so siloed, who were looking for help and support. And a lot of them seem to be in dire need of a tool like this. And I expected it was something that could be made available to them if it would be something that they would embrace very unhappily. Arts and Science is working on a similar project to webpack except for theirs is on a multi-site instead of a custom district. Okay, well thank you very much everyone.