 Welcome to our presentation. Today we'll be talking about the project that Evolvenwaub has been working on more than others. It's kind of a big project for the summer and it's almost done. We're quite proud of it and so we'd like to share with you what we learned in how the client here is the Princeton University Press. Don't be confused, it's sort of a nonprofit publishing house that's associated closely with Princeton University but it's not the same entity. It's on campus though and it's got a very famous tradition. If you have any academic books on your bookshelf, some of them might be from there. So about us, my name is Alex Durgachev. I co-founded Evolvenwaub 10 years ago and I'm Durgachev on Twitter and GitHub and I do lots of community stuff as well. I'd love to go all these camps. I know a lot of places here. My name is Matt Corks. I've been with Evolvenwaub for almost exactly a year now. I've been using Drupal since, as I usually say, since Pluto was a planet. My username on IRC and various other places is MVC. Our company Evolvenwaub has been it's 10 years old and we've been doing Drupal for about nine of those years. We've done projects big and small. We've got a great training program. We're based out of Old Montreal in Quebec and as usual we're always looking for good people to work with so if this is the kind of project that you like please you know reach out to us and start the conversation. Over the last years you know we've done we've done work for for some cool companies. Some ones we'd like to feature our for Linux Foundation, the press for Quebec originale which is the tourism Quebec website and lots of lots of others. So in terms of in terms of our approach you know we often we're a small team so we often work when we work with a larger company. We're very collaborative. We use their resources as much as possible and we try to teach them Drupal best practices. We have some hybrid estimation and agile approach so we're not just you know PHP developers but you know we most of us are very comfortable with Linux and lower level stuff and also with JavaScript so it's we try to have good expertise throughout the whole stack and you know we're open source guys so we actually we use JIRA when our clients demanded and use Redmine internally for pretty much everything and base camp for client communication slack and so on. In terms of what kind of stuff we do of course great design teaming but also infrastructure, multilingual, search, lots of migrations and lots of custom dev. So that's that's our company and our client is depressed. It's like I mentioned it's an independent publisher and you know they've been around for over a hundred years. They have published many tens of thousands of books and you know like 10,000 books in print and they're one of those prestigious academic publishing houses so you probably heard of also Oxford University Press and so on. So the current site as befitting a very prestigious institution with a strong legacy is a legacy site that's like 20 years old. It's believe it or not it's static HTML it's I think it was made sometime in the 90s or you know 15 years ago and and then been refreshed where they changed the header and the photo to insert the new logo but aside from that it's been a site that's been evolving so it has a lot of content on it it's got 9,000 published works it's got lots of Einstein related papers it has a lot of like sample chapters, table of contents, ancillary files, some blogs. So this is the site it's kind of like a static site with a lot of stuff on it. It does have some moving parts so it has some mini sites that we realized after the fact it exists. It's got a WordPress blog bolted on it's got a PHP list newsletter which they probably should move away from eventually. It has a like a little legacy Perl's TGI search application and it uses Google search quite a bit. They have the ability to buy books and believe it or not they have not have one but two built-in shopping carts one for the US and North American market that kind of the Americas and then one basically out of the UK from Wiley their partner and distributor and now one kind of handles Europe and Asia and Africa and so they have as you see two shopping carts that you can add stuff to. All of this stuff all of these books get generated a static HTML and thrown in an FTP server. In addition to this they generate an Onyx format XFL feed if you're in the publishing world you'll be familiar with Onyx as the metadata format for the public publishing companies use and that also gets thrown to other FTP servers that their distributors have that Amazon.com has and that's how Amazon has all the updated data and all the images. And yeah they have lots of audios and videos scattered and embedded throughout the content and their planning to move and standardize on Bimeo going forward. For this project what we did is we did a big analysis of what what they had which was a SQL server database and a lot of VB scripts that read this database and from this metadata that they extract all the relevant bits and pieces from the site and they throw it onto the static HTML which gets manually copied to an FTP server. We also did a content audit. We kind of are taking it back to find out how much other static content they had beyond what we were initially thought you know came from this VB script because they had 20 years worth of static HTML so stuff really crafts up. So we did that. We had then moved everything to Drupal. The mandate was initially described to us as a lift and shift and then we initially described to the client that there's no such thing as lift and shift but nonetheless the idea is to move to Drupal their existing site without changing too much. In part there's a secondary mandate that the organization is expecting to go forward with the big redesign where they're going to say what is their brand identity on the web in the 21st century and how should that be reflected on the site. So the idea was to take what you guys saw and move it onto Drupal. However they did recognize that in 2017 you need to be responsive. It's just crazy not to be. So that gave us an in to make some design changes you know just basically fix things and make it responsive was the instruction we got. The other thing is these DB scripts were quite heavy and not robust put it lightly. So we were asked to write that synchronization process that would get all their data from this from the central system and get it magically inside of Drupal synchronized every night. And after some consideration you know the team that they have there they don't have unlike other like other clients we've had or unlike even Princeton University when you have your own system in team the publishing house doesn't have a huge IT department web hosting is not really their specialty. So we cancel the conclusion pretty fast that they should be put on a host like Pantheon or Acrea or something like that that gives you managed Drupal hosting which has the benefit that they don't have to worry too much about it and at the same time evolving web doesn't really have a good lock in on this. This is as a small company our goal is to get clients locked in. So we were happy to put them in a good place. So this was the mandate. We had a couple of months to do it. So first things you know it sounds pretty simple right it's a static HTML website. How hard could it be? It's got no features just click around and look at books. The shopping carts by the way are all external like they're basically just like JavaScript integration. So we start the project. We had an idea what it's going to take but then we start. So we start looking at the database of this SQL server database and we were trying to figure out how we're going to integrate that with Drupal. And first of all we figured out that there's like 30 tables in there. Some a couple of the tables such as the works like the books table have like I don't know 50 fields of which maybe a third of them are being used in the website and the rest are used in other parts of their organization. So we have to do a big audit of all the tables of all the fields. What are they having them? We had some sample sample content from them so we know how to display it. We if they're bold you can't quite see it in the slides but if they're bold then they're used in Drupal. If they're not we can just ignore them. If they're bold you're being used currently in the existing website. Yeah. And the other fields are just used for other internal purposes but it's not the website. So we assume it wasn't used for the old website. We don't need it on anyone. The next thing so that started with the database and what's there. That's that's our source data. The next thing is they sent us an MSS access database that connected to the remote SQL server database and this access file contained a bunch of VB scripts embedded in it and it sounds not so bad right but it turns out there was like 30 to 50 scripts many were duplicate copied and pasted versions of each other because that was basically their whole code base. It's a static site generator embedded in these VB scripts embedded inside of this accessing with no version control or anything like that. So over over 15 years and of course the person who initially wrote it was retired. So over 15 years lots of versions of these started being around and we just have to go through everything and figure out what was there. It was really nice that the webmaster and the web director of the client were quite helpful in that regard. They rescued our butt because I don't think we could have figured it all out. We're not we do know VB script a little bit from our back back in the days but we're not the experts and it's pretty legacy stuff. Another little kink with this VB workflow that they had is that about right now it's only about 50 books but before it used to be more we're marked as special. So the HTML that was generated for those wasn't put with the same folder as the rest of the HTML generated. It was put into the specials folder and then a webmaster had to go in and manually merge whatever fixes or changes that had been done on production. So they were using dream reverend stuff for that. So a lot of it was because they didn't know how to fix the VB script to do what they wanted because it was just too complex. So we're happy to say that we're involved with a special workflow. Yeah, the MS SQL DB was called Ants Bibliot not to be confused with and was the web director not to be confused with the new system that we'll be moving to next year also called VirtuaSales Bibliot. They have this web drive. Let's see was an amount of drive that was accessible to all the other workstations inside as W. That's the web drive. And so that's what they're using as FTP. And like I mentioned the scripts were each individual script was between, you know, 50, 500, 1000 lines of code, but there was there was 30 to 50. So there's a lot of lines of code in total. We estimate that that was five to 10 real lines of code and the rest were all duplicates. So we did this audit thanks with the help from the folks at the press. So we had a giant spreadsheet. We knew that there was way too many scripts for anyone. There's literally nobody around who knew what these things are doing. So what we focused on instead is for every script that we could find, what files did they generate? And so at least we had a had an index of all the files and how they mapped. And basically, we had our list of templates, you know, the things that Drupal would be responsible for showing. And then we just open those things up like the HTML, and we said, where are these fields? What do they come from the database? Is it obvious? If it's obvious, we can have to look at the script. But if something if several fields were concatenated, or if there's conditional logic in this book show this field, not book show that field, we would have to go in into the vb script and check what the heck it was doing. To be honest, even now a few months later, I'm sure we only got like 80 90% of the complexity of those scripts. I'm sure as the site gets audited and tested some more, we will find little corner cases that somebody five years ago put in. Maybe they don't care about them. Maybe they do. We don't know. Nobody knew. We did it. And a lot of very custom databases that people used to get this drop. They were very responsible for reducing complexity, things that they didn't really need and would have taken us three days to extra work. It might have been a pet feature from seven years ago that nobody even remembered, but it was there for us to discover and figure out how to do it. So the existing workflow before we came along, was that we have the SQL Server database, and they have these vb scripts generating dynamic HTML pages. There's a bunch of file assets that's on the right. And there's also a bunch of static HTML, not just the special pages like we talked about, but there's just a bunch of other HTML because you have an FTP drive, you have a webmaster or three, you have 15 years, you're going to have a lot of HTML. So we try to change it up a bit so that we have Biblio, this database server, a SQL server, we export using just a very simple PowerShell script. We export all of the tables into CSVs. And we have that on an FTP that we can access. Then we have a group of migration that actually runs on Cron every night that goes in, checks to see if there's new versions of those CSVs, pulls that in, checks to see which rows in those CSVs have changed. And for any new rows, it makes records for any updated rows, it updates the records. And for any rows that were deleted, it removes the records. So we did quite a bit of work to write these migrations. We kind of scratched our head quite a bit about what to do with a giant collection of accumulated static files, images, PDFs, videos, source code. Don't ask me why some book had a source code, but it had source code. Did we decide to move it all into Drupal? No. We said that's just a recipe for madness. So we decided that the only thing we are going to move into Drupal are cover images because we have to have resizing and all this beautiful thing and meta tags with images. The rest, we told them your existing web thing can be its own FTP server. It's going to be on a different domain like assets.press.princeton.edu. And we're going to just update the links. So we shouldn't have any broken links. But we're not going to take all this craft and move it all into Drupal wholesale. And it's up to their existing web team to clean that up if they want or not, or they can manually bring it into Drupal. Yeah. Well, we also are aware that with Drupal, there's a lot more capacity to build dynamic pages because we have the power of views. So some of the things that were dynamic because a VB script generated them, like a listing of recent books, if they bothered to make a VB script for it, then we made a view for it. That was about half of the listings of books. If that listing of books was manually edited, we realized that we just didn't have the time to go back and forth and say, look, let's make it dynamically generated. Even if it's really easy, it won't be the same as their listing that was manually generated. And then you'd have to go to them and say, is the new one better or worse? Are you happy? Do we need to make exceptions? We didn't do it in this case, but let's follow up. This is what we wrote. For every single table, we wrote a query that just pulled out only the fields that are being used and generated these CSVs. So it's a very simple export feature. We did this because we knew that in the future, they're going to shift away from the SQL server database. They already said they're going to use virtue sales video. And we know that virtue sales video can make reports out of the data that's stored in there. As XML as CSV doesn't matter. So we didn't want to do anything complex also because this part is running by their assistant in a cron job on a server we don't control. So if there's any bugs or errors, there's kind of if you always play broken telephone trying to figure out what's causing it. So we had to make these export scripts as minimal and simple as possible. That's why we don't do any cleanup, any joining, anything. His job is to take your source data and generate these CSVs straightforward. Now we've got to move that into Drupal. I don't expect you to digest this table. But for static HTML site, you know, you think, oh, you know, with just a single content type basically books. Turns out, no, there's a little bit of complexity after all. So the complexity here is that the work content type has a lot of field collections in Drupal 7 terms, or in this case we use paragraphs, but however you want to think about it. These are entities that don't have their own URL, but they are aggregations of fields that belong in groups to each work. So we have the works, we have the author, which have many, one to many, or many to many relationship to works, and then we have a bunch of paragraphs related to reviews, links, illustrations, and so on, and then a bunch of fields related to these paragraphs. And of course, taxonomy So we then had to, we got access to their FTP server, and we realized that there's 77,000 HTML pages there. We're not talking about PDFs here, we're talking about just HTML pages. And we kind of had a small heart attack. This was after we already gave an estimate and then scheduled and all that for static site, you know, how hard could it be. But after we started digging, we realized that 20,000 from that audit, right, from the VBScript audit, we realized 20,000 are being generated by the VBScripts and are going to be handled by Drupal templates and views, fine. 54,000 are actually old versions of generated pages because they wanted to make a backup and threw like that back or something and they left it in place, because you don't just accumulate, it's not a dynamic, it's a static website. For all those people who love static site generators, keep this in mind. And then turns out that the actual number of HTML pages that had to be manually audited and considered was really only 2,500. Still quite a lot, but a human achievable task, since it's not something you can't do with, you know, a couple of weeks of work. In order to triage all these crazy folders, which of them were garbage, which of them were ancillary files, which contain HTML, which didn't, we made a big listing of every folder that has at least one HTML file, because if not Drupal won't touch it, it just goes on the asset server. So everything that does a HTML file, we collaboratively went through, did our passives trying to figure out what's there, then their web team confirmed, denied, gave us comments, input, and based on this we were able to cut it down to a much, much smaller list of essential pages, static pages to bring it. I also did a, what I think is a clever thing, where I have my own little FTP like static server with the backup of the site that enables directory listings after you password protect, and so I have a link to that. So if you want to see what HTML files are any folder, you can click right from the from the Google spreadsheet. So this was the audit. It was a lot of work. It scared us a lot, but fortunately, you know, we're good at this kind of stuff, so we fly on into the implementation. And we built the site. We got a, we got our designer to, we gave an instruction, make the site look as similar as possible while recognizing the fact that you're going to make it bootstrap based. You're not going to try to copy a design from, from the nineties, right? You're just going to, you're going to make it a modern design that works easily, but pretty much has the same elements. We got very limited mandate to change things or to make decisions because the creative team was not part of this. This is supposed to just be a modern version of the same design. So we still have the same slideshow. We still have just the feature publications on the home page. And yeah, the one thing that we did is the two search boxes that they had, we made collapsible, because we know javascript, so we need to click that little search box then, then the two things pop up. It's because it's Drupal, because it's bootstrap, because it's clean, it's got much better user experience, it's got much better SEO. This is not this spaghetti code vb system anymore, but now it's, it's a very clean minimalistic Drupal site with admittedly a very complex migration bolted on. But yeah, so they can, their web team can now just go in and update the content with WYSIWYG. Images are auto resized, users are much, much happier. So for the home page specifically, like I said, we stuck to the existing look and feel whenever there's any doubt we're not trying to change anything, we're just trying to take the cheapest possible way and make it look that broken and make it responsive. We did ask them for permission to change the menu. If you guys remember they had a 15 or 20 item left sidebar on every page and that actually would once in a while change, because when you get to a different section they would manually edit it, but for the most part it was just a static left sidebar and we asked them to make it into a drop-down menu. I'll just mention that that left sidebar was a usability nightmare and it wasn't consistent on other pages. It was way too long. We should have been in multiple levels. So there was like, and we'll show you the site soon enough, but there were pages you could only discover by following a trail of three to get some interior pages. That was what you see on the landing page and then certain sections of those would randomly change the menu of the left in a very unpredictable way. It took us a while to figure out the initial structure in order to be able to present a structure that made a lot more sense. Yeah, so we asked them to come up with a more limited set of items and we asked them to categorize it according to this drop-down categories which they were happy to do. It seemed to be uncontroversial, easier than we thought. And so that was a small step but I think a huge usability way. Yeah, we made a list of everything that was in the left follow menu on at least one page, put it in order by rank by according to Google Analytics data so you knew what was the most important. The things in the categories from there decided that they should be the first level and second level and there's only a few third level things. But the menu is actually the same on every page which is a pretty basic idea but not something on their legacy side. Great, so for the book templates themselves, like I said, there's about 10,000 of those so that was our main beast of a template. We moved things out of the, we have Yogan though, sorry, but we moved things out of the header. They just crammed pretty much every single piece of metadata that was in the description into that top header part which made it look really confusing. So we made it more consistent with things that you'd see in Amazon or something and yeah, we definitely are aware that different books have different fields and so we made a design that works whether or not they're there. We still have the same integration with the carts and Google Booksearch and videos and so on. Yeah, has anyone ever been in a design meeting where you attend a different department and you said one says my little thing has to be above the fold because if that means something on the website. So yeah, 10 years of that. Yeah. There's like 10 years of those decisions and we just shoved everything in. So everything became above the fold. So yeah, everything was with the priority which means nothing at the priority. Right, so then the actual migration of the book data, so like I mentioned, there was, you know, something north of 8,000 books, there was 4,000 covers. I think there was more than that, but there's a star. We, yeah, we made sure that all our migrations are incremental because it's a huge, huge migration and we wanted to be fast and only a few things change like they do, because obviously for years we made 20, 50, 100 books a new one. Yeah, we make sure that the migrations are running nightly and can be run quickly if they have urgent change and we handle removal out of print books. To give you a sense, I don't know if you can see the output of Drush Migrate status, but we have like 20 migrations there for these different entities, these paragraphs, these taxonomies, these images, and as you see some of these have like, the reviews table has 42,000 records. The eBooks edition table has a couple of thousand, the author's table has 10,000. Yeah, and these are migrating to different entities, so that includes redirects, which are entities, that includes paragraphs and then the very vast number of runs is the node which actually represents one book. So before that runs all of the different paragraphs and one of those is the editions, so that can be like the hardcover, paperback, DVD version, all of that's been migrated first and then the work node comes and puts it all together referencing the, giving the node ID from the previously run, sorry, paragraph I mean from the previously run migrations. Yeah, so you guys wrap your head around this migrating, you guys know Drupal Migrate, yeah everybody uses that framework, it's, it's came out as the canonical way to import data into Drupal and it served us really well. However we did have to write custom code for several things related to the migration, such as source deletion, that doesn't quite work out of the box. We also had the notion that a lot of Drupal entities, which would correspond to one of these migrations, need not one, but several CSVs that have the fields in it, because you have multibalue data associated to it, so our developers actually created like a plugin for multi CSV joining, as if the CSV data source is an almost SQL like where you can specify. Open these several CSVs and join them on this criteria, so obviously if you wanted to keep it more simple you could have just pulled them into a SQL database and, and just did that. In fact I don't know why we didn't. We have, we have now an impressive thing to contribute back. There would have been more custom codes right that way. Yeah, it definitely would have added moving parts for sure, because once you change those CSVs there have been more things to maintain. This, this pros and cons. The more moving parts the more points you get. Another thing that Migrate out of the box doesn't do is support for getting these source files of the source CSVs from FTP or SFTP. Kind of wackled back and forth what they wanted to provide it, so we were able to easily provide support both of them and this also manages to, for SFTP it actually connects using SFTP and does an md5 and sees if the file changed except timestamped. So try to be smart about detecting changes. Another gotcha was running all of this in cron, so Migrate out of the box did not run on cron, so we took a few iterations of getting it right and we thought we had it right. It was fine, we just ran it from our cron job and then we moved to Pantheon where the cron timed out quite aggressively and all of a sudden what we did wasn't fine, so we had to rewrite it to use Q workers. And then we think it's fine, but that was just this week, so it'll let you know. So stay tuned. There was of course encoding issues as you would expect from any Microsoft SQL server original data that we worked through and there's just a lot of fields, a lot of interrelations and a lot of legacy complexity that nobody could quite explain, but we muddled through it eventually. One thing that I'm kind of proud of is how few contribute modules we have. Remember this is mostly a static HTML website, no one's expecting bells and whistles. So what you guys see here are mostly migrate stuff and API helper modules, some basic site building stuff like context and active trail, a little bit of the empty browser for images and I thought so. Really this is not a site where we enabled a bunch of contribute modules and called it done. On the other hand, up until the last you know last two weeks I think we only had one custom module, the PEP migrate. So we also had to write no custom code except the migration. Eventually I think Jigar in the back had to had to write a views argument handler for some some advanced stuff. So we didn't change any URLs on the site. That was an auto scope. That was a scope that would require approval from too many people. So in order to have a view, to keep the views arguments looking the same, we had to write some fancy views argument handler. That was a bit of work, but I think there's one other custom module that didn't migrate. So we have a bunch of these lovely fields and then I'll just play correctly. We cleaned up a bunch of stuff of course. We had lots of PDFs linked all over the place and we had to make sure that they're being linked to that new assets sub-domain. We also in the source data, we had an ebooks table and a books table and they had like 90% of the same 50 fields so we kind of didn't we normalized it in Drupal. We had supported ebook they have apps and now we support that and we dealt with stuff. So that was that. We also took a couple hundred of the top pages according to Google Analytics the ones, everything that was linked in the menu anything that you really miss for being a soft launch and we imported it ourselves and we cleaned it up as much as time allowed. So we did that first pass the site doesn't look terrible so we moved off to their web content team to go through the long tail of everything else and fix it up. We did the search in Drupal views, we did all the listings all the taxonomy listings inside of views and some things like human print which was manually a list of books they can move into a view later. So the next section is going live I'll be glad to know. Sure. Yeah this is just our checklist of things that had to happen before the segment live. Right now it's existing on a beta domain we're expecting the actual hard launch to be in a couple of weeks. We already mentioned that we deployed on Pantheon which means we've been testing in that environment for two months now maybe month or two to work out any specific issues there. We created a second domain name that points to assets.press.princent.edu that resolves to exactly the same place as their current live domain that way so then we rewrote links to PDFs like their massive pile of legacy PDFs to use that domain name so it goes to the same place but after the launch that domain will keep working and point to the existing server so all of their links to photos and the zip files which are the source code for some computer science and whatever can still live on their old web server and they can migrate that over later. As Alex mentioned we had to spend a lot of time working on CRON timeouts as we were running migration nightly because we needed to do a nightly sync of their bibliographic data we ended up doing that by making a queue worker so that was interesting but that was the major difference between our dev environments where of course we all set the PHP timeout to be negative 1 and the memory limit to be negative 1 then you go to your production environment and hey, why doesn't the system want to do that? Oh, at least they were using Docker Yeah, yeah so we did have that control Yeah, we, I mean other things like these are just small like go live checklists that would be common to any site you made sure that Google Analytics was there we made sure to turn off DevL, we had to make sure that the live site's domain was whitelisted because through the latest of course as a concept of trusted domains we had to in and since the live site will be accessible by many domains like with and without the www by over port 80 or port 443 so with or without SSL we had to pick one canal called domain to make sure that everything redirects there for SEO purposes so that's something you need to do right when you go live we did a final performance audit so front end there are a variety of tools for that so the page loading time in the browser as well as backend that was with a tool called Blackfire which my colleagues Jigar and Alex gave a presentation on earlier today so I'll just assume you all saw that yeah I mean you turn on CSS aggregation at the end I mean the standard things Pantheon has their own launch checklist which is very helpful some of which is specific to that environment some of which is not and their own backup system so you just have to make sure that you click the button to say make me a daily backup we are doing monitoring with the service Pingdom so that'll hit the site once a minute and send us an email if it's down I mean you should do this for any live site with that or some equivalent service post launch so these are things that are yet to come so once the site goes live we will need to have a supporting agreement in place so the details of that are still being negotiated with this particular client but that's something that just like when you've got a car you always need the full number of a mechanic so we're the mechanics and just as you folks are future features will be fixing the right now the book URLs are literally title slash random five digit number dot html so which is amazing for SEO maybe that'll change there's other things like php lists for running their mailing lists they know they need to move off of that they could have better meta tags for discoverability on Facebook etc two really big things more automated views so the list of books that came out in the last week is written by hand currently in Dreamweaver and now it's going to be just here's a static page here's a basic page type in Drupal and I'm going to click at it and type in all the books that came out this week so maybe we can automate that since we have a database of all the books and their data publication I'll probably save somebody some time Virtue sales yeah Virtue sales is a very large software package that like this is a year minimum one year migration that they're in the process of doing that will manage all of their book inventory their sales will push data to Amazon and all their other distributors that will handle I think it'll even handle online sales I'm not sure about that but that'll be a massive change on their end and so we'll need to change the migration source to pull from that data source eventually but we did as much as possible to prepare for it so we did what we could when they are ready to launch that changes will be needed to the website and at some point they want to do a major design overhaul but they're not even planning to start that until like 2018 so we're going to have to be involved the website will need to be updated for that but they weren't ready so we did that with the current design as Alex was saying project management so here's the team there are a couple other people who did like a smaller number of hours but for the most part we had two people doing back-end development G-guards in the back of the room in the bright shirt and David was in here today we had one person doing front-end development so all of that so the parts of the views all the CSS we had an intern doing some QA so my boss Alex doing the business lead like negotiation contracts my other boss Suzanne was the lead on training for the content editors and the designer and then myself I was doing project management as well as code review because I come from a development background so we had several days of meetings on site seven calendar days, that's more person days than that several demos some in person, some over video calls we asked the client for a single point of contact and then I was the single point of contact on our end so if you don't have that, it's impossible I've had clients where there's like five different people you're supposed to talk to and you end up just playing referee that doesn't get you anywhere I'm sure you all have a horror story about that we had a backlog of tasks so we would sign contracts with them that included a long list of actually had our red mine ticket numbers we would provide them a list of things we needed to do that we thought needed to be done estimates in terms of hours and they would tell us, okay, do these in this scope these ones, these are nice guys I should jump into that that was for the last 40% of the time the first 60% of the time was us doing this out of figuring out what the hell is over there and then just saying, oh my god and then just saying, okay, let's see what we needed to do so the part I was looking at was not negotiable there's still a lot of different sites on the control list I've migrated that into Google I've told the data that we have that we kind of just did but once we got to the final point the little nice to have features such as deleted offerings they didn't need to have through the line they just happened to have through five years ago if you still need it, do you want a date for it or do you want to send your money somewhere else that's where we moved to this model yeah, and the client one thing the client was very good at was that also in the discovery phase we found the site was a little more complex than we thought and by the way, pulling in a very complex data set means that we had to use a very strange data structure inside of Drupal I mean, we would have come up with something that made a lot more sense and would be easier to build views around in Drupal if we had had the choice of the data model but since we had to retain their data model we had to do things in an awkward way in Drupal so it's doable, but it takes longer but other than that I would say there was no major scope creep from the client we had our list of nice to have and as we found complexity that we had to fix somewhere a bunch of those got crossed off and we were to have the time to deal with those other issues on the right is just a screenshot from a base camp which is what we were using to communicate with the client and we were using red mine internally this is a rough timeline maybe I'll think the next slide might be more interesting this is a rough division so the work happened over 5 months so here's roughly the percentage of who was doing which team was doing what work at what point coordination and code review is together one item because it was me doing it I'm not sure the exact rate down but otherwise it was about a third front end and back end the management reviews portion yeah do you want to do the summary? yeah sure so and then we'll show you the site yeah so basically you guys you guys saw we discovered things in scoping that were a little bit more complex than we initially thought we were kind of had our hands tied for the design but that helped because the design usually is the part where things go off the rails so that that didn't happen we just told our designer we've got 10 hours to make it look like the current site but modern responsive clean and whenever in doubt just make it look like the current site and he came back with something everybody liked we did a very complex migration with lots of fields and lots of records and a substantial amount of custom code to support the things that we talked about for the 9B sync there's a content clean up phase going on right now so while we're hanging out here at a fun little conference people are working hard and so we did the deployment we went with Pantheon just because it was quite economical they previously have been doing static hosting so they weren't quite ready to spend tens of thousands of dollars a year on an enterprise hosting account so Pantheon was a nice compromise between good level of service but not so much they have 500 or 600,000 paid views a month it's a really huge amount of traffic so we're going to see how that goes so we're doing a soft launch with that and for we'll be entering this sort of maintenance and I'm going building mode where in parallel maybe in advance of their big redesign and their migration to virtual sales we'll be adding these little nice to have features deep paradise during this phase or other things up there their stakeholders now that the site goes live now that the site has changed substantially for the first time in 10 years I'm absolutely sure there's going to be ton of stakeholders who come out of the woodwork and tell us what everything is wrong that weren't even part of the conversation so that's where the maintenance agreement is hopefully going to be comprehensive enough for we kind of caught off guard on this last minute checking migrations and caught on Pantheon with the client provided the credentials so we had to work on that quite last minute we were surprised by the complexity of their legacy codebase but I guess we shouldn't have been also the same thing with the content cleanup but fortunately the client's content team stepped up to handle the book of that and also we probably should have done the training for the editor experience a little bit sooner so we'd have more time to do that sorry that it was at the end we just did the bare minimum in terms of the usability for the actual content editors there's still a lot for them to fix manually or to update manually and the user interface is usable but Drupal can do better and we didn't put effort into that because we needed our time elsewhere and that's Drupal 8, it's not Drupal 7 so out of the box it's already 10 times as good so that's great and they're happy but we know that we could do better so we've given her a better experience than that so the timeline it did shift a little bit especially with having the self-launch especially with this content cleanup that's not what we quite expected but everyone's reasonably happy in terms of the vast majority of the development happened in May and June which for an organization like this happened 8 times as fast as anyone expected this has been in discussion for years before we came along so we kind of came in most of our team, we have a very small team we kind of came in and sliced and diced with machetes and got through all their complexity and really built a minimum viable product we gave them just enough input to say what's more important but not to add new crazy things and they were really good at not doing that so thanks to this really good collaboration we were able to achieve a lot at the same time we weren't able to achieve everything involving what we only do time and materials contracts so the client is perhaps justifiably a little bit miffed with us coming back and saying you have to choose between things guys this is complex stuff our schedule is what it is we can't add more weeks to the schedule without adding more budget we did a bit of a compromise we gave some more hours the client gave them some features and everyone is reasonably happy with the result this is kind of the project and just to give you the demo so let's show them the legacy site first so here is the site you guys saw a screenshot of it here's the lovely search two searches one is just the google books like full tech search the other you get to choose the title or the author or the ispn everyone works there uses the one on the left anyone else uses the one on the right two shopping carts us dollars and uk pounds and here is the site here's a book pretty simple one so the new site is on the page and so it just looks a little bit more modern it's like re-advertised there's a kind of shit in the show more you scroll it some of these are really long but it's not that long sometimes a book could have a hundred reviews or sometimes an author would be a list of all the awards that an author or a book have won and if somebody won a Nobel Prize he really wants that on that website I probably would do if it gave me a Nobel Prize but it gets lengthy and then you can't quickly scan the page and see the important contents I mean that's important information but it's also often pretty long so that's always collapsed by default for example great so with that I don't want to go into too much of the site it's pretty lift and shift and spare and it's supposed to be a great clean code base and a great clean platform for them to do their design overhaul so that's really what we achieved do you guys have any questions for us on this? Hi Why do you have so far two shopping cards? That's a business decision for them which this is just the way the publishing industry works often the rights to sell a book will be specific to geographical regions so for example Harry Potter the company that sells that I think it's Scholastic in the US Raimtree in Canada and I can't remember who in the UK that's just how the rights to the book were split up and a much better user experience would be to have a single shopping cart and then the order being fulfilled by the different things so that's on the backlog of nice tabs and subject to budgets and schedules but those two shopping carts there's literally a line of JavaScript each and the shopping cart is not managed by the site even though you see a number that's all external platforms and one of them is going to switch but the fact that there are two is not changing and that's just the nature of the book distribution business they don't have a separate website for different countries like Amazon they just have one so they need to have that listed there any other questions? I don't think we have time to do that and in fact I think Jigger in the back is the guy to pick his brain but I can tell you Chris that we get I know you know this we get that the Drupal module model is open source contributing back to the community so other people can fix your bugs for you in our issue tracker we've got a list of blog posts and core patches and things like that to give back some of this complexity to the Drupal community some are a little debugging modules that we think will be useful for people writing migrations another the doing a join between multiple CSVs that was our developer Dave who thought of that one that was a real good brainwave that saved us a lot of time so yeah this stuff will be showing up on drupal.org near you but it isn't all there yet but in the meantime Jigger at the back would be probably I'm going to speak for you Jigger would be happy to answer more specific questions about that if you have any after the talk any other questions? going once, going twice alright and before you go the most important slide and a slide that pays for us to come here and give this talk is my wife Suzanne is doing lots of upcoming trainings probably you have met her and went to her three talks and training yesterday so the most relevant one is going to be the Ottawa one September 11th 15th she's doing a whole week of training so if you have any interest in learning drupal 8th or if you have colleagues whether it's site building or virtual development she's got a week of training scheduled in Ottawa please send people our way and her training is pretty good better than our presentation she's practiced more so thanks very much