 So let's start. So today we're going to have a relatively informal discussion about how the Economist site, which I'm sure you all know, runs on Drupal, how the Economist site works. We're going to talk about things like how the teams work together, how the scrum process happens. We're going to talk about how DevOps works, we'll talk about performance, we'll talk about infrastructure. So what we're going to do is we're going to kind of go through the general architecture. We've kind of created a framework that describes the major aspects of the site and then different members of the team will talk about different pieces and then when we're done we'll take questions from most of you. We'll take questions from you about what you're particularly interested in and to say again that you know we had predominantly development team but we also but UX team is here and so feel free to ask questions about anything that you're interested in when we're done even if it wasn't a topic that we touched on that we touched on here. So I'm Diana Montelli in DuPuis. I am a developer at Four Kitchens and I worked on the Economist project as part of the Four Kitchens team which was also known as the Janus team which was also known as the Austin team and I think Austin team is where we're going to where we're going to stay and let me want to introduce yourselves or we want me to introduce you. You want to introduce yourself. Okay all right so this is John Jones and he's a developer and an overall harassment major from the London team and Ernest Berry who's from a developer on the New York team and this is Rebecca Moss who is the scrum master for both the Austin and the New York team. I have no idea how she manages how she manages that and Angelo Rapoli did I say Rapoli right? Angelo Rapoli who is the user interface developer. He is our front-end guru. I mean he's he's the yes the god of all things of all things and and Juna Anthony is a developer in New York and Dominic Laycock a developer in London and Caillou Sacks who is the one who keeps the entire infrastructure going single-handedly again I really have no idea how how how but he does or as Eric says we're just that good and this is Sharda Ram the developer from New York and as you saw Eric the CTO snuck in the back we weren't sure he was going to be able to make it and Anna can you say your last name for me? I would have butchered that and Owens I can say right Brian right Brian Owens and from London and New York team yeah so we'll when I'll explain how the teams work a little bit more let me just give you a brief introduction to the Economist online so the Economist was first published in September September must be almost end of the day in September 1843 to take part in a severe contest between intelligence which presses forward and an unworthy timid ignorance obstructing our progress which I would say we need more of in the US in May 2011 the Economist online had 6,421,376 unique visitors and 32,565,298 page views worldwide which even though I'd been working on the site for over a year those numbers still seemed cute I really had no idea and then this does not include the iPhone and iPad app download which 2 million plus but that's an older number what what do you guys think it is now and is it it's a lot more than that now and probably not twice maybe three and and I sorry I'm Android I'm an iPhone exclusivist so the way that development works is there's there's an Economist team in London mostly represented here and there's Economist team in New York and then four kitchens also has a team the Austin team and all three teams somewhat coordinate and somewhat work you know on their own to make the development team also recently was added what we've called the Continental team the Continental team who've come in to work on a new feature that's going to get built and so they've been they've been deep in the code review process for the for the last month or so and they're still saying so that's good and we keep in touch by Skype by IRC there's a Google groups that we call the dev list that has all kinds of has two hours worth of emails to read every day and also all the code review part goes to that we also actually speak to each other but that's less common and then in there'll be the develop you know the the literal developers on the team but also like for the Austin team Danny in New York is the designer and he will be the one that gives to us exactly what the look and feel of any functionality that we're adding or in the case of newsletter integration there's someone for production that comes into the team so other non strictly developer people will come in and out of the teams depending on the kind of work that the team that the team is focused on is there any other role that usually gets mixed in designer UX production it's about right occasionally editorial people okay and then okay and so Rebecca must she's going to talk about about the agile scrum process sure at Economist online we use scrum with which if you're not familiar with that is an agile project development framework it does come from the rugby term and basically the analogy is that we're moving the ball forward together as a team and essentially we work in two week iterations at the beginning of every two weeks the team gets together with with the product owner which I'll talk about in a little bit to look at what are the top priorities in the backlog list of features that we want to accomplish within a project and then the team meets daily for 15 minutes for the team to coordinate with each other for what they're going to do for that day and also very important in that is to raise anything that is slowing down or blocking the team that can be hey nobody's code reviewed my stuff I need somebody from one of the other teams look at it or it might be my development environment is busted and so we look at problem-solving either within that that that daily call or we have an action list of things to follow up that day to keep things moving forward so that's one of the very important things to moving forward at the end of every two weeks the team does a public demonstration which the other people in our group not just the developers but but editorial production marketing anyone's invited to come watch where the team demonstrates completely working software that is ready to release and that is the big piece of what we're trying to do with with iterations is that we're getting high value work chunking it into very small vertical slices with all the pieces the back in the front end and database all pieces so that at the end of two weeks we have something that we can say this is done this is ready to release and the roles within the team we have as Diana was saying we have a cross functional team so it's not just the developers we have UX designers and occasionally some of their roles come in another important piece of this is that the QA is done all within the team it's not that the developers just do coding and that's it throw it over the wall to to a separate testing department or someone who's a separate who's does the testing the team themselves and I think somebody else is gonna be talking more about our automated testing we think about that from the beginning and and it's the team's responsibility not to deliver code but to deliver working product so that means that that is tested just enough documentation and and that there's nothing else that would hold it up from actually going through the release process to live the second role within our team is a product owner and this is one of the key features of of agile and scrum which is that you have someone who is a representative from the business who is working with developers day to day on to make sure that what's being developed really does match with what the business needs are so so that product owner is attending the daily scrums the team is showing things work in progress so they can quickly get feedback and ultimately at the end of the sprint the product owner is is the decider is the one who says okay this looks good I accept this and and and and it's done so so that's a very key piece that it's not development working in isolation from the business but you have someone who's the voice of the business and the voice of the customer who is working with them every day and actually that's one of the challenges a lot organizations have when they try to move to agile is that's that's a big commitment to to have to have that kind of integration with the business and then the third role is the scrum master which is which is my job and the scrum masters role is it's a little bit different than a project manager it's more a combination between a coach and a referee I facilitate all of the planning's and the dailies to help keep things moving along but then as I said in the daily one of the important things is the team raises any issues or things that are blocking them and and the scrum master's job is to remove impediments so that's a big part of what I do on a day-to-day basis is going to the daily meeting hearing what's what's working and not working with the team that day and identifying what are things that are keeping us from meeting our goals and then I take that and and try to move things forward get the right people talking to each other and so that we can continue on with our work and then just one length of one last bullet about business value there's there's a slight misconception sometimes that agile agile processes are about getting things done faster and where that might be a side effect if you have everything going smoothly really the focus is on delivering the highest value of features sooner so if you compare this to waterfall where you might spend a couple of months in discovery a couple of months in writing up a whole big technical spec doc and then giving it to giving it to a development team who works on it for three six nine months before it's it's released out to live in this we're we're working with a product owner who has who because they're from the the business side they have a good conception of what are things that are gonna be valuable what of this big mass of features of different maybe different stakeholders are asking for prioritize it and and we look for how we can chunk it into smaller bits that we can completely release and get out and not wait six months for the whole thing to be released at once so that's really an overview of what we do so yeah I think that's great and we'll come back to questions at the end so which I'm sure they'll that's always something people have lots of questions with so questions about so that that works right with can you will you be able to hold your questions do you think or do you want to stop in between each piece for a few minutes all those who can wait till the end raise your hand okay wait till the end okay so so DevOps and I would say that in terms of development operations and also in terms of infrastructure that the Economist Project has for the most part now set the standard for the way big Drupal websites operate you know and it's being repeated now again and again on on other large sites as they as they adopt Drupal so that's that's gratifying in terms of that the work has was successful and has been tested and also means that to a great extent the team got to figure out all of the problems involved in the system to get the system the system working so I was thinking do you want me to put do you want me to show all of the bullets and you can go one and rather than cue me when it's done okay okay all right so those are a few of the things that we're going to talk about and Rebecca's gone through how things work from a scrum point of view and this is going to be slightly how things work for a developer what's it like for the development team that works at the Economist and we'll start off by saying that we use bizarre as a source control system how many people in here use bizarre bizarre kind of a minority people the Economist people anyone using it yeah lots of people using it good and SVN sorry yeah so we used to use SVN and we decided to move to bizarre and one of the things that we really liked about bizarre was the fact that using bizarre we also have access to a tool that goes with it called launch pad so our basic workflow as Rebecca said we break things into stories and stories are chunks of work so the first thing that we would do would be to create a feature branch that's going to contain all the code that we're going to do as part of that feature so that means I can work on my branch and not affect anyone else which is a good thing at some point we need to actually get that branch approved and one of the things that might be slightly unusual about the Economist is that no single piece of code will get made live without being reviewed at least twice so what launch pad allows us to do is when we're ready when when I'm happy with my code in my branch I basically propose merge it and it appears on launch pad with a diff and everyone can log on and have a look at it see what code I've produced and leave their comments about whether they think it's good whether they think it's ropey whether they think it's a really bad idea and I shouldn't have done it that way at all so our coding standards internally are exactly the same as Drupal core so for people that join our team we always say don't be surprised if you put your first branch up for review and someone says hey you missed a full stop from the end of that comment that's normal for us we expect that and and to be honest it works for us so the process is that I will get someone within my team to review my code and when they're happy with it I will seek one of the other teams because as we mentioned we have three different teams so I will seek the review from someone from the other team so that may be someone like Judah and the New York team I go hey can you have a look at this code let me know what you think and we have a dialogue via launchpad discussing the code what's good about it what's bad about it what we might have done better why did you consider this approach that kind of thing and it allows us as a development team to learn from each other and to make sure that the code that we produce is of kind of the maximum quality that the whole team got to as a collective so it's a really important and quite powerful tool and testament to that is is the fact that we we produce code that generally does not have bugs does not need releases to fix and it tends to be ribba robust code so it's a it's a really good model if you haven't had a look at launchpad I'm sure they maybe get equivalent I strongly recommend having a look at and checking that out so trunk stage and live trunk is basically what we take our feature branch branches off from trunk is pristine there is no bad code in trunk at any given moment at the drop of a hat we can release trunk to live and be confident that it's gonna work there's gonna be no problems and that's kind of important it means if something comes up that's really critical and the site is has a problem we can release probably within I'd say about 20 minutes we could get a release out because we have that knowledge and we have that certainty that trunk is good and trunk is always good so stage stage is a Calua system ops guy will talk later about our environment but stage is basically identical to our live site so we have a lot of testing environments on the way to stage that we can test things in they're mainly cloud based and they're not representative of our life site because they can't be because they're in the cloud so stage is really critical for picking up those tiny little bugs and glitches that might come in from a very very specific environmental configuration so basically how it works is every Thursday we will take a cut of trunk and we will push that to our stage environment and then every developer that worked on a bit of code that's in that release will go on to stage and test and make sure that that works and obviously anyone else in the business can also see that environment and test and make sure they're happy with it and on the Tuesday following the Thursday release we will actually push that code to live so that's pretty much how that works testing every developer is responsible for identifying the best type of testing for his code so I believe Judo will be talking a bit later about testing so Ernest will take you through the various testing tools that we have but basically the choice of which tool is most appropriate for that particular bit of functionality comes down to the individual developer and finally vendor branching bizarre does vendor branching really really well so any module that we use on the site sits in its own branch in source control so if we need to upgrade a module it's very simple for us to take the latest release or even like a check out of source control and basically merge that into trunk and bizarre remembers where that module sits what directory it was in all those kind of things it becomes a very very simple operation to just merge it in and that's pretty much all I have for the moment so if anyone has any questions like Dana said spoke till the end and that's it for me awesome right get does you I'm just starting to work on a project using using it get has a review process views GitHub right GitHub has a review you can do code review through GitHub yeah yeah get users anyone using get that has a code review process the GitHub process yeah okay okay so we'll let's talk about the the theme layer the the economist theme is it's certainly the most complicated theme I've ever personally interacted with I try and say we try and leave the theme to the actual people who know so a quick overview of our team layer it's quite basic actually we have two main themes one for the public face website and one for the admin to face actually we use Garland for the admin section and for the public face we use a version of the 960 grid is an adaptation of it because when we implemented it we didn't want to redesign most of the components so in order to prevent some issue with that we adapt 960 and we made it a little bit wider so it's not that big difference and it's it's working quite quite well for us to be honest and everyone is quite happy with it in terms of the structure of the theme layer we haven't changed much it's quite the default behavior so most of the template files are sit or in the theme folder together with the CSS and the JavaScript files when it comes to modules and in-house modules we change a bit the approach and all the template files CSS and JavaScript files needs to be placed inside module that will help us to prevent having CS files that live forever in our website whether when we decide to disable these modules and of course the downside of that is that we have a long list of CSS files as well as JavaScript files but then they get aggregated so it's not a big deal it's also it helps us to find the specific sector of styles when we need to and sometimes this actually has another downside that is that we have to duplicate some of the styles because some modules use styles that are the same as other modules and this is an issue that we will try to solve and try to solve in some way that is not giving us a lot of repetition basically and this is what it says about the module CSS about the future we are we started in November 2010 to implement HTML5 the first step was to change the dog type and then with the with our first project about the user registration we started to implement the new input types new attributes for the form section of the registration and in the next months we are going to start changing the structure and implement the new semantic tags and also try to focus on them on the responsive design so that it will help us to deal with different screen size desktop mobile and so on I think that's pretty much it about the theme layer and how it works okay it takes a minute to do this thank you Angelo so now we're going to talk about testing so part of the challenge has been organizing the right kinds of tools to be able to meet the needs of meet the the systematic needs and also to make sure that we're finding things as we go along so there's been kind of a creative a creative bringing together of different kinds of tools to try and match all of the different possibilities so Ernest you tell us about that yeah I'm going to try and go through the tools first and then talk about how the tools work together so we use selenium for our functional testing is everyone here familiar with selenium and we use simple tests for our unit testing as far as like Dom said it's kind of up to each developer as opposed as to which tool to use for which case I'll go in a hand testing after I talk about the tools we use grinder for our load testing who here is not familiar with grinder it's a grinder is a Java load testing framework and you write your test and gython so every developer has to be able to write a test and gython it's pretty straightforward and then I'll talk about the team sign off before each release so those are the tools we use for testing and the way and the way we use them obviously is when you branch off you decide which tool is going to be best to test this piece of functionality or piece of code or whichever feature you're working on and then so the hand testing goes in terms of a you as yourself as a developer tested and then when it goes into your proposal on launch pad obviously the tester then takes in your code and runs a test to make sure it runs is okay on their environment and then we have the stage release and so the third hand test that goes on is at each developer then goes in the stage environment and kind of test different features or you yourself go on the stage stage environment because that's where all the code kind of interacts as a whole to see if it's still working there so that kind of goes on to the team sign off before each release we in Hudson we have all of our tests automated when you when something's merged to trunk it fires off a build which then runs every single selenium test that we have in the system and they all must be green before we actually sign off that that that that is ready to go you should be doing that yourself but so that's kind of our team sign off is that you then run your tests and all tests selenium tests and unit tests all run now the grinder test is a little different everyone writes their grinder tests and we run them and we get results back from them if you're not familiar with grinder it gives you results of the meantime for each of your tests and how it's how it's performing that gives us kind of a kpi and then we might get a message from Cal you or since ops guy who actually gets that graph and he might be he might say something like there's a spike you know in one of the tests you know you might want to look at that to see what's going on there so that's what grinder helps us to do is to kind of keep a baseline of how the system should perform and also kind of to help look for spikes before we actually release its trunk so I think that covers most of our testing I wrote down some notes because I always forget things so pardon me while I go over them to make sure I covered everything we use PHP unit to fire off our selenium test that's our interface which is kind of the only way to do it and also we've written our own kind of internal custom framework to help you know kind of encapsulate the general functionality that we want to do with our selenium tests and we've also set it up such that every test has a video attached to it so if something does go wrong we can go straight to Hudson and we can look at the video to see what happened during the test to try and diagnose why it failed I think that's it for testing I believe and I'll answer questions at the end we also use it it's a slightly modified version of grinder which if someone's curious I can go into later but that's about it yeah so and and I want to reiterate one of the things why I'm beeping here is that when if I'm working on a feature branch and then I merge that into trunk and then the selenium tests are not all green or this are not green it's my responsibility it's each individual's developers responsibility to figure out why the tests are no longer passing and to fix them so that it's a collective responsibility of all of us working on the code that you know that every time we're adding code all of the tests are still passing so which is a dance it's a dance that's a dance there was a brief period of hell where we queued up a bunch of branches but never actually ran selenium tests against them and then when we merge that into trunk every single test failed and so for a week we're trying to scurry around all fixing tests and not getting a thing done because we have this policy that nothing gets deployed unless all the selenium tests are green that's why Dom has such confidence that our trunk can be deployed so yeah just kind of a note to keep your eye on those automated tests and a warning for new developers when Elliot joined our team he spent the first two sprints working on any time any time the selenium tests stop working we're like Elliot can you look at the test but it's good to then you know you know you know yeah okay so automated automated this is yeah so speaking of selenium tests and those other kind of things if ever you guys worked in large environments you know that there tends to be a whole lot of scripts that you tend to run to try to automate deploy all those kind of things and you get your little toolkit of scripts well sometimes it's a lot easier and a lot more transparent to have a GUI and a logging interface and what ultimately what you're looking for is an integrated development environment and are not an integrated development environment and continuous thank you continuous integration system and that is what our Hudson slash Jenkins slash Hudkins is Hudkins is our nice name for the distribution that is Jenkins but used to be Hudkins regardless basically what it does is it allows you to run code on your on the command line ultimately that is what it boils down to you can run arbitrary code you can run Drush commands any of you guys don't know about Drush I recommend you looking it up very great very good you can run a lot of things that would have ultimately be run on cron when we started putting things into cron we found that it wasn't at least in Drupal 6 it wasn't always the most elegant solution at least at the time we're doing things everything is run in a single process so if something actually like killed the process everything just died behind it and if you didn't fix what was killing it whatever was behind it didn't actually get run and there wasn't a whole lot of visibility behind it so one of the things that Hudkins provides us is you can create jobs for each one of these processes you don't have to worry about handling scheduling within the code because Hudkins Jenkins handles all of the scheduling it handles all of the workers it'll make sure that none of the processes run in concurrency so if you have a long process that every once in a while let's say it's indexing your site and it goes beyond let's say the hour long it'll make sure that it doesn't run that process twice which is really important because you'll end up indexing things twice and doing even more work and it'll start to build up so like I said it allows us to be able to run jobs independently so for example one of the things that we do is our content producers create content in their you know proprietary economists branded application and then and then we pull it down through from CCI into our website well all we had to do is simply create a drush command for running the import into our system we've done other things where we we index our search we can even deploy the deploy process will probably be talked more by our sys admin but we have things for every step of the process the deploy that can all be done using a GUI so that we can see the output we can compare it against what we're expecting there to be emails get sent out if things fail I mean in general having this system is a great idea should I talk more when you see tasks custom drush commands drush is a great thing one of the things everyone loves drush up DB it allows you to be able to upgrade your modules and if you deploy a lot of times the deploy process will update modules but one of the things we did find was we don't always want to take down the site when we want to run these long-running processes so usually when you're you're doing an up DB you have to take down the site and then you run the up DB check everything out take site back up well one of the things that we found out is all you have to do is really run our create a custom drush command so if you want to run something in the background just create a drush command you can even throttle it if you want if you're worried about the performance and that gives you an opportunity to run the command to let's say import something or update some data maybe port it from a legacy system all while the site remains running so yeah drush has been an integral part of our system thank you it takes a minute for my microphone to come back on again yeah one of the the awesome team the first time that this became huge for us is that we were integrating topics so that when you enter the economist you could look at a topic and then see articles related to that topic but that meant every article had to be sent to a third-party vendor who then you know gave us back the topics that were related to it and then that had to be saved and that's not something that could be done every time an article is saved that had to be a whole back-end process and by using a custom drush command I'm getting it out of cron meant we had control and using a custom drush command is if for some reason we needed to send just one article we could just put in the node ID of that article or we can send them all you know so that's an exam these kinds of back-end processes as Judah was saying like anytime you have to run one instead of putting it in the whole cron task thing and you can also add selectors right to make it to make it different and you can stop it if it's broken not mess everything else up yeah also in addition to like just running drush commands and stuff like that we we run our our test environments through Hudson as well so Dom I believe mentioned we have test environments we have our local environments we have our trunk environments we also have what we call our htis which are human tests in our instances where we can take any of our feature branches and create a version of our website with it with a database with the the source code of that branch and then pull it up on a web browser and have our product owners look at it sign off on it have our UX sign off on it and make sure that all the CSS is showing up correctly and in addition to that if we want we can run any automated tests against it so all of our selenium tests and simple tests can run against the environments as well yeah so we'll talk about the infrastructure now which Cal you will who keeps us up and running and reminds us when a query is reeking havoc and we all he'll he'll he'll send the email to everyone it says you know we've we've come across a query that's starting to back up and all of the developers they scroll down very slowly because you're just really hoping that it's not the table that you added last print so right so I'm going to give a very brief overview of the infrastructure that we have to run our production production system so coming from the customer side we we use the CDN to deliver all the static content like the images JavaScript and then the CSS files then we have a hardware load balance solution and we then have to varnish reverse proxy servers to cash the content then the request goes back to the load balancer and is balanced across 10 Drupal nodes 10 web heads and then we do have a memcache layer and the database layer just to go over the numbers we have two varnish servers 10 web heads and for database nodes so one master database server and three slave servers so it's a pretty pretty actually standard setup we also have a similar setup for the again not against more amount of servers but the structure is the same for the staging environment and we also have a DR environment that we replicate our data to and in case of a disaster we condense which is a disaster recovery site and that's pretty much it we use Jenkins then to automate all the building and deployment so deploying to our staging and production environment is basically just pressing a couple of buttons in the Jenkins user interface and then that's pretty much it and performance and so we're at 37 minutes so we still have another 10 minutes or so for so I think we're doing good in terms of times I love the spotlight okay so let me first read my notes real quick okay so this is near the end of the the Drupal week so many of you guys probably already heard a lot of the sessions about how to scale Drupal and how to make it perform like a rock star basically most if not all the things that you heard in those sessions we were also figuring out the hard way at the same time so you know normal Drupal install it's fine it's great it's great for shared environments we started building our site on Drupal 6 so Drupal 7 is a market improvement one of the things you figure out really quickly is you need things like memcache and you need things like master slave database so immediately we knew right off the bat that we needed to go with a different distribution of Drupal 6 that do that distribution is called press flow basically it's actually has some patches similar to Drupal 7 but it's back ported Drupal 6 and it offers things right out of the box like the master slave replication and also patches for memcache of course anyone who runs a server APC is kind of a duh opcache there's no need to actually compile your PHP for every single request it's just something that if you haven't looked up do it it's no loss very easy to set up so basically yeah the key to any kind of an environment is any kind of scaling is layers of caching so as we said there's the CDN and then you go back down and you hit the varnish and almost all of our anonymous is served directly out of varnish and varnish can scale it can go thousands millions requests we haven't even begun to get to tipping over varnish but there are times where you need to let people through they're authenticated or they've done some kind of action which potentially they need something different about the page so let's say you get through that page you have beautiful things like Drupal page cache and the block cache and those kind of things that that was built into Drupal and you know you got to take advantage of sometimes things don't always follow that it doesn't hit a view so you can't use view cache perhaps the person has logged in so they need to see things immediately so Drupal has its own API of cache layer you know you can just use cache get cache set and make sure that any specific query any specific rendering of HTML can also be wrapped in a in a in a cache set basically one of the problems that we found though in addition to what came just out of the box is we needed something in between the node cache which kind of caches the structure of a node and the content of a node and the actual page cache which caches everything so we actually came up with this and I think it's pretty novel it's we call it EC cache but basically it's kind of like the the rendering of the node cache and we've actually extended it so it can render any endpoints if you will so all of you guys who have built a menu hook you set up an endpoint and that endpoint basically outputs something so we hook into that menu hook and we basically cache whatever it would have rendered and the great thing about this is you can tweak it for how you cache it so in the same way you can cache base off of a roll you can cache it for the page the URL or you can cache it globally so this allows us to possibly even cache some of our our highly hit pages like we have our our subindexes we call them channel pages and our home pages that we just want to cache like we don't care if you're logged in we don't care if you're not logged in we want that cache because we can't afford to recreate that that node millions of times in a day so so that that provided us an opportunity and and they're given takes of that that we we had to stumble through and figure out you know if you're not actually rendering a node you got to kind of jump through some hoops to make sure you're also storing with that node not just the HTML but any other components of the page that is being injected into it but it has been a great help it has helped us to be able to scale the site up one of the things that we have played with and we are going to continue to implement is edge side includes it is part of the Akamai network and also part of varnish that you can cache the page and not a portion of the page or don't cast a page maybe the page has to be an authenticated so you can't cache it but you know maybe the header or the footer or something doesn't change this allows us to kind of handle the caching differently for portions of the page so yeah some of the stuff we found out the hard way but in the end Drupal can scale it's just how you do it and you have to do it smartly and you have to make sure to use a layer of caching yeah the one of the things that I was saying is going from working on smaller sites to working on the Drupal site was that caching used to be did you clear your cache in your browser that's why you're not seeing the change to every at every level from code writing to pages you you're always thinking caching always always always so so we're we're two minutes from where I wanted to stop for questions but we're going to go into that a little bit because of course one of the questions that comes up all the time in talking about the Economist was it was migrated from from cold fusion site and it took a bit longer than it was originally intended to take and this question comes up all the time which is how come it took you so long and part of the answer is because and I think this was very smart that the adding adding business value constantly during the migration process integrated together right so that both new featured development and migration were going on at the same time and recently the migration has been completed so JJ is going to walk us through the steps to how the migration to how the migration happened yeah okay so I've got two minutes to do this so we started a migration we finished and done okay now what we actually did is instead of going for the full big bang everything all at once we did it in steps so we took the comments and things like the recommends and moved them to my to Drupal then we did things like the articles so on the Economist website it's all article pages so what you're actually hitting for about two years was a cold fusion site but then every time you went down to an article you were hitting a Drupal site so it was a bit confusing for the end person but they didn't notice because we've got everything styled the same way then for the final part we did the users that was quite a large system for the Economist because we have subscriber details which come from various third parties we have two or three different third parties which can do subscribers we also had things like coming along like the iPhone the Android apps all of them wanted details and access to the subscribers various other third parties that also feed off the Economist user database so that was the largest one hence why that's now done we shut down the old system and we're fully on Drupal as we were doing this though what you actually notice is we did the articles two years ago people were still editing the articles up until about six months ago on the old system so although the end users were viewing the articles in Drupal editors were still automating changing in the old system so we had to have a continual synchronization process which took any changes from the old system and migrated them across and how we did this was we wrote our own custom modules to get the data out of the Oracle database and move them across but we also use the migrate and the table wizards so we could just pull various data across from our Oracle tables which we pulled into the MySQL and translate them straight into them my SQL tables for Drupal so the actual articles were now we're talking about the Denver Mountain this is very nice if you want to go skiing here okay so from a it's kind of throaty doesn't it he did deliberately just for me so when we were migrating the tools we were using with the Drupal migrate tool we wrote by I think Curve and MotionCo and the table wizards once we then migrated we turned off our synchronization process because we were only on Drupal for articles we kept it going though for the users because the users were still coming across so when you went and subscribed to the Economist you were going to the old system pulling it across when we were getting towards the end it was a case of all teams come together so as you noticed earlier we have three teams geologically located around the world for the last probably three months everybody was working on the one thing to kill our old system so everybody took a different part be it creating an index page creating a job site everything got moved across to the new system and that's kind of why it take took three years about three years so now we are two three weeks into the about three week four weeks we've been purely running on solely Drupal the on the morning when we actually did our code release which was a Tuesday we released the code about nine o'clock and by about half nine all of the old servers were turned off decommissioned completely nuked so there was no going back we were going forward and that's where we are yeah it was a boring day wasn't it a boy yeah yeah because it's a geological location the guys in the states came in and it was all done dusted no panics it was a boring day for us it was good for you yeah so now I think we're about to open for questions okay yeah so we're open for questions Eric is there anything that you want to add before we do questions he's good okay all right I think questions are probably more more what we want to do so let's see questions I will hear you know what we'll do like that well it's a very easy question how many models are you using on the economist side okay yeah so how many modules there's what seven million four hundred ninety two thousand does anyone know the exact number I believe around 40 or 50 40 or 50 customer contributed or both together both together I think contributed around 38 39 only because I'm doing that organization right yep yeah okay so probably fit about 40 or 50 all together half or custom half or contributed I'd say about 70 30 70 30 custom to contributed 60 40 70 30 there's a lot of custom a lot of custom code a lot a lot that's yeah a lot yeah so you have subscriber only content on your site yeah so you have to have a flint case you use or logged in subscriber roles how does that affect your performance when they come to look at articles that only that they have to be checked on against the database basically so you're asking about authenticated users who come through who have a subscription we yeah they're really just authenticated users because they have a role and that role allows them the subscription if you actually probably the paywall dog you probably talk more to this but we have a paywall also but so yeah so they're they're authenticated users we have a fairly small percentage of people on a regular basis that are coming in a logged in state a lot the bulk of our traffic is anonymous if that shifts dramatically we'll probably have to take some different views including you know some of the ESI's and the way we cash different parts of pages but right now it's a low percentage of people view the site in an authenticated state you know maybe using varnish in a different way or ESI is particularly yeah so now it's 10 minutes right varnish the expiry time five minutes five minutes okay thank you my question is do you do the team use some project management system and how do you make the work assignment thank you okay the when Rebecca we were you Rebecca is our scrum master and we use agile do you want to say a little bit was the question about the tools that we use like I said we we use we use a scrum which is an agile project or product development framework as far as the actual tools we've gone very low-fi and we mostly use Google docs to we have a backlog of what are all the features and that's that's in Google spreadsheet we do burn down charts with with spreadsheets in Google and basically we also use that as a wiki for for documentation we've looked at it and we've tried a few different tools for for organizing and we found that the time it took to update them often wasn't worth it as far as like the day-to-day of the team of who's what we're working on with the teams in in New York for instance we just use a task board a physical task board with post-it notes so we just list out what are the what are the user stories or the features that we've committed to for it for that two-week iteration and and and it's very low-fi it makes it we do the 15 minute meeting every day looking at that boards updated and move things across so it means that we don't spend a lot of time updating updating a tool and more time just everyone talking together and and it's very transparent where things are it also helps because then other people in the office can walk around and and they can see exactly what's being worked on for the Austin team because some of them were remote in Austin and some in New York we do that task board on on Google docs as well we just make a very very simple spreadsheet of stuff that hasn't been started not started in progress and done and very very simple yeah Eric we use unfit for bug management yeah okay okay I think how we did two more how do you work with estimations as to how to plan your job how do you reach an estimate there that you trust sure this is something that we do with agile scrum we use the concept of story points which is rather than getting into the trap of trying to estimate exactly how many hours or how many days something's gonna take the the product owner will put together here's here's a description of what we want we work with the team to make sure that's clear have some conversation about it and also come up with a what we call a how to demo description of like this is how if you went through these steps basically it's it's like the uat steps of what a user would do and we get together with the team to do a session of what we call planning poker and we size it basically is relative sizes on the Fibonacci sequence so it's like one two five eight thirteen and so we give them we give them relative sizes for each slice of functionality that we're looking at and with that we we take into each two weeks what we think we're gonna get done and after you've done a couple of sprints you you get an idea of like how much you can get done and and you only take in that number of points it gets us out of the game of like what exactly how many hours but more of what's the relative the relative effort and the commitment is then at in the planning of the team says okay based on this whole conversation we've had and what we understand of the relative size of these things then we commit to getting these things done in the next two weeks and that's that's the commit for planning and forecasting I mean when you when you go with an agile approach you can manage to time or you can manage to feature set so if the feature set is there's a minimum feature set you want then you have to be flexible in the amount of time it takes to deliver if you want to time box it then you have to be flexible in what you're going to deliver so for the past couple years we've we've kind of delivered on a minimum feature set and then said oh okay it looks about done for our next round of development we're going to be looking at more tightly time boxing it and the idea is to really get the teams and the product owners to really think about those bits of functionality that's that are going to deliver the most you're the biggest bang for the buck in our experience you know the Pareto principle actually works we get through about you know when we've gone and done these extensive huge backlogs of everything we can think of to do we usually get through around 20% of it and we wind up chucking the rest away so the thing about that is there's there was time and effort in that bottom 20 percent so we're going to try and approach this time where we say okay you know you've got a budget of four sprints or eight weeks you know what are the really high value features that we can do and can we get a viable product out and kind of challenge ourselves that way and that's how we try to manage expectations upward as well okay I'm fortunately we're out we've actually used extra time right we're out of time you know an hour yeah so 40 right because we started at 245 so 345 is when we're supposed to end for coffee we're keeping you from coffee and look you have stayed with us so thank you very minute you know this they found out yesterday afternoon at like four in the afternoon that they were going to be doing this today and I think they did a phenomenal job and great thank you for coming