 So, how to close a thousand websites and replace them with a Drupal platform? Yeah, as you can see we're quite a lot of people on stage, so it's going to be a bit information heavy. So, I hope you had a good lunch and sit back and be prepared to be bombarded. So, this session is for LSD enthusiasts. That's large-scale Drupal of course. It's government Drupal types and it's for strategists and content strategists and for anyone in between these groups. Just so we have a reading on you guys and what you're here for. How many of you are interested in large-scale Drupal? Okay, so that's all of you. Anyone from government sectors or something like the municipality? Yeah, so that's half of our colleagues. And any content strategists among us? That's a few as well. Great. So, what's on the agenda? We'll do a small presentation of who we are as individuals and of course as the municipality of Copenhagen and as well as a small mention of pro-people. Then why are we doing this and how we did it? And we'll use some examples from our current project, our big internet. And then finally, of course, we'll look a bit into the results. So, yeah, I'm the guy on the top left. I work as a digital strategist. I've been responsible for the internet and I'm working a lot with Drupal and I've been doing it for almost three years now. I guess, yeah. I'm Kaja. I work also with the overall strategy for our platform and today I will focus on the content strategy part that not so many of you are interested in, but I'll tell you anyway why it's important. I've worked with Drupal since Drupal 5, so I think that makes 5 for 6 years now. Please, applause. Hello there. My name is Rumania Donov. I'm from pro-people. I'm leading the Drupal operation in the Bulgarian office of pro-people. I've been doing Drupal for five years now and I'm intensively doing performance optimization infrastructures and basically creating a really crazy technique. Crazy technical projects. Thank you. I'm Yuri. I also work for pro-people and I was involved in technical part of the project and my area for expertise is services and communication between sites and we're going to have, we have a lot of them on this project, so that's where I was involved. So, a bit about Copenhagen municipality. I refer to it as KKs a few times. It's a Danish abbreviation, so bear with us. Yeah, we are 550,000 inhabitants in the city. We've got, we're a green city, we've got 650,000 bicycles. Over the next 10 years we're guessing that we'll be around or it's estimated that there will be another 100,000 inhabitants in the city. This is not only for Copenhagen, of course, but Danes are considered the heaviest population in the world for the last three years running. Let's see how that keeps on going. And that, despite the big tax raters, most of you probably know about already. And that's of course also interesting if you're working in Copenhagen and you notice that there'll be another 100,000 inhabitants. That's going to be some good projects. In the municipality we've got around 40,000 employees depending on how you look at it. And they're organized into seven distinct administrative branches. They each have their own mayor. I'll show you a small chart later. So, here's where it gets interesting. This is our IT fact. We've got at least a plus of 1,000 websites related to Copenhagen City in one way or the other. For those 1,000 sites we've got at least 500 editors, maybe even more. Not everyone is registered. And considering we have 1,000 websites, we have 500 editors and we have a lot of hosting, of course. That means we have a web budget that we are estimating goes above at least 10 million euros per year. And on top of that, to make things even more confusing, there's 1,400 different IT systems. Among those, there's like 17 time-locking systems. Probably even more. We don't have the exact numbers on these. We've got 70 system maintainers. Those are the guys that work with the systems on a day-to-day basis. And, yeah, we've got XX IT budget. It's not even... It's not an easy task to calculate this and we haven't done it. And some of you will then say, so we're doing a big system. How can we actually derive some sort of business case from it? Oh, sorry, actually, we... Sorry, just a quick... We introduced this slide a bit late. This is pro people, of course. We have to mention them. They're great guys and they're the IT partner of our project. So, yeah, why did we do it? As I just tried to mention, it wasn't because we have a big business case for this. I'll read this quote. Organizational anarchies are organizations characterized by problematic preferences, unclear technology and fluid presentation. Such organizations can be viewed for some purposes as collections of choices looking for problems, issues and feelings looking for decisions situations in which they might be aired, solutions looking for issues to which there might be an answer and decision makers looking for work. So, yeah, this is the garbage can model. It's sort of underlines that we didn't have this sort of distinct reason for doing the system if you look at it from a business perspective. This is one version. The second version is this one. We've got at least 250 site course sites. We've got 21 Drupal sites at the moment. That's growing quite a lot, of course. We've got 11 Plone sites, 44 WordPress sites. At least 200 homegrown, unidentifiable systems. We don't even know what they're built on. And then, yeah, the rest of the numbers are all the other systems. You know, so there's Synchron, there's Dynamic Web, UMNA SharePoint, Type 3 Oracle. I'm guessing if you can name it, it's in somewhere. Yeah, and that, of course, that translates into a hell of a CMS frenzy. And for a big organization like ours, that's more or less impossible to maintain. That's not only from a server perspective. That could be from a support perspective. Yeah, anything you can think of, it's complicated. And that, of course, translates into a huge overhead. So we're overspending on stuff we shouldn't be spending on. And it also, of course, if you look from a developer perspective, it means that we're doing the same integration to our Active Directory maybe a hundred times. That's expensive. So why we did it, why we did it version three? Well, some of our sites are really old. This is our now-buried internet, or what remains of it. As you can see, it's, yeah, it's old. So why did we do this in actual fact of the summary? It was, of course, with this goal in mind that we wanted to enhance and improve our whole ecosystem of web. That means we want better maintainability. We, of course, want reusability. We don't want to do the same thing again and again and again. Scalability, of course. We know we're going to expand. We need bigger and bigger solutions for some things, at least. We want to standardize stuff that makes it easier for us to support. Of course, quality. If we can put things on one platform instead of having a trillion sites, that means that we can make better quality. And, yeah, of course, we're a political organization. So if we can leverage any form of economy from scale, that's, of course, yeah, that's an obvious benefit. So why Drupal? You guys can probably give us a lot of better reasons than these, but Drupal is a great system for what we're doing. The major decision point in Copenhagen was actually, if you look at it now, probably mostly that it's open source. And, of course, that there's a big and a huge community. Us, all of us here. But it's not a glorified sort of, okay, Drupal is better than everything else in this term. So, starting to build this platform, we set about some mantras for ourselves. How can we do this the best way? One is, yeah, close more websites and applications than deploying a new ones. If you think about a thousand websites, you know there's going to be a lot of redundant old and useless information out there, and we want to kill this off. We want to replace three solutions with one solution that has better quality, basically. Second mantra, we want simple scalable structures instead of meeting specialized needs. This is, of course, quite difficult when we're working with politicians and in political institutions. Scalability is not sexy, and so is structure. They're quite boring. Politicians want good magic. They want to see some fancy websites, and they don't really care if there's any sort of reuse. Unless, of course, they get the obvious sort of question, do you care? Yeah, but they don't. So, the final one. We wanted to use existing country modules and contribute patches instead of creating our own ecosystem of modules and functionality. This is historically what we've done in a big organization like Copenhagen. We've used the image of a dog pond for the Danes among you. You know what this means. This is sort of a Danish way of looking at the world, or not looking at the world, I should say. Instead of taking part of a community, we have a tendency to maybe say, we are so big that we can do things ourselves. We wanted to try and kill off this way of looking at IT and web, at least in the Drupal Web Department. And, of course, this also gives us some obvious benefits. So, we're not dependent on solely one person in our organization who did one module 10 years ago or a specific vendor. So, we had some challenges, of course. When we started this project two years ago, one and a half years ago, there was no clear-cut strategy on web whatsoever. We still don't have a web browser strategy. It's not a thing that just builds itself. Another challenge. We've chosen a system that works best at least on a lamp setup, had no lamp setup experience whatsoever. And, of course, as I mentioned earlier, we're a political organization. That means we've got seven distinct branches. They each have their own mayor, their own management, their own finance, their own IT, and their own communication department. Whatever you can think of, they have it internally. And on top of that, or should we say across that, there's corporate services where me and Kaia, and a few of the people here, are working. We should try to be politically independent. So, our challenge is to sort of navigate these seven distinct political structures and their own goals for what they want to do with IT. So, this famed platform that we're doing, what is that? We want reusability. So, it means we want to look at user administration and access as a part of a platform. This is not something we want to reinvent and reinvent and do again and again and again. Same thing goes for UX, information, architecture, and design, of course. So, we're planning a project each time we do a new website. Content strategy, we want to be able to share content. Of course, a code base. Let's not do another Active Directory integration again if we can avoid it. And of course, infrastructure. So, to accomplish this, we tried to set up a timeline for how we'll approach this because there's a lot of things in there we need to solve when we're doing a platform. And as you can see, it goes on until sometimes 2015 and it'll probably go on forever. But the point of this being that we're a political organization where people want magic. They don't want to build a platform. So, we have to finance our long-term goals, the platform, the platform strategy from the individual projects. So, if you take a look at the chunk between 13 and 14, you can see that we're doing KK Intra. That's our internet. Our internal self-service programs are my page, a page to see some basic user information for you as an employee, our directory, and a few other things. And that then, if we go down to the bottom of our platform, it translates into some solutions that we can reuse for new projects. So, at this point in time, we now have our first LAMPstacks. We've got a lot of LAMPstacks now. We've got a taxonomy server. We've got Solr running. We've started Drupalizing our internal developers. We've communicated about this Drupal platform. That's a big challenge in an organization like ours to sort of convince people to go this one way. So, yeah, to exemplify some of the things on this platform, we want to look at our internet project because this made the first sort of big impact into getting this platform done. A major point, of course, being let's avoid early mistakes. We've got a long road ahead of us, let's try and fix everything all at once. Then we'll set ourselves up for failure. That's a given. Then we wanted to look at content strategy and scalable information architecture, which Kaja will look into in a second. After that, or along with that, that's probably the right term for it, we'll look into infrastructure and implementation, which Roman and Yuri will look into, and then prepare for the future. I'll tell you a little bit about how we think about content and content strategy as an important element in a platform strategy, as an unmissable element in a platform strategy. We have this, one of the first things I want to show you is this model, and it might be a little hard to see it from afar, so I'll explain what it means. It's a matrix of all the content channels that we are trying to incorporate into the platform. It's a matrix of the 1,000 sites that exist, and there are probably only 300 that will exist in the future. What the accesses mean are on the vertical side, on the vertical one means there's broad websites, that means they have broad target audiences, they have a lot of content, they have broad goals, and on the other end is focused on specific websites with either a more narrow target audience or a very narrow theme focus. On the other accesses, in one end, the internally directed websites, those are directed at employees, and on the right side is externally citizen directed websites. So to give a few examples of what's inside that matrix is on a very broad portal-like website directed at Citizens is our main portal for the municipality where citizens go to find service and business goes to find service and information. And as examples of more focused websites would be institutions like kindergarten websites, campaigns like Bike Mall in the City, or specialized long-term websites like open data websites or websites telling you about parking in Copenhagen or whatever, specialized narrow focus. On the internally directed websites that are sort of focused would be online employee magazines, institutions with login for citizens like for parents, and specialized small websites for small units of employees. And on the broad side, employee directed is our internet project, obviously. It's not one website, that's important to mention. It's, I think, now 13 websites because we have this decentralized structure of administrations. We ended up deciding it was better to have a number of websites so there would be a certain independence to each of the administrations. Yeah. So the goals of looking at content would be for us to focus time and resources for editors because with the massive websites we had now, editors had to learn a lot of CMSs. They had to log in maybe somewhere between two and 15 places to edit content. There was a lot of duplicate content around, so we wanted to get rid of that by enhancing more sharing and reusing of content, by making it possible to publish too many channels, by having a coherent UI for editors to separate form and content. I'll show you why in a second and to make this work across the platform, which means across the landscape of websites that we plan to have. This is our starting point for solving these problems, the internet that we have now replaced. It was basically two content types, articles and news. It had a very advanced, well not advanced, but a VCVic editor with a lot of freedom and tables to do the layouts. So every page would look different. Like this is another example and this is another example and it's things that we want to avoid in the future. So our approaches were these. First was structured content. I think that probably everyone here knows why it's important and has heard this song a lot. We work with it in this way that we learned ourselves how to love spreadsheets. And this is an example of one of the spreadsheets that we are now using to structure content for content types, for views of the content types, for node queues, etc. It's a fork of a spreadsheet that Larry Garfield has proposed. I can post a link on the session site if anyone is interested. So the consequence of working with that structured content is that for most editors this is what they will see mainly. The node edit forms with a very minimal VCVic. But some editors will have coordinating editors on some websites. They will need to be able to assemble pages in a more advanced way. So we have chosen a strategy where we have tried to find a somewhat clear-cut assembly model which is based on panels and the panels in place editor where editors are able to assemble views and they can also put in more custom content which we base on fieldable panel panes so that they're still structural and we can find them for the views later, etc. It's an approach that's very much inspired by this distribution called Panopoly that we like very much, but we have our own version of it which is a little bit simpler, we think. This is what the editors see when they assemble pages. I think that everybody knows that if you don't put restrictions to the amount of panes that editors can use, the system and the modules you use will output, I don't know, 200-plus panes that can be used and it's very confusing. So we've narrowed it down to the ones that actually editors need to use. The fourth approach we had was to support in our internet project models of sharing and reusing content. This is one first attempt of that which is the option for editors to publish news, important news across these 13 internet sites. So news that's important to more than one administration can be published without having to copy and republish, etc. This is another exploration we've done into reusing and sharing content. It's a pull. The first one was sort of a push model. You can push content out. This is a pull model you can import. This is a search result where editors can use the import button, the blue one in the middle to import content from other websites and the main fields of that content, those content types will be synced when it's edited on the mother side, the first side. So the fifth approach we had was to learn editors to love taxonomies, which is if anyone's done what we're working on, moving editors from a hierarchy-based CMS and into a taxonomy-based CMS, something that's very hard editors always ask, but what's the use of this? Why do I have to use all these taxonomies? So we developed, well, Ruhmann developed this very nice pain that lets editors assemble views. For me, it was sort of inspired by what was called Simple Views in 2006 that you scrape off all the unnecessary, very technical, difficult configurations in views and you make a very simple way of assembling views. They can choose content types and taxonomies, and they love it. It was a surprise to us too, but they really like it and they say things like, I understand now why I have to use taxonomies. How am I doing on time? Okay. Okay, good. So next phase, lessons learned from this content work. The first lesson learned is that we need to extend the number of specialized content types. We have now, I think, around 10, but we need to extend it with even more because if we need the editors to not edit too much in the VC week, we need to give them options to display and show content of very different types. So that's one, and what we specifically learned from the internet was that we were, we hated tables because we didn't want the editors to do layouts and tables in the VC week, but we didn't actually see that they used tables also for sensible things like custom lists of who to contact and what newspapers were around for specific targets. So that's just one example. Another lesson we learned is that even assembly models like panels with the in-place editor are very narrow. The focus can be misused. I'll show you an example. Creatively, I'll give him. It's creative. But he, you can see in the right column, do or here, it means you are here and it's an editor who used the in-place editor to create a fake breadcrumb because he missed it so much, the hierarchy of his old site. So he did that and that confuses a lot the user experience. Well, it's creative. I'll give him that. Next steps. Build more specialized content types. Integrate more content from other systems. We have the 1400 and there is definitely around 50 that could be relevant to display information from. Expand content sharing and publishing options. We're working on learning from those first attempts and making them better and build best practice guidelines for content editors. I have time for this. So the second thing I wanted to talk about as content was just a few points about how we tried to do a scalable information architecture on the platform. So the first thing that we wanted to avoid was hierarchies because they're not scalable. For a large organization with 40,000 employees and 1,000 plus units underneath, a hierarchy does evolve into this like church yard of old mishandled content that nobody looks at that's in the 10th level somewhere in the hierarchy. So we wanted to avoid that. We wanted instead to learn the editors to think about the site as many sites and not one site. If we have a thousand units, they cannot all be sort of put into one site. We have to create a sub-site thinking. The third thing is that information architecture is not going to solve your organizational, political and economic problems. So we tried to narrow down the amount of menu items, which is hard because everybody wants to be in the main menu. And in a political organization, politicians make really interesting choices about what to put in the main menu. So we tried to repeat that point a lot. And then we wanted to connect these 13 plus internal sites through a global menu and we're working on that now. I'll show you. This is the main architecture for the Internet. It has only two menu levels. The first one is the black one. It has a sub-level, so those are the two levels. And this is the next one you see underneath, the black one with the purple, is one of our, I think, very nice successes, is that we made it possible to keep the overall administrative menu in focus but also have a local menu for the specific unit that you work in. So this is an example of a center for competency development. And they have their own site inside the site. They can go for the common information in the top still. And the last point, oh, I skipped one. How do they find the units from the main front page? We developed this very little thingy that you can put on panel pages where you can, it ought to suggest your unit so you can go directly from the main site to there. And this is our next step. It is combining all the employee-directed websites with one common big global menu, which is the black one in the top. And the blue one in the right side is all your personal things. So those are, I think, six or seven websites we have that contain personalized information. And those will be always available to you. So we will develop this as a sort of service that provides the information based on who you are, so it generates the menu based on who you are and what roles you have in the organization. And it can be plugged into all the websites. So that's nice. And that's also the end of what I'm going to talk about. I'll let women continue with all the infrastructure that makes this possible. Hi. I'm glad that you're still focused. And I'd like to start with what we actually want to say, because so far we are bombarding, bombarding with a lot of facts, but for me, the message is very important. And when we make the presentation, we try to say, okay, but we should repeat our message. It does not work that well. Why we are here? We are here to encourage you to adopt Drupal in a large organization. And we are here to encourage you to do it and to show you how we did it because we consider this as a great success. And it's possible because of you guys. It's possible because of the community, because the community is the most valuable feature in the system. I'll show later on, but we use community to give back to the community after this project. But having the community will give you a power that no other system available can actually do it. So please feel encouraged. I really hope that after the session you'll go and say, yes, the big project can be done in Drupal and it's fun and it's a great thing. Okay, now for the boring stuff. Infrastructure, as you see, in an organization that has not been adopted the LAMP stack so far, the infrastructure was really tough, and basically what is IT infrastructure. The IT infrastructure is a combination of hardware, software, services, and tools that ensures smooth operation. Creating a really solid environment is much more a hardware task than just creating a branch of websites. And basically it's where it succeeded or failed. So let's start by defining the requirements for the municipality of Copenhagen infrastructure. Then is it a self-hosted solution? Because of data security, because of wall regulation, they need to host it themselves. But this also gives them a benefit to be very flexible in extending the solution because they are a public organization. As a public organization, any extent or new services requires a tender. And having the infrastructure inside will give them flexibility to extend the infrastructure much easily and skipping a lot of procedures. Of course, they use quite different systems, as you've seen, so you need to create an infrastructure vendors to start developing and integrating inside their environment. So basically when you have a new vendor, he should receive a box where he can actually start playing immediately and do the integration. They have very little experience, so controlling the websites should be very easy. They should have a user interface that is very easy to maintain and give them an overview of number of websites running, number of platforms running, number of servers running, and it should be relatively easy for them to actually create a new website, delete the website, put the site in maintenance. Of course, a system at that scale should be scalable, which means that you should be able to add more hardware and face the new demands, so you should not create an architecture that depends on a single unit of failure but that you basically can scale afterwards. And it should be as easy for maintenance as possible. Again, if you do great products but make them hard for maintenance, it will be a failure anyway because the people will face problems. We know that system fails and if it's not easy, if you don't have the procedures, if you don't have everything in place that will make your life easier, then the project will be a disaster. And last but not least, we meant this to be reusable and it's not like reusable in a single vendor context. We hope that we're creating things that other vendor can actually reuse and we even try to push to the limit so when people start developing things, they start thinking of sharing in the beginning because it's much easier to start to create something small and very custom that nobody can use but if you are forced to think global and to think about usability of your code, then you start making the things that actually people can use. Of course, we didn't reinvent the wheel. We used the best practices that are already available. We are using virtual machines. We are using predefined configuration, storing puppet. We automated the procedures of creating new servers so basically as the system administrator in the municipality states a new server can be born in a speed of a second. Of course, we established a development box for each developer so whenever a new vendor comes into the organization and needs to start developing something, he immediately receives an account for the environment and he receives a virtual box machine where he has the full LAMS tank ready and he's able to do all the integration without access without having this virtual machine. We have also established what I call the infrastructure strategy which means deployment procedures, failover procedures and all the documentation because when creating something very big, you need to document, you need to say, ok, if this is happening, you should do this. And in a large scale organization this is one of the most important things. If you don't have a clear information of what should be done in different situation then it will be a very difficult because there are too many people involved. Another thing that a lot of people miss is optimizing when you're doing software. People are saying, ok, give me a metric what is the expected world so I don't need to care about the cash, the world is war. And this is a completely wrong statement to me. We should try and get people to start performance thinking since the beginning. So if a content is cashable, even though all the users are logged in, why not cash them? Why not put them in varnish? They are technologies. So you should not be like, ok, it's logged in content, Drupal does not support cash for logged in content and we don't care. You should think about the performance. Otherwise at some place especially if the project is successful you will have a lot of problems. So we did all this work in the beginning to document the approach and we decided also to verify this one because the infrastructure is a really serious matter and we did everything we have the knowledge but we decided ok, it's a costly thing and we should verify it. So we invited our partner Acura to have a workshop with the customer and go through all the documentation we have made through all the plans. So to be sure that the infrastructure we are going to implement is actually without any mistakes. And the workshop went very well and I think that we've proven that what we are thinking is actually something that will work. We try to make the maintenance as easy as possible. Of course, I'm really saying we because I consider that we as a customer and quite we work together so we put a lot of effort in like having the same and share the same goal, communicating the same things and trying to work together to make the things as good as possible. So whenever it's possible we put file over on the places where we it's not possible, we make it easy to make the procedures what should happen if a server fails. We also establish well-known deployment strategy. We have a development box for each developer. We have a staging server where a developer can actually deploy something and show it too. Inside the organization once it's approved let's say we have a new feature and it is approved, then it's actually the KK that have the ownership and we can move the code from staging to pre-prod environment. The pre-prod environment is complete replica of the production environment so it has completely the same setup and you can run a stress test there and prove that the system will work good and then you move it to production and this is completely managed by interface. So how we try to push and solve the problem with the usability of the code. Solutions are pretty obvious but we say okay, we need to create one place for all the developers for all the vendors where they can actually see their work. We choose GitHub for it as a user interface but we also push it to start thinking not of a website but starting of creating actually a Drupal profile. Of course we have it's a natural thing because we need to create a factory that you create a lot of websites but actually we start considering every project as a web factory so instead of just creating and deploying single websites, we focus more on having a Drupal profile and a Drupal profile is something that is very easy for another developer to start creating they are using this and this in this module and I can go to this and this in this module repository and see to read me, see what actually this module is for and we enforce that a custom module have their own repositories because if you like combine everything into one repository, people start thinking of the components as part of the whole but if you enforce and says okay but this module should live outside, they start thinking okay how I can actually make it so it works in other environment also about the interface we choose, of course we choose something that exists, it's open source and it's completely Nordic A gear is in Nordic mythology got the oceans and if Drupal is a drop of water of course the ocean is the biggest holder for Drupal but why we choose A gear beside the name it may actually matches all the requirements, it's open source there is a community behind it there are like 40 plus modules developed for A gear and it's very easy for us to extend it what we have created so far and what is running on A gear is a distribution that we call KK internet factory we have now currently 12 instances that runs these distributions we have one distribution that manage and works as a central file manager so the editors from all these instances will be able to actually share files and use them we have one instance that is a taxonomy server for all instances that they can share a taxonomy which is a very useful thing for global searches inside the system because it gives you the same categorization for the content so it's really good for making searches we also have a very specific distribution it's called KK MyPage it's actually a personalized place for the employees inside the municipality where they can see a personal details like what's my salary how many vacation days I have left when was my last promotion did I get promoted at all and stuff like this so I'll just outline very fast some of the main features of the KK factory it has full web support including SSO which means that if an employee is working into the domain on the computer it's automatically working on the website it has both push and pull content sharing which means that an editor of a website can choose that this news or article is important and he can choose on which other websites it can be available another mechanism where actually the editor can use a search find an interesting article and create a local copy on it we are using Apache sour for searches so we have one Apache sour instance that index all the websites and it's easy to search across them all and of course we have a responsive team we have created a space team that is supposed to be used for all municipality projects and two sub teams and now we are going to talk a bit about migration thing and the services and I'll invite my colleague Yuri well if you might notice for such a huge project we can speak like whole day about technical details but it's not the main aim of this presentation so I will try to be fast because we needed to migrate so many sites all of them are in different systems but the key solution we have gone through is the feeds module so we build our solution around it and at the moment we have a lot of content types and I will go through like basic high level way we do migration so for each website that we need to migrate we spin up the site in Drupal infrastructure and then we import all the content from the site we are migrating from and it can be huge gigabyte file somewhere with a lot of XML files in it and in the new site we just point out where that huge archive is and then we pull that file unpack it and with feeds with feeds tamper and expats we just go through those XML files and pull all the content in and that has a lot of advantages because it has UI so theoretically it's possible for even content editors to do that but of course we involve more technical people in that process and it works pretty well and we can possibility to roll back so we use features a lot on this process so some statistics a lot of nodes terms, files, sites and another important thing I want to talk about is the infrastructure about the services because we had to make the sites communicating with each other as Roman told that we have special site that is file storage like central one and we have taxonomy and when the sites communicate with each other they use services a lot and okay for the services for example I can give you some scenarios if I upload a file and I want it to be shared between all the sites I will need to tag it so it's shared and then these files gets pushed to the file storage and everyone on the other sites they can search in the files that are in file storages and they can say like oh yes I want to use that file and it will be pulled the same so when I upload some content on my local site it got indexed by Apache Solar and all other sites have access to the same Solar instance so they can see like oh there is a new content there and if they build some like blocks with lists they will see that content appearing there but if they click they will go to original site but the idea is also that in terms of sharing I can say like oh I don't want just the title of that nice article I want if it's like breaking news or something I want it to be local so I can go to that article and I can download it to my local site so we have done a lot of fancy stuff in terms of services and it proved to be a very nice solution and it's extensible and we have a lot of experience in that area as well thanks so we were really fast to the technical solution because we don't have much time it's difficult to explain a 6 month project in 60 minutes but we are going to post Jopel case on Jopel.org by the end of the week and also you can always join the pro people booth and start asking questions the people will be more than grateful to answering you and I'll be really fast now just going fast to actually the effort made we've been doing the project for we use more than 10,000 main hours we use 200 hours in screen meetings we implemented 2,700 features and there was a small amount of people involved as you can see 150 and I would like to thank them all the result for Jopel we have created 70 custom modules 103 features we have contributed 58 patches back to Jopel we have reported 44 issues we have contributed one module and we have actually used 200 contributed modules so this is why I emphasize that the project is because of community we also created one base and two sub teams the results for KK we have managed to close 50 websites we have created four Jopel profiles and we hope that this is just the beginning of a very good friendship okay I'll invite Kaya now yeah so before questions I'll sum up real quickly what we think is the main themes of doing a successful platform one is to have connectivity so it needs to be easy to plug into and this I think is one of the main benefits that we have seen in Jopel is that it is easy for other systems and other data models to plug into in a meaningful way that we can work with it afterwards the second point is that for a platform of this size you need anchorage and attraction so you need to make sure to involve users involve decision makers involve editors in what they need this is why we emphasize a lot the constant strategy the bridge between the infrastructure and the business needs and the third thing is that we need to create value and we try to do this in our business model inside the organization so you get a lot from this site factory that you can't get outside of the municipality we're cheaper now and we give more than a standard system would give to the editors and the IT managers around the organization that's what we think we're doing if you have any questions you're very welcome to ask them now you can start thanks so I noticed in your numbers I think it was 58 custom modules that you wrote and that turned into one country is that right? we have 58 patches and we have contributed one module I just thought that there were a lot of custom modules that you created 70 and I'm curious about that ratio I'm fairly new to the Drupal to the whole Drupal scene and my understanding was that a lot of the work that the developers were doing was done as sort of part of a community project so I had a good idea good work would be reusable wouldn't be wasted so maybe I'm a little surprised that out of 70 custom modules only one was contributed back to the community and my question is for the other 69 custom modules was there not a contributed module that someone else might have written before that you could have reused? this is a good question but actually quite it's a question for proprietary systems that only exist in this environment so does not make sense to be contributed back basically and also a lot of the custom modules are just configuration over a stack of contributed modules that cannot be unfortunately exported in features just a question about this process of kind of going from a lot of websites to one sort of federated structure did you have a lot of fights along the way where you sort of had to maybe upset local editors and how did you do that or did it all go smoothly for you? smoothly is probably putting it a bit too optimistic there hasn't been any sort of real angry fights of people not seeing the benefit of this if you line out that we have a thousand websites on so many different systems it is quite apparent that this is a problem for us so in that sense there's good there's good backing for this project in the whole city state the whole city but of course there is when we get a bit above this sort of platform strategy what we're offering each of the administrations then all the specialized needs arise and then that's where we hit the real fights because then one for valedicts way of talking about IT might be completely different from another so there are fights but it's at a specialized level and not at the low level where everyone sees the benefit hi I'm curious about how you're sharing the text on me so it looks like you have a central manager for holding an authoritative source can people push and pull text on me or is it just kind of this is the one taxonomy and then the second question is actually how do how does each site get a hold of that taxonomy and use it so we have a centralized distribution that people can actually walk in and create taxonomy it's like a control environment so it's not like everyone create terms there it's just editors that knows what they're doing actually and this is distributed to all the instances of the KK site factory the websites are basically pulling terms but each site factory has like taxonomies that are global they're taking from this centralized location and the local administrator can decide to exchange the data and put their local terms okay just to follow up then so the actual pulling of the taxonomy terms is that through services is that how you're doing that yes I would like to know do you always update the modules when there is a new release or just keep this website after you finish it it's when you're talking about the contrib modules we have two update phases if it's a security updates they're usually not breaking the consistency of the module then we do the upgrade immediately and if we do have core contrib modules like C2 spinels that have a completely new release that they're changing the number like putting a totally new version then we are very careful in releasing these modules because most probably they won't work in the same environment so we need to have a procedure by testing the environment so that it's work and then we upgrade the module for example the Panopoly is a distribution of the profile and like one or two modules have new release maybe for security issues do you update the whole Panopoly actually not using the Panopoly we're using some of the concepts and some of the modules but we're not using the distribution itself we are creating the things again true contrib modules but we are using a bit different approach but we are not using Panopoly and one-to-one so I think we're running short we've already run short on time so one more question one question oh I had two now I have to choose one okay I was curious to know how you managed synchronization and management of your code base running multiple instances of the Drupal code base across multiple machines we're using with our multi-site solution we're doing a cloud-based thing with a single code base and it seemed easier for us to manage and I'm wondering whether there are obvious advantages to doing it your way too that we haven't considered well we are using AIgear for the purpose the only difference is that by default where the AIgear use a webpack of servers it needs to to have the same drive the same code base mounted on all the web servers but we found out that this is a performance issue because web servers reading a code on a mount drive gives like very bad performance so instead what we do we exchange AIgear a bit so instead of depending on mountain drives we actually are seeing the code to all the web servers so let's end it here and before we go out the door I'll just do a small advertisement for there's going to be a bird on a feather on AIgear right after this so go see that if you're interested in AIgear or you can thank you you