 All right then, it's time to get started. So I'll stand up for a second and not get blinded by the projector. Pardon me. So this is the migrating to Drupal session. This is, by the way, a non-technical session designed specifically to get your business people and your technical people aligned properly. And I'll give you a little bit of introduction. Nicole and I will introduce ourselves in a minute. We used to both do separate presentations and Nicole talked from the project management side. I talked from the developer side and we were at a couple of different events and we would present sort of next to each other or she would present the night present later in the day and we just decided, why don't we merge them together so that everyone can be on the same page? So Nicole can introduce herself. So yes, I am one of the vice presidents at Phase Two Technology. My background has particularly maybe the last six years been at the enterprise level in Drupal. I've worked on, let's see, my very first kind of large scale from the United States was Lifetime TV and it was kind of the first major site that was doing like 30 million page views a day. So at that time it was can Drupal in fact scale to that level. So I was kind of involved in some of the early issues of scalability around Drupal. And as I said, I've been in the web management professional since 1999. So even before Drupal, I was in the first bust, all of that. So that's me. I'm Ken Rekard. I'm the director of development and professional services now at Palantir.net, which is a Chicago-based Drupal firm. I was the co-author of Drupal 7 Module Development, which is actually a fairly advanced technical book. I've been working on the web since 1998. I'm an early enterprise adopter as well. I'm responsible for putting the first newspaper websites on Drupal back in 2005. So with that said, we're gonna get started and we're gonna take this in a, what did we say? Who, what? Why? Yeah, our slides give the order. So essentially one of the biggest things that you have to deal with in deciding or when an organization has made a decision to migrate and is the why. And it's important to establish the why because as I said, is that, as I said, is that one of the biggest reasons is due to organizational change. And what an organization is typically trying to deal with are time to market issues, integration issues, and political issues. And once you understand those organizational reasons, it actually makes establishing the rest of the project and the planning and as Ken will start to go through, much easier. And also another major reason is our budget issues. So what kinds of issues that once you've decided to migrate is what kinds of issues does Drupal kind of uniquely solve? What are those reasons that people choose Drupal? One of the biggest issues of course is around budget and licensing. And licensing, I've had projects where they've saved a few thousand dollars on licensing and I've been on projects that say they have a major oracle back in and they save several million dollars in licensing. So the larger enterprise, at least the clients that I'm working with are often moving to Drupal because they are dealing with major budget pushes by their organizations to save money. So licensing is one of the biggest issues. The second reason why I find organizations moving to Drupal is they have platform integration issues, so they have a lot of technology they're trying to package into their web solutions and they don't quite fit together they don't play nicely together in the sandbox. So people often choose Drupal because they're trying to solve problems of integration. And then the third one is people choose Drupal because I used to work in IT and one of the biggest problems I had in IT were the technology wars, the vendor lock-in wars. And so I had my Cisco guys, you know like the religious Cisco guys with my Microsoft guys and I had my Unix, you know, it was just warfare around kind of the different vendors that we would use and that I had people who were locked into them. One of the great things about choosing Drupal is you don't necessarily have those types of vendor lock-in issues. You may have favorite vendors that you're working with but it's Drupal. I mean, so essentially you're able to kind of pick and choose who your vendors are by and large. They may do it slightly different but it is in fact the same framework. So now that you've chosen Drupal and one of the things about Drupal though is it doesn't need a little bit of help when you decide to jump into that migration is that oftentimes there is a language problem when you're starting the migration project. And when I say language problem is you get tripped up on words and one of the first things that I often hear is there is a module for that. And this may seem like a simple thing as you're going into a Drupal migration project but it implies that your site is practically built. And unfortunately the truth is is that you know that your site is not practically built and that oftentimes that there's a lot of customization and flexibility that folks want in their migration. And that as you know what comes along in the language is that I want a flexible and customized CMS. The implication is that oh, it also is gonna behave like shrink-wrapped software. It's gonna have all the documentation and it's not gonna have any bugs. That's not necessarily the case. Anytime you introduce customization into any kind of solution it's gonna represent a lot of other issues that you are not necessarily planning on. Another big issue around the language issue is that the designs are final. I don't know how many of you worked, even to not in Drupal, the constant designer tweaking. And so one of the funniest things I thought that was a funny slide because I did a little bit of history on how the Mona Lisa was painted. And it was never really considered finished by Lena. It was never really considered finished even after seven years. And that oftentimes the designers are constantly tweaking. They're never quite happy. And the problem is that the implication is is that they feel that those small little design tweaks don't have any impact on what developers are gonna do or the complexity of the project. And that those little design tweaks do in fact change things, sometimes major things and it can change budget, it can change how long something's being implemented. I had one project one year and in fact it was a lifetime one where we actually had at one point the designers and developers were sharing the same drive where they were sharing designs. The problem is every time developers went to go pull designs, there was a new version in there. So ultimately I had to take the designs, pull it on a secure server. So every time designers sent me designs, I'd put it on the secure servers so that's where the developers pulled from. And the designers would be like, well, why didn't my changes make it there? And I was like, because ultimately when you told me it was final, I moved the designs into the secure server area. So yes, important issue, our designs are quite never final. Yeah, I would point out the first Drupal project I did as a consultant, I had to write a report to the client saying, well, between the wireframe stage and the final design stage, your design firm just added 400 hours worth of work. Yeah. So, and it's just, it's really important to be very diligent on the design side. The other kind of language issue, and it's related to why people move to Drupal, is Drupal is free. Yes, technically Drupal is free. There are no licensing costs. But for those of you who may have just started embarking on this endeavor of migrating, you've probably quickly learned that there are significant costs around the talent and around hosting and a number of other areas. So although you may be saving on licensing, you may be accruing a bunch of cost in terms of maintaining and securing your talent as well as configuring and setting up your hosting environment. And last but not least in the language area, it is really important to be very honest about a lot of these concepts. So like in the original one, I wasn't sure I was gonna put this slide in, is it's kind of like the Drupal out of the box. I had the house in the box. It's similar to there's a module for that, is you have to be really very honest with yourself. Is it, are you going to be truly happy with Drupal out of the box? Are you going to be truly happy with where your business requirements are and in fact how they match up against what typically Drupal offers? So in order to make your migration successful and establishing your why is getting very honest so that your migration is successful. Yeah, I would throw to that as we go into the what. The first Drupal projects we did for newspaper business is essentially we would say, it's gonna do 70% of what you want right now, very, very quickly. And then the question for the business is, how badly do you want the other 30%? So when we talk about the what, we wanna talk about, so you're planning the migration. So what Nicole's been getting at is what are the broader goals of your business? What are you trying to do? What is the purpose of moving on to Drupal? Now, you're going to need technical expertise to make that happen. And the key in this next part, when we talk about the what, is how do you make sure that the technical people are doing the right work? And I would say there are three things when you're defining the goals and the targets for your migration that you have to keep in mind. And we're gonna look at each of them in turn. And to reiterate what Nicole had said, I want you to be very, very realistic about this portion. You can, it's very easy to do pie in the sky. I have big dreams, I have big ideas. I wanna have everything in the world. And has anyone ever seen an enterprise SAP portal, by the way? They're brutally nasty conglomerations of iframes because they try to stick 18 different systems into one screen and they just don't work. They're awful because they're unrealistic. They try to put all the business data right at your command, right at your fingertips. But the data's all enclosed, systems and doesn't integrate properly. So if you're not smart about how you approach what you're trying to accomplish, you're going to end up with a really nasty system. The first part of it that I would ask you to start thinking about is literally your business process. And the slide here has people on it for a reason. Your business process is about what your people do on a day-to-day basis with the tools that you've given them, right? Process is very important and can be easily overlooked. And in particular when you're dealing with moving on to Drupal, if you've had an existing CMS, you typically have editorial processes that have already been established. And if you don't capture what those are, you will very frequently get bitten by the, well, we just launched the Drupal site. Why doesn't it allow us to paste a Word document in? Oh, well, no one ever put that in the requirements. That's a classic. So capture your business processes, understand what they're supposed to be. The second aspect to that is the business logic. The business logic is almost literally, sorry, my computer, she can't touch it. Sorry. Business logic is the stupid stuff that machines do, right? The business process is almost entirely driven by people. The logic are the hard, fast rules that you teach your machines and you're gonna program into your code. Capturing the business logic tends to be, as an outside consultant, extremely painful because people don't really know how to write it down. They know that they have rules, but they're never codified or you get cases where I just had someone say, well, we have an e-commerce product and there's 3,000 different pricing models, okay? We'll document those for us. Well, that process alone is gonna take them three months and they don't wanna do it, but we'll get to that later. The third part of that is business data. So the business data is really the raw stuff that you're bringing across. And when we typically talk about migrations, we talk about data migrations. But again, I want to capture all three of those elements and when we get to the actual implementation, we'll talk about it. And what I really wanna say about it is this is actually hard work. The biggest hurdle, and I'm actually fighting this with a client right now, this is not work that your external consultants can do for you unless you're willing to pay them a lot of money and open up all of your business processes to them. You can do it, but it's very, very difficult and it can put a lot of strain on the relationship if you're working with external vendors if you're not willing to put in the time. The fact of the matter is your people can do this work faster and cheaper than I can. If you have to bring me in to learn how everything about your business, that's gonna take a long, long time. Would you agree with that? Oh, definitely. So what we're gonna talk about are things that are gonna make it easier for you, make it a little more fun. I love this little slide. One of the things, and I think this is a theme that runs throughout the DrupalCon, so I think it's important to have this slide here, a culture of transparency. This goes, how many of you were at the keynote this morning, by the way? So what Anku was saying, I think applies to the kind of business logic we're discussing. If you really wanna capture exactly how to get the work done properly, you have to be very, very open and very, very honest about what's in there. And I use Drupal here as an example because communities like ours and software projects like ours will fail if there's no transparency. If people are making decisions in back rooms and when the developers don't know why or the marketers don't know why, it's gonna so distrust, it's going to lead to schism, it's gonna lead to a forked project. So these ideas that are very, very, almost, I wanted to say sacred in the open source community, the open sharing of information, a clear definition of roles, an agreement on key issues, and a sense of shared responsibility is really, really vital. So go ahead. I just wanted to add to that what's hard about working on a project and it really depends on the project. So although you may have established the why and that it may have matched kind of your budget decisions to move to Drupal or your integration issues, your organization that you're working with may not have bought into this culture of transparency, which makes the migration project even harder. So I wanted to add that. And in many cases, my argument would actually be the introduction of Drupal and open source technologies to give you a great opportunity to start this conversation, which can be very, very transformative in an organization. It can be very, very difficult, but I recommend it highly. At the very least, your team needs to be transparent. I love this little black box slide. I use it a lot. And I modified it for my German audience. One of the reasons that I love Drupal is I used to work with a bunch of C and Oracle engineers, and if I needed anything done, I might give them some inputs. I'd tell them what my business process was, what my logic was, and then they went off and did something in a black box. And the black box is where the black sheep hide. Where the magic happens. Where the magic happens. Right, but who knows what's coming out of the black box. You might get schnitzel. You might get a sausage, but you don't want that. You want a new website. So avoiding this, the key to that is open communication. Open and transparent communication by all the stakeholders on the project. The last part I wanna talk about when we're getting to this pre-planning from a technical standpoint is very, very rarely are you gonna have an open green field in front of you, right? Green fields are great. Green fields mean I can do whatever I want. There are no limits. There are no prior expectations. But you have to plan because you're almost, again, if you're dealing with a migration, you already have systems in place and you have to know where they are and how they work. And to do that, you have to ask the right questions. So I wanna hit on a couple of key questions when you're talking about what does it look like if you're doing a migration project? And the first of those is, are you doing an actual migration or are you doing an ongoing integration? And this is a whole separate topic that we could spend an entire session on. But a migration is a single use. I am taking data out of system A, typically something like my old Oracle system, and I'm putting it into Drupal and I'm never gonna touch Oracle again, which is how you save a lot of money. And integration is a repeated process, which says, no, I still need my Oracle data. I have people updating that frequently through a web app or some other legacy system that I can't get rid of or don't wanna get rid of. And I need to bring that over all the time. So you have to make that plan up front. I actually have a client right now too. It happened twice. We signed on to do migrations and they want continuous integrations and their budget went. Question number two is, is your data any good? This is what I call the hierarchy of pain. My SQL and PostgreSQL data for Drupal migrations is great. It's perfect. It's easy, it's native. Drupal knows how to read it, no problem. Any other form of SQL data is usually pretty good, although it can be a little tough to get data out of Oracle. That's not a big deal. XML data is great. I wanted a migration. It was 14,000 XML documents, not a problem. HTML is a little tricky if you actually are in the position where you need to migrate raw HTML. You can find me later, I'll give you some tips. And then in some cases people just have a bunch of old Word documents lying around that they wanna put in the CMS. And what I would say is, you wanna have a plan B. Everyone would like to have a nice automated process, a big shiny gleaming Porsche that goes down the highway and brings all your data across. But if you literally have a bunch of old Word documents, you're basically stuck with a broken down old pickup truck that won't do anything. And you might actually be better off hiring an army of interns to just manually enter all your data. We had a case where that just happened and in hindsight we should have hired the interns. The last two bits that you wanna think about is how do you review when you're done with a migration? This particularly goes to the business data because if you're doing an automated data migration, computers are not perfect, computers are in fact quite stupid, they can only do what you want. And we'll talk a little bit later about what do you do when the data's not what you expect? And really I would say, what are your editors gonna be doing, your marketing people, the folks who are responsible for your web platform? How are they gonna interact with Drupal in that sort of pre-launch phase after we've transferred everything over? So have a plan for that review upfront, right? No, those are the three major questions there. How will editors interact with it? What's the workflow going to be and how do you handle the updates? Very important questions. So after that you're ready to talk process, right? Who's actually gonna do the data migration? How often is it gonna happen? When is it gonna be considered complete? And from a project management perspective, what is dependent on it? It's that critical path of planning. If you need to be able to do editing, and sometimes this is very frequent, you have to do content proliferation on top of the data migration. And you can't start the content proliferation or the editing process of making the site ready to go live until the migration is done. So finding that critical path becomes very, very important and is really the job of your team. So once you've answered the why and the what, it becomes the who. And the reason why who is important is because people are, I could say 99% of the time, your biggest risk in a migration project and it's not the technology. From experience, I can tell you that. And the reason why is because actually deciding to do a migration is to do a Drupal migration and introduces quite a bit of change into the organization. And remember, the reason why you're going is that you may have budgetary issues, you may have time to market issues and those types of issues often kind of start to impact people and their job descriptions start to change. It starts to change who people are reporting to. And unfortunately, it often leads to people losing their jobs because they become obsolete. They may be working on a system that they can't necessarily make the skip over to Drupal or they don't wanna make the skip or they don't have the skills to do it. So the type of change that going in a migration introduces quite a bit of change on the people who are having to execute it or not execute it. The other thing that happens and why people are so important in it is it's the speed of change. And so if you go too slow to accommodate all the nervousness and all of the issues of the individuals involved, you may not make your targets for budget. You may want to avoid paying your licensing fees for the next year. So it's gonna make you rush a little bit faster than you should be. So rushing or going too slow can impact how people actually respond to the project. Your ideal situation is to find that nice middle road where you keep your people kind of psychologically happy and sane but at the same time that you're meeting the business drivers of why you've actually decided to do the migration in the first place. Right, but that's tough because the first migration I ever did. Yeah, it's never happened that way by the way. It had a hard deadline and no one ever said it but I am convinced that if they missed the deadline they had to pay about a quarter million dollars in licensing fees. Right. This is not something that's easily solved. It is quite a bit of massaging and one of the things, how I'm kind of recommending solving it is the next set of slides. It's not, you cannot necessarily control the speed of change or the type of change but you can work with the types of personalities that start to emerge out of the actual migration. And the first personality that actually emerges is your evangelist. The evangelist is the person who sold Drupal. They think this is, you know, this is the good idea. This is the technology that should be used and they're gonna sell it. They're gonna sell it to the organization and they're gonna say this is what we should be doing and this is why. The caveat is depending on how they've sold Drupal you may actually find yourself having to deal with the language issues that I talked about earlier. You know what, oftentimes when someone's selling Drupal to the organization they're saying oh you're gonna get all of this out of the box functionality. Oh there's a module for that. So they're selling Drupal but they're using language that's not clearly defined. So that is one of the hardest things about your evangelist is very important to the migration but you have to kind of be there to mediate the expectations that they're setting and clear up the language problems that are being introduced. The second type that always emerges is I call them the passive aggressive type and the passive aggressive is usually a fairly indirect person. They're not that person who's saying no I don't wanna do this, at least in public. They're that person who's sending the kind of sly emails, taking shots at certain people in the project and so that passive aggressive person although as I said be very quiet they can undermine your migration by just little simple things and how you have to deal with this passive aggressive personality in a migration project is you have to call them out unfortunately and then I know a lot of folks don't like to be confrontational it doesn't have to be an aggressive confrontation but you do need to call them out and have them understand that even the little things that they're doing is in fact undermining your efforts. The third person is not someone who's gonna send the sly little emails and do it behind your back. They're actually gonna be in the meetings openly telling you this is a terribly bad idea you should not be moving to Drupal I don't like the way you're doing this and they're openly hostile to your actual efforts. The unfortunate part is this person is usually motivated by fear and uncertainty it may be the type of change that's happened the change issues and so it could be that they feel the underlying issues that they may be losing their job they're reporting to someone that they don't wanna report to so usually your openly hostile person is in fact being driven by fear and how you deal with this person is you try to find a responsibility for them you try to give them a sense of ownership the problem is if you can't find that and it sometimes happens you have to get them out of your project you may not be able to fire them but you're gonna have to make a case to get them out because an openly hostile person it is so painful to do a migration and you have someone sitting in your meetings just confronting you every minute of the day telling you this is a bad idea it starts to take an impact on the morale of the group. The next person I call the nodal and the nodal and again no offense to any or you know just I usually call these my Java developers and so you and again no offense I've done a number of migration projects where we'll go into the organization or been my organization and they may have had a group of technical folks who are very experienced they may be very, very experienced on the web but they don't know Drupal and so you have a group of people who are telling you this is not the right way to do it and they're able to actually engage you technically but there are some differences there are some significant differences and that the way you would do a Java project is very different than the way you would do a Drupal project and so I call these the nodals and the way that you usually deal with nodals and I'm telling you this works 95% of the time is you have to train them just because someone is technical does not mean that they know Drupal and I have been engaged in projects where we're bringing Drupal in and they expect those other technical folks who don't know Drupal to just pick it up and implement and it just doesn't work that way they usually become very unhappy and dissatisfied but when training is actually introduced and technical training at that they are like, oh, okay so they feel a lot more empowered in the project so training is critical to this in terms of dealing with this type of person the other person and Ken talks about them and again there's a synonymous relationship between the apathetic personality type and the editorial staff your editorial staff essentially or your apathetic individuals will actually sit in the project on the sidelines for months and have nothing to say they never have any input on the front side never, nothing and you may even go ask them I've had projects where you have the business analysts the designers go in meet with all the editorial staff and what do you think oh, that's fine, love it, that's fine as soon as you've finished building something you get that administrative system set up and you roll it out they're horrified and then quickly they revert from the apathetic person to the openly hostile or a passive aggressive member so just how you, sorry, it's all right and how you deal with this person what is very important is in the upfront as Ken talked about is making sure you get their business process and getting their business process means extracting it by sitting them in front of a computer and showing them demos showing them other systems that you bought kind of walking them through what the interfaces may actually look like and experience that's one big deal, big way to impact it and the other is after each build every component that you put through don't ever think it's too soon to introduce something to the editorial staff they may be apathetic about it but at least you're kind of dragging them in front of the system and showing them pieces at a time so that's also very important now I can move on the next this is kind of an interesting member and this person actually originates in some of the larger projects and they're never like the CTO of the organization or the CIO it's just some kind of amazing like production manager who kind of arises out of the weeds who becomes the champion of the actual implementation they become that central person that the developers go well what do you mean by this and what happened to that part of the budget I call this person the protector and it's never usually someone in a significantly lofty leadership role they're just someone who can work hard and is committed to getting the project done and what do you do with this member and how do you deal with them and normally how I deal with this member is I take them out for drinks and dinner so you're smoozing with this person because they're the glue who is actually making this project get done and they are a magical individual so and you will know and just don't assume that they are a lofty titled person and then the last person that you're gonna encounter and this is kind of and this often is your well I won't say often it should be your project manager and this is the person who is going to understand the strategy, the language, the people the technology they're gonna work with your lead develop they are gonna be the person who pulls everything together they're not necessarily the protector they can execute on the pieces of all the pie and they can put everything together they are the only problem about this particular member of the project is they're not always well liked and the reason why they're not well liked is cause they have to be somewhat hard and make some decisions and push things through that people are not always gonna appreciate and that the main thing that you have to do with this member if you are kind of in somewhat of control of a project is to give them authority over the project you don't ever want your project manager to not have enough authority to actually fulfill this role in a project so last but not least is that people are your key assets you know any migration project they are they are so critical to you the success of your migration efforts do not underestimate kind of the roles that I talked about and how that speed and types of change impact how your folks will actually behave in a migration and as we go into the how and this is the execution phase when we actually get developers involved I would say it's funny because the most successful migration I ever had I was on site working with it we're running into some really nasty problems and finally for the first time in there were two people on the project who had worked in the same building for 20 years and had never really spoken one of them was an Oracle DBA who was responsible for storing all the data the other was the head librarian who was responsible for putting the data in the system but they had never talked about why the system does what it does and they had to sit together for about three hours and solve a really thorny technical problem for me and it was fascinating because they both went from sort of passive aggressive to protector very very quickly when they finally realized that they were actually in alignment and if they succeeded they were going to be rewarded that was great so in the how the first thing is actually to identify the team who's going to be involved in your migration process and part of it is is it an internal team or an external team I've been part of both I've been part of mixed teams also I just rolled off a very successful migration where my job was to coach the internal staff on how to do the technical implementation that worked really really well there are pros and cons to both of both effects or both approaches but you do need to be aware and these are the sort of roles that you'll need on the team you'll need someone who's a data architect which means someone who really understands all the data that goes to your system you'll need some programmers and you may need more than one of each of these roles you'll need a business analyst you'll need an editor you'll need a project manager and then you'll have numerous stakeholders and those people will all fall into the different personalities that Nicole has gone through can I add one thing to that slide now just understanding the kind of size of certain projects is that you will have people who wear two or three of those hats so it doesn't mean that there is a minimum of what is that seven roles seven people on a project you may actually only have two or three but you still have to fulfill those roles right but on the last so for example on the last migration that I did it was an external migration well even if you're hiring external consultants your internal team is going to have to do some work this is the big thing so on the last migration that I had to do personally I was both the data architect and the programmer and I was essentially having to beg the client to do business analysis it's like I need a business analyst because I don't understand anything about your data and there's no way for me to understand anything about your data because there's too much of it and it's all very typically domain specific which means very if you're an architecture firm it's all about architecture I don't know anything about architecture I know newspapers so then now you can write code you've got a team in place you can get started you just no no no no I have a guy right now who works for me he writes about he maintains about 50 modules on Drupal.org he loves to write code he's awesome and he's ready to do his first data migration and I won't let him because they haven't finished doing the analysis yet and I've actually done enough of these that I've come up with a decent format and I've cribbed it onto the next couple of slides this is all right now we're talking about business data bringing it across and how does that get into your Drupal site and these techniques will work even if you're not migrating on to Drupal if you're here doing due diligence and you decide after this conference that you're not going to go to Drupal you might go to Joomla you still need to go through this process so the first thing you have to do is map your data sources and these are basically very very simple worksheets that you should put together that everyone on the team can read and agree to now I don't have a lot of room on these slides but the notes are designed to be human readable the rest of it is technical information for your software developers so you might, this is our hypothetical database of stories from a newspaper system and we're pulling these out of the oracle so this is a map that says hey here's the oracle data that you're gonna see when you start doing this migration there's a title, there's an abstract, there's a last updated field and then there's some topics and the type is really just technical stuff you don't need to worry about it right now this import yes or no this is where the business analyst comes in it's like hey, do we care about this data point well last updated in this version is irrelevant the reason it's irrelevant is we're just gonna start over we're just gonna we know that everything's gonna get updated after we do the migration so the fact that an editor might have touched this nine months ago is irrelevant so let's just drop that and the notes are also interesting because the notes will help you capture information about how things relate to other where it says topics from topic list the implication here is you're gonna write these data source maps for every type of data you're bringing across and very typically you're bringing across multiple data types you might be bringing across some users you might be bringing across some stories and here I'm assuming you're gonna bring across some topics so a categorization or a taxonomy system and what I'm making a note here is hey developers note that when you hit this topic it refers to this other piece of data that we're gonna talk about in another slide well we're not gonna talk about it on this slide that's the full technical talk the second part to that is once and that part has to come first and I actually have a client right now who's really really angry at us because we asked them to do this because they handed us like 18 data tables and we don't know what they are now to be honest they can pay us an awful lot of money for us to guess and what they're gonna get if they make us guess is we're gonna hand them this document and then make them walk us through it to make sure we haven't screwed anything up something else in your phone so after you've done that your data architect can then map the data from the external system to the Drupal system Drupal works essentially in a series of collected objects that they call entities and so every piece of business data you have will map for the most part will map to an entity type and so here we've created a story entity type and it has fields, data storage that map to our external data and all I'm doing in this case is mapping the sources from the first list to the targets on the second list this is your roadmap for your development team because you're gonna be telling them I need you to create this type of entity and I need you to rip this data from in this format to the new system right this is written by whoever's doing the actual implementation then you review it with the client particularly the protector somebody who knows the system who knows what are the targets we're trying to hit and have we mapped the data appropriately and there are two slides I think that help with that like I said before you very typically have relationships between those entities so this is from a migration that I did where we would import a story but the story might be part of a collection of stories the story might also be tagged by topics and the story might also have an attached contributor and so knowing how those things interrelate becomes very, very important because quite frequently, well in fact always when you're doing a data migration in an automated way the order in which you do something matters so in this case I can't import stories until I import contributors because I have to attach the contributor information to the story information and that becomes really, really essential so getting a good understanding of those data relationships is vital and I found this slide yesterday when I was looking and I'm gonna actually stand up and explain it when I talked about business logic earlier this is a great example of business logic this is actually from the first migration I ever did this is foreignaffairs.com which is the leading political science publication in the United States essentially all US foreign policy in the last 60 years started in foreign affairs that's what they do and they have a very, very specific publishing rule set which basically says someone looks for an article online is it under copyright? Does foreign affairs hold the copyright to it? If yes, we go in one direction if no, we display a message that says we're sorry the article you're looking for is not under copyright we can't display it to you on the web here's the issue you can go look for in the library but is it marked as freely viewable because they're a pay service if it's free, we can display it if the user's authenticated against their registration and subscription system, we can display it they have a concept of user licenses which means that a university can buy a license and we know if you come from the university network you're treated like a subscriber which was a fascinating piece of code to write but this one in particular is an idea of business logic translated into code so capturing your business logic and being able to, everyone on the project not everyone, but the business owner the stakeholders have to be able to explain this to the developers in a rational way typically what happens is the developers will map this out and then say is this correct and you'll iterate through it until you get it right I would just add and I've had a number of projects where say I didn't early on collect these artifacts and when I didn't collect them and as the non-technical person I found developers having to go through multiple iterations that they shouldn't have had to go through to get the data correct and then again it could have been easily avoided had I had gone through these steps right and this is the argument I'm having with my client right now is they don't think they should have to do this they see it as a technical exercise right and they don't understand the value they don't understand how it's gonna save time in the long run is the weird part the next thing you actually have to do as a developer and it's helpful if you can get someone who knows the data but you're going to have to track what are called exceptions exception handling is the least fun part of programming I'll tell you flat out and exceptions are well in our Oracle data set everything should have a title but occasionally we might have goofed and there might not be a title okay well what do I do in that case skip the record and flag it flag it for review by the editorial staff but don't bother importing it what if the story title is in all caps I actually had this one on foreign affairs every once in a while they have a title in all capital letters and that might be correct but it's probably not and that's something you could correct programmatically but it really takes a human editor to know and so when we talk about editorial tools here in a little bit that's something you can flag for editorial review and typically when I do data migrations I'll have additional fields in my story concept in my story object in Drupal that allows me to tag specific exceptions so that when editors are going through to review things they can say okay find me everything that has this flag is having a bad title and they can fix all of those at once or find me everything where there's complex HTML embedded in my story right this last one is very important here for the Germans UTF-A Drupal does everything in UTF-A which is the language standard the coding standard for supporting things like the double S in Strasse non-UTF systems can explode if you don't properly encode the double S so you literally have to flag it in the code hey what do I do if I hit non-UTF-A data the worst case for this by the way is if you get a lot of Microsoft data from Microsoft Word which uses its own special encoding and causes us to cry so as you're going along and some of this you will do upfront and I will say this is actually Nicole's area of expertise if you want to talk about risk management you should ask during the question and answer or a little bit later but you should be tracking risks as a developer you should be reporting to the project manager hey I'm starting to see some really bad data here now when we do risk tracking typically there's a matrix of how important is it how badly should we freak out and essentially we have here the columns what's the chance that this happens and if it does happen what's the impact of it and then what you have to do is come up with a mitigation strategy so okay what do you do if you have bad data now we're trusting that we're gonna have good so there's low chance that this is actually gonna occur but if it does occur it's totally critical because it halts us on the critical path of the project the only mitigation strategy at that point is to delay until the data can be corrected right and I just had a project like this where it turns out that all the client was able to give us was 65 separate comma separated values list which were basically just exports from Excel we should have just let interns do it because we didn't have a smart way to analyze the risk of what was happening by the time we realized how bad the impact was it was too late to fix it so ideally you will set some risks up during the data analysis phase and you'll be able to say all right team especially to the chief these are the things I'm worried about so let's keep an eye on them right missing documentation I flagged as high here that's no one knows what this data is we're just trying to making it up frequently as a contractor when I come in you'll find like we're replacing custom systems that were built by someone 15 years ago who hasn't worked there for six years and no one knows how it works which is not a lot of fun so once you've done this pre planning then there's the go no go questions like okay are we ready to go I actually think you're ready to go once you have your targets mapped I think you can move forward with you can move forward without doing all the exception handling as long as you know that you have to do it I think you can move forward without doing risk mitigation as long as you know that you have to do it but that is something that the team needs to decide because if it is a risk any time you're moving forward without having all that information you're opening the project up to the risk of failure so it is a team decision to go once the team says go then you can write code and this is actually what I'm doing on the migration project right now is I'm the gatekeeper telling the programmer when it's okay to move forward and he's not moving forward until I tell him I'm gonna skip very quickly through there are a lot of Drupal tools if you're reading through the program or see or hear people talking about any of these things these are all useful things for you in your data migration the migrate module is sort of the gold standard right now feeds module can be used drush is very important Dvel is a developer's tool and views is great for editors and I did put in a little slide here to say hey these are some other sessions I think some of these may have already happened these are all technical sessions however about data migration data and business logic migration that are during Drupal and just a quick question just so I understand the audience I should have asked this at the very beginning how many of you are technical or just more business related so technical and then the business folks see this is why we decided to do this session together because that was what was always happening I would do a business session and I would have a group of technical folks and then I would never get technical enough and then so okay continue the first presentation I ever did that was any good was at a newspaper conference and it was how to manage developers in your newsroom and how developers should talk to managers because I was both a developer and a manager in the middle of a newsroom and it was about this very communications problem about making sure everyone knows what's going on so I'll just sort of throw out two very quick screenshots migration module is very very nice because when you're done doing all that data mapping and actually write the code it gives you a user interface to verify everything which is really really nice it also gives a lot of tools for automating the migration and scripting the import and duplicating the import so you can run it over and over again and test it over and over again which is really really nice and I would say views is great for building editorial tools and for the developers in there plan this out especially when you're doing exception handling when you run into things in the data that you're like this doesn't look right to me but I don't know what to do with it you can build in fields like I said to flag that so that then editors can go and search through the list of content and find things that they have to review very very helpful technique the guys at chapter three out in San Francisco actually pioneered this technique because I did a presentation with them in Boston the last thing that I say if I think is really really important about this is you have to track changes you do run into problems occasionally especially if you're doing an integration and not a one-time migration where your data targets the target mapping you did might not be accurate and you might find halfway through the project oh wait I have to make this small change to the way I'm handling the abstract field please please iterate your documents iterate your documentation excuse me so that the folks who come behind you know what you did once you did that you've got the code set up you're running it you have to test it you have to test it did the target objects get created correctly did fielded data come across correctly did the relationships between the data objects come across properly you have to check that all very very strictly and this is something that programmers can do and the programming team will want to do this before you unleash the editors on it the other thing you'll want to do before you unleash the editors on it is test it again can the process be repeated right yeah it works great on your system it works great on your test system does it work on the client system I just had a problem where we had a fatal non-replicable bug on the live web server and it turned out to be and I could not reproduce it on this very computer it turns out because I was using PHP 5.3 and the live web server using PHP 5.2 and there were significant differences that caused the process to fail on the live server yeah he just hung his head in shame yeah yep it happens and it can be performed by the team this is also very important we I mean we just went through a very delicate process of we have this tricky migration with 65 separate CSV files now they want to run it themselves so we have to train them up and how to do it when that's done then turn everything over to editor review and I've got to say you must schedule this step in human editors must review things because machines are dumb programmers are smart but they're dealing with dumb machines so you have to schedule time and editors remember they will remain apathetic until they see something so you have to introduce them early and so we didn't actually have one in our title but it we actually added this last slide it's it's not a big section it was essentially you couldn't actually answer when you're going to do a migration until you know the why until you know the what until you know the who and until you know the how so essentially once you know those four things you can then do the when you can then start answering things like schedule it's very painful it for someone who does you know migration projects or implementations or just projects in general when there's already a pre-set schedule and you don't even have the requirements of the plan so the when can only come after you've done all the for the first four steps we just had to do that on my the client was arguing we just had to extend the schedule because they couldn't they couldn't deliver the data in time to stay on schedule and last but not least and that we kind of thought about our themes in our presentation and the first one of any successful migration is don't rush there are many steps that have to be followed in order to get a successful migration if you skip any of them you are putting your project which is the second part of this is at risk and granted every project has risk and it is managing your risk and tracking your risk very important in a project also is planning planning and I actually it's planning and it's not just planning ahead it's just general solid planning and there is detailed technical planning the stuff that can went through is critical and why I'm so happy to be presenting with them today because those are artifacts that I have to on the project site must track in order to get a successful project as much as I don't want to have to push my clients to it and as much as I don't want to have to track it because it is very detailed it makes that all of the difference in the world and last but not least is communicating it is extremely important that you communicate that you saw how technical and how many steps are involved and all of the people issues if you are not communicating successfully in your project work it will not uh... succeed or it will actually be very stressful so without further ado we do have seven minutes for questions questions so I will repeat so the question was uh... do you have any suggestions for how to communicate or tools and how often how frequently that's actually an excellent question and I often write into our project briefs what the communication plan is actually formalize communication plans and communication plans first include where you putting the documentation what is where is where is your sales your central repository for documentation email is not a good central repository so the first is where you putting your stuff I don't care if it's a base camp a custom project management tool whatever the a shared server it should be clearly understood where all iterative and final documentation will go for the project and that includes project plans and it includes designs whatever it is the second part is is how often are you gonna talk to each other and so it depending on the size of the project so for my smaller endeavors I may meet twice a week thirty minutes which is sufficient enough and we go over you know the status that type of thing for larger kind of enterprise level I actually conduct a daily scrum or daily stand up and that there are fifteen minutes stand up I have the project the key project stakeholders who are involved in the date day to day decisions as well as the developers are on the call and and that is a fifteen minute stand up so that is part of the communication vehicle I've actually put another layer of communication in and that and and this is just up to you but I like the frequency of it is we use a lot of chat so we have like a lot of group chat where we are just in you know at each other all day I you have a quick question you don't have to wait for your daily scrum or you don't have to wait for that you know your weekly state you know your weekly status you don't have to wait for the email you basically if you have a question at immediate one you'll send out a chat so there are multiple levels of communication and I think a formalizing it so that everybody understands where they get to communicate is important because that is one of the stressful things if you have a question and it's an urgent question you don't know who to go to or when formalize it get it out there and you know make sure you follow it in your discipline with it so that would be my recommendation from the development side we we have dedicated IRC channels for our projects where we can just everyone who's working on it can hang out ask questions we did a huge project with Acquia and four other partners where we had a dedicated Skype hangout yeah just like some way to have an immediate way of asking questions so the question was can you deliver this in an incremental way or is it always one big go you know one big chunk you can do it incrementally I would say you have to plan that pretty carefully I have done I did one case where the project had such tight deadlines for deliverables that we had to do it in phases and we had to say okay during this phase we're gonna pull this data across and during the next phase we'll pull this data across that's a little easier to do I think with ongoing integration work then with actual data migration what I would say though on the incremental side related to that even in a migration that if if the owner of the content is comfortable with not having certain content make it safer launch that's I've I've often found that a client or the content owner has to usually wait to get all their content it's probably ninety percent of the time they don't get everything at one time so so on the incremental side of things it is that you could you see there's kind of anything technically in your incremental other than the content not all coming over at the same time you can you can increment the technical there's no reason not to do it incrementally you can do it you can do it either way yeah I would just say it's more about what what your end users could live with in terms of having the final data available you know and say your public but typically the problem we run into is that you have to get the core data in and then editors start building off of it and so very frequently we find ourselves in cases where no we have to bring everything over at once because it's on the critical path to getting the site but but that's again a business decision not necessarily a technical decision the tools and it's interesting the tools in Drupal are mature enough that it's not a technical barrier to do it in cases so we solve all the problems you're ever going to have that is very exciting but it also means that we get you to lunch early two minutes early yeah two minutes early thank you very much thank you