 All right, I think we can just get started. So this was a late swap in the program. So if you're looking for a user experience session, this is not that. Those folks couldn't make it today, so this is a totally different topic. We're talking about importing data from one Drupal site to another Drupal site. So typically it's going to be Drupal 5 or Drupal 6, but also Drupal 7 into typically a Drupal 7 site. And then towards the end, we will touch on what the plans might be for importing into Drupal 8. In general, we're talking about the major version upgrade path for Drupal and whether we can use this code instead of the traditional way of upgrading, which is update.php. I guess I should introduce myself, Moshe Weitzman, a longtime Drupal developer, about 10 years now. I'm the director of research and development for Acquia. Before that, I had a small consulting company called Curve, and we focused on data migrations into Drupal. We did lots of huge sites, migrated them onto Drupal. The economist, Martha Stewart, World Economic Forum, examiner.com, and my company was purchased by Acquia a year ago, and we continue to do data migrations at Acquia. So the project that I want to talk about is called Migrate D2D. It is a project up on Drupal.org. It's in Mike Ryan's sandbox. Mike is the maintainer of the Migrate module. This is an extension to Migrate that's specifically focused on migrating from Drupal into Drupal. Why do we go about building this Migrate D2D? When you think about Drupal, the way I think of it, if you're already on Drupal 6 or Drupal 7, there isn't a terribly large motivation to move to the next version of Drupal. You can build almost any website that you can dream of in Drupal 6. You can build it a little easier in Drupal 7, but if it's already built, you might as well stay where you are. However, after a few years, you often get unfriendly with your website. It's grown crafty. It's grown 100 content types, 20 of them you don't need anymore or 80 of them you don't need anymore. It's grown lots of filter formats you don't need. It's basically no longer serving the function that you thought that it would three years ago. That's completely understandable. Happens to all websites. What you really want to do after three years or five years is just build a new website that does what your organization needs it to do. That's when you do a major version upgrade. It really doesn't make a whole lot of sense otherwise, at least for non-trivial sites. You want to build it from scratch, build it just the way you want it and migrate your content into it. That's what Migrate D2D specializes in. We're going to just talk about how this could apply as the official upgrade path for Drupal going forward. Migrate D2D has as source plugins the knowledge of how to dive into Drupal 5, Drupal 6 and Drupal 7 sites and pull data out of them. It understands CCK for Drupal 5 and Drupal 6. It understands field API for Drupal 7, knows how to pull data out, knows how to pull even field configuration out and move that stuff into 7. For fields that are named the same in the Drupal 7 site versus the prior sites, it will do field mappings for you so the system knows how to move from one field to the next field or from the old field to the new field. It understands all of the weird data changes we've made over time between these different versions so the values and comment status, what was unpublished used to mean published and what's published now means unpublished. We fixed some of those things that UpdatePHP also handles for you. We handle them during this migration. A little bit more about Migrate D2D. This is its class hierarchy. The top most level is called a Drupal migration class. That gets extended by the major entities that you want to migrate so it gets extended by Drupal user migration, Drupal node migration and Drupal term migration and files and so forth. Each of those gets extended by version specific classes so Drupal term 7 migration, Drupal term 6 migration. There's the opportunity to handle version specific stuff in the class hierarchy and inherit anything that didn't change from version to version. Let's look at a little bit of code. This is excerpted from the Migrate D2D module and specifically a submodule in there called example D2D. Whenever you use Migrate D2D you create your own module that uses these classes. This thing is called example D2D and in the top line it's implementing hook flush caches which is a Drupal core hook that happens any time the system has a heart attack and you need to flush everything. We set up an array of common arguments. You'll see these happen in a few of these migration registrations. Let's talk about source connection. That's the first element in this array. What we are telling here, so the Migrate D2D module is something that gets installed in the Drupal 7 site, the destination site. What we're saying here is we're telling the Drupal 7 site how to get to the Drupal 5 or Drupal 6 or Drupal 7 site that is the source. Your source database connection is called legacy. Now we expect in your DB URL that you have actually told us where legacy, how to connect to legacy. Obviously for data migrations it's essential to be able to connect to the source data. This is how we describe that. We have to tell Migrate D2D what version of Drupal we're coming from. That's what the source version parameter is right below it. In the next array that we are merging in is just the description of a migration and a machine name for a migration. Here we say we're going to migrate the users from our D6 site to our destination Drupal 7 site and it has the migration name or the machine name of EX user, example user migration. The last line here is calling the register migration method on the migration class. This is a migrate module thing of how you create things called migrations which an example of a migration is take all my users from Drupal 6, move them to Drupal 7. Another migration would be take all my article nodes, move them to Drupal 7. We're registering a new migration here to move users. This is the big service that Migrate D2D is providing is the first parameter here Drupal user 6 migration. That's the class that's in charge of this migration. It's a class that Migrate D2D provided. We didn't have to do any work. We're using it as is and we pass in a couple parameters to register migration. The point is this is all it takes to have a new migration that Migrate module can work with and it will then move all your users from at least the user table part of users from five or six to seven. No custom code. If you guys have used Migrate before, you often have to create classes, a new class for users, a new class for articles and so forth and so on. In this case, we're able to just reuse the user migration and register it dynamically with some parameters and we're done. Similarly, here's how we would register the term migration. We have common arguments again and we merge in a description and a machine name. We just saw that and a couple of additional parameters that are expected by term migrations, namely what's the source vocabulary? That's vocabulary ID number two and the destination vocabulary is called categories in Drupal 7. That's all the information we had to provide to Migrate D2D. We're going to register this migration as an instance of Drupal term 6 migration and we're done after that. Migrate D2D really takes care of a lot of headaches for you. Here's a slightly more complex instance of using Migrate D2D. Again, merging in with the common arguments, a description and a machine name. Here we're migrating in blog notes from our old site to our Drupal 7 site. The source type, and this is a content type, is blog. The destination type has been renamed for whatever reason to custom blog type. Apparently that's something we wanted to do because now we have lots of blog types on our Drupal 7 site. The user migration that this depends on is called EX user. Clearly, if you're doing a migration of nodes, your users who own those nodes have to be in the system already. Migrate module has this notion of ordering your migrations so that users come before content and in general, you can have some control over how your migrations execute or the sort order of the migrations. It's a really good question. The question was, does your destination node type have to exist already or will migrate create it for you? The answer is it has to exist already right now. There may be other tools that will investigate your Drupal 5 site and do the same field API magic so that everything exists on the Drupal 7 site, but Migrate DTD is not yet doing that for you. In a sense, maybe that's helpful because the whole point of doing this was to rethink your content types. There's some value to the automation, but you wouldn't want to fully automate it. Here we do the same sort of thing. We call the register migration method. We pass it the Drupal node 6 migration class and the rest of the arguments and away it goes. It registers this migration with Migrate. We can now run it from the Drush command line or from the GUI. We're quickly developing a migration. We've done users and terms and now we know how to do nodes. We really haven't had to write mappings and other interesting things you have to do with Migrate typically. Your CCK implementation might have used field names that are not used on Drupal 7. You want to change them for whatever reason. Here's an example of letting Migrate know about those changes between your old site and your Drupal 7 site. Once again, where are we here? We have a class called EX article migration that extends Drupal 6 node migration or Drupal node 6 migration. We basically said that articles are a little bit special here and so we had to write a little bit of code. The little bit of code was called the add field mapping, right in the middle here, add field mapping method, and we're saying that the old thing that was called field publish date is now called field published. Similarly, there's a little mapping here from the old taxonomy system into the term reference field. So, you at times have to write a little bit of code for your different content types if you've done lots of changes, but it's rather straightforward. As is typical with Migrate, you will use a method called prepare row in order to munch the data on its way in. So I gave you two examples here. The field underscore address column in row in the new site is a combination of street and city. Basically that's the Drupal 7 way of doing addresses, let's just assume that. Let's say that in your old site your titles were mixed, some lower case, some upper case, and it was just annoying to fix that on the display layer. You wanted to fix it in the data once and for all. Here we have an example of taking the title field, running the UC first function, which capitalizes the first letter, and now in your Drupal 7 site you always have titles that are capitalized in the first letter. So an easy place to do data cleanup if you desire between if you're going to go through the effort, doing a major version upgrade, let's also kind of clean up your data along the way. There's a question in the back there. It really looks like this. Yeah. So okay, if you are migrating from a multi-language site, it may not be the symbol. Yeah. There's a translation layer that Migrate does to set up the node object so that it has the multiple, multi-value and the language thing in there. Basically you can say that these are French nodes and stuff like that. So Migrate's destination plugins are handling some of that complexity so you don't have to think about it here. One more feature that is in kind of a really early development stage is a GUI for writing this stuff that it will write it for you, essentially, and it will interrogate your old site and ask you, do you want to create this field in the new system, and it basically specs out the migration for you. It might be coming soon. It might be coming never. You know, we'll have to see. It would happen sooner if people got interested in it and helped work on it. Right here. Yeah. Yeah, so they were nodes in Drupal 6, custom entities in 7. It depends on how custom they are in 7. I'm not quite sure. I think it's possible you have to implement your own KLS. I mean, I think for the source side, you can just do it as a node migration. Right? And on the destination side, we may need some custom support in order to save them into your custom entity. There is some integration on the destination side with entity API, which gives a save method for all entities. So maybe it will just work with that destination, but I'm not sure. Okay. Okay. So migrate module, migrate D2D, our intention, Mike and I's intention is to have that ready when Drupal 8 comes out. Okay. So for sites that want to migrate to Drupal 8 right away, we intend to be there for you. There's another goal here, which is that, you know, I've been contributing for a long time and the upgrade path between major versions has become completely unsustainable in my mind. If you look at all the update functions that all the modules carry, they're getting crazier and crazier. And we're making, you know, really difficult rules around you can't use any of Drupal's API within these update functions because we don't know how much of Drupal's available basically when you're updating. You're in this horrid state where your code is all Drupal 8, your data is all Drupal 7 and we're like trying to make it work for you. And we don't have any of Drupal 7's APIs because the code's gone. So it's truly a difficult spot that we put ourselves in with the update.php. So as developers, I get that people might like update.php as users because you press a button and it fixes up your website. But I don't think that developers can actually deliver on that promise for another version of Drupal, even though we might try for 8. I don't think it's a good idea, but at least I'm hoping that for 8 or 9 we give up on that dream and make this the official upgrade path for major versions. Update.php is cool. It can certainly stay for minor version upgrades, the 7.14 to 7.15 and all that kind of stuff. I wouldn't propose to change anything about that. But I think for major versions, the migrate module and migrate D2D approach is solid. You can keep working on your upgrade path, roll forward, roll back, have your Drupal 8 site and your Drupal 6 or 7 site running independently and just move data along every 10 minutes so that people can see what your upcoming website is going to look like with all the latest data and get very comfortable with it before you flip the DNS and really go live on your new site. It's quite a bit different than the other way, which is probably take a database backup, change all the code, run update.php, pray, doesn't work, restore database, fix something, run it, restore database. It's just a terrible cycle. Migrate is specifically designed to make that cycle as painless as possible. There's a question right here. I don't mean to be pessimistic. Back to what you were talking about, your new sites in the freedom, maybe an example, Drupal 5 are using e-commerce and Drupal 6. I decided to switch to e-commerce because I heard all these great things about it. And you go to these major architecture. I'm not sure if migrate by itself is going to handle that. It's still going to need to make this process really about making decisions. And user, site builder, new decisions that I'm not sure I'm fortunate on. Well, it's definitely about decisions. Decisions are made in meeting rooms with people. Like, migrate definitely won't help with that problem. There are definitely organizational challenges around major changes to your website. And it can't handle that. I'm not sure if that's what you're getting at, specifically. Right. Well, one of the great benefits of migrate is that you have the option of switching platforms from commerce to Drupal commerce and Uberkart. If you don't have a good tool for doing data migrations, the people who say, just leave it alone are going to win. Because you don't have a good story for the other side. But with migrate, you have sort of the technical part of that discussion. The data migration part of that discussion is solved. So you now have technical merits and you have to show why Drupal commerce is better than Uberkart and so forth. It's not automatic. It's absolutely not automatic. Unless you're a one-person shop, it's not automatic. Let me go all the way in the back. Yep. Yeah, there's a great module that runs on top of migrate module. It's called commerce migrate that implements that migration. Okay, right here. Yeah, exactly. I mean, we're going to have to think about upgrade for configuration also, because configuration is totally changing in Drupal 8. And for now, we're kind of writing out those files and update functions, I believe. But it's kind of an open question where that ends up. Over here? Yeah, so the question was, what's the state of migrate for Drupal 8? And the state is we have not started working on it. So that's something that I think we'll work on, let's say, when it's like an alpha. When Drupal 8 goes into alpha, I think we'll probably start working on that. Sure. Yes, you could do that. You know, you have different priorities in that use case. You certainly wouldn't want like your node IDs changing, user IDs changing, all of that kind of stuff. And we don't necessarily solve all that for you here. But we solve a lot of things for you there. But I wouldn't really start trying to do what deploy module does here. You'll have to think more with this than you do with deploy, which kind of automatically figures out the dependencies for each of your pushes and so forth. Right here? I think that Drupal 8 is going to have to solve that. So yes, but I don't quite know exactly where that's at. I think we have to talk to the config folks what they're thinking there. Question? Well, this is an alternate way to preserve content across major versions of Drupal. So yeah, we're ripping out one thing and replacing it with another thing. And the, is it squared with the community? Is exactly what core conversations are all about. So, you know, I am trying to get some momentum behind this and get some help and get some, you know, leadership and from people, you know, here. So yeah, we're in the early days of this back here. Yeah, I mean, that's a little bit weird to me too. Basically, you just have to find a point in Drupal when you could register these migrations. And that's as good as any other. You don't want to do it on every page request. So we just do it there. You know, you could do it when you install modules or something like that, but hook flush caches seems to be good enough. People are used to just like CC all flushing caches when things don't work and then they get their new migration. So seems to work out right in the back, son. There are definitely many different ways to do it. The question was, would we want to put migrate module in core? I guess my current thinking is no, we don't put migrate module in core. So most users of this stuff would want to have Drush and migrate module in addition. The migrate D2D stuff could be in core. You know, it's open for discussion as to exactly how much goes in, but I kind of think of the migration tool as being not Drupal. It's a different tool that you use for a little while and you forget about it. I think actually the testing tool should be that also, right? So simple tests should not be in core. It should be a PHP unit thing. So that's how I think of migrate also. Question here. It could be a big deal. The question was about timeline and do we have to get all this done by December? It would be better to get it done before December, I think, because it's arguable whether this is a new feature of Drupal changing how we do major version upgrades. The upgrade path was a major reason why Drupal 7 took 18 months to sort out after feature freeze. So we might find ourselves staring at another year and saying, that's crazy, we need to do this. I mean, I think that could happen, too. It might happen that folks are like, sorry, we can't change how we do upgrades after feature freeze and this is a Drupal 9 thing. So I am a little worried about that from a timing perspective, but it'll happen as soon as enough people want it. There's an issue link down here at the bottom of the slide where this has been proposed and it started to be discussed. So if you have ideas or support, you might want to start talking there. No number 151052692. I should really alias it. Okay, why don't we stop here and we can keep on talking about it over the next few days. Thanks.