 Check All right Okay, so those were the goals of successful data migration Okay, I Think I'm I'm gonna start with the end product so you know what we're trying to get to And then I'll show you how to get there So the migrate module ships with a few sub modules one of them is the migrate UI So this is a user interface for Showing the clients on the web exactly how far along you are in your data migration in terms usually of how much you have built of the data migration the scripts and What questions you have about the data about the source data or the destination data? And so you can go to these web pages and talk about them and decide how to handle certain pieces of data And they provide a tool for the client to just see what's going on and feel comfortable at the migrations proceeding Which is really helpful? And lastly you can execute the migration using these web pages. All right, so we will I will show all these things right now Okay, so Let's see how this looks over here There's a big table The URL I'm at what I actually show this so that you can see is Admin slash content slash migrate Okay, so this is a URL you get when you enable the migrate UI module I have additionally enabled the migrate example module which comes with migrate You know migrate for people who don't know you download from dribble.org slash project slash migrate Okay, you enable migrate example and this is what your dashboard looks like. Okay, it has Looking at to the second column. These are the different migrations that have been set up by the migrate example module Okay, I'll talk about what a migration is in a second Each row is a migration It tells you the current status of this migration so idle means the migration is not currently executing It when you're doing a huge migration, you will actually get to see your progress using this tool There's only three rows in the source data for the beer term migration here There's three that have been already imported and there are none left to import. Okay The next row is in the opposite situation of there are a total of four and none have been imported Okay And if I just scroll over Some other helpful things are there's a messages column which tells you how many Error and informational messages have been logged Against this migration. Those are things you have to go see what's going on and make sure that they're not a problem The Migrate module when it actually executes a migration And it comes upon a certain node or comment that it can't save for whatever reason like Drupal rejected it or We realize that you didn't provide a title and that's not allowed or something like that It will log an error for that record and keep going Okay So this is how you figure out what records error it out and you can choose to handle it somehow in your migration Okay We keep a track of the throughput of the last execution you made and the date that you last imported this content okay So with this information you can kind of get a feel for how many records are left and what your throughput is and how long it's going to take All right So let's go into the detail of a migration. So if we click this link on beer term we get a vertical tabs Sort of thing I should say that just about everything I'm showing works for a triple six and triple seven Migrate is pretty much Feature parity between the two branches All right, so if you're going into six you should be in good shape if you're going to seven you're in good shape Okay, so the beer term migration has some properties that are listed here for Clients to look at and for the developer to look at with the client We try to get a product owner which is the client and put their name on here Okay, and their email address and that establishes Who to talk to about issues with this migration, okay, and it's important from a Consulting methodology point of view to get the client engaged and get them Looking at this stuff and working with you so that people are happy with the end product, right? The implementer is the developer In case people need to know System of record I'm gonna talk about later, so we'll skip it for now and a description about what this migration does And here we took something that was called styles in the old system and Those styles are becoming taxonomy terms in the new system. Okay, the Drupal All right, so let's keep going If this stuff is boring to you, we're gonna get into lots of code soon, okay? Okay Okay, so the next tab is called destination This gives you detail about what you're migrating into for this beer term migration Here the term it is a type taxonomy term like I said the vocabulary that you're migrating into is called beer styles Migrate example beer styles and Here are all the destination fields that terms have that you can you can migrate these things for terms They have names and descriptions and text formats and weights and parents You can name the parent by name or by ID and You can give it a path alias. Okay, so this migration that has been set up. You'll actually see that this is read down here What we're trying to say there is that there has been no path alias set up in the data migration So that Drupal field is not going to get populated and that may or may not be a problem, you know It's just telling you that You haven't done that yet. Okay on the source side It tells you here is your query that is actually fetching the data so I Should stop for a one second and talk about all this beer stuff that you see here The migrate example module has two files in it beer dot ink and wine dot ink and The beer dot ink is like a simple set of migrations that you'll want to look at when you're first getting started it is the canonical documentation for migrate module and So this this we are looking at the migrations defined by beer dot ink here And so beer dot ink This migration is a from my sequel database into the Drupal my sequel database. Okay Here we see the actual query that gets executed Okay, and These are the fields that get selected by that query All right All right, so more Perhaps more interesting are the mappings Okay, so we will start to look at how this is done in code, but I think it's useful to see it here that For simple columns in the source data. We can simply map them to properties on the term object in Drupal Okay, so the destination term name That is getting populated with the sources style column in them in it's my sequel database all right Just the description for the term comes from the details column and the parent name comes from the style parent, so What don't I just get out? This database browser it's gonna be pretty hard for people to see who are in the back here I apologize for that. I can't make it any bigger on OS 10, but If I go to the right Tables here, it was like What was it now Migrate example beer topic is the tape the source data. So here's the source data There's three records You can see it has a style column a details column a style parent column a region it has this source data in here Okay, and those are the columns that I'm mapping or that you're seeing the mappings for here Okay Moving on those are the mappings that are complete. They're in the done tab DNM stands for do not migrate Okay, so these are all of the columns that are not being used in the migration But we have explicitly declared them in the migration that they're not to be used. So that's why they show up on this tab In the source data, there's a hoppiness column in the query yet that hasn't been mapped to anything So, you know, this is a case where I'd want to talk to the client or maybe I have talked to the client We decided hoppiness isn't going to exist in the Drupal side. So it's marked as do not migrate and these things are Do not migrate on the Drupal side okay Here's a mapping that has been Categorized as a client issue, which means client you have to look at this stuff and decide how you want it to behave in the new system Here we have a region column in that mysql table and There's no corresponding cck field or Drupal 7 field where I can put this region information and so You know essentially developers asking for guidance here like are we going to do this or not? and We can even have client issues that have a priority in this case It's priority medium and you can put in a ticket number and it will hyperlink here to like the client's ticketing system where Perhaps that's where they want to answer questions from you. So That's sometimes handy to use Okay, so this is the detail about a given migration the beer term migration. We'll go back up to the dashboard here. I Want to leave this just for a little bit and go back to My presentation, okay So we talked a little bit about the detail of a given migration One of the key things you specify when you write out a migration class is Where do I get my data from? Where's my source? Okay These are the sources that come with my great module. Okay, so Lots of database platforms you see in the first bullet point there We've had good success pulling data from all those the second set has Sort of a little bit more exotics exotic formats that you can pull from But we have built-in support for these If your source data isn't in one of these systems or formats There's lots of example code for you to write your own source plugin There's in the migrate module you'll see a sub directory called plugins and so you can follow along and look at all of these See what they're doing and mimic them in your own plugin, okay One other key component of a data migration is specifying the destination of where this data is going to go Okay We have Really common ones like they're going into nodes or users And I've grouped those under entities, okay Triple seven has some Grouping there, so we call them entities Taxonomy terms is in that class too Within a node or a user profile you might have fields attached And so there's really good support for the core fields in triple seven and the cck fields in triple six You should have no problem mapping directly from your source data to field data and We have support for lots of contrib modules too So that you can migrate into organic groups and flags and user user relationships and private messages and You know lots of stuff like that You will find that integration code in the project called migrate extras Okay, it's not in migrate proper if for some reason you need to migrate from your source data into a regular old Drupal table and You don't feel like writing Node save type function in order to do so you can actually do your mapping and just tell Migrate that you're using a table destination and it will save stuff into the table for you Okay, I don't really recommend that for things that we have built stronger migration for like nodes and users but if you have something pretty exotic that we don't have my integration for you can use the Table destination for that and once again if the dates destinations that you need are not here You can create plugins for them Migrate module is a object oriented system You should be able to like look your way around the classes and and write your own Mike who's the maintainer of migrate is you know phenomenal about documenting code and Documenting the base classes. So I think that you know if you're worried about it give it a shot because it might be easier than you think All right Okay We're soon going to get into code, but I just wanted to highlight this piece first So basically you do you specify your source you specify your destination for a given migration And then you do mappings Okay in your code and there are simple mappings Like the one I'm showing here We call the method called add field mapping and We map the thing called show on the Drupal side to the thing called status in the source query Right, that's that's all there is to it and what the default value method means there is that if there's nothing Coming in from the source data, then it should default to zero Okay, it can get more complicated than this, but that that's all we need to know right now Okay, so let's look at a migration class. All right I Think I skipped over one piece. That's probably important to cover In this that's this notion of there's multiple migrations in your project A migration in our terms is like a collection of a given destination and a given source So you can start off with thinking like each Drupal content type that you're migrating into is a different migration Okay So you will just set up a new class for each of those content types and you will then say this is a Destination node and this is the node type We'll look at code like that in a minute and Then, you know, it doesn't matter if the source data for those Two different migrations is coming from the same table or different tables. You just write the query to fetch only that data that belongs in the Content type you care about and We found that it's quite helpful to have to split out your migrations and just do Your articles and do your authors and do your user profiles in your comments and handle them at separate migrations And so that's the approach that migrate modules suggest Okay, so Here we have a class called basic example migration Right it extends the built the class that migrate module offers called migration okay, and It doesn't actually do a whole lot it calls the parent constructor here and it just adds the The team members like the implementer and the product owner like we saw before So we can basically skip that the beer term migration Which is the one we looked at extends the basic example migration, which extends migration. Okay, so your classes Will extend migration either directly or with some class in between Okay, and so, you know Mike has this nice comment up here about what are the four key components of your migration class You have to specify a source a destination a map a fieldmappings Okay So let's see how that gets done Can people see the code back there? Not really All right, let's try to make it bigger. It looks pretty big to me here gigantic Okay, all right. Well, let's start here On this line 83 we have a query variable and We are going ahead and selecting out of the table that we looked at before my great example beer topic We're picking out the fields that we saw before and we have an order by statement there, so they come in the right order I can talk about order later but We have a query and then the next line is what actually specifies the source for this migration. Okay It's called It's the class migrate source sequel and all you need to do is pass it a query of dbtng type and it knows what to do from there. Okay So that's a simple way to specify your source Your your source looks very different for like XML migrations. We'll look at those later, but this is how a a This is exactly how anything from my sequel or or postgres would look because you would probably use the dbtng Layer to select out of it All right, your destination is also pretty simple this destination and Here it uses the class destination term Okay, which tells migrate that you want to create terms out of this and then specifically what? Vocabulary you want to use when you create terms? Okay So we've done sources. We've done destinations and now it's up to map mappings Here we add the field mapping and we do the stuff that we saw on that detail page The Drupal The Drupal field name is mapped to style in the legacy data Description details same thing Um parent name is is to the style parent And there's actually a description here, which is helpful for the migrate user interface can show a description to describe To the client, you know why you're mapping these two things together That's not really a code thing. It's more of a documentation thing You remember that some things were listed as do not migrate DNM This is how you do that. You just put it in the issue group DNM Okay Here something is in the issue group client issues and it's been given a priority and it's been given an issue number Okay So those are kind of advanced things you can do with field mappings All right, and you know this part of the code is not so much about Migrating the data. It's about communicating with the product owners More things that are DNM It's helpful to be explicit about what's DNM so that everyone sees it on the sheet on the UI and it doesn't show red and So forth. Okay, so that's a really simple migration beer term and You know we can do fun things like go run it Okay, so I just checked the column of beer term and You'll see that It already imported its three records So I'm gonna I'm gonna roll it back and import it straight away All right, you saw a quick little batch API thing happen. It doesn't take long to save three terms But it went ahead and did that it rolled back three It would have done so at a rate of 1700 a minute If we had 1700 and it processed those three terms and there were no errors Okay so The migrate module does a lot for you All you really have to do is write these classes that specify your migration and the migrate module is Takes care of fetching the records processing each record logging the errors logging the successes I Will show you what logging the successes really means you'll hear about the map table and how we do that and And then it moves on to the next record and then you know soon enough you're done okay, so You know naturally omitting some frustrating moments, but that's the overall flow of how things go Okay, I Want to look at more code so that was the end of beer term migration beer user migration Is really quite similar Okay It has a destination of migrate destination user It has a different select that it does It also uses migrate source sequel As its source I Guess now I can get into the map Okay, so the map is a special table that migrate maintains and It takes the primary key of the source data So Every user ID on the old system or every term ID or whatever it was Whatever that primary key was and it maps it to the primary key in Drupal for the corresponding object Okay, so here's where you actually specify that kind of stuff So AID is there's a column in this migrate example beer account table Called a ID which is short for account ID Okay, that just happens be what they call it in the old system So AIDs and here's like a little schema API array that tells us how to save it into Drupal tables Is going to be mapped to User IDs in the Drupal system Okay, so if I show you that table again apologize for the small Migrate map beer user There's four columns, but generally it's the source ID the destination ID and Then some more information about whether this mapping actually needs an update and when the mapping was last made So It's actually a really critical useful thing that migrate does by keeping track of what legacy ID mapped to what Drupal ID The number one thing is during development it tells you what you can roll back Because you might have Real data in your Drupal database that you don't want to blow away They might be real users in there There might be real other kinds of things and so we're gonna only roll back exactly the stuff that migrate put in there And roll back the number of things that you told us and so forth. So It's useful for development. It's useful for QA To have a record of what the IDs were on both sides so you can go and look at four or five Users and make sure they migrate it. Okay And you can find them on the other system There are some rare cases where you want to preserve your IDs between your old system and your new system if you're doing like mod rewrite rules To take old URLs to new URLs and there's some support for that We got that support into Drupal 7 actually for users and nodes so you can import them with specified IDs Which is handy. All right Let's keep looking at more code. So there's some convenience functions here add simple mappings Status and mail what we mean by simple mappings are like It's called status in the old system and it's called status in the new system So you don't have to tell migrate any more than you need to here Those are called simple mappings and these two columns are the name the same in both Drupal in the in the old system Okay It can get More interesting methods like this Ddupe For you can run into cases where your old legacy system Let users have the same username two different users Presumably they were using email as the login thing and they didn't care if people had the same names And that's not allowed in Drupal. That's that's expected to be a unique field. So It needs to be Dduped on its way into Drupal and we've come into that case enough times that there's now method where it will Basically find a free name by appending one two three four to the username that's coming in and save it that way Some other niceties like We know that the created field in Drupal That's the node created Well, no, we know we've written the handler for this field To know that it's a date stamp kind of field and so When you do a mapping here basically you can make posted it's okay if it comes in and anything that looks like a date The handler for created will convert it into a Unix timestamp, which is what Drupal expects it to be Okay So some of the the handlers have done nice work for you so that you don't have to worry about massaging your data before it gets into Drupal I'm gonna tell you what you do if you do need to massage data in just a second Here's one worth telling you about Here we're adding a field mapping To the field Field API thing called field migrate example fave beers. Okay, so imagine your content type has a multiple Choice field called favorite beers. Okay Maybe it's on your user profile or something like that And there's a column the legacy data called beers and That column just has a list Of different beers separated by pipes. Okay, you come across this thing all the time and so What we're telling migrate here is that go ahead and separate them on pipes. That's what the separator method means so they're now an array and Then map that array to field migrate example fave beers So that's how you can populate a multiple value field a Drupal multiple value field Okay There are other ways to do it, but that's you know one easy way. That's just one more method on your mapping Okay, so let's see if there's anything else on this that I really wanted to show you I Think that's it for beard on ink to go to wine. Yeah, we haven't come up with spirits Wine gets pretty complicated Yes, right Okay, so Here we have the wine user migration. Okay It uses one new feature that we haven't talked about yet, which are dependencies your List of migrations can't run in any old order for example You usually need to run user import first Because all the content of the website has to be owned by different users, right? And we need user IDs for those people or else we won't properly associate the content with those users. Okay, so Here we're doing the user migration, but it even has some things that have to go before it the wine prep migration The wine file copy migration and the wine roll migration now the file copy migration in this example like Gets their user avatar pictures and puts them into Drupal so that they can be associated with the Drupal user object that we're migrating into here the wine roll Takes all the rolls that were in the old system and Creates them in Drupal and so you know if you know about the roll system You clearly need your roll set up before you create your users or else you can't put them in the right rolls Okay, so that's why users dependent on those and you'll see lots of other stuff is dependent on user because it's about content Okay, so It sets up a map It sets up a query a destination It's got lots of simple field mappings This is kind of what I wanted to show you here So far we've only used one method in these migration classes. It's the construct method But we actually care about more methods in that Here is the prepare method so prepare runs After it runs on every row in your result set from your fetch from your source and It runs before the node save or user save or that stuff happens so Basically migrate module has applied all the field mappings But it hasn't saved yet. So this is a hook where you are allowed to massage the data before it gets saved Okay Which is absolutely critical because it's not all going to be simple mappings Okay, so this example I want to look at the second part of this So here we're looking at the row and the column called sex, okay and There's cases where that has a value of little m or capital M Okay, and what we need to do is basically make that a value of zero for males in Drupal and a value of one For females in Drupal. All right. So, you know, whoever set up this cck field Set up males and females as these integer fields. Okay It's quite a reasonable thing to do the way cck set up It just happens to be different from the way the old system was set up And so you need to be able to you know, remember we promised to have completely native Drupal data in there. And so that's why we better have ones and zeros in our fields when we're done here. So We just basically shove in the value zero into this long property on the user object, okay You know, if you guys haven't looked at Drupal 7 this might be a little bit unfamiliar to you but it's the field name and then the language and then The Delta and the value and then the actual value. So that that's how the The user object works and the you know, all these objects that have fields attached them look like that And here we are massaging the data before it gets saved. All right You might imagine here that we could add a case Okay, well he did actually do that if If it's not M or F It's some sex that we don't know about yet then he unsets it because we can't possibly save that into Drupal It's not in a loud value and And that will prevent an error basically So that's the prepare hook There's a similar one called prepare row And that's a chance you get to massage the data Right after it comes out of the database But before it gets attached to before the field mappings have been applied. So you're dealing with column names. They're not Drupal destination names Usually it doesn't really matter which one you use There can be some cases where it's helpful to use prepare row because it happens earlier Then other stuff can sort of happen automatically because you've massaged the data properly You can introduce new columns in prepare if you want like you can just set a Certain field Field API field to a certain value here. You can set node created You know, whatever you want to do and prepare There's a similar one called complete That is used a lot less frequently, but if you need to do some work after each row get saved To a node like you need to clean it up a little bit or you need to do some bookkeeping over here You get a chance after every record to run your own code All right So yeah for people who weren't interested in the code part of this we're getting toward the end of the code part So I'm gonna move down a little bit and show you other kinds of migrations All right, so in wine.ink we start to get into the XML migrations and I'll talk a little bit about that Here we have the wine producer XML migration and it's extending a class that Migrate provides called the XML migration and The principles are similar Your source in the case of An XML migration you basically have to tell it the URL to the File that holds the XML All right, and that can be a remote URL or it can be a file in your file system doesn't matter and Then you also have to tell it each item and What its URL is also so? There's an assumption in this particular form of XML migration that you have one file Which is like a list of IDs and then you have a file for each detail item So if your IDs are like, you know user one dot XML and to XML you have a file for each one of those If all you have is one giant XML file with all of your IDs and data in there There's a migration coming up later that you would extend that handles that sort of thing Okay, so we've told Migrate where the Listing file is and where the detail XML files are We've told it that we are migrating into nodes and this content type within nodes Okay, and then we go ahead and do field mappings now field mappings are a little bit more Interesting in an XML world. There's three things that we provide Like before we give it the destination field so we're doing it going into Drupal title field here and We have something on the source side called names and here is the x-path in The detail where you get the name out Okay So, you know some of you have worked with x-path some of you haven't You're lucky if you haven't because it's kind of pain in the neck But it is the way to query into an XML document and get certain pieces of data out all right so That's a general technique with XML migrations. This was The case where you have a listing file and detail files The next migration down is called wine producer multi XML migration I think multi means that you have many items in the same many details in the same file and in That case. Oh, yeah, you extend the same XML migration, but your source is a little bit different You here's the source you say you have a source multi items and And you pass it items class and fields. So what are those two things? Oh Items class you've already defined up here yeah, so basically you have you give it like a couple x-paths and the file where it's supposed to find this stuff and It will crunch through your file Find all of the items that you wanted to Migrate and then all of the little detail pieces inside of each of those items Map them to the Drupal object that you said in this case my great destination node again and Go ahead and save them So this is a pretty sweet infrastructure for doing XML migrations and like I said those can be Remote or they can be local. So you can definitely go hit a web service And that gives you back the XML and you can crunch it with with my great module See if there's something else I want to tell you about Yeah Let me go back to the presentation Okay, I Made mention of the different hooks that you have Here are a few of them just to recap prepare row prepare and complete That's where you get a shot at massaging your data Before it gets saved by user save and node save and all of those commands and This is just a graphic of something I talked about before Where? The data flow starts off with your source. Okay, you query your source you get And you you then process one record at a time out of that source Okay, there's a presumption in migrate that One record in your source data is going to be one record in your Drupal data Or not really one record. It's going to be one object in your Drupal data like node user taxonomy term file Etc. If your data isn't structured like that in the source There are ways of coping with it. So if you look inside of wine.ank it gives you some strategies for coping But for now This is how we process each record. We pick it out of the source. We apply the field mappings and We go ahead and save it. Well, we call hooks you massage it if you want we save it And then we either map it or we message it Mapping is what happens upon a successful save We take the ID from the old system in the new system and we stuff it in the map table If there were problems they get logged into the message table Each migration always has a corresponding map table and message table and migrate takes care of creating those on the fly for you as you create new migrations and When you view the Web page the dashboard page if it finds new migrations, it will go ahead and create Map and message tables for them and and go along with showing you that page Okay, so You saw me just a little while ago perform a rollback and an import of beer term so We actually never do migrations that way. We always use the drush commands and I would encourage you guys to use the drush commands instead of using the UI for executing migrations If you have small ones, it really doesn't matter that much But if you get large ones, it's a whole lot more efficient and faster to use the drush commands The batch API is like horribly inefficient So you would be much better off using drush. So let's take a look at drush and see how you do that Okay So what I've done right here is to list the drush commands that come with the migrate module Okay, so this is the dash dash filter option which people probably don't know about but I do and so Here you have like nine commands You don't have to use all them, but they have usefulness to them So if we run the migrate status command migrate status command Which for short is MS You get a version of that dashboard in your drush Terminal, okay, so you can see what's going on quickly in your migration Here we have imported all the terms And we have not imported beer user. Okay, so let's go ahead and do that Okay, so I ran the command migrate dash import past the argument beer user all right and Drush then went ahead and told me I Have verbose turned on in my drush. So I see more cruft than you might see when you do this But here we have we imported beer user and we actually processed for Four new ones were created none were updated zero failed zero ignored And it took one point nine seconds and here's our throughput and I Just wanted to Show off here. You know if I have time. I'll come back to this but There's xh prof integration in drush and develop modules Xh prof is like the tool for doing performance measuring So if you get into large migrations that start taking a long time and you need them to go faster you'll want to use xh prof to figure out why they're slow and You'll probably want to Get your develop module set up for xh prof and when you do your drush commands actually show you the URL Where the report is for this drush command you just execute it so if I go to my browser now And I paste that URL I Get the xh prof output for that drush request and I can see like what Drupal functions were slow during that request and You know none were slow in this particular request Because it's small and it's optimized here's user save which you would imagine would take up most of the time If you do exclusive you get kind of a better look at what functions we're taking time Actually, yeah, the Drupal 7 password hashing is really pretty slow So if you're doing a huge user migration you might want to think about Other strategies or faster strategies or deferring that to later Okay, so Getting back to our drush All right, so you saw me use the migrate-import command to run a single migration You know, let's say we're feeling confident now And we're just gonna say run them all all right all the beers all the wines everything Here it goes. All right So it's cruising through all the migrations processing all the things it has to fetch and Looks like wine wine takes a little bit longer than the others. I don't know why But We'll be done soon and It will let me know if they were errors and if there were I might want to roll those back fix them Roll them forward or maybe their errors that are not a big deal and I can just change my code a little bit and be happy with it So I'll show you some of the other drush commands besides Status and import which we've seen it's slow. I don't know what it's stuck on who cares Okay, so more commands Mfd migrate fields destination, I'll just show you briefly it shows you all the destination Fields that you might want to map to so this is information that's in migrate UI also It's a duplicate of that information But putting it in the terminal can be handier for the migration developer All right, so that's what like migration fields destination Migrate fields destination migrate field source is the source equivalent of that You can just review the mappings for any given migration with this migrate-mappings drush command Migrate-rollback can be handy In the situation that I said before that you need to make a change so you roll back your data And you keep going So if we look at the help for migrate import, which is sort of the real worker of the whole thing You can pass the option limit Which limits the number of records so you don't have to do all hundred thousand or one million at once You can just do five or ten or whatever you think And it will do that You can also limit based on number of seconds Which I haven't used but you can do that You can tell migrate to give you feedback every x number of seconds or x every x number of items And it will tell you how many imported how many failed and what the throughput was So I tend to set that at like every five seconds give me input and you get a nice, you know comforting Informative response out of migrate about what it's done for the last five seconds All right There are cases where you want to import particular IDs from your source system because You know, they're gonna be problematic like you know records before 1990 didn't have this right data in them and you need to make sure those work, too So you put in a few of those IDs here You can pass them as the option ID list. All right, you saw me run dash dash all that means go ahead and iterate over all the migrations in order and Remember that the order is specified by the Dependencies declaration in each migration So migrate does a sort based on all the dependencies that it dependencies that were declared and it makes sure that It does the one, you know, the ones that have no dependencies go first and so on All right If for some reason you one want to run the user migration before the role migration happened You will not be allowed to unless you pass dash dash force So that's why we have a force option on import So dash dash update So there are some more cool features about migrate that I need to tell you about So Basically what I've described so far as a way to In a one-time shot Bring over data from your source system in the Drupal You can get into projects where your source data is changing all the time It's a live website and you know, nobody's happy with you taking it down for a week while you like fuss with a data migration So we have developed tools so that you can migrate this constantly changing source data There's two concepts that you need to know about here one is called high water tracking so basically You can tell In your migration you can specify like how Migrate should do its query in order to fetch just the records since the last time it ran And so that's how it gets just new data and So what you would do is you know make sure that your source system actually has time stamps there about last modified and last added And you write your query so that it fetches that information And There's there's more detail to high water tracking, but you know, it's pretty well explained in The beard on ink and wine on ink There's an additional Like so beard on ink and wine don't ink are great for like migration class documentation Higher level concepts are pretty well documented at druble org slash migrate So that's one other thing. You'll want a bookmark and keep in mind for your migration projects And go ahead and help us with the documentation there So the second concept in synchronization is system of record So you the system of record by default is the source system Which really means that like if you let's say that you have already You're migrating for a second time the same record, okay By default Migrate will overwrite whatever node you saved last time That belongs to that record and overwrite it with whatever is coming in If you say system of record is the destination side Then you can be much more fine-grained about what your migration overwrites Basically, what happens in that case is that we will figure out whatever node or user corresponds to the Source record that we're dealing with we will node load that or user load that and only overwrite the mapped fields and So basically if you have some source data that's just changing in one column for example all the time The price column might be changing, but like other stuff doesn't change or you know Well, whatever So that's a way of just bringing over the new price every day Okay into your Drupal system and keeping the source data and the destination data in sync All right, so people can be editing the nodes on the Drupal side those very same nodes and making changes that you want to preserve and It's all okay because we will node load it We will just change a little bit that we're allowed to change That's been specified in the migration class and then save it back Okay, so you can even have like data changing on both sides at the same time in Some cases and and migrate can cope with that situation So other advanced features of the migrate framework Stub entities so You can come across a situation of migrations where you're migrating something like articles and They have a field on them called related articles and You will like start migrating articles and it will want to relate itself to something that doesn't exist yet And that's a tough situation to cope with because There's basically no way to avoid it. So the approach that migrate takes for this or that migrate suggests you to take for this is You can declare in your field mapping that if there is no corresponding thing to reference It should go ahead and create it Okay, and that's basically a new method that you put in your migration class called Create stub or something like that It's documented and You basically give it like a plain title like the title stub and the body stub and User ID one and it will go ahead and just save that for you and it gets more importantly gets a node ID Okay, so now this thing that you need to reference exists. It has a node ID and The thing you were trying to create in the first place can now have something to reference and then when we come across that thing That we stubbed we will just go ahead and node load it and fill it in with the right information later. I Hope you guys can follow that because a little bit of a brain teaser, but You will appreciate it if you need that Okay, there's a pretty good file handling For all of your Images and audio and video files and so forth You can you know there's integration with file field and image field There's immigration with file entities if you you know if your files are really important to your site Then you might want to make entities out of them and we have come across that and built support for those kinds of entities And their support for timers whoops So if you don't want to do the full xh prof thing you can just set up like start and end timers using some convenience functions in Migrate and then your drush commands will actually output how much time we're taken up by those timers So you can kind of figure out what parts of your migration are slow You know the first killers you'll come across our path auto and token module You almost always want to disable those Which is a little unfortunate, but I'm speaking about triple six now really For every Item that you save it wants to like render the body for it Which is a silly thing to do because we're not going to show it to anyone, but the tokens Do lots of work that wasn't even asked of them really and so that's why that slows down your migration there's a couple more example modules that you want to take a look at I Showed you just one called migrate example. There's another one called migrate example oracle If you're doing an oracle source, you'll want to take a look at that because it has a few little tricks about it There's another one called migrate example baseball Which is an example of migrating from a CSV file as a source It's also an example of something we call dynamic migrations Where one migration class actually can apply to multiple Migrations in the UI So it's like parameterized migrations So you can take a look at that if You find yourself needing like a hundred migrations that are all really similar because they only change one small thing You might be a candidate for dynamic migrations. So take a look at the baseball example I want to encourage people To publish your work. So write blog posts contribute to Drupal org if it makes sense About your successes and failures with migrate Go ahead and show people your code if you can like, you know on the one hand migrations are very personal Because it's your data and no one else quite has the same data in many cases But the techniques are common So I think it's helpful to write about the techniques you're using to get your data into Drupal There are a few cases where people have shared what they've done here migrating from WordPress to Drupal There's a nice module that we did that we published WordPress migrate module Type o3 migrate module exists also And go ahead and share your custom classes as well Both Mike and I work for Acquia now and are available for hire for migration projects So feel free to talk to me about that talk to our professional services people if you'd prefer and Sort of that's it for the talk part So I think I'd love to do Q&A for a little while you can go to that URL and find the session and rate it But does anyone have any questions? Sure Yeah, good good call The question is can the update functionality run? Using the UI or is it a drush only thing? I think that some of it can be done through the UI the stuff that is only Needs declaration inside the migration can be done that way, but I think For the full Range of update functionality you would be better off with the up with the command line There's that to there's the dash dash update option and the dash dash needs update option Which you need sometimes and you can't supply those from the UI You can't currently supply them, you know, you could also Do enhancements like form alter stuff to get them into the UI? Yeah, sure sure, so I yeah I didn't show a comment destination But there are a couple examples and beard on ink and wind on ink of comment destination of comment stuff So, you know to migrate it's the same as editorially generated content You just go ahead and query wherever that data lives in the source side and on the destination side It's migrate destination comment and and I think you have to pass the content type that it's commenting on Yeah, right Right, so good question. So I can show you in the code how we do that sort of thing Okay, so We are doing a Select here from the database table migrate example wine comment And we're taking out these fields and this is where the order by gets to be important We're ordering by Comet parent ascending which really means that We are getting the comments in the order of oldest to newest So we can never have a situation where we're trying to migrate the reply before the Source of that reply comes in So you have to get your sequel right in order to have the stuff be in the right order and after that it's a matter of field mappings So here's where the the relevant field mapping happens the The property on the on the comment object is PID Okay, okay last last answer is PID and in the destination. It's the comment parent. All right So once you set up that relationship you get your threading. All right. Thanks for coming everyone