 Today I'm going to be talking about the Migrate module. This presentation is really geared towards site builders. So if you've used Migrate before, this might be a nice kind of review or a different perspective on Migrate, but you might already be familiar with a lot of the materials I'm going to cover because it's really going to be an introductory session. And I'm also going to talk a little bit about Migrate from the perspective of why you might want to use it for different types of projects that you've had it considered before. So I hope you'll all get something out of this talk. Just as an introduction about me, I work mostly as a site builder and as a themeer. I do some Drupal training as well. My name's Suzanne Dergacheva and I work at Evolving Web in Montreal. We do a lot of Drupal consulting and development projects and we also do Drupal trainings and we've developed a training on the Migrate module. So if you're interested in more about that, please come talk to me afterwards. We work at Evolving Web on a lot of large scale Drupal projects which have an obvious use case for the Migrate module but we also work on smaller projects that we use Migrate for as well. So these days we're using the Migrate module for almost all of our projects in one way or another. I'll just talk a little bit about what Migrate is for those of you who are new to it. And just to get a sense, how many of you have used Migrate before on projects? Are there many of you? Okay. So some of you out there have used it before. So Migrate is basically a way to get content or data into your website programmatically. So if you're starting out with a brand new project, it's a way for you to pull content into your website. And it's an alternative to creating content manually. So if you are creating your first Drupal site and you're just getting started with Drupal, you're probably going to go in there and go to the content page and click add content and start creating your content that way. Which is perfectly natural because Drupal is a content management system. But as soon as you start getting a little bit more content and getting a little bit more familiar with Drupal, you might be looking for a better way of doing things. There are different ways that we have with Drupal of getting content into our websites. So like I said, the most obvious way is just to do it by hand. You have some content that you want to put up on your site. You go and click the add content button. But there are some other modules out there that help you create content programmatically so that you don't have to copy and paste the content in by hand. So we have the feeds module, which has been around for a while, that people use, especially when they're pulling in content on a regular basis. So if you have an RSS feed or some other type of feed and you want that content to appear on your site, you can use the feeds module. The realistic dummy content module is a module that people have started using recently to create sample content on your site. It works with the develop generate module to do that. But it's really not meant to be used as a permanent solution for creating real content that's going to live on your site once it's launched. You can also create a custom migration script. So if you're familiar with writing scripts and writing code, then that might be an option for you. But the migrate module has really become a standard. It gives you a lot of tools for adding content to your site. So you don't have to do a lot of coding, but it kind of helps you to automate the process of creating content. And so the reason that we go with the migrate module is that it's really become a best practice for creating content on your site. It streamlines the process of content creation. So instead of you creating content kind of as you feel like it, when you're building your site you're creating it always using the same method. And it's useful for a lot of Drupal projects. So I'll talk a little bit about that now about why you might want to use this migrate module. So let's say you're upgrading an old site to, let's say Drupal 7. So you have an old Drupal 6 site or you have an old website that was built using some other kind of system and you want to get all that content from the old site and put it into your brand new Drupal 7 site. In this case you might be working with hundreds of pieces of content, maybe even thousands of pieces of content, and it would be an impossible task to create all of that content by hand. So in this case using migrate is a really obvious choice just because of the quantity of content that you're dealing with. So migrate gives you a tool for taking all the content from that old database, whatever type of website it is, and moving it into Drupal. So migrate can also be used as a pretty obvious choice for websites where you're using third party content. Let's say you're building a website that uses open data. Are you all familiar with the open data kind of movement? So you have these databases of information that people have made available maybe from the government or maybe from some non-profit organization and you want to get that content into your website. So again you're probably talking about a lot of content and it would be impossible to create it all by hand. Now the third use case is maybe a use case that you're dealing with on a more regular basis as site builders you're creating a new website and you're just populating that website from new content or content that your client is giving you maybe by creating word documents or maybe the client is taking content from the old website and kind of rewriting it and then giving it to you to put on the new website. So this is a really common use case, right? You're building a new website, you want to refresh the content and so you need to take that content and put it on your site. And in all three of these cases I would say using the migrate module is really the best choice you can do because in each case you want to be streamlining that content creation. Now in the third case you might say well why would I use migrate to populate content on a new site? If the client is just giving me that content it's just as easy for me to copy and paste the content into Drupal as it would be to say put it into a CSV file and use a module to migrate it. And I think there's lots of good reasons why you might use migrate even for this kind of simpler use case. First of all you're creating content directly on a website and you're building that website at the same time. You're going to have lots of inconsistencies. So I'm sure you've done this before. You've created a Drupal website with some content types and then your client sends you the content and then you realize oh I have to change the fields in my content type. So then you start to change the configuration and then you have some older content on your site that's out of date. So it might not have all the same fields as the newer pieces of content. So if you're using the migrate module instead of creating the content right in Drupal you can put that content let's say in a CSV file. Or if you're really fancy maybe in a set of JSON files or something like that. Some more general format. And so you can develop that content and in parallel you can be configuring the actual website and doing the site building. And so this process of separating out the content from the site building it can give your clients more control over that content. So instead of you the site builder having to create all the content through Drupal, do all the configuration, you can tell your client okay go fill up this CSV file with all the content for the website. I'll show you how to do it. And meanwhile I'm going to be configuring the site and creating the views, creating the content types and putting that all in place. So there's a place for that content to live. And this might also give you the chance to start working with real content sooner. Because if you're responsible for creating the content in the site and doing the site building you're probably going to create some sort of fake place holder content on your site. Have people done this before? Lauren had some content for their websites. Not fun, right? You're not quite sure. I guess this is what they're going to put. It's going to be something like this. So you're not really sure what you're working with. So by kind of separating it out and saying okay I'm going to build the site but I'm also going to have a place for this content to live. You have an opportunity to maybe start working with real content on the website a bit sooner. And then finally what I've tried recently to start to do on projects is instead of having one database that I'm working with to build the site and create the content, I try not to have a master database. So I'm not sure if all of you are familiar with the term master database but basically it's when you only have one version of the database that's like the version you can launch with. The version with all the content and all the correct configuration. Instead what I'm trying to do now with our projects is to create sort of components that you can put together to build out the final database. So if you have a lot of configuration on your website, a best practice with Drupal 7 right now is to put it into features. People are here using features most of you. So it's like a way of storing all of your configuration in one place. Migrate kind of does the same thing with your content. So instead of you creating the content in one database and then if you lose that database or something goes wrong with it, then you kind of have to start from scratch. With migrate you can put all of that content creation into your migrations. So when you're ready to launch your site, you can create these migrations and your site is ready to go. So I think you'll see more how this works as I show you the migrate module for those of you who are new to it but these are all just reasons why migrate is useful for even simpler websites that you might be building. So how does the migrate module work exactly? There's a couple key pieces to migrate. So when you're creating content in Drupal you always have a source of your content. When you're creating content manually that might just be a word document or an email from your client. But when you're using the migrate module the source content is always some kind of structured file of information. So the example I'm going to be using today is using a CSV file because I think that's something everybody's familiar with working with spreadsheets but there's all kinds of sources that you can use with migrate as well. CSV is just one example. So you take that source and then you're able to create content in Drupal. And when you're using the migrate module we call this the destination. So you have your source which is where the content is coming from and then the destination is where the content is going to. So this might be a content type or taxonomy term user, comments, you know, some type of content in Drupal that's being created. And so with migrate you can, you're moving content from the source to the destination and you can create the migration or sorry run the migration to test it out. And migrate makes it really easy to undo that. So normally when you're creating content in Drupal you know you might create 50 nodes, you create a view and then you realize that there's some problem with the content you've created and you have to go back in, maybe delete all the content or edit it to make a change. But with migrate it's just one dresh command or one click in the UI to run your migration and then you can roll it back to undo the migration and then maybe make a change and try it again. So it makes this process of creating content really simple to test. And with migrate the one kind of caveat for site builders is that we have to write some code when we're using migrate. So it's not just a tool that we're using through the Drupal UI, there is some code that you have to write. And I like to think of this code kind of like a template. So I'm really a site builder, I'm not I don't consider myself really a backend developer but I do write modules for migrate to create migrations for our sites. And I think of it as being like a template. Like I always kind of start off with the same template for the migration and then I'm kind of filling in the blanks to tell Drupal where to get the content from and where to put it. So to use the migrate module and to create a migration the most important thing to know about is really site building. So the first step is really to create the configuration on your site where the content is going to live. You also have to kind of have an interesting in working with content so knowing about the content that's going to be created on the site is an important part of the process. And then it's also important to be able to write a simple module in Drupal so not being scared of that, that's an important step as well. Or an important part of being able to write a migration. So the example that I'm going to run through today is creating a migration from a CSV file and creating nodes in Drupal. And the use case for this example, it's a small university website that and the university wants to create a website that lists all the programs that they offer. And so far they don't have any place, like any database that lists all the programs. It's so far all been print materials that students have used. So they need to create a CSV file or a spreadsheet that contains all the program information and then we want to have an easy way to get that into Drupal. And we have to launch the site in about a month and so we don't really have time to create the content manually. The client has to be putting together this spreadsheet while the Drupal team is busy working on actually creating the site, doing the theming and so forth. So Migrate works with all kinds of different sources like I was mentioning. CSV files are the example I'm going to use today. And the basic steps for creating a migration in Drupal, you're going to want to start off by preparing the website for the content you're going to create. So at least having the content types set up. If you're creating taxonomy terms or users, you'd want to configure those elements as well. Then preparing the content source, so in this case our CSV file. Creating the Migrate module for custom migration, running the migrations and then testing and iterating is an important step too because it's probably not going to work exactly like you expected the first time. So preparing your site and preparing the content. Like I said before we have two important parts of a migration. We have the destination, where the content is coming from, or sorry, the destination is where the content is going and the source is where the content is coming from. And with a CSV file, it's pretty easy to visualize because each column in your CSV file is going to correspond to a field or some kind of property in your Drupal content type or whatever your destination is. And sometimes if you have things like multi-value fields, you'd separate the fields with a comma or some other kind of separator. So the first part is setting up your content type. This is the part that we're really good at. We're site builders. We're going to configure the site to have whatever fields we need, whatever content type settings we would normally have. And then the more tricky part comes maybe with preparing your CSV. So this is something that in the case of this example, I'd want to give to my client to do. I don't know anything about the programs at this university so I want the client to have a place to put this content while we're getting the Drupal site ready. So rather than just giving them this task and saying oh, client, go and create a CSV, they're not going to necessarily organize the content the right way. So what I would suggest is giving them some kind of a template for their content. So maybe create a CSV file, add all the columns that you would expect to have for the fields in your content type, and then fill in a couple rows of sample content. Like that work that you usually do of creating lorem ipsum text. Well, don't use lorem ipsum, try and be a little bit more creative. But put in some sample content into the CSV. How you would imagine, you know, what you would imagine the university's programs would be like. So maybe put in a program for, you know, teacher education or whatever the case may be. They might have already provided you with some of this in their designs, so it's just a matter of putting it into the file. For things like images where you have an image file and also an alt text, each of these can be separate columns. If you have things like properties like the author of the content, their user ID, you can put that into a separate column as well. So it doesn't just have to be fields. And what you want to try and do is have maybe one tab open with your content type configuration, and then another tab open with your CSV, and just try and make sure that the two things line up. So if you have a multi-value field in your content type, make sure you have a field where you can have more than one value in your CSV. And in the CSV files for the clients especially, I like to have two rows of the top, one for kind of the label, and then one with some instructions about what they're supposed to put there. So if it's supposed to be text or if it's supposed to be a file name or a multi-value field separated by commas, you can put that right into your CSV. So once you have all of this ready, you have some sample content, you have your configuration, you're ready to actually create the module. And so basically what you need for a simple migrate module, you need to have one file typically in your migrate module that contains the information about the migration. So in this case, my module is called migrate underscore programs, and that file is just going to be migrate underscore programs dot migrate dot inc. And then I also have a dot module file and a dot info file. So if you've ever created a module before in Drupal, this is going to be really familiar. If not, it's a good one. Each module in Theme in Drupal 7 has an info file, so you just have to include some basic information about the migration. And then in this case, we're also going to put a dependency on the migrate module because we'll need to have that enabled to run the migration. And we'll also add an extra file, migrate underscore programs dot migrate dot inc, that has the migration in it. And all the code that I'm showing you here, I have it up on GitHub, so there's some links that you'll see here if you want to actually go and look at the code while I'm showing it to you. So in the module file itself, you have to include a little bit of information about your migration. And basically all we're doing here is we're naming the migration. So if you're using this as a template for your own migration that you're writing, you could just replace the word programs with whatever identifier you have for what you're migrating. And down here in this function where we're defining the migration, the most important thing that we're doing is kind of telling Drupal what the name is for our migration, for our migration. So we have the word programs in the migrations array. We have programs and then we have our class name, which you're going to see in a second when we actually define the migration. And then another thing called a group name. And in this example, it's not too useful because we're just migrating programs. There's just one thing to migrate. But you can imagine on a typical website, you're probably not just migrating programs. You might be also migrating instructors or faculties or other information. And so you can put all of these migrations into a group so that when you create your site, all the content will be migrated at the same time. So now we come to the main part of our migrate code which is where we're defining our migration. So here we're defining a custom class called migrate programs. Which again you could just replace with whatever you're migrating. So this could be migrate instructors or migrate faculty. And basically this is just using a format that the migrate module provides to define your migration. So you're telling migrate what you want to migrate. What the source is for your content. Where you want to put your content. And then how to map these things together. If you can look at the code for here just to see as an overview. We have two functions that we're defining. A constructor function. And then a function called prepare row. So the constructor is the main function where we're defining the migration. And in here we're going to do basically four things. We're going to tell the migrate module how to identify our rows of content. And to do this we need to tell something that's unique about each row. And this is what lets migrate pull in all the content and then be able to roll it back. So that feature of being able to undo a migration that's only possible because the migrate module has a way to identify each piece of content that you've pulled in. So that's the first thing we need to do in here. And then we also need to tell migrate where to get the content from. So where is this CSV file. We need to tell migrate where to put the content. So what's the destination? Is it a node? Is it a user? And then how to map the source and the destination together. So those four lines of comments there those are kind of like placeholders for the four things we need to do. In the next few slides. So first of all we have to have a way to identify the content. So in the CSV example that I showed there's just an ID column. So this is kind of the simplest way of identifying your content if you actually put an ID on each thing. And this doesn't correspond to a node ID. It's an ID that migrates going to use to keep track of this content. So if you run your migration and then you roll it back and then you run your migration again and let's say you repeat it 50 times because you make a bunch of mistakes your node IDs on your site they're going to get really high. You have 50 nodes times 50 migrations. It's a lot. Your node IDs are going to be in the thousands. But the IDs here are always going to stay the same. So it's just a unique identifier that migrate uses so it can roll back your migration. The other thing we're going to do here is tell Drupal about the source of the migration. So in this case we have a CSV file and CSV files have these columns. And so what we're doing here is we're giving each column in our CSV file an identifier. So there's this array that we're creating called CSV columns. And the number that's the key in the array is just the number of the column. And it starts at 0. So the first column is the 0 column. And then we have an identifier for each column that we want to use in our migration. Now if you're using a CSV that maybe it's from some open data source or maybe it's CSV from your client you might have some columns that you don't want to migrate. So if you have a column in your CSV file that's just called editor notes you don't have to include that here. So not every column has to be listed so columns that aren't just won't, you won't be able to migrate them. And you can also see down here that when you define the source of your migration you can give the migrate module a link to the CSV file that you're migrating. And then you can also tell it to skip a certain number of rows. So if you have one row at the top of your CSV file that has the labels for the columns you'd want to skip that for sure. You might have a couple rows like I do if you want to skip also the row that has the instructions for the author of the content. So this is all to tell, migrate about the source of your content. When you're writing this code or when you're filling in this kind of template for your content source you're going to want to be looking at your CSV file and counting the columns to make sure that everything's matching up. And the third thing you'll want to do is to tell the migrate module where to put your content. So the migrate module calls this destination handler. And here when I tell the migrate module the destination I tell it to use migrate destination node because I'm creating nodes. You could do the same thing and replace the word node with user or comment or some other type of entity in Drupal. And all the core entities that Drupal has are supported by the migrate module. If you want to migrate other types of things that aren't in Drupal core there's lots of modules out there that provide support for migrate. And there's also a module called Migrate Extras that provides support for certain modules. So sometimes you'll need to install Migrate Extras and sometimes the support is just going to come with your module. So for example if you're using the media module and you're migrating in a bunch of YouTube videos there's a sub module I think it comes with media called Migrate Extras media. Or it might come with the Migrate Extras module. In any case it's a sub module that you have to enable to have a migrate handler for YouTube videos. So sometimes it takes a little bit of playing around with these modules to see what to use. But for things like nodes, taxonomy terms, users it's very straight forward you can just use the core migrate module. And so finally you've told the migrate module about your source and your destination and now you can map these things together. So looking back at the identifiers you used for your CSV columns and the names of your fields in Drupal you can add field mappings for each of these in your migration. So when you're adding a mapping the first thing that you pass to the mapping is the name of the field in Drupal and the second thing is the name of the source. So if you go back to your list of columns the identifier that you use there like program level, program type that's going to be the second value that you're passing to the mapping. And the first one is just you should be familiar with that as a site builder you know you have all the names of your fields sometimes with things like images or links you'll have extra attributes that you're going to map as well. So for an image you have the image itself but then you'd also want to have a mapping for the alt text if you want to migrate on alt text for each of your images. With a link field you have a title and maybe a URL. So those would be different attributes. And for the attributes you'll see there's a colon so you have the field name colon and then an attribute. So that's all that's kind of required to create a migration. In some cases you'll have to add one extra function to your class that you've created called prepare row and this is a function you can use to tweak your content before it gets saved in Drupal. So maybe in your CSV file you have a bunch of content and you just want to make you know you want to change one thing about the content before you migrated into Drupal. And I think the most common use case for this is if you have a multi value field so for example if you have a tags field that you want to migrate into Drupal but you want Drupal to store each tag individually Drupal or the migrate module is expecting to have an array of tags that it will create to create the individual tags. Whereas in your CSV file you just have the tags separated by commas. So in that case in your prepare row function you can take your comma separated value list and then turn that into an array. I'm not too sure what happened to my slide. Yes that would be great, thank you. You know to list what the source is and then to have to map that to the destination. Why can't you just do the mapping? And the reason is because you might have some fields that you want to make changes to before and they're actually mapped to your fields. So for example if you have let's say we're migrating in a list of authors. So you might have a field for the author name and then for the author image and then for the alt text maybe you want to use the author name again. So it's not necessarily a one in one mapping. And in the case of the tags that I was showing in the middle where you want to make a change to the authors or what you want to migrate. So when you're coming up with those identifiers for the columns in the CSV file that's just something that you are creating. Just like when you create a name for your fields. Usually I try and use the same name of the name for my fields so I don't get confused because I want it to be consistent but it could be anything. If one of your fields is a field reference to another node, can you migrate it then because I think the field reference is a node ID and it will be another node ID if it's on the destination. That's a great point. So I think the next thing I was going to show on my slides if I ever get this laptop turned back on were some more examples of different types of fields. So one really common example that most Drupal sites have is an entity reference field where you have another node that you're referencing and usually when you're creating a migration you want to be in the migration you want to be creating both nodes. So you want to be creating a node and the node that gets referenced. Say in this example we have a list of programs that we're migrating in. You might also have a list of instructors for those programs and then you have the entity reference between the two. So what you do is you create two migrations you extend the migration class twice and create two of them one for your programs and one for instructors. For many of your CSV files or programs you have an extra column for instructors and then you have another CSV file that lists the instructors themselves. You have the two CSV files and then in the instructors column you'd include the ID for the instructor that references the ID column and the instructor's CFC. And then when you're creating the migration and you're doing the mapping if you tell the migration to use a different source migration. If you look at how the mapping for free to mapping you can define a default value. You can also define a source migration which tells you to migrate to look at that other migration to look at which instructor to reference. So that means that if you don't have to know what the node ID is you can just out migrate what the...that will be the case of having programs that reference instructors. You might also have a use case where you have programs that reference programs. You might have a field of related programs. In that case the source migration community of things will be also programs. And you might run into some problems there because you might have a list of programs that you're migrating and then you have these related programs. But the related programs might not exist yet because you're just writing through the migration one item at a time. So the second program that you might write might reference a related program at the end of your list. And in that case there's one more function that you can add to your migration to create a stub. Node is just sort of a place holder node that migrate will create for you until that program is actually migrated properly. So it will create a stub with that identifier that's referenced from entity reference. And then when it gets to that program and the migration it'll fill in all the other details with all the other fields. So that case is a little bit more complicated but if you look at how to create a stub function with migrate it's looking for more examples. So the code I have it'll be online. I'll link to it from the section for the page on the website for this talk. But if you're looking for more examples the migrate module comes with a couple great example modules and they're called beer and wine. So the examples are like for migrating a list of beer and migrating a list of wine. And run into is when you have when we've had a set of documentation where the source was an external repository and we wanted to use migrate to pull that into Drupal on a regular basis. So it wasn't just a one-time migration but a migration that was going to run every time the documentation was updated. So we had to customize migrates so that the node IDs would stay the same. So that our content wasn't always being completely overwritten and wiped. So I don't know if is my colleague Dave here? Is anyone here? No? So I can't really speak too much to that just because that's not my experience. Well it's a question slash answer actually because I've tackled this issue so I want to double check and see what I think is valid. On the interface you actually have the source navigation. I can't read that well it's source navigation because I had the tackle with taxonomy term I migrated the taxonomy term and then my content I wanted to have the same taxonomy term as it had on my Drupal 6 site. So I think that if you put the source navigation there the node migration that the entity is referencing or the taxonomy term that is referencing and so on it can map the movement, the migration and it can still reference the same nodes slash taxonomy term slash entity. So I believe this is more or less how to do it. Slash question double check if it's a valid way that can answer the question. Can you turn but the ID might be different? But the migration module knows which node ID from Drupal 6 goes to what node ID in Drupal 7. And there you can choose which migration it can follow. So it's the same with files because you move files and you have a different file ID but if you put there that you wanted to watch the migration of files that you have done before then it can follow and it can actually map the same files of the same fields into the same nodes. Seems right? Yes. So definitely the references like I mentioned before when you're migrating an entity reference field Drupal or the migration module has these identifiers that it's going to use so those references should stay the same when you're migrating from Google 6 or content. But the IDs themselves will not be the same so like the node IDs, the taxonomy term IDs those will most likely be different. And so the problem that you might run into is if you have a code that's running that depends on the node ID being something in particular or if you have links that use the node IDs those are the situations I think where you're going to have. But the references should be fine. They're absolutely correct. Good. Thank you. What about if you want your site users to be able to upload a CSV file to your site and which automatically creates content to your site, can migrate what you'll help in this case? Oh that's a good question. Definitely do that with migrate. Again you probably have to write some more code. You have to write a little bit of an interface for allowing them to upload the CSV file and you have to have some code in your migration that manages where the CSV file is. But theoretically the CSV file that doesn't need to live in your Drupal site could be referencing an extra. The code back here where we tell Drupal where to get the content from. Here we're defining a link that's in Drupal but it could be a link that anywhere could be and have getting the migration to actually run. That's also something that you can trigger with code. So if you write migrations, we run them often through a deployment script. So we have a module that's called and this is the module that's responsible for installing all the features on the site, all the modules, and then writing the migrations themselves. So it's something you can control programmatically and you could trigger that to happen when something like the feeds module if you have content that's being updated on a regular basis. The feeds module is definitely well, it was written to accommodate the needs of content that change all the time. So it might be a good alternative to look at. So I think it's six o'clock. So thanks so much for coming to the session and apologies for the technical challenges. If you have more questions, feel free to come and chat with me later. And good luck to all of you using Migrate. I hope you all dive in heads first and use the module.