 It's 11.15 and got a lot of stuff to cover. Everyone that's here is here for the Drupal 8 migrations overview. They're basically going to be covering how to migrate content into Drupal, what are the strategies to use, as well as how to go about doing it. My name is Fulman Joshry. I'm a developer at Colfert Labs. You can find me on drupal.org under the moniker SJP-79. You can also find me on the Drupal slide group under the same name. And I also have a bunch of content on GitHub that you can find under the same name as well. If you guys want to win a Google Home, just come see me afterward. I'll have some stickers and some other stuff. So I'll tell you exactly how you can go about doing that. All these, OK. So all the slides and everything are at that address. All the code is there. All the name of the product is there. This is a migration you did at Statscan for people who need it from there. We did a Drupal 7 Drupal 8 migration. So all the code that was there is really in the reference field. You should also check the Drupal slide group. There's a lot of good knowledge on there for reference. And the session video is going to be here. OK. So today's agenda is basically we're going to look at some migration basics. What exactly do you need to do to get started between migration? We're also going to be looking at the different types of migrations that you can perform. Then we're going to go into actually how you actually go about doing a migration. So that's going to be all the migration breakdown. Then we're going to do a live demo for Drupal 7 to Drupal 8 migration. And now, so let's get into it. We have a lot of things to cover. OK. So before you start doing a migration, you've got to figure out what do you want to move over. How are you going to get there? Where's your starting point? Where's your end point? There's no quick way to get there, unfortunately. There are some ways you can kind of get there, but there's no real quick way to get there. But you can actually build a plan that can get you there in a timely manner. And there's also no one way to get there. You have to get there through a combination of ways. So this is basically the migration over. Whenever you have a migration, you have a source, a destination. And things in the middle are basically null odds up into one place. So in this example, we're basically migrating to users. We have a user with all this information. They have a destination that's applied to, and really it looks more like a migrant map. Something that's making all this information together. So for example, you're moving from one house to another house. You need to know exactly where all the different components of that new house is. So you can move everything over to you as well. In Drupal 8, unlike Drupal 7, you often need to understand what exactly can you move. Because in Drupal 7, what you could do was only migrate content over. Oh, before I even start. How many of you actually had users migrate on Drupal 7? Have you guys had Drupal 8 migration? No? OK. Yeah, I should have asked that first. Anyways. So when your planning is up, you need to actually understand what can you actually migrate over. If you don't understand it, then basically you need to make a checklist of things that you actually want to move over. You want to move over all the same content types. Do you want to move over all the text on them? Do you want to move over all your kitchen appliances to any location? That sort of thing? So a lot of that stuff you can do. We don't actually have to recreate that on the other side. So if you didn't want to recreate all your content, then you should export all that as configuration from your previous Drupal 7 site. But this is the sort of, yeah. Anyways, let's go. OK. So then there's of course the types of migration. There's a lot of different ways to get to your destination. You need to actually have some sort of plan, some sort of way of doing it. So you can load everything up into a truck and hopefully get there. Maybe you'll stand in the way of all your Ricardo. So there's basically three major types. Actually there's a fourth one. I'll get into that. There's three major types. There's manual labor. You're basically copying and pasting content. You're basically going into your source, looking at what's there, going to your destination, creating what's there. So there's basically a lot of manual steps, a lot of copying and pasting. Then you have your Drupal to Drupal migrations, which are built into Drupal 8's migration model. Then you have custom Drupal migrations. Now, any time you're doing a migration, it's not only one thing or one of these types. It's a combination of these two types. So the manual migrations, these are very slow and cheap, because you're basically copying a case. But it does allow you a lot of control in regards to what you're actually doing. So because you can go into your source and see, oh, I got my content this way. Maybe I can turn this into a paragraph type or something. So I can maybe do something different on the other side. But of course, it's fine for moving a small set of data over, but it's not really great for moving a large chunk of data. Then of course, you have your Drupal to Drupal migrations. These are, this is the Drupal upgrade path that's already built into the migration module. And it's very easy or not, depending on what you're trying to do. Because if you're doing the Drupal to Drupal migration, it can only migrate over what Drupal 6 or Drupal 7 core can actually understand that they actually have. And it's just a group of models involved in that stuff. Well, guess what? They're not going to move over. So you're going to get a partial framework. You're going to have to go into each one of those files and try to figure out exactly what needs to be added and what doesn't. So it doesn't allow for a great deal of refactoring. But there is a middle ground. There is what's known as a partial Drupal to Drupal migration. I know this is a little bit of a problem. This is a question, man, that has conversion. You pass in your sort SQL site, and you do what is a big problem. And what that does is it lets you move your framework of your site over without any of the content. So what that will let you do is basically you have your structure. Well, I want to just refactor a few things here and there. Then I'll start moving the content up. It gives you a lot more control. Of course, it's not like breaking a house in half, but yeah. Then you have your custom Drupal migration. It's completely customized. It's the way you want it. You migrate as what you need, when you need it. And it teaches you a great deal about how Drupal 8 works. Because you dive into the plugins. You dive into how entities work, how basically all the APIs work within Drupal as well. And it also teaches you why you should take Drupal 8. And also why you should appreciate Drupal 8. But you have to have a lot of time and patience for that. So now we're getting to the whole part of how we actually get our content to where it needs to go. You can use a manual way and just slingshot everything over. You hope everything lands where it needs to. Or we can try to understand how Drupal 8's migration stuff works. And the thing you need to learn right off the bat is migrations in Drupal 8 is general, which is far better than what was in Drupal 7. I can tell you that. It's all integration. And basically, if you are doing a migration, you can actually just do migration on the end of it. You don't have to write any piece you can. Which is really nice. And these are the modules you will need to actually just try there as well. You won't need the migratable thing, unless you're going to be doing the framework stuff. But it's there. Now the other thing in the migratable is that all the migrations are very structured. They need to be under the directory. They need to be under a certain folder. They need to be called something. So all of your migrations are going to be under the config install migration. They're going to be named by your plus sub migration. And then the name of the migration, it's going to not call my big plus sub migration. And then basically, Drupal is not going to recognize it. It's just going to be like, it's just next to the yellow box there that doesn't really belong to them. All your storage plugins, I'll get into the storage plugins a little bit, but they all go in there. Process plugins, destination plugins go in there. But this is basically, it's kind of hard to see. It needs to be bigger. It's kind of hard to see, but basically, at the top of the end, you're going to install that's where all the migrations go. And within your storage, you have basically all your source and destination plugins. So basically, the number you're bringing in is going to have a source plugin that's going to tell it, okay, this is the project I'm bringing over. I'm pausing it up. What format does it need to be in? So I'm going to do that for a second. You're going to have a process plugin that basically transforms that stuff into a useful stuff. Then you have your destination plugin. Most of the time, you don't have to implement the destination plugin. But if you need to, it's possible to do. Okay. So now we get into the actual migration YAML break. Because as you know, all migrations within Drupal 8 are YAML. This is the structure of it. At the top, you have an identifier that identifies what kind of migration this is. I know this is going to be a little bit boring because it's going to be going to a lot of code. So, well, yeah, configurations. Let's get through it, and then we'll get into the user. So, we're going to have the identifier. We're going to have a source, process, destination. These are the three types of plugins available for migrations. Then we have all the dependencies involved with each migration. Yes. So here's a sample YAML file. I didn't have enough room to say everything, so I can work it up to the side. But you get the idea. The blue section here at the top is basically the identifier. We're basically saying the ID, we're going to call it dn, dfcd, you know, article. Because this is an external content that we're bringing into Google. I call it that, you can call it anything you want. And migration tags. So basically, if you want to tag and want to start together, this would be important in one group. If you move up, there's also migration group. You can also import based on group type, but in that case, it means that it shares a configuration that only a migration in that group will share. Dependency. Now, this is an important section. So basically, let's say you change the integration of your migration file and you want to be imported. If you don't have that and you uninstall the module, it will still retain that configuration. And when you try to import the re-enable module, it'll complain about that it already existed, can't really update it. It'll try to recreate it. But if you do that and you uninstall the module, it'll actually remove the configuration and then it'll be imported. Then push a label. Then of course, after that, the yellow part is the source plugin. I'm just gonna give you that. So then we have the source plugin. Now, this is the plugin that we're implementing in our cloud app module. Which is basically, migrating over a lot of articles from our D7 site into V8. Then we have what is called target. And this is our database connection that we're using to get our stuff. Then we have our process section where we have ICD or processing, where we're mapping, trying to do a title, where we're getting type, the default value of commitment. Which is wrong. Then we have a body value. We're assigning body value to body, body part back to this value. You can actually assign some property to a certain field, a certain value. Then we have an angle destination, which is very, actually, no. We're only defining the type of commitment which should be our goal. And we have a number of components that basically depend on the user migration. But, anyways. Okay, I think I got to all of these things. Yes. Yeah, this is just the same one again. Just in a better format that you can actually see. There's a bunch of processes happening. So, in this one, like the other one, there's a UID stuff happening, which is like upon one of the vibrations, you get that value. Yeah, and then we have destination dependencies. So, these are all the destination plugin examples that are given within Drupal Core. Except for, I think, entity reference to vision and entity group. Those are contrib modules. If you wanted to basically migrate into paragraph, you would use entity reference to vision paragraph. And if you migrate from O.G. into groups, basically you'll use that. Okay. So, these are all the different types of source plugins that you can use within migrate. You can embed data directly into your YAML file to import directly. You can also use a SQL source which we are going to be using for the demo. And you can use file base. So, you can have JSON file, CSVs, XMLs, et cetera. So, this is an example of basically a embedded data example. So, basically if you embed data into a YAML file, the plugin will be embedded data. You can pass in default language to enforce the current language to the English or whatever the default language of the site is. If this is going to be a translation to an existing migration, then you're going to say that this is translation is equal to true, which is just passing in a primary and judicial resource. Parts sequence, but basically this is a structure of a embedded data looks like the YAML. So, you have basically key care for everything. For the body, you basically use a pipe to signify that this is using more than one line. And let's just follow there. And of course everything is separate. Oh yeah, I should probably just use this. Everything is basically separate with a dash to indicate each individual role. For the SQL plugins, you have basically the plugin name, which is the plugin ID. You also have a source migration. The source migration is basically a BD connection. So, as you can see in the example, given this is your settings of HP file within your actual group of eight site, that you will have your default connection. Then after that you can define your source migration. You can actually call it anything you want. And just specify as your target, I use source migration. And it's just building a connection to the actual site. So, in this one, we're basically being a Postgres to MySQL migration because the source is a Postgres site. So, this is basically an example of a SQL source plugin. My highlight is that the file name and the class name you need to be saying, this is the object language practices that I make mistakes sometimes. And you need to specify the main space for your migrations. They all have to be Drupal module name, plugin, migrant source. I use a previous Drupal camp migration example for this. I just modified it for Drupal north, so that's why it's using Drupal camp migrations. And of course, whether you're trying any sort of plugin, you'll have this dot block, and you'll have to have this annotation. If you don't do that, it's not, you won't recognize what the Drupal is. Yes, it seems like this is a problem, but this is actually a problem. Yeah. And of course, you have your article milk, which extends people base, and you can have all the functionality involved with that. Then of course, you have your file base to source plugins. I'm not gonna cover them because they're not part of my great core, and I don't have enough time to do that. Okay, now let's get into the process plugin. Now, the process part of every yellow migration file is basically a key value carrier. But it also allows you to transform a lot of the stuff you map in as well. For example, these are all the process plugins that are in Drupal 8 to get you started. Most likely you won't need to implement any of these, implement your own custom one, but you can probably use these. But anyway, so this is basically an example of a key value carrier. You're basically mapping field title with the source field title, right? Or you're mapping field organization value to the organization name from the source. Or you're mapping the field URL with the URI property, the URL from the source. Or you can do something like this, which is basically you're passing in, you're saying the type is gonna be a default value, and then you're just explaining what the default value is. And same thing for the field URL. So basically you can assign some properties within fields, certain value, but you can also process what exactly those things are gonna be. Or you can do something like this, which is basically you're saying that this piece of content belongs to a user, but you've already migrated the user through another migration. So we can just pass in what the source UID is into that migration. It can tell us what the new UID is, and it can map it back to this content. So basically, if a critical user owned content on the source site, now they own the same piece of content on the destination site. This is an example that we need to talk about. And I'll leave it to them. Then we go into the destination plugins. The destination plugins are basically for entities that you create custom that aren't part of Drupal 8. And you need some sort of migration path to that. They basically have a mandatory plugin feed, which pretty much all destination plugins have. And they have an optional one where you can specify default bundle of type. This is an example of a destination plugin. And basically it's using the same name spacing except at the end it's using destination. It's implementing, it's using a bunch of, bunch of classes from different versions. And at the bottom it's using annotations to say that this is my custom node. You can't read that, but this is just basically how it's laid out. I'll show you that more in the demo. Okay, now we just dive into the code. And we're making good time. Okay, so here I moved off. Okay, so here we have a Drupal 7 site. And here we have a Drupal 8 site. We can't like, block off, thank you. Okay, so Drupal 7, Drupal 8. First thing is of course we need to ensure that, oh, I'll just show you how the Drupal 7 site is. This is a base Drupal 7 site. It's got any translation enabled. It's got English and French turned on. And we're basically using the default article energy type. I haven't really added anything to it. Oh, sorry, that's Drupal 8. So here's the Drupal 7 site. We have articles, we have, I do enable title modules, so that the title is translatable. And basically it's just using body tags and image. I use the develop generate module to generate a bunch of temporary data to basically use. So if you go under structure taxonomy and list all these, you get basically a set of taxonomy data that basically is in a certain format. It's subjectonomy terms underneath each one. And then we also have a whole bunch of users. So I think I generate like 50 users just to have. And then we have content. I generate just two pieces of article content. You can add more. And basically what we're going to be doing is we're going to move users over first because they rely on their media to generate exactly who the owner of taxonomies are and who the owner of articles are. But we also need to move taxonomy over before article because there's a reference field on article. So basically it needs to know what terms it uses before it's actually there. So in order to do that, I'm just using a default Drupal 8 site as well. So basically all the same content types exist on Drupal 8. But I had to start doing migration. Oops, sorry, that's my alarm. That means I'm making a ton. Okay, let's see, it's not my fault. Yeah, okay. Here, it's hard to see, isn't it? Can you see that or should I make that bigger? Okay, so here I have a Drupal 8 module that basically does basically influence library by your class, by your tools. And all these are needed in order to basically have access to all those plugins. As well as access to all those source migration paths, as well as all the destination migration paths. You can also use migrate Drupal, which is the migrate Drupal upgrade path, but for today's example, we're not gonna do that. And then you have basically your layout. So as with any layout, you need a config install for all your migrations that you're gonna be doing. You're gonna have a source, plugin, migrate, source for all your source files. You can have a destination for all your destinations. So here's all the migrations that I wrote for this demo. One thing that I didn't cover in my slides is that you can create groups with shared configurations. So here's an example of a group I created a group called DNExternal, it's the same one I used from Montreal. So it basically says that all the migrations under this have this dependency, their source type external database and that they share a source key of legacy. Legacy is basically the database source that I've set up. I am, that's really hard to see. But anyways, I'm here defining my database connection as well as the legacy data detection to my D7 site. And so that's where that comes from. And so this group is basically going to be applied to all the migrations within here. So I set up my Texanian migration in this way where basically I set up my identifier, I set up my source as being Texanian term data which is right here, Texanian term data. There's my migrate source. It's basically being a SQL query to Texanian term data table on the D7 site. This cap grabbing all these fields. Then saying that I also, these are the actual fields that I'll be passing on as my source fields. And this is basically the key that I will be using. So you basically have to define your data source. And then you have to prepare a row. You're gonna basically say, okay, I kinda like cheated here a little bit because I didn't want to migrate all the categories over first, so I just said that if the VID is one, I just use tags because in Drupal 8, unlike Drupal 7, VIDs are now machine names for all the vocabulary. Then you, of course, you need your parent ID because some of the taxonomy are children of other taxonomy. So you need to capture all that information. And yeah, so I'm just setting that up as my source here. Yeah. Then of course I'm doing all my mappings. I'm setting TID equals to TID, VID equals to VID, name equals name, description value is equal to that. I'm setting the format to rich text. And I'm setting the weights. Then of course I'm doing the parent ID. Now parent ID requires two steps because there could be some taxonomy that don't have parents. So they have to be imported differently so that you don't want to sign and add ID to them. Whereas others, there could be taxonomy that do have parents. So in those cases, you want to basically tell migration that okay, I want to know what the TID of this taxonomy term is if it hasn't already been created yet. All right. So I would pass it in to the same migration to say that okay, here's a TID, have we imported this yet? And it will say yes, we have, here's the TID. Or it would say no, we have it. And it'll import it and then you'll get the TID. So that's kind of the recursion here. Then of course, we're basically, oh and yes. So parent ID doesn't actually map to anything on the destination site. It's just a variable that we've defined within our YAML file to hold the output of this process. Then on the destination site, we do have something called parent. We're basically gonna say parent is gonna be a default value of zero unless there's a source specified. Right, because so zero just means that this person doesn't have a parent and otherwise, we're gonna use this variable that we defined up here called parent ID and add that value there. And of course, we have our destination, take sign and turn and all the sources. Okay, so let me see if I actually did the, I'm using Dresch for pretty much everything. So Dresch is a really cool tool that lets you import items into Drupal. Basically have this form of Drupal. So I did term, take sign and turn migration first. I should have done user migration first. So let's go back and look at the code. Although, did I actually map it? No, I didn't. Okay, that works. So in order to run a migration, you can either use Dresch or you can use migration module menu within Drupal. Here, I declared my group as Drupal camp Montreal. And there's the unexpected error, that's interesting. What happened there? Subtune. Okay, interesting. Okay, I'm just gonna use Dresch for this example. Okay. So basically I'm gonna migrate in the textuality terms. I'm just making, here we can see that we have 10 textuality terms that need to be migrated over. There's 10 unprocessed, zero been imported. So Dresch MI, which is migrate import. Or you can actually write out migrate import. And then the name of the migration, d-i-e-x-t-d-z, taxonomy, oops, taxonomy, and term. Yes, you're right. Okay, and if you run that, you get this saying, oh, I processed 10, created 10, zero were updated, zero failed, zero ignored. And we can go over to our Drupal 8 site and go under structure, taxonomy, list terms. Here's all our terms with all their structure and everything. Pretty neat. And if you go on to our Drupal 7 site, go under structure, taxonomy to verify. We have all the same terms. The order is reversed because it brought in the first item's first and so forth. So there's a little bit sort order here. But all the, what do you call it, all the parents have the same kids and so forth. Okay, so let's do the user migration. The user migration is set up pretty much the same way. It's doing a lot of, let's see. Okay, yeah, so user migration is doing pretty much the same stuff. It's basically using the user import source plugin that I've defined using this target source migration. Yeah, and basically it's mapping all the fields, the part of me, and so forth, mapping back to the users. You know, it's also doing a fallback to site default for all the land codes, the truth, and falls for the land code. This is basically another parameter you can pass into a particular plugin if it defines it that you can be set back to the site default. Now I'm going to run the user migration, dreshmi, dmxt, db, user. Now this is going to work for this in a couple errors, which I couldn't resolve overnight, but there you go. So it created two, failed in two, because it's trying to import admin which already exists on the D8 site, and it was also trying to update admin which you couldn't do because of, yeah. So it failed twice, and basically created all the 50 items that are there. And we can see that if we go under people on the D8 site, here's all the admin and all the other users that are there. The reason that the create date is whatever is because the dual data used basically generates it for like old records, just fakes old data. Now of course, let's get into the article migration. Article migration is basically a node migration. It's basically called the UID from the user users. It's also going to, is this the article one? Well that's the translation of the articles, yeah. So this is the article migration. It's going to call UID, same thing. It's going to basically say that the language, line code is basically what it is in D8. It was language in Drupal 7. We're going to basically sign the language to line code. We're going to say that we want all the tags to then the tags field in Drupal 7. But because it could be a one too many value field, we're going to iterate over each one and process it individually. So we're going to pass it to the txoniterm migration and get back the TID. And at the end we're just going to map it back to entity node and we're also going to say that this migration depends on these two. Don't run this one, the last, those have already run. So let's run it. It's going to probably say, let's see. Oh yeah, node. There you go, it created two. So if we go into our site, here we have two records. Now if we switch over to friend side, it's just going to say welcome to Drupal because there's no French translations for those nodes. But if you look at the translation migration, the only things that the translation migration has been differently is that it's setting translations to truth within the source. It's also saying that get the node ID for this node from the source migration for this node. Basically, we've imported it already. We're going to grab the node that they created during that migration and assign it to the node in this one. We're also going to say that the language is going to be whatever the translation language is going to be. And at the end we're going to say that content translation source is going to be the default value of EN, which is the language coming from the source migration. And of course at the end there's the destination. We're going to say that this is basically adding a translation to a node that already exists. So that translations equals the truth there too. Okay, let's run that too. You guys see this? There you go. So it basically said I've created two translations for the content that's there. And if you go back to the site and switch over to French, we have the same content but in French. And you can see that it is a translation because we are in this node and we switch over to English. There's the English translation for the node. Okay, and if you go back, and let's say let's try adding another piece of content. Add content, call article. Hello, Drupal North. I can't spell right now. We can add a tag and save. And if you go back to our Drupal 8 migration status, you can see that since we added that one node, we have an additional article that's ready to be migrated over. And if you run migration now, it'll basically import just that one particular node that hasn't been done. But if you wanted to make changes to the existing stuff, you can also pass in a dash dash update and then we'll update the existing stuff too. So if we run this, we see that it created another one. Go back home. There's the hello, Drupal North 2017. Yeah, in D7 site. Oh, am I in D7 site? Okay. There you go. There's hello, Drupal North 2017. And if you switch over to our French translations, you'll see that there's no French translation, so you're only seeing the other two. And if you go back to the Drupal 7 site and actually translate this piece of content, in French. Sure, I know that. And we save it, that's weird. It didn't add it, that's weird, in English. I am using title field, that's weird. There you go. And if we do Drupal Trans, it shows that we import another one. And if we go back to our Drupal 8 site and refresh it, there you go. We have hello, Drupal North 2017, French. So there you go, Drupal Translations. Okay, let's start back in here and we are done, so any questions? Yes. Can we read this module extensively? If you, FID is inside the logic. Yes. What figure approach to that is it like a, I think it's a post-pro access for us that we have to work with, but. You can actually write a process plugin that I see read up, basically. You can write a process plugin to do that. There's also the UGID link module that's also being worked on. But what I've done in the past is I've just written a process plugin for the actual body that basically goes and parses to see if there's any file IDs and stuff like that. If it didn't, then it actually doesn't take the migration lookup for that file to see what the actual file ID is in Drupal 8 site and then just replaces that value. That's a post-pro sense. Yeah. You can do that on post-process too, but. Or you can actually do a separate migration to do that. I think that's probably the wiser way of going about it. To do it on post-process, yeah. Yes. Yeah, wait a minute, where is it going up here? Do you want it on? Do you update only one node? Yes. Yes and no. Yes, that basically use migrations to do that, but you can also run post-process operations other nodes related to it. So let's say you had a bunch of, what's content that relates to other content? Okay, but anyways, let's say you're basically importing a bunch of paragraphs and you want to assign them to a particular node. You can either have two migrations to do that or you can have like one and you can actually implement an event subscriber. Event subscribers are basically, if you're not familiar with each other, but in Drupal 7, there was like their prepare row, there was stuff to do after the migration runs. So all that stuff is now within something which is called event subscriber. It's a class you implement and basically after the migration happens, you get the formatted destination row back and you can use that to update existing nodes or existing content. To give you an example of that, I do have an example of that. So this is basically the open data migrations that we did for the government panel. In here we implement a migration subscriber. And in this, if you keep on going down, we have constructs, we have... So this is basically on migration post import. So after the import happens, execute this command. Right? So we're basically setting the home page to be something. We're also doing, on migration post rows save, we're saying there's some migration is this. We want to add votes to those migrations. So basically if you have a whole bunch of... We use the voting API and Drupal 7th side and we basically use flag on Drupal 8. And we want to basically import all the votes over and then we want to assign them to a particular node. So we import all the nodes and once the node is saved, we said, okay, are there any votes associated with that node, assign it to that node? So this gives you an example. All this code is on github.com.open-data.od. I'll go back to my slides and say you have all these links. But everything is pretty much there. Do you have any other questions or to answer your question or any questions? Okay, so make sure to come up here. If you want to win a Google home, I can give you some stickers. And also check out our booths. You can pick out one of these things. It's a snap bracelet. Now it's actually a wrist. It's really cool. And it's USB key too, so yeah, work with it. Okay, thank you.