 Good morning. We'll be talking about migrating content to Drupal 8 and like, I did press the button. Okay, so we'll be, but I don't see any animation on the thing. I don't know. Okay. So we'll be talking about migrating content to Drupal 8, the basics as well, and also, like, we'll focus mainly on multilingual content migration. So let's get started. This is me and my dog, Ryder. I'm Jigar. People also call me Jerry. I work at Evolving Web. I've been working with Drupal since 2013 or so, and it's been great. So these are some of the clients we worked for at Evolving Web, and I have done a lot of migrations. Okay. So what are we going to discuss today? Okay. A brief introduction to migrations and what they do. So what migrations are about. Basically, if there is somebody who has never heard of migrations, how many of you have used migrations? Okay. Quite a few. And so we'll just go through the basics first and then jump to what's so special about migrating translated content. And we'll focus on, like, we'll have two subtopics there. Like, in Drupal 7, we had more than one mechanism for managing translations. So we'll check out the core way of doing translations in Drupal 7 and how to migrate them to Drupal 8, the content translation module way. And we'll also check the other way of doing it, the entity translation module way. And migrating. And we'll also take a quick look at how to migrate, like, approaching migrations in general when we're dealing with other translatable entities, like custom things. I remember having built a football ground entity back in the days so we can migrate football grounds. Okay. So introduction. As the slide says, Migrate API helps us migrate data from a given source to a given destination. So we can, in Drupal 8, this has become really nice. We have separate plugins for handling sources and destinations. And basically what migrate does is we can specify, like, this is the data source and we want to take it and create nodes or create taxonomy terms. And we tell it and it does exactly that. And it has become, in my opinion, very, very organized in Drupal 8. In Drupal 7, we used to have to write classes and do a bunch of stuff. But in Drupal 8, it has become, like, one can write a migration in a matter of 5 to 10 minutes. And it magically works, like, and apart from that, like many of us might have thought, excuse me, why use migrations? Why not, like, do some custom coding and read a CSV file line by line and then import the content into Drupal? So initially I used to do that. And as I usually say, like, I'm a big fan of progress bars. So I use the batch API and make them work and used to do it that way. And then I figured out, like, migrate does a bunch of, excuse me, other things as well. Like, migrate maintains a map of things, like, where the data came from and where it goes into Drupal 8. So if, for example, you do a migration and then your data in the source changes, so migrate would be able to magically know that this particular thing has changed. So we have to update it in Drupal 8. And that's something, if we were to try and make our own module, it would be really hard to, like, do something like that. So migrate basically provides a bunch of tools which are, like, which we should not try to reinvent, in my opinion, because it would be a waste of time. And so, like, the third point says, like, we can concentrate on mapping the fields. Like, this data becomes this. Like, earlier it used to be called heading. Now we call it title. We put it in the node title. So that's basically what we focus on instead of trying to, like, figure out how do I get this, and so it's much better. Okay. So let's take a look at how, like, the scenarios where we might be, like, wanting to use this, the migration thing. So we can use migrate API and the migrate family of modules to, like, read data from CSV, JSON, XML, and any other, like, similar sources. And we can, like, pull it with ease into our Drupal 8 sites and make them into beautiful nodes for taxonomy terms. We can also use migrate to pull data from older versions of Drupal, because Drupal 6 is now, like, I don't know what term I could use. What do they say? End of life. I was going to use that, but end of life is better. So we can migrate data from Drupal 6, 7 into Drupal 8, and even Drupal 5, but we might have to write some custom code, but that's what we do. Import data from other content management systems, like WordPress and Joomla, if you want to. And the last one is a joke, but I'm not good at jokes, so I'll skip it. And, okay. So basically, like, when starting, like, thinking about I need to migrate something from some source into Drupal 8, what do we, where do we start? Like, first we create a custom module, that's what I do at least, because there's an alternative way of us writing config files directly in the config sync directory, and then we can use rush config import to pull them in, but that's, like, if you want to keep it clean, we make a module, and we put our migration files, YAML files, inside a config slash install directory inside the module. And when we install the module, all the migrations get, like, imported into the system, and they work. So, write YAML files to define migrations. Say it's the slide. So, yeah, like I mentioned once before, in Drupal 7, we used to write classes to define the source and get the data and map them, but in Drupal 8, we can easily write it with YAML files, which is much easier. And we don't have to have, like, someone with lesser technical knowledge or someone who's new to Drupal can also write them. For example, I started using Migrate last year, and I have, like, used it for a lot of stuff, and it's quite easy, so it's not hard, especially if you can write it with YAML files. It's wonderful. Okay, so, yeah, and for now, we do not have a UI for Migrate as such, so we do not have any progress bars. But we have to use Drush commands to see the status of the migrations and other stuff, so I'll just go through a few quick Drush commands which are really useful. The first one will be the Drush Migrate status. When we run that, we get to see a list of records which exist in the source and a list of records, like a number of records which has already been imported into the Drupal 8 site, and we also see, like, figures saying this many rows are unprocessed, so they need migration. Maybe some records were added to the source, so that's how it goes. And so Drush Migrate status, you'll be using it a lot if you do migrations to Drupal 8. And the second one is Migrate Import, so you can, this is the actual command which pulls data in and does the actual migration import, and we also have something called Migrate Rollback, so if you think, like, the data I got in is I need to get rid of it somehow, so let's roll it back. So all the records that got migrated will be deleted from the Drupal 8 site and you'll have a clean slate again, you can start over. And these commands usually have these parameters as well, like dash dash group equals dash dash tag equals, so we can, like, assign groups, like group the migrations into categories, and then we can run them together as per our requirements, like, say, we have a bunch of data about programs, academic programs, so we can group them with a tag called academic maybe, or maybe a group, and then we can do dash dash group equals this, so the command will only work, like, we'll only import a particular group or a particular tag, and the last command is quite useful, dash dash update, what it does is, like, it tells migration that the records which have already been imported, we need to update them as well. So, by default, what happens is every record is checked in the source, and if the source has changed, if the source record has any kind of changes, Drupal imports it, and if it has not changed, it ignores it. So it checks a row hash and does some stuff to do that. So by using dash dash update, we tell Drupal, like, we don't only want to create the records which are missing, but we also want to update the records which have already been imported. So maybe we imported something, and then somebody made a change, and we, like, and then they say, like, I don't want the change, I want to roll back to the thing that it was in the file, like the source. I want to re-import and override my changes so we can do a dash dash update, and those changes will get updated. Okay, and, okay. So now we jump into the code. So this is the basic structure of a YAML file that we write for declaring migration, and it is quite simple. I usually think of it as two to three parts. So this is the part where we define the metadata, and we basically tell the migrate system, like, hey, my migration's ID is called program data. So, and we can give it a label as well, so if you go to structure and migration in the UI, you can see a list of migrations that exist, and enjoy the view. But personally, like, labels haven't, like, if you are rolling back and importing, label doesn't matter much. You don't get to see it much, but maybe when we have a UI, it would be of much use, more use. Okay, then as I said, we have migration underscore group for grouping the migrations and putting them together in small groups. And we can also have tags. We can have multiple tags so that we can run a particular set of migrations which have a particular tag. And after that, we see a very interesting group which is called migration dependencies. So this is very important because sometimes we have scenarios where we want a particular migration to run before some other migration. Say we are trying to migrate a list of articles and we are also trying to migrate a list of users who made those articles. So when we are importing the migration articles, we would require to reference the user data. Say in your Drupal 7 site, there was a user called foobar with the ID 13. And we migrate that data and when we are migrating the articles, we want migrations to, we want to tell migrations, hey, there was a user called ID 13 in Drupal 7. We don't want the user ID of the new article, Drupal 8 article to be 13, but we want to use the new ID for that particular user. So you did one import. Please use the ID of the corresponding Drupal 8 ID for the user 13 and assign it to, as the author ID of my new article. So in that scenario, while importing the articles, we actually are using the data which came in through another migration. So that's a scenario where we have a dependency. So the article migration depends on the user migration. So by specifying this migration underscore dependencies thing, we tell migrate that we need to run the user migration first and then please after that, we will run this program data migration so that we can use that data. So it just orders things up. It's like module dependencies. So if you want to enable a particular module, you need to have a bunch of other modules installed. So it works in a similar way. So that's it. Now we go to the second group of set of configurations which we write in the migration definition. It would be the source. And here we define a plugin. Like where does the data come from? The data in this example will be coming from a CSV file. So we use the CSV plugin which comes with a module which is called, if I remember correctly, it's called migrate source CSV. So you have to install that module and you get this plugin called CSV. There are a bunch of other sources as well. Like we can read from JSON which the migrate plus module provides and a bunch of other stuff like XML as well. And it works really nice. So data sources. We tell them like this is my data source. We give it a path so that it can read the source data. Like get the data from this particular path on my file system. We can also use HTTP locations so that the data is read from remote sources if we want. And every source plugin has its own bunch of options like the CSV plugin gives us an option to have defined. This many rows are header. So that's not actual data. Please ignore it. So for that we do header underscore row underscore count equals one and it will ignore the first record as being headings. So it won't create a node out of headings. And that's good. The next, this keys thing that we see, the keys are really important because that's what tells migrate how to identify uniquely every record that we have in the source data. So in some plugins this particular property is called keys. In others it's called IDs. So we need to be a bit careful when we use the plugins. Like the other day I was using the URL plugin which comes with migrate plus module. And it calls it like IDs. So whatever it is like the basic concept being those, the things that we mentioned in keys or IDs are used to identify all the source records uniquely so that my rate knows which source record has become which node or taxonomy term in Drupal A. So that's what it is used for, for tracking the source rows. And then we have a bunch of field definitions where we simply like add some descriptions so that we can see it in the UI. And then we have constants, yeah. So constants, we'll discuss them in the next slide when we take a look at like it's something that we, I've seen many tutorials which use the default value plugin for getting data which is actually not coming from a source. So say for example you have a particular set of articles which you want to migrate but the article data you don't want to import any of the user IDs. You want all the articles to be created in Drupal A as if they were created by user one. So UI D1 is not in the CSV. So how do we bring it in? People use the default underscore value plugin but I found that like using constants is also a good way. I think it's faster than default value in some way because default value is a plugin, I don't know. So but constant is a nice way like you can define a bunch of constants and then we will assign them like constants slash the name of the property and it will get assigned. So we don't depend on the source data for that. Like we see in the UI D, we have constants slash UI D underscore root which we defined as one. So it comes from the constants as opposed to coming from the source data. And we have jumped to the next section of the migration file which is the like process section. This does exactly what the name says. It sets a bunch of processing instructions for the properties that we're willing to migrate. So for example in this slide we see title equals heading. So we are telling migrate like there's a title property for my node which I want to set as heading. Like the heading column of the source CSV file will be assigned to the title property of my Drupal 8 node. And we also do something similar with the UI D as we discussed. And so by default there's a plugin called get which just simply gets the data from the source and assigns it to the destination. But we can use like if you have ever worked on migrations or like if you have been a programmer you know things are never simple. Like somebody tells like the client says move this chair from there to there. But then you realize that there's like a bunch of like maybe you have to cross a river to get the chair from point A to point B. So when we have to cross a river we also have to use a boat. So this is the boat. Like we can massage the data, source data and process it the things that we used to do in the prepare row function if I recall correctly in Drupal 7. So we massage the data and then we can assign it. So as we see for the status what we're doing is calling the str2 lower function on the source column which is called published and then we are using like if the source data says yes then we mean one. If the source data is no as a stream then we change it to a zero and then it will be assigned to the status. So we can like do a bunch of stuff like we can add more plugins as well. So we can it's called plugin pipelines. So our source data goes through a bunch of processes before it gets assigned to the destination which is very useful. And it's quite easy to define a custom process plugin as well. Like you only have to create a class and define a method and it gives you a bunch of flexibility. You can do a lot of things with that by writing a transform method inside your own process plugin and it's great. So basically this is what we do to get the data like assign the data from which one goes where. This is what we do here. And the next section would be destination. I think I should have put this one before the processes but it's okay. The destination plugin again the destination section tells the migration. We got the data, we processed it but what do we want to do with it? So this is where we tell Migrate. We want to create node entities with the source data that we pulled in and if I do not mention the node type in my processes by default make them a program node. So this is what we are doing here. And as a matter of fact like a few days ago I figured out there was some update or something I don't know which made it like trigger the errors but the plugin definition that you see the entity colon node has a colon in the value so it's a good idea to wrap it in a set of quotation marks so that it doesn't break things otherwise there is something which starts complaining and tells you like there's a colon value so you will get a white screen with an error message PHP so be careful about that. Like if you're using, if your value in YAML starts with an at the rate sign or if it has a colon in it then try putting it inside single or double quotations to make sure that your code doesn't break. So yeah, so those are the parts. We have a source, we have a destination and we tell Migrate like what we have to do with the properties before assigning them to the destination. So that's pretty much all we do in defining a basic migration. So that said, we will jump to the next topic which is Migrationis multilinguis which means multilingual migrations. So what's the big difference between a normal migration and a multilingual migration? First, just read it. We migrate things in the base language. So while migrating multilanguage content first we do a simple migration which only gets the base data. Like I'll explain the concept now. So when we're building a migration we usually like as we do while we are learning a language or something. So I'll give that example because I'm a language learning enthusiast. So we first have our mother language which I would refer to as the base language in this presentation. And according to that, first we download that so we already grew up with the language. And then we want to have translations of things like how do you say chair in some other language? We just relate it mentally like chair in English equals seizure in Spanish. So we have a base translation and then we build upon that and attach the other translations to that. So if you suddenly start learning some other language you associate that to your base like mother language. And so that's what we do here. First we get the content in the base language which usually like we tend to use English in the majority of the projects which I've handled. So we have the content in this example as well. We'll treat it as if English is the base language. So first we get the data in the base language and if it doesn't have translations then we'll just import it as is without worrying about any translations. And the next step is where it gets a bit complicated but not too much. After having the base language translations we get the translations. So first we get the English data and then we ignore the English data and get only the data in other languages. So say for example we have a site where our data source has like we have a data source which is in English and in Spanish and we want to treat English as our base language. So first we get the English data and ignore the Spanish data and in the next step we write another migration and we do exactly the opposite. We ignore the English data and we get the Spanish data. So that's pretty much what we do. So we'll take a look at an example to get a better idea of what's going on but before that we'll discuss a bit about the two different methods as I mentioned of migrating things from Drupal 7 to translating content in Drupal 7. So as you can see on the slide there's a core way of doing translations in Drupal 7 which is called content translation. And the other way is a contrib module which is called entity translation. And in Drupal 8 we use moreover the way that the entity translation in Drupal 7 used to work but it's now in core so we have a standard way of translating everything. So okay and we'll start with the content translation way which is the core way in Drupal 7. So as of Drupal 8.3 if I'm not mistaken support for the content translations is built into Drupal core. So you don't have to write any classes or you don't have to do anything like which a developer does to do like D7 to D8 migrations with the content translation thing. So we'll take a look at how the content translations work in Drupal 7. In Drupal 7 when we create a base node like a base language we get a node ID like we see the first record there our node ID is 20 and the language column says English and the title is in English. So that's the node which came first. And after that when you go to the UI and say I want to translate this node and make a Spanish translation for the node so when you do that and you save the translation we get another node which is the ID 22 in the slide and as you can see in the language column it's Spanish. So in Drupal 7 basically every translation is a node on its own if you're using the content translation module which comes with the core. So if you make another French translation of the same thing you will get one more node with a different node ID and so this is how it used to work so for migrating this thing we first migrate the base data so for that we write a simple migration like we see here programs in base language as I labeled it and the plugin that we use the source plugin we use the D7 underscore node plugin which comes with the migrate Drupal module and we tell it node type article so what this means is we want to read all the article nodes from the Drupal 7 database and we want to do something with that and I have a yellow line with the highlight it looks a bit weird on there but it's yellow actually so translations are supposed to be false at this stage we can ignore that line we don't need to do anything special at this stage because by default the D7 node plugin migrates only the base content like I say like the base translation so all the nodes in which came first like English and the nodes which do not support translations they will be migrated if you do that and then we see the destination section I also have the translation false thing and it's striked off so we don't need to do that here we don't need to mention it and if you want to mention it, mention it as false so what this means is that we are telling Drupal 8 for the nodes that we are trying to migrate we want to create fresh nodes out of every Drupal 7 node we are not dealing with translations these are just nodes, make them nodes so for every record that we read we will end up with a new fresh node in our Drupal 8 site which will not have any translations after having run this one this is the first step and the next step is migrating the translations and attaching them to the base nodes that we migrated in the previous migration so in this stage as the label says program translations so we are migrating the translations and we do mention the translations as true in the source and the destination so when we mention translations as true in the source it means like we are telling the Drupal 7 D7 underscore node plugin that hey, I'm not interested in the base data but I only want the translations so every node which came into existence as a result of translating some other stuff will be read so English data like the base data that we were talking about will not be considered in this migration so we will only get the Spanish and the French and other languages known here since we mentioned translations as true in the source plugin and after that we see the destination area where we have to the right hand side I'll skip the processes for now and jump to it after the destination so in destination we mention translation equals true so what this does is like it tells migrate we do not want to create new nodes for these things that we are trying to migrate but we want to attach this data as translations to nodes which already exist in the Drupal 8 database so for example in the first migration we migrated a node with the title German Shepherd Dog so here when we read the Spanish data we are telling German Shepherd Dog is the original article we want to associate this Spanish article which is called Perro Pastor Aleman and we want to attach it to that node that we translated before migrated before so that's why we mentioned the translations equal true so I'll repeat this once again in the first step we do not do translations equals true we ignore that line and in the second stage we do actually set translations equal true in the source as well as the destination and apart from that there's a tiny little change in the process area everything else remains the same in the processes but we actually specify a node ID which we usually do not do in migrations because that's weird but we specify the node ID here because as you can see we are using the migration lookup plugin how many of you know about that like what it does does anybody know what the migration lookup plugin does? okay so since the first one since we migrated some data in the first migration and we are interested in referring to that data the data which came into existence as a result of the first migration we use the migration plugin which has now been renamed to migration underscore lookup so what that plugin does is we give it a migration name like it says migration program underscore base so the plugin will actually go to that particular migration data and it will check the records there and the source TNID, the line that you see it will use the property which is called TNID lookup the database and it will spit out the Drupal 8 node ID for the content which got like migrated so I'll just jump back a few slides to say that English node has the ID20 and it gets migrated into Drupal 8 as node number 3 for example so the migration lookup plugin will actually go referred to its ID map or whatever it does behind the scenes and it will pull out the new node ID that it got in Drupal 8 and it will use it as the node ID for the translation that we are migrating so what will happen is all the data that we pull in using this migration will be attached to the node ID 3 as we got in Drupal 8 so the migration lookup plugin does that it looks up another migration and it translates the old ID into the Drupal 8 ID so that we can do cool stuff with it so this is why we are specifying the node ID here so at the end of the day when we run this migration what will happen is we'll read all the non-English, non-based data from Drupal 7 we will look up a node which like look up the corresponding Drupal 8 node for that data and we will attach all this data which comes from like the translations as actual translations in Drupal 8 so we won't be creating new nodes we'll be migrating the data and attaching them to existing nodes so that's why we do all this NID migration lookup and all that it's easy like if you start doing the migrations you'll know that it's like how it works so it's a good thing and so after having run this you get all the base nodes and you have all the translations attached to those nodes so we are happy programmers because we just migrated everything from Drupal 7 to Drupal 8 and any questions so far? like, yep? Drupal 8 node table actually I'll show you a node table as it would look with the NTT translation module in Drupal 7 and in Drupal 8 it looks more or less similar because it's the same logic that we use for managing translations in Drupal 8 so that's the next section I'll tell you, I'll point it out okay so I will jump to the next one okay so now we switch to the other way of managing translations in Drupal 7 which has been adopted in Drupal 8 core as the default way of managing translations so the difference is I'll actually skip this slide and go to the database table extract that I have so that we can like if somebody wants to read this the slides are available on I don't know where I put them I have Google Drive I think so that's it okay so NTT translations module this is a glimpse of how things work behind the scenes so first when we create the original node of a German shifter dog node as I always prefer so we can see that the NTT type here in this table okay let's talk about how I got this table we install the NTT translation module and we do some translations for anything that we translate using the NTT translation module the module behind the scenes has this particular registry where it maintains a record of all the translations that exist so as we can see NTT type node NTT ID 1 in the first row and language is English and so that came into existence the first row came into existence when we created our base node in English and as we can see the source is empty because there is like it came into existence on its own it isn't a translation the source column is blank but if you go to the next row when we create the Spanish translation for the node we get a new record NTT type node NTT ID 1 and language is ES and the source is English so what this means is somebody used the English version of the node translated it and we got this Spanish node was born so that's what it means and the same for the French node if you see the third one and I wanted to like say German Shepherd dog in French as well but I forgot how it said so the third row is the same way the source is N the language is FR and the NTT ID is 1 so the new thing that we see here as we like we have only one NTT ID as opposed to the other method that we saw so there's one node which has a bunch of translations so which is more logical to have instead of having various nodes for maintaining various versions of like language variations so this is the main difference that we have in Drupal 8 as well so in Drupal 8 when you're translating content no, he asked about Drupal 8 8 minutes left okay we're close to the end okay so the thing is like in Drupal 8 as well we have this thing we have one node and then the translations are attached to that node so we get only one node ID and the translations are maintained somewhat like this in the fields table or some node field data table and all that so this is what happens so for the bad thing about the NTT translations thing for now is when you're trying to migrate something which was translated with this module in Drupal 7 you don't have an out of the box solution in Drupal 8 so we have to do something like some weird stuff like write code so we have to write a migration source and in the migration source we do a bunch of stuff like I named my source plugin as D7 node NTT translation and in there we write a bunch of code to take into consideration the NTT underscore translation table so that we can consider the translations nicely and we write this plugin to basically create a provision for the translations property that we have that we used in the first migration translations true or false so that's not supported for NTT translations yet so for that we have to write a custom source plugin like this I have a blog about it as well so if you want you can read it on our site evolving web.ca because I'm running out of time for this one because I went into too much detail and so we basically consider this table we do some inner join and we tell migration that all our source data is not only identified by the node ID but also consider the language parameter to identify the rows uniquely so and we do some other stuff as well to read the source data in the correct language which doesn't happen out of the box but I'm trying to write a patch and after you do this just use this particular source plugin that you wrote as your source plugin and do the rest of the things just the way you did in the other like content translation module migration so use translations false first then true and you will have two migrations and everything will work okay so this brings us to the end of the main topic and we have some tips for migrating other NTT types as well so what we do is basically the same thing first get the base data then get the translations in another migration so that's what I'm trying to say there okay I think I'm out of time is that what it means okay Austin Powers is sad okay so I have a bunch of slides a slide like the last slide is a bunch of links where you can read more about it and you can always go to evolving web.ca where I have now I think I have more than eight articles on migrations and you can go through them and leave comments if you have any questions and that's it and try doing some migrations if you want to improve like I always use this example I tell people I can swim but I can only swim theoretically because I read some Wikipedia articles saw some videos on YouTube but I've actually never been in a swimming pool so you cannot be like that if you want to do migrations you have to do migrations and then you improve like I will be doing the same thing with swimming so that yep and that's it merci beaucoup thank you