 Hi, I think we can start now. Hello everybody, thanks for coming. This is a talk about migrate and well, I hope you learn many things about migrate today. I will introduce myself, I'm Ignacio Sánchez, I'm a Drupal developer at BlueSpark and if you have any questions you can contact me by Twitter, email, Drupalore or even my blog. I'm a member of the Spanish Drupal Association and an individual member of the Drupal Association. I will show you migrating four steps. One is the which requirements we need to start a migration, which is the anatomy of a migration. After that we will see which is the migration framework itself and we finally see some performance tips to speed up our migration. Okay, let's see requirements. Migrate is in core, yeah. There are three modules in migrate in Drupal core. One is migrate, another one is migrate Drupal and the third one is migrate Drupal UI. We only will see migrate. Migrate Drupal and migrate Drupal UI is a module to migrate Drupal 6 or Drupal 7 to Drupal 8 with a button. I don't like that because you haven't got too much control, but it's very powerful. When you install migrate, first question, how many of you use migrate in Drupal 7? Great, and anybody use migrate in Drupal 8? Great. When you use migrate in Drupal 7, you have a dashboard of all of your migrate, of all of your migration with how many nodes are migrating, how many nodes have been migrating, how many are not migrating yet, but in Drupal 8, you don't see that. You install migrate and you don't see any page of admin, you don't see anything, okay? So if it's not UI, if it's not DRAS, how can I execute a migration? You need to go country, okay? There are two modules, migrate tools and migrate plus, and you have to use DRAS and Drupal 8.1 or superior, Drupal 8.2 right now. Okay, let's see an anatomy of a migration. The easiest example. The workflow is you have a source, you process that source and you have your Drupal 8, your destination. Easy. In files, we have four definitions, okay? YAML files or custom files. Those YAML files are your own custom files. And for plugins, you have PHP files that could be or core plugins or your custom plugins, okay? And you have many types of plugins, source, process, destination, builder, and ID map. Let's see the easiest example. In config install, we have the config file, article note for example. We have an ID, our ID of our migration, and you have three blocks. Source, destination, and process, okay? Source is where we take the data. Destination is our entity, and process is which file relates with what other file, okay? Title is title, ID is our node ID. Easy. And this is our PHP plugin. We need to create a plugin that extends SQL base or Drupal SQL base if we enable Drupal Migrate. And with this annotation, tell here article note. This article note is exactly the same as this. Is this one okay? We add our query as usual query. We have another function that our fill. Here we define some fill that we provide to our jammer file or our migration. And get ID's. Get ID's is how we relate all nodes, all data with new data. We configure the connection to the legacy database. Local host, old app, as usual, okay? And with migrate tools, we have drash command, okay? And we can execute migrate status, migrate import, migrate road back, migrate stop, and execute our migration. Article note, we have 128 nodes, execute that migration, status, all of the nodes are imported and we can make a rollback, okay? This is the easy example. With an article, no more fills, the easiest. And for me, it's easier than Drupal saving. We have only two files to edit and our drash commands. Too easy. But let's see the migration framework itself. The most common plugins as source plugins and process plugins. Source, we take the data from somewhere. Process, we place the data, store the data in our Drupal 8. And we have some destination plugins and if you want to see how many plugins are, you can take a look at this directory, okay? Source, here we tell SQL base, or Drupal SQL base, which database is the source. In the source, we have our plugin, article node, and the key and the legacy. Key and legacy are these places, these items of the RI, okay? We define our migration and that's fine. And in the source, which plugin will make the query, okay? As we say before, in our query, we select articles from the legacy database and these five fields. We define these fields here. These fields are promoted to, well, we see later, are promoted to process a block of the YAML file. Get IDs is where we relate the old IDs with the new IDs and prepare row. Prepare row was in Drupal 7 too. And we can modify some data and calculate some data as we want, okay? And destination is how and where we store the data. Normally, we use an entity, node, user, file, comment, normally, okay? But we can create our new destination plugins. But the common is entity node, entity user, entity comment. Need more destination plugins. Search for destination in core. And you will see many, many destination plugins. If you need to see another example. ID mapping. This is very important because the most powerful tool about migrate is this for me. You have always the information about the old row with the new row. The old data with the new data. The node 7 in our KPHP is the node 100 in our Drupal 8. So this table told this to us. And because they have a hash as Drupal 7, that could track changes between the old database with the new database. Okay, process. How we transform each field file data, okay? If in our old database we have a location object with latitude, longitude. Maybe we want to transform it to Drupal 8. We have to add a new process plugin, okay? And here you are able to map fields. Same field in both sides. Modify. Add new fields, calculate it or by default. For example, map fields. We in the query select in the legacy database the title, okay? So in our process block, this title is our column name in Drupal 8, okay? And this title is the column name of the legacy database, of the legacy table, okay? The same value. Title is title. Easy. Default value. For example, in our KPHP we haven't got any content type. We are nodes or articles. So by default we are a default value and the default value is article. Body format, plain text for example. User zero, status zero as we want, okay? Collable. It's another plugin for process. You want to call a PHP function. In this example, it's here at the time. Here you want to use this plugin. This function has to be defined if you put it in the index PHP of Drupal, okay? If this function isn't, you can call it in index PHP, you can put it here. The dupe entity. If, for example, the username could be the same in the legacy database in Drupal 8, you can't add the same user ID. So you can the dupe entity with postfix and entity type, fill. It's an example of Drupal, for example. Migration. For me, it's the most important plugin process in Drupal. This is how you use another migration in your migration. I mean, for nodes, you need a user, maybe you need a file, and maybe you need related nodes, for example, okay? So when you migrate these nodes, the user has to be migrated before. So this is the plugin, okay? You migrate 10 users and the 7 users has the user ID 10, okay? This user 10 creates a node with a node ID 11, for example. So this plugin relates the 10 users with the 11 nodes in our Drupal 8, okay? If you don't use these plugins, these migration plugins, you could have problems with rollbacks. It's very important to use that plugin. Maybe it's difficult to understand when you start doing migrations, but spend time here, very important, okay? For example, this term is a fill, but there is no here. So where we take this data? We create a new fill in our fill function. We define here the terms, and we add here the term itself. In our prepare row, we select in the table terms, our term with condition 1 or whatever, and set source property, our terms with terms, okay? Here you put this value to here, to here, and finally to the migration, to the node migration. Another example, hierarchical taxonomies, the parent term. You can use the same migration to migrate this taxonomy term, okay? This is an example taken from migrate examples that are already in code. Custom plugins, okay? I see many plugins in code, but I need a specific functionality to do that. So you can do your custom plugins. For example, in this plugin, I copy some files from our Drupal 8. Okay, we query those files, create our plugin, and this path name, this path and this name, we take it from here. And in our prepare row, we calculate this file path as to Drupal, and our filename as to Drupal, okay? And transform. I want to show you the difference of transform and prepare row. In transform, you can process some fields, but in prepare row, you can do that. What is the difference? Prepare row is executed before any migration, before you execute any migration. You do a migrate status, and prepare row is executed. You do a drash migrate import article node limit 10, and you have 1,000 nodes. Prepare row is executed 1,000 times, but transform is only executed 10 times. That is your limit, for example, okay? Transform is used when you want to copy some things that you are already doing that migration. I mean, I want to copy one picture, and I want to prepare the path of that picture, or I want to crop that picture, prepare the name in prepare row, crop the picture in transform, because you're modifying things, okay? And you do anything. Here I put an example of print, because you can debug with xdebag, your drash, or your PHP scripts, but maybe it's difficult for some people, okay? It's difficult to configure. So with print, you can see which variable or value has this variable. You want more core plugins. There are many core plugins, concat, extract, flatter, get, and more, and you have the examples all in core, okay? In core, you have many, many, many examples, a lot of examples, and you only have to search plugin and that's a plugin itself, or search for plugin action, okay? You want more. You can take a look at core modules, migrate, src, plugin, migrate, in every module, if it has a migrate template, you have many examples in colon. You can search, again, with migrate source, destination, migrate tags, really, full of examples. I learn migrate with migrate examples and core itself. And I want to give you some performance tips that, for me, are very useless, but maybe it's not your... If you have a little migration, maybe it's not needed, okay? The first task you have to ask you, do I have to migrate everything? If I have a commerce, I have to migrate every product? Yes, maybe yes. Would I have to migrate every order? Maybe I only have to migrate the orders since here and three months before. I don't want to migrate since 2006. Maybe it's not needed. Is this migration fast enough? I mean, this migration spends one hour. Is, for me, okay? I need to overperformance that migration? Yes, I need, okay. We can start by dropping indexes on every insert. If you have index, the indexes are rebuilt. So, at the start of any migration, you drop all of the indexes, and after all of migrations, restart all of the indexes. And this is a kind of slow process, but when you have to migrate a million nodes, this is very useful. Very, very useful. And it's not so much code. You have to open PHP Miami, MySQL Warbench and see which fields or which columns has the indexes. Drop it and start again, okay? Can I have some hardware improvements? Can I migrate our database server to SSD? Can I put it to RAM? In RAM, in TMFS? Can I do write change? Can I use a better CPU? Maybe, if we are in a cloud, we can change our server to SSD, execute the migration and get back, okay? But I think every database server needs an SSD. So, disable functionalities. Can I disable some modules before the migration? Maybe one that creates many hooks or do some things that I don't need. Can I disable hooks and modules? Yes, no. Incremental migration. This is very useful, but it's a bit slower than a common migration. Track changes. When you activate it, every row is hashed and stored in the column hash. So, if there has changed, migrate nodes that, hey, this node changed. I have to migrate it. Instead of migrating everything a thousand times, the same migration, okay? You can do an incremental migration of a million nodes, start on Monday and stop on Sunday, for example. I have to execute every week or with track changes every morning with these million nodes. I only have to migrate or update a thousand or 500. Maybe this is very useful, okay? More things. Multithread migration. When you execute a migration in Drupal 7, it's the same. Trash or PHP executes only in one thread. And if your server, if your front-end server has four CPUs, there are one CPU at 100% and three CPUs are either. I want to use the four CPUs, okay? There is many articles in Internet that shows you how to configure Trash to execute on many CPUs. Two, four, as you want, okay? But I use that. I name it Multidivided Migration. I don't know why, but... The reason is you have to divide your million nodes. This is an example of a previous project that I have to migrate half million nodes with no comments, no comments, half million nodes. And we break our migration in four migrations. For one, two, a thousand, a thousand to two thousand, okay? Why? Because I can execute with an ampersand of more consoles the same migration anytime, at parallel. So if we have four CPUs, I execute. I prepare my migration with three cuts of the database and execute the four at the same time. This time, the database was a bit stressed, so we have to divide in three blocks. But our migration initially took three days and with this solution took only one day, okay? It's many, many times. How can we do that in Drupal 8? We use the same plugin, CMarticle, for example, and we add here two parameters, 400 to 800, and here we take the mean is 400, and we add in the query function, and we bypass, we cut the migration. If you use this solution, it's a bit tricky, you have to take care that there are no more nodes related with between 400 and 800 and one thousand and two thousand. If the same node is used in node 500 and one thousand, you have to take care, okay? You have to see the queries very well because you can get some inconsistencies, okay? And I add here an example because I don't phone anyone here. Here, custom migration is you create a Drupal 7, generate a thousand nodes, and put that migration enable, and you can see if you execute three times this migration, you could see the three CPUs at 100%. You need three CPUs and another more for your database. You can do it in your localhost, maybe it will broke, okay? And for me the most important thing, common sense, okay? We are programmers and we use common sense for many things. Here, it's very important to use common sense. If you smell that this migration could break in some case, or we can improve in another tip, maybe we have to try. Because migration for me is a very critical process because you execute it in the final stage of your process. I mean, it can be wrong or it can spend too much time. If it spends two days in a migration, maybe you could have problems. If you prepare it for hit one button and the migration is ready, it's better for me, I think. And that's all. I think I'll be a bit quicker, but if you have any questions... You can hear me quite loud. When you were talking about dropping the indexes and restoring them for performance reasons, you had them on a pre-important, post-import function, but I couldn't see which class that belonged to when you were overriding that. Where did they live? I don't remember exactly the name of the class, but it's one of the tips of Drupalorg. I can't tell you later. I don't remember exactly the class. The other question I was going to ask is, with the strategy of having four separate migrations with different IDs for performance reasons, does that mean that the migrate maps will have separate tables and then your use of the migrate process plug-in won't be reliable? Well, I think in this tricky solution you have to store the same ID of the old database. And if you have the same ID, you map the old ID to the new ID, the inconsistencies maybe couldn't be happened because you use the same ID. I don't know if I ask you a question. No, I don't know either. Well, I'll have a think. Thank you. Thank you. Thank you for your presentation and the model, of course. I have a question about content which might have a relation to self. For example, I have a list of budget numbers that relate to some other budget numbers, but they are all in the same list. From what I've experienced, I didn't experience any problems with that, but that might be coincidental or can you tell a little bit more about how to approach that? Yeah, I think you could use two migrations. One, to migrate every node and another one to migrate this field itself. So if you have a related article, for example, I think I could use the migrate the nodes and after that another migration to related that. More questions? Okay, you can talk me in the hall if you have any questions. There is no problem. So thank you and it was very helpful for me if you evaluate this session. If you like it, if you don't like it, if, okay. Thank you.