 10 pages, so if you want to check them out. As I said, I'm Oliver Davis, I'm OP Davis, very much everywhere. I'm Drupal.org on GitHub and Twitter everywhere. I work as a senior developer for Atmavation. I'm a Drupal call contributor mentor, I've been taking various contrib modules. I'm also the co-organiser of the Drupal Bristol and PSP South West user groups. I'm one of the family and co-organisers of Drupal Camp Bristol, which is a little bit relevant, because this talk talks about how we migrated the previous Drupal Bristol user group site onto Drupal 8. So a little bit of backstory. Drupal 6 to 7, to go from Drupal 6 to Drupal 7, there are a few different options. The first was there's an upgrade path in call, so you take your own Drupal site, you'd copy the new files onto the old one, you'd run up date.php and it would do all the database updates behind the scenes. Essentially, your old data is still in your database, so it's not removed, it's not clean, it's still sat there. I worked on Drupal.org, I used to work with the Drupal Association, and there were still database tables there for I believe Drupal 4, back from when the original site was built, because it'd been upgraded in place every time, that data is still there. There's a lot of crap still sat around in your database. The other option is to do a migration. So for Drupal 7, there was no migration in call, so it was provided by one or possibly two migration control modules, the main one being aptly named Migrate, and the second one was a helper for Migrate and also called D2D. So for Drupal to Drupal migrations, particularly help with upgrading from Drupal 6 to Drupal 7, for example, whereas if you're upgrading from a custom database, another CMS or custom thing, as we did in this case, the Migrate module is fine. It takes a different approach. So you start from scratch, you start with a completely plain Drupal site, you build it all up again from the ground up, and you import your data into it. That gives you the option to only migrate the data you want to migrate, not everything, and you can also perform various transformations on the data while you're doing the migration. So essentially both the modules provide base classes to extend, various things like this. I think a class actually called Migration, and the idea is you write your own custom module, you extend the base class in your module, you tell the migration how to work, and then you run it. The Migrate D2D module sort of provided a layer in between that, so essentially you'd have a Drupal 7 term migration class would extend the one that the Migrate module gave you, and it would do some stuff for you. You'd extend that and then just do the bits you need to do, rather than have to do everything. So essentially configuring the migration, this is how we have to do it in Drupal 7. So in this case we've got a module called My Migration module, and you'd have a file that's the name of your module.migrate.inc, you'd have a function that would implement the Migrate API, and then you'd have this huge array of information that you're passing into the migration. So in this case we're specifying the API, so the Migrate API version 2. With defining groups for our migration, this was taken from the Migrate module example, so there's a lot of beer and wine references, but not mine, honest. This continues on. As well as your groups, you've got a array of actual migrations with class names, group names. I think we all agree this is, especially this one, quite verbose, a lot of text. Most of your, as you can see, basically all PHP, a lot of quite descriptive, but fairly long method names we have to use multiple, multiple times. So essentially something of migration in Drupal 7 is a lot of verbose code, a lot of nested arrays because Drupal, and yet again a lot of descriptive, but fairly long method names. So in Drupal 8 things have changed a little bit. There is no upgrade path, so you can't take the Drupal 7 site, put Drupal 8 on top of an updated PHP, that no longer exists. The Migrate module is now in call. The Migrate D2D module has been renamed to Migrate underscore Drupal, and there's also a Migrate Drupal UI module. There are still additional modules in contrib, so Migrate Tools is one, provides some of the backport and functionality, essentially, from D7, and also there's another one called Migrate Plus, Plus Accent, the thing. Tools essentially adds various drush commands and things you would have been used to if you'd have done a migration in Drupal 7. And rather than having all the verbose PHP code all in classes, there's still PHP classes, but we're also using technical annotations, which essentially are comments that actually pass and read as code, and also using YAML configurations because Drupal 8. So, if you look at the Drupal 8 site, you'll see this on your module screen. You'll notice that they are within a group called Experimental, which means they're not technically stable still in call. And I found this, when I initially did this migration probably a year, a year and a half, maybe two years ago, things have changed during that time. Essentially, experimental modules are modules in call but don't adhere to semantic versioning particularly. So, they can be breaking API changes within minor versions, which isn't normally how semantic versioning works. So, that's just something to be aware of. So, in my particular use case, I had to build a source database because I didn't have one. We were migrating from a project on scalping. Anybody familiar with scalping is a project. You guys I know are. So, essentially, scalping is a static site generator. So, people are familiar with Jackal or Octopress. It's along that line. It's written in PHP. It extends various symphony components. So, essentially, what I ended up doing in this case was I extracted the data from YAML. It stores it's configuration in YAML and then created a separate database based off that data using a custom script with a few more symphony components again. What I ended up with was a database with a venues table, an events table, speakers, and talks. So, this is how it used to look. We had a venues.yaml page which had a venues key and then each of the venues were listed. You'll notice where we've got SIFT and we've got Proctors. Those were essentially RIDs. In this case, we just stored the name, essentially a label, and on the website to link to. I got transformed into this. So, I used incrementing IDs for 20, 20, 90, 30, etc. The name got mapped into its own column and the website. I also kept the old ID so essentially the key string for reference, essentially. Likewise for events. I've done the thing. I've got an events key with a list. It contains the titles, the dates, the link to the GDO group and the location of the meetup. Again, you notice that these keys referenced the IDs that were set in the previous slide. This is the resulting table. There was some magic that happened in the background and you'll notice that venue IDs match up with the IDs that were created in the previous table. Nice relation of database. From the database, I could tell which events happened, where, what talks happened to what event and which speakers gave which talks. We had a source database. We had to add it to our Drupal configuration. This was in my settings.php file. This is the fairly standard Drupal VM settings file with Drupal, Drupal, Drupal everywhere. So essentially I had a second database. This one was key with Migrate. That was Drupal 8.0, knowing where do I get the data from in order to import it into the Drupal 8.0 site. All I had that, I had to start writing my custom migration. So I had to write a custom module. This was the structure of said module. So you notice that there's an info in the animal file. If you're not used to doing Drupal 8.0 modules, Drupal 7.0 had dot info files. We declared information for our module. Drupal 8.0 is info.yaml files. There's no dot module file, because we just didn't need it. It's optional in D8. There was an install file. So you notice that the top, we've got some configuration within the config directory, the install directory. Those are custom migration plug-ins that get installed around when the site of the modules are installed. So, within the install file, we've got the uninstallers for them. And then within our SRC directory, that's where the PHP code knowledge has created four different events sources. So each one, one for events, speakers, et cetera. So in that case, the bottom half is fairly similar to Drupal 7.0. If you ignore the file structure, everything in the top half is pretty different. So this is my info.yaml file. It's also got the name, and the description is a fairly familiar for doing Drupal 7 development. You can just specify a type, in the case it's a module, as opposed to anything else. It's Drupal 8.0. Put it under Drupal Bresel package. Again, very similar to Drupal 7.0. And I'm defining some dependencies. So we're going to depend on the migrate module provided by Drupal Core. I'm also going to rely on, depend on the migrate plus and the migrate tools modules. So the first thing I need to do is create a migration group. This is one of the things that changed between the first implementation and the more recent update to it. So in this case, this is provided by the migrate plus module. So you'll notice the file name include migrate underscore plus and followed by the key migration useful group with the name of the group. So Drupal Bresel got the right amount. And this is just a very bare bones implementation. So I've literally given it an ID and a label for other options that you can pass. These are the minimum I can get away with essentially. So now we've got the migration group to create a migration source. So this is the part where we're interacting with the plug-in API. The plug-ins are given to us by the migrate module. So this class lives with our SLC directory and it lives within it's got a namespace. So that namespace needs to match the directory in the case it actually lives within a plug-in directory and then migrate source and then the name of our class. The class is venue-term. So the file name is venue-term.php and we're going to extend class because we report it from a database for our various others depending on what you're extending from. And this comment block that we see is the annotation. So this is something that's used quite a lot in other PHP projects like Symphony. It's a comment but it's a comment that does something. So it gets passed by the PHP interpreter form for better word and the data gets extracted from it. And it's using the migrate source annotation and we give it an ID of venue-term and we'll see later on where that actually gets used. Within that class is a query method so that tells Drupal how to pass the source database the migration database and we're going to select everything from the venues table the alias to v just do less typing. We get the ID, the name and the website columns. If anyone's not familiar this is using PHP short-racing tags so the square brackets are the same as doing array open bracket, close bracket. Also we've got get IDs method so essentially we're defining primary keys as a way to try and think of it. In this case we're going to use the actual ID the table that was automatically generated by the table and we're defining it as an integer obviously. I could have passed actually the string in but they went to the integer. In this case then we need to tell it which fields we're mapping to so I've got the ID of the venue I'm not sure that was actually using the end but the name was and we're just providing some labels for Drupal essentially to know how to pass it. Again there's a site Drupal 8 difference rather than just calling the T function and do this arrow T just to get it from a class. Function is the same thing. If you've done Drupal 7 migrations you might be familiar with the prepared row method so what's actually like saying is we've extracted the data from Drupal 7 before we put into Drupal 8 we can do some transformation we can massage the data and we can alter it. In this case actually this is what I'm doing originally the speaker ID so if we had multiple speakers giving a talk they were comma separated in one column whereas in this case we're going to map them to a multi-field entity reference essentially so in this case we used to explode the string so if it's 1 comma 2 comma 3 in the source table it will pass it through to Drupal 8 and it's going to raise those values so that's still there so in terms of actually adding the migration these are all done in the YAML files and I said they run on install so they live within the config install directory we can give it an ID label so in this case we're importing the venue the venues will map to taxonomy terms and live within the Drupal bus of migration group we're giving it a source plugin called venue underscore term that matches the ID that was in our annotation so now we know how the two map together and we then need to give it a destination plugin so in this case it's using the entity plugin and it's a taxonomy term we're mapping it to taxonomy and continuing on from that there's a process and essentially this is how we map the fields so we're going to map name to name makes forward we're going to map the vocabulary ID we're going to use a plugin called default value and we're going to pass it the default value of venues so this is how we basically think the Drupal we're using the venues taxonomy vocabulary and then we're mapping the website value to a field underscore website so that's the link modules in core that allows you to have both an address and a title so we're just specifying that it's mapping to the URI within the field the field website field it's being cut off so what you'll see is if you go to the structure page you get this migration that's what happens all the time the migrations thing there you'll see a list of them this is the Drupal Bristol migration group we're then going to open it and see all the migrations in this case most of all the migrations and you can see that it's able to get from what we've defined is able to pass it to migration find out how many rows there are to import so we've got 62 events 9 venues I didn't have the original speakers or talk so I've just got one it tells you how many you've imported how many you have left to process so again if you've done Drupal 8 migrations it's fairly similar and again you just see a more graphical view of all your mappings and things once there haven't noticed it'll run the migration through the UI I will run them through Drush on the command line so I'll show it in a minute but I was expecting to see a big button that says run this migration but I couldn't find one so you're actually wearing the migration the Drush the Drush commands are provided by the Migrate Tools module again if you use Drupal 7 migrations fairly similar you can use a Migrate Status command or MS for short that gives you this output of essentially the same thing we just saw in the UI we can run Migrate Import we can run the migration we can run the migration by group so we can run all the Drupal Russell migrations together we can just run just run so we shouldn't say group we can just pass Drush Migrate Import then the name so venue underscore term just run that single migration or we can just run dash dash all we can run every migration regardless of the group the nice thing about the Migrate API for not where it also has rollback functionality so there are various each migration ethos on mapping table where it maps the original ID for the new ID and essentially you can just roll back and it will delete the nodes if it's created and it keeps track of all the things so the same functionality exists with Migrate Tools Migrate dash rollback for short and then there's also Migrate stop sometimes if you just do a stop it gets a little bit confused it's just a status column in the database it gets a little bit confused you do have to use refat status just to put it back to be pending or not run or whatever the original status is we're trying to do a demo not hosted on Amazon so we should be okay so we've got right this this is my drip late site it's a very small resolution so we can see I showed it to login so the only thing I've done with this drip late site is modules so there are actually let's start from the beginning so there are some content types so I've already added in I've done some basic side building so we do have a venues tab venues venues taxonomy this is currently empty we haven't done any migrations yet and then within structure we do have an event content type speaker and a talk and these are also all currently empty awesome so inside my vagrant box already and trying to remember if I've had able this module no so carrying no migrations just label this installed awesome and then if you run migrate again again ok this is north ok so you can see that we've got our migrations you can just go ahead and just run just run the venues first and you can see it's gone through process nine items so what's on there in the back is just made nine taxonomy terms if you go back to the side we should be able to see there are taxonomy terms send it to go back now we've got that we can run the event node migration this is actually where it's surprisingly quick it's gone through and made 62 items in a matter of seconds pretty cool now we've got that we can go content so there are events so it's pulled in yeah ok so this is actually from the database that I've created so yeah so if I should be able to look at this there we go so these are the four tables that we've got these are all the events that it's pulling through so these are the same as the slides that we saw at the beginning so there are nine nine venues there are all our events and then I've just made a couple of test speakers and talks just to combine say ok yes we've got the event nodes created if you edit those you should be able to see it's mapped the titles I've didn't map the dates on this occasion but it has mapped the thing to the website and also it's mapped the taxonomy reference so it knows which venue it's relating to and so so again let's just run drush.ms let's check where we are ok it's not done that we can run by a great let's run the speakers speaker content ok there's our speaker it's mapped in my tagline my Twitter handle and my IDs and then finally we can run the talk so again we've just got the one talk so go back here there's the talk edit it it's mapped the body copy in we've passed through a strictlyt html as a default value also within the original script it's mapped that it's mapped the speaker entity reference to the correct node it's mapped the event to the right event taxonomy and then it's also the sideslinks and the obey code all mapped across essentially the end of that middle demo title so yeah just a few takeaways I guess the code for the actual migration module is on GitHub so that's the latest one there is one in the Drupal Bristol organisation but this is the newest one with all the migrate plusfixes and things in it so all in all I found it I did this, we had very little Drupal experience this I think was my first Drupal project but I thought I'll try and put this together then all the site building bits I did a few migrations before from Drupal Center Drupal 7 custom source Drupal 7 especially with the migrate tools module those two extra contributors in there it felt similar I was used to having the commands there but obviously quite different you've got the ammo files you've got the annotations and you've got the config entities but yeah for many of us I was able to figure it out probably in a few hours essentially maybe longer and yeah just a point and we're excellent within the original migrate module they had its own set of examples and then the D8 examples module doesn't but the migrate plus and migrate tools module do so they've got some very good examples there of how to put these all together or we'll see if there's three poses as an example quick plug we've got Drupal Camp Bristol and then 30th of June to the 2nd of July business day sessions sprints again and yeah any questions ok I think so what it does do at least is it does store a mapping table so it knows that venue one is equal to taxonomy term three for example so if you were to run the migration again it would know it's already imported that term and then potentially skip it I don't know whether there's a way of just saying import I think oh actually will it do it and update it I think one thing I have tried doing is to avoid doing that and I have run migrations on several which I think if you do run I think if you run an update it will just skip it because it's been already imported well I do think maybe the D7 version had an update flag you could pass it and it would update it so I'm not sure that still exists in D8 in the trip there are also advantages to having almost like a teardown rebuild every night so you can then ensure that your configuration is up to date as well so that is a valid way of approaching it ok there we go so we can it will do an update but it looks like it will take the whole structure like it will import everything again from scratch potentially rather than just picking and choosing well as we saw it went through 63 records within a second or two so I don't think that would be too it's not going to run for hours it's got lots of records ok it's completely horrendous the product resists and I would imagine you would be able to do having done that I think you should be able to use the same method so you set it all up so you know that as long as your source structure doesn't change you can run an update and it has to go through all the records as they will change and it should just update it and then you'll finally in a you can kill the original but we did a huge I mean that was a massive place because my question was actually if migrate tolls is provides the drush does that mean that you can't run migrate because you said there's no view you are yeah I'm not sure how to do that I'm not going to do that I'm just expecting a little button and see I don't see it I don't see anything else so you can set that yeah so if I go into migrations so what I was doing was looking here's the group but then list the migrations and there's nothing there's nothing in there I don't think it's on its widest just when you got around to the experimental still yeah I'm sure I used enough emoji for migrate power and I'm sure that provided exactly what the old point was so I solved the code update a great I'm sure it was something like that I think what that creates is a lot of pleasure the CSV stuff in it the text files and things like that or things like that write that out of the box I'll have to find it I've got my VM on the end I've got my VM on the end yeah so the things I haven't really used the core migrate one is too much I haven't used the migrate the Drupal yet because also this is going from a custom database in so it might be something but in that migrate Drupal UI that I'm just not seeing because I'm not going to my root but yeah yeah so if you want to migrate from Drupal 7 to Drupal 8 and you've got a panel heavy content so you have a node with a title and everything that node is panels and what is the best way to extract those nodes to get them into Drupal 8 do you have to go and join tables in order to get the right result possibly it does if I go back there was a point where I was joining tables and I wasn't joining them but somewhere yeah so this so to tell you this is just a query I think it's the query interface but possibly you have that I'm sure you could just run join methods against and join what you need to get from there so if your page is quite heavy and has different sections and uses different sorts of panels and UI and will that make it a heavy process to it can so I've worked on one of the sites I did migration for Drupal 6 to Drupal 7 that was part of that migrating from a gallery content type and a photo content type and then we used a logical skull in Drupal 7 which had its own entity system and that it had to be quite a fair amount of transformation in order to get it from what Drupal 6 had to what Drupal 7 had and that obviously adds time that will increase the time for migration and migration to accept what I was to run to run this at various points but this was where is it there's the prepare-row which is what I'm looking for I think it's I can't find it there's a prepare-row method that we were extending so I was doing it with the I don't know if I'm finding it there's a prepare-row method it's in there somewhere there's a prepare-row method that I was using to take the comma separated values from speakers to the array and then pass them into Drupal so if you had that you'd do your table joins in the query method to give you the data that you need and with the prepare-row method you can then transform it and say we're going to concat it these two together and spit that out into something else and then map that to let's say a body field but yeah that's going to take that's a really similar technique that we used to migrate from Oracle Drupal 7 to export everything we've joined the tables that need to be joined exported to a CSV then double the CSV separately which was then fed into Drupal 7 just to migrate would you map it all up? yeah when I did do that it was quite a lot of transformation to get to what the other and then I think probably as a result of doing that we'd have to run it over and they'd come back in the next morning and then see the result of migrating it so yeah I think there's a trade-off there between the more transaction you do and the longer it takes to run briefly showed the default value transferred so are those plugins that can be applied on the field basis so that's quite similar to the prepare-row but you can just have them as plugins if you want or is it? yeah so the default value is one that I found when I was going around and trying to find something and I think that came from the migrate tools examples I think originally so whereas before you'd maybe just pass a string into this vocabulary you'd have to use the default plugin there's a whole list of standard plugins that you probably could use in that scenario but what I was doing then when we did the ammo file was just to find our own plugins on top of that so yeah you probably could even do it that way isn't it? yeah I think in that case it was just a case of this is what I'm familiar with from Drupal 7 and it was natural to me to go oh prepare-row method we will sort of use that whereas there may have been another plugin or some you probably could have written a comma separate load plugin and just have that as a separate library essentially and just use that so that probably could do that in either way in multiple ways because Drupal was just the way I attached went about purely mostly based on having done migrations before on D7 and trying to find I guess similar or more natural things to me I guess to you that way so I'll see what we'll find out so default value is a plugin but then we're passing value into default value yes let's see two slides so yeah we're telling it it comes a little bit confusing initially we've got a plugin the name of the plugin is default underscore value but then the next key is then default underscore value and that's where you pass the actual value to give to default value to give to the thing so when you're passing venues as a string into the default value plugin then using that to populate the field and it's not the field but the property migrating files so anything to watch out for I didn't have to do with this migration I did it previously in the Drupal 7 version so I need to speak to that a little bit and if I remember correctly I literally tried to take the file serialize it and copy all its data and make a new file on the other end which was obviously quite resource intensive so the best thing I found for doing that was just to take the physical files off disk put them on to the new server essentially and then just have to skip out that whole process so if you don't bother doing this bit the files are already here and then just map them to the same on the destination side as well so a lot of the cool stuff will be handled within the Drupal to Drupal migrate if it's 7 not for a Drupal 8 it's got a separate migrate Drupal there so it's renamed slightly so a lot of that it should sort of handle or at least give you maybe a starting point for reference if you do need something more custom there is like a map in a username slash how do you find out all the possible options or is there something how do I do that I think I think I may have done it once to the UI then sort of reverse engineer to find out what I'd set it to afterwards if it's probably a better way of doing it if that's probably the way I found it I was fairly familiar with that sort of syntax from doing was it one of these if you're doing body the body field which I did do on the other slides but when you map the body field you have to map it to the format then the value is the actual text and the format is full html obviously full html actually I must speak in some way so I did it this is the actual thing this type of format so I was familiar with this type of thing having done dribble 7 migrations essentially so you're saying we're mapping it to the body field but the value part of the body and in this case then this is using the restricted values plug it in to say the format is going to be a default value of restricted html which is where it got that from when we saw it on the dribble side yeah I think it literally must have been either got it from the examples because the examples are very good I think maybe made the node once manually and then we'll make it back to us find out what they were set to field collection I was looking at field collection recently for a project I think field collection is more or less deprecated to paragraphs now so I don't know where the paragraphs comes without house migrations I've seen html to paragraphs migrations quite cool yeah no and hopefully not have to do it again we've had multiple migrations so if you could have site A migrations and site B migrations and run them together and then take nodes 100 maybe from this side 100 from 100 to 200 from that side into a third side yeah exactly in that case maybe you have I've only got two databases so I've got the dribble A and the dribble seven one but if you're migrating from additional sources you could either just put more databases in put adam in your sayings or PHP and that makes the migrating module aware of them and then in this case maybe sort of the query fields and then you tell it which table to reference so I've called them migrations actually based on having a dribble essentially yeah but it means previous versions it meant you didn't have to carry across the cruft on a better word than your previous sites you didn't have to still have the dribble six tables set the database but yeah for dribble A you have to as far as I'm aware still to start completely from scratch rebuild it all create all the types of hand again and then migrate your data in so you did migration with the scull module what are the issues that you have? just scull expects things in a very certain way and I think it was a case of previously it was just a couple of years ago but I think they were dribble six it was a photo content type and then a gallery content type and then one reference to the other by no reference if I remember correctly so it was just a lot of the this and sort of massage it into the way the scull wanted it to be it was great it went across but how did you saw being upstone migrating it? yeah I think we were running lightly migrations at that point so I was working on something we were taking this was for a fairly well known site within its community to say that way so it was quite imperative that everything works after migration so we ran through it several times and we'd taken the database of life put it back onto a safety box re-run the migration overnight so it was taking several hours and then just yeah a few minutes afterwards but yeah it's a lot of massaging to get it exactly the way scull wanted it to work just to get the peering if I remember right but it was it has its own entity so if I remember correctly so you have to sort of work out how that works and how does this gallery fit within that this photo within that gallery etc well the issue we're having at the moment is you need to migrate a few separate sites that you score module into one domain access instance so I don't know whether you've been through that sitting or looking forward to that no we've gone from a say triple six two concepts into one scull module I'd imagine if you're doing that that might be easier because already in the same the right format to begin with rather than having to get it into the right format I'd imagine much better this is something I've considered before I did the initial migration so I thought I've got the files in the ammo do I want to just write a ammo parser and then take it straight from the ammo into duplicate or do I want to have that extra step of writing the scripts and using it essentially was using symphyllys the ammo parser and PHP's a PDF object so pass the ammo into an array essentially write that into a database so I decided to go down that route because I sort of thought that was going to be the way to approach most of them so this would sort of be a good trial run for how to do this in the future and I'm more likely to do it that way one of the slides we did have is extending an SQL base class so depending I think on what source you get it from you extend a different class at that point and see what it gives you various helpful methods and things to go from there I think the most inherited sub-interface, the most of the methods would be the same which is what it does behind the scenes would be slightly different yeah there's I think yes originally we had these so you can sort of do this and this was the talk migration so yeah the talks you have to have events you have to have so you can before you can do talks this is in the talk mode so this is within this one here so yeah this is within the migration plug-in so after we've done all the field mapping it's the migrate dependencies at the bottom so I seem to remember in 7 there was sort of a hard and a soft dependency so it had to run or not but I think this is just something called required and if you try and run it without having it it will just say this can't be run or it can't be processed or something at that point yeah much I'm assuming you can pass other options other than required so I think there was soft and hard dependencies in this case they had to run so you have to have it's not quite a migratory talk if you don't have A the speaker or B the event of mapping to has to run it before you can do it anymore thank you all for coming lunchtime