 Okay, we might as well get started on a high note there So I'm Mike Ryan and I was the primary developer of the Contrib Migrate module you know and mostly love in D6 and D7 and We're going to talk today about Where we're going with my grade in eight what what does my grade in eight mean? What's done so far and how you can help us? Get the rest of what rest of the way there? So also on the panel today. We've got Ryan wheel we've got chicks and I never I can never And oh a late addition not on the slide be done friend and Mike and Ella So just a little thought Yeah Yes While you ponder that up just say I'm just going to go very quickly over What the status of my grade in eight is Main thing here is going to be the conversation. We're a core conversation. We're going to converse so Once we're done feel free to line up at the mic and We'll talk about where it's going where you'd like to see it going any questions you have and Some some of the acknowledgements the Just listed a few of the key committers here and I do want to point out that Actually, I I have not Contributed all that much code so far to the D8 effort mainly. I've been working on developing the simple upgrade UI so far chicks it's been the technical lead so far and Many other people have been contributing patches or contributing in other ways Want to call out Melissa Anderson even though she isn't here she did a great job doing project management as we Got started on the project So the basic idea here is that historically Upgrading between Drupal versions was done using the same update dot php used to Update minor versions. It's an update in pick in place and it as Drupal has become more complex that process has become more problematic and It became more and more difficult for a contract modules to keep up and It's in particular d6 to d7 Getting cck upgraded to Fields in core was a big problem. So It was proposed and finally accepted the that our approach going forward word would be a migration approach That is what we do is Rather than run an update process within your production site What you'll do is create an empty Drupal 8 site and migrate the data From your old site old Drupal site into your new Drupal 8 site so Core itself Drupal 8 core is going to support direct migration not not just from Drupal 7 but from Drupal 6 This is one of the advantages of the migration approach that we can leap over versions It's just a matter of manipulating data after all and that's easy And the framework that will be in core will also enable us to in contrib Import from other Drupal versions we'd be able to do side side imports d8 to d8 as you can do today with d7 to d7 And Drupal 5 and if anyone's really ambitious you can go back to 4.8 or whatever And Contrib modules under this new architecture instead of implementing update hooks they will implement migration configuration they'll ship with yaml files describing how To transform their data from their Drupal 6 Drupal 7 versions of the contrary module into the Drupal 8 version and More complex ones may need to go a step further and implement plugins Which I don't think if they're complex enough to need a plug-in for this migration their update hook Probably would have been pretty ugly anyway So Schedule-wise the the plan is that the underlying apis the the classes and base configuration for performing the migrations will ship with Drupal 8.0.0. I Have started working on the a very simple UI for doing Upgrades with no customization and so forth just a straight upgrade from Drupal 6 Drupal 7 that that is in a sandbox now and That right now it's going to be in a separate sandbox in Contrib and we'll when Drupal 8 is ready to go to our C release candidate The decision will be made whether or not the full the upgrade UI and the full migration framework is Ready to go into 8.00 if you saw the keynote today Dries talked a little about the semantic versioning We're going to be using for Drupal 8 which allows us to put off adding functionality, so it might not We could comfortably postpone it to 8.10 if we are not a hundred percent Sure of that upgrade path so the the pieces of The the whole puzzle there is a basic framework for migration that is part of core the migrate module and If you know the Drupal 7 migrate module, this is basically the the the base classes that you find in the includes file and migrate as well as destination plugins for core entities and new with Drupal 8 is destination plugins for configuration That that's one big plus over migrating Drupal 7. We will be migrating configuration such as content types and fields and so forth variables In addition to content Also in core there's the migrate Drupal module and this is the framework that is specific to pulling data in from a Another Drupal instance With the with the core migrate module you'll be able to pull in from Jive or WordPress or anything you can now This is the specific configurations and plugins that Understand what a Drupal 6 database looks like what a Drupal 7 database looks like So as I was mentioning migrate upgrade that is a very simple UI. It's basically prompts for the database credentials for your source site what your source site URL is so it can find the public files and a couple button clicks later you should have all your content in Drupal 8 and Migrate plus that's not that work hasn't yet started But it I think it is critical to the Drupal Ecosystem to have all the functionality we have now with the migrate module available as If not when 800 comes out as soon as possible afterwards and that's so the migrate plus will include all the parts of today's Migrate module that aren't going to be part of core it so it'll be the full UI Which includes the dashboard the field mapping editor and so forth. It'll be additional source plugins your XML and JSON and so forth CSV plugins and Hopefully we I expect that we will be able to take it quite a bit farther in Drupal 8 especially with the configuration API that we have in Drupal 8 that makes it a lot easier to build very flexible UIs So a lot of people who don't have the technical ability to do migration on their own now A lot of those people will be able to do it through the UI There'll be that much less coding necessary and A lot of migration development will be writing the ammo files rather than php code So where are we now we have the migrate module in core? We have the migrate Drupal module in core with the support for Drupal 6 sources and as I said, I'm the Migrate upgrade in my sandbox and as of about two hours ago I've got it working again with the latest core chasing core is so much fun With with one just one little glitch is that you need a core patch for files to migrate but other than that it migrated my site Yes, my my personal site is still Drupal 6 So what's left? Drupal 7 we did Drupal 6 first for a couple of reasons one is that There's a lot of anxiety in the community among people who are still in Drupal 6 I don't know how many of you are following the epic issue about continuing Drupal 6 support but we You know going Drupal 6 to Drupal 7 and Drupal 7 to Drupal 8 as you used to have to do is going to be extremely painful so we want to jump start that and The other thing is that Drupal 6 and Drupal 7 The the data structures are not that different and of course Drupal 6 is a little farther away from where Drupal 8 is now So the Drupal 7 migration should be basically a simplification of the Drupal 6 Migrate upgrade module. It's very bare bones now It needs a lot of testing in different use cases and we need to work out exactly how bare bones It's going to be how much information are we going to present whether we need to add any additional configuration for that process We have not yet mailed down What implementing the upgrade path for contra modules will be like for now You can look at the migrate Drupal module and see how it's doing it but we we need to Document that and start getting contra modules working with it and giving feedback on that process and the developer experience for developing those upgrades and The number one thing is testing there there's My my little simple blog type website is not as sufficient test there's You all have so many different Modules you've hacked your Drupal sites in so many different ways. We we need people to feed Feed everything they can through this and find all the holes that are left to fill and On that note, what can you do test? test test test and I think this is the point where Michael would like to talk a little about Sprinting sure. Thanks Is this yeah, that's okay, so Friday is our big sprint day obviously our community sprint and The focus for the migrating core team For Drupal con Austin the sprinting is the test sprint as Mike alluded to we're really focused on D6 to D8 migrations And a lot of that stuff works But at this point we really need to perform some focus testing to find out You know does every variation of a text area in D6 migrate cleanly into D8 I'm relatively new to the project Okay, there we go, so Something other than text areas then does that that'll be fixed in I guess 45 minutes or so But I'm relatively new to the project So I'm I'm very familiar with or it fresh in my mind the pain points and getting going from zero to a point To where you can test so we've actually written some documentation that you can get to if you that second bullet point the Click the start here link on the imp group on groups dot triple org There's a link that leads you to a page that basically describes Kind of in the big picture what you need to do if you want to help us test and I broke it up into kind of two Big things you can do number one is get your system ready Which is get a d8 clone get the latest version of drush installed get a fresh d6 site up and running I think there's an PHP 5.4 in order to run Drupal 8 once your system is ready at that point You know Find one of us find either you know Ryan or Brent or myself or find us You know in person up in the sprint room. I think sporadically between now and Friday one of us will probably be there Most times Friday for sure will all be there You can find us on pound Drupal migrate Pretty much anytime. There's a number of people who aren't here in Austin who are going to be helping as well But the idea is that we need people who you don't have to know PHP If you can get your system set up to run these migrations find one of us We will help kind of show you how to do it and there's a couple screencasts there to kind of walk you through at the first time But the idea is that we will kind of assign you something to test So obviously a lot of what we're what we're all focused on when we do migrations is did my nodes in my fields come over So in addition to CC, I'm sorry d6 core. We're also providing support for cck basic cck fields or cck core fields as well as the link field So we're testing all of that those things And one thing you didn't mention hopefully if everybody knows something that's different about Migrating core as opposed to different or previous upgrade paths is We are migrating not just content, but configuration as well So you don't have to set up your d8 site with your content types and your and your fields and do any mapping you can basically do a fresh install of d8 and migrate from your d6 site and You know the the goal is to migrate all of your vocabularies all of your terms all of your content types all of your fields all of your field configurations and Pretty much anything else that's in d6 core So I don't know how many migrations are there total right now. There's just about 90 or so Yeah, 90 different migrations one exception though that I got asked at a previous presentation, which is views Views is like configuration, but it's kind of like you might want to just re-implement those right? Yeah, it's not it's not there yet, or will it be there? We don't know so Good I'll add yeah one more note if you've never installed d8 and you do plan to attend the sprint on Friday There's a session in the morning from 9 to noon I believe and there will be volunteers with gold shirts. I'm told and we have Environments that are ready to go that you can download Using BT syncs so we don't have to use a lot of external bandwidth And those people will help you get set up so that you're familiar enough with onboarding to d8 that we can then transfer you over To a table where there's my great people so if you've never used d8 look for some volunteers and then proceed from there and One other thing was you need the latest drush and it needs to be like drush 7 dev So that involves installing composer as well. And so those Mentors will be able to help you get that stuff ready to go because at the moment we do need to use drush to do the migration Of course because this is going to be Drupal 8 It's a product that you're getting the migration path wheels will be 100% supported in the UI, but at the moment we still need drush so Before warned right so I think the only thing we didn't mention is Hopefully if you do plan on contributing and you want to stick with it We do weekly Google plus hangouts every Thursdays at 6 Eastern and I live on the East Coast so I can't calculate time zones because I guess I'm very East Coast centric Yeah, and that's it so the if you're interested in contributing put yourself in one of those two groups and those two last bullet points and Be there and we'll get you going and hopefully it'll be pretty smooth So thank you, and I apologize. I have to run out in a minute. So I apologize. I don't want to seem rude Yeah, a very important man. He's double booked One two more things that I would like to note one is that There is one configuration Very important configuration that we never get migrated. It is impossible to migrate and we never do it And it is your modules You need to enable your modules yourself If you think of it It's a it's a kind of a chicken egg problem, right? You have a contra module the only way that it could run its migrations if it is enabled, right? So for this reason module the module list is a configuration that is not migrated Unlike views which which could be migrated. It's just not it's just not written yet And and and I can't guarantee that it will be written anytime soon Maybe somebody will take the afford. It's a huge The other thing is just the second the other thing is that If you are on the other hand somewhat familiar with Drupal coding Doesn't really need to be Drupal 8 knowledge. We need people who take the results of the testing on Friday and Write automated tests out of them these tests are going to fail But once you have failing test fixing them is very easy. I tell you One more note is that I think maybe this was mentioned briefly earlier But there's a lot of things now that are in core that were not previously in core and like one example being a Lot of the multi-lingual support and there is plans to support all that stuff that was previously in contrib So we're actually grabbing a lot more data than we were previously So there's aspects that will need to be tested in the future as well And for any questions because we are recording Please line up at the mic and ask Well in just second besides this week sprinting this week going forward We will need help testing and Extending the migration system over the next several months so coders The all this migration work is under the migration system in the core issue queue Some of the stuff that's not quite ready to be Integrated into core like the Drupal 7 support that development is going on in the imp sin bot imp Sandbox and the simple UI in the migrate upgrade sandbox there links to those sandboxes from the GDO imp group and Again Go in there Whatever sites you have around sites you've abandoned Anything you've played around with give it a try and if you have any trouble Let us know drop drop an issue in the queue or Paint us in the Drupal migrate IRC channel. I will let me add I will say search to issue queue obviously It's a you know Because there are known issues. I mean like the link link fields I can tell you don't migrate yet from six to eight It's a known issue and there's an upstream patch. That's in core And there's an issue open for it and you know Mike mentioned the file fields, you know, don't migrate yet But there's a lot of little things like that that we know about but we're looking for a chx mentioned is our Mike You know, we're looking to find the holes that we don't know about yet One more patch I'd like to point out that if you set up your d6 instance on your local machine with different database credentials than your d8 Instance there's a patch for that So if you are stuck right at the start, please ask and we'll direct you to that Hey, thanks a lot for great module and So it is quite new to have my great dealing not just with content with with the configuration or the structure of the site and usually we take the opportunity to of the migration to do away with a crap that didn't work and And and just maybe figure out a better destination a better structure While preserving the content that is worthwhile So how separate are these two things in this migrate module? Yes, well Extremely separate one of the big advantages of the migrate system compared to the upgrade system. This is extremely loosely coupled You can Run basically each of these 80 something migrations separately if you want This is this was extremely important that is very easy to change them it is very easy to just run as few as you want Because What we wanted was exactly a scenario like this so that you actually have a very good Start a base it also gives you a good base There will be I suspect there will be very very few people who actually run every migration with the simply why because Very often that won't work for the cases you say, you know, you had the site you built it seven years ago Maybe you want to rebuild it now Yeah, the the migrate plus module in contrib will be your go-to when you want to customize your migration One additional thing that exists that was in D7 But I don't think was as often used as I suspect it will be in D8 is there's a hook that you can tap into that in D7 would affect your registration array and So you'll be able to affect some things that way without actually having to roll your own custom Like migration, you'll just you'll be able to affect some of the settings Thank you I'm hoping you can speak to two popular requests that I get Doing Drupal 7 migrations And if you expect this to continue to be a popular request in Drupal 8 since it's in core The first is migrate so cool Everyone wants to use it to consume everything so oh I have a CSV and I'm gonna consume it every day I'm gonna write a migrate now. Here's my XML. I want to consume it forget feeds that takes forever I'm gonna do migrate. So I want to know if that's Anticipated that that's gonna continue to be a thing and and if there's anything built around in anticipation of that and secondly related the system of record I Hear this a lot that You know would be really great. We're building a new site Would be so awesome if we could just keep migrating every week until we're done so system of record is Changing the destination. Are you anticipating that? Are you trying to put a stop to that? Well the system of record for those who don't know the Contrib migrate module has a concept of Defining a migration with a system of record being the destination that is you take rather than taking the source data as the definitive definition of your node or your user account and so forth and always overriding it completely the Destination system of record in theory allows you to say The Drupal side is the definitive data except for particular fields that you're going to overwrite and the The implementation of that in Drupal 7 Contrib is not very good. It's it's it's pretty shaky So We haven't addressed that yet. Have we no no it definitely needs to be rethought. It's it has its place, but We haven't figured out how to cleanly implement that so what Drupal 8 does At least friend it is But all your data is going to be entity so that's not a problem that it is Overriding only the fields that you specify in your configuration so This is already I feel that this is sold. I think this is actually sold it will just work. Yeah, it just works We need someone to test that Another note for testing that is probably a good segue from here is There the rollback functionality and things like that are not yet there So expect that you will have to re-migrate and when we're testing and you're like, okay, I've added a new node yeah reinstalled Drupal 8 and Then run it again. So keep that in mind We're not really doing iterative iterative migrations at the moment and I think some of that stuff is planned to be in the contrib modules if I Yeah, anything that's not in core will be in contra right right now. I Do a lot of drush SI Actually well that's Let me take the opportunity to take a risk here Well different formats yeah, that's definitely going to be in contra The context of that was though was not the fact that there are different formats, but that is a continuous migration, right? Yeah, and that that should continue to work in terms of you know when there's new content In your source that gets pulled in So Let me show you what I've got for the simple UI which is very very simple This is with Get pull of Drupal core from like three hours ago. So it might be broken if we pull again, but We'll see So we're starting here with an empty site no content Only the standard profile article basic page no fields and so forth So Like the traditional update dot php install dot php the path for the upgrades is at the top level And Oops, I didn't mean to click there note. It is not update dot php. It's just update. Oh In the eight they finally changed that Or are you I was just lost a few days ago when I was like where did the path to get to the migrations go? So it's just slash upgrade right So you get your database credential form. This is actually reusing the install form You know nice thing in Drupal 8 is that we've got an object-oriented form controller and The install system now implements a class that we can override for our purposes here. So Works just like when you install I'm going to install my oh, I should I suppose I should show you what I'm importing my My sites such as it is I've gone from posting about once every two years to posting about once a month. So I'm getting much better about it See that one is how many years old? All right, so credentials and The address of the site to find the public files and the the plug-in the file plug-in will use the domain and The public file directory Configuration from the Drupal 6 site to figure out where the actual files are and import them So it's really really simple two steps first. Let's import the configuration And so this will Get all your settings that are in variables It'll set up your content types your fields your vocabularies Yes, I do and they're the same credentials. So I'm not running into Ryan's problem there But they could be seven line in the body or in fields. Are they image fields or in line images in the body? Well, we're migrating the files from the files table the d6 files table to the file manage table we're pulling the physical files over and The image references just work awesome All right, so we got our config and You know, this is one of the things we want to get feedback From people is how much feedback do you need here? Temporarily we're using the watchdog To log what we've done and very minimally And one thing we know we need to say more than that It's not really useful and the reason for that here is that I don't have The profile module on the d6 site So what we really want here is to skip the user profile migration, but so far so good Let's do the content It doesn't usually stop there Okay, you know a couple missing files. I've never seen a Website of any size where there weren't files that were mysteriously missing when you went to migrate Do do do do I have a question for Mike? So without the drush, I mean the d7 version for drush Well, I'll let you run really long migrations without worrying about memory limits in the batch I think you recommended using it over the batch API Since we have to support the UI in eight Do is there a plan for? really long migrations or Well the it should the UI should work like it did in seven it should do You know break it up into Batches that fit into the you know php max execution time and if memory is running low it should Create a new batch And it's slow it will be as it is in d7 slower than running in drush But it should work regardless of how large the site is But if you got a big site you should be using drush and you and in in the d8 world you'll probably Want the migrate plus module which will give you a lot more tools for selectively running your migrations and so forth Huh, I'd Just a little bit slow this time Here we go Question does the d6 source site have to be upgraded to the latest a minor D6 version. Yes. Yes, and no I This is a complicated this is an extremely complicated question that doesn't have a simple answer How my great works you need to understand it runs Cuverries against your database tables If your database tables Have the same structure and in the case of configuration the same meaning then we are good so Which is entirely possible that you can get away with an order draw ball Because either there were no changes or the changes were in material to us You know, there are changes when a column gets longer Previously we had only 64 characters and now we have 128 That such a change is in material because we do not migrate date Table structures Maybe we added an index to make Drupal faster in material if we Audit a new variable and that's a worry about that we want to migrate Which happened in one of the seven minor versions if you remember that we split the field bundle variable Then you need to be there So the safest is always to go to the latest But we actually put in some effort to make people To let it people avoid it Because the whole point of the exercise is that we presume that people are stuck on sex So there is not a simple answer to this but in short Not always you are safest You can try you can try and most of the time you will succeed To follow up with that. Is there any issues for having a check Kind of run beforehand to look for those instances. I mean, it's obviously defined that you're Checking for certain say like a column name changes, you know, you're checking for those things. Is there any? Thing that or should someone post an issue that said hey, we shouldn't have that I would say just try it out like you're gonna you may have to re-migrate anyway So like keep in mind that you'll have to re-migrate possibly One thing that we do want to highlight with testing is that if you've previously upgraded the site So let's say you have a Drupal 6 site that used to be a Drupal 5 site or a Drupal 4 site There could be some data integrity issues from those past upgrades And those are the sites that are probably going to have the flakiest data structures if at all so If you have sites that you know were previously upgraded those would be really great candidates for testing in your own On your own environments and then posting bugs and explaining the history of the site if you suspect that it might be You know some oddities in your data structures That was a cool example Could you show the granular migrations because these are bundling everything in the both the structure on the content Could you show this the either the UI or in dress where you have broken down in in the individual items, please? We we can look at the logs that lists all the migrations that were run If you were to go into the migrate Drupal folder There is a config folder in there that has a list of all the migrations. So that's another route you could go Oh, we got a got a notice there Maybe we should do something about that But yeah, well, this of course this is in reverse order, but you can see the configurations Basically variables and upload to user profile, which of course failed And then the content you'd see we Had those two warnings files that no longer exist we but then we Overall the file migration succeeded text on me terms So you can see the granularity here and it is actually There there are a lot of migrations To get the reason I ask is because orders sometimes matter like you probably need to migrate first Vocabulary for the user the files of calories and then a content type or Yeah, there's a there's a dependency system. So then you have all that and and sometimes it's wonky like I mean you have the chicken and egg things and the stubs and everything worked out, but Sometimes it just I feel like I need to Select some are maybe the system has organized my migrations, but I know better Because it's not working and I need to either update a migration or do things like that Well, the the migrations are all defined in YAML files the D8 configuration like the the block migration here and The great thing about the D8 configuration System is you can override configuration So, you know, it won't be in core, but the migrate plus module will let you edit The configuration of your migration save it to your active migration and those override the defaults here So here's your dependencies So so you see that the required dependency means we can't run this one until that one runs The equivalent of the old hard dependency optional means we want it to come after it So that's the old soft dependency I'm actually that that's something we didn't talk about but it's you know all those ad field mapping calls You used to have to stick in your constructors and so forth now. It's all just YAML is just Write your source field in your destination field, let me see And that's an unusually complex example there, but okay destination field theme source field theme No, we didn't taste it you on the droop late side your Your met menu or your block label is going to come from your Drupal 6 title field boom It's a you know, this really lowers the barrier of entry for people developing migrations Compared to using a fluent api's and constructors Any more questions? All right. Thank you for your time