 All right, well, I guess we're good to go Thank you for coming. I'm Andrew Morton Gonna do the can you hear me? Is that working? now At all. Is this better? At all anybody hear me? Louder can anybody hear me? How's this? All right, it sounds really loud up here either way. So, you know So I'm Andrew Morton gonna be giving a talk on the migrate module and You know getting stuff into Drupal I'm one of the maintainers the project The guy that started it Mike Ryan is here in the front. I think I don't know if most is in this room But they're they're also giving another I think more of kind of a business oriented talk after this in the same room So if if this one's a little too technical, then you know, maybe that's that's the one you want So You've got this old website or you know CSV file or You know something that you're trying to get into Drupal I'm I'm guessing you're here because you Want a better way to do that? Could we get a show of hands of or like, you know, just shout out what you guys are doing right now? Like, you know, what's what's your strategy? You know, do you have the intern copy and paste things or? Oh So node import feeds CSV file with like a custom script or do you do with feeds? Oh, you want to know? Okay. All right Cool, so Yeah, I've definitely done some kind of you know, the full range of things used feed CSV files a little custom script and whatnot and and I think migrates a much better solution so Here's kind of my goals for this presentation the the migrate module is Kind of complex, but it's complex for a reason and What I want to do is give you kind of like the mental framework to understand it so you can understand, you know Why it seems as inscrutable as it does and then we'll kind of step through the basics of like how you would build a migration module and You know, if you guys are contraband tanners And I'm going to try to convince you that it's worth your time to provide support for migrate in your modules So we're going to talk about migrate 2.0. There was a 1.0 version that was Primarily views driven you'd use, you know views to kind of do all the mapping between source and destination migrate to is Primarily code-based. There's a there's a UI and there's a drush interface, but all of the setup and configuration is done in PHP classes and It's got a steep learning curve You know, you might call it the migraine module because I spent two weeks just banging my head against it before I got to the point Where I was like, oh Okay, here's how you actually get this to work So what I really don't want is you guys to have that experience. Hopefully, you know, you'll get done with this and The the the system will make a little more sense I'm what I'm not going to do is demonstrate the UI or the drush interface They're both pretty neat I'm going to show you the screenshot to prove that they do exist and encourage you to check those out in your own time There's also a Drupal 6 version that's basically a back port of the seven in in a lot of ways It uses DB and G DB TNG and auto load So you can kind of jump back and forth pretty easily Except that then you're back in Drupal 6 and like wondering why nothing works, right? There's the migrate extras module, which is sort of the You know dumping ground for Contrib module support that isn't in those contributed modules. So, you know, like a dress field and I'm gonna space out on thinking of like web web form You know those kind of things that haven't been put into the module those live in module in migrate extras So you want to take a look there if you're trying to Figure out how to get something into into Drupal And one other thing that's kind of important to point out is that the the best documentation for it aside from this presentation is in the The beer and wine files in the migrate example module Those get updated pretty much with any kind of change and they they serve as unit tests for the for the module So give that a look. There's a ton of a ton of really useful information in there It's all really well documented and You know and you can kind of copy and paste a lot of stuff to get started I've given this talk a couple times and this generally comes up like when I just use feeds and you know I'm a pragmatist like if it works for you use it It's it's a lot easier to set up It's you know, you can kind of almost hand it off to the admin to go, you know kind of map the fields But my grade is definitely going to be like a higher performance solution You're gonna be able it's it's gonna run a lot faster. You're not trying to have it be cron driven and Where feeds kind of falls down is where you've got a couple of different things that you're bringing in and you need to Have some kind of relationships between them And that's where Migrate is a better solution So we'll kind of talk about the The theory of this and like I say kind of explain why my my great is what it is so you got a You know you got a source and you got a destination right you got something you're coming out of and something You're going into and you know those have you know fields in the you know in in the like say sequel sense on the source side And on the in the Drupal field sense on the destination Well, not necessarily So, you know your source it's it's kind of the interface into into whatever you've already got if it's a you know CSV file if it's a You know Microsoft sequel server You know other another my sequel database You know a web feed that kind of thing and and what it really provides is just a list of fields and Optionally descriptions about them and and then it just needs to be able to step through them in in order. It just needs to be iterated across them on the destination side, it's it's a little bit more complicated because It needs to handle writing it and so for the most of the the Drupal entities that You know come come with Drupal core the support is already provided You know that the entities like users and nodes You know those you're not gonna in front with but you can also do other things on the destination side There's a destination that'll just let you write sequel rows and one thing that's kind of important to point out is the The destination is first for a specific type. So if you've got say a table full of user information that you're going to use to create users and then to create You know profile to profiles that needs to be two different. That's two different destinations and The migration so you're actually gonna end up with two migrations You're gonna end up with one that creates users and one that creates the profiles so the next kind of step is you know, you've got Fields on both sides what what goes where and so they the concept for that is a field mapping and The way that kind of Works is you just give it the on the source side you give it a field name on the destination side You give it a field name and it encapsulates some other information And as the kind of example shows like you don't have to map all the fields, you know, some things maybe no There's some stuff that you know, you just don't care about on the source side And you know, maybe on the destination side, you just want to hard code default values So, you know again, it kind of holds on to you know the names of the fields on each side and It has some the ability to do some kind of basic transformations if you've got like a comma separated You know field it can it can split those out so that then they can be treated You know handled as an array like if you're going into a multi-value field on the destination side And and it has the ability to hold arguments that it can pass the destination And we'll say more about those later So with those first three pieces you basically have what feeds gives you right you've got You know some kind of abstraction on the source side you've got a destination and you've got you know field mappings and and what I'd said was that you know that Feeds kind of falls down when you've got more than one type and you need to have a relationship between them and so One of the the way that kind of migrate Addresses that is with a migration map and it tracks the The source ID and then what it becomes in the destination ID So if you're importing something and say You're importing you know articles and users When you bring them into Drupal, they're probably gonna have new IDs You know your WordPress user ID is not gonna be the same as your Drupal user ID So when you go to import your articles you you know you you you're given the WordPress ID But you want the Drupal user ID So what the the my excuse me migration map does is let you look up what the old ID is and kind of translate back and forth So to do that it needs to know What's your what your keys look like? What what are the IDs look like? So it'll support composite keys, which is the case where you'd have An ID that's in two separate fields so And then it has a couple of other nice side effects since you're keeping track of All the IDs that you've created you can do a rollback and delete all those records And then the other is that you could rerun a migration So you imported those articles and you realize that oh the You know that the time zone is all screwy so you can just you know fix that in your migration Rerun it and then it'll update those articles in place. So you don't have to you know run up your node ID counter or whatever And Then then the next piece is is a migration that kind of wraps it and puts all the pieces together and and that's the that that's actually what you'll be building you you implement a migration class and It has a couple of other little helper functions that I'm just gonna like mention here so that you know kind of Visually where they live But I'll say more about it when we talk about building a migration And so yeah, as I said, it's it's it's where you Configure all the pieces, you know the source the destination the map and then all the field mappings and then the next kind of piece is the The field handlers and so if you think about the way that Drupal Kind of builds entities there these Like the fields are typically these like nested arrays and it's it's a non-obvious structure It's it's kind of complicated And if you think about what the source is going to be the source is going to be a simple value It's going to be a string. It's going to be an ID You know and it's and so how do you get it from that single value into that nested array? And that's the job of the field handlers and They also are what consumes the the data that you put into those arguments on The field mapping There's there's one other kind of piece that is more Kind of an implementation detail, but is Of interest and just to be complete here. We'll talk about it. It's the destination handler What that does is let you kind of extend a destination without the destination having to know about all the possible handlers Or if there's a module that gets turned on Like that the obvious and example is the comment module where it adds a couple fields to the node that if you Don't have common enabled you don't care about so No, doesn't have to care about the node destination doesn't have to care about the comments the Comment destination handler can take care of that some of these names get a little long so You can blame Mike Yeah, sort of the same thing and the destination handler is one of the things that contribute module maintainers might need to create So let's look at this in practice, right? You know sounds good, but what what does it actually look like? The way that you go about building the module is the first off just put it in a separate module Don't put it in with the rest of your custom site functionality because what you want to do is be able to just turn this off Once you get everything into your site turn it off There's some other use cases from migrate where you might have it doing continuous imports, but For the most part you still want that in a separate module So go ahead and create a new module. Don't just do it in migrate example Create a new module with your namespace and then you can leave my great example alone to you know copy and paste from And then you need to implement the the migrate API hook You know create a class sometimes I'll just get if it's a single migration I'll just put it in the module file be a little lazy And then in the constructor, you know you set up the the source destination map in the you know field mappings And if the class is in a separate Dot ink file then you want to make sure you add it to the dot info file so that the auto loader can find it So here's kind of what the constructor would look like There's there's the four or there's the the three properties and then one function that that you'll be calling to do that setup And we'll go through each of these in in turn So here's a like a sequel source It's a dbt and g query, you know, it's pulling up the what beer topic table it picks out a couple fields It sets a sort. I don't really know why we did that but so you take that query and you pass it into the Migrate source sequel class and then it It's able to go and turn that into a count query so you can figure out how many things you're going to be importing and You know does does some other stuff, but that's that's kind of the basic The most basic use of a sequel source one thing you'll notice is that you're not specifying a List of fields this or you know this the source fields. It's able to pull that out of the query. So Like when you contrast that to like a CSV file Because there you just kind of have numbered columns You want to give them like a little bit more readable names so for a CSV file you'd build an array of the names and then you know optionally in descriptions and I think that's I think this this might have a little typo in it, but Then you give it a path to the CSV file and the column information and then that's your source for for that example There's other there's there's quite a few other sources for doing like XML files Jason files For those you're probably gonna have to do some tweaking right because if you're trying to pull in some Like something out of an API for like a Jason file You'll probably end up subclassing one of the existing sources and Overriding some stuff, maybe adding pagination or adding caching. You'll probably definitely want to add caching And there's a couple of base classes to help with those I'm not really going to dig into it too much I'm gonna leave this slide here so that afterwards when you realize you need them you'll go download the slides and You know kind of read about it the next is the the migration map and As I'd kind of mentioned it needs to know what those the keys on the source and destinations I look like so you do that using the the Drupal's schema Like hook schema format and you'd give it an array of your Your your fields on the source side on the destination There's typically helper functions. So the destination class and then it'll be The get ski get key schema function So that you don't have to manually type those in one other thing that I should mention is that first parameter to the sequel map is the migrations machine name and and the reason it needs to know that so I'll just say what the machine name is that's basically the class name But if it has migrate on the end it just deletes migrate so that it's a little little shorter and And the reason you want to specify that is so that When you're having another migration look up a key in it, it knows what it's called So let's look at the destinations These are a little simpler right because there's not as much variation Drupal knows what you know what the what the term how to how to build the term how to you know build an article and so To figure out what you're gonna pass into the constructor on those like on the on the term That's a machine name for a taxonomy vocabulary you know the nodes is going to be a node type user, you don't need one, but I'd say pull up the Pull up the class for that and you can usually see sometimes. There's some other options that You know, you might might find useful and the the last kind of piece is the is the field mappings And so that's how you get You know source to destination one little quirky thing is the order the destination is the first parameter The source is the second and and the reason for that is in that next example. There's the the default value Where so that you don't have a source so it's a It's less awkward having it be destination default value rather than you know null comma destination and The adfield mappings function returns a class that you can then call the functions on and those are those are chainable so that you know, you can specify a separator pass some arguments in and Have a bit more compact syntax So the one kind of problem with Migrate right now that there's there's you know We're trying to kind of figure out how to address is that if you have a field that has multiple kind of sub values in it Say the node body. Well, yeah like a node body It has you know, the body text it has a HTML or a, you know, a filter format And it has a summary. So there's three different pieces there But my migrate really only sees it as one field So the question is how do you pass those other values in and the kind of the hack that We've come up with is you pass it in through the arguments Not great, hopefully we'll get that sorted out pretty quick, but So it has a little bit of a kind of quirky format to it if you pass in just a literal value Like in the in the example there, you know format of one that's going to be hard-coded as one But if you use that Array notation where it's an array with a source field key Whatever the value of that key is Will be used as a field name and it'll go look up the value of that that field. So in this case, you know, we're importing We're taking the teaser field on this row and that's going to go into the into the summary on it on the destination side One thing I guess I should mention is if you guys have questions or something doesn't quite make sense, you know Just shout something out because Otherwise, I'll just kind of keep steam rolling along here So some of the some of the arguments have little helper functions to build them I Personally think that first example is much harder to read when you have to remember the order of the things I think the second where you pass in named parameters in the array is a little bit easier But Yeah, I think I think that's us trying to work around a failure in the API by kind of putting another band-aid on it So let's talk about how you would do that look up of of user and Art and the user an article example that I that I mentioned The way you do that is you specify a source migration on the field mapping so in this we'd be creating our articles and On the old site it was authors author underscore ID on the new site It's UID and so what we want to do is we want to translate that using the migration map from the beer user migration and So for this to really work you kind of want the beer user migration have already run so you use a dependency for that you can you can pass an array of migration names and Migrate won't let you run this migration until the dependencies have been fulfilled and And so with those those pieces you could pretty much do quite a bit of quite a quite a bit with migration But there's a few other functions that you can use to kind of customize the flow Confusing Lee their name prepare row and prepare and complete The best I've been able to figure is that prepare came first and then we decided we needed something else So we had to prepare row So we'll try to make it clear. What does what? prepare row gets called When the when the sources is looking at records and You are able to skip a record. It's kind of the first pass. Say you're importing files and you want to You want to skip over temporary files? So what you do is you just return false if it's you know Whatever the status field is temp, you know, then you return false that row gets skipped and you don't end up with anything on the destination side You can also make changes to it. So, you know say you want to convert a you know date string into a timestamp or You know fix the case on something This is an easy place to do that because this will happen before it gets transformed into the gets transformed by the the destination field handlers into That array structure. So it's a real easy place to do it. You can just you know fix it on the row the next Function you can implement to affect the flow is prepare and it hands you two pieces it hands you The the entity that's been built up. So If you're building a node, it's going to have all the fields in that crazy array syntax And then the second parameter is going to be the source row. So, you know, whatever came out of the database or out of your CSV file and This is a great place if you've got a field that you don't have a field handler for You can just go ahead and hack it into that array structure and you know, or if you need to make some other changes It's a it's an easy place to do that. And then the last one is complete which gets called once the Entities been saved. So if you need to, you know, say send out an email or something with the new ID, you could do that here One thing I will point out is if you're calling node save or entity save on that if you're resaving the thing you just imported You're probably doing something wrong The one kind of useful thing that I'd seen was in Commerce migrate they needed to there's the orders and the line items and they both need to know the IDs of each other so What they were doing in there was after so you'd create the order then you create the line items And then you'd go back and tell once as each, you know line item finished it would go tell the order Hey, here's a new line item or it might have done in the other order But that that's one kind of like legitimate use of that that I've run into So one one other kind of thing you run into and this is this is one of those words This is what makes migrate really really nice is dealing with circular circular dependencies You know, this this is like one of those that I kind of bang my head against with some custom scripts You know say you've got well one one example. I ran into you with another commerce migration was If you're there they've got like kind of a user profile for for the building and the shipping information And a user right so the user has two entity references to the building and shipping information and the The shipping the shipping and billing profiles both have a user ID So which do you create first? How do you get that in there? And the answer is you you know use a stub? so You go ahead and if you're importing, you know, it's gonna depend on which one you want to import first but You You you know just go ahead and create a dummy user and then you know get your profile set up And then you go back and you know match them up or or whatever And so the way you do that is you specify the source migration on the field mapping and then The the the migration that Sorry, I'm gonna screw up the naming on this because I need to give them some names So if you've got your first migration that's running along and and what it's gonna try to do is it's gonna say Okay, go look up this ID in the source migration for me It goes and looks it doesn't exist So then it it calls that second migration that it just checked and didn't have and it gives it a chance to go ahead and Create a new record and it doesn't really put anything in there It just creates it gets the ID saves the ID back into the map and then lets the first migration continue on And then when the second migration gets run it because it's got the ID in the map It just goes ahead and updates that record fills out the rest of the information and then you're able to kind of move on So yeah, that's that's that's basically that process So now I'm gonna just try to kind of summarize the way that the import flows through here I've got this next slide that I'm gonna kind of do a little reading off of because It's sort of hard to keep it all in your head. This is like a nice little reference slide So that the way that it's gonna start is the Migration is gonna ask the source for record and it's gonna keep grabbing a record And then it's gonna look at it in the prepare row and if it skips it It's gonna keep skipping and until it finds one that you know is is okay and then Whatever changes you've made you prepare are gonna be kind of carried forward The migration will go ahead and apply all those field handlers to transform the the data into the Into the you know sort of the entity format All those nested arrays Once that's done the migration is gonna call the prepare function and let you do any final modifications on the entity Then the then it hands it to the destination the destination will go ahead and save that entity and then it gets passed back to the migrations complete function and Then the migration saves the new and old IDs into the migration map so Yeah, so if you've got a contrived module There's the two two kind of places where you can help one would be providing a destination so like And if it's something where you've got your own Structure like a webform submission, you know if you provide a destination then You know people will will love you If you're doing of the other would be a field handler, so hopefully you won't need to if it's an entity There's actually some pretty good work. That's been done on that as part of the commerce migrate module so there's that issue that you can go look at and and Kind of just base it off that But if if if not Migrate extras has some pretty good sample code that you can use to kind of figure out how to go about writing a destination On the other side would be like a field handler So if you've got a cck field like you're doing Well address field and you know email field and phone field have support, but if it's a single value field like if you've just got you know One value that you write you can extend the migrate simple field handler and very easily provide support if you look at What we actually do in migrate things like the user user reference and the Node reference those those fields use that So I'll just put this up and kind of ask what what kind of questions do you guys Actually grab the mic back there in the middle of the in the middle of the row I just wondered if you would make the slides available to us Yeah, they're they should be on the presentation Node on the conference so you can download them up there It says a PDF right now. Oh, is it they had a they had a multi-value File field on there like two weeks ago And I had like a bunch of different versions and then sometimes since then they changed it to a single value field and So okay, I'll put the I'll put a PDF up there Well, I want to thank everybody who's worked on the module my use case was importing a hundred and fifty thousand records every month And on a continual basis, so migrate is hefty enough to do the job I found a hook uninstall in the migrate install file that was commented out some code that It got rid of the tables for the migrate messages And that's how I was able to reset and begin a new monthly Import each time and it would clear out all the messages all the stuff that was there So again again, is there a better way to do that to wipe it clean wipe it fresh and then start another So on your source thing do you don't want to keep the ones you've imported all the information once it's imported I want to start a new migration completely fresh I don't want to do any all the mappings and everything else is gone Are the IDs changing on the there are actually no IDs that come in it's all Basic text itself, but yeah that once they're created it in Then I can just drop all those so that it was really nice It said you know select from the schema information Table name and then drop those but does anybody else That you know of any use cases for that. Oh de-register migration Mike says Okay, I don't know if that if that helps if not come up and chat with us afterwards Yeah, so de-register because I keep the all the module information there But de-register then we'll clear out those tables with the messages and he's nodding. Yes, so okay. Yes, I'll take that Yes, thank you I'm working on a site that has a content type that really should have been three from the start is Migrate something that would help me split all of those existing nodes into three nodes or more You could yeah, so one thing that migrate I guess I you know in my whole selling on migrate one thing it doesn't do great is Dealing with like Drupal sites because there's like the 15 different joins as soon as you start adding CZK fields So if they're actually if it's on the same site you can kind of Get you can you can sort of be a little sneaky and you could in your source Just go build your query to select your nodes and then just select the node ID And then in your prepare you can call node load in there And so that that's how you could kind of get it But then since it's all in that that crazy nested array structure, you're gonna have to do some flattening on that But it would be pretty it would it would probably work pretty well for that because you could go through To find your fields on the source type and then Be able to specify, you know, you would be it would become three migrations, right? One for each of the node types that you'd be creating but you Yeah, I mean, I think I think it would probably be a you would be able to do that pretty It's one of those how many nodes you have to that's kind of the other thing because if it's like 50 nodes There's there's about a thousand at this point Yeah, then you know, I think it would be a reasonable fit for that So my question is about dependencies if you if you've got a road that essentially has a column on it that Would indicate a node reference, but you're essentially free tagging them, right? So when it comes across it that node reference might be by title, but it may or may not exist yet I Guess could you speak to how I guess out of the box those field handlers? I guess we're handling that or would you do sort of a chain dependency of first of all go through all the records and Create those nodes and do you get where I'm sort of going with that? Say a little bit more on kind of you saying sort of the say you say you have you know five rows and You know row number one has a has a column that's out to node a but node a doesn't exist yet Three four and five point to node a is that what I just create that what I just create that programmatically at some point here That's where that creates dubs would probably come in handy because it sounds like If you can't order your source so that those things, you know Sometimes it's kind of like obvious the way you're gonna do it, right? Like just the node IDs are gonna be or that you know source IDs are gonna be in such an order that like you Can kind of get the parents ahead of the children, but if not that that the The Create stub It doesn't have to be a different migration. It can be it can kind of be in itself That's sort of what the taxonomy terms do because for the hierarchical ones You you know you can kind of get these these orders where the parent doesn't exist yet But the child is kind of pointing to it and you want to know what the new idea is so that's That's exactly what it would do it goes in it's gonna create it It's gonna create it with like a empty term name or whatever and then you know when it finally gets down to it It's gonna go ahead and fill that one out. It's gonna it's got the ID already in the map So it knows okay Here's the one I need to fill out with that Information so that makes sense on the first row But then is is the stub implementation smart enough to know that it already created a stub and Okay, yeah, cuz it looks in the map when it when it when it first thing it does when it pulls up a record Is it goes to look in the map to see if it if it's already been imported and that's how it can do the updates rather than You know duplicating a bunch of of records. Thanks welcome My use case is to migrate a phpbv forum into Drupal commons discussions And my question is To create something on a destination and this is triple seven Do you always rely on the fields API or how do you know how to create the record properly? I'm guessing that's not a good idea to just Write SQL and write rows to the actual Drupal database, right? So there's got to be a better way of creating records. Well, so for the destination side for for like a comment There's already a destination for that. So It it knows the kind of built-in What are they calling like properties on the entity right those fields that are on that base table and then it The way that the fields are actually implemented There's a destination handler that goes and like looks up the entity and figures out what fields and then it kind of passes through to the to loading the right field handler so It's kind of a two-step thing right there's the fields that are on the base entity and then there's all those I Was going to call them cck fields, but I guess that's you know Drupal fields That then get kind of populated so I think it should be setting those up for you And and that's one of the things with the UI is really handy Is it'll show you all the fields that the destination is is listing and so you can kind of basically go through and copy those in and then You know go look at the destination tab and go copy and paste and then kind of match them up that way and You know turns out I probably would have had time to To show some of that. I mean I guess we could probably go into that if you know if we get through the questions of people Still want to see some stuff. Thank you my question is really to performance and like Based on your experience using this module like what's a kind of a good number? Before things time out on running this process like if you have like million Customers, which we want to import from one system into Drupal. Can we use this and if we can Can we go in one go or do we have to like schedule it for like? That and sorry in related to that The second question is if for example it breaks in the middle like when we are importing does it roll back everything? I am or Then this this it might actually be worth kind of showing it So there's the two ways that you can do it through the UI it uses the batch API Which is slower, but Just if you're familiar with Drupal's batch API it kind of runs for a second stops runs for a second stops so that you avoid the PHP time limit the Drush mode just effectively disables the time limit and Well, actually I think it's got some kind of trickiness so it kind of spawns sub processes to avoid that but Basically it you don't you don't have to worry about timeouts with it, but If if you stop it in the middle because it keeps track of everything in the migration map It knows what's already been imported So then it'll just kind of scan through the records until it finds one that hasn't been imported and then carry on from there So you should be able to You know resume it with without any problems But in the middle of the migration if there is an error which happens or something does it roll back no it well If if the record gets created the record got created on the destination side And in some cases it'll be sitting there just spewing errors and you know depending on what you've got going on in your code It might just be spewing errors and like, you know, so fields are not getting set up, right? But nodes are getting created But you know and so then you control C stop it and it's those nodes are still going to be there It's not gonna Not gonna do that. You can you can manually roll those back and have those deleted or you could rerun the the Migration and there's a there's an upgrade or update flag you can specify to have it Update the records that are already on the on the on the site. So that's an easy way to kind of resolve some of those You know if you have a if you have a bug in your migration That's how you can kind of fix that this this module provides some kind of logging that we can check that what's completed What's not you know, like, let me just let me hop out of here and go Let's pull up Yeah, because it does it does log Okay, let me oh, you know what I'm gonna have to do I'm gonna have to switch the I have to because otherwise I'm not gonna be able to see the thing while you're seeing it, right? So here's our Drupal 7 site and So the migrate UI lives in a tab on the content page and It'll list out all of the migrations it knows about show the amount of you know rows They've got what the kind of status is and it'll try to track the throughput and when the migration was last run And then you can go kind of click into one of them and get a little bit more detail I think there's a bug in here that I had I noticed earlier, but didn't really want to try to trick down So to list the overview it actually has some kind of abilities to kind of track Sort of metadata about the migration you can kind of associate people with it I Found that kind of maybe be a little bit more trouble than it was working worth But I guess it depends on your organization and then you know you can kind of look at the destination so if we Let's just go and like run a Couple of these and then just you can kind of see the You know see what see what this half, you know, there's there's there's here's the list of operations that supports You can do an import you can do a rollback You can stop a migration that's in progress or And so sometimes like if it they'll they'll get kind of wedged and then you can reset it But here's sort of the here's what come the batch API looks like and so it'll tell you you know what got created and then Inside the migration. There's Where do the log stuff live Mike? Wouldn't there a log tab? Yeah Okay, so Where am I I'm totally missing this oh Yeah, okay, right. So here's the so here's where the message you can see kind of like So this was probably thrown as a warning And so those that's where those will get logged Yeah What a next question I guess kind of how do you deal with attachments like images or other uploaded files? kind of a pain The the That's actually one of the kind of the drivers of the sub of that kind of the fixing the arguments thing Because that that little example that I showed in that one slide was sort of doing an image field So where you give it a path and you give it that kind of stuff it works, but there's definitely some kind of issues with it What I would sort of say for is is looking the beer and wine they they both have samples for dealing with files but It does kind of suck So you mentioned that this wasn't really Easy to do for taking content from one Drupal site into another Drupal site And that's actually my use case is migrating an old broken site into a new rewritten awesome site So other ways to make that an easier transition or is there something other than migrate that you would suggest trying to use? Well, there's some new stuff that do you want to just talk about that Mike? Do you want to talk about the d2d because one of the things that's getting Probably just grab one of the mics One of the things that that's kind of coming up in in Drupal 8 is dropping update dot PHP To do the major version migrations and using migrate instead And and so you've been working on some kind of like stuff There is an experimental module migrate d2d for doing Drupal to Drupal migrations. It's in my sandbox So it's a little bit hidden But if you go to my profile find me Mike Ryan on Drupal that org Under my projects you find my sandbox and right now it works for your basic Drupal 6 to Drupal 7 objects other people are contributing to Drupal 5 to Drupal 7 and It's early, but it I've just used it successfully on a client project I've done some kind of like nasty things where we had one that we sort of had to keep the site up and running So I ended up kind of writing this custom web service module to just export them all as Jason because I found it a lot easier to Have Drupal tell me what fields were on there rather than trying to join everything manually and just then I had the code kind of flattened that It wasn't great enough that I really want to release but Yeah, and then you kind of have two problems you've got That you get to deal with the Jason trying to talk to it, so I I'm excited to have an excuse to try out Mike's project. I have a source that I'm migrating to to different Type of nodes to different content types So I'm running it through the prepare row so I can return false and having two separate migrations Is that the best way to do that or what's it stored in? Is it a sequel table or is it like a csv file or is it something what type table? Oh? So then coming from as the source is a sequel table I would probably try to do to restrict that using where clause on the the sequel query Because then you avoid then the counts are going to show up, right? Yeah, okay I think that would be the best and then you you know then you'll show the counts because otherwise You'll see all those numbers skipped because it'll win the logging it reports And then it keeps skipping the ones that it and so it's going to slow it down because it actually loads all those records Hands it to your migration and then right you say nope. No, thanks And then it goes to the next one, but you would still do two separate migrations because you have to declare the content type in the Yes, you because I have mostly the same fields Would there be a an option to in the prepare to change the no type what I would do is do a do one If it's basically the same fields, then I would do a I would just do a subclass of the migration and then you could say specify everything but the You know you could just you could just have separate queries And then pass those into the source separately like you could have the base thing where you set up all the the fields And and whatnot and then in your subclass you the only you change would be the source but definitely two different two separate migrations Yeah, yeah This is kind of a follow-up question It's interesting a couple people have already kind of touched on this It's something that I was wondering about myself And so you've mentioned that Drupal to Drupal maybe isn't the best right now But then Mike was saying but there is something in the works that maybe they would take six to seven or like Five to seven and so I'm wondering if is there any discussion like with Drupal core in terms of like the way that like D8 and D9 and whatever's to be built so that maybe We can have like better integration so that like one person was saying so like if we had our Drupal six site That's you know, we've got the certain ways that things weren't Fieldable entities so we built the site in a certain way that now that seven allows you to have the field of Entities is there any kind of discussion going on in the Drupal community now about building Drupal in a way that it's going to be able to then plug in Cleanly to using something like migrate because that's what it seems to me like as well as like if you can take a WordPress site into a Drupal site Why can't we take a Drupal site into a Drupal site and go just avoid this whole pain of trying to say well I'm on Drupal whatever and then I've got to get to the next and the next and the next why can't I think that's exactly the Conversation that that's kind of starting and you know and then the kind of difference between the WordPress one is it's Oh, you know, you've got a lot. It's sort of the same reason. It's easier to do themes in WordPress, right? There's just a lot less, you know, dumb stuff you can do to kind of configure it in strange ways Whereas in Drupal, you know, you we give you all sorts of rope to you know, hang yourself and You know, you get you could put 50 different fields on something and you know, I did one migration where they had, you know literally dozens of content types and then like hundreds of fields and That was the one that I did that Jason thing on because it was like there's no way I'm gonna do all these you know queries to or figure out how to do the sequel queries to do that. So I think that's definitely part of the conversation is figuring out how You build sources in such a way that you know, and the nice thing is if it's locked to a to a version, right? You can say okay. Here's what CCK on D6 look like and so it can go do the introspection in the tables Figure out how to do all the joins and then kind of just build that for you Versus like right now where you know the you know The D7 migrate doesn't really know what a D6 site look like and so You know, it's just that that first pass the first the kind of the first thing That's easy to do is get all the sequel stuff and then the kind of you know building that layer on top of it And so I think we've got you know some kind of like good building blocks And then it's exactly what you're talking about like how do you start setting it up so that you know Drupal 7 knows how to You know hand itself off Yeah, well because I'm thinking I mean what's done is done in the past five was built Oh, it was built six was built seven But I'm wondering like going forward here if that's kind of like the thought processes so that I'm not saying that we should be able to take a Drupal 5 site to a Drupal 10 site But starting with eight or like our nine something that it's kind of be built into Drupal itself that like this whole upgrade process of Getting from eight to nine for instance or from eight to ten or from nine to ten We'll just be an easier thing for us so that we aren't kind of caught in the situation of when we're building a site for somebody and thinking like oh God, you know what? That's kind of more what I'm after is like looking forward to the future here If we're going to maybe have a just sort of a more natural kind of plug-and-play way that Drupal itself is Aware of the migrate module, so it's just like it's much more of a snap to get from There is a core issue to put Something based on the migrate API into core for Drupal 8 So there's an issue out there and tomorrow 1 o'clock I'm hosting a bof to talk about that. I think it's room 210. So if you're interested in that on a drop-by Thank you I guess the one thing I would kind of add to that is that you don't necessarily know where you're going to be so it's sort of hard to future-proof because You know 640 case should be enough, right? My use case is that I'm using the address field module with three addresses a specific number and So the CSV file that I have has three address three different sets of fields one for each address. So and That gets that would be Using the geocoder module as well to query Google and geocode that I'm wondering if that if you know If you've had experience doing that type of thing I actually wrote the status of a field handler for the address fields And that's kind of where I realized that our argument thing is totally busted because what it does is it you For the for the field you specify the country code and then you pass everything else through the arguments and it's Kind of tedious, but if you look at migrate migrate extras In the I think it's in the 2-3 version. It's the one that just just recently came out So that might have been why you missed it Yeah, I mean if you want to look at I haven't actually haven't done any research into this Oh, okay. Yeah kind of a good session for me to get to because this is on my plate this week So, oh yeah, it it it's a little tedious to set up, but it'll definitely do it And it you see you just kind of key all the arguments based off of I forget what there It's actually pretty well documented So I would say I would say pull that up and give it a look because it should it should address that pretty well I feel part of the fun any any other questions Do you guys want me to poke around and drush because I think we've got ease your question and back to your original slide steep learning curve Our migration I believe is much simpler not 10,000 entities Coming out of another database into XML and then into Drupal It would seem that there might be an easier approach. I think we got a one-to-one match Can you comment on that because it seems that the learning curve to get to use migrate is Migration is going to be pretty difficult here. What else are you looking at using? I'm guessing feeds I mean, how big is the XML file? Excuse me. Do you know how big the XML file is? Is it a single file or is it? Well, they're about They're about 30 fields I'm just in in total because like 30 fields per inch per entry and about 10,000 entries So like 100 megs 200 megs because yeah, I mean I had one where it was trying to do a 900 meg thing and Yeah, I don't think feeds is gonna be happy with that Because it because the problem with so one of the things that we ended up building in I think the recent version of migrate is Once you get the big XML files like that, it's hard to because PHP runs out of memory So I mean you can kind of just bump your memory up really, you know to okay, whatever gig kind of thing I think my problem is not as trivial as I was hoping it was Yeah, you can do I'm just spacing on So he's suggesting in feeds you can do the background Background process it just depends on the size because that's that's gonna be the one that because like trying to do An X path queries get slow to is the other thing like I'm just spacing on the name of it There's a there's a way you can do XML files Like X Whatever basically you don't have to load the whole file It just runs through the tokens and then you can kind of grab subsets of it and then just process those And depending on the size of the file that might be worth it if it's small enough or if you can chunk it up You know, but Yeah, give it give it a try with feeds if it's a single type then you know if you can get that working. Yeah Anybody else do you guys want to see me poke around with Josh a little bit or okay? All right, so Oops. Oops. No, it's too small Okay So Boy, this is there is really no worse way to try to type that in front of an audience Okay, so We'll just do Drush help to list out the The kind of the the commands so there's this big block of migrate commands, right? Most of them have little aliases I'm just gonna kind of show a couple that So, you know Drush MS for migrate status is is kind of you know nice for just sort of seeing where things are at Right, you can kind of see that okay the term finished user finished And so to run one you can it's a Drush MI for migrate import And so it'll go off and you know tells you when it gets done tells you how many it did You know what you know what what worked what didn't and give you kind of a throughput And most of these have you know some kind of switches Boy, that's not really readable So one thing that like a lot of times you'll be kind of debugging migration and you want to keep kind of running the same record so what what I'll find myself doing is Doing an update because that'll rerun the migration and update the records and then you can specify a limit This I hate you have I think it got fixed, but you have to So one item okay, so now you can just type one Okay So So it'll just keep running that same one record so if it's kicking out a bunch of warnings You know that's kind of handy. I Wonder if I can get something that's going to take long enough that I can cancel it I guess I could stick a weight in there or something like that, but Do you set spell right? It will just do minus minus all which will just grind through all the Imports that it has so you can see it's going to kind of look at the ones that are already done It'll just work its way down the list that we hit baseball there Okay, so we got something going slow, so you know, I'm like, hey, I don't know what's going on here Let's just kill it and then we can do you know drush MS and it'll show us Okay, so oh it thinks it's still importing right well, it's not importing because I hit control C so We're gonna go Copy that Drush MRS Drush misses drush my great reset or something. I don't know You just kind of memorize and so then and oh for the status you can actually specify migration status So you don't have to pick through the whole list. So okay. It was going along. It got kind of stuck And and that's sort of how you'd reset that when they when they fall over What else am I what's what's oh, yeah, let's do a rollback so So we just oh M. R. Yeah, just drush mr. They're both stern Stern folks one resets and the other deletes So yeah, it's gonna sit there and you know kind of grind along and then when it gets done It'll tell us, you know, how long it took to delete all that stuff Kind of let it run a little too long. I think yeah, so, you know, it'll it'll tell you kind of what what it did Any other questions everybody happy. I think like said, there's another Migration session in here directly after this so, you know Stick around if you've been having fun