 So this is a core conversation, I was saying earlier before, if I wanted to do a talk on migrate and tell everybody everything I knew about migrate, I would have just submitted a normal talk. So this is a lot of the subjects that we don't know much about, things that are outside of the realm of what I do day to day. So I really want to get some feedback, especially from other core developers, people who are developing against the API in Contrib, people who are using it, especially people who are using the upgrade path, you know the one-click upgrade path stuff like that. So I'm the Drupal 8 migrate maintainer, there's three or four of us in core, in case you can't tell I've got an accent from Northern England, but I've been in Australia for about 10 years. So the four areas that we're going to look at is current problems right now like what we really need to get fixed to kind of get out of the experimental phase, what features we'd like to focus on next, if we dare add any more features, possible API improvements. So this is more focused at Contrib and core developers. And then dare I even say it, but what are we going to do in Drupal 9? All right, so which areas currently need work? This is probably more here than anywhere else in the rest of the conversation. First one is documentation. So I've learned a lot about documentation even in the last 48 hours. This is my first ever Drupal Con, and in Australia we're very isolated and working on the migrate API. I don't think I've ever read the docs once. Being a core maintainer, you find yourself, you don't need to use the documentation, it's maybe not that relevant to you, especially when you wrote half of the things. You don't really pay attention to it. And I've had a bit of a look through. We've got some half-assed documentation around process plugins and things like that. And a lot of it looks like curtsun, if it's that chicks posted into IRC and they ended up on a handbook page. So I really am interested in getting some documentation. I had a really good chat with a variety of people yesterday. I had a chat with a Drupalize.me team, and they were obviously preparing documentation and video tutorials. And then I had a chat with some development agencies from Germany, things like that. This seems to be a real confusion around how core does things, because core solves a really complicated problem that you don't have to solve for an everyday migration. The biggest thing in core is we're dynamically discovering everything about the source site at the point of running the migration. So we need to figure out what bundles you've got. We need to figure out what fields you've got and what the name of all them fields are. And one thing that people seem to misunderstand, and three or four people mentioned it in DrupalConnet, and somebody asked me last week as well, they feel that they need to try to run the core migrations and then somehow hook in to change the bits that they want specific to their situation. Like if they've got some custom fields, they feel like they should be hooking into that. I think there was a bit of a complete misunderstanding that you can just copy the D6 Nerd YAML file into your module and just hard-coded the source fields in your site. If you're just migrating a few thousand nerds in a little agency just doing your migration, just copy that file, forget about all the field storage, config migrations and all the purview mode migration stuff, you don't need any of that. Just hard-coded the fields in there, and you can just run that file. There's nothing else that you need to know. And I think we need to work on some possible tutorials, style format documentation. I know the documentation team have been doing things like that. And there's varying blog posts out there. I was somewhat horrified to hear about someone's experience with the contrived flow where we had the migration templates, and you had to somehow convert that into an entity and then change it and run it. And it's not that complicated anymore with 8.2 and the migration's going into plugins, you can just get the file and run it. So I think that is a really big part of some of that we need to work on. The next part is the upgrade UI. This is something that is just foreign to me as like a developer. I work at an agency and everything we do is pretty much bespoke. I've never even dreamt about running update.php or replicating a Drupal 6 site to a Drupal 8 site. And until Dries told me last night that definitely people use it, I barely believe that anyone had ever used that UI like I truly had them. So I was somewhat thankful that I haven't wasted the last three years of my life. But I would really like to get some feedback from people who use it and understand why they use it, whether you don't have the technical expertise. The example that he gave me was actually universities, they often build sites, they've got low budgets and they need it to last ten years. And their content model doesn't change, they're just really big content sites. And they just want to move to keep up with security updates from one version to the next version and not change anything. Primarily because they don't have the money to do it. So I am really interested if people are using that to hear about why you use it and whether you contemplated using an agency to do custom migrations and things like that. The next blind spot is multilingual, again, I'm still struggling to learn English. I never mind a second language. And we don't build any multilingual sites in Australia. I've never even come across one. Migrate is such a big subsystem that we really need more maintainers and people who have knowledge around things like multilingual to step up. I was saying to Adam earlier today that the maintainers.txt just puts people under migrate maintainers, but migrate is so big, or you need so much knowledge in certain areas like with the translation stuff, we almost need subsystem maintainers. And now we don't have to document that in maintainers.txt, but we do need to kind of try and get people with experience with multilingual sites. One of the big things when you volunteer in your own time is you've either got to be scratching an itch, so you're using it or you've got to have some enjoyment in what you're doing. And it's very difficult to get motivated to work on things as complicated as multilingual, which it is when you never actually use it yourself. So yeah, if anybody here is, I know a lot of people in Europe, obviously lots of languages, lots of multilingual sites. We really do need people to get involved. Gabor did tell me that a couple of people have been working on some multilingual stuff, which is great, but we really do need a lot more input there. So the next one is Drupal 7 to Drupal 8. We did the D6 upgrade path first, and the D7 one is getting pretty good now, but we do still have some work to do. I think one of the biggest blind spots is around reference fields, entity reference, user reference. But I would like to hear if anyone's run any migrations, you know, upgraded from D7 to D8. I imagine one of the biggest problems is like customized field types. I don't know whether anyone's tried to migrate a site with field collection yet or address field, something like that. You know, let's have the war story. The Contrib Tools I think is a big place where we still need to improve a little bit. We're built really early on. We were keen to get the Contrib Tools in place, and they still have a few artifacts around the migration templates, where you have to configure it and turn it into a config entity first. I had a look today, I was a bit horrified, but Migrate Plus actually disables plug-in discovery and uses config entity discovery for migrations. I can't imagine how that's working out for some people, and I really don't want to explain it. So, you know, I have some really interesting ideas about what we can do for Contrib Tools. You know, like I think we should... One thing I thought about was possible, pluggable runners. You know, like everything right now is based around Drupal, but Drupal consoles get in a lot of traction. It's got really good multi-lingual support, as we heard in the keynote this morning. And most of the stuff that you need to do when you've got a runner has nothing to do with Drupal itself, so there's no reason we couldn't architect in such a way that we just use console or Drupal as the display and interaction of the runner and maintain a similar code base. So, we can support both. And the final bid is another call-out for help, I guess, but early on we had a project manager called Melissa, and she was probably one of the best things that ever happened to Migrate, and do wish she would come back, but I know she's busy on other things now, and I think it's the Migrate team. We are very developer heavy. Migrate's been doing a really good job with some spreadsheets and, you know, organizing the calls and things like that, but it's not quite the same. He's spread very thin doing development, working on contrary. He's got to do his job as well. So, you know, a project manager is definitely something that I think would be valuable and people that, you know, can help facilitate the development process. I've put a discussion slide in between every section, so I've got a few more sections to go, but I wanted to, you know, try and have a bit of a talk, then we can move on and we can do it like that. Has anyone got anything they want to say regarding any of the what needs work? Hi. Okay, so one of the things that's been driving me kind of crazy is the way we're doing error reporting in the migration system. When we run migrations in Drush, like when you, or more like when a plugin wants to notify about an error condition or kind of save an error condition or you want to get more information about why a migrate exception was thrown somewhere, I am a migrate maintainer and I have no idea where that information even goes. Like throw a migrate exception and then it basically just disappears into the void. So I'd like to see us do something about that because, I mean, we're dealing with data here. It's, you know, these are data loss bugs that we could potentially be having and so not having really good error reporting is kind of critical. So that's one thing I think could use some more work from us. Yeah, I agree. Has anyone seen the migration requirements failure ever? Has anyone run a migration here? Yeah. So yeah, I'll ask you one. Another one is anyone run a migration and just somewhere as a missing? Just don't know why. In fact, tracked it back. Skip row exception somewhere. Just never logged, disappeared entirely. Yeah, I agree. There's a lot we can do about that. It comes back to the migrate executable stuff. Is Alex on that? Y'all on that? You're on Nertjewee. No, I agree. Anything else? This is probably getting a bit more subjective. I still think the first section was where we need to put most of the efforts, probably 70% in the need to work category and we're getting into the fun stuff and it gets better as we go on. But what features next? Drupal 8 to Drupal 8 is actually a pretty huge one, I think. Not only for people going from Drupal 8 to Drupal 8, which will start to happen in the next year or two, maybe something like that, but also it's probably a key part of unblocking what might happen with Drupal 9 because we're always gonna maintain the destinations, the classes that are aware of how to save data into Drupal, they can never really break and they'll follow ahead as we move on, but what we need to do is we need to write some classes that understand the Drupal 8 database format, where it is today, so we can pull that data out of Drupal 8 and put it into Drupal 9. So that is gonna be a key part and that will probably be something that is quite fun to work on. So any new contributors next year that'll probably be something quite interesting. Another thing is a migrate UI. You know, we built it, or I built it for 8.0 around config entities and then we moved from config entities to plugins and it's now broken, doesn't work at all, but is that something people are interested in? Like in D7, you know, not the UI, that's the dashboard, but the UI that allowed you to do the field mappings and things like that, you know, quick show of hands has anyone ever used it? Couple, yeah, a few, a lot in a room this small. So yeah, so that is something that we, you know, we could consider as well and that again feeds back into the contrib tools that need work. I was gonna put the migrate UI in the needs work basket but really it's a stat again, so there is more of a new feature. And anything else anyone wants to suggest on the new features? Felt like there could have been a few, but I'm almost glad if there isn't, I'll skip past that. So these are big ones. These always cause a lot of stress. It's very subjective. Anyone who's involved with migration, config to migration plugins, probably it's taken me this long to even think about a new API improvement. It was that kind of level of hard work. Modernizing source plugins, this is, I think this is a really good issue, very valuable and I think it would be a big improvement. It's also is, it could be quite a change at the moment. We're doing our standard. This looks really easy. It's just a new plugin type. We'll keep backwards compatibility but experience tells me that we'll get 80% of the way there and it'll become a lot more difficult and all of again, we'll be begging to see if the committers are gonna put it in. What we're pitching here is a lot of the features that were in the Drupal 7 module, they're just all in the source base. So it's all just the base class for sources. These things like high water, marks, track changes, features like that. You know, anything that needs to do something to the source like even mapping the source query over to the map table. It's all just built into that one super class and if you don't extend from that, I imagine the API just doesn't work at all. So what this is pitching that we do is we take source plugins to become just a really simple query again. They just pull the data out as part of like, you know, wherever it's coming for me that they'd read their CSV file or they'd just query the database and then we would have a new plugin type and we've come out with all kinds of names like filter plugins and stuff like that but essentially any logic like high water, mark, you know, where it wanted to have a look at when the last time the import was and figure out what needs to be imported and whatnot, they would all be abstracted out rather than just magically kind of being coming down from the inheritance chain. It would allow people to do a lot more things like you could have your own strategies then and your own plugins on when you wanted to update or not. You know, all of a sudden, you don't just have to take a high water mark which is a timestamp. You might have something entirely different that decides on whether the import runs could be an API integration, you know, you could look at a remote system to decide whether the data was going in it. The possibilities are fairly big around what we could do there. The next one, source destination needs to get to a config, terrible sentence because it wouldn't fit on one line. But this is another important issue. Like right now, we kind of half solve this with the runners, you know, we've got the UI that exposes a form and it just perfectly manipulates all of the keys into the data that it needs to be and passes it into the source or the destination and you just need to know that they need them values and that's it. If you don't know, things break. You can also hard-code the source destination configuration into the YAML files which actually comes back to my earlier one about the tutorials and stuff. Everybody thinks they need to type in all of the, you know, CLI flags when they're trying to do the important stuff like that but you can just put that in your YAML file and commit it to your Reaper. Very few people seem to know that at all. But this is a big issue because we need to find a way for source plugins to say that I need to get the source-based path of where the files exist on disk and something like that and obviously it needs to be in such a way that it's going to work for a CLI runner or a UI or whatever it is. They just need a generic way and it's a difficult problem to say I need this information and who's going to give it to me? And then the last one, I put that in there just for Adam but if anyone wants to get involved in the, you know, migrate in general, we've made a fairly big clean-up around source tests or really Adam and we're just moving them over to the data provider pattern which PHP Unit uses from what was previously a home-grown mega-array structured test. So it's actually a really good opportunity to learn a bit of PHP Unit if you're interested also learn a little bit about migrate as well. Any new contributors again? Good place to start. Any thoughts on the API stuff? I had one thought that I had regarding the configuration of plugins and getting configuration into the plugins and you just said like we want to be able to do it like with Drush or we want to do it with a UI and one of the things our plugins currently don't do is implement configurable plugin interface. They just kind of expect to have their configuration at instantiation time and that's it. You can't reconfigure them, you can't change them. So that would be maybe, I don't think that would necessarily break backwards compatibility or anything that might be a quick win that we could do just put configurable plugin interface on source plugins and one other API improvement I would suggest or that's been bugging the hell out of me for a long, long time is when we are using, like a lot of the source plugins use database as the back end from which to fetch data and for some reason the connection like a actual database connection object is not passed into those plugins with dependency injection and that means they have to do all kinds of terrible things where it's like, oh, let's look in the state system for the database fallback key so that we can build a connection object. Alex, you have this expression like, like, oh, no, not this again. No one's meant to mention that. Well, yeah, it's exactly, it's painful. I mean, like, that's, I mean, it may not be realistic right now to do this. It's just one thing that I would like to see as a really big API improvement. I think it would fix a lot of things and make things a lot easier for us if we could just have, if we could have the calling code be responsible for getting a source database for SQL source plugins. I think the priority is, and this is an open description, I think the priorities are the first slide, like fixing what we've already got but that's everything I put in here is the stuff that I don't know really well like, you know, if someone was here saying, we can't move to Drupal 8 until we get Drupal 8 sources or something or whatever it may be, then maybe priorities can change and that's the whole reason we're having this core conversation. But yeah, I fully agree. The first slide stuff, the D7 to D8, you know, and the contrib tools, I think, are the two biggest things right now. I feel that the core APIs actually a bit better than people think and the contrib tools are maybe a fraction worse than people think, but everyone other than about five or 10 people just kind of bundle the whole ecosystem together. We kind of need to make it all a little smoother. I'd underscore that the things like the Drupal 7 to D8 migration path for entity reference, multilingual sites, the user interface that we're all on the first slide are very, very important to the ecosystem as a whole. They like, these are things that are actually preventing people from dropping Drupal 8. There's all kinds of trickle-down effects from that. So I would underscore that I think that, you know, the API improvements are great because they improve the long-term maintainability, the developer experience and so forth, but as much as we can do to fix those things that are preventing migrate as a suite from being stable and core now more than a year or about a year after the release and probably gonna be another four to six months before we get there for the user interface and for the Drupal 6 and 7 migration path. So the more help we can get on those things, the better. There's a big problem with the UI, like in that no one in the core maintainer team, like for migrate maintainers actually really wants to maintain it because they don't use the Drupal UI and like I always laugh, I remember having the discussion with Mike Ryan because he built most of the UI to begin with and when it was going into core, I was saying, like I was linking him an issue and he kind of said, I don't wanna look at that. And I was like, well, I thought you wanted to build it. And then it's kind of turned out that he'd somewhat been, I wanna say pressured, but like asked to do it. And he ended up only building that UI because he was trying to help. He actually didn't wanna maintain it either. And that was a bit of a ground breaker for me because I was like, well, I don't wanna maintain it. So, you know, I know it's difficult to, this is another reason for the conversation, but like I feel so far it's been difficult to find people who are maybe using that UI and also capable of helping maintain it. So it's a difficult, it's a bit of a balancing act there. So also with the UI, there's been an issue to add back incremental migrations to the UI and I've been pushing back on that quite hard because people think that it's an essential feature to have in the UI, but some people think it's an essential feature to have in the UI. And one of the things that catches people out when they create migrations in Drupal 8 is that they don't realize that unless they configure their migrations in a certain way, it won't necessarily map node IDs or IDs of objects in the way that you would expect. Like the core migrations that are provided by core, if you just go copy the D6 node file, as you say, that the node ID maps to node ID is not saying. And what that means is that if you go and create content on your new site and it has the same node idea as your old site, when you do the migration incremental again, it's gonna overwrite it. So we've got quite big problems with incremental migrations and that currently provided migrations in core. I don't know how to solve it. Apart from, I might think that we probably don't wanna do that mapping in core. Yeah, preserving IDs is something we talked about multiple times. Like the fallback argument that everyone always went to was imagine you as migrating Drupal.org and all of a sudden everybody's user ID changed. There would be a world crisis. But yeah, it also was a workaround early on before the migration plugin come from using, I know the migration lookup plugin before that was fully functional, preserving IDs allowed us to kind of push on a little bit there. Yeah, removing that from core could definitely be something worth considering. But it's just, it's exceptionally painful because of course if you're a Drupal 7 site and you haven't done URL aliases for whatever reason, all your node IDs are node slash one or whatever and they all change and then Google is like your whole search engine ranking and everything just changes. So I don't know if there's a nice solution apart from actually detecting that there's a node on the new site that wasn't part of the migration and just stopping for that case. Skip. Well, yes. Use a different format for node IDs. I don't know. And I get like, you can do some logic like that. You know, you could try to have a look in the map table and stuff, but it's hard to say they could have migrated first and then added content and then migrate again. And it's a very difficult problem. The whole one click upgrade, which is one of the reasons I was against some of the incremental stuff and things like that. When we very first talked about it, we talked about for people who want to use that one click upgrade initially, they'd install Drupal and then they'd run it and then they'd have a replica and that was what our initial goal was. And that can change if people want it to be different. And I think the goalpost did change a little bit with the UI. We had rollback and we had incremental changes and they all seem like really good features on the surface, but they're quite complicated to maintain. So we've got to be sure that it's actually worth maintaining them because people like using them, you know, it's a lot of effort and we're already spread thin. Oh, my favorite slide. So I think I may be alluded to this already, but I was speaking with Mosh yesterday in the sprint room and we were saying like, we're super low on resources, hard to find container for the UI. You know, does anyone actually use it? Like, and then I had a chat with Dries last night and he gave me the example, but I spoke to a few of the people who didn't use it. And like we talked about some funny ideas, like maybe we could set a flag when you do use that form and send that information back with like, you know, like the status, like the updates status information, stuff like that to try and track how many people are doing it. But like, I'd be interested to see how much use that actually gets. And I'm sure people have got something to say about that. And the last bit is, yeah. But the last bit, I barely want to talk about it. But it's something we did mention early days. You know, there's a generic kind of extract, transform, lured concept that, you know, in software development people talk about. And, you know, it's a software pattern that we could write a library around and we could write that in a generic way. And the migrate API would just become a consumer of that. We're talking about rewriting the whole migrate API, but someone created an issue and it's nice and subjective. So I thought I'd put it in my slides in case anyone has a point of that. You know, it kind of falls in with the getting off the island. But I really do think that it's hopefully not even Drupal 9, maybe Drupal 10. All right, that's all of the main points. So any old discussion from here? If anyone's got any questions, unrelated to this stuff, I'm happy to answer them as well. If you just simply want to move information to Drupal 8, not migrating, just moving information, for instance, at CSV5, what is the easiest way to do that? Yeah, that's a good question. So the question is if you wanted to bring some basic content across from a non-Drupal source, maybe like a CSV file. So in the Migrate module in D7, I think it was almost all in one module. There might have been a couple of side modules, but it was all in there. And of course, we went all granular, we split them all out into sub-modules. So there was like a Migrate Source CSV, Migrate Source XML. Then we changed our mind and we moved them back. So they went back to Migrate Plus, but not everybody agreed. So there's half and half right now. So there's a Migrate Source CSV module, which will provide you a source like for reading CSV files. And in your migration YAML file, you can just map the fields in Drupal to the columns in the CSV. And it's pretty straightforward. I did one the other day for a business directory, and there was like a thousand entries and something like that. It took me an hour and a half. It's pretty straightforward. Does that answer the question? Yeah, Migrate Source CSV. All it does is provide a source class. Do you know what migration YAML file looks like? Yeah? Yeah, well, you can specify what the source is, you know what the thing is responsible for querying the source data. If you download the Migrate Source CSV file, you can reference that as the source. And then you just pass in a bit of configuration. The configuration is just a map between the numerical column in the CSV file and the name of whatever you want that to be. And then everything else is the same as a migration. You just map the source fields to the destination fields. The feeds module isn't used in this... We don't talk about the feeds module. No. Yeah, the feeds module has nothing to do with the Migrate stuff. Obviously, it provides a way of solving similar problems. And I know many people use it. I've always had a soft spot for Migrate. Even in D7, if we were consuming remote APIs, we used to use Migrate. I just feel it's a better API, but that's preference. And I've not even looked at it in Drupal 8, so I really can't comment. So there's someone at the microphone first. But the configuration, say, the path to the CSV path one time. And you can update quickly this name and do all days one CVT file import. So how can we make dynamic configuration for Migrate? So you want to change the file name, right? Why can't you just change it in the file and run it? Nothing else. Right, OK. So what are you using to run the migration? You've got like a cron drop that's calling out to like Drush, MI. So there's obviously a specific flag for passing in the base path, which obviously wires that into the source config. You would need a generic way to pass config options, either at call time. That's the only way to do it. As far as I'm aware, I don't think it exists. It's just channels. Yep, I don't use Migrate plus at all. I'm using a competitor called Migrate Drush, if anyone's interested. It's on the opposite end of the scale. It's almost too simple, but it's another option. So I wanted to talk a bit about the maintenance of the UI and the fact that the current team of maintainers who are building and maintaining Migrate and Core don't really have a need or interest to actually maintain that. And the proposal to remove it from D9, which I don't think is going to work. So I'm definitely sensitive to the fact that people contribute what they need and what they get value back from. So if I don't want to put that burden on maintainers who are already, as you've said, limited resources, struggling to maintain the parts that they need. One suggestion that I have is for anyone who is doing migrations on actual data and testing out what we have in Core so far is once you have all the different migrations that you need set up, try using the user interface to do the migration and see what happens, because that way we get bug reports from it. And maybe you'll find that you have ideas on how we can improve the error reporting, for example, which is I know that Adam mentioned this is a problem in general, but definitely in the user interface, the way that we surface information about whether or not you can run the migration, what migrations are available or not is very likely just put the minimum amount of information in there that we could in order to get that user interface into Core in 8.1. So that's something that you can definitely help with on your own. It doesn't have that much time to your development stack. It's like, run it, let it go through the user interface, and 10 minutes or 10 hours later, see what happened and see if it gives you the information. You need to solve it, help with that information back in the queue. That's a really great way to contribute. Obviously, anytime you run into a problem with a migration in general, also making sure that the problem you run into is somehow look for the issue in the Core issue queue that reflects the bug that you had or with the migration of a contributed module, if that was the case, to make sure it's there. Another thing is we don't want to, a few of us, we don't necessarily want to maintain it directly, but that doesn't mean we wouldn't be prepared to facilitate people. If someone stepped up and said that, they'd be happy to work on the Migrate UI, and they had some experience but knew nothing about migrating call, I'd be super happy to answer every question and help you deeper. I might send you flowers. But you know, help you deeper, complicated problems and things like that, we would, you know, we would definitely help with that. Like, I'm prepared to invest in training other people and getting people up to speed. I just don't want to be the person triaging that you are issue queue. Yep. Very long time, use plus one on the need for work. Most of the migrations that I use. Yep, so I think the man. To repeat the question for the recording, the comment was that for a long time user of Migrate in Drupal 6 and 7, the need for documentation is definitely important. The difference between those previous versions on Drupal 8 is substantial enough that it was difficult to figure out how to do the migrations. And so, especially for cases where it isn't just a complicated use case, where it's not just a one click migration. And the way to contribute, obviously, the handbook pages are publicly editable, but I think the key to things is creating issues in the issue queue to propose a part of the documentation to enhance. And then jumping on to IRC, which, as we had in the keynote, isn't maybe the best for everybody, but at the moment that's how we do it. And the same as the UI, like everyone I know maintains Migrate is super excited when people propose documentation improvements. And we can read and vet content and give you pointers and all the technical information. We're just not all grand authors, you know, content, so that's probably the best way to contribute. I have a thought about the documentation issue. So you're familiar with Migrate in Drupal 7 in previous versions. So I started using Migrate the first time with Drupal 8 and I ultimately at NEDCAMP last year in New England, I went to a presentation about Migrate in D7 and I actually learned a hell of a lot. That was a great presentation. I learned so much, okay, because what I realized there, thank you. Yeah, I was like, oh, I'm a Migrate maintainer. You're like, yeah, don't say anything. So I didn't. I listened very carefully. And I learned so much because what, and the real thing I took away from there is that Migrate, sort of the basic flow of a migration in D8 versus what we have in D7 is not all that different. Like you set it up differently. Like in D7 you have your one class and you do all this bullshit in the constructor and set up everything, but you still have a source, you still have mappings and process, a process chain for each of these things and a destination. And that's all very much applicable in D8. So maybe one idea for documentation is that we actually, do you think it would be easier if we wrote our D8 documentation initially that just kind of illustrates the difference? It's like, well, in D7 you do this, in D8 you do this. And I think for me personally, that would actually make it super duper clear if I knew Migrate in D7 what to do in D8 and it might actually really solidify things really, really quickly for those who have experience and kind of bring more people in. Sweet, cool, thanks again. I'll talk to you about it. Before and after documentation. Take what we can get. Anything else? What's your Drupal.org username? I thought I had mollusk. Yeah, I remember. Okay. That's not my great specific, I'm afraid, like the core contribution process. I work with some really talented developers who only really do contribute because they find core's contribution process challenging at times. And it can be a little bit like that, but I do feel if you persevere with it, it is also a really good way to learn a lot yourself and work with some really clever people. And writing tests, it gets easier. That's all I can say. Yeah, I really agree. Writing tests isn't easy on its own. It's a learning curve. And I'm very eager to get over that learning curve. But writing tests for migrate is even a level higher. And that's something which I think is a barrier to get new contributors for migrate in the eight. Have you tried again recently? Because we did do quite a lot of clean up on the test to make it easier with like learning the fixtures and things like that. Adam did a lot of good work on that. I wanna hear more about that actually. Yeah, I'd love to hear more about the problems that you have learning the tests because that actually drove me absolutely batshit crazy initially when I started. I like hated the tests more than anything else on Earth. And I did a lot of work to clean them up and make them easier. So if there are ways to make them even easier and I bet that there are, I would love to know more about that because I care a lot about that. I would also agree that the migrate tests, the update.php tests and the migrate tests are the two hardest test suites to write coverage for by far in core. So I would definitely agree with your experience there. One thing to keep in mind is that you can always, if you have a bug fix available, you can post that patch in the queue, tag it and need tests and hope that someone else can come along. But also we do have a sprint tomorrow. I would strongly love, encourage to see you there, whatever that didn't make sense. But they're like look for people in the room who are also working on migrate, sit down at a table with people. It's so much easier to get over those hurdles when if you have like, you get like that first 20% of the way through and then it's like I have no idea how to solve this problem but Adam's there and so then you can get over that little bit. It would be really, really helpful. And then that's also another place where developer documentation for writing migrate tests in particular is good. We have for update.php, we have a very antiquated handbook page that someone spent a lot of time on in early 2011. And then we made that much less important by making migrate the primary rate upgrade between major version. So a repeat of that too would also help solve that contribution problem. And I'm not saying the migrate test can't be improved because they definitely can but it is also quite a difficult thing like you're testing, you know, like a unit test can't get any simpler. But even like a browser test that clicks a button and asserts some, there's a low bioventury, right? You know, you log in and you click it with the update and the migration system that's setting up entire copies of Drupal in one state and then trying to get it, you know, into a new version of Drupal and then so it is also just a difficult area. Getting the fixtures for the test right and yeah. But that's good feedback. So with the migrate UE that used to work and then broke, is there a golden age where you can go back to an examiner and see it working with the old API to actually work out what the end goal is to that we're trying to get to with it and actually is that the same end goal or are we actually looking to make changes? So whether it's exactly what it needs to be if we rebuild it, that's open for question. I did a talk last year and I think I demoed it but if you just wanted to get it up and running, you just check out 8.0 and it'll work with all of that. Basically some point to the 8.2 branch we changed migrations over from configuration entities to be plugins and that was. It was 8.1. Was it 8.1? Yeah, it was 8.1. Shortly before the release there. Okay, just before everyone. Yep, so that was the doomsday for the UI. So if you get back before that commit, it'll work, it should work. And feel free to ping me. I hope it helped you get it set up. The basic idea was the YAML files that you write, it would allow you to build them in the UI and it supported all of the process pipelines and stuff so you could kind of visualize the pipeline that would be applied to the data. If you had a string and then you was exploding it and then you were running it through another filter and then you might implode it back together, you could kind of see the three steps in the UI in the context of that one field. But it was just a UI that replaced the YAML file and half of the reason I haven't rewritten it is I don't find YAML that hard. But not everybody feels that way, I'm sure. So beyond the need for maintainers for the UI, is there a defined roadmap of what you want to see happen with the UI beyond just supporting what was there in D7? Are there proposed features, can you estimate that sort of thing? I guess there's two things to say there. I'm gonna let Jess speak on this, but there's two UIs, three UIs. Is it the UI in core, which is the one-click UI which Jess can probably or Alex can probably tell you more about the roadmap for that. And then there's the UI that we were just talking about then which is a country project for building out migrations from the UI. And then there's the third UI, which is like the dashboard style UI in D7 where it showed you how fast it went and how many nodes have imported 600 out of 800 and things like that where you could click through to see the messages. They're the three UIs and the latter two are both in contrib. I would add that there's actually a fourth UI, a user interface which is Drush. I mean, it's not a user interface in the browser, but it is in the interface by which most people are going to run migrations and that. The, yep, I think that the Drush user interface is more robust and complete, but it also is currently in contrib and we want to put it in Drush core because when the migrate API is in core, it's unexpected to have to install a contributed module only for the Drush commands to interact with that contrib API. Currently that's blocked again on test coverage for Drush is why most hasn't accepted that to my knowledge to Drush core yet. So that's the fourth user interface in quotes. For the first user interface, the sort of MVP, the minimal thing that someone needs in core to do of this basic migration, there is an issue in the migration system queue in the core queue that's something like, it's actually like a roadmap issue that's something like follow up, like plan for the migrate UI that lists every single follow up that we identified during the initial patch. A lot of the work has been done already thanks to like Benji and Mike and others just sort of like suffering through and doing the work of things like basic code cleanup and error reporting and so forth, but there's also a lot of more substantial work outstanding there just in terms of what needs to be reported to the user. And it's possible that so the sort of like full use case of like lining up fields that the, think the second interface that you mentioned does, there's that doesn't necessarily need to be part of the core user interface, but there is also potentially a use case for some of that functionality being in the core user interface as well. It's debatable, but it's not part of the minimal thing that we need to get done, but some of those features might have a place in the user interface in the future as well. So that's something we could discuss if you're interested in helping with that. If anyone's done any duplicate migrations, I'd be really interested to get a, like a 30 second overview about when, feel free to step up and contribute that. Tell me how good our body went. So I, for a while, I was helping out with the migrate. He's a maintainer, he's talking it down. Am I technically a maintainer? Yeah, yeah. I really shouldn't be at this point. All right, I'll create an issue, but. Oh no, we're gonna get you back in the video. Yeah. I was in no escape. But I've done a few, mainly from D6 to D8. And you know, so I've got the inside baseball knowledge because I'm lucky enough to work with everyone and really understand the system. I actually think there's a big fear that's unfounded about actually writing a custom migration. Because if you're just migrating content, I forgot someone mentioned it, like how would you just migrate simple, just content and not configuration? There's a lot of stuff that you can migrate without having to write PHP. It's like, that Benji was saying, it's just YAML. And just getting over that hurdle of understanding that, oh, I've got a content type on this side that I wanna map to a content type on this side and move these fields over. And if they're granted, if they're CCK fields in D6 and supported by Core in D8, it's really straightforward. And you use migrate plus and migrate tools, thank you. And you have your drush migrate status, migrate import, migrate rollback, that all works. So it feels a lot like D7, as Benji was saying again, the concepts are the same. You have a source, you have a destination, you have your pipeline. So for me, it hasn't really, once I learned the syntax, it wasn't an issue. And I have a couple of fairly sizable clients that I'm running D8 migrations for. And I do it every few weeks, the D8 site hasn't launched yet. So every few weeks I basically just take their source database and rerun the migration against the delta. It doesn't update old nodes, which is a little bit of an issue, but that's the high water stuff you were talking about. High water's broken. Yeah. So that's one of the kind of the manual hurdles we have to overcome, but it's. How did it? It's fixed. High water's fixed. Okay, so I should, good for me to know about that. It was broken for long enough. Yeah, it was broken, so I didn't even consider using it. Yeah, it was broken for over a year. Yeah, but I think a lot of it is, it's just getting over that initial hump and writing that first migration. And if you're using migrate tools, the Drush commands are exactly the same. That all seemed like some really good feedback, but I think the main thing for me was I came here to plead to get more people to contribute and we just lost them in there. So that's it. So please don't step down, Mike. Mike contributed a lot to the sort of like issue management very early in migrate as an initiative. The first few years, you know, we did a lot in the sandbox. I don't think everybody knows that. It was a bit of a different approach, but we were going solo in the sandbox. And yeah, there's a lot of valuable contributions from some people like yourself, you know, prior to it coming into court. All right, well, I think that's it unless anyone's got anything else. Okay, it's been tomorrow. I'll be about.