 ac yn y dyfodol diwrnod drwg wedi eu gwreithio. Bydd y gallu unrhyw ddwybodol ymwneud yn gwneud o'r ddwybodol, a sut rydw i'w gydag o'r rhaid i'w gyda. Felly, rydw i chi'n fwy o'r ddwybodol, byddai'n gweithio i ddwybodol a'r ddwybodol i'r ddwybodol i'r ddwybodol. Cysylltiadau'n gwybodol i'r ddwybodol. Dwi'n cael ei ddwybodol. Mae'r ddwybodol ar y dyfodol, dw i'r f wondered i'r gondiwlad. Os ydych chi'n wneud i rai'r gondiwlad iddy, efallaint yn rai'r gondiwlad ar y cyfllyn crefat. Rydw i'w ddoch chi'n siarad yw'r bwrdd, rydw i'w ddysgu o ddweud bryd o'u wneud, yn yr anrhywg wrth fy ngynghwng. Rydw i'n mynd i ddechrau cael ddwyllun o'r swydd yn hyn, Ac i ddysgron maen nhw ychydig ar hyn, ond hyn yn gweld profodi fel iawn. Mae'r profodi yn ffordd i ddau allan yn ein masnedd. Ac rwy'n ei chwestiynau'n aún i ddim o'i gwaith. Nid olygu, rwy'n rhoi, am ymgweithio hwnnw bydd wedi'n gwaith yng nghyddon. Rwy'n gweithio'n cyhoedd, oedd ychydig i'n gweithio erosлаen ffaseraidd, sydd yn Cys. Mae ymddangos hwnnw'n gyffredin, oherwydd rydyn ni'n gweld hynny. Felly mae'n gwychwch, mae'n gwychwch yn gweithio'r link oherwydd. Tess, y cyfleid o edrych, 8 miliwn gwahodd ymlaen, mae'n gweithio'r llun i'n cael ei fod iawn, felly mae'r ffordd o'n cael ei gwaith o'r cyfleid o'r cyfleid sy'n gweithio'r cyfleid, wedi bod i'n gwneud i'n cael ei gwaith. Yn ni'n gweithio, yw'r cyfleid. Yn ni'n gweithio'n gweithio'n gweithio, mae'n gweithio'i gweithio ar gyfer. Can everyone see that?] So what's this presentation about? Now we're getting on to the interesting bit, all the background's done. So it's basically, a talk about the migration strategy that we implemented at TeZ. A we're going to look at some of different migration sources. We're going to look at some example code. Talk about some of the issues that I've had. ..I'm hopefully going to do a live demo of migration into the customer migration, ..which is I've written not working as the case may be. So hopefully that'll work. You probably heard people talking about migration.. ..in terms of extract, transform, load. And that corresponds to three different phases of migration. So when you look at configuration file we also use resource, process, and destination. Ond ond, mae wedi cael ei hunain'r ffordd i'r gymhau yn ffordd trwy'r bandwg, rhai o'r teimlo, rhai o'r proces, rhai o'r log o'r dyfanodau o'r ffordd. Ond o'r ffordd o'r confir ymblifio, mae'n ddefnyddio i mi allan. Rydym gynig i'w chyflwydd i ei wneud. Y rhaid i chi'n eftan ar ôl, a fddwn yn rhaid i chi gyddi am mhawr. ond mae'n cael nod i'r llyfrnodi nesaf, felly ddwy meddwl am ddo lokwch yn taedd. Mae'n ddweud yn gafio chi yn gweithio na'n creu pethau â'r ddwy. Mae'n gofio'r modgol yw'r creu tynnu'r modgol yn eu bod yn ôl. Mae'n braw wneud y modgol yn gyfer, ac yn hunana am rhaid, clontyridd modgol a'r grwphaen mynd yn llyfr y modgol. Ychwanavew eraill yn gweithio'r wybodaeth yn y gweld y gallwn modul yn ein iawn. Roeddwn yn llwyddo chi'n wybodaeth y myddwch yn fwy modul. Yna'r edrych yn y gweld mae'r bobl yn meddwl yn gwybod estae, wrth gwrs ei meddwl yn ei'r bobl yng nghylch. Mae'r bobl yn gweithio'n meddwl yn wych yn gweld yw'r bobl yn ei wneud yn bwysig i'ch gwerthbarth ei fwy dangos. Mae'r bobl yn llwyddo chi'n meddwl yn ei wneud yn ei wneud yn dangos bod yn ein bwysig i'ch gwerthbarth, As I'm going through the presentation, that's a really good one to look at. The migration we're looking at is going to be an SQL migration, although I did do a CSP migration as part of this process on a world touch one as well. The only reason I'm saying that is because the SQL base class is the one we're using for database to database migrations, and that's actually in core. But if you want to do a CSP migration, then you'll need to use the contract module, which I'm going to mention as we go along, and that's my great source CSP. My great tool is the Drush goodies for doing migrations, so you'll need to install that. It's also implementing the UI for migrations, which we'll have a look at as I do the presentation. So it's a bit of a background to the migration. We've got a current D7 site. It's around about 250,000 nodes, of which 215,000 are news articles. So we're doing a partial migration, just migrating our news articles. We've got around about 16,000 authors of news articles, and around about 55,000 managed files that relate to those news articles. The business was proposing a redesign of the news file at the site, and that gave us impetus to actually push for a droop-late migration. Because there was a return on investment, the business didn't see any value in us migrating, but obviously we had this big bit of work, so then we can push for DA on the back of it. This is still a work in progress, so we're actually only partway through this migration. That code hasn't been working live yet, so that's why it might not work when I share to you. So the first phase of push-up down a bit. Before you actually start typing away, developing your migration plug-ins, or designing your content types or any of that kind of stuff, you really want to look at your existing data. You want to make decisions on whether your existing data is actually worth migrating. I've put down the automated versus manual migration. Some bits of the data, you might want to, there might be a small, let's say you've got galleries or something, and you've only got a few galleries. It might be worth doing a manual exercise for those, rather than automating that migration process. The reason I say that is that the manual migration has the benefit that the end-users who are going to be editing the content get familiarity with the CMS, the new drool plate editing functionality as part of that manual migration. The other thing to think about is users will often say of all the content, but if you look at the content and then say to them, well actually there's not that much of that. If you want it, you enter it yourself, they might then actually reconsider whether that's actually worth taking over. So that's one of the things to think about. The other thing as well is that the data analysis that we do, excuse me, moving on, data analysis that we do drives the migration strategy. So it drives our content types that we're going to build, but it also might drive our migration strategy. So that's another thing to think about. I've also mentioned here about getting feedback from existing users. The reason I mentioned this is that we had some ambiguous fields on our news article node, and the end-users didn't actually know which of the fields to use. So when you actually talk to the user and say, are there any issues with your editing content, do you have feedback to you, and that also might drive your target content types? This is the bit where I talk about our test CSB migration. So the first issue we had was we had a load of themas. We wanted to design the front-end stuff, so I didn't have any content, so we needed to get some test content on there. So we did a source CSB migration to staff, just like 100 news articles. So I'm not really going to touch on that very much, but I've put all the code up on the repo so you can pull the code down, have a look at it, and have a look at some of the plug-ins and all that sort of stuff. So source content that we wanted to migrate, so that's not too high for everyone, is it? Let's see that, can we? We had some different vocabaries we want to pull in. We had some author nodes, so the people who were writing news articles, news articles themselves, a load of managed and unmanaged files. We also had embedded content within our news articles, so where they'd actually added media files and stuff like that, one to pull those out as well, and we also had some news images. We had two main images on our news articles, a hero image and a teaser image, and the end users never knew what the difference was between the two, so we'd actually simplified that to actually have one image on our news articles. This was kind of a source data type, and I'm going to show you, hopefully, okay. Hopefully you can see that. So that's our source news article content type. There's quite a lot of fields on it, and there was some ambiguity about some of the fields, so they had, for example, a teaser image. They never knew the difference between news article images and teaser images. We'd actually allowed them to have multiple images, but we only actually themed for one image, so that was a bit of a bug, so they were adding multiple images and we're a bit confused why they weren't seeing a teaser image. So we looked at this content and what we ended up with, a simpler content design, which hopefully will load very shortly. I'll go back to the slide, okay. So we ended up moving from that in the process of moving from that to this content type. So a much simpler content type. Okay, so that was all fed from our data analysis phase. Plus, we also had to, because of the design constraint that we were given by our UX department, we also had some additional fields that we were introducing to meet those requirements, one of which, then if you can see, can you see up there we've got our mobile title. What they wanted to do is they wanted to actually have an article titled for mobile. So that was one of the design decisions that was fed in from our design department. So we're doing a database-to-database migration. So what we've got to do is we've got to define where our source data is coming from. And the simplest way to do that, the way to do that, is to set up an entry in your settings file. We're using the settings.local.php file. I don't know if anyone is using that for your local database connections and stuff. So I've basically set up a migrate database. Put the connection criteria in. Now, unfortunately, this is one of the gotchas. Migrate is the default name, the default database name for all migrations. And because of that, when you actually enable the migrate module, you end up with loads and loads of migrate tables. Because what it does, it goes to that database. It says, is there a database called migrate defined? Oh yes, there is. Let's go away to that database. Look at all the entities that are there and create mappings for those on your target database. So you end up with shed loads of migrate tables. So this was one of the gotchas. That was the wrong choice of name. So I'll show you that in a minute, but we end up with loads of tables that you don't need. So I wouldn't specify all my migrate databases as migrate, call it something else. And then in your configuration, you simply point your configuration to that source database. So obviously, as I said, don't really want to call it migrate, call it something else. So all I'm going to do is I'm going to just switch to my code view bit. So in our code view bit, config files. So basically, so this is our configuration module. I'm sorry, our news migration module. Again, it doesn't have to be, you know, it's the wrong choice of name. It's because we've already done a CSB migration, called News Migration. So because I was then doing a live one, it was called News Migration 2. Created this migration bit module. Created a load of config. Now one of the things that the migrate tools and the other contrary modules for migrate do, they give you the ability to group migrations together. And one of the benefits of grouping migrations together is that you can define a shared configuration. So for this migration I called the group News 2. It should have really been called News, but never mind. And I've added this shared configuration section. So in this shared configuration section you can actually add, define anything that you want your migrations that are in this group to actually use. Now I was using some custom plugins to get my source data. Because I'm doing testing, what I wanted to do was I wanted to define a range of nodes I was going to pick up. Because we've got 215,000 nodes and I didn't want to run that migration on my tests to wait to see that it hadn't worked. Because I'd still be there now doing it. So I added some selection that would then be shared between all the other configurations that were within that group. I specified there, this is another bit of custom configuration and you'll see this when we look at the plugin. We're populating a taxonomy, the tax tax, tax vocabulary. But we've already pre-populated some other vocabularies and what we're doing is we're going to be using on our taxonomy field on our news article. We're going to be mapping multiple taxonomies to that field. So if a term already exists in one of these other taxonomies, other vocabularies, we don't want to actually load it. So I've added this cross-reference list there and what it does is when it's migrating taxonomy terms it looks in those vocabularies to see if that term name already exists and if it does, it doesn't actually migrate it. But then when it comes to actually load it, it also pulls that name corresponding term ID from those taxonomy terms when it comes to actually load it on related entities, but I'll show you one of that. Again, this is all in shared configuration, so nice and easy, one place to do it. I also want to specify what my source domain is. I want to specify a default, what my source public folder is. Again, used in the custom plug-ins I've written and where the default target folder is going to be. All of this is shared and this last bit here is another shared config. What I want to do is I want to, before they actually run the taxonomy migration, I want them to make sure that those vocabularies have already been populated. I'm passing that through into my plug-in so that you can check the requirements phase of the actual migration. You can actually check those vocabularies and make sure they've got terms in them. If they haven't, it won't run the migration. Does that kind of make sense? Then this last bit are constants. All under the source bit, you can actually define constants that you want to then also use in other parts of the migration. One of the things that I was doing was we're not migrating users at the moment, so we're actually marking up all of our users against our instances that we're creating as being the admin user. In here I'm going to be just defining a user that I want to actually pass through as a user ID on those nodes. Ultimately I might want to actually pick up users and migrate them as well, and then I wouldn't be using that constant. We've got defaults for our process, so we're defining a default value for the language, and then we've also got an enforced dependency. Now what that means is, let me actually refrain, and one of the cultures that I had, which I'm going to talk about later, was that I wrote the module, I installed it, sat up on the config, and that's fantastic, and then I uninstalled the module and tried to reinstall it, and it fell over. That's because the config existed in my database, in the config table, and it tried to create that config again, and it fell over and it was trying to reinstall it. Now, if you put this dependency on each of your configurations in your migration, and when you uninstall the module, it will delete that config from the configuration table. If you don't, you have to do a hook uninstall in your install file to manually delete them. So that's a bit of a gotcha. I'm putting my hair up without to start off with. So you'll see that in each of the configuration files on each of the migrations. So use groups, the ability to group migrations is really useful, and we'll look at some of the other migrations stuff in a minute. So I'm going to just delve in into code and all that sort of thing, if everyone's okay with that. And in the meantime, we'll sort of enable the module as well. So I'm going to one of the easy ones. So tags, tags are nice and easy because we're basically migrating just the name and description. So here we're using a custom plug-in, custom source plug-in. All of our database related migrations have their own source plug-ins. And the reason for that is because you need to tell the configuration where to get the data from. So you'll see a plug-in for each of these sources. And they live underneath the plug-in folder right down in the tree, right underneath source. And you'll see it down here. So that's the plug-in ID. And when you define the plug-in class can be called whatever you like. A bit that actually maps it is this migrate source bit. So that's where you're defining the ID for that migration. If you see errors in my code please don't shout it out. Just message me afterwards. So this is where the tags bit is being done. So if we go back to the actual tag migration file, nice simple mapping. We're mapping the name and the description. We're using the vocabulary we want to map to. We're using this core plug-in default value. And we're giving it the value of tags. So we're just actually default value enables you to put a default value against the field. There's a whole load of core process plug-ins. There's a good page including some links in the presentation. A good page of lists of core process plug-ins. But you can easily write your own. So that's my problem. I'm just going to move this to one over at the moment. So we're there. So we're pulling in, as part of this migration, we're pulling in three. We're pulling in I've got some pre-populating vocabaries which are region space and subjects. We're pulling in three different taxonomy fields. Because on our own news article we had three taxonomy fields and we're mapping those to a single taxonomy field on our new content type. So these are the three fields. Now, if the name from the taxonomy exists in any of those pre-populated vocabaries, which are these, then we don't want to migrate it. We want to use the term ID that's in that. And we have a migration requirement that those vocabaries, these ones, have been pre-populated. So we're switching back to the code. So what you'll see in these database to database migrations are at least the ones I'm showing you. Much of the transport logic is being done in the source plugin. So in the prepare row bit you can do all the stuff that you might do in your transformation in the process bar. You can actually do it in there. And I'll just find that a little bit easier. But you can write process programs as well if you prefer. I'll show you examples of both of those. But when we look through some of this code you'll see that most of the logic is done in the source. So this, as I say, is the tags one. We're extending the core SQL base class. We've got a check requirements function. So in here what we're doing is we're actually this bit here was just because what I wanted to do was make sure that the source database existed and it had tables that were populated because I had a script that I was doing to build by that environment and I had a bug in that script and I was dropping my source database and recreating an empty database. So my tables didn't exist and I was scratching my head and I thought I'll just put this in there. Shouldn't need to do this really. All this does is actually checks that the source database exists. The tables that I'm expecting exist. This bit here is looking at a configuration and it's looking for required vocabulary IDs. So you might be able to define that in our shared configuration in the groups. So if that exists and what it's doing is passing those vocabulary IDs over to this check ref data populated and all that's doing is going through those vocabularies and making sure that it's got terms and if it hasn't it returns false and we throw a requirements exception. So what it will actually do if you're running it from the UI when you run it actually let's run it from the UI and you'll see what it does. So this is an empty database I haven't even enabled the module yet. So what I'm going to do is I'm going to go into enable the module. So this UI that we're seeing here is what the Migrate Tools contribution provides you with and I was using this for most of my testing. Obviously if you're doing it live you get much more flexibility through the Drush commands but obviously for the testing that I'm doing it was absolutely fine. So I'll do my list migration. So those are all the migrations that are defined in the group news2. When you look at it in Drush I'm going to show you on the Drush side as well. When you run it on Drush like yeah, okay run it on Drush it's doing more or less the same thing as Drush. The difference is that the Drush version takes account of any dependencies that you've defined in your migrations. When I say dependencies let's say for arguments like you're loading a news article but that news article you're actually populating to third field with entities that you create in other parts of migration like taxonomy terms like image files, file entities and authors. So your news migration is then dependent on those other migrations having been run and what Drush does is it will order this, it will work out based on those dependencies which order these migrations should be running and then show you those migrations. Now I don't know if you can see we were just looking at the tags migration weren't we just now. You see that's not actually there and the reason that's not there is because I haven't run the pre-population for that part of the migration so it's failed to check requirements so it's not actually showing. So I'm just going to see here we have some other migrations that were written by somebody else that are populating this what we call in breath data and I don't know if you can see so in the composer file, the composer Jason Farre they've defined a script to run those files. It's quite a nice little way of actually backing up scripts like that because then you can actually run them as a script and I'll show you how that looks I mean it's shown in the presentation documents but if I'm just showing you how that looks can you see can you see that file there so that's the composer Jason file and this is the script section so you can actually define the different scripts that you want to run and so when I did that in pull breath data it was running CD to web and then all of that all those drush commands so just a little to see whether it's useful to you or not just that initial so can you see that's running and it's populating a load of taxonomy terms and when it finishes what should happen and it's populating all the leaves taxonomies down here what should happen when it finishes when it finishes ok so if I now was to run my group and obviously I need to be in my web folder I should see can you see I've now got my tag migration at the top which I didn't have before because it failed the check with Climax so it's actually clivering that way won't let you run things if they fail the migration the UI will let you run the stuff it just won't do anything it runs and says no nodes loaded and you'll actually then need to go to the db log to actually see the quiet error message so what I can do now is go to my migrations this one I'm going to run from the UI so I'm going to run the tag migration so 284 nodes and if we look at the we look at the code so the code that 284 is being pulled back by the queering method so it's running this query and it's getting back a total number of rows in it so it knows there's 284 rows that it's going to migrate or it thinks it's going to migrate because you can actually stop it from migrating stuff in the prepare row by returning force so I'm going to try to execute that and before I do that I'll just show you the prepare row there so in my prepare row there there, what I've actually got is this bit here so what this bit is doing is it's going away and it's looking for that name in a load of vocabulary that exists in return force so the idea is that it won't actually migrate that row so with that says 284 I'm expecting it to actually migrate mess so I'll click on the execute thing, I'll go to the execution page and I don't need to tick any of these because this is the first time I'm running it if I run it before I'll probably want to tick that on a day some stuff so I'm going to click on an execute that will fire off a batch run and as you can see it's going through and it's doing something it's doing something so while that's running I switch back to my tags so we've talked about the source bit and we've kind of talked about the process bit because that's where our mapping is being done what it's actually the target entity that we're creating is a taxonomy term so we're using the the entity plug-in entity taxonomy term plug-in so the stuff that comes out of our process bit is going to be chucked into that class to actually do our load so switching back to the browser 64% through the minute is whether it's actually worked or not it always seems faster through Drush as well, that's the other thing so probably that's been finished now so actually it's created 245 taxonomy terms because this other 39 already existed and we can prove that by looking at the existing but we can look at our tags so this is what we just populated so we've done the first part of that migration and you can see when you look at the migration UI we've now got a last imported date as well as that we also get entries in our migration tables so what actually happens is on the right place so this is where I was telling you about all these loads of migration map tables can you see that? shed loads of them so don't call your database name migrate, call it something else so if we look at our tags migration map so what it'll actually do is this contains a mapping from our source ID to our target ID so when we do our other related migrations we can say to that migration and say use the cross reference in here to get the target ID and we'll see about all the different migrations so our tags are migrated now back to here if we were to run that now we again see the same thing that we saw through the UI so the next bit we want to do looking at this list here we want to actually put in some our file entities probably run out of time actually so I will try and speed up a bit so our file entity ones two of these screens are done so if we go to our file entities which are what's called all the images we want to do so all three images what we're going to do here is we're going to we're using another source plugin this time called d7 news file if we look down here this is our class that we've written for that we've extended the we've extended the file class I've just given it an alias now just we've got some other requirements in here so as well as checking that the target database I'm also making sure that we've got we've specified folders for the migration so we've specified a source domain we've got a public folder we've got a target folder all that configuration exists we're also using some source SQL and the reason we're doing that is because we've got lots of files that we're going to migrate I've just decided I was going to write a utility where I could pass through the names of the fields and it would actually then do the appropriate drawings to the managed files table so instead of writing an individual a bit querying logic for each of the different file types just actually specify the source tables in here and then my source plugin would actually do the magic for me which it would do down here if you can see that so it builds the joins and all that sort of stuff I've added an additional we're actually extending the file class which has its own set of fields but I'm also adding some additional an additional dummy field to it or custom field which is this destination path and the reason I'll do that is because from my source plugin I'm saving to the migration where I want the file to be saved and I'm building that destination path in my prepare row does that all make sense yes no so in my prepare row I'm going away and putting a load of stuff and you can probably see down here so one of the things I'm doing is I'm taking my source file name and I'm trying to niceify it so I'm trying to where it's got embedded spaces or mixed case or stuff like that I'm trying to make it into a nicer format name what I'm then doing is I'm going to be building up you can see it down here this destination file path so from the folders that I've passed in in configuration I'm going to build up the name of where I want it to be saved if that will make sense and then I basically just have to set that property back on the source row so then that property becomes available that menu property becomes available in my process part of the migration and you'll notice that in my process part of the migration if we look at our we'll help those looking at the migration too many screens open let me shut this one down let me say this one so if we look at the images bit if we look at our process part there I'm using the file destination not just build to point to where I want the destination of the file to go so as you can see what I'm doing is I'm trying to do most of my logic in the source plugins I could have done this in the process one as a word but I just find it easier to do it in the source ones so I'm going to just run that migration so that is this news authors images if I can find it, my eyes are terrible now going back to here what we've got is we want a target folder of authors images so we're looking at my site you know my site so I've got no real folders in there so what I want to do is I'm going to create this migration should create that destination folder which is author images and should populate it with the author images for the file migration so I'm going to run this and fingers crossed it will work so two parts of this one part is go away use the download mechanism to get the file create the folder it's going to go to save it in that folder create a file entity and if I look back at my site's folder I'm hoping I'm going to see my author images which I am and if I go back to my database what I should see so let's have a look 18 files, 18 migrating what I should see file entities created on my talk of database probably this ends at 10.2 I'm pretty much there again actually I'm sorry about that I'm just going to quickly show you one process plugin to just give you an idea of two process plugins and then if you've got any questions can you just remove the CD on sorry that's because I yep so this is where this is where having stuff in gear is great because you can just invert it that's a trouble with having two of these screens open as well then if you've found that typing in the wrong room and window sometimes okay so I'm just going to quickly look at a couple of process plugins so the first one I'm going to look at is this D7 stomp has anyone heard the concept of or anyone aware of the concept of stubbing crank stumps to things so what you'll see in most of the migrations stuff I will share this URL with everyone in fact you can actually just take a note of it so this URL here that contains all of the that contains all of the source code that I've shown you as well as all these slides and a presentation another big read me file with gotchas and examples of things that went wrong for me so you can pull that down I think that they're going to actually publish the links to all this sort of stuff at some stage I don't know when but you'll see that most of the migrations that I've got are specified in no stub of true which basically means I don't want you to create a stub entry for these because I'm going to do the migration in the sequence that it's intended to be so that when I get to the point where I'm doing a migration I shouldn't need to that entity should exist the only one that's the exception for that is the news articles because they reference themselves because they've got a related content entity so you do need to create a stub for those and so when you look at the migration you'll see that there's no stub for that that is commented out and when you create if I look on here when it creates stub entities can you see some of these lovely names can you see that really rolls off the tongue doesn't it that's the sort of naming convention that it does when it creates a stub entity so what I've done but I only did it this morning I had actually changed that I've written a a little a little process plugin called stubname and what I've done with that one if I just open up my recent so what I've done and the reason I didn't do it on my problem migration stuff is because I didn't want it to break it I don't know if you can see that alright but what I'm doing here in my title I'm using my D7 stubname plugin with a source of title so the source of that plugin on source value is the title found and it prefits in