 Okay, well, hello everyone. I think there's still some people coming in, but we'll get started. So today we're going to be talking about migrations. And this is the one thing that we like to do before we start a presentation on any topic is to ask, well, why? Why would you want to migrate? And in the case of migration, the opposite question, why wouldn't you migrate is actually a very good question, because you have a website. It works. It's functional. Why would you want to migrate? And, well, sometimes you don't need to migrate. You don't always need to migrate. Sometimes you can just leave your website as it is, or sometimes, well, as you mentioned yesterday, you can also just shut it down, and that's the end of it. But sometimes you need to migrate. And Janice found this quote from a general, I think, who wanted to stop a Russian politician who wanted to stop bird migration. And no, you cannot do that. It's impossible. And in the same way, when you need to migrate, you know that you have to migrate. So if you're stuck with a system, this could be an enterprise preparatory content management system. It could be a novel platform that's unmaintained. When you need to migrate, well, you just have to migrate. It's a natural thing. And, well, when you can't migrate or when you really have to, well, it kind of feels like being in prison. The most famous prison is, of course, Alcatraz, which gave its name to the title of our session. On our escape from Alcatraz, we have two fellow criminals here to assist you on this way, this way out. So hello, everyone. My name is Florian Lorten. I'm a software architect at Wunderkraut, Germany. And I'm Janice. I'm a developer in Wunderkraut, Latvian office, so he's welcome to our criminal session. And now let's get started, not with the way it is now, but with things, well, what are the different ways that people have done migrations before? And actually, we took it in a way, let's, let's, we took it in a way, really, we looked back in the history, how people actually migrated away or to some things. And we looked back really, really far away in the history. And then we actually looked back, you know, back in the days, about 4,000 years ago, people had some writings. So they basically were writing things, not in a computer or a paper, they're writing things on stones. So you had to do a pretty heavy stuff. So you had to rely on quite reliable animals. And firstly, we used donkeys, mules, and camels to migrate things from one place to another. People used to migrate data. Right, we moved on. And then we, we became clever. So we invented typing machines. It's a great way to migrate. The next thing, when we became more clever, they actually started to use more clever options, things like copying and pasting. Has anyone done this? No one? It's okay, it's okay. It's okay, fine. Yeah, you're great people. We're starting copying and pasting. A friend of mine copied a book of 2,400 pages from a web page to an Excel spreadsheet. So it's just for your inspiration. If you can't get things done, go back to old days. Right, if we move to the next stage, we probably have raised so big in our organization, so we don't have time to do this copying, pasting. So what we do, we hire somebody. Well, actually, we don't hire anyone. We just say somebody that we are going to hire that person. And at the end, we just say, I'm sorry, we don't have budget, you will be unpaid. So, and we'll give you this wonderful option to be an intern and a massive big organization. We don't do that, but some do. We go more there to the right, more to the direction we want to talk about later. Then obviously, we start to move things into various sophisticated systems, things like Drupal. And then when we moved to Drupal, when there went back in the days, we didn't have any migration support to kind of maintain and stuff like that, we used to do a custom script. If you can read it thoroughly, you'll get a chocolate for me. Right, this is my own creation, my own script. It went very well. By the way, there were 1500 items, files and some texts. It went well, but it had no backwards compatibility. By that, I mean it had no rollback options. So, you created nodes and they were kind of stable, stubborn. And finally, this is the thing I mostly like. I've done this many times before, actually, start to do things in a more clever way. This is insert into statements. Has anyone done this? One, two, three, ten. Don't be afraid. This is weird. Yeah. Don't be afraid. You can be honest here. Yeah. We used to do insert answers, by the way. If you can create this, you can create users. You can take users from some weird database or a system of tables and just put them into a Drupal database system. For instance, this is Drupal 7. Good. Since we've covered the history, we will now see how to do things in a proper way, in a Drupal way. Let's go ahead and look at the way how we can do it. Yeah. So, in the modern Drupal ecosystem, there are two main modules that can help us with migrations. Most popular one, I would say, is the migrate module. It is the one that we will focus on mainly today, but it is worth mentioning the feeds module, which is also an alternative. The migrate module and feeds module work in different ways, but you can do pretty much everything with both of them. But they have very different directions, different approaches to the task of doing a migration. The migrate module is really about migrating all the data from an obsolete system that you really want to get rid of. You get all the data out, and you put it in your system, leave the old one behind. The feeds module is more about synchronizing data between an old environment and a new one. It works well to do that. And you can actually do a migration as a synchronization, or you can do a synchronization as a migration. But I think it's important to know what kind you're doing. And if you want to do a synchronization, maybe have a look at feeds. If you want to do a migration, look at the migrate module, and this is the one that we're going to talk about today. So no matter which one you use, migration always has three basic steps. The first one is to get your data out of the old system, then to do some processing of the data to make sure that it matches the format that you want to have, and then you want to save that data somewhere. So you have a source, some mappings, and then you have a destination. In our example of getting out of prison, not the legal way. First, you dig a hole, you get out. Then you have some transformation that's where most of the work happens, crawling, running, swimming, as fast as possible, and then you want to make sure that this stays the way it is, so you change your identity or something like that. I'm not an expert. In terms of doing this as an actual migration from an external source to a Drupal website, for example, we could be importing from a CSV file or from a SQL database. So the first step is really to get the data. The second step is to figure out how the old data will be mapped to the new data, and then we actually want to save the node or the user or anything that we want to save in the Drupal system. Here it's very interesting to see this separation because it makes it very clear why it's helpful to use a system like the migrate module. We have sources, we have destinations, and we have the mapping. The sources, it's generally something very common, a CSV file, a SQL database, an API from some external system. The destination, it's also very common. It's an entity. It's a node. It's a file. It's something like this. And the only part that's really where the business logic lies, the thing that's really product specific, is the step in the middle. And when you use the migrate module, for example, you can actually reuse an existing source, an existing destination, and just focus on the mapping part. So how does it work? Here's an example. Well, actually, the migrate module itself has a very good example module. And it's an example that migrates beer, that creates beer nodes. The thing, though, is that it only has three different kinds of beer. And I don't think that this is relevant. And especially being in the Czech Republic, we thought that, you know, three beers, we've already had more than three different kinds of beer at the bar. So we want to have something that's more relevant. So we found a website that documents Czech beer. And as you can see, it's a beautiful website, modern web design, fully responsive. But it could use a migration. And so what we did in this case, we did a copy-paste, and we put this in a spreadsheet, which we exported as CSV file. This was the only copy and pasting step involved in this example. And then we decided to create a module. So the migrate module, it is a module, but it does not do any work for you. So it does not declare any migrations. It declares the tools. And in order to use the migrate module, you have to create your own module, which will build on top of the migrate framework. So here, this is just the info file. Nothing special about this. We depend on the migrate module. And we have an include file, which I'll get to in just a second. So there's a module file, the mymodulename.module. This one stays empty. You don't need to have anything in there. The migrate module, well, a migration really happens, and it's a separate thing. It does not happen while your website is running. It's a separate step. So for this, we have a separate include file, the mymodulename.migrate.inc, and there we implement hook migrate API. There's two things that we need to define. First, the version of the API that we want to use. This is pretty self-explanatory. And then we want to define a list of the migrations that we want to do. So one specific migration in this case would be I want to import content of this type. I want to import users. I want to import a list of breweries, for example. So we define this, and we just indicate the class name. This class is going to be in a separate include file, which we included in our info file. And here, we just extend the base migration class from the migrate module. There's only one method that we need to override from the base class. It is the constructor. So we just override the constructor, call the parent constructor first. And everything else that follows is happening inside of the constructor. First step, we just give a good description. I think CheckBear is enough in this case. Not very descriptive, but it works. The second step is to define what other migrations need to happen before this. So typically, if you want to associate nodes with users, you would migrate the user first, and then when you import the nodes, you attach nodes to the proper users. In this case, well, we could have had an example with, well, you want to import breweries first, and then import the beer, something like this. This is the place where we would have done it. But in this case, nothing. Nothing needs to be done. One more thing that we need to do is we need to establish a mapping between the old pieces of content and the new pieces of content to figure out what were the identifiers in the old system and what are they in the new system. This is also something that's really important so that we can roll back the content that we imported so we can test the important multiple times. And the migrate module has a helper class that pretty much does all the work for this. We just give it the idea of the old system, the ideas in the new system, and it just goes ahead and creates a table and establishes the mapping as the migration gets run. Now we get into the part that I talked about before. We have a source and a destination. We're going to start with the source. In this case, we have a CSV file, so we just indicate where the CSV file is located on the file system, and we define explicitly which columns are available in the CSV file. You can also use the headers from the CSV file that works too, but sometimes it's better to define things formally, so you know exactly what to expect. You know exactly what ideas you're going to have. And then we just add the source to our migration. Same thing for the destination, but this time it's even simpler. We just want to save this into nodes of the type beer, which we have defined previously, and this is the only thing you need to do to set the destination. Now comes the mapping, not the mapping between individual pieces of content, but really about the fields available on these pieces of content. So in this case, we want to map the columns of the CSV file that we are importing to the fields on the nodes that we want to create. So this is something that we can do in code, and this is also something that you can do through the user interface, starting with the 2.6. So it's a release candidate at nabond, so if you just do drushgl migrate, you will not get this version, but it should be out soon, and if you explicitly want this one, you can definitely get it. Very stable. And here you can set the mapping not through code, but through user interface. So if you're a developer, you will not get a huge advantage of that. But what is very nice is that, well, this mapping is something that really often requires input from the customer, and Yanis will talk a little bit more about that later. But having a user interface like this actually makes it possible to take this into the meeting room and say, well, you have this, and what should this be mapped to? And you can really have a conversation with your customer through this user interface. This is not something that you can do with writing code. And finally, through user interface, you can also select the migration to import, and here we see 16 bear types were imported, not very much, but for example, it works. So as a proof that it works, we have functional code that does this. This is the same data exported from that web page into a Drupal website. All the nodes were created, no copy-based involved in the migration itself. And of course, you also have a Drush command that can run the migrations, can roll back the migration, all the operations that you can do in the front end, you can also do them through the command line interface via Drush. And this is very helpful, not because, well, it's cool and you can use the command line, but this is something that you can trigger as part of a script, like a deployment script, or a continuous integration script, or if you want to have something that runs regularly, you can also use a cron job. So this is a very, very powerful tool, and you can really integrate this into much bigger workflows. And now we're going to talk a little bit about the less technical aspect, but just as important, if not even more important, planning and collaboration before the migration happens. As we, as Florian just demonstrated, the migration module itself, it's quite easy to set up things. You can set up field mappings, use various sorts of SQL, CSVs and things like that, but if you move to more sophisticated systems, you get maybe more data or you get more sophisticated data stored in many columns or many databases, you would definitely need to go a couple of steps before you actually start to do any coding or even starting about thinking about coding. So let's go ahead and move to a very important subject planning. And I've noted myself down from experience what things you have to go through, and what I would recommend to go through before you actually move to any coding creating modules or maybe thinking what to do. First thing, field mapping. You showed an example. It's easy to do in terms of an interface and the programming, but field mapping here, I mean, you have to agree with your customer which fields go where you sometimes and have not like we had five, five or six columns. Yeah, five or six columns. That's that's not much. That's really easy. But if you can have sometimes 10, 20 multiple columns, you may need to join other tables. So the system you create at the end, it needs a really, really thorough look through it and just make sense to discuss it which fields in order to agree with a customer. What is the end result he wants to see? This is an initial thing you have to do. You have to do field mapping. The one of the easiest things you just create a spreadsheet and you if customer has given you some data, you create a selection of fields you can offer from the source and a sexual action of fields you have developed on your Drupal project and you just go to a meeting and agree which fields go where you can use the same interface we've just demonstrated. Second important thing is time when when you do migration. It really matters. Sometimes it your migration is a crucial part of your work. You are the way how your project is set up. You're not able to do any further work if you haven't done migration if you haven't pulled the confront from somewhere. So it needs to be planned when how you need to discuss with the customer things like what systems and stuff like that what time maybe the sites the project sites they're not allowed to have a massive downtime they can have only very short periods you can switch off the site and think various things like that. So plan it when agree on that amount of quality of legacy data third another crucial aspect first thing amount so how many how much data you have the type of data whether there are files whether there are files like by files I mean images documents PDFs etc etc the amount of data how many how many are those files or how much rows you have in your database tables you want to migrate or how much how many rows you have in your CSV file or XML file you need to go through them and just see data quality the fourth one Florian will demonstrate you an example about not the best source data quality but these things can happen so be make sure that you're the data quality you are satisfied as an organization as a implementation organization the one who implements the site does the migration in your workflow make sure that you are kind of less more or less satisfied with the quality of data any weird character sets we will talk about character encoding and things like that they have to be whether removed or encoded decoded or whether just discussed with a customer that they might be some issues and migration workflow that's that's a crucial import all those five steps this is your workflow when to do and how to do and things why to do and in my example a little bit later I'll demonstrate why it matters the next thing collaboration this is a crucial thing because when you migrate this is like anyone can imagine it's like a building a house you built a house and you have various stages you the initial stage you have to set up your fundament all the concrete works you had to dig the pipelines so you make sure that electricity is there all the wired networks and things like that but your client he doesn't see actually the result you just do something you do lots of mess and modern things like that so there should be some collaboration involved you should demonstrate your customer what's going on why it is important and things like that and by the way and in about don't know 95 percent of cases customer will not use the end result of your work by that I mean that your code your code will not be reused so collaborate between team developers you can do more you can work concurrently on the same tasks you can have multiple migrations one person can do a worker one migration other can do the other one collaborate so what dependencies you know which should go first with clients always try to demonstrate them as long as you're ready with just a simple part of migration try to demonstrate them so it works to build trust workflow integration again try to integrate within your project's workflow try to understand where you are why you are how to do it what when it's the best time to do it and things like that and yeah those are kind of three basic rules for your correlation and let's now go ahead and move to more interesting part let's look at some examples of our things we've done minus first this is just recently finished and this is the one which I can't show you because this the work is still going on we've done the migration but the work is still going on this is our workflow so I can't show you what I'll just go through this is a Swiss Red Cross website about 1500 nodes and about same amount of files we migrated maybe not a big figure but it was pretty interesting and it was migrated from type of three yeah it was migrated of systems of type of three mixed together of various custom builds so actually if you look at table systems or the dumps we received from a customer would you didn't have any prefixes typical to typo systems they were just some tables and the way our kind of customer wanted to be them displayed in a various way so you got those 1500 rows about 35 columns and he wanted to pull the data he wanted to create entities file entities nodes node types different things taxonomize them and things like that it has to be done via migration it had to be done sorry nevertheless and what was the interesting thing customer knew Drupal so he knew possibilities of Drupal I don't know why maybe he read lots of blogs about the migration but he knew very well that that's possible so he knew Drupal in the same time we were always getting the quick feedback so anytime I send him some some queries some messages I don't understand your what you're saying to me or I'm not fully sure whether this has to go there or whether this has to be categorized it was so easy to work with him easy easy feedback he spoke perfect Drupal language perfect Drupal language he understood everything customer it supplied us field mapping so we just created a small spreadsheet having you whatever will we just stop you had a list of fields we have in a source you had a list of fields we've we've created based on your specification and can you please just add a third column and give us what you want to see if you did that reusable meeting notes perfect thing he created a taxonomic sorry categorization but so how I was able to make them reusable so what I did I downloaded that small really small spreadsheet as a CSV file and I created a parser which just parsed the data in that CSV and that gave me the categorization of items because sometimes in the source you may not get the data you want to see in a destination so there is a there is a possibility for you to apply some callbacks and function which you can parse the data and then put it back to a system and obviously based on all this we built an initial trust and a very very good communication it's kind of here is bonded so quickly there were no emails at all we chatted on the skype mostly so nice and easy work and I think next one is a bit more interesting yeah a bit more data yes so this is for a different project we were not migrating from a website but we were migrating from a commercial database so we needed to have a list of all commercial addresses in Germany it's about three and a half million addresses with metadata and it's a lot of data so we we ordered when we bought this the place where we bought it they say okay it's going to be there in three days and like why what does it take so long like well that's the time it takes for the mail to deliver the dvd like okay that's so just for a reminder this is what a dvd looks like so it's a round disk that stores data and it can be a good amount of data um so it's three gigabytes over three gigabytes of text data and this I mean it's a CSV file so relatively standard format but excel can't open it because it's too big no no no uh excel alternative can open it um and well it's also too big for uh for php to open the file at once so the only way to do this is to go through chunks and by default the migrates uh csv source does not do that so we we ended up just killing our web server uh well local environment but but still uh trying to do that important there's actually a quick trick which I will get to um but there's one well a couple more issues it's um well the the imports well when you import a hundred items or what I showed in my examples you see exactly how many milliseconds it took and this is very important because originally we wanted to import the data we also had a separate source for logos for each of the companies that we wanted to represent and we wanted to save the files and we realized that at the speed uh at which things were going it would have taken two and a half months of just letting the script run if we wanted to migrate all the data um so this you know the the the time it takes and how many you get uh how many items important per seconds or per minute this is very important information because if you have a lot of data it's going to matter and in the end well we ended up making some compromises uh separating the logo import from the from the the just plain text import and we were able to optimize this to run into well in 20 minutes so it's still a good amount of time but it's still a lot less than 20 uh than over two months and also the file was encoded as um as a windows 8 encoding and so all the special characters all the accents which of course matter a lot in germany well they they were all messed up and so we it's pretty easy to fix you just need to fix the file encoding before you import it but it this is really something that you have to notice while you are testing the import because there is really no way to to to fix it later um well there are other ways but it's really you really don't want to be doing that so it's it's very important to recognize text encoding issues first and and really be able to address them um when you want to migrate a very big data source and this applies to the csv source but also if you want to migrate a lot of data from from a sql file well from a sql database and you want to make a query if you query for everything it's also not going to work that well so what you want to do is in well and the thing that that actually causes the problem is that when you try to import a csv file uh the migrate module will try to figure out how many lines are in there and the way it does that it goes through every line and just counts them so if you have a lot of lines like over three millions then it will just run continue until it runs out of memory uh so what you really want to do there is to just say well i know how many how much data is in there more or less and i well don't count them just go through it and there is this skip counts parameter that you can pass to any any source and it actually makes it work so that you can really advance chunk by chunk and it works really well now we have one more example and lucky to you this is live you can see this website uh this is this is quite interesting well it's a property website just there's why not fully understand what it does you can sign up you can post your ad and then others can rent it so it's a fairly interesting system and this is an example of migrating from one drupal to another one it's kind of we are we've heard a dry speaking about the drupal 8 so i think there will be a pretty interesting subject in the next six or a month how to migrate from all drupal systems to new and this is exactly an example this is drupal 6 to drupal 7 uh not kind of a straight migration site had to be re developed so it's it's a different completely different look what it was to so uh yeah i think we faced a couple of challenges a couple but uh it was a sophisticated list of destination fields the way our customer wanted to display and the way here we wanted site to work we had a lots of fields very in a various formats we had a lots of field collections which is one of the uh challenging situations in the migration we had a lots of entities creating when the node gets saved and things like that it was people don't do that these days but this was a requirement uber cut to uber cut migration so two to three uh we did that yes we it was a challenge in the first instance because uber cut it doesn't kind of comply to any typical drupal way it just it's kind of an entity but in the same way it's a custom table so it's kind of you don't know what it is but you can do there's a lots of custom entities as i mentioned already creating getting created when the when when you create and when you add a new property let's put things through uh kind of as they are if you create an entity a lots of other entities are created lots of messaging system then the messages are stored in a table then they're getting emailed and things like that various stuff uh long set of process yeah it was really long and painful to set it up unmaintained files it was the most funniest thing so we had files as you know drupal stores managed and non-managed files managed files are the ones which drupal keeps the file path is in the database system so can easily find them you just run a query and just hope that the files exist actually in the file system but we had a lots of unmaintained files and unreliable data sources so uh in terms of the columns were a bit different the data stored in a drupal 6 obviously the data was different from the way it should be stored in drupal 7 and things like that right and i'll go through a couple of code examples how we resolved them first thing drupal this was done with the help of another module which is really great one it's called migrate drupal to drupal it's a yeah helpful one if you want to migrate in between drupal uh releases and the way you do it it's a bit easier way uh this is taken from the example module which is in the drupal migrate to drupal drupal to drupal migrate uh the way you apply it whenever i flush caches or caches then it just runs through my register migration register and checks whether there is something new uh you can do it anyway but that was just a convenience thing for us then uh i'm not going to kind of browse through it thoroughly but just a couple of things what you do here you divine a source which is a separate database legacy source version this is crucial you have to say which version you are mating from migrating from on this case was six and then you set up the first thing we migrated kind of having done all the mapping collaboration thinking planning that we did users and that was kind of planned everything else depended mostly on users and we define the users we define some array of arguments which are some are pretty obvious you define description and the machine name and then you just this is a static method in the migration module a bit one version downwards 2.5 you just register the migration and apply those arguments you just created and another example which just i would demonstrate you how you can use this as a dependency so we had some node types we had a source type i've just renamed them because i can't disclose you the node types in the original system and the destination type so we had those types created and you just create again you create an array of arguments various things you want to do class name description that appears on your UI whether you do it's on drush or a web interface and then machine name and stuff like that and then here and down in the bottom we apply the user dependency so all migration depends on a user migration so users have to be migrated first and then we can say afterwards we can basically apply those users to a node pretty tough stuff and then yeah that's it this is probably a bit weird example but that's again this taken from the example module within the Drupal to Drupal migrate and it just allows you to to merge those drusers to various node types you may register various node migrations like not only one in this case but many then just run through all of them apply the user dependency and then register the migration so what we did the other thing what we did we we were not migrating all entities using migration we were creating custom entities on the ply and then you can use hooks so there are various things which which kind of various processes which takes place when you create something in Drupal users nodes so in this case i use an example of hook user insert and then here is the crucial thing is this first three lines of code for with a comment you'll have to check whether you are in the migration or not otherwise if you don't do that your the following the following script will probably run any time when you will create a new user so this is just a simple check of whether if we are in migration cool and i'm just doing a negative statement because i had a lots of code following and then i just say if that's not set stop stop the script in this module and then we had another problems with data consistency and the the validity of data so i just used try the type is some type this is a bundle as you know entities in Drupal have types and bundles in nodes case entities node and types are just content types and you have to say which type you want to create in the profile so in this case we were creating profile to entities this i just description away of some fields for Drupal more fields like address first name surname and things like that and then i ran a specific function i just put the code in a function and then if it doesn't work it just carries on to the next iteration that's it and if you want to find more about that where i took it from this great blog post gave me a great inspiration and this is the actual function but i do i just demonstrate to you that i assign some values to user and to a profile entity and then obviously i return it so it gets created yep that's basically it and now we will talk about the tricky subject if you want to move away from Drupal so i think this uh well this might be uh maybe a little bit controversial but sometimes Drupal is alcatraz an example of this i think there's many people who have a Drupal based blog and they're realizing that well it's a little bit too much of a hassle to have a server that runs Apache and PHP and all that where you just want to have a blog that's just read only and so people use static site generators the problem is how do you get your 10 years of blogging history into such a system and well um actually the the same thing uh the same thing the same principles apply so instead of migrating from a different system to Drupal well your source is Drupal you want to do some transformation and then you save this into into YAML files which is the most common standard used for static site generators and one thing that's well static site generators can be pretty cool but they don't really have that many tools for migrating there's some scripts that are around but you know they're of the same quality of what we showed earlier as an example of how things used to be done in the past and so the tools there are really not that great and we happen to have the migrate module which is very agnostic about which sources and which targets it's used so you can actually use the migrate module to migrate away from Drupal so you define your source as the your Drupal site could be nodes could be users whatever you want to migrate and then on the destination you just save files and then you can take those files wherever you want to go um and so that's a tool I think that is pretty familiar to to many of us and it's a tool that actually does a great job of this so now we have a quick checklist of what kind of things you want to make sure that you include when you're doing immigration um I think the first one is make sure that you have redirects in place for any of the pages any of the URLs that you use you used to have on your old system because you want any anybody who's following a link to the old page to come to the right page on the new system this is especially important for SEO if your website realize heavily on on SEO and has some great content that bring in a lot of traffic if you don't put this in place you're pretty much losing the entire value of your of your website so you want to make sure that every single website every single page gets redirected properly um you wanted to talk we've talked about that a lot already validity with data make sure that you your data and both ends are valid so the the way what we mean here basically you have to demonstrate your customer that this this is the data he wants so he just comes back sometimes saying well actually we've talked but this is this is not the way I just I don't want those fields you have to be sure that the data you migrate and then the smaller projects if we demonstrated with a beer example you can just use your own use a testing there are 16 items you just browse through them and just compare it's nice and easy but there is a sandbox text module available a test module available which you can apply and use a Drupal one of the built-in systems for testing an automated testing to run through those tables you migrated and check whether your data match with the source that's one of the stuff referential integrity make sure that all the references are created correctly so whether your source and destinations if there are some references in both ends they reference at the if there are some references in the source they reference in the destination integrate with other modules there can be various stuff so you make migration runs by 90% at the end smoothly and nicely but sometimes obviously you have other custom things you've built on things like that you make sure that they integrate very well string encoding that's we talked about that a lot your last saving one of the last saving kind of things is you can apply whenever you do a field macking you apply a callback and you can do something with that string this is one of the saviors just in case and obviously when you've done with your migration you can literally kill everything with relates to it just disable the module uninstall it and probably if you don't need it you can just remove it from your repository you don't actually need it if module the migrate module is constructed and architectured in the way that it just whatever you've migrated if that's done it's finished it stays there so whatever you do with the module it doesn't do any harm to the data to the new data and actually have one more that I realized we haven't covered in any of our examples and it's something that belongs in here dates make sure that your dates are correct because different systems use different ways of saving dates sometimes with time zones sometimes without time zones and so it's also it often happens that you don't realize that everything else well everything that all the dates are like two hours later something like that you want to make sure that the dates match this is something that happens pretty often yeah this is we've we've done a small list of the helper modules which might help you in your migration there are various stops which are already set up like I think migrate x-tree is one of the cases it provides some more sophisticated things to work with entity migrations I think it has profile integration and whether whether various other stuff and migrate migrate x-tree is one of the things that is inspired to migrate the inspiration is taken in commerce migration so to migrate once some some source to a dupal commerce distribution they are ready modules to migrate from away from dupal 3 sorry type of 3 wordpress and juma and just simply take them I've demonstrated already dupal to dupal migration it was a migrate redirect sandbox this is the one I've just put on this week there is a destination class that I've created which will help you to make your migrate your redirect so you can take your source data and then create redirects on the fly because redirects are entities in dupal 7 so that's a convenient thing and then a blogger migration this is not kind of it doesn't stick into a migration system but it there is a module which will help you to migrate your old blogger blog post into a dupal it uses a bit different approach but anyway it migrates and there's a long list of other modules probably you may check the dupal.org forward slash project forward slash migrate there are lots of other modules which might resolve exactly your case if you can't do it you may apply those examples we just showed you yeah we've got a conclusion thing so I think that's well is anyone convinced that well when you need to migrate well you have to I think anybody objects okay good well so yeah as a conclusion when you need to migrate well you have to so it's one of those things that's kind of unavoidable but it can be fun to do as a developer and it is also something that does not bring value to the customer directly because it is not a feature it is something that it is a chore but if you present it to the customer correctly if you communicate it well enough you can do so in a way that they really understand the value that they get out of this and this is something that is very important and the migrate module is a great solution that really facilitates doing this in a great way thank you do you have any questions in terms of questions any good question will be rewarded I've got some chocolate so please do not so to answer the first question the the migrate module actually has well when you define the mapping you can either say I wanted you to find a mapping between fields or you can also say this is actually referencing the entity created by another migration so it will for example if you say well I want to migrate this and it's it's a reference to a user it will look up the reference in the mapping table for the user so you need to make sure that you import the user first and it will actually use this to make sure that the referential integrity is there so the same way you add a mapping and this is also something that you can do through the user interface the same way you add a mapping between fields you can add a mapping between entities it's called the source migration you apply the source which from which you come and the second question about estimates it's it's difficult because typically there's one end of the migration that you cannot control so you need to have a look at the data and I would it is the kind of estimates where you need to have a bigger buffer and it really helps to have a quick look at at the data and and see well look for encoding look for what is the structure of the data so I would never make an estimate without having seen the data at least but it is something that is difficult to do the red cross how much time was well they did the home I think it I was involved in this for about three weeks it's including planning collaboration and then the migration itself it was plus minus it really depends on the complexity of the data we've got three weeks with how many people was that or was that just you it was just me it was a team of three guys a scrum master and a customer side but the migration part the migration part was just between me and the customer we we called them project owner and this in this project so it was just only two but chatting yeah we're going to need to throw it apart so what's the situation with the three collection because I run into some problems about six months ago and I decided to write my own code what great and the second question is what's the best intermediate format to have I chose my square but where do you prefer my grade from if you have a choice well the first question was about field collections field collections actually it's it's uh there are entities so it's a list of entities that each of them had filled so if you look at the way it's stored in the back end I know there are some some yeah that's that blog post that blog post that one of the slides we'll put this one on the on the website jupo con that there is a blog post actually that blog post is about the entity collection migration and it uses the hook when you insert a new entity then it creates field collections and yeah so I think this is the way you do it basically what what happens if you migrate basically it's kind of it's an it's almost a messaging thing you create a message in the source by just creating your entity it doesn't exist and then you send it to Drupal and then Drupal just uses a node create and then node save thing and it receives those fields from the source which should then be passed to those entity uh entity sorry field collections sorry so pretty much you you create the the entities on the fly so you are not doing two-step two-step migration because well it's really the the entities attached directly to the the base entity that you are importing and the second question it depends on the kind of data that you have csv is very simple but it's only it can only do things that are later so you have like multiple valued fields you cannot do this with csv or you do some some crazy like multiple values inside the cell or something like that so it's I mean it could be done but there's really no need to to go through that so if well sometimes you can just migrate directly away from the the original source if you need to have an internal an intermediate format SQL would be fine I think it's use something that already has an existing integration and use it well see if it matches the data that that you have well it was a custom entity that uh that was created so we defined our own custom entity it the performance in this case was not really a concern because actually all the querying were done through a search so using the FHG solar so the querying of the database was not as much of an issue okay follow up I want to migrate several groupers 15 to 18 entities something like that after three minutes migrate goes down with only two entities permanent every single time if I migrate this is good question and is it do you use a web web web interface or drush no no but when you when you actually come to migration do you do it via this migrate you why yes where we are yeah try drush you may see some of the problems uh we've got a guy and then we can do yep okay um I got a question regarding continuous integration good one would it be a good idea to use migrate um to migrate the depth side to it's all about the production side to another new version of the production side inside so the same Google version just with new models that work to go over the screen for self is it is it possible to have such a framework is it usable at the time um so to get the content from the production side to the staging side I mean it's good work but generally you really want to not to get only the content you want to get a full copy of the database because you want to make sure that your your upgrades work you want to make sure that everything well you want to have an actual copy of the live site you don't want to just have the content so I think that in this case having an actual copy of the database would work better we've got two two questions okay yeah yeah you what now that I'm migrating from all the uh websites it's positive that there must be too much uh tags or bed tags inside the text that's a good question sorry from the body where they are in the ocean it's yeah it would be possible but it is in this case it is not a field that is being mapped to a field it is a field that you take you do some transformation you extract the data that you want and then you put that in a different package so if you want to do this then you would well of course you would need to create your your own migration and there's this this method called preprocessed rows this is on the on the source so you you create your own source that that extends whatever typical source you have and you preprocess each of the rows using regular expressions or whatever the the right tool would be to really convert to convert that data my last question do you have any experience we had an experience with the large sizes like a hundred thousand articles on relative to this question that uh it's just two articles in a minute so one notice that if you uh if you have the script you're running time for like two minutes the job will run only for two minutes and they can use that and each process in import so it increases for like 10 hours it can run for 10 hours and it increases because the problem I have is that we have a hundred thousand articles large as well with lots of content and a lot of database then as well as complex it takes a long time to run to get all articles and you can't map it in the local database so um do you have any experience what experience are you in especially if you have like hundred thousand articles so minimizing the minimizing the downtime yeah um well you can I think in this case it's worth taking a look at really what is making the system slow so the same way you do a performance analysis for all scalability and high traffic you can do a performance analysis of what what queries are making this work sometimes you well when you save a node typically there's a lot of things that happen uh you want to save fields you want to save images um and sometimes the only thing that you need to migrate is not the entire node just like the basic data and this is something that I mean it is a performance improvement that you only need to do if you are entirely sure of this that sometimes you can write directly to the database and get a huge performance improvement uh this is up for something that I mean comes with the responsibility of writing directly to the database so be careful with this and it can actually make yeah that's it guys thank you for coming we've run out of time yeah if you have more questions we are here all week uh we have and please please fill out an evaluation we would be very happy to hear what you have to say and I think the DrupalCon organizers also want to know thank you enjoy the rest of day