 Let's you know. First, a few things about me. I am a Drupal developer. I come from Europe, from Slovenia. It's a tiny little country near to Italy where the red dot is. I live on a seaside. I'm currently working for examiner.com. Before that, I was a Google Summer of Code student and mentor, both with Drupal. And I used to work with biggest media company in my country where we migrated websites from their custom CMS to Drupal. So FIDS. FIDS is a very powerful module. You can also call it a framework because it basically gives you tools that you need to get your data into Drupal website. It's very pluggable and flexible. So we can stick it to almost any use case that has to do with fetching data from a third party source and importing it into Drupal website. It can import the data or FIDS of various format types. You can import XML files. You can import things from HTML pages. You can import RSS FIDS. You can import from CSV. And when you get data into FIDS, you can modify it a bit and then save it as nodes or users or taxonomy terms. You have a contrib module that integrates FIDS with Drupal Commerce so you can import products, things like that. This is the project page. It's very easy to find. Just go to the Drupal org slash project slash FIDS. And if you follow the link for documentation, you can also find the list of all contrib modules that extend FIDS. So it's really a place where you start, download it, and if you find that your use case doesn't fit into plugins that come by default, go to documentation, check if there is a contrib module that does what you need. This session will be based on demos. Demos are intended to show you what is possible, what can you do, and to get a basic idea how it functions. First demo is a simple case where you build a website. And you have a customer that has a lot of employees and they want you to create a lot of user accounts. They send you a CSV file, and because there are hundreds of users in there, you don't want to import everything manually. Let's have a look at the CSV. It's very simple. We have a header row. First row is user. I mean, first column is user. Second column are emails. And third column are desired passwords that these users want to use them for their login. OK, first, let's go to module page. We already downloaded FIDS, and we search for it, and we enable it here. And we also need to enable admin UI for FIDS, because otherwise we're not be able to configure FIDS. When we enable these two modules, we get a new menu item under structure part of the menu, which is called FIDS Importers. You can see it up there. When we click on this link, we come to the page where we'll find a list of all FIDS Importers that we have on our website. So right now we had an empty page, so there were none. And later we'll see that the one that we will create will appear there. We clicked on the link to add the new importer, and we were redirected to this site, where we enter a name for our importer and a description that will describe what this importer is about. And now we are at the FIDS configuration page. So you can see here that on the left side, on the left sidebar, we have various steps that apply to FIDS workflow, kind of. We have few steps on every importing process that will fetch our data, load our parse data that was fetched, load data into nodes or taxonomy terms, and then save everything as an ordinary Drupal object. And each of these steps has its own configuration part here at this page. At basic settings, we see that we can still change name and description, and we can decide whether to use a standalone form for our FIDS importer or if you want to connect this importer to a node type. Usually, if we have one FID importer that will fetch data just from one source, it's OK to go with standalone form. If you want to use one FID importer for more sources, like we have an importer for RSS articles from various websites, we can attach this to node type and then create a node for every RSS feed that we want to import. Because we will only upload one CSV file, it's good to use standalone form here. And below, you also see that we have a dropdown for periodic import. It's possible to configure FIDS to go and check if there is any new content on your source on every cron or every one hour, two hours, and so on. Here, we are at the fetcher configuration part. Fetcher will know how to get your data from the internet or from the local server. Or as we can see here, we also have a file upload fetcher, which means that we want to upload our file, which is the case in our user's use case, because we have a CSV file that we need to upload. Each fetcher, basically each plugin has its own configuration. And with upload for a plugin, we can limit file types and define where files should be saved in our system. Then we have parsers. Parsers are responsible to get data and recognize what is data all about. So if we have a XML that is a text file, basically, parser will go and find fields that we're interested in. And by default, we have these four parsers in FIDS. And for our use case is CSV parser, which is very conveniently there already. So with CSV parser, we have just a little of configuration. We can define which is our delimiter character. So if we don't have commas, we can use semicolons or anything else. We can also have CSV files with headers or without them. So our CSV file had a header, because we saw that first row was kind of names of our columns. If we would have CSV file without header, we can just name our headers with indexes from 0 and incrementing on. Now, we are the last step. We have processors. And because we want to save our items from CSV to users, we will use user processor, which knows how to handle users. We can configure whether to update or replace existing users. That's handy if we get another CSV with passwords changed for customers' users, things like that. And if we configure FIDS to update, it will recognize which users have changed and only update their fields appropriately. With this specific case where we have users, we can also configure roles. So if we want to import these users as a specific role, if you want to import users as a specific role, we can configure that down below at the end of the form. We'll just import users as normal authenticated users. And here we've come to the mapping part. Mapping part is a configuration where basically tell feeds which field from source should go to which field on our user entity. So we can say that user will be translated to username of our user entities. And because we have a CSV file here, we have to enter these source header names manually with the different sources. We can also have predefined, and we have only dropped down as we will see later. So we map user column to username. We map mail column to user's email address. And we will also map pass column to user's password. And you can see that it's written that it's unencrypted password. So feeds will take that password, hash it, and save it into database appropriately so we can use it for logging. It's also important to define a unique field. Unique field will be used if we import our users and get a new file with new users and changed emails or passwords. Unique field will be that field that will tell feeds that we're having the same user and different email, for example. So if we mark username as unique field, everything that will change in a row where user is the same as username of some user in our database, well, have it updated. So now we'll go to the import page. Import page is at slash import. And we'll have a list of our importers here. Since we have only one, it's just this one. And we come to the nice page where we can still override some of the settings and we can, for example, we would get CSV with semicolons, we can change this here. And now we can upload this CSV file to this page. And when we click import, Drupal goes, sub parses this file and tries to create users. And we see that it's created 15 new users. Let's check the user list page. And we see that we have a lot of users that have never accessed our website and they're all created at the same second because they were created at the same request and only one user that's actually used right now. So now we'll try to log in. I'm in log out and we'll see if log in with one of our new users works. We'll take this Corina user with Corina 637 password. We'll see that it logs in. It's normal authenticated user. So it's no admin menu or anything but you can see from the menu up there that you have a possibility to log out. So yeah, that's first case. Any questions here maybe? Okay, with second demo, we'll try to import on RSS feed. We'll import Drupal Planet. As you can see, we have no nodes here so the site is empty. But we created two node types that we're going to use here. Feed node type will basically store the URL to RSS feed that we're importing because this time we will not use standalone form, we will use feed importer that's connected to a node type. And we have another node type which will store each article that comes from Drupal Planet. We have few fields there like title, description, author name, and original URL. Let's create importer again. It's the same procedure as before. This time, as I said, we connect our importer to a node type so we can add more feeds that we will import and we'll import it like every hour or yeah, every hour. So each hour, Drupal will go and check if there are any new articles. This time, we'll get our source from the internet so we use HTTP fetcher and we can also, if the URL that we provide is not actually RSS feed, we can also use an option for feeds to try to recognize RSS feed inside of a HTML page. We have RSS and ATOM parser already in feeds that need no special configuration. This time, we'll use node processor because we're importing nodes, right? And now you'll see that configuration page for node processor is a bit different than the one for users. First, we have to decide which content type will be used to import our items. We can again decide whether to update existing nodes or not. We can decide about text format that is used for text fields on our node and we can also decide that nodes should expire. That means that after a certain amount of time, they will just be deleted. We've come to a mapping page. Now you see that we, for source fields, we don't have text area anymore. We have dropdown because in RSS feeds you have predefined fields that come with them. So we map title to title. We map description to body field. We'll map publish date which also comes in RSS fields. We'll map original URL of the article with link field that we've put on this node type. We will also use so-called global unique identifier which is also defined in RSS feeds with internal global unique identifier that is used by feeds. And we will use this as unique field which makes most sense because title can change, URL can possibly change but unique identifier should stay the same all the time. So we have our mapping defined. We go to slash import again. And this time we'll see that we are not provided with a custom form. We're basically see the normal node edit form here. And we give our feed a nice name. I have to put planet. And as feed URL which is not a normal field API field, it was added there by feeds. In that text area, we will copy URL to RSS of Drupal planets articles. And when we save, we'll see this progress bar again. And this time Drupal will make a request, fetch this feed from Drupal servers and import 30 nodes. And as you can see, now we'll have few articles here that came from the internet to our website. Okay, my slides are online. And all these demos are on YouTube embedded with slides. So you can watch them again. Okay, let's go through concepts. You already know that we have four basic plug-in types in feeds. We have fetcher, we have parser, we have processor, and we have mapper. Fetcher knows how to download your data on your site and it doesn't always downloads it. You can also upload it. It's basically how you get your data to your website. You can use HTTP download, file upload, we used both of them. You can point to a certain location on your server. You can use an SQL fetcher which will query against SQL database and get data from there. And you even have plugins for things like IMAP or pop free to import emails. So as with every plugin type, you can also write your own. I had an example with X Client, which was a cable TV provider of the region where I live. And they had a VOD database of like thousands of movies and Sirius and all that crap. And they would not give you one XML file for each movie, right? They just spread everything through thousands of XML files. So we would need to load more XML files to get information about one video. And obviously no standard fetcher can do that, but we just implemented custom fetcher plugin and we could use contrib or default plugins for all the other steps. After fetcher, we have parser. Parser gets usually a string data that comes from your source and searches for fields that are interesting to us. By default, we can parse RSS, CSV, OPML. We have a lot of plugins. One that I found find most interesting is Xpath. With Xpath, you can basically write predicates to describe where is your data inside an HTML or XML document. And in case you don't know Xpath, predicates look like this. First one matches NEH1 element, and second one matches all the div elements that have ID set to article content. If you're interested in this, you have plenty of resources about a syntax. Very similar is QueryPath that is PHP Querying Library and you have also plugins on drupal.org that can parse YouTube feeds, Vimeo feeds, Excel files, things like that. When we've parsed the data and mapped the data, we have to save it and that's the job that processor does. We can save it as nodes, as terms, as users. As I already said, we have commerce products. It's probably you will not need to write any new plugins for this except if you create your own entity or something like that. And one of probably the most interesting for ordinary day work is Mapper. Mappers are plugins that provide links between parsed data and the drupal object and you use them mostly with fields, right? You want to usually, in 99% of the times, to map your data into fields that are in your nodes. So default fields that come with Drupal Core are already supported. Some of the fields that are in Contrib are also supported by default and for the others ones, you have Contrib modules like for entity reference, for example, you have few solutions that handle integration and if you create your own fields, you can write your own plugin and just tell feeds where to put parsed data inside your object. Okay, now we have another demo. This one is a bit more complicated because we don't have a standard source feed. We have an environmental organization in Slovenia that offers XML feeds of current weather situation in our biggest cities. And this XML file can be accessed to this link and it looks like this, that it's a normal XML file and then we have blocks which define every city. For example, this was city, Celia and then we have temperature and wind speed and humidity and pressure for this city and we have like 15 of these cities in this feed. It is, this is another city because we had another block and another one and so on. So this is a normal XML file but we don't have an importer that knows how to parse this. We created one node type where we have title as a city name, we have temperature, humidity, pressure and wind speed fields and we want to import that using feeds. We will create a new feed importer and we'll use standalone form again because we have only one XML source for this. We will set it to periodically import like every three hours. Fetch or that we'll use is again HTTP fetcher because we'll go to their servers and directly fetch this file and we've come to the parser and we have no parser that could parse this. We have sitemap parser that doesn't help us. CSV is not useful at all. So we have to use something else, right? And I decided to use expat parser, which I mentioned before because it offers you to make custom queries against XML files and I already downloaded it. It's on Drupal.org, it's on the list of plugins when you come to the feeds documentation page and when we enable this module, which is called feeds expat parser, we'll return to the feeds configuration page and we'll see that have some new options when it comes to parsers. Okay, we have the default ones and we have two new parsers. One is XML parser with expat and another one is HTML. We will use XML parser and you can see that we have no special configuration here but we'll try to go on and see what happens. We'll import into nodes because we've seen that we already created a node type and our node type is called weather observations. So let's configure this and our nodes will never expire because we'll keep updating with new weather conditions. And when we come to the mapping part, we still have a dropdown but we only have one option on the source side which calls expat expression and we can add as many as we want of those. So let's try to add them. We'll map title is our CD name, we'll map temperature, wind speed, humidity, air pressure, I believe. And each CD in this XML feed also has its own ID and that ID will be used for global unique identifier in case that, I don't know, case of the CD name changes or something like that we'll still have the same ID. And now we've configured mapping but we still didn't tell feeds how to parse this XML file. But now if we go back to the configuration page for parser, we see that we have all these fields that we mapped appeared here. So we can use these fields to enter expat predicates to tell parser where the data you need is located inside the document. We also see up there that we have a context field. Context is context inside XML document and when we match context, we will basically tell feeds which part inside XML defines one item that should be imported. So one item in our case is one CD and you can see that every CD, so below that, we have CD name, below CD name we have unique ID of that CD and then we have geographic location and all the weather information. And context in our case is an element that wraps entire block that defines CD. So in our case, it's that element that's called metadata. And you can see up on the top that if we click, I have a plugin in Chrome and if I clicked on this element, it would give me the most basic expat predicates that leads me to that element. And because we also have one parent element that wraps everything, we have two elements that wrap every item. So data is the first element that wraps entire XML file and metadata are elements that wrap every CD. Is that clear? So we use slash data slash metadata as our context definition. We'll copy that and we'll go to configuration page and just enter this in this tech field that expects context. And now when we map individual fields, we are not doing absolute expat predicates anymore. We're just doing relative to our context. So it's really easy because everything that is inside our context is on one level and we just need to basically get names of our elements that define CD name, temperature, humidity, wind speed, air pressure and things like that. Can you go to the microphone please because of the recordings? Or yeah, yeah. All the units here are metric because we're parsing feed from Europe. So, but it's the same, it's only numbers. The fields that are available here are from when you created the content type or the node type with your observation. The fields that appeared here are when from the dropdown list earlier that you mapped. We created a node type. And then when we came to that mapping page where we mapped fields from dropdowns, when we selected expat expression from dropdown, it basically created one text area here. So we just go and check what the names are and copy everything to these fields. And that's basically it. This is how we tell expat parser to find your data inside an XML document. So we have the main ID, which is like node ID for a CD, but we will not use it as node ID. And our importer is ready now. So we check once again if all the predicates are right. We also have option here to enable down there. There were some check boxes and you could individually check fields that you want to get debug information from them. And debug information will basically mean that you will get printed the data that expat parser got from your predicates. So if you created a wrong predicate, you will see that something weird is printing out. Let's go to slash import again. Typo. We have importer here. And we just copy URL of this feed. And when we save it, site makes a request, gets an XML, parses it and create nodes that store data. So we have 15 nodes because we have 15 cities in this feed. Let's Port Roj is place where I live and we can see that it was 21 degrees at that day when I was recording this. And let's see Ljubljana, which is capital city. It's 20 degrees Celsius. So we managed to import completely custom fields or without any coding in five minutes. Okay, we have plugins that can extend our functionality. But we also have helper modules that can help make feeds even more powerful. One of probably the most interesting is feed stamper. It is a module that allows you to alter your data before it gets saved. So when you, for example, fetch city name from that feed, which we just used, you could transform more characters to lowercase or uppercase or do search and replace init or combine two fields together into one field, things like that. And feed stamper, it's also pluggable. So if you can't find plugin for tamper that does the job that you need, you can just write a plugin. Another interesting one is feed scroller. It's a fetcher plugin, which will know how to follow pages of a paged content. So if you have a website with 10 articles on a page and you have pager below, you can configure feed scroller to find other pages and import content also from page one to 10, for example, in one step. If you want more plugins, you have documentation online, it's that URL. And as I already said, these slides are online. So it should be published by now. So you don't have to write down or take pictures. Another example, which I did when Google announced that nasty decision that they're shutting Google Reader down, I was just wondering if it would be possible to create replacement for that with feeds. And that meant that I need to export my subscriptions from Google Reader and import them to a Drupal site. And then I want to fetch data from these feeds regularly. And I created a proof of concept, which it's a bit more complicated, but I invited you to play with. It's a module that depends on features. Is anybody here that doesn't know what features are? Okay, so basically it has feature that gives you two feeds in porters, one for subscriptions, one for fetching your articles from your subscriptions. And it creates a view that is very simple and it's not as nearly as user friendly as Google Reader, but you can browse articles that came from your subscriptions. And yeah, you can download it and play with it just to see what's possible. Resources, project page is the main resource when it comes to feeds. You have a link to documentation there, short description of what is possible. In documentation you can find a list of plugins, which you'll probably need. And all the demos that I showed today are recorded, are on YouTube, so when you go to the slides, you just play them. And all node types and all importers that I created here in the demos are exported as features and are on the third URL. So if you miss something or you want to play with it, change something, see what happens. You can download this feature, enable it on Drupal 7 site, and you will get everything that I showed here. Yeah, and I'm done. So now it's time for questions and please go to Drupal's website and evaluate because it gives a nice feedback. Please, yeah, Mike. Hi there, my name is Lynette. I was just wondering if you had, if you could talk a little bit about the limitations you've experienced with feeds. And maybe when you've had to do anything more custom, have you ever had to work with the migrate module? Yeah, with my previous job, we used both. When you have dependency between importers, we usually went and migrate because you can have more migrate classes that depend on each other and it will automatically know that one class needs to be migrated before the other one. If you have more simpler things like this, it's much easier to do it with feeds because you usually don't need to code. With migrates, you can't do anything on clicky bases. So if you have a simple use case that can be covered solidly with feeds, it's much easier to do it with feeds. If you have something really complex with a complex business logic, it's probably better to go with migrate. And it's just the reasoning. When you come up with this case and you know limitations of both, yeah, but I would say more simple ones with feeds. And if you have something very complex with complex business logic, it's maybe easier to go with migrates. Hi, thanks for presenting. My question is, I have a local CSV file and I have products and I have a nested taxonomy terms for product. How would I import that? So how do you have nested terms in CSV? It's under one vocabulary. And how do you store information about taxonomy terms in CSV? Well, that was my question. Like, should I make a column and then add the taxonomy terms separated by... You can map for taxonomy terms can basically match a label. So even if your structure is not in only one level and you have unique labels, if you have... It will find it and reference it. So that could be one solution. And do you have taxonomy term this categorization imported before or you want imported when you import products? I have it imported before but it can alter in case there isn't one already. Okay. I think that you can create new nodes. You can probably play with tamper to see how if you could alter it there, things like that. Okay. Does it help you or? Yeah, I think I have to play around with tamper. Yeah, yeah. I mean you basically have to alter data before or create these entities before that. Okay, thank you. Thank you. Thank you for presenting. I was wondering if it's possible using feeds as it is to actually import an image that you have the URL for from the source of your... Yeah. Yeah. The mapper for image field will take a URL and create a file and save it. Will it? I think so. I just checked code this morning or it's just file field. I'm not sure but I think that... So image field does it. Yeah, there is a code that if it's new URL it will download file, save it as a file object and reference it to a field properly. Okay, thank you. Should work. Hi, thank you for this. This is great. I have two related questions. The first one is if you have an atom feed with custom defined fields in it, is there support for parsing that out of the box as an atom RSS feed or would you recommend going through the XML... I was using expat in this case. Okay, all right. So don't mess with me. You could probably, I don't know, extend our parser for CSV from feed score and add these fields but that would mean coding. So if you're fine with that, it's nice. But if you just want to configure it, I usually just went with expat. Okay. It seemed easier. My second question is you did the XML based import of parsing it with expat. Is there a parallel way if you were doing an HTML import instead, how would you parse your HTML input? Yeah, when we enabled expat, we got two new parsers. One was for XML and another one was for HTML. And they basically work the same way. They just treat the content that comes in a bit differently. I think that XML parser is a bit more strict if XML is not valid and HTML parser is a bit more, I'm okay with that. So yeah. And there's one extension which I forgot the name. It's purify, I think. If you enable this extension, feeds can go and use this extension to try to fix your XML and HTML files. And configuration that is used on XML and HTML is different because it's different kind of content. So that's the only difference. But the whole configuration process and how do you define XPath predicates, it's completely the same. Okay. Thank you. You're welcome. Hello. Thank you for presenting. I don't have Drupal yet, but when I do, it's gonna have a SQL server backend. And currently I take Excel spreadsheets from my boss and I throw them in my backend using a stored procedure. So my question is, can you use these Drupal feeds and run a stored procedure so it does things, whether it's inserting or updating? So to use procedures, you would probably need to have your custom processor plugin because processor deals with saving. And, but the easiest way to do it, it would be just to save it as Drupal nodes or something like that. It depends what you need. You also have this contrib module that is called data that is meant to save batch data like this because sometimes it's not very nice to save everything into nodes. It's so, I mean, we have mechanism that are better. So maybe that one, it's more convenient. It also has support for views so it can nicely list things. But I tried them data once and it didn't work very well. So I'm not guaranteeing that it will work 100% but it was some time ago. Thank you. You're welcome. You might have mentioned this. Does feeds have a means of doing things the other way that if you needed to create a feed, a custom feed that some other system needed? You would probably go with views and some plugin for views that can create XML or something like that. Or if you have more complex stuff, you have a services module that can be used to create service endpoints very easily. So if you only need to give data to some other website, it's probably the easiest way to go with views and use some plugin for views for data format that you want to expose. If you want to have business logic, then you need services. Thank you. I've had a problem with feeds import for a couple months but I've just kind of learned to live with it. Basically, once I set it up for a periodic import, it only imports, it's a local file import. It's a folder full of XML files. It imports one every time I tell a text USA every 10 minutes. Every 10 minutes, it'll bring in one even though there's 30 in there. Unlocked, the feeds get locked. Added the unlock button based on a patch on Drupal.org but I didn't know if perhaps you would run into that issue with feeds import at any time. No, do you have your unique identifier set correctly? So... It does work. Like I said, it just does it one per cron. No, I never run in such a problem. I just noticed when I was playing with this Google reader thing, I had like 50 subscriptions and every subscription would import. 50 nodes, each of those subscriptions lead to one XML file, all right? Yeah, yeah, so it's just one big file with many objects in it, not many files. Yeah, okay. But there is some limits, how much items will be imported at one batch. And then I noticed with the Google reader replacement because I had like 50 subscriptions and every subscription had like 50 nodes, for example, and that's kind of a lot. And it's needed like few cron runs to import everything because it's just, I assume that because of safety reasons, it doesn't go and import everything, prevent thing from time outing. Right, thank you. So I just ran few cron runs and when I had all content in, it was just importing changes, which was fine. So it's maybe it's this, same issue. Hi, I was wondering if you had any experience or suggestions with kind of the workflow back end of some of these feeds. I've run a website that pulls a feed from a calendaring system and so we have a few hundred events that pull in every week and we're having a big problem with just a sheer number of nodes that get imported. So I was wondering other than using views and kind of going through each one, if you had any other suggestions or thoughts on any other ways to improve the work. Sorry, I think I didn't get your question. Can you repeat please? If you had any suggestions for any other ways other than using like a view or something to kind of go through the feed with a feed with many different nodes that get imported every day and kind of go through and have someone approve or deny or whatever. If you have any feeds that have a lot of nodes in them. Not really probably with contrib I would say views that you can try to filter appropriately or something like that. Otherwise some custom code that, it just depends what you need. You need to go through every item that gets imported and approve it or not. Basically, yeah. Yeah, it's a calendar so the marketing aspect of the web has 10 of the 100 events that they want to highlight but so they don't have to type in everything all the events get imported first before they go through. Yeah, I mean it's manual work, right? So if you view doesn't work for you then you can create some custom form that loops through all new unchecked items that were not approved or disapproved or unapproved yet but it's custom solution, right? So yeah. Thank you. Hi, I noticed recently the lock and unlock feature that was added and I was wondering what causes it to lock. I don't know, I didn't follow that, sorry. Maybe somebody else knows if, I can check it out and see, yeah. Okay, thank you very much for attending.