 Hi, my name is Yoris. I'm Boris on Drupal.org and on Twitter. And I will be talking about search API. I work for a small company in Belgium called Dazzle. This is our wall. And the slides are already online. They're on that URL. You can also find it from the session page. That is the session page. And if you want to follow along on your own machine or you want to grab the slides for later, they're already up. So I will be talking about search API, fast API, and related other modules. I will start by explaining a little bit about search API, then search API solar, facets, a couple of other contract modules that are in the same space. And then I'll shortly discuss some custom code and how to write it. But first of all, what is this actually about? It's about building a search solution. So you have a couple of entities that you're looking for, a view that gathers those entities, and keywords, and facets, and sorts. Those are all things I will be shortly explaining a little bit more about. But that's the main thing that we are going to try to build in Drupal. So about search API. I'll start with a short introduction. In 2010 for Drupal 7, we wanted to make suggestions to Core Search to improve it, to make it a little bit better, and mainly to provide a generic and flexible way of searching to index not just nodes and users, but also custom entities and taxonomies and other kind of things that you wanted to index on different search engines, such as Solar or Elastic Search, not just MySQL. And also to provide different types of user interfaces than the user interface that Drupal Core ships with. And we have a pretty good tool of making user interfaces and lists. It's called Views. And in Drupal Core for Drupal 7, it was not using that. So Search API is using that. So building views of the entities that are indexed in the back end with advanced features, such as keyword search and facets and sorts. That's the main goal. Looks like the batteries on this are almost dying. So I'll just do it like this. About three years ago, a little bit more by now, the Search API module for Drupal 7 and the Apache Solar module for Drupal 7 figured that it would be better to create one module for Drupal 8. So Search API for Drupal 7 and for Drupal 8 got together with Apache Solar to create one search solution. So for Apache Solar, that means that there will be no Drupal 8 version. There will only be Search API in Drupal 8. So we had a crowdfunding campaign to support the main maintainers. We had a couple of Google Summer of Code students to help out. And we had a lot of Sprints, a dedicated Sprints in Belgium, one in Switzerland. We had Sprints at DrupalCon. This is DrupalCon Amsterdam, Sprints at DrupalCon Barcelona, Sprints here yesterday, and today, and tomorrow, and on Friday. So basically every day of the week. So some influences, because Search API is quite mature module, especially for Drupal 7. And we had a lot of help. One of those things that influenced this heavily is Drupal.org. The Drupal.org issue queues are powered by Search API. It's a Drupal 7. And it's using the MySQL backend. So that means that we had the opportunity to work with a data set and a scale that doesn't fit on my computer. It's a big data set. It means that the MySQL backend got more stable, got more reliable, got extra tests, and got a lot of optimizations to be faster, because we all like Drupal.org to be fast. This also means that if you're building a custom website for your clients, and you want to use a search engine, and you don't need special features, and you just want a basic search, then MySQL will be good enough as a backend. You don't need solar for a small website. Or well, Drupal.org is not a small website anymore. But it's not the size of your data set that should determine which backend you're going to be using. And I can't talk about this and just think that I did it all on my own, because I didn't. These are a couple of people that helped heavily in the development of Search API, and a lot of them are here. So, thank them if you have the time, because they did an awesome job. This also didn't come overnight. This chart starts in July 2013, six, sorry, three years ago. And as you can see, there were a couple of sponsor sprints, and Drupal Dev Days in Zegat was also quite successful in helping us create code and get extra code in, as well as DrupalCon Amsterdam and Barcelona. And probably when I update this chart, there will be also a big visible blob for DrupalCon Dublin. And we can't miss out on those two last ones there, because Acrea has the module acceleration program, and they were kind enough to sponsor both me and Thomas Seidel to work on Search API and facets and solar to actually get it into a more usable state for all of us. So, thanks for that. So, I'll shortly discuss the architecture and the way that Search API is built. So, you have three concepts that you should know, or need to know, or could know depending on how deep you want to go. There's an index, which is basically, what are we going to be searching for? It is nodes or users or taxonomy, or you can click whatever you want, and filter by bundle or something else. There's a search server. A server is where the data is stored, such as MySQL or Solar or Elasticsearch. And that's basically one plugin that delivers that. And you have a display, and that's how the data gets displayed onto your website. So, we ship by default with support for views, but there's also another module called Search API Pages that does just an entity load and doesn't have the added complexity that views might have. So, for simpler results, that might be sufficient for more complex solutions, views is a good way to do it. There's three modules inside of Search API. Well, there's Search API, which provides the framework. There's Search API DB, which provides a connector for MySQL, as well as SQLite and Postgres, because well, those are all database backends. So, that's one backend for those databases. And there's a Search API DB default, which sounds scarier than it is, because it's actually providing you with a server, an index, and a view already configured. So, you select those modules and you have that configuration on your website. However, this will also already index a couple of fields into the index. That means that we need to know what fields they are. So, this depends on standard installation profile or rather the article and the basic page content types and the fields defined on them. So, it only works if you have selected standard while installing and haven't deleted any of the other shipped fields or content types. So, I'll give you a short demo. I started by installing a new website on my local machine and I generated some content. And then, I will enable database search, Search API and Search API default. I did. It will install all the modules and will also import the configuration into your system, which takes a while, but not very long. Then you have to import all the configuration because if you go to search slash content, which is the default view that we ship, you won't see any data. So, back to the admin and you go to Search API. You select your index that's being shipped. Default index, default content index. And you index the nodes into the index. Index, index, I know. Go back to the overview and this is the result. This was 48 second video and that leaves you with a Search API installed on your site provided you're using the standard installation profile. So, this was quite easy. No actual configuration needed other than enabling the modules and downloading them, of course. But you probably also have to index other fields or add other fields into the index so that they can be searched on and they can be filtered on. So, I'll show you how that works. Note that this UI probably is not the most intuitive. It looks kind of scary, but we're working on improving this. There's an issue open about this, but you can actually go into, for example, tags, which is a field on the node, and go into the term ID or even deeper and you go into related fields. For example, the user of a paragraph, of a paragraph, of a node, of a related entity, of a taxonomy, you can go on anywhere you want, but you can add those into the index and have them be something to be filtered on. And you can also change the name if you want of those fields. Note that every time you do this, you have to re-index all the content because the schema changed, so we have to refill the database or the solar backend with new data or the same data again. But because additional fields are required, the action has to happen again. For a site with 50 nodes like this one, that won't take very long. For a site with 100,000 nodes, that might take a little bit longer, but it's batched, so it shouldn't fail in theory. So there's a couple of things that we still need to fix. One of them is hierarchical entities, but that's actually in a very usable state right now. We also have to improve the administration UI, the UI I just showed you, and especially the fields UI with all the pluses and add buttons because that needs some love from someone who has some insight in what is good looking. And we also need to provide config overwrites for search entities, which is quite a complex thing that we haven't figured out yet. That means that you can, through settings.php, configure a different backend on your local machine compared to on your live or your QA environment so that you can, for example, work with MySQL on your local machine and solar on your production environment so you don't all have to install solar. And also documentation and tests because documentation is good and tests are also good. However, that doesn't mean that we don't have any tests. You can see for Drupal 7, there were 24 tests. Right now, we have 229 tests, 292 tests because one extra went in yesterday, but that means that we already are more confident in the state of the code and that we already have a bunch of tests. So, search API solar. I'll also give a short introduction and then I'll give a short demo of how it works. So, Solarium is a PHP library that's being written by the Solarium PHP group on GitHub and they have provided a library, a package to work with search API or to work with solar queries. So, we don't actually have to write native solar queries. We have a PHP package available that allows us to write native solar queries from a PHP interface which is easier for us to do. It's also very well tested. It's very well used. It has a very wide install base. So, that means we can actually ship out a lot of responsibility to them, which is less work for us to do and that's good. So, basically we have a plugin, a backend plugin for search API and a couple of query alters that transform information, the way it looks from a search API query into a Solarium query which then transforms it into a solar query. But, Solarium actually powers a lot of that. We also have a bunch of tests and those tests don't run on the Drupal.org infrastructure. They run on Travis CI because we also have to use an actual solar instance and the Drupal.org test spots don't provide that but we have Travis to install the solar and to actually run real tests with a solar instance so we can be confident that our code actually works. So, I'll quickly share with you how that works. So, first of all, because this requires Solarium, which is a PHP library, you can't just install the module. You need to have Solarium in your vendor directory to actually be able to use it. So that means you have to configure Drupal to use either the Drupal Composer packages or the new Drupal.org Composer facade. And that way, you can quite simply from the terminal to Composer require Drupal slash search API solar 8.1.x and it will install the Solarium because that's required by search API solar. It will also install search API solar because that's the module you wanted and because it also depends on search API, it will also download that if you haven't installed it yet through Composer. So, as soon as you've done that, this thing kind of beeps, I'm sorry for that, you have to enable the module, start your solar server and then you can start configuring it. On the left side of the screen, there's a terminal output. This is a tail of the solar log. So you will see stuff appearing there. You don't have to read it, but it's just so we all can confirm that actually things are happening on the solar server. So, we start by adding a solar server in search API configuring the part. In my case, it's slash solar slash the eight on my local host, saving the server and something has actually happened. So it's actually writing to solar. Then changing the default content index that we already created to use solar instead of the database server, re-indexing all the content because nothing was in there previously. It's a new thing. So there was actually things appearing in the logs. And you see one last line and that was the actual query to solar. This was another 40 seconds. Of course, you have to configure and download the module, but it shouldn't take very much of your time once you've actually figured out how to install the modules to configure solar as your backend for your search API search. So facets, I'll also give you a short overview of the architecture. I'll give you a demo and a couple of issues that we're working on. So there's two entities in facets. You have facet and facet source. A facet source is actually the same thing, more or less as a display from search API. This is not called search API display because we also support core as a search provider to create facets with. And if in some kind of future, some other module also wants to have something, then we should be search API agnostic. A facet source is the display of a view that you've created, for example. And on that display, you're going to be creating the facets. So that means it's a block or a page or a REST display, one of those three. And that's the facet source, and then you'll see a list of facet sources, and you add a facet to the facet source on the correct field, and that's the actual facet. There's also different points where you can change the behavior of facets. One of them, the most important one, probably, is a processor because that's the way you change behavior. There's four points where you can actually do it, changing the behavior. It's either before a query has run, after the query has executed, on building a facet into a widget, or to sort it. Sort is one that you will probably all have to configure every time you create a facet because you either want to have it alphabetically or in reverse order, or by the amount of results, or any other kind of thing that you can think of. We ship with four defaults, kinds of sorting. And those are the actual behavioral changes that you can add. There's a widget, which is the way that a facet is being displayed onto your page. We ship with four, three defaults for now. One is links. Another one is checkboxes, which is links and some JavaScript turned into checkboxes. And a dropdown, which is links and some JavaScript to turn it into a dropdown. And there's one more that's being worked on, which is a range widget that actually allows you to do minimum and a maximum and drag between them, or just one slider, depending on what your use case is. And there's also a URL processor, which is another way that you can change the behavior. We ship with one default, which uses the query string arguments to change how facets work. Well, it actually appends to the query string on enabling a facet. And there's one extra that's in Contrip, which is pretty parts. We know that a lot of people actually use that, pretty parts, but we won't be shipping with it by default in facets, because it needs a different kind of speed of development that might be different. So there's three modules inside of facets, one being facets, the other on REST facets to provide a way to have a REST display on a view and to add facets into your search results. So you can do a decoupled app that can search that can also show facets as well. And there's also core search. That way we can also use Drupal course search but add facets on top of it, if that's something that someone would need. So I'll show you how that works by enabling the module first. And then these are the facet sources. In this case, there's only one, there's only one view. This is the same site that I used. And this is the default view that we ship with. So let's add a facet. You select the facet source and then the field on that source that you want to search on. I think I went for content type. Let's see, content type, okay. So we'll auto fill the name and the machine name. So you don't have to worry about that. It'll go into more configuration. These are settings and sorting. But by default, it should ship with a relatively sane configuration. It also provides you with a block so you can search for a block. It's called content type. Same name as the facet that you've created. You place it into a region on your site. You refresh the page and the facet appears. This is the data that's coming out of solar. So this is actually using the solar backend. And that was how you create a facet. So this isn't done yet. We have to implement or rather finish pretty parts. We have to also do hierarchy, which is a way to have a parent, child taxonomy term and to display that as a hierarchy in facets as well. However, this is almost done, I think, Jimmy. It's done. That's being worked on this week. As well as current search. That's also one that's being worked on this week. One almost on the bottom. We also have to add in support for dates because that's not supported yet at this time, as well as the slider that I mentioned earlier. We also have to add extra documentation and tests because we all like documentation and tests. So a couple of other things. This morning, we tagged battle two of search API. So this actually will provide an upgrade part when we enter beta. Search API page is lagging a little bit behind but it's on alpha 11 and it should theoretically still work. I haven't tested it since we tagged the new beta so I can't say it definitively. We have search API solar that's on alpha six right now and we will probably tagging another alpha version of that this week. There's first alpha available for search API sorts which still works as far as I know and we will either be fixing it if it doesn't work anymore or also move it into beta this week or at least that is the plan. There's a development version out there for elastic search as well as search API autocomplete. So you have autocomplete on your searches and actually Daniel Wiener is working on that one for Drupal 8 so we figured that's in pretty good hands. There's an alpha four out there for search API attachments which provides ticker integration so you can actually attach or well index PDF or Word documents into your solar server and search in them as well as the notes that are provided and that worked the last time I tested it. Again, we tagged this morning so I haven't tested it since and there's a development version out there for search API exclude entity which is the same thing that exclude node in Drupal 7 and there was also exclude taxonomy I think and this one just combines it. There's an alpha four for facets and a sandbox module for pity parts. We will be tagging an alpha five for facets this week I wanted to have it done by now but the tests are still failing and I need to figure out or we need to figure out a way to fix them but by the end of the week I'm pretty confident that we can actually tag alpha five and that will work with beta two of search API so we can actually we're going to try to provide a list of modules that work together in most of the release notes of search API there's a link to what version actually 100% sure works with it but right now the development version of facets works with battle two on my local machine and a couple of other people's local machines as well. It's just a test spot that disagrees and hopefully not for long. I'll show you how to create custom code for search API. This is using the plugin system so it's actually working the same way for facets and for a lot of other modules out there as well. So I create a very small custom module and I'm going to be creating a search API processor. It's going to be implementing that interface. I won't have any settings for it and no configuration form and it's a very simple and quite naive implementation of a plugin that will not index any node that has pipe ignore in the title of the node. I'm not sure if that's a use case that any of you have ever had to do. I'm pretty sure you haven't but you can figure out from the example code how to write something more useful hopefully. So this is it. This might be a little bit more readable. So you define a class and have an at search API processor annotation on it. You have given an ID, a label and a description so you can find it in the UI later on and you give it a stage. Stage is one of the different kind of hook places where you can change the behavior. In this case it's going to be before we're indexing entities into the index. So we have to make sure that this only works for data sources that have a node which means that we implement support index and check if the data source has node. This is just to make sure that this can be portable into other things. If you're absolutely sure that your code will work for your use case, you can just return true in this method and it will also be enough. And then this is the actual work that's being done here which is getting the original object, the node out of the search API item and then just using string POS to check if pipe ignore is in the title and if it is, then set it from the area of items. I'm pretty sure no one will actually have to use the exact example that I've given here. I use it in production. But this is a way to have like if you have some kind of weird way that you have a taxonomy term that's linked from a node and you have to check on if it's four, then you can't have it in the search index. This is a way to do it because that would be a very horrible thing to build into the UI. But in code, it's actually quite short class. It's only about 60 lines of code, I think. So the question most of you probably came here for, can I use this right now? Well, according to Swentil, yeah, but he did say this, we can start using it in October of last year. That was a little bit optimistic. However, with search API in battle two, you can definitely start using that today. There is an upgrade part if something changes in the storage, we won't break any of your settings. You might have to reindex all your data when you upgrade, but that's everything we're going to, well, the most that we will break. We won't break any of your configuration, any of your settings. And we will provide an upgrade part if we do. So you would have to run updates, but nothing will be absolutely broken so that you have to start up. So for search API, absolutely you can start today. For facets, if you use development version, you could, but we're still in alpha phase, so we won't provide an upgrade part. So if you don't mind reconfiguring the facets when you upgrade, then you should be fine and you can start using it today. It's actually usable. And there's a couple of sites already doing so for search API solar. There's also people already using it. And we feel relatively confident in saying that you can start using that as well today. There's also an upgrade part provided for that if we ever have to break any of the configuration for it. Yeah. I haven't tested FASTA for elastic search, so I can't say if it's in a usable state or not. I'm sorry. Yeah, for FASTA, it doesn't matter. The backend actually provides support for music facets. So if the elastic search backend supports FASTA, then it's fine. Okay. Thank you. Thank you. Yesterday, I checked the usage on the Drupal.org website for all these projects. There's about 3,700 people using search API in production or sites using search API in production, about 1,600 for solar and about 1,400 for facets. So if you choose to go down that path, you're not the only one. So does anyone have any other questions? Yeah. How's the multilingual support? Is that for solar or for? So there's an additional module, search API solar multilingual, and that's actually being maintained by Marcus, who's also here, but I don't see him right now. He might be in the sprint room. But there's actually, he's using that in production. I haven't tested it, but I assume since he's using it in production, that he's quite happy with the way that it looks right now. For search API, I don't think we added any specific handling for multilingual, but as long as it gets indexed into the database, then you can get it back. So I don't think there's going to be any problems with it specifically if you, yeah, thank you. Maybe just to add on, so the search API multilingual offers an option to generate schema based on the languages you have enabled in your site, so that the schema you download is compatible for all those languages with all the specific processors that are very specific to the language. So I'll repeat it for the recording. The search API multilingual, search API solar multilingual module provides a way to have a schema generated with specific handling for the languages that are enabled on your site. Okay, any other questions? Okay, so what about calculated fields? That was your question. Is it a field that's only for search API or is it like a hook info field kind of thing? Hook field info. So there's a way to provide a processor to actually provide arbitrary fields to a search API backend. I haven't tested it with hook field info fields, but that's a way you could provide it. For example, we have an at URL processor which adds a URL to the output of a search API result and that's a computed field. So you could have a look at the way that that works and see if that solves your issue. Yeah, first of all, thank you for those awesome modules. Thank you all guys. We are one of those neatly guys that used it like last October in production so it still works, but I'm graced with the task to get it into beta, so. Okay. Yay. No, just a simple, easy one. Do you plan any additional thing that like you've showed just a little check box on my notes, for example, the four page will always be indexed as well so that I can exclude a certain notes from the index. Exclude one note from the index, that's your question. But on a note basis for the content manager. On a note basis for the content manager. No, we don't provide that. However, the way that I showed you that processor, instead of SDR pause on the note title, check the field value and that should be the same thing. So you can download the slides, that's just a code that you can copy out. And so with that small modification that should work for your use case as well. Well, thanks of course for me as well. I have a question about geolocation. When I want to build, for example, a store locator in a scale from say noob to savvy backend developer, how difficult would it be to add geolocation search functionality to search API? At this moment, you would have to be a very heavy backend coder. The answer I was afraid of. However, there is a proof of concept. There's a proof of concept out there that actually worked at Drupal Dev Days in Milan two months ago, three months ago, something like that. But that was not very, it was just a proof of concept. However, the maintainer for the geo module is here and he might be working on it or he might not be, but there's a proof of concept out there. There's a way we know how to do it. We just need to have time or people funding us to build it. That guy's a freeloader? Well, I can find any resources. Okay, so actually someone here is already using it. So it's possible, apparently. Okay, so additional support for Acrea and Pantheon? Okay. There's the Acrea connector module out there and I'm looking at Jacob. Is that in a usable state? It's getting there. It's getting there. So for Acrea, it's getting there. For other hosters, I don't know if they need a custom solution or a game. Probably in October, it should be working, but we all know how software go, so maybe end of October. Yeah. Are there plans to get search API to replace core search? Sure, in the ideal world, we might try that. Let's first try to get search API in a stable version and then we can try talking to people and see if there's actually a need for search API as a core search functionality or to replace it because it is more complex than what core search currently supports or currently offers. And then let's talk to the core maintainers and let's talk to the maintainers of search API to see if they actually want to support a core module because it's a somewhat bigger responsibility than just a contract module. However, this is not a simple contract module, but so if we have enough time and it's in a usable state and people are all for it and we can convince core maintainers and the maintainers of search API, then maybe, but that won't be for 8.2 because no additional features go in there anyway, not for 8.3 either. So probably not for eight. Maybe 8.9 if we ever get there, which is in three years. So in a future world, maybe, but no, there's no talks about it right now. All right, back to multilingual discussion. So oftentimes I find that languages that feature, websites that feature languages other than English provide quite poor results. For example, if user used a word with different endings, for example. So my question is does a search API provide any kind of support for work stemming to improve this situation? So stemming is something that's only possible in solar.