 All right, we'll get started Well, thank you all for being here. I'm super excited to be here in front of you. It's great to see everyone and Doing this session in person. It's been a while. I think last time last time I was up here was a few years back So so it's it's great to kind of be here up again and meet people in person So my goal for this session right for you all is really kind of hopefully after the session You know, you will be more knowledgeable about kind of what is a good and a bad search experience You know, what do you you know? How do you kind of see the power of open-source search and how do you utilize it on? You know some of the search experiences on your own Drupal websites and then finally And hopefully some of the community members here. I can enlist some support to help me make search for Drupal org a little bit better for the community and and you know help me kind of launch the Drupal search org I Will keep you know a few minutes or probably ten minutes at the end of presentation for question and answers So if you know anyone has any questions probably take it up there and with that, why don't we why don't we get started? So just from introduction side my name is Samir Magan. I You know, I work for a company called search tax. I focus on product and growth At that company engineer from the background, but you know over the period of time I've kind of moved a little bit on the on the product side of the equation and and kind of that's my role there There's my Twitter ID and and my Drupal ID for those of you are interested Please feel free to follow me if that's of interest It's a little bit about the company search tax so search tax is You know software company we try to make it easy for You know developers as well as marketers to deliver in a really interesting dynamic and powerful search experiences We offer search infrastructure Solar as a service for those of you who haven't you know who don't know about solar Hopefully we'll introduce that today. We offer that in a pass and a SAS environment And also we have a SAS software for search analytics management and optimization Distributor team of developers across United States Europe and India about 500 customers across the globe and growing rapidly and You know search tax is one of the largest Apache solar farms out there in the world and We're coming back to you know the Drupal side of things We have plenty of customers that actually run Drupal Acquia Pantheon search on top of search stacks with various different industries that actually You know power their search using search stacks So today's session really kind of outlining You know we'll talk about a little bit about importance of search We'll kind of talk about why search is important. What's the cost of a bad search experience? Then we'll kind of take you know that and then go back to Drupal.org website itself And then the community website and then you know kind of see what search looks like on that website some of the issues on the current website and then potentially improvement opportunities on Drupal.org Then we'll dive into best practices around the search experience. What does a good search experience look like? And then we'll segue you know from there into a brief introduction to Apache solar for those of you who may not Have experience or be aware of that and then you know after that we'll kind of go through a developer's journey So over the you know over the last couple of weeks. I've tried to put together a You know a a search experience that kind of takes the content of some of the content of Drupal.org And then tries to look at you know kind of the pitfalls and drawbacks of what Drupal.org search is and then see How you can actually use Apache solar to make that search experience much better And then you know kind of from there we'll kind of go through what the next steps are in that journey For me and then you know options for getting involved as a community So just before we kind of dive in a quick You know shoe of hands. How many of you are aware of Apache solar? All right, so I'm talking to an audience that's you know fairly familiar with it and how many of you have actually you know Kind of coded or you know written queries and configurations and stuff in Apache solar Okay, so maybe about you know 15% of the people here. All right great. So Yeah, I think that's that's helpful. So why don't we kind of go and segue into you know importance of search So why should we care right? I mean we we know search and when we talk about search right at least when I'm talking about search I'm talking about search on Companies websites right or this is Drupal.org website. This is you know, you're probably you know employers website That's public facing people go to that website and they may or may not have a search bar in that particular website And that's kind of the search I'm talking about and not the you know Kind of the google google search or the web search or the crawler, you know the world worldwide web crawler In in this scenario So these are these are some of the research. Let's look at the research and facts You know today, you know search abandonment costs in excess of 300 billion dollars annually just in united states alone. So what does search abandonment means, you know, someone's coming to the website Executing a search not finding what they're looking for and then going away from the website because they didn't find what they came for On you know on that particular website So, you know, this is a research that was done by Harris poll surveyed across, you know 200 website managers and about 10 to 15 000 Consumers who actually utilize websites and then that's kind of the output of the survey Other research and and statistics around, you know, why search becomes important is You know today 84 percent of the companies fail to optimize and measure search on their websites So that's a pretty large percentage. So pretty large percent of the users who spend a lot of money You know utilizing Drupal and other content management systems to build websites But then 84 4 of them Just don't have the means or don't have the means to measure How search is performing on their website and how do they actually optimize or make the search experience better Statistics also say that about, you know, if a user actually performs a search They are almost approximately two times Highly likely to convert than if they did not perform a search on the website So that's a pretty again significant for folks who have websites where conversions is a pretty important metric Whether that's a call to action from a marketing perspective or you know in an e-commerce settings It is a purchase of a product So the conversions matter a lot and a hundred percent or almost a hundred percent improvement on conversions is, you know, most of the time a marketer's dream, right? Um And finally, um, for you know, I'm sure every one of you of you have used, you know, google 75 percent of the search queries that are performed on google are actually influenced by Autosuggest suggestions that come from google. So when you start typing Something in google search box. It starts providing you suggestions on, you know, what to search for So it says 75 percent of the users actually pick the suggestions as opposed to typing the entire search words or search terms that they're trying to look for So these are some of the examples of, you know, kind of Lack of understanding of, you know, how users are using search on the websites How can you improve conversions and then some capabilities like autosuggest and stuff that can actually impact and help improve You know and end user search experience But then when you look at from a consumer or the user perspective, you know, google has done a fantastic job in Um, you know providing kind of the results that people are looking for from a keyword perspective And then they've also upped the ante, right from a from a consumer perspective on what to expect out of a good search experience But when we go when we take that to companies websites That experience is not at par to kind of where google is today, right? And that experience today sucks and i'm i'm probably going to show you A worst example here in the drupal community because you know, it's our own website From a drupal.org community website, but i think it's helpful, you know to kind of see What the search experience on a website that is full of content, you know, whether it's modules documentation forums All sorts of content And and search becomes a pretty powerful tool to kind of find what you're looking for Um, you know, there's so many kind of drawbacks and and areas of improvement In what users are so expect, you know expect out of google search and what you know today the websites kind of have to offer So why don't we go into a live example? To kind of look at you know drupal.org Just from a disclaimer perspective. My intention is not to You know not to highlight kind of the deficiencies of kind of what the community has built, but I think my pure You know pure motivation is to actually use that as a learning exercise to see You know, what are the areas of improvement in drupal.org from a search perspective and What can be done in a fairly simple fashion with the tools that we're already familiar with In actually making that experience better And then and then, you know, my hope personally is finding enough You know enough support and work That can be done to actually have a fantastic search experience on on drupal.org Sorry, I gotta Make sure my screen is projected like this will I gotta mirror it Otherwise my neck will be All right, great Okay, so You know, we all know um drupal.org So let's uh, you know, let's do some example searches on on drupal.org You are all here wonderful event after a you know, a couple of years of Having remote events. Let's go to the website and actually search for drupal con Does anyone in the audience kind of see an obvious problem here? No, okay, I'll What about in the results does anyone see any kind of obvious Things missing on the results All right, I'll I'll I'll give it up. You know, there is another drupal con that's happening in europe And if you go to the website and actually search for drupal con That that web that page and that event is not even in the search results The site if you go back to the homepage, you know front and center promotes kind of both the events Right the one event that's happening in september and the one that we are here today on portland And if you go to the website search box, you know, such a major event where you're kind of Bringing the community together and if someone were to search for drupal con on the website That event is not available on the search results So that's one example of how you know kind of where In an example where one of the content pieces that is you know from a community perspective an important event Is you know not found in the kind of top results of of drupal.org Let's go and do a couple of more searches You know, let's uh, let's search for you know, we all work with customers. We try to Do personalization. So let's say if someone comes and start searching for personalization The first four results have nothing to do with personalization There you go that that one, you know, kind of beyond as kind of seem something that's that's kind of more relevant, right? But that's another example of you know, what you look for Let's you know, let's search for something that's as basic and as common as You know drupal and I'm going to misspell it Um, I'm going to use the you know use the character that is just On the right of character you and I'm just going to say You know, I'm just going to search drupal And see what the site says The site doesn't even recognize that we're looking for drupal What it says is do you mean triple? Let's look at some more examples. Let's look at You know the word site search So if I'm a user I want to improve my site search I go to the community and I search for site search in the hopes that maybe there are some modules Maybe there's some documentation. Maybe there's some forums. I can look at I'm disappointed because I don't you know, I I don't get that right off the bat or I have to work really hard to kind of go and do some filtering Do some pagination And other means to actually get to what I'm looking for On top of that what it also recommends me is a size search And you know just from a basic keyword analysis of of the website today We actually know that site search is more popular in this repository than size search But drupal.org is recommending us size search as opposed to site search Let's look at another example Where I search, you know, I'm searching for website search, which you know in most sense Is very synonymous to site search. So what I would expect typically is if I'm searching for site search Or website search. I'm actually you know In in in most cases looking for almost similar Almost similar results, right So, you know, again, this is another area which you know from a search terminology perspective, you know You call synonyms is an area of improvement on kind of the drupal.org So these are some some examples where we kind of see if if you look at, you know, kind of drupal.org's website Oops All right There's some some issues, right? Again, I think let's see if we search for events, you know There is an events that is up top and center You know, when I'm looking for events, I'd want to look at all the events that are going on Um, whether it's a meetup whether it's a drupal con or any other events If I search for an event events on the website Again, I'm not seeing the two biggest events that drupal community hosts I don't see um, you know, I don't see Drupal con uh, Europe. I don't see drupal con america. I don't see any of those things So, you know, that's you know, that's kind of uh areas, right? I think where where if you go to drupal.org The the search there's con there's vast amount of contents, right if you look at on the right hand side There's forums. There's documentation There's almost, you know in total of about Probably 7200 100 150 000 items in drupal.org But when you start doing some very very simple basic searches, you know, it kind of fails to fails to perform And and you know, there's very kind of little experience around that So with that, let's go back to our, you know, kind of Aspects of how do we, you know, what do we do about it? So what do we do about it and what are some best practices in search experiences, right? So today is right in a modern search experience if you if you look at some good examples of You know modern search experience on a website. What are some of those best practices in in the search experience? You know, what you see more and more these days, you know There's really kind of three key ingredients to kind of making search Really good for a given website, you know one involves You know with kind of making sure that you have an understanding around What's being searched for how users are interacting with it? What are they? What are they getting out of it and then where are the areas of improvement that can then be taken by You know a developer and then applied to a website that can actually make the end user experience much, you know Much smoother and much better And then there's the second aspect is our key ingredient is around the capability set, right? If we go if we look at the drupal.org Common capability sets like, you know, like google auto suggest The related searches if you go to google on the bottom, you'll see related searches Spell checking all these kind of things that are You know outside of kind of your basic tech search when you do a search you have a result set These are some of the capabilities that have become pretty much kind of the norm in some of the good search experiences And then finally, you know, which is a very ignored aspect of You know search experience on websites is really kind of the relevance in the ranking Which is primarily the examples that I was showing On drupal.org is when you're searching for something and you're getting a set of results How are those results ordered in a way that provides the maximum You know value to the user and get the results to the top of the Top of the stack to really make sure that what's presented to the user is what they're looking for And help them get to where they want to go as quickly as possible Right. So that's really kind of the area that that aspects, you know, kind of Revolve around the relevance of the of the search results, right now I don't know and you know, I don't have the answer to the question whether drupal.org Website today deploys any kind of analytics that Um, you know allows, you know, the the webmaster or someone who's managing the website To look at how people are using search. How many searches are there? What kind of conversions and click click through rate, you know, is being delivered from search What kind of latency is there on the search side? And then if people are coming and searching for something and they are ending up in an empty page What percentage of those searches are actually ending up in a bad experience where they're getting no results at all And you know, what are the options or at least get gain the insights to see what improvements or what are the Lowest hanging fruit improvements that can be made on the site to make that better, right? So that's kind of the analytics aspect of it But there are tools out there in the market now That provide you with that kind of a detail by simply integrating google analytics itself has You know some detail that allow you to get, you know, kind of basic search metrics Um, you know, you have other tools in the market that allow you to also get search analytics And that kind of becomes the base of one of the key ingredients where You know marketing teams and website owners are becoming more and more sophisticated about Learning how users are interacting with their website But search, you know from a search tooling perspective You know most of the time people kind of ignore completely that aspects or area of Of the website analytics framework You know the second is really kind of where it goes to What are the capabilities set that is a good search experience, you know typically now in bodies, right? And these are, you know, of course, you know You know, however from a ux perspective, do we have a Large search bar for obviously, you know the sites where there is a large amount of content and you know, can they search? You know, can they find the search box easily? And can they start typing in that search box and get some of you know, some of the capabilities that you expect from google Right. Do they have different options to view search results in a different view or format? Do they have facets that you see on the left hand sides here? You know, do they have You know capabilities that allow users to provide feedback on the site search and then when users are searching for something Do they have the capability to kind of really pick the snippet or the area of cord Where this, you know, where the keywords were found and extract that and show that to the user so that they have some context In in in, you know, why a particular search results were search result was returned for them And then finally, you know, there is the relevance aspect of it is, you know How do you make sure the ranking and the relevance of search results is appropriate? What kind of, you know capabilities are there for making sure the right results are shown to the top? And then if there are no result searches, how do you make sure that, you know, you can actually show Suggestions that are more meaningful to the user for, you know, for what they're searching for So if these are the best practices, how do we make it better, right? So as a developer, you know, if you have all the content in in Drupal or on your website, how do we make it better? Right. It seems almost 100 percent of the people here are aware of Apache solar Just kind of a refresher You know, Apache solar is an open source fast powerful flexible search platform widely used across You know across the website I think the last statistics is about 97 percent of the fortune 500 websites use Apache solar to to build their search All sorts of capabilities You know that you can imagine and the capabilities that I talked about whether it's faceted search whether it's filtering Whether it's geospatial search, you know plugin architectures query suggestions spellings Full text capabilities ranking relevance, you know, all those capabilities are already There in the tools that you know, you guys are most familiar with So now what you know, what what I'd like to do You know today is kind of say, okay, let's kind of walk through You know what I did And you know as a developer, you know, what was the journey to kind of say Let's take the content that Drupal.org has and let's make it better using Apache solar So let's uh, you know, let's kind of let's kind of walk through walk through a little bit of that All right, so So here's what I did, right? So I followed through a series of things to kind of get to You know a search a search result You know kind of examples, right that I showed around Drupal con and events and things like that So here's what I did. I took a subset of the website Unfortunately, my goal was to actually index all the content on Drupal.org and present that here Um, but you know Drupal's webmasters are pretty pretty good. They actually blocked me from Crawling all the content of Drupal.org And so, you know, I wasn't able to crawl all the content pieces But what I was able to do was create a very kind of focused Um crawling set that actually crawled about, you know, two to 300 pieces of convent to kind of demonstrate Some of the things that I wanted to demonstrate to kind of show you how you guys can actually You know make the existing search better using just, you know, the same plain plain vanilla apache solar So if you see, um, some examples, let me see if I can I can share Share that here All right, so, you know, this is an example of some of the content that I've scraped Um or downloaded, uh, some of it is handcrafted Oh, great. Let me see if I can mirror the displays again Okay So this is some of the content, um, you know, so what I did was I actually went to Drupal's website And I went to the events page and I You know handcrafted and I downloaded some of the content of all the events that were on the events page Uh, put it in a simple structure a JSON file that you know contains as a list each and every record For each and every item that I downloaded and as you can see it has, you know, the events page the the portland the prog And all the other meetups and and some of the other content that's already there So it's about I would say it's about 30 to 40 items That have been extracted from the events page I also did a few searches Just to demonstrate some of the examples I did a search for site search I did a search for personalization. I did a search for Um a couple of other examples And I you know got the content the content is fairly simple All I have is a url of the page. I have the title I have the description if that's there on the page I have the you know published date modified date Category whether it's a module whether it's an event Whether it's a documentation Whether it's forums or issues And whether it's a a web page, you know page or article So, you know, that's a category and then I have a subtype just for my reference perspective What other you know, if there's some some deeper example for example, this is an article, but it's of a type case study So I have some of that content that's been extracted out of the out of the web pages and have as a list Pretty simple. I have these jason jason files with me and you know if I had access to the entire content base I would just probably get that jason file And and then what I have is a solar server That's you know, that's running On you know, I've I've used search stacks to kind of host solar, but I have a solar You know running that you can see it has a couple of collections I have a Drupal search 572 collection That I've created for indexing some of the content that I've downloaded And you know, it has a basic configuration files and such So then what I did was I you know simply utilized a curl command Simply utilize the curl command to upload the content into my search index Um, you know, I used the the jason file that you see I actually uploaded the events jason file I uploaded the site search jason file. I uploaded the the personalization jason file And you know, all that all that content is now there on on the website So now, you know, I I slapped together a front end on top of it And this is an example of You know what that front end looks like I was planning to host that on Drupal search.org But because I didn't have the content so I kind of paused on that before I have all the content So But you know, I I have this front end that basically if I if I do some of the example searches that we were looking at, right? So let's say now if I search for Drupal, I actually now start to see auto suggestions These are built using solar The solar has a capability called suggestor that can actually build These suggestions or auto suggestions that can be utilized to power And and and give to the user and auto suggest or search suggestions based on as they're typing So if I typing, you know, dru Pal it'll show me, you know, Drupal and Drupal con, right? So let's say if I search for Drupal con It's the same content that we were seeing on Drupal.org But now if you see Drupal con if you search for Drupal con You know, you're actually finding Both the portland and the parag Events right on on top of the screen along with, you know, any other content pieces that you have And this is just basic apache solar using apache solar putting it into the contents into the solar You know environment and making sure that you can search across titles descriptions And category and that's pretty much it fairly simple Next let's look at You know, again, I also configured a spell checking component inside solar And once I configured the spell checking component if I search for Drupal I'm actually getting a correct response It's been smart enough to actually say, you know what most of the content that's already been indexed It has Drupal as the majority of The words that are being indexed And when you're searching for something that does not have any content pieces associated to it I'm going to give you a recommendation around spell checking and actually show you that did you mean Drupal So that's two out of the things that we saw on Drupal.org Let's now search for you know personalization I actually get personalization On the search results And this would be different from the search results if I actually search for person Which is what I was seeing when I went to Drupal.org Let's search for site search Intuitively now I'm getting actually results that are More relevant to what the keyword I typed as opposed to you know, what I was getting earlier Then I also did was I also told solar with the synonym capability that Site search and website search are exactly synonyms to each other So now if I search for website search I get the same results It's smart enough to know that website search is also site search And it's showing me the same results for site search as it's showing me for website search results Let's take it a bit further Solar also has some really interesting capabilities that can actually detect and and parse the words keywords that you're searching for and You can identify some parts of the keywords and actually make your query a little bit more sophisticated So in this case what I'm going to do is when I search for site search It's actually showing me everything today, right? So I have some articles on the top There are some modules and these icons kind of refer to those types. There's some articles This represents a module this represents a forum um another module another module I have no idea what a what a truck is but it is some Some sort of a some sort of a content type. I have to dig in as to what the truck, you know is So what let's let's now get a little bit more advanced and say what if as a user I was more interested in just looking at site search modules Can the search be smart enough to identify and detect that what I'm trying to do is only finding modules There you go So what internally I've done is I've actually utilized solar and I've said when you're searching for something and if the keyword term Matches one of the categories that have identified on the left hand side I want to give more weight age to all the documents That have that are of type modules so that my other results don't get You know hidden but at the same time I'm elevating all the modules that are available to the you know within the content base So in this case, it's the same set of search results that were there earlier But now what you have is you have all the modules on the top And all the you know forums and other things kind of much lower on the relevancy rank aspect of things So, you know, this showcase is kind of, you know, a bunch of capabilities on the solar side Now let's take it a bit further, right? Let's try to personalize it You know, we went to the events aspect of thing. Let's say I'm searching for an event If I search for events, I'm looking at Drupal cam 2023 right What about if I was in Europe and I was actually looking for events What do you think should be the first result typically right now again The the answer could be, you know, quite Um, you know quite contextual based on the person. Maybe I traveled to Europe But I am actually, you know from America, but I was trying to look for events in America But most of the time, you know, if you want to make your search personalized You can actually utilizing solar Personalize your search and in this case, I don't have a UI, but that I want to show is I've actually created a model called geo affinity inside solar And what I'm doing is I'm saying if someone's coming from and this geo affinity model, what it does is It's actually right now hard coded the fact that whenever I'm utilizing this model I I'm actually going to make sure All the I think in this case all the events in Spain are actually ranked higher than all the other events that are there in the system So so this geo affinity model what it's doing is it's actually You know, so from a simulation perspective if you're in Europe, your UI can pass to solar A parameter that tells it that, you know Use the geo affinity model and make an assumption that a person is coming from Spain So in this case what you're seeing now in the results section is That your first result that's come back is a Drupal camp in Spain. That's uh, you know, that was on april 4th So this is an example of you can actually just use the same tool that you are You know that you're aware of and you're familiar with to actually do personalization inside Inside Drupal as well and all this you'd be just kind of vanilla solar that comes out of the box with just some Customizations done to solar based on the capabilities that it's exposed So that's uh, you know, that's what I have so far So from from a kind of a journey perspective, you know, what's what's next for What's next for me to kind of do this is I think the first thing for me was kind of make sure I get the sponsorship for the support for the project, you know the hosting the time my time You know the license and everything that I need to kind of run run the search. I've actually acquired that I think next on my you know next on my list is to figure out a way to get support from drupal.org website To crawl the website and get access to the data Again, this is a call for help You know in the community if anyone Can help and knows how to navigate that world. I would be appreciative of getting some help and how to get that data Um, and then finally I think this is a simple one I have a drupal search.org where I want to launch this page that you saw and start to see how I can Kind of enhance that and start to make all the analytics that people or the community is actually searching for publicly available to the community um I just need to kind of get a permission on the trademark site and I think Dries owns Drupal as a trademark. So I just want to make sure I you know dot the i's and cross the t's from a From a permission perspective And then just make both the website as well as the analytics and insight available, you know available to the community So that's kind of the next on on my docket If anyone of you are interested in role, we're hiring, you know In our customer engineering team both in us and europe Both, you know, we're all a remote organization. We're fully remote organization We have a booth at at the pavilion. Please feel free to come chat You know, there'll be current or me or someone on the team If you're interested happy to happy to chat about it And with that, why don't I open up with some question and answers? yeah I think that's a great question. I mean just for for the folks who you know Just kind of repeat the question both for the recording perspective. I mean the question really is You know, I've demonstrated most of the example here directly going through the api and actually, you know interacting with the api The question is, you know, what's my opinion around utilizing, you know, the the search api module um And where credit it's to I believe drunken monkey is probably the author of Of of search api module versus kind of the api From my perspective, I've used both I've actually collaborated with Thomas sidle as well on the search api work that he's done And you know, he and I have actually exposed some of those capabilities for some of our customers Including natural.gov and other customers Using search api to do a lot of these things I think the more headless approach where a lot of the You know Drupal websites or or or such are going kind of in a headless mode where you don't want to have another api layer in between You can directly interact with the, you know, search Search apis and wire up wire up a search experience based on your needs. The example I showed was a view.js app Which is a headless application that is directly talking to the search api From solar and returning, you know returning the results both ways work You know, I think there are some limitations. I've seen in my experience with the search api Especially if you're using, you know solar and indexing faceting and stuff I think what goes inside solar is only ids sometimes not the entire Strings That actually go in solar So sometimes that creates a problem if you're trying to let's say create auto suggestions based on Some of the, you know, some of the facets that you have so so, you know, you have to do some You know headstand kind of work to kind of get that get that working Um, but both both approaches are are viable. You know, I haven't seen any issues there Go ahead Is Um I I've used suggestor. I've used spell check. It's a separate component. Um, I've used The models one I've used Elevate component To in the elevate component what you can do is you can pinpoint certain items And you can make sure they always come on the top based on certain queries that you're searching for So so let's say if your marketing team wants to do some promotions And they want to say hey this research report is very important Whenever someone searches for a research report I always want this document always on the top for the next, you know 30 days because it's super important, right? And I want to drive traffic towards that So you can use in solar what you call elevate component to do that And then finally, um the affinity, uh, the affinity stuff that I showed Is actually a custom plugin that's been written using a query query parser That that has been written in in java and then deployed on solar to do that Any other questions? All right. Well, thank you all for coming. I hope this was helpful and uh, you know, have a have a great rest of the rest of the event