 Hello We'll come to the session with the Josh's Apologies that we all lost our voice a little bit after last night. I guess that's that's what's happening every in every conference, you know So we're gonna talk about elastic search and Drupal Start from introducing ourselves. My name is Josh Lee. I'm a Drupal developer I'm based in Canberra. So in my entire Drupal life. I'm dealing with the government and Yeah, hi, so yeah, so I'm Josh as well Yeah, so I'm a technical support engineer for elastic Which means that I just get to work with a lot of awesome people and helping them do awesome things Through the use of this elastic stack we have which I'll go into a bit more right a PhD in astrophysics I did research a long time ago, but that's that's far behind me. So don't ask me about that all right, I'll jump in and Dr. Josh Ridge will introduce elastic search later. So the Drupal we know Yeah, it's really popular now and if you guys read the blog from trees that you guys know that not like just adopt Drupal 8 in their Investor Relations website. That's big. Yes, and From my experience that the government like to use Drupal For example, they About security website in department of agriculture. They put all their biosecurity news in Drupal and The ABCs and SPS they all use Drupal to sort their news Ecomers, of course, that's really popular if you guys been into the session that my friends got interested in worth giving about at this Everything pretty much people like to put into Drupal why because it's awesome and Yes, you know, we have a lot of data in Drupal The way that we keep the data, you know, structured is you know, everyone of take down the returns Views, and if you're lucky enough, you just use those faster API with such API And that will actually give you the block that you can place on the side bar Hopefully that you can actually calibrate the data based on different field or properties of that entity and Anyone use I'm pretty sure everyone use entity reference module Reference module and I have seen people using both in the same content type. I don't know why But you know when people started to use entity reference module or you know, no reference or user reference There's a kind of like a structure like a you know, likely you have a parent and you have children between the content type or entities and For most of cases or if you're using field collection, I Know that people tend to put the entity reference field or you know every kind of reference field into a field collection and For most of the cases Field collection is not really, you know, it's unlimited value It's not really, you know set only one value for that kind of times and similar to program module Also flag module is another entity that provide a flag entity that allow people to subscribe to whichever note With everything installed in your content time, you know to content type of whatever in your site It's kind of like hard to Find out the right the right entity you want to find it For example, if I want to find out the reference entity in whichever Content type that happened to be in one of they feel collection field Is I don't really know where can I find it? After I have this kind of a complicated Content type my manager can come, you know, can I get a report of our website? I don't really know what that means because Well, that means a lot that connect that can be, you know, I Can give the total number of each content type to my manager. So he knows what's going on your website or Maybe I just add another filter on top of it, you know per term Maybe he's happier or if we happen to have organic groups, you know Site and you know, I can just keep, you know, how many knows per group or how many uses per group But I don't think he likes that because it's just boring numbers and you know, just the least for most of the terms that The boss it doesn't really have any technical background all they want to know is a nice Paragraph a nice diagrams that can come you can compare with the number of each content type perhaps By the way, this nice diagram I got it from the elastic search website Or in the case that you have Parents and children relationship between the two entities in Drupal that You know, this case always happening in our real life that client who may just remove the parents somehow that made the children often and Sometimes you know it depends on how you compose your model then that were recent errors So the manager would like to know how many contents, you know in percentage that is often in this case, you know There are too many often children and sometimes the client jump on board, you know, I Want to have a visual visualisation dashboard and I don't really know what I want in the dashboard. I just want a nice dashboard So They don't really know much about what they want. I Thought you give me something nice, you know to start with I said you give me that dashboard I will know where to start That's what I was facing in the client. I feel like I'm talking too much about the PM thing I was and and that that's another thing, you know, I can't really name the boss But you know, they always want to explode it and print it and because they want to scan it and you know email to whoever That's what I thought after I got those requests First thing maybe I'll just you know dive into my my circle and Join I don't know how many tables Find out those numbers and least Or if the requirement is easy enough for me I can just you know use a relationship in the views and of course, you know, we can use With that module call you can generate the CSV from views And luckily enough we have faster API installed Right, so we have a lot of blocks based on different categories. Oh, that's actually giving me a lot of nice Aggregated data. I can actually pass that URL to my boss. So he may just you know do it himself And I heard about these three module Not sure if you anyone of you use these remodels this remodels is available in triple seven I'm sure about to put eight and it has the integration with the views. So if the views is easy enough, then These remodels can actually generate a simple diagram for you Print friendly. Yeah, I'll try to print that out in a PDF So All of these jobs, I feel like okay. That's just one single report. I feel like I need to spend like two weeks or something Before I talked to my manager. I had a drink with this guy He actually taught me something different Yeah, thanks. Yeah, I was actually yeah, it's probably more than one drink actually. Yeah, so Yeah, cheers So I might just start with an audience poll because I love audience polls Who's heard of elastic search? Cool. A lot of people. Who's heard of log stash? Cool, some less people. Who's heard of cabana? What about beats? Okay, dropping down. Cool. Cool. Cool. So let's talk about elastic All of those things I just mentioned now fall under what we call a company we call elastic We used to be called elastic search because we used to just have essentially one product which was elastic search Now we have a whole bunch of products, which we just sort of pulled around through and a bunch more which we've also got But at our core, we're an open-source company We have around 70,000 community members. We love our community and we're on discuss and IRC all the time talking with them Getting pull requests from them. They're filing issues. It's great, you know Like our community really helps us out and we thrive and live with that community, which is really awesome We have like ridiculous amounts of product downloads across our stack Like you know like people are just using elastic search all over the place and working in support It's I'm just always amazed at what our customers are doing and what people are trying to do with elastic search and the rest of the Stack as well It just blows my mind or use cases they find out of it and that's that's you know And on the commercial side we have you know people who rely on us for support So to help them out with like not only you know like when it's something is on fire Which hopefully doesn't happen very often but also for like you know helping them design and architect a solution with that stack So that's what we are. You know, we're we're trying to help you people You know with these get this like lots of this data put it into some kind of way which they can search aggregate visualize it Filter it enrich it do all these kind of things with their data like take it in some format Maybe just take it raw just stick it somewhere where they can then sort of like access it because basically a lot of people nowadays You're all probably facing this situation where you've got too many data sources You've got all sorts of like information coming at you and it's just too much to process manually You can't sit there tailing locs anymore. You can't sit there You know like trying to analyze your data by hand, you know, you have to have some way of like filtering and searching through it and That's where we can sort of help so as I mentioned and as we just pulled there We have like this whole stack so did another poll did anyone used to as anyone heard of the elk stack Cool cool cool. So we used to call it the elk stack, which is really cool But now we've got all these other products here So we've got like beats and cloud and you know like elk is an acronym that stands for elastic search log stash Kibana What do we do? Like, you know, like how do we call it? Do we call it? Belk Elk B? You know like, yeah, I've got an Elk B here But yeah, it's like, you know, this is just getting madness and then we add a C in there Maybe we add an X in for our X pack our commercial offering. It's starting to become crazy So what we like to do now is we just like to call it the elastic stack and the elastic stack is composed of all of these things at It's core is these open source products Kibana elastic search log stash and beats all open source You can go and download them a patchy license a patchy to license I'm going to download them play with them. We've got all the nice cool features, which I'll be showing you in a demo a bit later If you want to like have it hosted like you don't want to worry about the management of that kind of stuff You can use our elastic cloud offering. That's basically elastic search and Kibana is hosted for you So you just throw data at it and you can just have Kibana to do visualizations of it And then on the side there in addition when you like basically buy a support subscription Which means you get someone like me helping you out You basically get access to some more commercial features for you. We won't talk about that today We'll be talking about basically the traditional elk stack. I'll let I'll let it slide for today Um, so, yeah, so as I said, like, you know, it just amazes me like what people are using it for like a lot of people probably who Used it for a like a logging solution just sticking logs into elastic search. Maybe one or two people. Yeah, cool Cool. So that's you know, that's like a pretty bread-and-butter sort of solution for us But you know, we're seeing more and more use cases and partly what we're going to talk about today It's like this operational analytics and business business informatics like kind of stuff, you know Where you've got your managers coming at you to you know, like they want reports They want to see how the site is performing and all that kind of stuff And then at the same time you've got your users and that kind of stuff that you know, you're like, you know You have a lot of data and you're you want to give it to your users as like fast And you want the users to find that information really fast I think I was in a talk yesterday where they were talking about the attention span of modern humans It's like eight seconds or something like that So you kind of want, you know that information to appear there within eight seconds So search is probably the best way to do it And you know this like and this covers a lot of different people, you know Like in your organization like, you know, when we're talking about logging, that's you guys maybe you're looking at those logs The performance of your website, you know, your Drupal watchdog all that kind of stuff You know your managers they want to know some of that information in a more condensed and pretty format And then your users they just want to access the data really quickly So yeah, so like that's a pretty much an overview of elastic the stack that kind of stuff I'm sorry. It's waffling on it's a little bit. I guess like high level, you know What does it mean? That's I guess what we're after, you know, like here like maybe if I can just show you some demos and some Pretty pictures and things like that that'll like basically get it across a little bit better So let's start off with knowing our audience So the people we're going to be talking about today in which Josh just talked about earlier was basically two types of people You've got your managers and your users and I don't put these pictures up here to disparage these people because we're also these People like that's me on the left when I'm trying to use a website, you know I just want to know the information, you know, don't show me like, you know all this flash and stuff like that Just give it to me as quick as possible. That's what I want I don't have the time to like look through, you know, all your pretty pictures and that kind of stuff, you know I want, you know to get to it as fast as possible at the same time. I'm also the person on the right, you know I like pretty pictures Sometimes I don't understand what's behind the pretty pictures, but I do like them. So I like my charts and graphs So yeah, so know who you're trying to like target with this information So with that in mind, let's talk about the search aspect first So this is where I guess like users and managers sort of like can both you like rely on this kind of like a feature In terms of elastic that's elastic search. That's you know the search and why would you use elastic search? Well, because it's really fast We do what we call near real-time search, which basically means you index a document and it's pretty much available to be Searched on pretty much like I think the default is one second. So within a second you can get it in terms of like, you know Like regular searches on your website people get like basically down to the millisecond, you know response time for their searches So yeah, so it has that near real-time sort of capabilities. It's highly available and it's Distributed out of the box and what that means is basically you can stick elastic search on a single server And then once it starts like, you know, the resources there start to get a bit constrained and that kind of stuff You can just spin up another server stick elastic search on that and those two servers can talk to each other And they'll just spread the data across them. So basically you can build out a Solution as your data grows really easily because we just have that built in for you and it's like ridiculously easy to use It's nice for developers because we just have this nice restful Jason API. It's Jason in Jason out You talk to it Jason. It speaks back to you and Jason So you can do all that nice optimization of you know, HTTP and all that kind of stuff So it's really easy to talk to and we have a whole bunch of client libraries for all different languages PHP is there pearls there Goes there. So it's all over the place And then finally we have like a little bit of analytics and aggregations on top and I'll show you some of that Mostly through a tool which we call cabana and I'll show you a bit of that a little bit later So in terms of the search aspects Who uses Wikipedia? So as an example of how what you can do with elastic search is Wikipedia that search you're typing in that search box That's going to hit elastic search. It's going to be sending those results to elastic search And it's going to be sending them back to you That autocomplete you see in Wikipedia. That's also elastic search doing that autocomplete functionality for you Um as another example of another I guess a local shop which is using us who uses seek who's using seek right now No, I shouldn't say that I'm who's user who's use seek? Yeah, great. We've all used seek. We'll look for jobs You know, that's how we make our living. Um, well seek is this isn't particularly how seek are using elastic search They do use elastic search. They don't use it for this search front-end right now But I like the seek website because it's a nice example of faceted search Instead of you just having a general search box where your user can just type in something and they get a whole bunch of results back You can filter down the results a little So the users can like sort of click and choose a few categories and subjects and they can refine their search a little which is Really great. And then finally as another example domain who uses domain Awesome, I don't actually use domain. I'm from Canberra and we have this thing called all homes And we're just like weird, you know like all the rest of you use domain and people I'm like What's the main, you know, we're like in our own world anyway domain They use us for you know when you do that properties near you kind of search that kind of people property is near this property That's all powered by geo searches in elastic search. So that's really cool So there's an example of that there So on the other side of things we talked about search And what we're going to talk about a little bit is the like the business analytics those charts and graphs which our managers love Um, you can do that with the elastic stack as well using Kibana Kibana is our like essentially our visualization tool. It sits on top of elastic search. It speaks to elastic search It actually uses the power of our aggregations framework in elastic search And it basically queries elastic search Gets those aggregation results and makes pretty pictures for you So, you know, it has all these capabilities of like dynamically filtering down into the data So everything on the chart is clickable and I'll show you some examples of that a bit later You can filter by time you can click on subjects and things like that. It's a nice search box So you can just do full-text search as well. You can share those dashboards up and coming in a Actually right now in our b5 stack, which we just released Yesterday or the day before yesterday Yeah, so it's basically got a reporting framework So you can basically schedule reports and have like basic dashboards with a whole bunch of visualizations sent fire email to your manager So basically you can automate yourself out of a job So, yeah, so you can do all this kind of fun stuff with Kibana and all powered by these aggregations which are all in elastic search And there's some examples and I'll show you some more examples. So we'll just skip that slide And then I guess the other thing is it's like it's all great to talk about like searching your data and Visualizing your data, but how do you actually get it into elastic search? Well, we have some tools for that as well We have lock stash Which is something you've been using probably nicely for a logging sort of solution So lock stash is basically like a tool which can go and grab data from all different kinds of sources It can grab it out of databases and grab it out of logs and files It can listen for like information coming over a TCP stream It can go pull another like rest endpoint or another like, you know soap endpoint or whatever you want You do all sorts of crazy stuff. It can take that data. It can manipulate it filter it enrich it Do all the kinds of like crazy stuff with it add geo information to it all that kind of stuff And then it could send it off somewhere It doesn't have to be elastic search. He can send it off to a broker He can send it off to some other like I know SQL data stores can do all sorts of crazy things So that's what log stash can do is very powerful sort of generic or all purpose tool for like sort of getting your data from somewhere And sticking it hopefully in elastic search Beats that's like sort of these are like I guess, you know, like our lovely like Unix command line utilities like LS And all that kind of stuff. They do one thing. They do it really well LS list files and directories beats are kind of like that, but for data So basically they take some data source and they stick an elastic search So they're like a lightweight version of log stash They don't do all the enrichment and filtering that kind of stuff They kind of just like basically grab the data source from somewhere and send it off as quickly as possible So kind of really good to put like in, you know Your endpoints and that kind of stuff where you can't really run up log stash or it's not really effective So and beats can send to log stash so they can work perfectly together If you want to do that filtering you can send your beats stuff to log stash And then yes, I do I don't really talk much about that. That's just basically, you know, if you have HDFS You can grab data in and out of that Demo data, that's the fun stuff. Let's do that because that's fun. All right Out of there. Oh, I've got messages. Oh something failed All right, let's go. All right. Actually, let's go back to that first All right demo data So ABC local online Stories, so this is like some I better see the mics that I can record me So this is basically metadata about a whole bunch of stories written by the ABC and local online like basically ABC reporters around Australia This is just a summary information. So this is basically a URL to a picture So a photo which went with the story a little caption about the photo a brief description The title of the story was associated with a link to the ABC website which contains the story So it's just this metadata about it. It also contains, you know, some information like which state reported it The which station which local ABC station reported it and some geo data to say like the photo was taken You know here the story actually happened here in the middle of the outback or something like that So just like a bunch of metadata, you know, maybe something like you might have in your data store in Drupal or something like that You know like which you might search over and or present in your in your website And I just grabbed this off data gov. So it's up there. You can go grab it as well 84 that 8,500 records just go and grab it pretty small. It's easy to use CSB file or Jason file I use the CSB file And how easy is it to do and use this? Well, it's really easy. So this is basically what I did It's only three steps. Normally I put a three step a four steps in there and I have a blank three step and profit You know, that's the funny joke. But anyway, yeah, no, it's like download the CSB file I basically index the data into elastic search using log stash I configured it, you know, basically with and like, you know to read the file do a CSB filter Send to elastic search really simple really easy to do the config is really small. I can show anyone later on I didn't want to show it today because you know when you show code on a screen It's kind of ugly and everyone just like sort of nods off. I would call the cold later. Don't tell that Yeah, no, so we you know, it's a very simple config So just basically stick it into elastic search and then I just made some pretty visualizations in Kibana And I'll show you that dashboard Right now I do apologize because like my laptop is horrible. It's like a Dell and it has this dongle and the resolution gets crazy So Kibana is a bit squished, but we'll have a look at it So this is Kibana v5 version of Kibana and this is the ABC local online dashboard I created so you can think of this as like, you know You had this information in your website and maybe you're your you want like some interface where maybe like, you know your Managers or like, you know, just your devs can sort of like query and look at the data in a more simplified way of out going through the website or something like that So basically, yeah, you can see the data here. I've got like a tile map which contains some geo hashes geo grids Of like the density of the stories. So the red or it is the more stories were reported in that location I've got a like just a little bark bar chart over there, which has the state distribution So the count of stories per state Who likes pie charts? I love pie charts I don't know why all the hate for pie charts. I love the pie chart So I do a pie chart of the subjects which were tagged in the stories and down the bottom there I've got like the actual pitches. It doesn't quite fit, but I'll scroll around So, um, yeah, and basically this is Kibana. So it's just taking that data And it's just interrogating elastic search in real time for this data So it just goes and does a search and then presents it on all these charts All of these charts are diff individual aggregations which are sent to elastic search It gets Jason back it takes that Jason and renders it basically using D3 a bit of JavaScript magic stuff I don't understand, but it's really great And you know, this is all you can like basically if you have time base study you can filter this by time So basically you can choose, you know, like the the time region for it. As I said, it's all searchable. Assuming I've got internet I do Let's give it a try Look at that. I've done a whole bunch of searches. Awesome. Let's try Gold Coast There you go. We've got a high density in the Gold Coast of stories for some reason We've got in the middle of like Australia maybe something to do with gold and then up in Darwin Maybe they're trying to claim that they've got a Gold Coast up there But anyway, we can see the stories about Gold Coast and we can see, you know The Queensland state state reported most of these stories We can see the subject about, you know, human interest that kind of stuff and we can see the pretty pictures You know, like somebody's metal detecting and the caption there, you know, like all these stories about the Gold Coast Likewise we can search for cattle because I'm a Queenslander I can say that I love that. And yeah, there you go. Look, it's all in Queensland Awesome. Probably up there. That's Rockhampton beef capital Australia. Whoo. Yeah, so there you go And you know, you get the stories back So it's quickly searching that data and presenting us back that data really quickly, you know Essentially, you know in almost, you know, what was that like maybe a second less than a second? I don't know and this is all filterable So I can like go and I want to like, you know, look at stories which are about like rural beef stock Cattle and now it's filtering it down even further I can pick, you know, another category. Let's go Disasters, maybe I'm a bit morbid Yeah, and it's filtering down the stories even further. It's doing this all in real time going off to elastic search querying it doing this filter sending the data back presenting it in a visualization for you. Awesome So that's like one example of like, you know, some kind of content, which is pretty pretty normal, you know Like this this applies across all sorts of different data So here is like for example the LAPD crime data for back in I forget what time period this is over a couple of years ago You can see LAPD and you can do things like filter by crime, you know, where should I not live, you know Maybe like resist traffic crimes and you know the graph changes and all that kind of stuff that updates in real time So, you know, this is kind of like if you had a bunch of data and you needed to present it internally to like, you know A different audience. This is something you could give them They can make these pretty dashboards and graphs and things like that and then they can go and we don't want to break We can basically, you know filter this data and they can like play around with it Do all kinds of crazy stuff with it manipulate and see it in real time, which is really awesome And then just, you know, like obviously What have I done here? Terrible with my mouse, you know, obviously you've got like the traditional sort of like apache logging sort of situation We've got our nice graph here, you know, like we can see the logs from our like web server So you can think you could stick your DB log in there or you know You're apache log stick it all into here and you can have your internal, you know Like dashboard to monitor your website from a back-end sort of way You know, like in again, this is all filterable The other cool thing about all this is when you like to go and filter this things You can export this data into a CSV file so you can go and take it and manipulate it and analyze it in another place If you want to but yeah, so you have these nice dashboards To like, you know to look at your data and then just as another example, you know Like going back to sort of the manager side of things, you know, your Twitter analytics So this is actually our Twitter feed like taking our Twitter feed of I think the hashtag elastic And elastic search and maybe ours products hashtags Basically looking at it over the last 60 days and we can see the distribution of where the tweets are coming from You know what people other tags people are using some various graphs about it, you know And as I hover over I can see like, you know, these values are changing. It's all dynamic all that kind of fun stuff So yeah, so there's a lot of different like ways You can like basically if you look at extracting the starter out of Drupal or whatever form wherever it's coming from And sticking it to elastic search can be quite powerful so And that was all done all of those things are basically done with Logstash Where's my clicker? There it is So yeah, so when you think about it, like if you've yeah, we've got those two different types of audience You know, you've got your data sitting there Which your users going to search over and you're going to have a nice search box somewhere Maybe some faceted search and some filters and things for your users But then at the same time you can you know, you can get that data out put in elastic search And you can give your managers and that kind of thing Kibana or your internal users, you know In different groups access to that Kibana and they can go and generate their own reports and visualizations and all sorts of do their own Searches and that kind of stuff you don't have to design another website because you know in some ways You don't need to maybe go to that full-on website sort of design for those internal users So yeah, you've got this power to basically You know Like I guess please both sides of the story Um, so with that in mind, I guess like you know, there is You know solar is a big thing in the Drupal community and so how does elastic search compare to solar? We're essentially the same product. You know, we're both based on a patchy leucine You know, we have a like we basically I went I went disparage solar. It's a great product But what elastic search has it's based on a patchy leucine. It has a nice restful Jason API It's built to be distributed out of the box It has these real-time search capabilities and most importantly, we have this aggregations framework on top So you can actually do things like histograms and averages and pipeline aggregate Aggregations so derivatives and all sorts of crazy sort of mathematical things. You can do that in elastic search You just tell elastic search I want you to aggregate over this search result and it will give you back that aggregation result as well Compared to like things like a relational DB, you know, our postgres or whatever we're using for our data store for Drupal I see that as complementary MySQL and relational DBs in general they make different guarantees about your data We make guarantees around search. We want the search to be really fast MySQL and that kind of things they make more guarantees around storage, you know So basically to like guarantee that your data is there and it's correct, you know and all that kind of stuff So I see there's a complementary thing there and basically you're going to have a lot of data So you probably have more data in my SQL and less data in elastic search because there's only probably a limited amount of data You actually want to search on you don't you don't want in all cases to search on every single bit of your data There's just a certain fields and information you probably want to search on so there's a complementary use case there And they're completely different tools. There's no way you can say this versus that So there is some there's some complementary use cases there And that's likewise for any kind of stories like other no SQL things like couch base and Mongo Whatever you like to use So yeah, so the bottom line and hopefully you through this demo and that kind of stuff is you know You use the right tool for the job and in this case what Josh is going to talk about Here in how to integrate it more into Drupal is that it's elastic search I'll head it back over. Thank you, Josh You like his search is fantastic from what I what you said, but I don't think that actually helped me so far Because you know everything Josh was was demo, you know was talking about that has a condition that we have to have the Content in elastic search How can we get that content from Drupal to that elastic search? I'm going to talk about that. Otherwise we can't use Yeah, you know we all from Drupal community and there's a sentence in our community thing that there's a module for that In fact, there's there are three But Those are not really completed So the first one elastic search connector That's fully working in Drupal 7 and it has different components The first one is the repa for the elastic search PHP library So using that repa you can actually use the API to perform a crowd or search in Drupal It also has a component components to actually ship the watchdog log into like a search Also, it has a search API integration It has the views integration all available in Drupal 7 not Drupal 8 So well last time I was trying to install this module in my Drupal 8 side. I actually broke my side So it's under development The second search API elastic search is not available in Drupal 8 yet Last time I was checking the progress. It was kind of like a quite far behind the third one is I think that's the most Closest to the production. It's called elastic search helper what it does is it provides a Plug-in in Drupal 8 and that plug-in give you the full function about you know the crowd and search function in Drupal And and that provided repa What do you need to do in your side is you need to create your own custom module in your module you implement that plug-in You don't have to do anything in your module You just implement that plug-in that API becomes available and in the module file in elastic search helper it has You know all of the entity hooks in there. So when you update the entity then it can fire up A query to update the entity in elastic search such as that but All of those threes they all need work and I do not really see You know currently a lot of active works on that so Well, you know we all the Drupal community and We should actually contribute to those modules so we can actually leverage the functions from elastic search If you don't really want to use those modules, of course, you don't have to They elastic search PHP library is Maintained by elastic company and they have you know that the library is well documented So if you want to create your own module all you need to do just you know add the library into your composer file So just use the composer to install this library and in your custom module you can just start it to use the elastic search PHP API That provide a full crowd and search and including the aggregation and API is I didn't really show that code so if you guys are familiar with the query that in elastic search is really just a JSON that you can actually convert it into a ray in our module and They key can be you know Aggregation and you want if you want to aggregate by whichever term or whichever field you just put the key in there and You know the value in there. So Pretty much just passing their race to elastic search and it will just return the JSON That's the code example. Yeah, I don't want to show too much about the code And this example is actually from elastic search helper module So what you see here is really just three hooks and in those three hooks We actually use a service and to you know index entity or we insert entity. That's how we trigger the performance in elastic search and There's another way you Josh. I've been talking about using lock-dash to Index the content into elastic search Lock-dash is a wonderful tool is open source and is written in ruby I'm not a ruby programmer. So I can't really read that Unfortunately, but it has a lot a lot of plugins. One of the plugins is called Drupal DB log That's really awesome. So what it does is pretty much Shipped our work dog log into elastic search automatically. So a lot of actually it's a command line tool. That correct, right? Yeah, so all in your command line You just need to run the query in lock-dash and that will actually read your Drupal site and ship all your work dog Into like a search. Also lock-dash has another plugin called RSS reader We all know that we can actually can we can configure our views into to export RSS feed so if we can manage to actually Expose our RSS feed from our views the reader from lock-dash can read it directly and input that into our elastic search Database and this morning. I just learned something new from George There's another plugin called HTTP polar HTTP polar. Yeah, that's actually more advanced that You can actually pass the URL to the HD polar You know that URL can be JSON format it can be whatever format And in the lock-dash configuration file you define which field I want to you know push to elastic search or You can create a filter in there. So I want to convert this for example In triple seven. I'm not sure if you guys know that in triple seven if you do an entity load and In elastic search what it sees is if it's an NID node ID what we see is actually a string It's not really an integer So lock-dash provide a filter that you can you can actually create your mapper if it's a node ID And you need to tell you like a search is a integer is not a string something like that And then yeah, and I guess just finally and the other way you can get is the really rough and tumble way is like lock-dash Has a JDBC input so if your database has a JDBC driver You can just hook that up to it to find a select tool whatever query to like grab the data out of your database and stick in the elastic search And do the filtering and everything you can do magic you can do a flow Or if you if you want to Fill to the data before you actually reach elastic search you can do it from the the module level from the application level All right, any front-end developer here Right I kind of like I know what are you guys thinking about? It's you guys probably thinking about you know We have a really nice interface from elastic search because it's all JSON All you need to do is just to actually send the query to elastic search That's possible that you can actually create Angular app and use that app to contact elastic search directly Yes, that's possible because elastic search provide a really mature JavaScript library and Also, I think there is a plug-in or what you call more module in Angular that's available that integrate These three and elastic search that actually make that possible that I just want to have a local browser app And that app can actually contact my elastic search database There's no true possibilities here But we can't do that Why I was trying to well Just wiggle a bit. You try hitting it really hard I haven't I haven't talked in and I'll fix it. Yeah, right Oh Good time Yeah, I was trying to you know play around the Angular so I fair up my own server and I in some elastic search in there And I started a really really simple Angular app, you know just pretty much give give us some filters and Some drop-downs I choose whatever drop-down a real return the JSON I'll use they this way to generate a really nice pie chart or whatever chart I want to because it's just great a job that was working I was really happy after that night I got I got an email from my service provider saying that my server is down Because I left my elastic search exposed overnight I'm not going to tell you the IP though So look up and show Dan We can't do that it's too risky to actually expose your elastic search database to all of the browsers So what we need to do is to actually provide a rapper in the middle to do at least the authentication That's what I thought Well, I was using Symphony to create that create that rapper but since to pull it actually is simply and it is really simple I'm pretty sure you guys all know how to create the end point in Drupal 8 to provide a repar One of the module or two of the modules I was mentioned about They elastic search helper and elastic search connector They all provide a nice UI that you can actually type in your authentication details in Drupal and that will actually do the Optimization job for you. So I reckon, you know, why don't we just install those modules in Drupal 8 and we have all Well, you know custom modules provide the end point. So our Angular app will contact our restful Endpoint in Drupal 8 and Drupal will handle the rest will convert, you know, whatever queries you type in and you into elastic search and do the do the search and You can use D3 or you know, you can use Angular or whatever polymer to render whatever Pie chart or bar chart you like to questions Yes So elastic search has its own graph API. It doesn't use graph QL or anything similar to graph. It works a little bit differently Sort of like think of it like, you know doing searching through these nodes and things like that and looking for the uncommonly common connections So rather than getting super nodes back you sort of get like the actual relevant sort of connections because this is all based on essentially search So it's all about relevancy between these two nodes. So there is language. It's all JSON as well So there's actually a whole API built in on elastic search and we actually have a UI as well So you can go and explore it and you know, make you pretty graph diagrams and things Not that I'm aware of because graph is so new. So, um, yeah, I'm sure you can like, I mean, it's an API Just you can go and hack on top Kibana is is built by Angular is using Angular and D3 So there's if you want to well, for example, if you want to display a nice diagram from Kibana to your website There are different ways that one the easiest way is actually Kibana can export iframe Then you just copy the iframe to your site, but that's static. If you want to display the dynamic data into your site You can either grab you can you can export the query from Kibana that is used to generate the diagram use that query so, you know, you know what character query elastic search and Use your own front-end library which whichever D3 or something to render the JSON into different charts Yeah, so I'm in the next version of Kibana or actually in this version of Kibana because we've got v5 out now So basically, uh, yeah, we we will be building upon like a plug-in framework for Kibana So the visualizations I showed there, you know pie charts cool bar charts and all that kind of stuff You'll be able to add your own visualizations in there Some of our community members are already doing that for radar charts and swim lanes and all kinds of really cool Visualizations on top of that we also have a tool called timeline Which allows you to have access to more of the advanced features of elastic search like pipeline aggregation So you can do those derivatives and moving averages to smooth out noisy data and see rates of change and that kind of stuff and draw all kinds Of crazy like arm charts, you know more complicated charts So yeah, there's a lot of work going on right now for that kind of stuff And I think people can actually contribute to Kibana using Angular, right? Yeah. Yeah I mean Kibana is an open-source product just gets on good hub my question is maybe you know, there are some other questions about that is beat and It's beat and they are the they are the input to open source. Yes So like log stash Kibana beats elastic search all open source all apache 2.0 Questions questions more questions. Yeah, sorry. Yes, I was I was using express in Node.js There's there's I think there's a library from elastic search in Node.js that actually provide they The full rep are in in express so it's really easy to fire that up and I found that was quite useful But it really depend on the requirements So if if you can actually handle the traffic in Drupal 8 that that's that's cool because Drupal 8 actually gave you that The ability to actually measure data, you know, that's what Drupal them Yeah, so in elaborating further on that out language clients often have support for things like connection pooling and that kind of stuff So you can because you know elastic search distributes you'll have a whole bunch of nodes So you'll just like, you know, if you're using one of these language clients You'll just say all these IPs connect to these and it'll just round robin the requests out to them And then in elastic search There's also architectures you can build out where you can have things like nodes Which just act as coordinators to send queries off into the cluster and that kind of stuff So it's pretty easy to build this up at scale in both your application and also in elastic search So there's plenty of architectures you can go through and you can go down that rabbit hole really fast So there's no more questions. I'll just mention just quickly. We're going to have a tour in Sydney shy the founder He's actually going to be in Sydney in November the 29th. So come and see him. I have to talk after him I'll be really nervous, but yeah, it's shy you can go and hug him You probably can't hug him, but you can go and see him. He's a real person So come see us in November the 29th elastic on the day of like things our customers will be talking We'll be there all our tech guys in Australia will be there You can ask him all sorts of questions and then next year if you can make it San Francisco We're in San Francisco a big user conference three days. It's going to be intense all developers are going to be there So if you use anyone like you know who wrote code, which you don't like you can go and smack him in the face So there you go