 Hello everybody Good morning Everybody is still more or less asleep, right? She's a bit like it. So let's see where we can take this Who is already you doing a lot of live dashboards anybody? Okay, so what is everybody using for live dashboarding is anybody using Kibana already? Okay, a couple but not everybody Let's see what we can do then so thanks for the introduction Yeah, I'm working for elastic the company behind Kibana and a couple of other projects So we kind of started off with elastic search the thing where we store the data and then we built a lot of Ecosystem and tooling around it basically Maybe you've seen this one. This is very widely used if you have logs or any monitoring and Metrics and things like that and it's like elastic search log search Kibana the three original products basically that we had Together they formed the infamous elk But it's not only logs and it's not only elk and that's kind of the idea of this talk That we want to go a bit further than just the classical logs because the second a lot of other things And we want to take a look at that a more recent picture of the stack would look something like this We have added beats, but we have added lots of other components as well For example, we have tracing now, but there is also security with seam And there are many other things in this stack basically and today I want to be a bit more on the business analytics side So you have some business data and you want to live show and dashboard that that's the general idea You can run it either on our cloud service or you can you use docker? Which looks something like this that would be the minimal docker example. This is what I'm running for this demo So this is all you need to do with that. You can just run that thing here. Okay, so Enough slides or let's we'll skip to the slides back to the slides every now and then but if you've never seen Kibana This is Kibana So I hope that makes more or less sense for you and so that this is very reproducible for you what I'm doing is we have this Sample data set in here So we have three possible sample data sets and one is more like business data We have like an e-commerce shop We have orders in our e-commerce shop and that is the one I want to add and we'll use for dashboarding So if you want to try out what I'm trying out You just need to load the sample data and you have all the data already there for you And you can just go wild with that pretty much and it has added that and it has added a couple of things in the background as well But let's quickly at first jump into the data set. So you can see in the last 15 minutes. We had one person buy something from us That's not a lot of data to dashboard. We'll probably have to switch to a bit of a different time frame So let's say we're going back like the last 12 days um Days ago This is the one I want and we can also go like I don't know 15 days into the future since this is a generated data set you can look into the future, which is kind of very convenient, but Normally you won't be able to do that but switching to that time frame you can see This is what we have today Like the red line is where we are today and this is the data that we have So we have around 4,000 or so shopping events in our data set That's the general idea where we are and now I can unfold this one And you can see this is all the data that I have in one specific data set For example, you see somebody was buying something out of meant closing and meant shoes The customer full name was Jackson Fletcher and you can see for example where He was from from California and you can also see like for example How much money did they spend with us? 126 7 whatever dollars is what they spend and you can see these were the actual products They bought so you could just have here the total quantity and this is the kind of data that we have This is very timeline based so you can see every order This is generally sorted by when an order was placed. This is all the data that we are collecting here Taking a step back again now So we've seen this There are two ways to Look at your data in general One is we call that entity centric and the other one is more time series based What is the approach that we have looked at so far? Pretty time series based because you you saw in Kibana when I looked at it You had like how many items were bought for every single day What would be an entity centric approach in comparison to that? It's more for example, I would the entity that I want to focus on would be for example the end user So I want to aggregate like I want to have one entity per user and that contains the entire shopping history for that user It's not their individual purchases But it's for one specific user what they have bought overall so have more focus on the specific person doing something And this is also something we can easily generate out of this one here So just to show you this is the time series based approach that we had so far But you can basically take the data because these are all the events and now I want to aggregate them together into these entities How do I get to that? That is one of the more recent things that we have added and we can quickly put that together So what you have here is it's called transformations or transforms and with transforms you basically can create a job where you out of The individual events that we had over time we want to aggregate that entity centric view together So what I'm doing here is I take that data set that I have it shows me the individual entries And you could for example see here we have five of the 28 fields We could switch them around and show others But what I want to do for example is I want to group on the individual buying something and to keep it simple I will just assume that the name is a unique key here, which is not true normally But to keep it simple We have I think we have a name a field called full name We'll use that we'll aggregate based on the full name and then we want to run some aggregation What do we want to get out of one individual user? Maybe how much money they spent with us? so we could have and I'm never sure what the fields are called. I think there is something Products tax amount is not what I want tax full price Text less price. Let's go for the text less price and I want to have the sum of the text less price And now you can see this is how much money this specific user has spent with us in total And maybe I also want to add to that. I want to have a count of the transaction IDs and I actually don't want to have Product ID We want to have a value count of the other IDs and then you could for example see how many times have they been shopping with us and Let's say this is good. I go to the next step. I say like this is let's call that the Transformation is the user View Let's keep this lower case because it's nicer And then I can just say this is We throw this into a so-called user and then we can just look at that user and I also want to run this continuously So every time Based on the product order every time a new document comes in with a 60 second delay We will aggregate that into that specific user again. So we can just Start the transformation It's calculating data for us and now rather than having this Time series based approach. So this is the one that we had before the time series based approach I can now switch to the user You see the time series is gone because I don't have a good time field anymore What I have instead is you can see in here. We have the user How many orders they have placed and how much money they spent with us So for example, I could make this a bit easier to read like a toggle this column and a toggle this column So now we have here what we see is the customer name and how much money they spend and then I could check out like Who spent the least money with us that was Jim Pratt He only spent seven dollars with us or who spent the most money that was Vakti and he spent more than two thousand dollars, which is kind of weird because everybody else only spent a couple of hundreds So maybe this is an outlier, but this is the entity centric view where we basically continuously aggregate out of individual events We go to this entity centric approach to see how what everybody is doing Which can sometimes be exactly what you want because you don't you're not really interested in the individual events You want to have more this entity centric view like who is the user spending most money with you? And you just have that in one entity now. So that's one approach and that's what you can aggregate together here The next thing that you might be interested in is you might want to graph something out of your shop To see what people are actually doing here. So to create a very simple visualization Let's go for a line chart because it's one of the simplest one and I take the original data set that we've added here and By default it gives me account of all the transactions that are had in that time frame So you can see from 12 days ago to 15 days into the future. We have 4,000 shopping events That is not really very helpful. So I want to show on the on the X axis for example I want to have the x axis and I want to make this a data histogram So first off now we see the individual transactions per day How many transactions did we have today? So this is a simple count and then Besides the amount of items bought or transactions We could also say I want to have a different aggregation here For example, I could say I want to sum and let's go for that tax less total price Let's aggregate that again and now you can see on the one axis here We see how many items did people buy that's like 70 to 80 or so You can also see like how much money did they spend in our shop every day Which that might be a nice graphic that you want to put on a dashboard somewhere around the folk selling stuff So they know like okay people are buying stuff or they're not buying stuff and we're in trouble One thing that is maybe a bit annoying in this view here is that here We have the same axis for both and this is very small and you cannot really correlate How is one doing in comparison to the other what you could do for example is on the metrics and axis? You can say that this one here I want to go have this on a new axis and This is going to the right-hand side now And now you have them kind of like side by side and you can also see like how do the number of orders and The total money spent with us correlate So this shows you basically how is your shop doing right now and we can by the way rather than calling this count here we can call this items sold and We'll call the other one let's call it money made and This is what you have here and you can by the way if you say like we want to have this in red You can sort simply switch the color here And this is the very first visualization that you could build and you can see from a business perspective This is showing you kind of like some of the most interesting things already in your data Let's save this one and say I'll call this custom Line We'll get back to the custom line chart Another thing that you sometimes want to see for example is you would want to figure out like who are your biggest spenders What might be one visualization how to figure out your biggest spender? It could be the infamous tag cloud. So for example, we take our data set here again Right now it aggregates everything together and we can see all is here and now I want to Bucket that and first off I don't want to have on the account. I want to have the Buckets and here I go for a specific term and the term I'm interested in is the full name What you have now is these are the people who did most transactions and these are only the top five people who did the top transactions Let's switch that to 25 This is the top 25 people with are based on a number of transactions But maybe you're not interested in a number of transactions You might be more interested in how much money they spend and then you can switch this one around here from count to a sum and again, you can just What did we have the tax less total price if I run this one here? Now you basically see who is spending most money and maybe 25 is too much. Let's reduce that to 20 Okay, it looks kind of decent You can see who is spending how much money with you and maybe you want to send special love to those customers to tell them Well, we like you please spend more money with us Just to figure out like who are your top spenders and obviously this can always be selected per time frame Like if I change the time picker here I could see like who spent most money with me yesterday, for example, or the last week or whatever time frame you have So this is should be very interactive as well. What we can then now do is we can save this again I call this custom tax The final thing that we might want to add like just to do a third visualization is I want to see The average price per item that people spent with us so we can see Does our average price go up or down and maybe you change something in your data set and then your price is changing over time So what we do for that one here is We create yet another visualization if you've not seen this one. It's called TSVB is called visual builder. This one is a bit more Advanced in terms of you can click together a visualization and even do calculations and aggregations in it in a very visual way I'll show you how that one works in the lines together. So by default it shows you the number of transactions again over time, but that is not what we want. I Want to have an a view of let's call it average price That we want to have here. How do we calculate the average price? Probably we need to have that count to know how many items people buy and then we aggregate the price together or Sum up the prices of that time frame to see how much money they spend so we have a count This is what we need in addition to the count. We also want to have a sum and in the sum I have the Text less total price for example now It's just showing you the text less total price and now I need to combine the two to combine the two You add another step here. Basically. This is all just one step followed by another step followed by the next step what I want to do here for example is I have a mathematical formula and This is a bit tricky but follow along I Basically say like I have a count and I give it the variable name count and I have a sum of total price And I'll give it the variable name price and what you can then do is you can reference those variables params dot price divided by Params dot count so this params dot count basically references this variable here that falls back to the count that we've defined up here and Params price is the same thing the sum of Texas price in a price variable and we have that price variable this one here basically so this is what we have then here and This should give me some data. Let's see Unless I did something wrong And I must have done something wrong What did we get we have the count of the items we have the sum of the text less price? Let me see do we have a better field product price. Let's try this one Okay, somewhere Somewhere I have I'm in the right data set. That's good All the data is also the field we want so we want to have this based on the other field That's the one I want We have the average price Params dot price, let me take this one out again Okay, this is the price that we have in the last 24 hours This is what we want and we want to have that divided by params dot Count It doesn't like me for some reason right now Okay, let's go just with parents that price for night doesn't really matter So we we see how much people bought in that time frame, so that should still be correct Let's take this one and save it as well So we've saved all of those The three visualizations that we have what you then normally want to do is you want to kind of combine them because you Want to have a bit more of a general overview and for that we have dashboards So yeah, there are some dashboards that we have prepared So with this sample data set you have some dashboards that you can just get and you can see these are the prepared ones But we can also just look at the ones that we have created ourselves now So for example, what we can do here is we just create a new one And in this one here, you can just add the custom visualizations that we've added So that's why I've kind of always put custom in front of them. So I can just click all three and it will Add those and now we could for example say like I want to have this here on top And you can see how prices are developing and then you can see this one here with the average price We could have that in comparison below it and then we can I don't know keep that one in the middle here And then you could just say like oh, this is nice. This is the dashboard I want to show on some big TV screens and we see how much over this time frame are our sales developing like for example In the last seven days. These are the curves that we have here So this is nice what you can by the way also do is from on this visualization here, for example You could annotate it. So that's another thing that sometimes is not super obvious how to get there is here You can say I want to add it this actual visualization again You can see this is how it's going and then you have for example under annotations You could add an annotation so the annotation that you might want to add is or let's take a step back What do I mean with a notation? Annotation is for example, you have a sales event like you have some you start the fall sale or the spring sale now Whatever and then you can just annotate that in your graph and then you can say suddenly Oh, my average price decreased, but the number of total sales increased afterwards because we changed the price point what you want to add into this one and I'll quickly cheat and Copy the right document because you don't want to see me type that in what we want to do here is I'm adding a raw document. So these are the raw documents that you can annotate this could be For example here. I'm saying We say on the 12th of November we start the fall sale if that is really the thing that we want to do Here is doesn't really matter. So we have that fall sale as an event that we have added and what you can then do in this visualization here You can say we want to take this out of the events index. So this is what we've just created We've just created this annotation event the events Based on that timestamp and then I want to have this tag and Basically the field that I'm interested in is the description and then I can just use description here and use the description field here and then if you have the I don't know. Let's say two days from now and 10 days in the past Here for example, we started the fall sale and then you can see how the sale changed over time Obviously it didn't change in my example because We just set the tag here, but this is how you can Update your graphics We confirm and overwrite this one if I go back to the dashboard And you have the right time frame for example here. You could have this annotation then and you could say here Okay, this is when we started the sale and then for example I want to zoom in how did sales just change right after we started this sale and you can see all of this is Connected and interactive so if I zoom in here the tag cloud here adapts to that and this chart up there also kind of changes according to that one So that is where we want to head with that one One other thing that we recently improved a bit was maps This one. I'm not clicking together. I'm just relying on the sample data that we have in that one here This one for example shows you based on specific regions where you are And The actual map material is only available when you're online. So let me quickly do that here So the data is in your instance, but the map in the background That's actually just open-street map. That's what we normally fetch When we refresh that dashboard So now that we're online, we can also see the countries So you can see basically this is the UK this is where we had some sales in the UK and Then you can for example see like how much did we sell in this place? So we had 130 sales events and we filled $9,500 or so in the last seven days in this specific region And this here sales revenue is just to show you this is just falling back on The sample data set that we have I have the geo IP location So basically I know where somebody is coming from and I just show this as a point here And then you have the total sales we have once you zoomed in enough. You can see the point you can see okay We want to have this course grain and we just have to count of the sales events and then we have a sum of the sales event on Textful total price as well And this is where all of this data here is coming from and how you how you see how much you sold in the UK in the last Yeah, seven days or in Morocco for example So these aggregations are Here and easy to use Okay, with ten minutes left and probably a couple for questions. Let's finally come to canvas. Has anybody used canvas before? Has anybody seen canvas so the idea of canvas is maybe you have this problem that you have this report that you need to generate every now And then and then you have some charts or whatever and you take a screenshot And then you put them into your own PowerPoint So it is kind of like part of a nice presentation and it's kind of yes You do this every month because you have the monthly presentation on some sales numbers And the kind of idea is what we want to solve with canvas is that we want to basically give you the chance to Skip the copying stuff to PowerPoint. That's the general idea here. It is Your canvas is pretty much this white space that you have here Just to show what this looks like this one here is this and this is just you have a space And you can just arrange stuff any way you see fit for example Here and you can see the total sales This is a metric number the value of the sum of this price and this is backed by this query Now you might be confused. Where is SQL coming from suddenly? Because we're using still elastic search which didn't speak SQL for a long time But we have a SQL read-only interface So since a lot of people are used to SQL you can back all of these charge with charts with SQL now So for example what I'm doing here is over the total lifetime of this entire chart We select the sum of the Texas total price as some total price from our index And then we just say like okay This is a value and this is the total price and then you can format it like you can see here the number formatting You could then just say like Yeah, this is a currency and we just abbreviate 250k with a K at the end and we have the dollar sign before and you could for example change the font size How you align it etc and any one of these elements is live data and driven by our data set in the background And you can also say like for example this so it should refresh every 15 seconds So you could just say like this will refresh every 15 seconds And then whatever happens in the background in your live data set These charts will always be fresh and you'd never have to copy out kind of like a screenshot from Excel or whatever BI tool you have into PowerPoint to present like that But you can just do it like here and you can have multiple pages as well And you can just style it any way you want so you can totally bring that to your own corporate identity And then just show how these work To give you an idea how to create a new one This is sometimes a bit tricky. I admit that canvas is It's very powerful, but it's sometimes a bit challenging to use so for example here You can say okay We want to have this in light green like this is our background color as a company And then for example, we want to add the logo of our company and to keep it simple like the one logo That we have built in is our company logo, so I'll just keep this one here So this is for example The logo that you want to have and then you can just add any other elements that you see fit That you might want to have so for example to keep it simple We can just add a line chart here by default any data set that we have in here is backed by a Custom demo data set, but you can just switch that to a different one So you could either target raw elastic search documents or you could just do an SQL query again So right now I'm doing all the data from the Kibana and sample data set Who likes SQL? good How do we get back to the how much we sold over time? Probably we want to have something like some and then depending on what field you have and I always forget my field names Let me quickly check in the field names. So Which one does make most sense? We could for example say textless total price. Let's take this field here Bless you. We take some of the textless total price as total price We can save this one and then in the display you want to say that that one here is actually a date and The y-axis that is then the value of the total price that we had and the x-axis is Wherever my date picker is But first probably we need to add a date picker anyway So what you can also say here because so far we haven't picked the date yet What you want to have here is you want to set a time filter for example where you can say this one here And this is then interactive as well was just the last 24 hours of time and I actually want to have that on the order date and Then you can just fix that down to a specific time frame of that order date And that's how you can build totally independent dashboards off that just look exactly how your company logo looks and With the data that you want to have in here and with that I think we're getting pretty close on time So to kind of wrap up what we have here is What you can do in Kibana and also elastic search is more than just logs. It's more than just full thick search It's very much on that I have some business data and I want to visualize that business data and correlated maybe with my log events for example That is possible We've covered on that entity centric versus timeline view the timeline view is always nice If you want to have like this is what happened over time The entity is like who are my top spenders for example, and you can just extract those Then you have various visitations that I've shown including canvas where you can Customize exactly the way you want to see data for your use case and that's Pretty much it and I think we have like four or five minutes left on any questions or anything else you want to see Yes, do we have a microphone? Yes Hello, I Like for you to go over again for in the transformation part and you create kind of a new Dataset called user. Yes. Is that something that you can reuse later? Might not be a elastic search or is something is a process running. I Didn't have the full grip on so what do you have here? And you can see this is the job that is running continuously So what this basically is doing and I want to say this one here you can actually see This is deep Let's go to the Jason. This is kind of that what is really happening The UI is only showing you part of what what you're doing. You're taking this index here You take all the documents out of that index and you put it into a new index called user And that is basically you run some aggregation and you aggregate from one index into this other index so Then here you can see this is what we aggregate on we aggregate on the customer full name and We then extract these two metrics and what you do here is let me show you the raw documents if you go to Get user Search oops So this is one of those documents. So you basically go from Get what is my original index called Kibana sample data e-commerce this one here So you have the individual sales events and from those you go to this Entity-centric view where you just for this this specific user This is whatever they have done in the past So you create a new index with new documents and then you can build on top of those and if you configure it or enable it This will be continuously updated. So every 60 seconds basically it's a bit like a batch job every 60 seconds We look like what new transactions came in update all the the documents that we have aggregated and run and update those It will continuously keep doing that for you In elastic not yes, well So you can configure like Kibana is just a fancy UI tool All the hard work is basically done in elastic search always So elastic search is running the aggregation Kibana is just to configure the job and then show you the results But yeah, this here this this is basically the elastic search side when I go to the raw queries Just to show you what is really happening in the background of those Sure. Yeah Hello That's elastic have a choice. Can you do a join between two indexes? so that's actually a Good question because this is so the general answer is there are no joints because any distributed system like distributed joints are not something that is working well, but What is kind of a slide work around? Maybe you've seen something that is called join in the maps thing that I've shown you because they're basically This is an application site join So we don't have a joint in elastic search But of course your application and my example Kibana is the application basically Kibana can basically fetch some data and then fetch some more data based on that But it's more like an application site join It's not in the data store that it's doing a proper join because kind of like the requirement that we have is If it works on one now, that's fine But it also has to work on a hundred nodes and while we could make a join work on one node Distributed join over a hundred nodes is probably not what people expect in terms of response times So that's why we don't have those. Yeah, thank you Any other questions? By the way, if you want stickers, I've thrown the stickers there If you want to take stickers take stickers Final question otherwise Thanks a lot. If you have more questions just come to me and find me. Thank you