 We have one final session for today before we can go out and enjoy drinks and food and go and see some of the exposition outside. Real-time log analytics game is what we're going to hear about from Kresden Krabzo. Right here, give him a hand. Thank you. Thank you. I'll talk a little bit about myself as these slides come on. I'm Kresden. I am happy that so many people are here. This late in the afternoon, this is the worst slot, so hopefully I can keep you awake. I'll jump a little bit if I see you falling asleep. So I'm PhD in computer science here from Orhus, 20 years ago. Before that I was at NEXT in Silicon Valley, NEXT Computer, Steve Jobs, so I've worked with him for a couple years. And then after finishing my PhD, I've been 20 years, or actually 17, 18 years, building a consultancy called Triforc, I don't know, 700 people now. And it's not that I want to say anything about that, but I felt that I needed to do something new because at heart I'm a technical guy. So I started this co-founded its Humio project with a couple of friends three years ago, and that's all I do right now. So here we go. We've been, there's just one slide, this is the commercial slide, so bear with me for the day, right? This was set up three years ago. We have a very experienced team. We have 37 employees. Most of them are in the U.S. and we've got a bunch of VC-money, so we're doing full speed rocket start-up thing, and it's all-encompassing. And we have some real, these are real reference customers that are using this and production large-scale setups, the ones that I can reference. There's a whole bunch more that we can't talk about. And particularly we're proud of Bloomberg, which is using a really, really large scale. So let's see, what's the reason for this? The reason for this whole product and what it is is that today, my colleagues at Triforc we realized that when you're building these software systems today, they're really complex. They're using Kubernetes, Pass, SaaS, Cloud, many servers, and there's all these layers of complexity that make it impossible to understand really what's going on. And you're sitting in this tranquil development environment and all the software is running down there in the basement, in the data center, and there's complete disconnect. You can't see the software. When I did my undergraduate work, I was sitting at a computer and you could feel if something was wrong, if the disk was being written to, and there was this interconnect between the network and the sound card. So if there was network activity, there was actually this small hiss going on because there was network traffic going on. That feel, the feeling, the hum of the system is completely gone. We're completely isolated from that. It's really difficult to connect with how is the system doing? So the first tagline for the company was feel the hum of your system. We realized this was one of the first large-scale systems we built at Triforc was all the back end for the back end for this large healthcare infrastructure called a blank. For all drug prescriptions in Denmark, go through the centralized system. So it's built for the drug administration. So every time you get a prescription of some drug, every time the pharmacy picks up a prescription, and every time a nurse administers some drug to you, it all goes into the same big-event database. And this was really critical infrastructure that we were building at Triforc. And very soon it was like several hundred servers and integrations to 50-plus different other systems, hospital systems and data sources, et cetera. It becomes so complex. It's impossible to understand what's going on. So we started using these gathering logs to understand what was going on in the system. And it proved really useful. And in a sense, what we're trying to do is very similar to, you know, here's another example of a very complex system which you can't really touch and see and feel. Now we have some rocket or moon mission and you want to understand what's really going on up there. So the way they work with it in the operations room here is you have these dashboards sitting up there that gives you kind of these high-level, high-view, high-level view of what's going on. And then if something goes wrong, a valve or something triggers or one of these graphs looks weird, you go and talk to one of these engineers down there. They know how to dig into the data and how to understand what's going on. So that's the experience. That really works. That really works as a way to understand large complex systems. And actually anyone doing large complex software systems in large scale. As we also heard from Simon Simonson, the network management, the network is a large complex system that you need to understand. You need this kind of thing. Large application deployments with many different moving parts, you need this kind of infrastructure. So that's Fumio. This is kind of the simple diagram. There's all these different systems, they send logs, you put them into a big database of sorts and that's what we built. That's a box in the middle. And you use it for two things, metrics or monitoring for these dashboard, alert, like live kind of things. You want to notice if the latency is too high or if there's too many errors over a given time interval, those kinds of things. And then when something goes wrong, then you need to dig in and see what happened. Okay. So here's an example. Oh, did I, I showed you, was this the one I showed you right now? Anyway. So here's an example of a customer. Very similar to the talk we heard before about network security. Here's a customer doing about a million events a second from 30,000 desktop PCs, a bunch of ADs, various servers and network traffic monitoring. It adds up to around 20 terabytes of data a day. And at this scale, the likes of Elastic Database, for instance, is having a really hard time following along. So it turns out there is really a sweet spot in the marketplace for a product. We realize this. You know, it's something where you can record everything. Basically you want to record a lot because when something goes wrong, the things that go wrong are exactly the things you didn't think of beforehand. So you're always in this situation where you say, I wish I indexed that. I wish I pulled that data out and kind of set up a monitor or a trigger to realize that. But when something goes really bad, it's always the case that you didn't think of it beforehand. Right? So because of that, you want to be able to lock everything or as much as possible. Then it's really convenient to generate metrics out of the logs directly because that's a super easy way to integrate with the range of products because logs are almost always available from database servers to cloud services, et cetera. You can always pull them in. It's really convenient to be able to do interactive searches, to find stuff in your data. And particularly for our use cases, because I was telling you we were in this health sector, it's really important to be able to install it on premises for privacy and security. And you also want it affordable. And if you want all of these, there was no product like this. So that's what we set out to build. And so I can't help myself but talk a little bit about technology, but at least conceptually. If you look at a traditional database, I'll call it an indexing database. It's set up so that you have a relatively low volume. It performs optimally when you have a relatively low volume of ingest. And it's okay to have a high volume of kind of queries, data coming out of it. Because you put in a lot of effort to build these indexes. And you spend a lot of resources on CPU and disk space to store these indexes. So that because there's lots of people asking queries, all the asking searches or queries all the time, and because you put in all that work upfront, it's really, really fast to get the data out again. That's a traditional mindset of what a database is supposed to be. So the opposite would be a no indexing database, where it's super, super, super easy to put in a whole bunch of stuff, but then it's really hard to get it out again. So that would just be just append it to the end of a file as the data comes in. It's really quick. You take the data, put it in the end of a file, and when it's full, you start a new file and you just keep appending. And then it's, of course, it can be kind of cumbersome to find it again. But actually, it's not too far off what you want when you do incident response. Because in incident response, you want to store a whole lot of data, and it doesn't happen that often that something goes wrong in the system. And maybe if you have this stored in a large cluster, it's okay that there's one user, he's asking these questions, it's not 100,000 customers on the front page of Amazon or something like that. There's one user, the operations guy doing the queries, he can use the full cluster to still get a reasonable performance out of that. And that's kind of the idea in Humio. We also play some other tricks that make this bearable. For instance, as data comes in, you can set up these triggers or alerts. You can run real-time queries that run on the ingest pipeline. So as data comes in at high volume, we can still do some certain computations over that, like keep a time window of percentiles of response times, and then trigger on that, for instance. But you could also do historic queries at a somewhat slower pace. So that's the recipe for Humio. You take a fast MapReduce engine, a stream processing engine to do the real-time, searches a real alerting, et cetera, and you add some query languages and data sketches and visualizations, and then that's basically what we built. And our users love it. And so I kind of hinted that it isn't really that fast, maybe, but that's kind of the last part of my talk here, I'll try to argue that it really is fast anyway. Because it's amazing how much hardware is being wasted. So Henry Petrovsky, he's a failure researcher. And one of the biggest failures is all this waste of hardware. We run function as a service, save me. Function as a service, imagine that you have these functions, and every time it has to call another function, you persist it to disk, move it over the network, and read it up from disk again to call another function. How terrible a waste of hardware can you do? Anyway, so there's a horrible waste of hardware. And as it adds up when we compare Humio to our competitors, so these are the main competitors in this data space of on-prem systems. There's lots of players in the SAS field, but we focus on a system that you can install on-prem. So kind of in terms of hardware, to get a similar experience, Splunk is about 3X, and Elastic is like way off, 20X often. And the reason why we're so fast is we cared about this hardware all the way down. So as an example of that, let me just take you through what is an I.O. efficient algorithm. I mean, the core of what we do is just really running a grep, like search for text in a file. So if you take, say you want to search for a text in a gigabyte of data, right? How do you do that? You first read it from disk, the disk is fairly slow. It takes a second to read a gigabyte of data off a modern disk, and then you spend a little bit of time researching, right? So one way to improve this would be actually to pre-compress this, right? So if you compress the data as it comes in, it loads much faster. Well, maybe it takes some time to decompress, but all in all the time, to do the load decompress and search is much faster. And plus it has the added benefit, and this is the major reason, and major reason why you don't need that much hardware, is it also takes much, much, much less disk space. You don't need as many disks. It's just much, much easier to manage. So of course, modern hardware, you have a multi-core system. So say it's a four-core desktop-class PC we're running this on, right? So you load the compressed data, and then in four different cores, we, of course, we run a parallel search on each core, where we decompress and search. So then we're a lot faster. Now we're quite significantly faster, but you can do even better than that, because it's, imagine that you're at your CPU, right? And you're doing one of these things, you're one of the cores, right? You're decompressing, remember that yellow represents a quarter of a gigabyte of data, so it's 250 megabytes, and by the time you're done kind of decompressing a quarter of a gigabyte, definitely your CPU caches are all wasted, right? You lost all the data, because the stuff you decompressed in the beginning is back out in main memory. So instead of splitting it up, this way you split it up in tiny little 64K chunks that you compress individually, and 64K chunks so that you can load it up on the level 2 cache, decompress it on the level 2 cache, and then run the string search on those 64K and then move on. Then we get to, then it becomes much, much faster, because then bottleneck is really how fast can you move compressed data from main memory up onto this level 2 cache. And then on top of all this, of course, all the compressed data we have, typically it's already in main memory, right? Because memory is relatively cheap, you can keep most of the data, but just keep it in the page cache of the file system. So often, you know, you don't even need to load it. So actually, this is 30 times faster than in that eave, grep, that you run from the command line. So if you want to run a terabyte of data, and our customers actually do this, theoretically with these computations, maybe 150 modern cores to do a terabyte a second. In practice, our customers, this is SpaBank. In Norway, they use this for large network security setup. They run through a terabyte a second. And so that's one of the keys, the brute force search, where we really, really carefully engineered it all the way down, right? Instead of doing it the CPU load at query time, you do the CPU load at query time instead of an indexing time. You compress it, this gives you really, really nice ad hoc queries. The other part that I didn't have time to talk about, because now I'm going to be kicked out, is on the way in as data enters, you can also do stream processing in Humio. And that essentially allows you to keep these materialized views in memory of over data ad that has flown in or is currently flowing in. And this services dashboards and alerts. So that's it. That's all I have for now. I couldn't help myself doing some technical stuff.