 From San Mateo, it's theCUBE, covering Scalar Innovation Day, brought to you by Scalar. Hello, I'm John Furrier with theCUBE. We are here in San Mateo, California, Special Innovation Day at Scalar's headquarters. We're here with Steve Newman, the founder of Scalar and Jeff Mathis, a software engineer. Guys, thanks for joining me today. Thanks for having us. Thanks, great to have you here. So you guys introduced Power Queries. What is all this about? Yeah, so the vision for Scalar is to become the platform users trust when they want to observe their systems. And Power Queries is a really important step along that journey. Power Queries provide new insights into data with a powerful and expressive query language that's still easy to use. So why is this important? So at Scalar, we like to think that we're all about speed. And a lot of what we're known for is the kind of the raw performance of the query engine that we've built that's sitting underneath this product, which is one measure of speed. But really we like to think of speed as the time from a question in someone's head to an answer on their screen. And so the whole kind of user journey is part of that. And kind of traditionally in our product, we've provided a set of basic capabilities for searching and counting and graphing that are kind of very easy for people to access. So you can get in quickly, pose your question, get an answer without even having to learn a query language. And that's been great, but sometimes the need goes a little bit beyond that. The question that someone wants to ask is a little bit more complicated or the data needs a little bit of massaging. And it just goes beyond the boundaries of what you can do in kind of those basic, the sort of basic set of predefined abilities. And so that's where we wanted to take a step forward and kind of create this more advanced language for those more advanced cases. I love the name Power Queries, I think it's power and it's got to be fast and good. So that aside, query's been around people know search engines, search technology, discovery, finding stuff. But as AI comes around in more scales and that the system seems to be a lot more focused on like inference, intuitive what's happening. This has been a big trend. What's your opinion on that? Because this has become a big opportunity using data. We've seen log file companies go public, we know who they are and they're out there, but there's more data coming. I mean, it's not like it's stopping anytime soon. So what's the innovation that just gonna take Power Queries to the next level? Yeah, so one of the features that I'm really excited about in the future of Power Queries is our autocomplete feature. We've taken a lot of inspiration from just what your nav bar does in the browser. So the idea is to have a context sensitive, predictive autocomplete feature that's going to take into account a number of individual, the syntactic context of where you are in the query, what fields you have available to you, what fields you've searched recently, those kinds of factors. Steve, what's your take before we get to the customer impact? What's the diff, what makes it different? Where is Power Queries gonna shine today and tomorrow? So it's, it was kind of both an interesting and fun challenge for us to design and build this because you're, we're trying to, by definition, this is for the more advanced use cases, the more when you need something more powerful. And so a big part of the design question for us is how do we let people do more sophisticated things with their logs when they have that use case while still making it something, kind of preserving that speed and ease of use that we like to think we're known for. And in particular, that's been something where step one is go read this 300 page reference manual and learn this complicated query language. If that was the approach, then we would have failed before we started. And we have the benefit of a lot of hindsight. There are a lot of different, there's a lot of history of people manipulating data, working with the sophisticated different and different kinds of systems. So there are, we have users coming to us who are used to working with other log management tools. We have users who are more comfortable with SQL. We have users who really, their focus is just on more conventional programming languages, especially because one of the constituencies we serve are, it's a trend nowadays that development engineers are responsible also for keeping their code working well in production. So they're not experts in this stuff. They're not log management experts. They're not telemetry experts. And we want them to be able to come in and kind of casual, come in casually to this tool and get something done. But we had all that context to draw on with these different history of languages that people are used to. So we came up with about a dozen use cases that we thought kind of covered the spectrum of what would people bring people into a scenario like this. And we actually gamed those out. How would you solve this particular question if we were using an SQL-like approach or an approach based on this tool or approach based on that tool? And so we did this like big exploration and we were able to boil down, boil everything down to about 10 fairly simple commands that pretty much covered the gamut. By comparison, there are other solutions that have over 100 commands. And obviously there's just a lot to learn there. At the other end of the spectrum, SQL really does all of this with one command, select. And it's incredibly powerful, but you also really have to be a wizard sometimes to kind of shoehorn that into... Yeah, even though SQL's out there, people know that, but people want it easier. Ultimately, machines are going to be taking over. You get the 10 commands. You almost can get to the efficiency level of simplifying the use cases. What's the customer scenario look like? Why is design important? What's in it for the customer? Yeah, absolutely. So the user experience was a really important focus for us when designing Power Queries. We knew from the start that if a tool took you 10 minutes to relearn every time you wanted to use it, then the query takes 10 minutes to execute. It doesn't take seconds to execute. So one of the ways we approached this problem was to make sure we're constantly giving the user feedback. That starts as soon as you load the page, you've immediately got access to some of the documentation you need to use the feature. If you type incorrect syntax, you'll get feedback from the system about how to fix that problem. And so really focusing on the user experience was a big part of the... Yeah, people don't factor in the time it takes to actually do the query. Write it up if you have to code it up and figure it out, that's time lag right there. You want it to be as fast as possible. Interesting design point, it's logical, right? Right, absolutely. So Steve, how does it go fast? Jeff, how does it go fast? What are you guys looking at here? What's the magic? So let me, I'm gonna step over to the whiteboard. Chalkboard here, and we'll... So Chalk in one hand, Mike in the other will evaluate my juggling skills. But I wanted to start by showing an example of what one of these queries looks like. I talked about how we boil everything down to about 10 commands. So let's talk through a simple scenario. Let's say I'm running a tax site. People come to our website and they're putting their taxes together and they're downloading forms. And tax laws are different in every state. So I have different code that's running for people in California versus people in Michigan or whatever. And it's easy to do things like graph the overall performance and error rate for my site. But I might have a problem with the code for one specific state and it might not show up in those overall statistics very clearly. So I wanna get a sense of how well I'm performing for each of the 50 states. So, and I'm gonna simplify this a little bit, but I might have an access log for this system where we'll see entries like we're loading the tax form. And it's for the state of California. And the status code was 200, which means that was successful. And then we load the tax form and the state is Texas. And again, that was a success. And then we load the tax form for Michigan and the status was a 502, which is a server error. And millions of these mixed in with other kinds of logs from other parts of my system. And so I wanna pull up a report what percentage of requests are succeeding or failing by state. And so let me sketch first what the query would look like for that. And then I'll talk about how we execute this at speed. So first of all, I have to say what, I've drawn just the relevant logs, but this is gonna be mixed in with all the other logs from my system. And I need to say which logs I care about. Well, maybe as simple as just calling out, they all have this page name in them tax form. So that's the first step of my query. I'm searching for tax form. And now I wanna count these, count how many of these there are, how many of them succeeded or failed. And I wanna cluster that by state. So I'm gonna, clustering is with the group command. So I'm gonna say I wanna count the total number of requests, which is just the count. So count is a part of the language. Total is what I'm choosing to name that. And I wanna count the errors, which is also gonna be the count command. But now I'm gonna give it a condition. I want to only count where the status is at least 500. And I don't know that you can see that, but behind the plant is a 500. And I'm gonna group that by state. So we're counting up how many of these values were above 500, and we're grouping it by this field. And what's gonna come out of that is a table that'll say for each state, the total number of requests, the number of errors. Oh, and sorry, actually I left out a couple of steps, but actually let's draw what this would give us so far. So it's gonna show me for California, maybe I had 9,152 requests, 13 of them were errors for Texas, I had, and so on. But I'm still not really there. That might show me that California had, maybe California had 13 errors and Rhode Island had 12 errors. But there were only 12 requests for Rhode Island. Rhode Island is broke. I've broken my code for Rhode Island, but it's only 12 errors because it's a smaller population. So this analysis is still not quite gonna get me where I need to go. So I can now add another command. I've done this group, now I'm gonna say, I'm gonna say let, which triggers a calculation, let error rate equal errors divided by total. And so that's gonna give me the fraction. And so for California, that might be 0.01 or whatever, but for Rhode Island, it's gonna be one, 100% of the requests are failing. And then I can add another command to sort by the error rate and now my problem states are gonna pop to the top. So real easy to use language. It's great for data scientists digging in there. Practitioners, you don't need to be hardcore coder to get into this. Exactly, that's the idea. Group, sort, very simple commands that just directly, kind of match the English description of what you're trying to do. So then, but you asked a great question then, which is how do we take this whole thing and execute it quickly? So, actually I'm gonna erase here. So you're getting at the speed now, right? Yeah. So it's gonna be like the, how you get the speed actually. Exactly. Speed is good. So. Simplicity to use, I get that. So now speed becomes the next challenge. Exactly. And the speed feeds into the simplicity also because step one for any tool like this is learning the tool. And that involves a lot of trial and error. And if the trial and error involves weighting, and then at the end of the wait for a query to run, you learn that oh, you did the query wrong. That's very discouraging to people. So we actually think of speed really then becomes some ease of use. Got it. But all right, so how do we actually do this? So you've got, you'll have your whole mass of log data, tax forms, other forms, internal services, database logs, whatever. You've got your whole, maybe terabytes of log data. Somewhere in there are the really important stuff, the tax form errors, as well as all the other tax form logs mixed in with a bigger pile of everything else. So step one is to filter from that huge pile of all your logs down to just the tax form logs. And for that, we were able to leverage our existing query engine. And one of the main things that makes that engine, there's kind of two things that make that engine as fast as it is. It's massively parallel, so we segment the data across hundreds of servers, our servers. So all this data is already distributed across all these servers. And once your database, so you guys built your own in-house, okay, got it. Exactly, so this is on our system. So we've already collected, we're collecting the logs in real time, so by the time the user comes and types in that query, we already have the data and it's already spread out across all these servers. Then the first step of that query was just to search for tax form. And so that's our existing query engine. That's not the new thing we built for Power Queries. So that existing, very highly optimized engine, this server scans through these logs, this server scans through these logs, each server does its share. And they collectively produce a smaller set of data, which is just the tax form logs. And that's still distributed by the way, so really each server is doing this independently and is gonna continue locally doing the next step. So we're harnessing the horsepower of all these servers, they each only have to work with a small fraction of the data. Then the next step was that group command, we were counting the requests, counting the errors, and rolling that up by state. So that's the new engine we've built. But again, each server can do just its little share. So this server is gonna take whichever tax form logs it found and produce a little table of accounts by state, this server is gonna do the same thing. So they each produce their little grouping table with just their share of the logs. Got it. And then all of that funnels down to one central server where we do the later steps. We do the division, divide number of errors by total count and then sort it. But by now, here we might have, you might have trillions of log messages down to millions or billions of messages that are relevant to your query. Now here we have 50 records, just one for each state. So suddenly the amount of data is very small. And so the later steps may be kind of interesting from a processing perspective, but they're easy from a speed perspective. So you solve a lot of database challenges by understanding kind of how things flow once you've got everything with the column or databases there. Just give a perspective of like, what if the alternative would be if we, this is like if I just threw in this sort of database and I'm running SQL, trillions of log files. I mean, it's not trivial. I mean, it's a database problem. Then it's a user problem kind of combined. What's order of magnitude difference if I was gonna do the old way? Yeah, so I mean, the truth is there's a hundred old ways now, right? It's a complicated world. How much pain is there out there? If you're gonna, if you try to just throw this all into one SQL server, my SQL or Postgres server on one server, it's just not gonna happen. You've got terabytes of data. And by the way, we're glossing over, the data has to exist, but also it's to get into the system. So when you're checking, am I letting everyone in Rhode Island down on the night before the 15th, you need up to the moment information. But your database is not necessarily, even if it could hold the data, it's not necessarily designed to be pulling that in in real time. So just sort of a simple approach, like let me spin up my SQL and throw all the data in, it's just not even gonna happen. It's not even gonna happen. So now you're sharding the data or you're looking at some other database solution or whatever. It's a heavy lift either way. It's a lot of extra effort taxing on the developers. You guys do the heavy lifting. Okay, what's next? Where's the scale features come in? What do you see this evolving for the customers? So Jeff talked about autocomplete, which we're really excited about because it's gonna, again, a lot of this is for the casual user. They're a power user of JavaScript or Java or something. They're building the code. And then they've gotta come in and solve the problem and get back to what they think of as their real job. And so we think autocomplete. And the way we're doing it, where we're really leveraging both the context of what you're typing, as well as the history of what you and your team have done and queried in the past, as well as the content of your data. We think a little bit like the browser location bar, which somehow you type about two letters and it knows exactly which page you're looking for because it's relying on all those different kinds of cues. Yeah, it seems like this is foundational, heavy lift, minimize all that pain. Then you get the autocomplete, start to get into much more of AI, machine learning kicks in, more intelligent reasoning. You start to get a feel for the data, seems like. Yeah. Steve, thanks for sharing that. There it is on the whiteboard. I'm John Furrier. Thanks for watching this CUBE conversation.