 Hi everyone and welcome to the Cube's ongoing coverage of SuperCloud 5, The Battle for AI Supremacy. I'm your host, Rebecca Knight. We are joined today by Ken Exner. He is the Chief Product Officer at Elastic. Thank you so much for coming on the show, Ken. Thank you, Rebecca. Great to be here. So today we're talking about Elastic's new launch of dedicated query language, what it means for Elastic's customers and what it means for the wider development community. Ken, why don't you start by talking a little bit about the impetus for this new language? What was the problem you were trying to solve? What was it that led to the development of it? Yeah, so we're happy to launch ESQL, which is the new Elastic Search Query Language. It is a new search engine and a new search language. So it's a piped query language that allows developers and practitioners to query various different datasets, hold it together all in one query language and be able to construct fields on the fly, to be able to do math and operations, to do joins on different datasets. It's a really powerful new query language that makes it possible for people to do things in one query language that used to be only possible in different query languages. So if you are an observability practitioner, for example, an SRE practitioner, you might have one query language for logs and another query language for metrics and another query language for traces and APM. And you have always had a hard time of how do you pull data together from different datasets and do all of that in one query language. Or we're happy with ESQL that customers can now do that. They have one query language that allows them to work across the different datasets and pull things together and do things that they couldn't do before. And because it's a new query engine that's built directly in Elastic Search, it is native in the engine and it is super fast. Customers have always loved the speed of Elastic Search for search. And now with this new query API and query engine, we have blazing fast performance for queries as well. Okay. So, but it is 2023 and really the buzz is around generative AI. So how does this new language relate to AI? Does it relate to AI? It does. So I hear this question a lot. Thing about AI, generative AI, is it only as good as the foundation is built on? So when you're using cogeneration, for example, you need to build on top of APIs, on top of libraries, on top of frameworks. If you're using a cogeneration tool and say you're trying to build like a thumbnail service on top of like an object storage system, like something that creates thumbnail images, it's not going to build the computer first. It's not going to build S3 first. It's going to start from some foundational primitives and then build up from that. So if you're building the thumbnail service, it's going to build on top of S3 libraries, they build on top of S3. The same thing for these cases we're trying to solve for here. So if an customer wants to do some analytics on different data sets, they want to build it on top of some foundational primitives, and that's what ESQL gives them. It gives them a really robust foundational language for how to pull data from different data sets, whether it's structured or unstructured data, and be able to do different types of aggregations, set alerts based on whatever kind of studying you want to come up with in your language, create detection rules that run on the real time. All these things can happen on top of the data that you're bringing in. And if you want to, on top of that, build a natural language interface, do some other things like that, we can do that too. In fact, we actually have done that with our AI assistant in Elastic. The AI assistant allows you to use natural language to map into Elastic query language. You can just think of how you want to construct a query. I want to pull data from this data source and I want to look at all the hosts that have CPU above 40% that have had a particular type of latency, greater than a two-second latency, and where there have been errors more than 50% over the last hour. Constructing a query like that across different data sets and doing it in natural language is possible now because of the foundational capabilities that we have with ESQL. It really sounds as though it is addressing a lot of the complexity and the fragmentation and the data landscape. Are there any other unique capabilities or features that differentiate it from other languages? You're absolutely right. One of the things that it's doing is it's giving you one language that works across structured and unstructured data sets and does this in a very natural pipe language format. If you think about query languages that are out there, they typically are for structured data sets like SQL or for their unstructured data sets like logs, for example. You had different types of languages that you use based on whether it was structured or unstructured. You use SQL and people love certain things about SQL. It's very readable. It allows you to do joins on data sets. It allows you to do various SQL statements, but if you go to the unstructured world, it's totally different. What we've done with ESQL is brought the best of structured and unstructured into this new pipe query language that allows you to iteratively build up queries and pull some of the best capabilities. Do lookups and joins and unions. Be able to construct fields on the fly. Be able to do math operations on data. All this on unstructured data too. That's a super powerful new capability that data practitioners have not had. Observability and security practitioners have not had the ability to do that before. Your recent announcement says that this is available in technical preview. What does that mean and how can companies and organizations use this language? Yeah, so it's in tech preview. All that means is that it's freely available to everyone to begin using, but because it's in preview, we're still refining the APIs and there are still some known limitations. We want to be able to take customer feedback and then lock down the APIs, and that's what we're going to give GA. GA is when we make a commitment that no API changes are going to exist after that for that version. People are free to use it now. In fact, people, even before this release, have been using it in private preview. We have a large number of customers who have been giving us feedback in private preview for the last half year or so. Customers can begin using it now, whether it's part of the private preview or now in the open public. People can start giving us feedback, and then once it's GA, we'll lock in the APIs and move forward from there. Okay, so what are you hearing from that feedback that you said you've had for about six months now? What are the kinds of things you're hearing from customers in terms of how they're using it and what are they seeing? Yeah, so one of the biggest use cases that we see is security analysts, security analysts that want to do threat hunting in logs and other types of unstructured data sources. What they typically are doing is pulling data from over here and over here, and they're wanting to run queries on top of multiple datasets. And the typical workflow that they go through is an iterative workflow. If you think about a threat hunter, they're thinking, I'm looking for, I want to see all the login attempts for the last hour that originated in a particular IP range, maybe from Vietnam, and I want to see login attempts from accounts that were created in the last day, and they iteratively construct these queries. And what has happened before is they had to do this across numerous different tools and numerous different datasets. So what the ESQL release allows them to do is do that in one language, just build up this iterative query that sort of matches how they think about threat hunting and do this in one language, one paraphernalia language. So we're hearing a lot of feedback about how that is greatly improving the threat hunting experience for security analysts who want to be able to have a tool like this that allows them to pull data in different data sources and iteratively build up their queries. It matches how they think and how they do their job. So we're hearing a lot of positive feedback from that. So it sounds as though it is a pretty seamless integration into existing workflows that for organizations, particularly who are relying on various sources of data. Yeah, if you have logs and metrics, if you have network logs and application logs, and they exist in different indices, you're able to pull these things together and figure out how to manipulate that data in one query language. So whether you are trying to build up a dashboard, this is a query language that can help you build up that dashboard across different datasets, or if you're trying to set an alert so that you can alarm on a particular case, or you're trying to create a detection rule for looking for particular vulnerabilities, you can construct these things all in one language, whether it's logs or metrics or any other type of dataset, whether it's structured or unstructured. So it's quite powerful that these practitioners now have one tool, one language that works across different types of data, and allows them to construct these complex queries fast. So what are your metrics for success, and how are you going to determine whether this is having the impact you wanted to? You used the great example of the threat hunters, and they're able to do what they want to do faster, more efficiently, and more effectively. But what are some of the other things you're going to be looking at to determine whether or not this is having the impact? I expected to replace the existing DSLs and existing query languages that have existed in the elastic community. A lot of them have been sort of bolted on, on top of the search API. This is a new build from the ground up query language that has a new query API. I expected to replace a lot of the existing use cases. So I'm looking to see if people adopt it, people start constructing their alerts and detection rules based on this. So success for me is if I start seeing customers using it and getting a lot of value out of it. Success for me is also customers asking for more features and more capabilities, because then we know that we have something that resonates that customers are starting to get invested in. So far, so good. We're hearing a lot of positive feedback, a lot of requests for new capabilities. People are now kind of, their imaginations have kind of been let wild and are starting to think about what can we do now that we have the ability to pull different data sets and do this all in one query language. They start imagining new possibilities. I love that, because then we get brilliant new ideas from our customers about how to make it even better. Well, I really love what you're saying there, because that really is about bringing this kind of curious, open mindset to innovation and how companies are then able to think even more expansively and ambitiously about new products, innovations and services. How is the introduction of this or how do you foresee it influencing the larger development community? I mean, you said you expect to see other languages maybe lead to rest because of this new one. What kind of impact do you expect to have it on the larger community? Any good product experience? It's a collaboration with your customers. Someone asked me, are we done with our product? And I'm like, no, you're done with your product. If customers are asking for the things, you're constantly iterating. So development of good products is a joint effort between customers and the company producing that. So I always think that we need to have a partnership with our customers, be listening to what they're asking for, trying to get feedback constantly. And with ESQL, that's one of the reasons we had a private beta for the last half years. We were wanting to get feedback and iterate on that. This is why we're doing the technical preview right now. We're wanting to iterate and get feedback before we lock down the APIs. Again, it is an iterative experience with our customers to develop a great product. So I always want to hear customer feedback. I want customers to tell us where to go. And I think ESQL is sort of the product of that collaboration with our customers where they were telling us about their workflow and how they work and how they want to work. And we were trying to build a query engine and a query language from the ground up that matched that workflow, that matched what they were trying to do address some of the limitations that they had run into. So if you're an SRE practitioner and you said, I love being able to use Elasticsearch and build dashboards and set alerts, but here's my problem. I need to work across different tools when I'm looking at metrics versus logs. So we thought, how do we fix that? So the same thing going forward. We're going to continue working with our customers and building more capabilities into Elasticsearch and into ESQL based on their feedback and based on their usage. I'm excited for the collaboration we have with our customers. Well, describing the hand and glove collaboration you had and how it really was a constant feedback loop. Did you face any skepticism early on with maybe some who said, do we really need another language? Sure. We face that skepticism internally too. So over the years, we had built support for multiple different DSLs and query languages on top of Elasticsearch, but they're always kind of bolted on. So we have the original search API and then we built mappings to a query DSL, which is a JSON representation that feeds into the search API. And then we started supporting, to a limited extent, various other query languages like EQL and SQL and KQL and Canvas and painless, a bunch of different, I think there's like nine or ten different query languages that are at least partially supported, but they were always bolted on. They mapped to query DSL, which mapped to the search API. So if we're adding another one of those, yeah, we don't need that. We don't need to do that. But that's now what we're doing here. What we're doing here is creating a new query API that is built into the query Elasticsearch engine that has a new language that is designed from the ground up around how our customers work. And once we realized that that's what we were trying to accomplish, once customers realized that that's what we were trying to accomplish, they were all in. They were excited because it wasn't just another thing that was going to follow the same pattern as before. This was built in the ground up around their workflow, around how they wanted to work, around what they were trying to do. Are there any specific industries or sectors where you see ESQL making a significant impact on its release? I think two sectors in particular. I think security practitioners who use Elastic and Elasticsearch as a security analytics platform, I think they're going to greatly benefit from this because it helps them with their threat hunting. I think also observability practitioners. SREs and DevOps practitioners who are looking for root cause particular issues or creating dashboards, who are setting alerts, they're going to be able to do all that work in one new query language rather than having to work across different tools and different query languages. They're going to be able to have one query language, one powerful query language that works across metrics, logs, traces, APM data, and more. I think those two sets of customers are going to benefit the most. I think there's going to be creative uses even beyond that. Once people realize that you can manipulate different data sets, they're going to start coming up with business analytics use cases. They're going to come up with search use cases, AI use cases beyond what we see. I know that observability and security practitioners are going to love this. You are an industry veteran. You were at AWS for a very long time. You're now at CPO of Elastic. What are some specific AI related innovations and trends that you foresee having a significant impact on cloud services? Not only in 2024. We're rounding out 2023 now, but even in the years to come. So many. Do you have all that? How much time do we have? I'm very bullish on this. I'll talk about some of the things that we're doing inside Elastic. I think it's an example of how we're embracing AI and generative AI and where we see it going, but I'll also talk a little bit more broader. Within the observability and security space, there's a few things that we've seen over the years that are transforming these three areas. One is around auto detection. The whole promise of AI ops was to help customers with detection of issues. We introduced anomaly detection, both at Elastic, but also in the industry. People were starting to use automatic detection as a way to make people more productive and find issues before they stumbled into them. We're now also getting into the area of getting to diagnosis. How do you find root cause faster? If you're an operator or an SRE professional, you spend a lot of time and once you've detected an issue, you're trying to figure out root cause. Same thing with security. You're trying to figure out how do I get to the... Once I have a security issue, how do I figure out where it's coming from and who it is? You're trying to do diagnosis and then you're trying to get to remediation. I think across detection, diagnosis, and remediation, remediating operational issue or remediating the security incident, generative AI is going to have a huge impact. We're investing at Elastic and making detection, auto detection, we're making diagnosis, auto diagnosis, making remediation, auto remediation. I think generative AI is having a huge impact there. We're also at Elastic focused on helping customers leverage the power of LLMs and generative AI on their private data. Earlier this year, we launched SRE, which is a new relevance engine that helps customers create a bridge between their private data and public LLMs. Lots of customers are wanting to figure out how to take advantage of the power of generative AI and LLMs, but they have lots of private data that they don't want to... They want to keep private. So how do they work with large language models? Well, one option is you can hire a bunch of applied scientists and build your own LLMs or do your own training, but that's hugely expensive. So what we've done in Elastic is we've helped customers create that bridge to these large language models, keeping their data private, but figuring out how to pass the right context to an LLM so that they can build generative AI applications. And we're seeing tons of interest and adoption in these capabilities where customers are able to take their private data and start building generative AI applications using SRE and the tools of Elastic. Wow, exciting times. Thank you so much, Ken Exner, for coming on The Cube. Thank you, Rebecca. And stay tuned for more of The Cube's coverage of SuperCloud 5 on your host, Rebecca Knight. You're watching The Cube, the leader in enterprise technology coverage.