 From theCUBE Studios in Palo Alto in Boston, bringing you data-driven insights from theCUBE and ETR. This is Breaking Analysis with Dave Vellante. Enterprises are fighting a dual mandate of operating inside a tight IT budget envelope, while at the same time transforming their organization into an AI-first company, balancing macroeconomic headwinds from inflation and uncertainty with driving innovation. It's an exciting challenge for IT decision-makers. To deliver the goods, technology leaders are stealing from other budgets to fund AI, they're trying to find quick wins with pretty unspectacular use cases and in some cases, swinging for the fences with ambitious AI training initiatives to drive revenue from things like advertising of course or complex examples like drug discovery with 10-year plus break-even times. The reality is 16 months into the Gen AI awakening, there are lots of hype and tons of experimentation going on, but success in AI is far from assured. Hello and welcome to this week's theCUBE Research Insights powered by ETR. In this Breaking Analysis, we dig deep into the numbers and look at the macro spending climate. Then we go into specific spending patterns around generative AI. We're gonna look at how budgets are being funded, how Gen AI ROI expectations are shifting. We're gonna look at some common use cases and the adoption of some of the more popular LLMs. And we're gonna close by asking somewhat controversial and sometimes annoying question, is it cheaper to do AI in the cloud or on-prem? First, let's look at how the sentiment on IT spending has changed in just a few short months. This chart shows the results of 12 quarters of sampling, the ETR survey data, IT decision-makers and their forecasts of their annual tech spending growth. The ends on these surveys consistently range between 1500 and 1800 with very high overlap, around 75% of repeat respondents. So as you can see in the upper left is we exited COVID and the Fed started tightening, expectations kept decelerating and finally bottomed when the Fed stopped tightening last summer. Rob Williams, who's the Senior Vice President of Investor Relations at Dell, commented to me on this chart that this probably tracks the two-year treasury yield and he's absolutely right. It's basically inversely proportional to that metric, meaning the rise in two-year yields corresponds to a deceleration in IT spend expectations. And what's notable in this data is IT spending grew at about 3.4% last year. And in the January survey, decision-makers expected it to jump to 4.3%, but as we cautioned at the time, the quarterly growth expectations were very much back loaded. Well, sure enough, if you track the two-year, it's popped up again lately as the higher for longer theme reared its head. And as you see in the red here, the expectations have dropped from 4.3% growth, that was January to 3.4% in the April survey with Q1 and Q2 forecasts also decelerating. So we would expect IT spending to grow at least one, perhaps two points faster than GNP with the new CPI numbers out. We could see GNP forecast and growth expectations growing, but it looks like right now they're coming together with these tech spending forecasts. And with the AI mandate, you would certainly like to see tech spending maintain that one and a half, two-point gap. But the reality is, well, AI hype is in full swing, AI monetization isn't. And so the Fed remains a factor at scene. So given the tight macro, how is AI being funded? And as we've indicated in previous episodes, this data shows us that 42% of customers in a survey of 898 respondents say they're stealing from other budgets. And when we dig into that data, the money is coming from business apps, non-IT departments, productivity apps, no surprise there that GNAI could disrupt, other IT spending like kind of legacy machine learning, analytics and not surprisingly, legacy RPA. Now remember, this data represents percent of customers, not budget amount, but when you look at the big spenders in large companies, the figure jumps 50%. This data shows the results from 224 global 2000 respondents in the survey. So at more than 10% of the G2000, pretty representative set of respondents, the point is because of the macro climate and perhaps other factors that we're gonna discuss, it's not like the CFOs are across the board opening their checkbooks. Now, anecdotally, I will tell you that some CFOs are being very aggressive about AI spending. But again, it's not across the board. I'll be excited to look back five to seven years from now to see those aggressive companies, those ones that are spending aggressively on AI and those forward thinking CFOs. I wanna look back and see how they're faring. The other thing to note is ROI expectations are becoming somewhat less aggressive. People naturally listen to the hype and they think, oh, this AI stuff is easy, which in many cases it is, but it's not as easy to drive measurable results to the point where you can throw off enough cash to fund to gain share and fund future investments, not yet anyway. So ROI expectations are shifting to the right as shown on this chart, meaning they're becoming less aggressive, still inside of 12 months. But as you see, there's also uncertainty on ROI timelines in the far right there. The other thing we see is ITDMs, IT decision makers, they're squirreling away some of this budget and waiting as they try to determine where to place their bets. But as the call out notes, the big spenders at large companies are even more conservative as shown here. And look, why not? Why sign up for a short ROI when you're still experimenting with Gen AI and basically applying chat GPT like use cases to your business? For example, this data that we show here tracks how Gen AI is being used in 1800 accounts. And in no way is this remarkable by any means of what you'd expect, right? Tech summarization is the most popular, the fall by customer chat support and then code generation and writing marketing copy. There's some image editing and design work which is becoming more functional and it's got a low single digits of other which includes some of those more complexion cases like the ones we mentioned up front. Advertising to drive revenue and things like drug discovery. Now these use cases are expensive and they're very training intensive and require access to GPUs which either you're gonna get from cloud providers or those alternative GPU clouds like we've talked about in the past like CoreWeb or Genesis or Lambda. Or you're gonna buy GPUs and do the work on-prem but if you're not willing to commit to spending $10 million, it's a good chance you're not gonna see an NVIDIA GPU for a long time unless you buy it from an OEM. So the clouds are an option or increasingly if you're gonna go to the Dells and HPEs of the world to buy GPU powered servers, you're gonna increasingly have access to those. Now the other column on this data is look to the left. We've highlighted the red in red, the percent of customers not evaluating GNAI and LLMs. You might say, huh, that's what I said when I first saw the data. And while the number has come down quite significantly in looking into this we confirm this is actually the case. These respondents said, no, we're not pursuing GNAI. It is not necessarily that there's no GNAI happening in the company, there probably is but there are a number of folks that we talk to that say, look it's moving too fast right now it's too complicated to pick winners. So we're gonna wait. We're gonna step back, let the storm subside and then we're gonna pick up the pieces by wearing from others. Now the other thing we heard from these folks is shown in the insert that people just don't trust the models right now. They elucidate too much and they're too risky given the edicts or compliance and governance in the organization. You know, to both of these we'd say at the very least you need to start thinking about your AI platform architecture. It's gonna be very different from the way you support your general purpose workloads of CRM and ERP and collaboration software today even if that's in the cloud. So we advise thinking about your AI platform and how to construct that. Look, you're likely a hybrid shop today so how do you evolve that into so-called hybrid AI with a combination of cloud and on-prem? And how are you rethinking your data strategy to support AI by things like unifying metadata or rationalizing disparate data types with semantic layers and so forth? Things are moving so fast. Here's some data from ETR on AI tools adoption. It shows for each AI vendor the percent of customers in the latest survey adopting the platform new. That's the dark blue in the top of the bar. Investing spend, investing more spending in the, that's the light blue, the second layer of the bar. Spending flat, which is that powder blue. Spending less than you did the previous year. That's the orange and churning. Departing from the platform, that's in the red. You subtract the orange and the red from the darker and the lighter blues and you get net score, which is that dark blue line. You see going down to the right, you go from left to right with the number of vendors there. Now what this data is telling us is that OpenAI and Microsoft are, they're like the iPhone and AT&T in the early days with a partnership that's getting a lot of attention and frankly working. Anthropic has made a big move up recently in the last few months in our surveys and ETR surveys to surpass llama in spending velocity. Cohere, you see has momentum. Amazon SageMaker is still very popular for a lot of AI applications. The Gen AI, it was not really well suited for. We talked about that last week on breaking analysis. Google interestingly is closing the gap on AWS when you look at Google's spending momentum. They still don't have as big a presence in the cloud but their spending momentum is closing the gap on AWS. AWS, a lot of AWS's activity is probably through bedrock so some of Anthropic is certainly through AWS. Databricks here is very likely their legacy ML actually highly likely and that's doing very well. It's not DBRX which they just announced that's not in these numbers. Hugging face is also very popular and then you see the pre-chat GPT firms like H2O and others and they're having to evolve their portfolios. Now IBM is interesting here to us because for the first time in a long time we're really excited about IBM and the prospects and how the company, Arvind Krishna, that really has the company on the right track in our view. The data is not as friendly here though. We suspect this because IBM is still cycling through the old legacy Watson which was positioned in places that really shouldn't have been. It was very largely services led but they're ramping up Watson X and its corresponding services around Watson X for data and Watson X for governance, et cetera. But IBM is definitely in the game as is Oracle which is not shown on this particular chart. And look, we sympathize with the complexity of the situation right now. As an example, Databricks announced DBRX, touting benchmarks of an MOE, a mixture of experts model that beats Mixdraw who popularized that methodology. Then, Mixdraw counters like days later with a new high watermark and an Amazon ups its investments in Anthropic but they're building Olympus which they want that's their supposedly codename for their internal LLM which goes beyond their existing LLM and they want that to be better than Claude which is Anthropics LLM. That's what the reports are anyway. Then Microsoft does an AccuHire of inflection AI. So yeah, it's complicated and you got open source and you have closed source and so there's a lot of flood going around. By the way, to think about architectures that serve your business. So that's really what we're thinking here is you want to build your architecture that fits your needs and that may mean focusing primarily on the processes and the people versus the technology which is fine but the folks that are telling us, wow, it's just moving too fast. You could still start with, okay, how should we be rethinking our organization? What does this mean for our processes? The technology, the underlying technology at some point is you're going to figure that out and how to leverage that but think about the architecture of your business so you can compete and you want to think about that now and bringing in the expertise and partnerships to help you build that for your purposes. Now, the last thing we're going to talk about is pasta cloud versus on-prem and everyone's debating this. It's kind of obnoxious because the answer is always it depends that's the answer you're going to get from us today. The cloud platforms are moving very fast and let's face it, that's where most of the monetization action is right now with Microsoft pulling off the open AI coup and they've forced Google and AWS to respond with their internal code reds. And as you see the GPU clouds that we talked about earlier popping up and meta throwing off its open source weight around with Lama and the hyperscalers, you know, they're winning the CapEx wars because they're swimming in cash and so they're buying as many GPUs as they possibly can. And at the same time, they're building their own silicon chips. But then you see a GTC and video conference, there's Michael Dell sitting in the first row getting a call out from Jensen. Jensen says nobody's better at building end-to-end systems than Dell so you want an AI end-to-end system, go to Dell, you'll take your order. Right in the first row was amazing and it's true because Dell has by far the most comprehensive end-to-end portfolio of anyone from laptops to high-end servers but how do you think HPE Lenovo and Supermicro feel about that call out? So HPE just announced that Jensen is gonna be speaking at the sphere with Antonio Neary at HPE Discover. Now the reason I bring this all up is adding to the confusion the cloud players are saying come to us, we have optionality, we have tools, we have our own LOMs, we got a partnership with OpenAI, we got innovation, we have scale, come to us. The alternative GPU cloud guys are saying no, no, no, don't go, don't go to the clouds, they're not built for the AI error, they're built for general purpose computing, they're all about multi-tenant, AI is different. And then the on-prem cloud is saying you're gonna spend a lot of money in that cloud so come over to us. So we got a hold of some data that we found interesting. It's data from a study commissioned by Dell and conducted by ESG, now you have to be warned. This was paid for, paid for study. And you know how these things are, they're gonna be used in marketing. But the person who led the study, his name is Aviv Kauffman, he's very well respected. And from what I'm told, and by the way, I was not told this by Dell, I didn't bother asking Dell that, but I know a lot of people who know this individual who said he would not compromise his ethics and very proud of building models that show, you know, that are fair and defensible. And this guy went to WPI, he's got a strong engineering background and worked in labs, so let's give him the benefit of the doubt. Now, coming to the chase, just to add a little more fud into the conversation, this data shows that doing inference on-prem for a 70 billion dollar, sorry, 70 billion parameter Lama 2 LOM using RAG is far cheaper on-prem than in the cloud. And the point of the study is if you're driving your RAG via token-based API services from the likes of OpenAPI, you're going to pay a price over doing your own RAG on-prem with open source models. And RAG, let's face it, RAG's not that hard to do, we've done it. It's hard to get right and it's hard to necessarily monetize, but this data suggests that doing RAG-based workloads with Dell-powered GPU servers is more cost-effective than using IS on AWS or other clouds. Now, I love the ping-pong matches, whether it's benchmarking or these TCO studies. So the first thing AWS is going to say to this as well, if you want lower cost inference, and I think the study was using NVIDIA GPUs, I think they would say, well, you should use our custom inferential chips. You can only get them in our cloud. And I'm not sure this Dell study did that. I don't think the Dell study used that, but trying to do an apples to apples with NVIDIA chips. So hence the methodology becomes really, really important, but that's what this back and forth is all about. The point is, this is one of those, it depends moments. Cloud company A is going to tell you it's much less expensive to do IT in the cloud than on-prem with all that heavy lifting and the on-prem guys will say, well, that may have been true in 2010, but we replicated the cloud operating model on-prem. And it's a hybrid world. And so the debate goes on, and it is hybrid world, by the way. Now, it may very well be more expensive sometimes to do work in the cloud, but oftentimes the developers in an organization are so in love with their cloud that nothing you can do. It's almost worth that extra cost. There's potential disruption. And if you try to force them to go back on-prem with certain workloads, or there's access to new services or innovation in the cloud that are often better. At the same time, as we all know, cloud bills are times opaque. They're oftentimes very expensive. They're unpredictable. Hence, that's why we saw all this cloud optimization the last couple of years. And moreover, the data you want to use for your AI may not be in the cloud. So in all likelihood, well, the cloud going to continue to grow faster than on-prem, and that's where a lot of the AI, if not most of the AI action is today, are customers, most customers are living in a hybrid IT environment, and that is going to extend to hybrid AI. But I'd like to speak with Aviv and learn more about this study, so I'm going to reach out to him and hope he talks to me. Okay, we've got to wrap, but let's leave with a few thoughts on some of the things that we're paying attention to and some of the barometers that you can look at around enterprise IT adoption. Now, despite all the confusion, one thing is clear. The big money in AI right now is in consumer, and it's in superchips, and big memories, and fast interconnects, and training. And B2C in use cases, they're like no-brainers for AI because the bigger the AI cluster that these internet giants can build, the better AI and ad targeting that they're going to get. So they're literally printing money. But those big complex problems in healthcare and climate and the like, they have very long investment horizons that are going to take a long time to pay back. So they may not see ROI for a decade. But they're going to see it for a decade. Mainstream enterprises, mainstream enterprise use cases are focused on productivity and quick hits that are very chat GPT-ish in nature, quite frankly, document summarization, and ideation, code generation, et cetera. RAG, as we said, is not that hard to do and it allows for domain specificity, domain specific inferencing. There are a lot of experiments going on with RAG today, but the big money use cases, they're not that easy to find. Their RAGs are fun, they're cool, the chat GPT-like stuff is really, you know, catching a lot of attention. But down the road, when you really start driving this into your business, cost is going to be a factor. We think low-cost inferencing at the edge is going to be the dominant AI use case. Now, it's not necessarily going to create monopolies. In total, it's probably going to be a bigger market, but capitalizing on that for one company is going to be a little bit more difficult. It's going to be much more distributed and narrow examples. You're not going to, you're likely not going to see an Nvidia-like monopoly, and Nvidia does have a monopoly, by the way. You're not going to see that in a merge at the edge. The question about Nvidia is how long can it hold that monopoly? We'll come back to that in some other day. But there's going to be a lot of inferencing at the edge. That's the point. And collectively, it's going to drive a lot of revenue, big opportunities, and particularly opportunities for consumers of AI to really drive differentiation in their business. And as we've said for years, it's going to be based, most of it is going to be based on the ARM standard. Now, finally, forward-thinking CTOs and this emerging role of what we'll call an AI architect, they're thinking about their new platform strategy. The waiting for the storm to clear and subside may not be the best strategy with respect to architecture. There are some knowns, like the type of workload patterns AI requires and how training and inference work may have different requirements. And much of this is about governance. It's certainly about your data and your data quality. That's where you're going to get your differentiation and your competitive advantage. And of course, it's about other corporate edicts that you have to comply to. And cost is going to come into play. It always does. Look, remember the math at ROI. It's really simple. It's a benefit divided by cost. And we all know what happens when you drive the denominator toward zero. The result goes to infinity. But the size of the benefit matters, too. So, for instance, 1,000% ROI on a project with a $100 net present value isn't nearly as interesting as a 12% IRR on a billion-dollar net present value. But if it takes 10 years to get payback and get that crossover, you know, a lot can change in 10 years. Confused about what to do with AI? Well, you're not alone. So think about the people effects and their dramatic changes in your business process that AI can bring. Then get going on architecture, platforms, iterative development and learnings from the experiments that you're running today. But don't stick your head in the sand and hope to figure it out down the road or your company may be out of business. Okay, that's it for now. What do you think? Does this data we shared reflect what's happening at your organization? You haven't all figured out or are you struggling to keep up? Where are you placing your AI bets and how are you funding them? Let us know. Okay, thanks to Alex Meyerson and Ken Schiffman on production and Alex does our podcast. Kristen Martin and Cheryl Knight help get the word out on social media and in our newsletters and Rob Hoth as our EIC over at siliconangle.com. Remember, all these episodes are available as podcasts wherever you listen. Just search Breaking Analysis Podcast. I publish each week on the queue research.com and siliconangle.com. You can email me. When I get in touch david.balante at siliconangle.com or DM me at dbalante, I'll comment on a LinkedIn post. Please do check out ETR.AI for the best survey data and the enterprise tech business. This is Dave Balante for the queue research insights powered by ETR. Thanks for watching. We'll see you next time on Breaking Analysis.