 Hello and welcome to this CUBE Conversation. I'm John Furrier, host of theCUBE here in Palo Alto, California. You got a great conversation with Robert Nishihara, who's the co-founder and CEO of AnyScale. Robert, great to have you on this CUBE Conversation. It's great to see you. We did your first Ray Summit a couple of years ago and congratulations on your venture. Great to have you on. Thank you, thanks for inviting me. So your first time CEO out of Berkeley in data. You got the Databricks is coming out of there. You got a bunch of activity coming from Berkeley. It really is kind of like where a lot of innovation is going on data. AnyScale has been one of those startups that has risen out of that scene, right? You look at the success of what the data lakes are now. Now you got the generative AI. This has been a really interesting innovation market. This new wave is coming. Tell us what's going on with AnyScale right now as you guys are gearing up and getting some growth. What's happening with the company? Yeah, well, one of the most exciting things that's been happening in computing recently is the rise of AI and the excitement about AI and the potential for AI to really transform every industry. Now, of course, one of the biggest challenges to actually making that happen is that doing AI, that AI is incredibly computationally intensive, right? To actually succeed with AI, to actually get value out of AI. You're typically not just running it on your laptop. You're often running it and scaling it across thousands of machines or hundreds of machines or GPUs. And so organizations and companies and businesses that do AI often end up building a large infrastructure team to manage the distributed systems, the computing to actually scale these applications. And that's a huge software engineering lift, right? And so one of the goals for AnyScale is really to make that easy, to get to the point where developers and teams and companies can succeed with AI, can build these scalable AI applications without really, you know, without a huge investment in infrastructure with a lot of expertise in infrastructure where really all they need to know is how to program on their laptop, how to program in Python. And if you have that, then that's really all you need to succeed with AI. So that's what we've been focused on. We're building Ray, which is an open source project that's been starting to get adopted by tons of companies to actually train these models, to deploy these models, to do inference with these models, to ingest and pre-process their data. And our goals here with the company are really to make Ray successful, to grow the Ray community, and then to build a great product around it and simplify the development and deployment and productionization of machine learning for all these businesses. It's a great trend. Everyone wants developer productivity, seeing that clearly right now. And plus developers are voting, literally on what standards become as you look at how the market is. Open source has driven a lot of that. They love the model, love the Ray project, love the any scale value proposition. How big are you guys now? And how is that value proposition of Ray and any scale and foundational models coming together? Because it seems like you guys are in a perfect storm situation where you guys could get a real tailwind and draft off the megatrend that everyone's getting excited. The new toy is ChatGPT. So you got to look at that and say, hey, I mean, come on, you guys did all the heavy lifting. How many people you are in? What's the proposition for you guys these days? Our company's about a hundred people, but a bit larger than that. Ray's been going really quickly. It's been, companies use it like open AI uses Ray to train their models like ChatGPT. Companies like Uber run all their deep learning and classical machine learning on top of Ray. Companies like Shopify, Spotify, Netflix, Cruz, Lyft, Instacart, ByteDance, a lot of these companies are investing heavily in Ray for their machine learning infrastructure. And I think it's gotten to the point where if you're one of these type of businesses and you're looking to revamp your machine learning infrastructure, if you're looking to enable new capabilities, make your teams more productive, increase, speed up the experimentation cycle, make it more performance, like build, run applications that are more scalable, run them faster, run them in a more cost efficient way. All of these types of companies are at least evaluating Ray. And Ray is an increasingly common choice there. I think if they're not using Ray, if many of these companies that end up not using Ray, they often end up building their own infrastructure. So Ray has been, the growth there has been incredibly exciting over the, we had our first in-person Ray summit just back in August and planning the next one for a coming September. And so when you asked about the value proposition, I think there's really two main things. When people choose to go with Ray in any scale, one reason is about moving faster, right? It's about developer productivity. It's about speeding up the experimentation cycle, easily getting their models in production. You know, we hear many companies say that they, you know, once they prototype a model, once they develop a model, it's another eight weeks or 12 weeks to actually get that model in production. That's a reason they talk to us. We hear companies say that, you know, they've been training their models and doing inference on a single machine. And they've been sort of scaling vertically, like using bigger and bigger machines. But they, you know, you can only do that for so long. And at some point you need to go beyond a single machine. And that's when they start talking to us, right? So one of the main value propositions is around moving faster. I think probably the phrase I hear the most is companies saying that they don't want their machine learning people to have to spend all their time configuring infrastructure. All of this is not productivity. Yeah, it's the other- It's the big brains in the company they're being used to do remedial tasks that shouldn't be automated, right? I mean, that's- I mean, it's hard stuff, right? It's also not these people's area of expertise and or where they're adding the most value. So all of this is around developer productivity, moving faster, getting to market faster. The other big value prop and the reason people choose Ray and choose any scale is around just providing superior infrastructure. This is really, can we scale more? You know, can we run it faster, right? Can we run it in a more cost effective way? We hear people saying that they're not getting good GPU utilization with the existing tools they're using or they can't scale beyond a certain point or, you know, they don't have a way to efficiently use spot instances to save costs, right? Or their clusters, you know, can't auto-scale up and down fast enough, right? These are all the kinds of things that Ray and any scale, where Ray and any scale add value and solve these kinds of problems. You know, you bring up great points, auto-scaling concept early days, it was easy, get more compute. Now it's complicated. They're built into more integrated apps in the cloud. And you mentioned those companies that you're working with, that's impressive. Those are like the big hardcore, I call them hardcore. They have good technical teams. And as the wave starts to move from these companies that were hyperscaling up all the time, the mainstream are just developers, right? So you need an interface in. So I see the dots connecting with you guys and I want to get your reaction, is that why you see it? That you got the alphas out there kind of kicking butt, building their own stuff, alpha developers and infrastructure, but mainstream just wants programmability. They want that heavy lifting, taking care for them. Is that kind of how you guys see it? I mean, take us through that because to get crossover to be democratized, the automation's got to be there and for developer productivity to be in, it's got to be coding and programmability. That's right. Ultimately for AI to really be successful and really transform every industry in the way we think it has the potential to, it has to be easier to use, right? And that is, and being easier to use, there's many dimensions to that, but an important one is that as a developer to do AI, you shouldn't have to be an expert in distributed systems. You shouldn't have to be an expert in infrastructure. If you do have to be, that's going to really limit the number of people who can do this, right? And I think there are so many, all of the companies we talked to, they don't want to be in the business of building and managing infrastructure. It's not that they can't do it, but it's going to slow them down, right? They want to allocate their time and their energy toward building their product, right? To building a better product, getting their product to market faster and if we can take the infrastructure work off of the critical path for them, that's going to speed them up. It's going to simplify their lives. And I think that is critical for really enabling all of these companies to succeed with AI. Talk about the customers you guys are talking to right now and how that translates over because I think you hit a good thread there. Data infrastructure is critical. Managed services are coming online, open sources continuing to grow. You have these people building their own and then if they abandon it or don't scale it properly, there's kind of consequences because it's a system. You mentioned it's a distributed system architecture. It's not as easy as standing up a monolithic app these days. So when you guys go to the marketplace and talk to customers, put the customers in buckets. So you got the ones that are kind of leaning in that are pretty peaked probably, working with you now, open source. And then what's the customer profile look like as you go mainstream? Are they looking to manage service, looking for more architectural system architecture approach? What's the any scale progression? How do you engage with your customers? What are they telling you? Yeah. So many of these companies, yes, they are looking for managed infrastructure because they want to move faster, right? Now the kind of these profiles of these different customers, there are three main workloads that companies run on any scale run with Ray. It's training related workloads and it is serving and deployment related workloads like actually deploying your models and it's batch processing, batch inference related workloads. Imagine you want to do computer vision on tons and tons of images or videos or you want to do natural language processing on millions of documents or audio or speech or things like that, right? So I would say that there's a pretty large variety of use cases, but the most come, we see tons of people working with computer vision data, computer vision problems, natural language processing problems. And it's across many different industries. We work with companies doing drug discovery, companies doing gaming or e-commerce, right? Companies doing robotics or agriculture. So there's a huge variety of the types of industries that can benefit from AI and can really get a lot of value out of AI. And but the problems are the same problems that they all want to solve. It's like, how do you make your team move faster, succeed with AI, be more productive, speed up the experimentation and also how do you do this in a more performant way in a faster, cheaper, no more cost efficient, more scalable way. It's almost like the cloud game is coming back to AI and these foundational models because I was just on a podcast, we recorded our weekly podcast and I was just riffing with Dave Vellante. My co-host on this were like, hey, in the early days of Amazon, if you want to build an app, you just, you had to build a data center and then you go to the cloud. Cloud's easier, you pay a little money, penny's on the dollar, you get your app up and running, cloud computing is born. With foundation models in generative AI, the old model was hard, heavy lifting, expensive, build out before you get to do anything, as you mentioned time. So I got to think that you're pretty much in a good position with this foundational model trend in generative AI because I just looked at the foundation map, foundation models map of the ecosystem. You're starting to see layers of, you got the tooling, you got platform, you got cloud, it's filling out really quickly. So why is any scale important to this new trend? How do you talk to people when they ask you, what does chat GPT mean for any scale and how does the foundational model growth fit into your plan? Well, foundational models are hugely important for the industry broadly because you're going to have these really powerful models that are trained that have been trained on tremendous amounts of data, tremendous amounts of compute and that are useful out of the box, that people can start to use and query and get value out of without necessarily training these huge models themselves. Now, Ray fits in any scale fit in a number of places. First of all, they're useful for creating these foundation models. Companies like OpenAI use Ray for this purpose. Companies like Cohere use Ray for these purposes. IBM, if you look at, there's of course also open source versions like GPTJ created using Ray. So a lot of these large language models, large foundation models benefit from training on top of Ray. And but of course, for every company training and creating these huge foundation models, you're going to have many more that are fine tuning these models with their own data that are deploying and serving these models for their own applications that are building other application and business logic around these models. And that's where Ray also really shines because Ray can provide common infrastructure for all of these workloads, the training, the fine tuning, the serving, the data ingest and preprocessing, the hyperparameter tuning and so on. And so where the reason Ray and any scale are important here is that again, foundation models are large, foundation models are compute intensive. Doing, you know, using both creating and using these foundation models requires tremendous amounts of compute and there's a big infrastructure lift to make that happen. So either you're using Ray and any scale to do this or you are building the infrastructure and managing the infrastructure yourself, which you can do, but it's hard. Good luck with that. I always say good luck with that. I mean, I think if you really need to do to build that hard in foundation, you got to go all the way. And I think this idea of composability is interesting. How was Ray working with OpenAI for his tech? Take us through that because I think you're going to see a lot of people talking about, okay, I got trained models, but I'm going to have not one. I'm going to have many. There's big debate that OpenAI is going to be the mother of all LLMs. But really, people are also saying that there'll be many more, either purpose built or specific. The fusion and these things come together, and it's like a blending of data. And that seems to be a value proposition. How does Ray help these guys get their models up? Can you take us through what Ray is doing for say OpenAI and others? And how do you see the models interacting with each other? Yeah, great question. So where OpenAI uses Ray right now is for the training workloads. Training both to create chat GPT and models like that. There's both a supervised learning component where you're pre-training this model on doing supervised pre-training with example data. There's also a reinforcement learning component where you are fine tuning the model and continuing to train the model, but based on human feedback, based on input from humans saying that this response to this question is better than this other response to this question. And so Ray provides the infrastructure for scaling the training across many, many GPUs, many, many machines, and really running that in an efficient, performance, fault-tolerant way, right? And so, this is not the first version of OpenAI's infrastructure, right? They've gone through iterations where they did start with building the infrastructure themselves. They were using tools like MPI, but at some point, given the complexity, given the scale of what they're trying to do, you hit a wall with MPI. And that's gonna happen with a lot of other companies in this space. And at that point, you don't have many other options other than to use Ray or to build your own infrastructure. That's awesome. And then your vision on this data interaction, because the old days monolithic models were very rigid. You couldn't really interface with them, but we're kind of seeing this future of data fusion, data interaction, data blending at large scale. What's your vision? How do you, what's your vision of where this goes? Because if this goes the way people think, you can have this data chemistry kind of thing going on where people are integrating all kinds of data with each other at large scale. So you need infrastructure, intelligence, reasoning, a lot of code. Is this something that you see? What's your vision in all this? Take a look. AI is going to be used everywhere, right? It's, we see this as a technology that's gonna be ubiquitous and is going to transform every business. I mean, imagine you make a product. Maybe you were making a tool like Photoshop or whatever the tool is. The way that people are going to use your tool is not by investing hundreds of hours into learning all of the different specific buttons they need to press and workflows they need to go through. They're going to talk to it. They're going to say, ask it to do the thing they want it to do. It's gonna do it. And if it doesn't know what's being asked of it, it's going to ask clarifying questions, right? And then you're going to clarify and you're going to have a conversation. And this is going to make many, many, many kinds of tools and technology and products easier to use and lower the barrier to entry. And so, and this, you know, many companies fit into this category of trying to build products and trying to make them easier to use. This is just one kind of way it can one kind of way that AI will be used. But I think it's something that's pretty ubiquitous. Yeah, it'll be efficient. It'll be efficient to see up and down the stack and we'll change the productivity equation completely. You just highlighted one. I don't want to fill out forms, just stand up my environment for me and then start coding away. Okay, well, this is great stuff. Final word for the folks out there watching, obviously a new kind of skill set for hiring. You guys got engineers. Give a plug for the company for any scale. What are you looking for? What are you guys working on? Take the last minute to put a plug in for the company. Yeah, well, if you're interested in AI and if you think AI is really going to be transformative and really be useful for all these different industries, we are trying to provide the infrastructure to enable that to happen. So I think there's a potential here to really solve an important problem, to get to the point where developers don't need to think about infrastructure, don't need to think about distributed systems. All they think about is their application logic and what they want their application to do. And I think if we can achieve that, we can be the foundation or the platform that enables all of these other companies to succeed with AI. So that's where we're going. I think something like this has to happen if AI is going to achieve its potential. We're looking for, we're hiring across the board, great engineers on the go-to-market side, product managers, people who want to really make this happen. Awesome, well, congratulations. I know you got some good funding behind you. You're in a good spot. I think this is happening. I think generative AI and foundation models is going to be the next big inflection point, as big as the PC, internet working, internet, and smartphones. This is a whole nother application framework, a whole nother set of things. So this is the ground floor. Robert, you and your team are right there. Well done. Thank you so much. All right, thanks for coming on this CUBE conversation. I'm John Furrier with theCUBE, breaking down a conversation around AI and scaling up in this new, next major inflection point, this next wave, is foundational models, generative AI, and thanks to ChatGPT, the whole world's now knowing about it. So it really is changing the game in any scales right there. One of the hot startups that is in good position to ride this next wave. Thanks for watching.