 Hello and welcome to this CUBE Conversation Exclusive. I'm John Furrier, host of theCUBE. We have Matt Garmin, SVP of Sales and Marketing for AWS. Matt formerly ran for a long time, the EC2 team, which we know is the compute in the cloud, which has really changed the game, is a core product. We want to continue to get more compute, big part of it. Of course, Matt led that team now. He runs sales and marketing and global services for AWS. Of course, he's a CUBE alumni. Matt is here to talk about the surge in generative AI and Amazon's role in bringing that value of the enterprise and the ISV. Matt, thanks for joining me today and joining our podcast. Great, John, nice to be here. Thanks for inviting me back. Great to see you. I know you guys got a lot going on. I reported the other day that you guys announced the AI initiative and a lot of people were jumping on the Amazon's catching up kind of bandwagon. You guys have been doing AI for a long time. Many have saw that and even Jim Kramer from CNBC publicly walked back his comments acknowledging Amazon's deep work in ML and AI. Let's start out by clearing the air in AWS position, you know, it's history. Andy Jassy always said there's no compression algorithm for experience, you guys have that. Briefly explain the history, the trajectory and the experience AWS has with ML AI. Yeah, happy to do it. So at Amazon and AWS, we've been super focused on AI and ML and have long felt for frankly, 20 years we've been working on the space and have known that this has been and will continue to transform how companies do business. And so like you said, we've had an expertise in this for a really long time. And at AWS in particular, our focus is always on how do we help our customers get the most out of new technologies that come up. And I think recently lots of folks, us included are incredibly excited about the potential that generative AI has to fully transform lots and lots of industries and businesses. And when customers think about that, you know, we wanted to make sure that we don't just toss a brand new technology out there, but we wanna really be sure that customers can leverage it in a safe, effective way that makes sense for their business. And that's how we really think about this space. It's an area that we've been deeply investing in and an area that we feel passionate about will help our AWS customers and all customers all over the world really transform their business. And we think the approach that we're taking in AWS is ultimately how most customers are gonna want to consume and build generative AI into the applications that they run. Yeah, I definitely want to get into those approaches and some of the experiences you've had. But before we dive into some of the questions around the product opportunities, I have to get your reaction to all these conversations on banning, banning AI from the enterprise, Goldman Sachs to Apple, who even just last week banned employees from using chat GPT, even regulation is rearing its ugly head. What's going on here? Why are people freaking out about AI? I mean, it's too early to ban in my opinion or even regulate. What is your position on this? How do you see this playing out? What's going on? Yeah, I think ban is a fun word that people like to use, but that's not really it. I think what you read is that when chat GPT came out, it really inspired and caused a lot of broad swath of people to really understand what the power of AI was. And that's where a lot of us have been working on this for a long time, but it did a great job of really kind of bringing into the public consciousness of what's possible. And so I think you saw a lot of people get really excited and want to jump in quickly. And I think when you look at what some of the banks are doing or what some of the companies are doing, they're not so much banning the idea of generative AI. They're encouraged, I mean, they're putting the brakes on their own teams to be careful about putting their own IP into those systems. Part of how those systems learn, like chat GPT and others, is that when you enter questions in, when you put data into that system, it takes that system, integrates it into what it knows, and then it builds a broader corpus of knowledge that it can go answer questions from. And so a lot of companies are, you know, banning, they're putting the brakes in so that they have the right controls and security in place so that their own IP doesn't leak into those models. And I think that's appropriate. It's in fact, when I talk to customers and enterprises, one of the things that they're most worried about is that they understand that in the future, their own IP and their data is actually what's going to be one of the most valuable and differentiating things that they have going forward. And so what they're putting in place is controls to ensure that they have that right set of controls over their IPs so that their employees don't inadvertently share it into one of these models and it gets kind of uploaded and then available for everybody and they kind of lose that IP. They don't realize what you're saying is they don't realize that they're actually contributing to the revised corpus with their IP which then comes into all kinds of issues around IP rights and releases it essentially. That's right, that's right, exactly it. And so, you know, I think if you're bank number one, you want to make sure that your data doesn't get loaded up into the model and so that bank number two can learn from what you're doing. A lot of great possibilities. I think one of the things that I've observed over the past 13 years covering you guys and 10 years at re-invent looking forward to this year is the makeup of your customers, right? You've had a mix of customers from early startups and enterprise quick adoption and then massive growth, more higher level services. You guys serve essentially every kind of customer at this point in every industry and those customers want different things. And if you look at the enterprises today versus say ISVs even of yester year and today, enterprises are merging with this kind of like super cloud mindset or ISVs just want to do SaaS. They all want different things. What are some of the key differences on how enterprises want to consume generative AI versus say how an ISV wants to consume it, generate AI? Yeah, I think, well, the first thing that you mentioned is spot on. I think everyone is going to want to use generative AI and appropriately so it is a powerful technology that has a potential to help us be more efficient, more effective and really change customer experiences. I think when you think about those differences and how a startup thinks about things or how a large enterprise thinks about things or a SaaS provider thinks about things. You know, a lot of them are not totally different. As you might think, their stages of adoption may be different. I think if you're a startup, you're trying to figure out how can you get out there fast? How can you iterate quickly? How can you get access to some of these technologies that may only normally be accessible to really large companies? And that's one of the things that cloud and AWS enable. And so you see startups like Hugging Face, like stability, like runway, I can go on and on anthropic building on top of AWS because they can get large scale capacity quickly. They can iterate quickly. They can learn and they can grow. So that's where a lot of startups love to use the cloud. And that was, as you know, that's where we kind of grew up from the very beginning as the value proposition and generative AI is no different there. I was talking with Swan. Okay, that's right, interrupt, go ahead. I was saying, when you go look at larger scaled ISVs, it's really not that different of a story. I think one of the things that they love is the ability to scale, the ability to test new capabilities. I think if you look at, you know, large ISVs like Adobe just launched last week new generative AI capabilities inside of their creative cloud. Really cool stuff that these larger established ISVs are doing and rolling out really innovative new technologies and capabilities all based on generative AI. So go ahead. Yeah, I just was just going down the same road I was thinking, which is enterprises have a little bit different needs than say a developer or startup that's growing rapidly. Enterprises might want SaaS like experiences like Code Whisperer, right? For example, or developers wanting like say, bedrock for the building blocks. How do you mix that together? What's your take on that? Do you see that same thing more SaaS for the consumption side? Developers want to build with bedrock. Isn't that kind of where the action is? I mean, where is the, because my question is, do you believe that to be true? And where's the action? Yeah, I'll tell you like, you know, John, I think really our take is there is no such thing as a homogenous customer. Customers all have different ways that they want to consume this technology. Some are going to want to consume it at a package layer. Some are going to want to consume it all the way at the infrastructure layer. And I think that's where AWS really shines in how our product strategy is, is that we want to have capabilities for everyone. For people that want to build their own models, we build our own silicon. And I think increasingly that is going to be a competitive advantage for us to have choice. We have, and for a long time, have been the best place to run GPU infrastructure. And so customers love running large scale GPU clusters in AWS. But we also build our own infrastructure that we think has costs and performance advantages in the sense of training for large training clusters and inferential for running large inference clusters. If you think about SageMaker, it's the development platform of choice of almost every single ML developer out there to do things like make sure that you're doing safe by AI, make sure that you're testing various different models to see what actually works well with your applications. And then Bedrock is providing an easy to use API so that the variety of models, whether you're using, we think that over time, there's going to be a large number of these foundational models that folks are going to want to be able to use for a different set of use cases. And they may even want to combine different ones. And so Bedrock provides a really easy to use API so that customers can combine those. Now, the one thing that I will say is consistent across almost every single customer that wants to use generative AI is that they want to make sure that they do it in a secure, safe environment where they know that their IP is safe, where they can have explainability, where they have as much information as possible on how the model was created. And really that's where our focus is, is how can we give enterprises that assurance that they have the highest performing infrastructure but also the best and most secure platform in order to go build that generative AI so that they know that their data and their IP doesn't leak out to places where they don't control it. On the security thing, how do you ensure that? What's the key value proposition there? I mean, sounds good. Back that up, be specific. What's the security compliance? Is it more regional thing? Is it you guys have with your architecture? What's the security, I guess, how can you seal an approval from AWS? How do you ensure that? I think there's a range of things. For our first party models, we have our own models, which we refer to as our Titan models. And those, we're very careful from a copyright perspective of which data has been used to build that model. And we're very clear about that. So customers know that they can be assured that the data that went to build that model is something that we have the rights to use to go build it. We provide things like open source models inside of Jumpstart. And when you're running on some of those open source models, many of which are becoming really, really powerful. And in many cases are actually outperforming some of the proprietary models today. Customers are able to run those entirely inside of their own proprietary VPC or networking. And so they can run that model. There is, they can isolate that from any sort of external connectivity and know that anything that they use in that model stays inside of that model, stays inside of their VPC. The same with Bedrock, where people, anyone who uses any sort of a tuning to tune Bedrock models, which is one of the key features that we'll have inside of our Titan models. We ensure that that data doesn't leak back into the core foundational model and stays inside of the customer's VPC. So many of the controls that they use for the rest of their enterprise data work just the same for their general AI capabilities. And we think when we've talked to a lot of customers, they've come to trust AWS and our security models. They trust their data inside of AWS. And now when they run their generative models on top of some of that data, we can provide some of those same controls to help them understand how their data stays inside of their environment. I think that's a really great point. In fact, we've been talking about this whole prompt engineering wave where it's essentially a call. I mean, as a prompt is a call. In prompt tuning, that's operational. And then obviously auto autonomous is just software. You mentioned choice earlier. I think that's a fundamental comment. I want to just double down on that. You guys have been known as a company, even Amazon from the early days of selling books choice. Now you guys got a broad selection of general. You mentioned a few first party models. That's your model and open AI has theirs. It's not on AWS. We'll come back to that in a second. Third party models via bedrock, which you guys announced, which is getting a lot of traction. And then the recent wave of open source innovation just in the past like month and a half, you saw a huge surge. You guys got a hugging face out there. Some of these individual models are clearly very more prominent. Get that and important. Looking ahead, when will customers want to use the prominent models that you guys have? And when will they want to use some of these long tail bedrock like products and open source? How will you balance those? Yeah. I mean, our goal is to give customers both the choice to be able to run what's best for their application because the model that's optimized for a financial services customer may not be the one that's optimized for genomics data may not be the one that performs best for e-commerce or images or any of those other things. And that's why stability AI is a great model for images right now, but not for text. And by the way, they'll change over time and they'll add some of those. And so we want customers to be able to pick and choose what the best model that they want to use for the best use case. And that's part of where SageMaker plays a big role. And we make it really easy for customers to AB test things. And in a cloud, you can do that. You don't have to spend billions of dollars to go build your own model. You can leverage some of these others and test if model A performs better than model B, or if some combination of models is actually the optimal one for you. And I think over time, that's largely where people will land is they'll tune and kind of build on top of some of these foundational models and they'll have their own model that they tune and then condense from those. And that's the thing that they'll actually use in production. And we want to make it super easy for them to do that process, but then also cost effective and secure in order to actually use that and scale that out because cost is one of the long things people are looking out in the future and they're worrying about the cost of generative AI is going to be. And so we focus on all areas of that to try to make sure that we can meet all of those concerns and have the best option for them. Whether it's first party Amazon models or models open source or other proprietary ones. Our goal is over time to support every single model out there. That's awesome. And this whole conversation reminds me of early days of AWS when you had the same. Do I build the data center and provision all this stuff or do I put it in the cloud and get instant value, variable, elasticity? I mean, same kind of thing happening dynamic here with the cloud and foundational models. It feels the same. You can stand up your own if you want. Good luck with that or mix and match and code your own. That's right. And look, and over time, you'll see us leveraging generative AI more and more in some of the applications that we make available to customers as well. I think you mentioned earlier code whisper is a great example of that where it's a coding companion but still with that enterprise in mind, right? We're still, we have automated reasoning built in to make sure that you're building secure code. We have the ability to highlight if we're showing you code samples that come from open source, what is the licenses and to ensure that you want to use the code sample that comes from open source. These are, it starts from that fundamental starting with the customer that we do and working backwards. And we really like to think what are the things that customers are really going to care about when we roll these out and our focus, which is a little different than others which is we are laser focused in AWS on how can we have generative AI make our customers successful and a little bit less, we're not distracted by productivity suites or search or any of those other things. We are laser focused on how can we make sure that our AWS customers can take best advantage of these technologies and we start with those use cases and then work from there. So Andy Jets his shareholder letter, he was very optimistic and bullish on this basically saying this is a transformer type Adam Sileski's comments as well and what kind of what you're getting at it reminds me of the old days of AWS early days. You know, Andy's would say, undifferentiated heavy, we automate away the undifferentiated heavy lifting. Well kind of AI can do the same thing for differentiated heavy lifting. What's your reaction to that? Because now it can do both, right? You got the cloud for undifferentiated all the toil provisioning and all that stuff. Now you're seeing AI take on more tasks shifting the human augmenting the human capabilities. So differentiating, seeing a lot of conversations today around how AI can actually automate and differentiate for companies. This is a big part of the refactoring on the business side. What's your reaction to that comment? Yeah, I mean, I think, look, I think generative AI is an incredibly powerful capability that has a chance to make us much more efficient, much more effective. You know, it's not going to replace people anytime soon, you know, I think that's a long way off and a lot of people are worried about that. But you know, every time a new tool or capability comes out that's kind of transformational in what you can do, I think people will worry a little bit about that. But if you think about Code Whisper, Code Whisper is not going to make it so that you don't need developers anymore. It's going to make it so developers don't have to write bespoke code. It's going to make it so that developers can write more secure code, but they can focus on some of that piece that's like, what is the innovative customer experience that I can go deliver for my business and for customers and not have to worry about, you know, the blocking and tackling of necessarily writing code? I think, you know, like the future coding language is probably going to be English and that's okay. You know? Yeah, exactly. But it's going to be saying it in English and then the tools will translate that into code. And so, you know, the expertise may not be understanding the nuances of Java or C++ or anything like that, but that's okay. I don't think it doesn't make, it just changes some of that skill pieces. Now you got to think about the parts of your application that you want to go build as opposed to how you build it. So- Yeah, humans plus AI is better than AI by itself. 100%, yep. And that's going to be like that for a really, really long time and probably forever. I think that's the big thing that we want to get out there is people shouldn't be afraid of it. It's not opportunity. I think it's one of the biggest ways we've seen combined all the other ones and we're going to report it heavily. You mentioned GPUs. I want to jump on that real quick. So supply seems to be a bottleneck. Nvidia stock is all high. And I think they're kind of like hoarding all the GPUs in my opinion, but I won't get into that. There's demand for the training and inference. What do you see as the core constraint in the industry and what does usually have to do to have a line of sight to relieve the pressure? There's more demand. You mentioned you've got the GPU service. How long do you expect it to take to clear this up and get more freed up? You know, I think that's a good question. I think there's a number of constraints here. I think one of the things that's key is that it takes a lot of compute power to go build some of these foundational models. It takes billions of dollars of GPUs, but not just GPUs, servers, networking, data center, power, electricity, all of those pieces, right? And we've been building a lot of those things for a long time. We have the largest GPU clusters anywhere in the cloud. We have the best performing GPU clusters in the cloud. And long term, I think that power is actually one of those things that you have to really think about because these clusters have the potential to use hundreds of megawatts to gigawatts of power. Now, by 2025 we'll be running all of our global data centers on renewable energy, which helps a lot because there's a risk that some of that power causes environmental issues. And so I'm super happy that we made that investment and commitments 10 plus years ago to do that. And that's great. But we're also gonna wanna think about how do we scale those in a bunch of different ways? And I think that's part of where our custom silicon comes in. GPUs are awesome, Nvidia does a fantastic job of building a really good product and they're gonna be super important in this space for a really long time. But we also think there's space for custom design silicon. And we think that things like products like Tranium have the real potential to help customers lower cost over time, reduce the power of footprints and improve performance. And there's a lot of work to get there and there's a lot of innovation that's gonna happen in the industry because of so much focus in this space. But we feel like we're at the forefront of that and can have a competitive advantage for our customers and for our business by having that low cost option for customers that actually in some cases can outperform what GPUs can do. Yeah, I think you'll be a little bit humble there, Matt. You guys have had, I'll give you props to silicon work and the physical layer. You guys have been squeezing every ounce of physics out of it at AWS for years. I've reported many stories on that with James Hamilton, Peter DeSantis. A lot of great work there. But that brings up a good point. You know, AI is all about chat GPT which shows a ubiquity of, if you're always writing a paper for me, blog posts and tweets, but the action of AI is up and down the stack. It's physical layer. Reminds me of the old OSI model back in the day. You mentioned physical. This AI up and down the stack and it's gonna be startups are gonna leverage this not just for the application layer, but there's work to be done. Can you just share your thoughts on the kind of generative AI that's happening up and down the stack? I think that there's just gonna be innovation across the board. I think every single industry, there's gonna be innovation at networking. There's gonna be innovation at the compute layer. There's gonna be innovation at the tool layers. There's gonna be innovation in supporting services like vector databases and other things like that. There's new startups that are popping up every day focusing on different parts of that tool chain. I think all of those things are really interesting. And as we've talked about all the way up to the application stack where there's all sorts of new technologies. So I think it's a technology that can be applied almost anywhere. And that's part of what makes it super exciting and it's an incredibly fast moving space. And frankly, whatever we talk about this month may be totally different six months from now is there's a lot of folks out there innovating. And that's part of why AWS is great. We give people a platform to go innovate. And I'm not the one to guess what all those folks are gonna go build with the capabilities we give them. Other than I know that they'll build some stuff we don't expect and that's part of the fun of it. Well, just our surprise on SiliconANGLE and theCUBE is we stored all of our transcripts for 35,000 interviews in the cloud on Amazon. And we've been using transcribe and other services. We have an index, turns out it's a large language wall. Hey, great, we're turning on a CUBE AI right now. So that kind of never would have been available. How do we not been leaning in? And this is something that I want to ask you because what I'm seeing in my reporting is that there's two types of customers right now on the AI side. There's ones that have been into the cloud and ones that aren't or not, they've been listed in shift but not truly in the cloud. The pandemic showed us if you were leaning into the cloud you had a tailwind, if not you had a headwind. With AI there's a feeling that if I don't lean in I might be caught flat footed like the folks that didn't get into the cloud with the pandemic. What's your reaction to that? Do you hear that? And what do you tell customers? Because it's not like just jump in because you have to. Like there's a benefit for getting in there. I'm hearing that from customers saying I'll put the toe in the water, I'll jump in, I'll play around, I'll explore, discover but I don't want to be flat footed like the pandemic where I didn't have leverage. That's right. I think that you're exactly right. I think getting all of your data and your workloads in the cloud enables you to adjust to changing trends and technologies. And I do think generative AI is one of those that every single customer and company has to really think about how they're going to integrate into everything that they do. And it's harder if your data is not in the cloud. And so almost like a step zero is to make sure your data is in AWS that it's available in a data lake that you can look at that your compute workloads are there that you have your structure around it. And so many of the customers who've already jumped in that cloud journey are in a good place to move fast and others are hustling because they realize that this is capability that just not going to be able to do in their own data centers. There's just no way of doing it. The scale is just not possible. The speed, the technology is moving. It's just not possible to do in your own data centers. And so that I think this is further evidence and impetus for people to move to the cloud quickly. But I also encourage every single customer to be thinking about how generative AI is going to change their business over the next, many years. Or if you got a stack of GPUs, keep those and sell as a service opportunity there. I'll link it. Last couple of questions just to end things. I really appreciate your time. I know you're super busy running operations over there at Amazon field and global services. Fun questions, thought exercises you can answer any way you want. I mean, they got open AI, which is not available on AWS. Anthropic, which is available on AWS. I've been talking to insiders and VC firms and some of the top enterprises. And they all want open. They want choice, okay? Many complain privately. They would like to see open AI run with bedrock. Would you ever offer Sam Alton and lots of customers for open AI via bedrock? Sure. There's no, I would love to show. I really like, I think all customers, I want to have choice. And so I would love to have every model that customers are interested and excited about running in bedrock and AWS. All right, there it is, open. Customers and developers also want open source. So you're starting to see that. There's been a big surge just in the past month and a half. A lot of fruits of the labor from the open source community, really jumping on AI big time. We reported explosion of developer and entrepreneurial innovation and value creation. This is the next area where the startups and the unicorns are gonna, the next drop box is gonna be coming out of these communities. All took advantage of AWS in the early days. You guys got the big models, the long tail of open source models are emerging. What's your view on the mix of the models, the big prominence to the open source long tail? How do you see the mix playing out? Do you have an opinion? You have a visibility of kind of how that might shake out in terms of mix. Yeah, it's hard to say. I think some of, we've seen awesome results from some of the distilled open source models recently. Facebook's llama model was awesome. I think there's a new model that just came out this week that's light on, I think, which is an even smaller model that's outperforming Llama now on the open source world totally trained on AWS. I think there's a lot of this interesting, a lot of interesting innovation that's going out there. I think there's also always gonna be need for these really large core models, too, that help distill some of these open source models and specialized models. But I think it's such a fast moving space. It's hard to say. That's why I think that the choice is so important. It's, you know, there's, anybody's guess as to exactly which horse to bet on. We'd prefer to make it really easy for customers to switch horses if they find the one that they like better later. Matt, thanks so much for your time. Final question, as people ride this wave, it's a big one. If they're not on the wave riding it properly, they're going to be driftwood as we've been saying on theCUBE. What's your advice? There's a lot of change. You mentioned just some of these really good examples of, you know, carbon footprint, which by the way Stanford did come out of the study saying it's worse than anything else, even crypto. So that's a good play for the cloud, for you, congratulations. And, but for startups out there, what's your advice? Because the entrepreneurial track is not the same as it was during gen one. You got to get customers, but scale is a huge thing. We've been saying, how do you see this next gen wave hitting? What's your advice to startups and for companies? There's a lot of change. How do you keep on top of it? What would you advise? Part of what we think about is, is that we think AWS is a great place for startups and all sorts of customers to actually as a channel to get to customers. They're, you know, the vast majority of enterprises and companies out there are running their businesses in AWS, but we're not going to go build the broad swath of innovative new technologies. We'll deliver a lot of stuff, but there's a lot that we won't build and partners are key to everything that we do in AWS. And so we have a lot of programs from marketplace all the way through to some of our channel programs and certifications to ensure that our partners are available for our customers to use in a really easy to use, really easy to integrate way. And so I'd encourage all of them to look at some of those programs that we have in the partner ecosystem and in marketplace, as we're seeing that that's one of the ways that a lot of enterprises want to bring these tools together to be able to use broad swath of things. Awesome, Matt Garman, senior vice president. CSM marketing and global services for AWS. Continuing this next level. It's legit next gen AIs here, super exciting. It is transformative. We love it and we love the change and accelerated towards business transformation, Matt. Thanks for coming on theCUBE, exclusive conversation. All right, thanks, John. Happy to be here. Thanks. Bye. Hey, this is theCUBE. I'm John Furrier, exclusive customer with all the thought leaders and executives in the business, making things happen. Generative AI is the hottest, changing trend happening right now. Obviously, data is at the heart of it and cloud scale. John Furrier with theCUBE, thanks for watching.