 Good evening, hardware nerds, and welcome to the beautiful city of Denver, Colorado. We're here at Supercomputing 2023. My name is Savannah Peterson, joined with my co-host and cube founder, John Furrier. John, what a wonderful night to be talking hardware. Yeah, we're kicking off for four days. This is actually five days if we count today, but this is kind of like election night covers, team covers. We're going to go to the late nights here. To me, the exciting conversation is how networking and interconnects is going to change the game because AI will solve a lot of problems in multi-cloud, but in how HPC, where ExoFlops are going to kind of max, I won't say max out, you see a lot more architectural changes, and we've got a great guest here from Broadcom who's going to break it down for us. Yeah, I'm super excited. We're kicking the night off with an absolute bang with our friends at Broadcom. Hasan, thank you so much for being here on the show. Thank you for having me. Great to be back. How does it feel? The excitement's already starting to buzz. Awesome, it's packed. You know, I got lost while coming here. It's huge. What's your big focus with Broadcom? The VMware story's hitting, but you guys are the middle of all the action. Ethernet, and you guys got the Ethernet Alliance, I saw that big time. Been around for a long time, very good standard, but networking is the hottest area. Actually, that's not talked about much, but everyone's like GPU more than gold and CPUs, but networking, everybody agrees that's the problem. Talk us through why that's so important with AI. Yeah, so, Tim, you know, at Broadcom, we work with a lot of the hyperscale customers. They're great partners. We spend a lot of time understanding their requirements, and it has enabled us to help them build some of the largest clouds out there in the planet. You know, they've been doing AI for the longest time. It's not a new app to them. What has changed the whole environment are all these LLMs? You know, GPT-3 was 175 million parameters. GPT-4 is a trillion plus parameters. So they're huge, they're complex. And so when you're trying to train them, right, you can't do this on traditional compute. You can't do this on a GPU, a single GPU. It's, even though it has thousands of cores, you need hundreds or thousands or tens of thousands of these GPUs. So what connects all of these GPUs? Because they have to communicate in order for the job to complete, and that is the network, right? So we call it, it's a huge computer, and network is the computer for this particular environment. Now, what is the interconnect for this? We believe Ethernet is the right choice. Ethernet has been around, it's the 50th, Ethernet is 50 years old this year, or 50 years young. And, you know, it is based on open standards. It is open, it is a huge ecosystem to innovate on. And then, you know, it offers the lowest cost of ownership, right, you know. And that is why I think there is all this hype around Ethernet. And, you know, we'll talk a little bit about the Ethernet consortium that's also getting formed. We can get to it. What is the AI workload challenge? What makes AI different? It's the gift that just gave the industry, I think, a revitalization. It valid, it vindicates HPC all those years. AI hardware is booming. Do I just bolt on a GPU with an accelerator? Can I have the chips on the same board? What's the architecture? What's the AI workload look like? As you guys look at this 20-mile stair, what's different about these AI workloads and how do you guys look at it? Sure, absolutely. So, to just understand what the AI workload looks like, let's just see how, you know, when you're training a neural network, what it looks like, right? You have a GPUs doing a lot of compute, and once they are calculating a lot of gradients, a lot of parameters, and then these GPUs are exchanging these amongst each other. That's where very high bandwidth communication happens. Then you synchronize, then you iterate. You repeat the process over and over till the model converges. Now, what's different in this case is when these GPUs are communicating, the flows, you don't have a lot of flows. In a traditional workload, you will have TCP, hundreds of thousands of flows. Over here, you have very few flows, and they are huge, very high-banded flows. And they basically are coming on the network at the same time, very synchronized. Another thing unique about this is, you know, these jobs, they don't finish in five seconds, one minute, even a day. They take on weeks, right? And lastly, I think an important thing for these workloads is job completion time is the most important metric, right? When we compare with HPC, people talk a lot about latency. In this case, it's the tail latency, because tail latency is when you get the last flow, right? Then you iterate again, and this is what you'll hear, like, you know, at OCP, Alexis talked about from Meta, how when you're building these out in this communication network is taking 57, up to 57% of the time in training, right? GPUs are sitting idling at that point of time doing nothing. So if you can improve the performance of the network by 10%, you know, we say, look, essentially in a cluster, your network is probably 10% of the cost. It's not the most valuable asset, but it essentially pays for itself if you can improve the performance by 10%. So, at Broadcom, you're touching companies across verticals all over the world. You're just talking about these massive AI workloads. What type of customers are you seeing deploy these, run these? So, of course, right, the biggest focus at this point of time is the hyperscalers, right? You know, these guys are building huge clusters that are out there. You know, we are talking about tens of thousands of nodes, but we are also seeing a lot of AI as a service companies, right, popping up, right? You know, and it's amazing to see their scale itself, right? You know, so they are kind of the second segment that seems to be following this. I would say from an enterprise perspective, it's some of the largest enterprises are also looking at this and deploying this today. But if you look at the journal enterprise, you look at the financials, you look at the insurance companies manufacturing, all of them are excited, they're looking at it, but they are probably in the proof-of-concept stage and how they're going to leverage this and how they're going to use this moving forward. So, I want to talk to you about the hyperscalers. I'm going to, after this show, I'm going to head up to Seattle to see Andy Jassy and Adam Sileski for my pre-re-invent sit-down. And I'm going to ask them a question, a little preview here, I'm kind of blowing it, but I'm going to ask you. The cultural conversation right now is cultural and technical. Cultural mean like, AI's a no-brainer. I mean, when the cloud came, it was cultural. You're doing a startup, you didn't build a data center because you just go to the cloud, you didn't have the cash. So, it wasn't about the technology, it was storage and servers. So, it wasn't about the tech, it wasn't about the culture, the outcome, the workload. But here, with the hyperscalers, if you have to optimize a choice between choosing to optimize for more CPU, GPUs, TPUs, or more better, faster networking, culturally right now, where is that falling? You'd say by reading the news that GPUs are all the rage. They are, but when you start bringing in this end-to-end conversation, workloads and interconnect systems talking to each other, complex distributed computing, this is an interesting optimization question. What would, how do you look at that? Optimize for more power or more networking? So, interesting question and you know, 10 years ago, a decade ago, we had a similar conversation with some of the hyperscalers and they said, you know, this networks are getting in my way. You know, they said, network is not the most valuable asset, it is 10% of the spend, but it gets in the way of getting the best optimization out of my computer at that point, which is CPU. It's exactly the same conversation, it's repeating itself, but this time the workload is different, that's the only difference. And as you are building this out, right, you know, because of what we talked about, you know, the workload uniqueness of AI, you know, you have to deal with a lot of things. You know, how do you manage load balancing? How do you do effective load balancing? How do you do effective congestion management? How do you manage failovers in this environment? And so, if I look at all of this and if I look at some of the data points I told you about, customers are coming us, hey, my GPUs, I spent billions of dollars on these GPUs, but they're sitting idle and your network is the one who is doing all of these things in the middle. Can we improve this? So, absolutely, it's a no-brainer for me, you know, the investment should be on the networking side. A 10%, like I'll again repeat the number, 10% improvement in performance. I'm not even going to ask the VMware networking question because I know you're not going to answer it, so I'm not even going to ask it. They got some pretty good networking over there in the software defined data center. But on the floor here, we're going to interview the CEO of Liquid, they're a partner of yours. You're seeing a lot of the ecosystem, you guys partner with them too, not just hyperscalers. What's the ecosystem going to look like? Because we're seeing a formation of a new kind of architecture, the combination of CPU, accelerated on the same board, networking interconnects, thought about taking the creative way, kind of redesigning, not just on motherboard or backplane, but an architectural network. What is the ecosystem doing? What state of the art, what are you guys seeing, and what are some of the best practices that are coming out of this show? Yeah, so a few things that are happening in this case. The first of all, from an ecosystem perspective, we work with a lot of ODMs, and they are extremely active. We have announced at least two products over the last one year. One is the Jereco A3 AI fabric, the other is Tomahawk 5. So you can go to these ODMs, they're building these boxes packing in 51.2 terabit in a couple of our use. The other thing that we're seeing is, you've seen Nvidia as this DGX box. A lot of companies like Dell and Supermicro and others, they are also building similar kind of infrastructure, which is compute, but you need, there are two kind of things in AI. One is a scale up, where within that box itself, how do you do networking there, and then how you scale up, which is basically, or how you scale out, how are you connecting all of these things together? So what we are seeing is the ecosystem is, we're working with the ecosystem to enable networking within these compute nodes, but also for scale out, build out a lot of this infrastructure, which is based on ODM hardware. And the Jericho was announced, what, this year, earlier in the- About seven months ago, we announced Jericho. What's the feedback been on that? That reduced the bottleneck, is that a bottleneck killer for networking? Is that- Absolutely, right? So basically, again, we have seen all the tests that we have done. Jericho is giving about 12% better performance than even Infiniband, right? What's unique about Jericho is, so there are two schools of thought out there on how do you solve this problem? You know, are you going to do all of this congestion management and load balancing within the switching infrastructure itself, or do you do it at the endpoint? Do you do it at the NIC, or you do it- The NICs are also going to get integrated with the GPUs moving forward, but Jericho is basically, it solves some of these problems in the sense that it's, it does what we call perfect load balancing, right? So every packet gets, you know, sprayed across every link that's out there, so you are 100% utilizing the network. Perfect is a bold word to use in this space. Perfect load balancing. And then, you know, it's basically, it has, it's a scheduled fabric, it does congestion management within the fabric itself, that the sender cannot send till it has credits from the receiver. We have sub-10 nanosecond failover, right? Microsoft published a paper recently which talked about that a 0.1% drop rate can degrade the performance, RDMA performance by about 60%. So sub-10 nanosecond, I was told this number, I'm repeating this here, it's the failover. And you know, with Jericho, you can build one domain, which is a 32,000 node cluster, and then you can replicate it, right? So you can get a lot of stuff. So repeatability, flexibility for deploying it for your partners. Absolutely. It's not a general purpose, then they could use it any way they architect it. Absolutely. And I think, so over there, our ecosystem is active as well. There are a bunch of OEMs who are building this hardware. Well, OEMs, people like Arista who are building this, but there are software desegregation partners, people like DriveNet who are writing software for this architecture. So a lot of activity. I'm a big fan of Fabris. I love the flexibility you put in the architecture again. We believe that this interconnect combination of how chips are going to be managed and at the semiconductor level will offer up options for the ecosystem to build on top of. They need agility, optionality, and architecture flexibility. I got to ask you while you're here, from a Broadcom perspective, what's the most exciting thing for you? And what's the most important story the audience should pay attention to this year? It's Supercomputing23. So what are you most excited about from your perspective at Broadcom? And then what is the most important story happening in your opinion at S323? So from a Broadcom perspective, of course we are excited about the VMware acquisition. This is not the topic that we have going on here. But I would say we are really excited about, networking is some new life has been breathed into networking once again, with all that's happening around this AI and HPC becoming more mainstream. We were just talking about that. What you'll also see is I think in this forum, we formed this consortium called the UEC, UltraEthernet Consortium, which is 10 companies, basically they looked at, we talked about this AI deployments, but three years from now, people are asking us already, how do I get to a million AI accelerators and how do I connect them? So that's where Ethernet is working very well today, but UEC formation, to make sure we solve this problem for tomorrow. And I think at this, we will be talking a lot about that forum here. We will also be, of course we have, we are not the 10 founding members, right? It was three component vendors, it was AMD, it was us, it was Intel, you had Arista, Cisco, NVIDIA, and HP, and then you've got Microsoft and others, but a lot of people get inducted in here. So very excited about this. Again, I'll say Ethernet is 50 years young and it continues to win and all of these players coming together to make sure we move this forward is very exciting. Open standards are key and the most important story happening in here to show that people should pay attention to, just on a macro level, what do you think is happening? I mean, again, I would say. It's AI everywhere. It's AI everywhere. It's certainly an AI party. AI. Well, it's typical people like it. It's all AI hype. Well, that really is kind of happening. I want to ask you a question before we go here, that talks about our 50 years young friend, Ethernet. What is the formation of the UEC, the Ultra Ethernet Consortium? What's the role of that in solving these scaling issues for HPC and AI? Yeah, no. So, you know, just Ethernet itself, if you just start from there, you know, it has won many battles in the past, right? You know, you go back to token ring, FTDI, packet of arsenic, all these technologies. It has, you know, it has the interconnect of choice for front-end cloud networks. And what we also saw was, if you just look at the last year, since GPT-3 came about, we are excited about our two announcements, but, you know, Cisco came out with a 51.2 terabit switch. Marvell came out with a 51.2 terabit. Even NVIDIA, right, came out with a 51.2 terabit fabric. So, this ecosystem is thriving, right? And that is the beauty of the Ethernet. But UEC, as we talked about, was really built. How do we, three years from now, how do we build these million AI accelerator interconnects? And how do we modernize Ethernet to do this? You know, one forum, I would say, one problem that UEC is looking at is, how do you modernize RDMA? RDMA is kind of the, when you're doing memory transfers from one compute node to another, it's the technology of choice. There is Rocky, which is RDMA would converge Ethernet. And so, the team is looking at things like, you know, it does not have multiparting, like we talked about, right? How do you do, you know, load balancing is important? How do you do multiparting in RDMA? How do you do out-of-order packet management? Whenever you're spraying packets, you will have out-of-order stuff. So how do you manage all of this? How do you do selective retransmits? Right now, there is a go back end technology where everything gets retransmitted back, but how are you selective? So, the good thing is, so many people are now coming together, this problem will get solved, it'll be standardized, and the entire ecosystem can use it. You don't have to go to one technology vendor where you're locked in to get this technology, you can get it from multiple vendors. Open standards, this congestion, contention problem has been there, caused a lot of side effects. But if everyone agrees to open standard, they're going to implement it. And you can see that all the big names are out here, so it will get implemented, and you know, others will use it. Hasan, thank you so much for being back on theCUBE. It's wonderful to have you here. John, thank you for your insights and your fabulous questions as always. And thank all of you for tuning in to this, thrilling coverage all week long here at Supercomputing 2023 in Denver, Colorado. My name's Savannah Peterson, and you're watching theCUBE, the leading source for technology news.