 Good morning and welcome back to Dallas, ladies and gentlemen. We are here with theCUBE live from Supercomputing 2022. David, my co-host, how are you doing? Exciting, day two, feeling good? Very exciting, ready to start off the day. Very excited, we have two fascinating guests joining us to kick us off. Please welcome Pete and Armando, gentlemen. Thank you for being here with us. Thank you for having us. I'm excited that you're starting off the day because we've been hearing a lot of rumors about Ethernet as the fabric for HPC, but we really haven't done a deep dive yet during the show. Y'all seem all in on Ethernet. Tell us about that. Armando, why don't you start? Yeah, I mean, when you look at Ethernet, customers are asking for flexibility and choice. So when you look at HPC and, you know, Infinity Band's always been around, right? But when you look at where Ethernet's coming in, it's really our commercial and their enterprise customers. And not everybody wants to be in the top 500. What they want to do is improve their job time and improve their latency over the network. And when you look at Ethernet, you kind of look at the sweet spot between 8, 12, 16, 32 nodes. That's a perfect fit for Ethernet in that space and those types of jobs. I love that, Pete. You want to elaborate? Yeah, sure. I mean, I think one of the biggest things you find with Ethernet for HPC is that, you know, if you look at where the different technologies have gone over time, you know, you've had old technology like, you know, ATM, Sonic, Fiddy, you know, and pretty much everything is now kind of converged toward Ethernet. I mean, there's still some technologies such as, you know, Infiniband, Omnipath that are out there. But basically they're single source at this point. So, you know, what you see is that there is a huge ecosystem behind Ethernet. And you see that also the fact that Ethernet is used in the rest of the enterprise is using the cloud data centers that is very easy to integrate HPC based systems into those systems. So as you move HPC out of academia, you know, into enterprise, into cloud service providers, it's much easier to integrate it with the same technology you're already using in those data centers, in those networks. So what's the state of the art for Ethernet right now? What's the leading edge? What's shipping now and what's in the near future? You're with Broadcom, you guys design this stuff. Yeah, yeah. Right. Yeah, so leading edge right now, I got a couple, you know, things to show and tell here. We love a good stage prop here on theCUBE. Yeah. So this is Tomahawk 4. So this is what is in production, is shipping in large data centers worldwide. We started sampling this in 2019, started going to data centers in 2020, and this is 25.6 terabits per second, which matches any other technology out there. Like if you look at, say, Infiniband, highest they have right now, that's just starting to get into production is 25.6G. So state of the art right now is what we introduced, we announced this in August, this is Tomahawk 5. So this is 51.2 terabits per second. So double the bandwidth of any other technology that's out there. And the important thing about networking technology is when you double the bandwidth, you don't just double the efficiency. It's actually one of the being a factor of six efficiency. Wow. Because if you want, I can go into that, but. Well, what I want to know, please tell me that in your labs, you have a poster on the wall that says T5 with some like terminal kind of character, because that would be cool. If it's not true, just don't say anything, I just want to, I'll just, I'll just actually shift into a terminator. Well, but so what are the, so this is from a switching perspective. When we talk about the end nodes, when we talk about creating a fabric, what's the latest in terms of, well, the NICs that are going in there? What speed are we talking about today? So as far as 30 speeds, it tends to be 50 gigabits per second, moving to 100 gig PAM4. Okay. And we do see a lot of NICs in the 200 gig Ethernet port speed. So that would be four lanes, 50 gig. But we do see that advancing to 400 gig fairly soon, 800 gig in the future. But say, state of the art right now, what we're seeing for the end nodes tends to be 200 gig E based on 50 gig PAM4. Wow. Yeah, that's crazy. Yeah, that is, that is great. My mind is actively blown. I want to circle back to something that you brought up a second ago, which I think is really astute. When you talked about HPC moving from academia into enterprise, you're both seeing this happen. Where do you think we are on the adoption curve and sort of in that cycle? Armando, you want to go? Yeah. Well, if you look at the market research, they're actually telling you it's 50-50 now. So Ethernet is at the level of 50%. Infiniband's at 50%. Interesting. Yeah, and so what's interesting to us, customers are coming to us and say, hey, we want to see flexibility and choice. And hey, let's look at Ethernet and let's look at Infiniband. But what is interesting about this is that we're working with Broadcom. We have their chips in our lab. We have switches in our lab. And really what we're trying to do is make it easy to simple and configure the network for essentially MPI. And so the goal here with our validated designs is really to simplify this. So if you have a customer that, hey, I've been in Infiniband, but now I want to go Ethernet, you know, there's going to be some learning curves there. And so what we want to do is really simplify that so that we can make it easy to install, get the cluster up and running, and they can actually get some value out of the cluster. Yeah, Peter, talk about that partnership. What does that look like? Is it, I mean, are you working with Dell before the T6 comes out? Or you just say, you know, it would be cool. What would be cool is we'll put this in the T6. No, we've had a very long partnership both on the hardware and the software side. You know, Dell has been an early adopter of our silicon. We work very closely on Psi and Sonic on the operating system. You know, and they provide very valuable feedback for us on our roadmap. So before we put out a new chip, and we have actually three different product lines within the switching group within Broadcom. We've then gotten very valuable feedback on the hardware and on the APIs, on the operating system that goes on top of those chips. So that way, when it comes to market, Dell can take it and deliver the exact features that they have in the current generation to their customers to have that continuity. And also they give us feedback on the next gen features they'd like to see, again, in both the hardware and the software. So I'm just fascinated by, I always like to know, kind of like, what, yeah, exactly, exactly. You start talking about the largest super computers, most powerful super computers that exist today, and you start looking at the specs and there might be two million CPUs. Two million CPU cores, exoflop of performance. What are the outward limits of T5 in switches building out a fabric? What does that look like? What are the increments in terms of how many, and I know it's a depends answer, but how many nodes can you support in a scale out cluster before you need another switch? What does that increment of scale look like today? Yeah, so I think, so this is 51.2 terabits per second. What we see the most common implementation based on this would be with 400 gig ethernet ports. So that would be 128, 400 gig ports connected to one chip. Now if you went to 200 gig, which is kind of the state of the art for the Nix, you could have double that. So in a single hop, you can have 256 end nodes connected through one switch. So this T5, that thing right there, inside a sheet metal box, obviously you've got a bunch of ports coming out of that. So what is that, what's the form factor look like for where that T5 sits? And is there just one in a chassis or what does that look like? It tends to be pizza boxes these days. What you've seen overall is that the industry's moved away from chassis for these high end systems more towards pizza boxes. And you can have composable systems where in the past you would have line cards into the fabric cards that the line cards would plug into or interface to. These days what tends to happen is you'd have a pizza box and if you wanted to build up like a virtual chassis, what you would do is use one of those pizza boxes as the fabric card, one of them as the line card. Okay. So what we see the most common form factor for this is there tend to be two. Let's say for North America, most common would be a two RU with 64 OSFP ports and often each of those OSFP, which is an 800 gig E or 800 gig port, we've broken out into two 400 gig ports. So yeah, in two RU, you've got, and this is all air cooled, in two RU you've got 51.2 T. We do see some cases where customers like to have different optics and they'll actually deploy a four RU just so that way they have the face plate density so they can plug in 128, say QSFP 112. But it really depends on which optics, if you want to have DAC connectivity combined with optics, but those are the two most common form factors. And Armando, Ethernet isn't, Ethernet isn't necessarily Ethernet in the sense that many protocols can be run over it. I think I have a projector at home that's actually using Ethernet physical connections. But what, so what are we talking about here in terms of the actual protocol that's running over this? Is this exactly the same as what you think of as data center Ethernet, or is this RDMA over converged Ethernet? What are we talking about? Yeah, so RDMA, right? So when you look at, you know, running, you know, it's HPC workloads, you have the MPI protocol, so message passing interface, right? And so what you need to do is you may need to make sure that that MPI message passing interface runs efficiently on Ethernet. And so this is why we want to test and validate all these different things to make sure that that protocol runs really, really fast on Ethernet. If you look at MPI, it's officially, you know, built to, hey, it was designed to run on InfiniBand, but now what you see with Broadcom and the great work they're doing, now we can make that work on Ethernet and get, you know, it's safe performance. So that's huge for customers. Both of you get to see a lot of different types of customers. I kind of feel like you're a little bit of a, looking into the crystal ball type, because you essentially get to see the future, knowing what people are trying to achieve moving forward. Talk to us about the future of Ethernet in HPC in terms of AI and ML. Where do you think we're going to be next year or 10 years from now? You want to go first or you want me to go first? I'm sorry, yeah. He feels ready. So, I mean, what I see, I mean, Ethernet, I mean, is what we've seen is that as far as on the starting off of the switch side is that we've consistently doubled the bandwidth every 18 to 24 months. That's impressive. Yeah. Nicely done. Casual, humble brag there. That was great. That was great. I love that. One of the benefits of Ethernet is like, is the ecosystem, is the trajectory, the roadmap we've had? I mean, you don't see that in any of the networking technology. More who? So, I see that trajectory is going to continue as far as the switches, doubling in bandwidth. I think that they're evolving protocols, especially again, as you're moving away from academia, into the enterprise, into cloud data centers, you need to have a combination of protocols. So, you'll probably focus still on RDMA for the supercomputing, the AIML workloads. But we do see that as you have a mix of the applications running on these end nodes, maybe they're interfacing to the CPUs for some processing, you might use a different mix of protocols. So, I'd say it's going to be doubling of bandwidth over time, evolution of the protocols. I mean, I expect that Rocky is probably going to evolve over time, depending on the AIML and the HPC workloads. I think also there's a big change coming as far as the physical connectivity within the data center. Like one thing we've been focusing on is co-packaged optics. So, right now, this chip is all the balls in the back here, there's electrical connections. How many are there, by the way? 9,000 plus on the back of that, everyone. 9,352. I love how specific it is, it's brilliant. Yeah, so, right now, all the CERTIs, all the signals are coming out electrically based. But we've actually shown, we have this actually, we have a version of Tomahawk 4 at 25.6T that has co-packaged optics. So, instead of having electrical output, you actually have optics directly out of the package. And if you look at, we'll have a version of Tomahawk 5, where it's actually even a smaller form factor than this, where instead of having the electrical output from the bottom, you actually have fibers that plug directly into the sides. Well, cool. So, I see, you know, there's bandwidth, there's radix increasing protocols, different physical connectivity. So, I think there's a lot of things, and the protocol stack's also evolving. So, you know, a lot of excitement, a lot of new technology coming to bear. Okay, you just threw a carrot down the rabbit hole. I'm only going to chase this one, okay. So, I think of individual discrete physical connections to the back of those balls. So, if there's 9,000 fill in the blank, that's how many connections there are. How do you do that many optical connections? What's the mapping there? What does that look like? So, what we've announced for Tomahawk 5 is it would have FR4 optics coming out. So, you'd actually have, you know, 512 fiber pairs coming out. So, you'd have, you know, basically on all four sides, you'd have these fiber ribbons that come in and connect to these actual fibers coming out of the sides there. You wind up having, actually, I think, in this case, we would actually have 512 channels, and it would wind up being on 128 actual fiber pairs. Because... It's miraculous, essentially. Yeah, yeah, yeah, yeah. So, you know, a lot of people are going to be looking at this and thinking in terms of InfiniBand versus Ethernet. I think you've highlighted some of the benefits of specifically running Ethernet moving forward as HPC, you know, which is sort of, just trails slightly behind supercomputing, as we define it, becomes more pervasive, AI, ML. What are some of the other things that maybe people might not immediately think about when they think about the advantages of running Ethernet in that environment? Is it connecting? Is it about connecting the HPC part of their business into the rest of it? What are the advantages? Yeah, I mean, that's a big thing. I think one of the biggest things that Ethernet has, again, is that, you know, the data centers, you know, the networks within enterprises, within, you know, clouds right now are run on Ethernet. So now, if you want to add services for your customers, the easiest thing for you to do is, you know, the drop-in clusters that are connected with the same networking technology. You know, so I think what, you know, one of the biggest things there is that, if you look at what's happening with some of the other proprietary technologies, I mean, in some cases they'll have two different types of networking technologies before they interface to Ethernet. So now you've got to train your technicians, you train your assist admins on two different network technologies. You need to have all the debug technology, all the interconnect for that. So here, the easiest thing is, you can use Ethernet, it's going to give you the same performance, and actually, in some cases we've seen better performance than we've seen with Omnipath, than, you know, better than Infiniband. That's awesome. Armando, we didn't get to you, so I want to make sure we get your future hot take. Where do you see the future of Ethernet here in HBC? Pete hit on a big thing is bandwidth, right? So when you look at training a model, okay? So when you go and train a model in ADI, you need to have a lot of data in order to train that model, right? So what you do is essentially you build a model, you choose whatever neural network you want to utilize, but if you don't have a good data set that's trained over that model, you can't essentially train the model. So if you have bandwidth, you want big pipes because you have to move that data set from the storage to the CPU, and essentially if you're going to do it maybe on CPU only, but if you do it on accelerators, well, guess what? You need a big pipe in order to get all that data through. And here's the deal, the bigger the pipe you have, the more data, the faster you can train that model. So the faster you can train that model, guess what? The faster you get to some new insight, maybe it's a new competitive advantage, maybe it's some new way you design a product, but that's a benefit is speed. You want faster, faster, faster. It's all about making it faster and easier for the users. I love that. Last question for you, Pete, just because you've said Tomahawk seven times and I'm thinking we're in Texas, stakes, there's not going on with that. Making me hungry. I know exactly, I'm sitting up here thinking man, I did not have a thinking of breakfast. How did you come up with the name Tomahawk? So Tomahawk, I think it just came from a list so we have a Trident product line, which is a missile product line, and Tomahawk just being kind of like the bigger and batter missile, so yeah. Love this. Yeah, I mean, it's not the stakes. So do you let your engineers do it? You can just name it. Had to ask. It's collaborative. We want to make sure everyone's in sync with it. It's not the Aquaman Trident. Right, right. It's the stake Tomahawk. I think we're good. We've cleared that up for all of you. Armando, it was really nice to have both of you. Thank you for teaching us about the future of Ethernet and HPC. David Nicholson, always a pleasure to share the stage with you and thank you all for tuning in to theCUBE. Live from Dallas, we're here talking all things HPC and supercomputing all day long. We hope you'll continue to tune in. My name's Savannah Peterson. Thanks for joining us.