 Hello, and welcome back to theCUBE and our SuperCloud 4 event on GenAI and more. I'm Rob Streche, managing analyst with theCUBE Research. Today we're excited to have a personal friend in AIOG, Frederick VanHaren, CTO of HiFens, joining us today live here from our Boston studios. Welcome, Frederick, how are you doing? Good, thanks for having me, Ro. Well, I mean, again, you've been around AI for quite some time before all this hype and before chatGBT really blew the lid off of everything that's going on. You bring a lot of perspective on this as well, and especially from the fact that it comes from an infrastructure perspective up as well. And I think that, you know, the data science part as well as connecting back into that. You know, help us kind of get acquainted with what the history has been and where you've been with this and help us understand what you've been up to in AI, you know, for the last decade or so. Right, I mean, so it's interesting what has been happening. I mean, we always talk about AI as something new. The reality is AI has been around for a long time just like HPC. And I think the big trends in the early 2000s is that we went from a software-centric or code-centric approach to a data-centric approach, which basically means for enterprises that there's software used to be their IP. Nowadays, it's data. And that's a complete shift, not only from how you approach the problem, but also how you deploy, acquire, and do innovative work with hardware. Yeah, I think that what's interesting is, and you know, we talk all the time about, you know, is it becoming data products and data product management and how you take a more data-centric, you know, like you said, that's their product, is their data now. You see a lot of different companies and what they're up to in this space and trying to get started, you get brought in by a lot of the different vendors to help them, you know, get their feet under them, consolidate data into a data platform. What are some of the mistakes that people are making out there that you just, you see and you go, okay, this happens over and over again? Yeah, it's an interesting question. Mistakes happen at all different levels. I mean, the first level is really where AI is becoming a commodity in the sense that a lot of enterprises want to do AI or have to do AI and they start from a corporate IT perspective, right? So the cultural approach and the hardware approach and the methodologies are significantly different. So one big thing we see is corporate IT people or organizations kind of doing the same thing as they always used to do. I'll give you an example, somebody might say, well, I understand I need to buy a GPU but I have an old server lying around, I'm gonna put that GPU in my old server and then I'm gonna do some AI, right? And then they quickly realize that first of all, it's not compatible and that there's a lot more going on. It's not just a GPU perspective problem, it's a data problem, it's a network problem, it's a latency problem. Some of those issues aren't really not new, it's just that they come in in a new context and people have challenges with those. The challenge of course for enterprises is once you start it's very difficult to go back and fix it but everybody is at a different stage. We see people acquiring GPUs and then say, now what, we have other people that do the opposite, right? Where they do a lot with CPUs where really they should be using GPUs. We see a little bit of everything but it's challenging in every organization kind of reinvents the wheel if you wish. And maybe that's a better way to say it is that a lot of organizations try to reinvent the wheel. Yeah, no, I think that from that perspective what we hear and we've talked to as part of this SuperCloud 4 event and we've been talking to a lot of end users and I think part of it is they see a lot of hype and they're being told, hey, go do this or their CEO is getting behind the saying we're going to embrace this but we're gonna do it in a safe way. And that kind of leads to your comment about the data and things like that. A lot of time what I'm seeing is that there's a data cleansing or data quality issue in there, garbage in, garbage out. What are you seeing with that and what are some of the challenges that you're seeing with companies trying to deal with that? Right, it depends if what they're trying to process is something that is already well-known like a vertical market that's well-known or if it completely is something new. If it's not, if it's completely new the challenge is there is no good data yet or at least you don't know what good data looks like. So you will start compiling and collecting as much data as you can. I can just remember in the late 90s where when I was in the speech business we actually drove around in cars with tape recorders trying to record our voice at different speeds. I mean nowadays we laugh at it and say that data was horrible but there was no other data collected from within the car. So that data at that point was very useful to us. And I think a lot of organizations go through that perspective, meaning they collect data and trying to figure out what's good, what's bad. If you look at the more advanced AI organizations they don't have a lack of data, they have too much data. And the challenge with too much data is you don't know the value of the data unless you process that data. And processing data comes at a decent cost, right? I mean buying GPUs, they're not cheap, they're long wait lines for acquiring those GPUs. So in the end data management becomes your key focus, right? For at least for the organizations that are relatively mature. I mean I really could write a book about data management and data challenges but it's not easy. It's not like I'm going to collect a lot of data and I'm going to be done. It's collecting a lot of data that is targeted to the domain you're trying to figure out. It's not because you have data collected from within a car that vendor A will be able to use that for vendor B. Well I think what's interesting about that is, and again it goes back to the garbage in, garbage out and knowing not only the quality of your data but where it came from, the lineage and what is the business value of that data as well. And I think that the two don't seem to always go hand at hand or almost never. And it seems like a problem that we've had where you either do ETL or ELT and one way or another you're trying to figure out how do I optimize for cost or performance and that seems to be a common theme within AI is what is the business value and mapping that to the use case as well. Right and you brought up a good point. I mean data lineage comes up a lot and it comes up from different angles. One of them is bias. If you have a model you kind of want to understand what data went in there. But at the same time you want to use data lineage to figure out if the data you used, you were able to extract the value you wanted and if not maybe it's worthwhile to look at different types of data. And those are kind of challenges that organizations have to deal with but when you look at organizations that want to do AI or have to do AI because nowadays you see a lot of organizations start in AI not because they want to, because they have to. It's a time to market concept, right? If you and it becomes extremely challenging if you have a lot of money it's easy for you to buy a lot of hardware. It's not easy for you to buy a lot of good data. Right, yeah and do you see people saying well I want to do it in-house versus take it to a cloud or to one of the hyperscalers because they're looking at it and going it is considerably cheaper to train internally closer to my data and it's just legacy where they've had the data internally versus trying to fight the gravity to move it up to a cloud. Right, I mean to answer the question we have to split the AI question in two pieces. One is training and inference. So training is really where all your data sits and typically depending on the organization they might decide on premises for security reasons or maybe cost reasons, right? The public cloud is great but something 24-7 can turn out to be really costly. If you think about the inference of the production side that's where I see the public cloud really shining because the flexibility and only pay for what you use concept is really key there. And so the answer is not simply AI is good on premises or AI is good in public cloud. I think you do have to split it and then of course there's also where your workload sits or where your data sits, right? On the edge, right? So what do you do on the edge? How do you extract that data? But the public cloud and inference slash production is definitely my favor. Yeah, I think we've talked about this. I think, again, the edge is a really interesting piece of it because I think a lot of people are trying to figure out maybe I have sensor data, car data, what have you that's moving around or and or does not have a lot of compute out there? And they're trying to make decisions there and do you end up with a hub and spoke model where you're pulling some data back but you move inference out to say a co-location like an equinex or digital realty or something like that closer to your sensors. And are you starting to see people where they're doing their model training back so they have to bring some amount of the data back to train the models and get it up and running and then they push out the inferences as close to the edge as possible? Right, so I do see two areas. One is the data ingestion, right? So imagine a video camera filming a security in a building, right? So you could consider that at the edge but it doesn't make any sense to send 12 hours of nothing happening in the firm. So you want something at the edge that gives you the good data only being exported to the data center. And then from a data center perspective, I mean, that's really where you can work with the data and create this flexibility and scale to process that data as you wish and how fast you want it, right? Yeah, no, I think that makes total sense. And I think that to me, when I see people organizations talking about AI, I always ask, okay, what's the use case or what are you trying to solve for? Are you seeing people finally starting to wrap their heads around why they're using AI or are they still being pushed to just go fast, get something with AI up and running? Right, I mean, everybody wants to get up and running as fast as possible. The reality is, and even when I'm being asked to speak at conferences, it's really about basic AI, demystifying AI, the concepts behind it that it's not voodoo but explaining, there's data behind it, there is math behind it, there is scale behind it. Why do we need GPUs? What's happening to the market, right? So we all know that every three years, the amount of data doubles, what do you do with that? And so if today it takes you a month to build a model, you're gonna know that in three years, it's gonna take you two months to build that model. On top of that, the algorithms, the software we use to do all the math, those become more complex every year times 10. And so if you put those together, it's almost a race against the time, but most people really are looking for better understanding what AI is, right? I mean, no surprise that all the accelerator hardware vendors are telling you, you know, buy our stuff and the more you buy, the better it is. The reality is that GPUs or FPGAs by themselves are not gonna solve your problem, right? I think there is a definite need, certainly now in the situation right now, is to explain what AI can do for you, but also explain what AI can't do for you. Yeah, no, I think that is a big key. And I think it's starting simple, starting small, understand your use case, then grow how you're using that. If it fits or not, you have to make that determination from a business perspective. We've been talking to a number of different companies where one friend in particular that I talked to was starting their CFOs organization was starting with an LLM. And part of this was they do 10Ks and 10Qs and quarterlies and all kinds of different reporting for the SEC and stuff like that. And they wanted to, it's a form, it's pretty normalized, but the way that they do their calculations to fill that form is kind of proprietary to that company. So they were building on-prem their own LLM for the CFOs organization. They had to be very secure. What are you seeing from, or what are some of the considerations, I guess, for organizations to think about when they're starting down this journey from an infrastructure perspective in particular? Right, I mean, LLMs are a good example. I mean, it used to be that organizations would have to build their own models and they actually would deploy their own models, right? And so I think now we're entering a period where training is being done, or let's call it pre-training is being done, like an LLM that understands what the English language is, what words you expect to see, and then bring your own context, right? And so if we go technical a little bit, that we call that transfer learning. So you take a neural network, which is pre-trained, you cut off the tail end, and then you create your own little small neural network and then you glue it together. And this is really where AI is being applied today is not everybody has to build a large language model from a trained perspective. You actually can do that, right? There is Lama nowadays, I mean, there is Bard, there's so many different options. And I think that's a trend that the LLM people brought forward, but I see that in other industries too, where it's more the application side, if you wish, as opposed to the need to build a large language models. I'm not sure how many organizations build full blown large language models today, but I think it's a handful, right? It's not like when I, as you know, I used to work at New Orleans Communications, we would have to do everything ourselves. We had to build the models, we had to deploy the models, we had to retrain the models, we had to do everything ourselves. Nowadays, you can get, you know, you can go to a website where you download the large language model and just build your own application. Yeah, I was going to say, I think hugging face with the amount of open source large language models or models in general that keep coming, it's something, some ridiculous thousands a month of new models show up in there or something like that, which is kind of crazy when you think about it. Yeah, if you think about it, I mean, like what I said earlier in the conversation is that it used to be where the source code was the IP. I mean, if you think about it, hugging face and others, they give it all for free, right? So the source code is almost insignificant. It's open source, you share, everybody improves it, but the data is still king of the hill, right? That's what it's all about. Yeah, and I think, you know, just to bring it back to that, I think that, again, it's one of these that people may or may not have data because I think, you know, having both of us have dealt with storage in our past lives and, you know, you got to a point where you were spending so much on storage and you had to determine, okay, do I tear the storage? Do I age out? Do I just expunge it completely after a certain time? And then there's actually regulations where, you know, certain regulations and sent you to get rid of certain data because it becomes a liability to the corporate. How do you think that companies are gonna be able to kind of deal with all of that data management at that level? Yeah, I think it's a big challenge. I think organizations, some organizations already have a, I wouldn't, I don't know how would you call it, like a CDO or like a data officer or something like that. I think a lot of organizations will go towards something like that where they have somebody who understands that concept and at the same time, data is not something you per se have to own, data can also come from your customers, but also I see a lot of open source data nowadays, right? If you want to build maybe a large language model or data for self-driving cars, there are public data libraries out there. So it comes with a whole slew of different problems, but I do think that data will become more embedded in organizations and better understood than something like we just need a lot of data. Right, yeah, I think it was Bloomberg and has one for financial services and obviously it's slanted towards their view of financial services and I think Stanford actually has a healthcare one or somebody does or is out there, so that makes total sense. And I think hopefully the people watching got a good view of where they can get started and what kind of infrastructure to get after. You don't have to buy hundreds of thousands of GPUs. I think that's a takeaway is to look at that and I think also I'm last thought, I mean, as people get towards inference, you start to see that they can run on CPUs for inference. I mean, that's what we're seeing is smaller models. Once the model's trained and you're just using it for inference, you can push it out usually depending on the size. Right, I mean, historically, at some point before the GPUs, I mean, training and inference was CPU, both CPUs and then there was a trend where training was more and more GPUs and inference still CPUs. I think we're now at the stage where CPUs are being pushed out on the inference side in favor of GPUs, but it depends, right? If we talk about the edge, nobody's deploying a large GPU cluster on the edge. But yes, definitely, I think the CPUs had a good run. They're still effective in certain areas, but I think the accelerators in general are kind of winning the battle. Yep, that makes sense. Well, thank you for joining us today, Frederick. I really appreciate it. I always learned something when we get chatting and we could probably sit here for another hour and talk. I know that, just a way that both of us talk. So it's great. So thank you for coming on board. Thank you, great conversation and thanks for having me. Excellent. Well, and remember, you can stay here and stay up to date with all things AI by visiting siliconangle.com. Thank you for watching theCUBE and this episode of SuperCloud 4, the leader in high tech enterprise analysis and coverage. We're here for you. Keep tuned.