 So welcome everyone to ask the experts panel for the machine learning and artificial intelligence track. Today, we're hoping to discuss how open source technologies and the communities behind them are driving new innovations that are shaping the future and resulting in solutions that meaningful impact to the world. Our panelists will be providing insights into this concept of how data centers are changing in response to the amount of data that's being generated and processed and utilized. Questions on things like edge computing and AI and machine learning are very much welcome here and appreciated. We'll also be talking a little bit about how Red Hat is working with open source communities to develop these cutting edge technologies that are integrated into your open source stacks. Yeah, great. So with that, I'd like to introduce our panelists. Our first panelist is Marcel Hild. Marcel? Oh, you wanted to introduce me. So I should introduce myself. I'll call out your name and then we'll introduce yourself. So yeah, Marcel Hild, I'm dialing in from Germany on the internet on that beautiful switch-like platform. And I'm managing a team of data scientists and software engineers in the office of the CTO at Red Hat. And I just started working on AI like two years ago. So I'm not really an AI expert, but given my background in software engineering and systems administration and my focus on applying AI to these domains, I think I'm a good crossover person to see both sides of the game. Because AI, just for the sake of AI, is pretty boring. And systems operations, just without the help of AI, can be really tough. So why not combine the both? And that's the topic that I've been working on for the past two years, something called AIOps. Very cool. Next up, we have Sanjay Arora. Thanks, Anish. So I'm also in the AICOE in the office of the CTO. And most of my work is more focused on machine learning for systems. So one of our big projects is tuning network cards, for example. So this is pretty low level systems analysis that includes both what I would call classical data science. So analyzing log data, understanding things like the details of the TCP protocol and seeing what's getting reflected in the data, as well as more black box machine learning solutions like reinforcement learning for learning policies. Great. Thanks, Sanjay. Finally, we also have Eric Orlundsen. Hi, I'm Eric. I also work in the COE. And I'm part of what we've been calling forward-deployed engineering. And a lot of what we do is helping to enable both customers and also Red Hat field people in what it means to do application development for intelligent applications on OpenShift. So essentially, machine learning workflows and DevOps on OpenShift, I think that's very, very important for a lot of people today, figuring out how to get started with these workloads. I guess I should introduce myself as well since I'm the moderator. So yeah, my name is Anish Astana. I'm also an engineer in the AI Center of Excellence, and I'm working on the Open Data Hub. Really, that operations side, which Marcel mentioned, and that's kind of my focus and interest. So I'll start things off with some freebies. So anyway, I'm going to take these. Can you guys explain to me what the difference is between artificial intelligence and machine learning? Is there a difference? All of us would have something to say about that, but just relax. I'll start. So artificial intelligence is technically broader than machine learning in the sense that all machine learning is a kind of artificial intelligence. But there are forms of AI that really aren't ML. I think back to the 80s with expert systems. They were a kind of artificial intelligence that weren't about algorithms learning from data. They were typically humans encoding complex interacting systems of rules, where the behavior emerged out of those. But it wasn't learning from data in the way that was done, at least. Yeah, I completely agree. And just to add a bit more to that, if you look at if someone asks you to define AI, and you said, OK, what does artificial mean? What does intelligence mean? One can define it to say, can I write programs? Can I write code or algorithms that solve tasks that involve perception of some kind? So it could be, of course, classically it's computer vision or it's speech processing, natural language processing. But really, at an even deeper level, it is reasoning. Can I get a computer to prove that there are infinite prime numbers, for example? That's an old proof. It's a classic proof. Can computers do that? And like Eric said, there's really a priori, no reason one would say, well, let's look at data. You could very well say, yes, I'm going to come up with a very smart algorithm with some rules and with some propositional logic that says, just try and combine these maybe on a tree of some kind or a search, and you can prove these things. Machine learning or what's also often called statistical learning is exactly what Eric said, which is the idea that, well, let's not bother with trying to figure out these rules. Let's give you enough examples. Let me give you examples of images or let me give you examples of proofs in some computer readable format. And let's focus on algorithms that can infer patterns from the data instead of algorithms that can solve our tasks directly. So of course, we now take it for granted that machine learning is the dominant way of solving AI tasks. It doesn't have to be data centric, but that works very well today. And I would add to that the catchy thing of AI you would use when you try to get some funding for your business nowadays, or if you want to attract salespeople to listen to you. And if you talk to an engineer, you probably talk about machine learning to get them interested in the tools that you're doing. And if you're talking to the real data scientists, you probably talk to them about statistics because in the end, what we're doing is steroids. So like Eric said, this AI field has a really long history. And it was even back in the 80s when we tried to build up expert systems. And I think today we're really using advanced statistics, or as somebody said, regex on steroids to do a really, really common and narrow task of identifying some patterns that humans are not capable of identifying and then actually solving this task really, really good. But is that intelligence? You could argue, right? So I would really say, let's define what intelligence is. And if we can come up with an artificial intelligence, maybe we need to rethink how we approach this field anyways today. Because right now it's really more about at least in the commercial world at solving narrow tasks for getting stuff done and growing your business. I'd like to harp on that business aspect a little bit. I've seen some of your talks in the past talking about AI ops. Can you talk to our attendees a little bit about what that really means and how you see that helping companies? Yeah. So when I started working on or trying to understand what AI ops means was, OK, let's use AI to fix operations. But we pretty soon hit a wall there because we can't just fix operations with throwing data over the wall and applying AI to this. You don't get interest from the operational folks because they are pretty happy right now with the tooling that they have of identifying patterns and coming up with thresholds. And you really need to convince the AI folks to get an interest in operational problems. So right now I think it's a grab-back for commercial providers to sell their ops tools in order to just prefix it with AI. And you get more eyes on it. And you get more funding. And you get more visibility, basically. But then a lot of these tools are basically just a little bit of pattern analysis and time series. And you still need to do the ops thing manually in an operational approach. So I'm right now seeing it more as DevOps, the DevOps movement where we combined or brought the developer mindset and the operational mindset together. So developers using some tools from ops to set up environments and the ops people using tools like Git and other things from the development world to build out their tool belt and bring up their capabilities to another level. And if we stack some AI on top of this, I think we can answer some questions better and solve some problems more elegant than with just the tools that the DevOps people right now have. So I think it's more an educational thing. How do I get Jupyter notebooks into the hands of the operational person or to the developer person? And at the same time, also getting AI folks interested in operational problems. Right now, they are pretty much focused on identifying cats in images or identifying Alexa. Although Alexa really seemed to have dementia right now. She's getting worse and worse, at least in my own. I want to verify on something you just said too, which is that a lot of people think of artificial intelligence as like, oh, I'm turning over the whole thing to some algorithms. And what you talked about earlier where it's like, well, you got some pattern recognition algorithms presenting you with some results. But there's a human in the loop that's like translating that into actions. And so at least to call it, I don't know, AI assist more than full AI. But there's a huge continuum there where you automate certain things just to make the human component of the decision making easy. It's not closing the entire loop with a bunch of code all the time. Yeah, I think a big disservice has been done to the work of most data scientists or in the financial world, it's quants, right? Or actually many data driven scientists. A big disservice has been done by this idea which is that you can get a data set on Kaggle or some image data set like Marcel is saying, cats and dogs. And now you have a static in the sense you have a notebook and you just sequentially write your code and you get an accuracy of 80 something 90% and you're done. That's machine learning or data science. And at least in my experience, the most successful data driven projects involve so much back and forth with between the data scientists, the software engineers, and the domain experts. And in our case, of course, we are lucky that the software engineers and the domain experts are the same. But as a very concrete example, one of the projects that we are working on with Boston University, it's literally two people who have a lot of experience in systems engineering and me as a data scientist. And we literally sit on a phone call sometimes for six hours, eight hours, not in one go, but in one day. And it's literally looking at the data and these guys can say, I know what the TCP protocol does. That doesn't make sense. And I'm sitting here saying, well, maybe I can use a decision tree to do this and extract some patterns and analyze that. And most real data science projects have that collaboration between the domain expert and the data scientists. And it's the same thing with the AI ops world. It's really understanding what the operational person deals with every single day, what are the issues they encounter. And initially, machine learning or statistics or even very simple mathematical rules just help them improve their workflow by a percent here by 5% there. The fancy reinforcement learning policy gradient thing comes closer to the end. And one cannot do that without really understanding what that operational person deals with every single day. So would you say, well, within a given project then, would you say that the domain knowledge itself is more important than the data? This may be a loaded question asked by someone who doesn't know what he's talking about. I mean, it depends. So I would say this. For many projects, data and domain knowledge take you 90, 95% of the way. You can get away without knowing machine learning most of the time. And it's only in a few cases where I think you can, I completely understand why computer vision that has been revolutionary to have these deep neural nets doing this. And there are many tasks around the world from things like you're exercising and it's your form correct, something as simple as that to maybe sorting fruits or something. I can understand in those cases why you get such a big impact from just picking someone else's models. But I think in our world, sometimes having a hard-coded rule does take you 80% of the way and that's completely OK. You don't have to put a model in. So I would say they are definitely more important than the machine learning bit. But there are certain projects and they're not the vast majority of the projects where getting someone with a quantitative skill set, whether it's machine learning or physics or math or statistics, can really make a big difference. And in some ways, we are working on that cutting edge. Even if people don't believe that, what AIOps is doing is at that cutting edge. There are some companies that have automated, I'm sure at Google internally, they have many tools that do automated anomaly detection, better load balancing, things like that. When you're working there, yes, it helps to know linear programming well. It helps to know stochastic processes well. But the vast majority of cases, I would say, domain expertise and good data take you a long way. Yeah. I think you definitely need the domain expertise because it's not this throwing data over the wall yet, right? So once we really have general intelligence or we can deal with unsupervised learning problems much better way, maybe then AI can kickstart us better. But right now, the data scientist needs to understand the domain knowledge and that's only possible by talking to the domain expert. And at least in that's what I'm, yes or no? Yeah, and the only other thing I would add to that, Marcel, is you need the domain expert, of course, to make sense of the data many times. But you need the domain expert to tell you what problem to solve. It's like me showing up in a hospital in an oncology unit and saying, give me all your scans. I'm going to do something with this. And maybe I'll say, oh, you know what? I'm going to automatically find tumors or something. And maybe the oncologist will say, that's not useful to me. I like looking at this myself. And our error rates are very low. And it takes me five minutes. Now I know nothing about medicine. And I picked a bad example because actually that's a use case for machine learning, automated tumor finding. But an oncologist still looks at it. They don't know the scan. But in our world, if I show up to the AI ops world, certain problems jump out at me. And they might be completely useless to the operational frontline person. They might say, I don't care about the anomaly detection on this one metric. So I think they really need to, it's a balance. The domain expertise has a bunch of problems they would like to solve. A subset can be solved by data science or machine learning. And you need both people to identify that subset. Yeah. I think it also brings up a skill set issue, which is that successful data scientists are people who are good at that kind of interaction. Being able to reach out to domain experts and talk to them and actually understand their concerns. And it's not always easy to like pull out what's important, you got to be able to chat people up and iterate with them. And you really have to have like that kind of interpersonal skill set as well as the technical to do the data science successfully. Yeah. And I think that's the great times we're right now living in because data science becomes easier to apply to certain problems. It's a little bit like comparing a sampler language to C and then JavaScript, right? So getting into the software development world now is much, much easier than like 20 years ago, right? Because we have so many easy onboarding tools and online courses, et cetera. And nowadays I can apply K-means clustering without being able to implement K-means clustering. But if I'm grasping the intuition and I can do that was just watching a five minute YouTube video to understand what K-means actually is doing. And then I can prepare some data and with some lines of code, I can apply it to a dataset. And I think that's the educational part. So I think some of the mission is also to take away the magic from AI and from advanced statistics and machine learning and make it ubiquitous and usable by also the average software engineer. Yeah, I'm interested what you guys think about this because there's like two sides to that. One is, yeah, the democratization that's happened here has been really powerful. On the flip side, occasionally if you've been in the position of actually implementing these algorithms, you have certain intuitions about like their failure modes and what kinds of tuning parameters and what they're gonna do on certain kinds of data that you don't always get if, you know, you did what you described is, it's like, I don't know, it's become very easy to get some kind of useful result. But having like gone into the guts of the code and the algorithm gives you certain kinds of insights that can be useful and I'm curious if you guys see that kind of trade-off going on. I mean, I struggle with this question because like my first introduction to programming was in a physics context. And these are, you know, molecular dynamics and you're running these simulations. And that's the first time when you work with that or data, you see the difference between a compiler emitting an error saying, you didn't declare the type or something like that. At least I can fix that, right? I can, it identified it for me. But logical errors are harder and errors in anything with data, any scientific computation, the problem is like you said, Eric, it would emit some number. And unless I understand the context and what I should expect, it's very hard to know if this is the right thing or the wrong thing. And in the physics case, it's easy because you spend your studying physics and then you say, okay, when I plot this quantity, I know what to expect roughly and you can do all the checks and balances. When you do it with, so when a software engineer does it with data, they might get fooled by something that the algorithm is doing that they don't understand. But they at least understand the data so they can say, well, I clearly, the act packets that I'm getting on my network clearly should not be 400 bytes or something. And the data scientists can get fooled because we don't understand the domain. So we say like, sure, the act packet can be four kilobytes, it's okay. And I don't know a solution to that except to say they should work together. Which is... And I think it's a matter of interface design, right? So a very well-defined language prevents you from making errors like syntax errors and the type errors, et cetera. And I think the same is also should be true for scikit-learn modules, right? So if you get back a certain number, it should prevent you from interpreting that number in a really wrong way. And also coming up with best practices, how to interpret these results and feedback loops. It's, again, an educational thing. And you can make this easier by designing the interface to these models or these things where you throw in the data and get your numbers back just more meaningful. I do think the interface can help a lot in terms of just at least controlling the, for lack of a better word, the ranges of the numbers that come out, for example, right? Or a silly example, this key means I say negative five clusters, and of course it says, are you crazy? But it gets fuzzier when it's, it's like, you know, like Shrey's talk earlier today where he was clustering and systems and you see five clusters versus you see 10 clusters. And I would have zero idea, maybe that's a bad example, but you can, I don't have a good example in mind right now, but maybe there's a case where you should see five clusters and maybe because of the way you featureize your data or something, you see 10. And that's something I think there's no general rule or interface that can capture that because the big part that's changing all the time is the data that's going into your algorithms because the domain's very so much. And there I agree with you Marcel, it's an educational process where at the end, the most important thing is having that scientific mindset of being very, very skeptical and doing a lot of checks and saying, am I fooling myself or is this a real thing better than my data? And maybe I'm saying this because of my engineering background. So I saw myself growing better at software development by reading just more and more books and practicing all over again. So it's test-driven development and that was way better than just writing spaghetti code when I grew up, right? So applying these best practices to software development made me doing fewer mistakes. So maybe the one question could be asked, is there, can all these best practices that we know from software engineering or from operations, can they also be applied to machine learning and AI problems or is there a different set of best practices that we are still learning given that AI is such a young field and maybe in 10 years from now, these problems just don't exist anymore that we have five clusters versus 10 clusters because we have, I don't know, feature-driven engineering for AI or cluster-driven engineering. That's definitely, it's only true in a course. I mean, just the phrase ML workflows on OpenShift or cloud-native development for AI applications is just those, just those phrasings, is basically assuming the idea that software, existing software development practices apply equally well to the software we do with artificial intelligence as a more traditional software. And it's a huge socialization task because a lot of, some data scientists are fairly hit to that, others definitely are not. And it takes more than just showing up at a customer and saying, oh, we're gonna do some DevOps on machine learning. It's like, they've got a stack and we may be able to point out all kinds of like process issues as the way they do stuff but they're used to doing it a certain way and telling them to like, they're gonna have to like do a bunch of churn to like change all their tooling and log into some platform. Intrinsically, they don't wanna do that because for better or worse, they know how to use the tools they got. You know, I think getting data scientists to realize what they're doing is still software development and like Marcel said, test driven development, repeatable builds, all these things that we think of is kind of bread and butter because we're like immersed in it, people still need to get their minds around it. Yeah, absolutely. You have a question from the audience. Shrey is asking, do you think that data scientists should be diving into only into problems that they're willing to be domain experts in or should the onus be on the domain experts to start learning how the algorithms are returning the results they're returning or why they're doing that? And where do you think that balance lies? My personal opinion is the data scientists needs to be a little bit more willing to interface with the domain expert. I mean, sometimes if they're asking you to do something which is not possible or defies the laws of mathematics or something, you may need to like explain something like that. But generally speaking, the domain experts typically, for no other reason politically, the domain expert is typically the customer. And so like you're there to serve the customer and you try to meet them more than halfway whenever possible. That's just how it works for me, usually. Yeah, I would agree with that too. I would, I mean, I can only speak for data scientists but I do see, I mean, one major reason I came to Red Hat initially was because I was interested in systems engineering. So I do want to understand how the Linux kernel scheduler works. And that helps a lot because I want to do it. I'm happy to read a paper or look at the code or something. I have been in situations in my past careers where I'll give you a concrete example. I worked on an investment banking team for some time as a data scientist. And they wanted to do all kinds of things with IPOs and companies raising money and I did care, not one bit, right? But I'm there, I have the state I'm supposed to do something. And psychologically it gets in your way because really what I should be doing is going to the people who actually help companies go public and say, what do you do? What do you look for? What's the process? And when you enjoy the domain or you're interested in it, you go and find out all this information that makes you better at interpreting and working with the data. So it definitely helps with the caveat that of course you won't enjoy every single problem that you solve as a data scientist, right? Every single data set. But it's still, I think the onus is on us to go and seek these people out and at least try our best. You will meet a lot of domain experts who don't care, who say, look, my job is, I'm happy in my job. I've been doing this for one year, five years, 20 years, 30 years. I think all this is hype. This is fake stuff. I don't want to talk to you guys. You can help me go away. That happens. And then you say, okay, let me see if I can find someone else. And I think Marcel would agree with that too. Absolutely, I think so if you take one definition of intelligence, which is the ability to adapt to unknown or previously unseen situations or problems, I think the same is true for this balance, data science and domain experts. So if we have domain experts that are just ignoring the data science part, at some point, well, you can't adapt to it anymore. And you're so old school and you're just out of the game, right? And the same from the other end. So you always have to adapt and you have to learn from what's going on on the other side of the wall, on the other side of the river in order to progress, right? Nowadays, if we have a, we're just looking for coders that also understand operations, right? So nowadays, the huge ballpark, the huge market is for DevOps people. Certainly you still have assembler programmers and prolog programmers and programmers that don't understand anything about the ops side, but it's a shrinking domain. And the same for the old school ops people that don't care about the development side. So I think the same will be true eventually for data scientists that don't understand the subject matter domain. Certainly they will write the latest and greatest research things, but I think where we apply it in the market at customer problems, you must understand both sides. So speaking to that a little bit, right? Like the research side of it, a lot of these machine learning models, algorithms, and techniques have been around for a really long time, right? Like since the 80s, 70s, whatever decade you pick, what's caused that recent uptake in like the interest in machine learning, right? Like go back two decades, maybe it was because I was a little younger back then, but like you didn't hear much about like things like Alexa or grocery stores that let you walk out without having to meet a cashier, right? Or these kinds of things. What's really helped drive that innovation if those core algorithms are still the same? So my one sentence take is that the advent of cheap use and cheap processing and large matrices, multiplication being ubiquitiously available made those algorithms actually work at scale. But that's just my naive. No, and you can, I agree with Marcel 100% on this. If you look at the first, I think major breakthrough in what's called deep learning now, in this century was, I think in 2011 or 12, was something called AlexNet, which was a neural network, a convolutional network that was trained on something called ImageNet and called much better performance than all the classical computer vision results still there. And the innovation was driven by GPUs, by this large data set, ImageNet was a large one that was collected, I think by people at Stanford. So having that large label data set, having a GPU and plus epsilon, where epsilon is small tricks, right? So there were things that in hindsight look trivial, like if anyone in the audience has trained a neural net, there's this activation function and it always used to be sigmoid or a hyperbolic tangent. And they said, let's make it easier. Let's take this thing called a value rectified in your unit. When you look at it, you say, mathematically, I can teach this to maybe a sixth grader or something. But that small thing made it much easier to train deep neural nets, right? It's much faster to execute the instructions in that one function. So it's small things like that, but the 2000s were a period of uncertainty in the neural networks community. They were trying things and I often go back and read papers from, it's less, it's like 10 years ago, 15 years ago. And it's surprising how much uncertainty there was. How do I do this? How do I train this thing? Now we can all go get someone's repo, clone it and run it. And it wasn't that black boxy then. So I think those are, and data too, just having large labeled data sets made a massive difference. That was big. Nothing but agreement with, you know, Marcel and Sanjay are saying, I think that in addition, a couple other, I don't know, social changes had an impact. I mean, just, you know, we talked earlier about the fact that, you know, these things are all open source tools now and anybody can download them. I mean, when I was first doing this, you know, if you wanted to be doing AI, you like downloaded some papers and or went to the darn library and photocopied some papers and, you know, spent a few weeks coding some stuff. I mean, that's how you got anything to work. Now, you know, you just get a Jupyter notebook, pip install some stuff and, you know, like you say, look at an example or two on the internet and like go to town. So like the barrier to entry for smaller, you know, shops with less manpower to actually do anything in the space has gone up enormously just because of the open sourcing. I think that, you know, Sanjay talked about data. It's more than just even the size. Oh, that was huge for deep nets. It's just that everybody has data now. That was not always true. You know, it's like everybody's online. Everybody's got telemetry of some kind that could tap into or they've got stuff that's digitized that didn't used to be. So just, you know, you can't do machine learning without some data stream. And just the fact that now everybody basically has access to that kind of data plus tooling is cause an explosion in the number of people who just have the wrong materials to do ML. Yeah. So one question to Eric and Sanjay, do you see a next leap? Oh, where do you see the next leap? And is that right around the corner or? I think that for me, my guess, my best guess is like the next huge leap would be in like synthesizing what we've been calling machine learning, which is very, like you say, very statistical in nature. You start a lot of data at something, you know, and a model that you know how to optimize something for, which is very good for some things, but not others. And maybe like unifying that with reasoning, which is an awfully loaded term, but just being able to take that kind of low level perceptual stuff and combine it with being able to like, you know, make informed decisions from it automatically. Yeah. I mean, the short answer is I don't know, but the thing that at least I'm very excited about is reinforcement learning. And it's partially because I've been staring at it a lot the last few months, but so there might be many other things that I just have no idea about, but there have been very interesting results doing things, all kinds of things, but I'll give you a couple of examples. You can define a very simple instruction set and abstract instruction set. These are the allowable operations, things like swapping elements in an array or adding numbers or that stuff. And you say, can I train an algorithm to sort an array? Simple, right? Quick sort or something, but except you're not telling the algorithm what to do, you're saying, here's an array, here's a neural net that can generate moves and most of the time, of course, by chance, it won't just get a sorted array, but you have a reinforcement learning algorithm called policy gradient in the loop and it discovers essentially quick sort. A similar thing was done with neural net architectures. You have a neural net that generates an architecture for a neural net, trains it on an image net, gets an accuracy and that feedback is sent back to this agent and it improves the architecture. And that one, today you need a lot of compute power to do these experiments, but you can still beat the state-of-the-art performance on that data set because it finds very novel LSTM cells and all these things, novel architectures. The problem with reinforcement learning is it's where, in a very crude sense, it's where deep learning was 10 years ago, where if you actually work with these things, you have to tune things, you have to really dig deep into the internals and say, why is it not working? What's going wrong? Which is how deep learning used to be 10, 15 years ago. But my hope is partially because it doesn't need labeled data sets, it just needs something that can say yes, no, good, bad, or even give you a reward, which is a continuous score. If one can get reinforcement learning algorithms that can train on small and smaller data sets, you could use it in a lot of places where we just can't use supervised learning today. Okay, so I don't know where the time went, but we are pretty much out of time now. We have one more question from the audience or two more. The first one is, what helps you keep up with the pace at which AI, ML tools and methodologies are evolving? What's kind of your preferred source of information? I guess it's 8% generic, more so, but. Eric? You're primarily, primarily it's, I'm gonna call it conferences, but just venues where you interact with people. The social aspect of keeping up with what people are doing is super important. This is gonna sound completely ridiculous, but I cultivate certain people I follow on Twitter and doggone, if you can't find pretty interesting new stuff going on that way. So it's amazing enough that you can do more than just see cool pictures of dogs with social media. I think the short answer for me is I don't even attempt, and I should clarify that. What I mean is, I go to hack and use a lot, and I have an aversion to Twitter, but there are many leading machine learning resources on Twitter posting what they're doing. But at some stage, what worked for me is to say, let me pick two or three problems that I care about, right? I'll work backwards. And so that problem says I need to go deep into some reinforcement learning algorithm. I'll do that. If it means I need to learn something about linear programming, I'll do that. And generally just keeping an eye out, right? But the fact is, if you look at the news, every second day there's something like AI does X, Y, and if you follow all those things, you go insane pretty fast. So I just try to ignore it, except for things that are really outlier, that are really interesting. And I try to talk to Eric, Sanjay, and all my data science folks as a proxy for this. That's the most efficient method, having multiple people. The reinforced Sanjay's comment is it's like, to a certain extent, there is no really keeping up. I spend every day feeling like a caveman. It's just the amount of things that are happening right now is stupendous. All right. So with that, I think we wrap this panel up. Thank you so much for coming. I think it's very, very informative. I, at least I enjoyed it. Hopefully we'll have you again at DefCon. It's easy or something. So as an announcement for everyone else, there is a wrap up party slash trivia thing happening soon. Just go to the closing ceremony under tracks and you should really find it. Yeah. Oh, well. Thanks, Anish. Thanks, everyone. Thanks, Anish. Marcel, get some sleep. Good night. I know I can stay up. I think I'll play a round of two. See you guys. Adios. Take care.