 I also have with me Jeff Squires of Cisco Systems and one of the authors of OpenMPI Jeff. Thanks again for your time Hey Brock love doing these kinds of things. You always get to hear something new You can find us online at www.rce-cast.com You can find all the old shows on there see a list of people we're looking at talking to and if you have contact for any of Them blood is please let us know you can also follow me on Twitter at atfrockpalon and you can find that off of the website Me too, I think I'm at Chase Squires and Well a little note to anybody who's listening out there if there are things about MPI that you would love to see explained Let me know and I'll do a little bit about it on my blog because just off the cuff I did something about eager limits a week or two ago and Brock sent me back afterwards. Hey, that was really useful. I can show that to all my users And so you know if you gentle reader out there have anything you want to hear about on the MPI sphere Just let me know be happy to talk about it Yes, that that was very useful Also, I'm gonna be at the terror grid conference, which is in Salt Lake City this year That is coming up in July it is July 16th through 21st of any you guys are gonna be there for sure to track me down I'll be walking around as a campus champion there representing the University of Michigan So I'll be there if anyone else is gonna be there All right, so let's get on to our topic today I think you have a special affiliation with our guests. So why don't you give the the introduction honors here? Yeah, yeah, so our guest today is Rodney Maka of Hyperlogic He's actually gonna be speaking to us about a lot of the questions we have about Windows HPC server I've personally never ran an HPC server setup Rod's unique in that he actually got me into this business Rod hired me back in 2004 2005 as a student employee at U of M Running a bunch of Mac and Linux clusters there. So this is how I got started in this business So I owe Rod a lot for getting me on track in a great industry Yeah, thanks. Thanks, Brock. I said I knew you would be excellent employee And I'm glad to see that you know run the podcast of a small small world And I also work with Jeff in the past on lamb and then open MPI so it's got to great to see you guys Yeah, we go way back and before we were recording I was saying that people need to stop using lamb Yeah, please upgrade to open MPI and stop asking us questions about lamb. Yeah, you can do like the Microsoft IE 6 web page, you know stop using I6 Using lamb and you're like you can point point where in the world people are still using lamb Well, but kind of a segue here We're talking to Rodney here because he his company does a lot of consulting around Windows HPC and Windows HPC is not something that a lot of us know about so it'd be interesting to hear about it because At least my particular bias is mostly focused on Linux and other POS XE kinds of OS is And the HPC in those arenas now Windows HPC is getting more popular. So we figured Let's hear something about it. So Rodney, I wonder if you could give us a little the background here about why you can Talk to us about this stuff. Sure. Yeah, we have a consulting company hyperlogic So we do a lot of Linux consulting. We also do Microsoft HPC consulting mostly for Manufacturing and automotive because we're out of Michigan. So we do a lot of CACFD a lot of commercial office shelves shelf apps We're an HP HPC elite partner. So we put together clusters support them manage them And for Microsoft, you know, definitely like you said Linux has been you know Kind of de facto standard for a long time and as Microsoft's came out with their stack, they're now in the third version It's gotten to the point where it's actually Very interesting and we've seen a lot of customers in the manufacturing space adopting it for different types of Point solutions and even in larger companies in tier one automotive. We starting to see a Microsoft HPC server So we've been doing we have tools and solutions in that space. We do development and consulting So thought we could come today, you know, having a kind of a mixed background We see both sides of the fence of done operations development. We could maybe hopefully, you know Talk about talk to the H Microsoft HPC space Currently listeners that might be interested in kind of what's different or what is what is Microsoft doing in in this arena? Just stress that a rods a good candidate for this because as I mentioned at Michigan He was running in all Linux and Mac clusters. So and he still does Linux quite a bit of Linux still actually right, right? Yeah, yep. Yep. We still do a lot of Linux. So, you know, it's We can talk about like what you know, what is a decision point when you go to when you're trying to decide Linux or Windows, you know often You know, there's a there's some criteria that may make one more preferable over the other depending on your situation Well, yeah, actually, let's dive into that. I mean what what why would a customer choose Microsoft HPC? And I don't mean that nearly as confrontational as that sounds But you know a big chunk of the HPC industry is Linux. So what would lead somebody to the to the Windows solution instead? Well often what happens we go to a customer side the engineers, maybe they're using a Linux cluster They don't have necessarily Linux expertise or they face problems like they'll have they'll create an input file on Windows And they have to deal with like Unix to Windows line conversion all those like little things having to FTP files back and forth And so the the Linux for the end user the engineer, they almost don't care it They're just trying to get their job done. Whatever is the easiest way to get the job done. So often that barrier is That Windows reduces that barrier a little bit if the engineers are Windows focused now other shops there the engineers I'll want to use Linux. So it just depends on the environment the same thing goes on the flip side the for the IT administrator Often they if they don't have Linux skills and Linux comes in we often bring Linux into a Windows only shop in the past and You know IT, you know corporate IT is has maybe a more difficult time handling Linux than they do Then they do a Microsoft solution now I should say that just because you have a Microsoft solution in place doesn't mean that having Windows skills makes you an HPC person You definitely need to learn, you know the all of the part of HPC But that does lower the barrier of entry for folks who if they if they aren't familiar with HPC and they aren't familiar With Linux it's at least one of those two you knock out. You knock out if you want to consider HPC And kind of the third realm that's new is that if you want to like utilize a lot of companies now are going to Windows 7 So you see a lot of Windows 7 workstations out there and especially in engineering these things are almost like mini supercomputers I mean they got 8 cores a whole bunch of RAM, you know what they were they were more powerful than the clusters We used to run a U of m back in the day and they're sitting under someone's desk HPC server let you basically schedule on those boxes seamlessly So you can use it even if you have a Linux cluster You could use it as a point solution to do cycle scavenging on Windows boxes and you could do this in the past with third-party solutions But it's it's way way easier now with HPC server than it other ever was before with like having to buy off a third-party app to do it So a lot of people are used to seeing Windows machines, you know a desktop that kind of interface Is there anything about running an HPC a Windows HPC server set up that is going to feel very desktopy if you've been running Linux things before Well kind of the dream of HPC server is that the that you actually submit from your native application and you should never really even realize that you're submitting to a cluster necessarily So applications like Ansys there's some good apps out there that do have that you know dream realize they've using Microsoft API They submit to the cluster. It's it's basically transparent if you want to go back to where your application doesn't have that level of integration Then it's not that different than using a Linux. There's a command line API. You can do like job submit You know the the big difference between how you submit maybe to an HPC cluster versus a Linux cluster is what are you using instead of writing bash Maybe using bash files or PowerShell So so it's definitely a little bit different skill set, but it's not vastly different There's a command line you can you don't have to use the GUI to use the GUI tools though They have all GUI interfaces web portals standalone thick client, you know command line so whatever you prefer they don't you can choose choose any of the above Okay, so that sounds like they've actually reproduced a lot of stuff that we figured out in other batch environments for a long time What about actually moving code over that was originally developed in a Unix Linux environment? Do they have POSIX compatibility? How hard is it? Well, they do have a there's this thing called SUA, which is basically a full POSIX compliant Implementation on top of the Microsoft kernel so you can install that it has like AUX said grep You can basically compile any application with very little change But that's not really the way if you're gonna do that. You're almost I think you might as well stick with a stick with Linux But if there's a lot of toolkits like PGI make some nice tools that allow you to easily, you know, take a pure Linux Unix heritage code and make it work on Windows You know a lot of the changes are just like on Windows you have dot OBJ instead of dot oh you have to change your make file slightly for that PGI actually has a pretty good tutorial on on migrating over some classical HPC codes over to Microsoft just to kind of show you What the steps were that you had to do, you know, what did they have to you know change? So it's a it's but you know, it's it can be non-trivial or it can be You know, it could be an effort just depending on how many assumptions or code makes about the fact that it's running on Linux Like how do you do your directories? You assume slash temp is there, you know, are you using fork, you know stuff like that? All right, so along that line, you know, my particular bias is MPI, you know and hypothetically the MPI API is exactly the same between Linux and and Windows and a lot of people have experience porting between different versions of MPI simply because you know There are fundamental differences for example between open MPI and MPH to two big popular open source ones So I would imagine that the MPI portability is not so much the issue. It's really all the other little things Is that kind of an accurate statement? Yes, that's correct. The MPH to layers the portability issue is not there The the Microsoft MPI is based on MPH to so if it's compatible All right, and what version is the HPC server based on are there different versions, you know What what do people use out there? There's basically one version. There's the eight there. They've had three versions of the product But the version that you get is HPC server 2008 R2 so that's like a whole huge mouthful But that that's the main version and it's broken down into a couple subsets based on what functionality you need Okay, so what things are included in there? I mean, you know, typically and again I can only speak for my experience on the on the Linux side You know, there's a scheduler and a resource manager and a Cuing system and things like that, you know, are these kinds of things involved on the Windows HPC stack as well? Yep, that's correct. Actually, they have let's call the a Pack express pack and it has all those things basically for free if you already have a Windows server that you've played for us paid for a server 2008 R2 It has message passing job scheduling Reporting sys admin deployment all that stuff is included for free Now they also have like an enterprise pack that gives you some additional functionality like being able to run on Windows 7 workstations Or using Excel on a cluster and some other functionality like that So I've heard some stories and speaking with Microsoft reps when they come by about infrastructure needs and So that's been some confusion for people like I need an active directory setup Or I need a quite a large Windows infrastructure to even support this is that true and what is really needed? You do need active directory But there's very few businesses that I have encountered that don't have active directory installed But it is true active directory is the main requirement You could always install active directory separately if you needed it for some reason your company didn't support active directory But but that is a requirement Okay, so I because I had heard things like SQL server was required to be able to collect statistics because I stored it in a SQL database So none of that's is that just an optional requirement and it's not needed to actually run the server It includes SQL server express which is free But if you want to run let's say a top 500 style cluster Then you would want to put full-blown SQL server on there if you want to collect a full Full debugging etc. That that would be true Speaking of top 500. What is the largest Windows HPC cluster you've seen slash been involved with? They have they just did a pediscale cluster the one over the subami over in Japan So that was I think 1.1 Petaflops and over like 1300 nodes. I think it was number four on the top 500 last time I had checked though To be honest the majority of the installations we're doing them You know top 500 are like few and far between but the majority of the clusters we see in the Manufacturing etc. You know under 256 nodes, you know I've seen as large as 512 nodes that we've done with H with Microsoft HPC server So it just depends on the implementation for Windows workstation nodes We've done up to a thousand Windows workstation nodes with being scheduled under Microsoft HPC But but yeah, it's it's it's it's it's certainly they can do top 500 obviously top 500 dominated by Linux right now All right, so they're not really targeting it though It'd be fair to say they're targeting more the the commoditization market at the bottom 50,000 Kind of market is that accurate? I can't speak for Microsoft I'd say if the people wanted to buy in the top 500 we're buying Microsoft they they they can scale to that level They certainly designed to that level and they've shown through you know doing top 500 runs that they they don't have Scalability problems in any part of their stack whether it be MPI or or just loading the reporting tools All are go scale up to the up to the top 500 level. It's just that There's not like I said the top 500 customer base. That's 500 customers in the world There's a they're trying to grow the market I think where I see where we see personally us deploying Microsoft Microsoft HPC server our customers that would not have done a Linux cluster Certainly would have done a you know top 500 type cluster and but we need a manufacturing needs to reduce the time to To engineering they need to reduce solving time They need some kind of cluster technology and so I see it enabling a market that maybe wasn't there It wouldn't have been there if we would have stuck to just traditional you know putting xcat or you know some type of Linux solution out for customer where that that would have been a difficult sell to that that user base or that IT department So I want to back up a little bit you mentioned a tsunami cluster at Tokyo Tech The first version of that just ran some links derivative But I had also seen some numbers of that running Linux. Is that like a dual boot environment or something? They're doing there. Do you see that kind of environment often? I? Personally don't like the dual boot solution though. You can certainly a lot of customers do that I know BMW and there's a bunch of universities that are using like stuff from adaptive computing and platform That'll allow you to dual boot based on what you need, but I personally like Linux or Windows I don't actually come into a scenario with customers that they're that they want both Now why do they want both what kind of requirements would that be? What's the benefit of that? Well, one would be if you have an application that has a very nice native integration with Microsoft HPC server that would be one one one area where I could see that beneficial or you're developing on.net Microsoft is investing in different types of languages and technologies to improve Writing parallel code. I mean you guys develop code, you know, it's hard to write Parallel code and so if you have an application like that that would require basically Microsoft platform I could see that be another requirement, but Normally when when we're going into customers, they want one or the other We don't have we're not trying to decide for them which one that that they need both So for the cluster administrator if they hook a console up to Compute node in the Windows HPC world will have like a full desktop on it Or is this like a lean stripped down version of Windows that goes on to compute nodes? You can strip it down if you want to but it's basically server 2008 R2 What they do is they you're not allowed via licensing if you want to run the cheaper version of HPC You can't put them like your active directory mail server and everything on it That's just they give you a reduced price purpose built basically to run computational Loads, but but yeah, it'll look and feel like what regular windows you don't have to But like just like on Linux if you install red hat it comes with the Bluetooth enabled and all those types of things So you still want to tune it down if you want to get maximum performance, but it's certainly not required There all right. Here's a random question out of the blue. I am on the mp3 Fortran committee for Really obscure reasons because I don't really know a lot about Fortran, but somehow I'm in this group is Fortran supported on Windows Yes, they support Fortran now It's either with PGI or Intel or a third-party compiler vendor But yeah, they do they have Fortran bindings in their MPI library and you can use PGI or or Intel all right, there is nice because there's a you know a huge boatload of existing Fortran MPI codes out there They might not very well be the target since you know portability is not necessarily the main issue here, but Certainly particularly with modern versions of Fortran. It's actually pretty kind of kind of analogous to C++. It's a fairly Modern language, but a couple people have asked me about that whether it's supported on Windows or not So I figured I'd give an ask here. Here's another obvious question. What networks are supported by Microsoft MPI? They support the standard fare Infinite band, you know gigabit ethernet 10 gigabit ethernet mirror net, you know all the major interconnects They have a network to direct providers. So now while what Microsoft doesn't actually produce like the OFED stack Obviously, that's you know a consortium of people developing that so you would obtain your infinite band layer You could go download that but they they have a committer on that that helps and I'm sure you're you're involved with that as well So but yeah all the major interconnects. Yeah, I actually do I'm asking a few questions that I know the answer Microsoft MPI guy The MPI forum together and he and I get along pretty well, but Yeah, so still give a little explanation of what's network direct because I assume most Linux people won't know what that is Yeah, when they when it's basically a direct to hardware when they first came out with the very first version of this product They actually had a like abstraction layer between that and the hardware and it made it so that their performance numbers weren't good So when you saw benchmarks, you know It reinforced people's preconception that windows must be slower So they fixed that in the next release with this network direct that basically gives them us just a small shin that That lets them have direct access to the hardware And then you get the latencies down and the things that you need when you obviously you want using a low latency Interconnect and and now you see the numbers when you see Linux Windows benchmarks You know, they're they're almost identical within a few percentage either way and that's because of that change they made and 2008 So what kind of latencies are you seeing? I mean we've had a lot of focus on our DMA type networks and stuff on The Linux platform and there's this big existing infrastructure of Linux HPC systems out there their latency focus What kind of numbers are the Microsoft guys getting the same? I mean, that's not they're not limiting They're not being limited by the fact that it's windows. Let's say with the network direct you get the same Latency kind of numbers when you look at the application performance You almost can't tell the difference in fact some sometimes they went sometimes they lose but it's it's not a deciding factor anymore I have never gone into a customer and they said we need Linux because it performs better at least for us That's not the that doesn't need that conversation doesn't happen anymore in the old Yeah, I'm gonna I'm gonna go off on a little rant here too saying that you know it is real application performance that matters micro benchmarks have their place and There is certainly some chest dumping going around saying I have the lowest latency around and things like that But you know, I've said this actually many times on my blog as well like, you know, low latency is great High bandwidth is great, but it really matters. What's your application gonna do and there are a lot of latency sensitive applications out there But there's also a whole lot of them that aren't even though they think they are that if you run it Over TCP and you run it over in finna band You're not even gonna see much of a noticeable difference. And so I would say I would assume that rod is right, right if They've got to infinite man latency down in the one to three microsecond range or even maybe even four Probably doesn't matter particularly for the target audience that we've talked about a little bit here on on the podcast Yeah, a lot of times customers. They don't they don't come to us and say we need one microsecond second latency between nodes They say I need this application to get this model ran in this amount of time Whatever technology meets it. That's that makes them happy. So we often, you know, don't talk about Talk about the those micro benchmarks with customers. We're talking about at the application level So what about file systems with small linch clusters people have people start with this NFS But sometimes they move into things like PVFS luster Gluster a few of the other like a more exotic parallel file systems a lot of those I don't remember seeing Windows support for does Microsoft have their own solution for high performance IO operations Nope, they don't they still rely on third parties for high performance I mean you could certainly put in something like a fusion IO card, which is you know if you need really really fast bandwidth to one particular node, but Yeah, they don't have a parallel file system of their own They definitely depend on third parties to provide that kind of support So something I probably should have touched on earlier in the conversation is how is it licensed and what are the costs of this? Because one of the deciding factors in HPC is cost now We're kind of moving away from that as HPC is getting commoditized, you know It's exactly what you said that the engineers, you know, they don't care about whether the job finishes at midnight or 3 a.m Just as long as it's done before they get in in the morning and in you know a traditional enterprise kind of world Cost isn't the major factor anymore where it used to be in the you know the academics and the researchers They have the the least amount of funds and they're trying to get the most bang for their buck So how did Microsoft approach their licensing and their cost structure? How does that work? It's basically licensed per server So you have a they don't need Cal or with the access licenses But yeah per server if you're using Windows 7 workstations per workstation in terms of the cost in a corporate environment You know like you can say okay, we can use central assets free But a lot of times in a corporate environment, they want to have red hat they want to have that hey I want to have support and then you layer on the other pieces well We need this commercial scheduler that has support and then we layer on another layer We need or some kind of reporting mechanism that comes from your scheduler vendor by the time you assemble those pieces I don't really think cost basically becomes an becomes an issue Because you end up having really on the on the way it is today The most expensive person or the most expensive part of the solution is the software and then the person running it At least in our space Let's cost us like the OS and they even the hardware nowadays has gotten to be you know, not the most expensive piece of the puzzle Okay, so taking all that I mean and you coming from a Linux e maxi kind of background I mean what is is your take? I mean is this a good product is this something that people can get worked on and have confidence in How's the support on it? You know all these all these kinds of things Yeah, I really like 2008 R2 I think deploying and standing up a cluster if you have Windows skills is definitely way easier But of course if you come from a Linux background, you know, you have the same, you know ex-cat There's lots of toolkits to choose from you know some there's disadvantages advantages of both with Windows HPC It is nice. You have you know one stack. You don't have to wonder does this version of MPI work with this version of Linux I mean Intel's trying with Intel cluster ready to basically kind of make a Monopolized stack there to create the same kind of consistent ecosystem But I think it reduces the amount of testing you have to do in that sense The one thing I really love I like command line So I do like they have PowerShell and and I like you know But being able to use being able to use the break open a command window and and but there's different skills So one will sometimes people know when they're trying to especially an app that was ported, you know They say while there's it's not just point-and-click not every application on HPC servers point-and-click. So Just realize it at That you still you know for Linux people are trying out Windows. You're still you're gonna use your skills It'll be just a different technology you're using So very early on you mentioned something about Cycle scavenging picking up excess cycles from machines at night I'm guessing just like you know your your desktop at work. Can you tell us a little more about that? Yeah, basically if you have a win it has to be Windows 7 But if you have a Windows 7 workstation you in and you basically install this little cluster Compute pack on it that and it joins to the cluster and then that's available basically for scheduling jobs So you can set up things to say if the load is too high or during certain times Don't schedule on those nodes, but they basically become a resource and Actually a lot of Linux only shops that have Linux clusters We've been deploying the solution because you just basically put a head node down at their site They already have all these Windows 7 workstations that they've paid the capital expense for and now they're able to actually utilize them off hours Why before all they were doing is costing money right sucking up energy not being utilized And so that has been a quite a popular use of HPC server Especially in the last six months where we've seen a lot of Windows XP to Windows 7 rollouts We've been doing quite a bit there So do you see these even being used during business hours like you know the the secretary or even the engineer who really all they're running is you know Outlook and you know, they're they're barely using their machine Are those types of environments being used during business hours to say like take over half the cores on their machines I mean machines are coming with two and four and eight cores standard these days. Anyway, yep We have a lot of we have so several projects Currently going on where the customers are trying to avoid actually buying any kind of cluster because they're like We already have the equivalent of a 512 node cluster here just in our engineering department Why don't we just reuse those resources and so that that is a is an area that people are interested in They're trying to avoid even buying a cluster just reuse the resources. They already have No, that's fascinating. Are they trying this with desktops or with laptops? I mean, does it work with laptops since they they come and go and you know, how does that? impact a running job for example if someone it's running on a laptop and then I close the lid and walk away the majority of the ones We've been doing are actually workstations like HP workstations But you could do it with a laptop But that is true if they unplugged it, you know, the job would crash you could schedule it to re queue but But I haven't done one with the laptops It's probably not they're not as good a use case not as power powerful as a workstation still today So when people are running this cycle scavenging version, are they generally running serial codes or single node parallel jobs? Or you know, what's kind of the more common way of using this? Often they they use SMP if you have eight cores the limiting factor often is the price of the software licenses to be honest like Some of these third-party ISV applications are extremely expensive They don't actually have enough cores to run a large job many big clusters are just running lots of little jobs And so for them, yeah, they can run an eight CPU job on one box. Great. They don't need to run across I do have some folks that are running across to gigabit ethernet though and doing a distributed parallel to get the memory down So farming out all these jobs isn't you know in the input data in the output data and everything else Isn't starting to put extra stress on the corporate network Often in places we go the because I'm in manufacturing. I should Say that in manufacturing the engineers often have their own dedicated network They're usually gigabit because they are large transferring large files around to each other So no, it's not it's not like it's going out on the corporate backbone and back You know, it's on it is still on a dedicated network So are people running mostly ISV codes or is there any homegrown stuff? Or is it mostly focus on you know your abacus is an influence and things like that Well from I know and and this is this is where from from our perspective We live in manufacturing and also an oil and gas so between those two a lot of that is Is ISV application some of those have like Fortran We because they're Fortran based codes that where you are putting in like user-defined libraries But certainly some customers are running their own codes. It's just our customer base is mostly ISV ISV apps financial services though They're all basically custom and HPC servers is becoming pretty popular and In that space because they also support SOA and some other types of models that are popular and financials Now I know that Microsoft is testing heavily in trying to and you mentioned this even earlier in the call in the call here that you know They're trying to make it easier to write parallel code Do you have any of your customers doing that or is that something that you can talk about it? I'll give some examples because we all know that it's it's difficult to write parallel code I mean, I guess I'd be interested in hearing. How are they trying to make it easier? Yeah in a previous life, you know, I worked at a compiler company where we you're developing tools to help people write parallel code That isn't the market. I haven't been in anymore. So to be honest, none of my customers are trying to write parallel code Though I do follow the space that I still interested in and developer in the developer community We definitely develop a lot of our own code, but it's not, you know HPC So yeah, I couldn't speak to that. I think it's very interesting what they're doing in the space But I still think the majority of applications are MPI base today Open MP So I'm gonna go off into the weeds into a really weird spot here. This is a relatively small market Really really large SMP machines. We saw SGI take the altix from the itanium platform Which windows did support back to you know, your regular nihilum chip that we get on a you know Two-socket server from Dell Or do you see anybody using windows HPC on a 128 core SMP machine or anything like that? You certainly could there's no reason that you couldn't though Yeah, like I said, majority of our customers aren't trying to buy a big scale-up SMP solution Mostly because it's still cheaper to buy a bunch of smaller cheaper boxes than in to buy one big expensive box And especially in manufacturing if that one big expensive box goes down you're out of business And you have a bunch of little boxes a couple go down. You're not out anything. So I don't see big SMP boxes too often in our space. So Rodney, I know you're not Microsoft yourself, but you know anything about the you know upcoming releases What are some new cool neat features that are coming and when are they coming and things like that? Sure. Yep, they're having another release called sp2, which is gonna be out Maybe by the time this podcast is available. You can download the beta currently and try it out Some of the new features that they're gonna have one of the big ones is they're gonna have support for Azure So you're gonna be able to actually just put a head node, you know at your site and run your MPI jobs basically on Azure Which is like a cloud slash hosted Solution so that that's kind of the big functionality They're gonna support VM roles and you'll be able to actually see your on-site file shares remotely on Azure and be able to access those That sounds cool with all the standard cloud HPcish questions. How do I get my data there? How do I keep it secure and there's all kinds of things like that that we talked about when we had Nicole Hamstoth on here with HPC in the cloud Yes on a while ago, but those solutions don't you know the cloud that you know There's that whole cloud talk though I can't say in terms of cloud security you'd be surprised I you know how insecure a lot of customer sites are the cloud may not be any less secure than their site already is or you know, but But I could say it's a it's a nice solution for people that maybe only once in a blue moon need to run a larger job You can spin up and like you had mentioned earlier. They just care about getting the job done Does it run slower? Yep, sure does but they get the answer by the time they need it, which is the important part. I Can't say I've heard that argument about cloud security before but it's probably I wouldn't doubt you Hey, Rod, thanks a lot for your time and clarifying a lot of these questions We've had we've had Microsoft reps come through here and you know, they're not necessary They're the same Microsoft reps are selling desktop So we've we've had some HPC specific people But it's it's great talking to someone who's actually stood it up and used it and actually got stuff running on it So Rod, where can people find you and how can they get ahold of you? Yeah, you can reach us at hyper logic comm It's hi for high performance comm and we do like you said if you have any questions We also have a website just dedicated to our tools for HPC server called total CAE dot com So either either place you can reach us in Okay, well, thanks again for your time, Rod All right, thanks for your time, Rod Yeah, thanks for having me guys. It was a lot of fun