 All right. Well, good morning. It's 9.50 and this is probably about all the people I want to embarrass myself in front of anyway So thanks for coming I'm here to talk to you about high performance open stack for science and data analytics in a hybrid cloud environment very long title The content's actually not as long as the title Steven Carter We've got Jason Grimm and Josh Lothian. We're all from Cisco Systems All of us have in past lives been HPC practitioners And so now at Cisco we tend to help customers do HPC things on clouds We all know that open stack is is useful for doing science that the problem is that open stack is a constantly evolving constantly changing Animal if you're always looking for that next feature to tweak your performance just a little bit You risk having multiple science experiments your primary science experiment like genomics and your secondary science experiment like Open stack, right? We really believe that as it stands today and really as it stood for the last year along with other public cloud Environments, it's it's very conducive to do in science. It's just you kind of have to do things maybe a little bit differently So what we'd ask is is maybe a little bit of attention We'll go over what we've done with our customers and hopefully that information will help either you if you were a Computer scientist trying to run science on clouds or if you're have users trying to run science on clouds Two points of order actually we have some of our slides are our eye charts really right because they're heat templates and whatever And I'm not sure if you read them read them all that well We that when they post the videos what we intend to do is make a comment with with a link to where you can get The slides and some of the templates and to use as reference material right because you're probably not gonna remember much of this Anyway, it's all it's all a blur with everything else. You're seeing the other thing is at my heart I'm actually still pretty much a network guy, and I try to infuse humor into my talks The problem is I send all of my humor via UDP. So I don't really know if you get it or not See that that usually only works in a network crowd. So you got clearly some network people in here Really the goal of this Presentation is to help people get science done, right? You know, let's let's not worry about the infrastructure too much. We're gonna go over Things to watch for right when you're using clouds particularly around networking and storage how to run workloads on these clouds We're not going to get too in-depth, right? We've got 40 minutes. We can't cure cancer But but maybe some of the information learned here can help others cure cancer and we will take credit for it science in the cloud Why right a lot of old-school HPC practitioners like, you know, that doesn't compute the the problem is that there's this explosion of data across all disciplines and and Many of these disciplines are trying to analyze these data this data and there aren't Enough HPC clusters out there, frankly, and so they're going to all sort of platforms. They're taking Resources from wherever they can get it and if but if you think about it HPC workloads, right depending on how you define HPC I usually say HPC with a small h maybe our very conducive to cloud cloud one of the one of the tennis of cloud is Scalability statelessness, right? So HPC workloads have been one of the one of the first stateless workloads out there You you bring in the state you compute and you you you spin it all down, right? You you can achieve economies of scope with other cloud companies. So you've got companies like Google and Facebook and eBay developing Mechanisms to scale software to to cloud scale right to to massive scale if you use these Mechanisms you can take compute workloads and scale them to massive scale and and science is multi-disciplinary It's not necessarily especially nowadays. It's not about putting an HPC thing in a corner and doing some some work on it It's really about collaborating. It's about taking in data. It's about disseminating data There are many things that you have to do other than compute and so in a cloud environment that in a cloud environment in which you can do these Computational workloads you can also use this infrastructure to do the collaboration to do all of the other parts of what science is nowadays Differences that tends to be very data intensive and compute memory intensive So when you when you when you're looking at hybrid cloud usage, right? I mean there's there's goodness in the hybrid cloud You put the workload where it needs to be the problem in the public cloud component of the hybrid cloud is they really tailor They don't tailor to this use case right? They don't tailor to the high CPU the high memory the large data movement So if you're if you're not using the right part of the cloud in the right way You're going to get big bills and potentially bad performance and that's why hybrid clouds a good thing because again You put the workload in the place that's most conducive to undertaking that workload And it's not so much batch right a lot of the cloud technologies out there more streaming than they are batch but you can you can still do batch and You know one of the things we work on that we don't cover in this talk is really making your Application the core of your application more cloud native and using beat to be able to use things like Kubernetes and streaming analytics and such When You know the sort of first step we're not asking you to go rewrite all your applications to make them more cloud native It's about meeting the application where it is But you can still infuse cloud principles and how you use the cloud and one of those is Portability right write your application your workflow in such a way that it can be portable and I'll talk about that a little bit later Elastic make sure that it can scale make sure that it can grow to to to consume these resources That might be available to you for bursting into the cloud for example and and infrastructure agnostic There is certainly value. I'm not trying to dissuade You know the science projects of making better infrastructure. That is that is goodness all around right, but it is in itself kind of a science If you're really wanting if you're wanting to adopt cloud principles most cloud native applications They don't really care what infrastructure they run on right? It's about the application. It's not about the infrastructure So take that mindset make your application make your scientific application about the application and not the infrastructure And you know don't be a hero right some all tweaks to a cluster will yield good results at large tweaks again They yield a science experiment. So leave the science experiments for what they need to be but when you're trying to undertake science Maybe take a more pragmatic approach use it use it where you can use the optimizations where you can But where you have to get things done where you have to get science done Take a more pragmatic approach and it's about you know for us It's not about optimizing time to solution a particular time to solution for one workload It's really about if you're in if you're an institution. It's about optimizing time to science It's about scientific output and it's about giving people access to the resources they need to undertake their scientific workloads And I promise I will not spend that same amount on time on every single slide This is an example workflow. We worked with we did two of them in this particular for this particular material One is a whole genomic sequence seen pipeline the other one's a farmer job But it's it's very much. It's very similar to many many workflows And that pointer does not do anything on that screen doesn't to many workloads, right? So in this case we have an alignment from from a lime in in the alignment phase We take a reference genome and we take the genome of a person Does anybody know na one two eight seven eight by any chance? That's yeah It's actually me that this researcher promised he could he could cure my mayor my male pattern baldness if they did some analysis on my genes, but The UDP I don't even really know if you got that or not So there's some initial data that you pull in right and then it's it's a pipeline of several tools It's a tool tool chain and and the the first tool follow you know feeds the second tool second tool second And the next tool next tool and then it it puts it into two objects It puts it in the storage, right? We use object storage in the in the form of Swift in this particular Experiment and then that's where you kind of you can collaborate and and disseminate data and and interact with with participants in the program really But that's just sort of the that that's very much like pretty much all the jobs that are run in this sort of space and the embarrassly parallel loosely couple it coupled space that that tends to be very good for For cloud if we start looking at some of the bottlenecks to avoid You know this this first on the left is is what you'll usually find in a open-stack Cluster right now, you know, especially when that's not tuned You end up having some network node that network node could either be a silicon-driven node a hardware node or software node often It tends to be software node The problem is you you can run into some bottlenecks there right software is great and it fees great But but in general it tends to be about a thousand times slower than it's then it's equivalent hardware It's equivalent hardware component You can try to address this to scale But but you know again when people do open-stack it's hard enough it is as it is to get it up and going You know they might not have tweaked it you still want it to be useful though So just be cognizant that that network node could be a bottleneck Malt you know tenets a tenant you also go through that network node by default Where you don't go through the network node is within your tenant network, right? So within a tenant you tend to not be exposed to that network node as a bottleneck Depending on your overlay could add in a bottleneck right, but we won't get into that But you know generally that's going to be your best friend with node to node There's you know of late you've been you've had distributed virtual router as an option distributed virtual router helps out quite a bit In that tenets a tenant. It's handled locally Source not you still kind of you still go through that router, but floating IP you don't right So that could be a way of getting around a bottleneck So if you have data transfer nodes for example You might give them the floating IP so they can pull data in and out whether you have a formal data transfer node or something Like globus on it or just a node that you end up doing your FTPs from and then on the right is is what we find That we have the most success with and that's just a simple plain old provider VLAN or or just a provider network Generally in the form of provider VLAN, and so you don't have to worry about any of this You you kind of exposed to the VLAN And you get kind of full bandwidth to your to your instance If we go over storage it's it's important to know your path to how you're using that storage if you're using a Femoral that's generally local that's local to the node So if you're using a femoral for for scratch You're reaching down into the net into the nodes hardware if you've got fast disk a nice raid or SSDs or a combination of the two You'll get some pretty good performance. You don't have to worry about any of these bottlenecks If you use sender now, it's not just not you know an absolute statement But generally sender is accessed via NFS or ice-cazi or something over the back-end network of the nodes You don't have to worry about bottlenecks from the neutron router because it's not one of the tenant networks So if you have like dual 10 gigs or dual 40 gigs, you can generally get some pretty good bandwidth the storage If you're using where this is not an absolute statement like for example some some open-stack clusters will use a converged Storage right so they'll run staff on the same as as the compute so sometimes your sender might be local Which which could also have another another aspect of goodness And then when you're trying to access shared storage for example swift or NFS or cluster This is when you have to worry about your network bottlenecks right a lot of a lot of HPC job Still use some sort of shared POSIX file system to get data to write out results And just you have to be cognizant that when you mount that file system when you do I owe to that that Shared file system that you're doing in such a way to avoid these network bottlenecks If we look at flows right so HPC Data workflows that is how you get your data how you put out your data This is the the object storage tends to be good in a cloud environment Mostly because you decouple yourself from a POSIX file system You can still use a positive or I should say a shared POSIX file system You can still use a POSIX file system for your scratch base right that doesn't really stop your your job portability But if you're thinking about portability, you might not want to have your application Just always assume there's a shared file system and mount it because if you want to move that to Amazon or some other public cloud or community cloud That file system is not going to be available to mount But if you get your data via object storage swift for example, which is very often co-located with an open-stack thing You can ingest that data Operate on that data use local scratch and then exfiltrate that data to your object storage And this way you know say you have a swift cluster at your institution You can push your jobs in any cloud and it's it's a pull right you pull that object out of your cluster with the appropriate Authentication and whatnot into for example Amazon where you don't pull for you don't get charged for pulling data into right you do your work You get that data out of there So you're not charged for the the storage now you're charged to get data out of Amazon, but still it's actually a good It's a good mechanism for job portability and kind of most conducive You know financial model with respect to doing your workloads You've also got as I mentioned shared file system So you can certainly co-locate an open-stack cluster with a shared file system like luster a GPFS or luster or PVFS or whatever And that works perfectly fine It's just it does limit your portability and you have to worry about the bottlenecks a little bit And then there's a shared virtual file system to some of the tools that we present that you could use Will actually spin up some of these right so if you if you if you take advantage of the fact that generally within a tenant network You have very you know very many very many fewer if that's correct grammar bottlenecks Essentially what you do is when you spin up your worker nodes you might also spin up You know we kind of demonstrate spinning up some slurm scheduling nodes You can spin up some Gluster nodes all of which use Local ephemeral storage and stitch that together as a shared file system and this helps for things like genomics So so genomics Pipelines they tend to have you know you end up do Processing these genomes that come in right that you have to do this this Sequencing on you compare them against reference genomes and actually a lot of reference files So instead of if you have ten workers or a hundred workers instead of each one of those hundred workers Pulling in those files separately you can basically make a shared file system a temporary shared file system in this virtual environment Pull it down there and then I'll share it between the workers right and they might they might they might pull the genomes out of object Storage and and even the the shared stuff in the shared of the shared data and shared file system might be pulled out of object Storage and then just for completeness. We haven't actually tested this one the In the kilo time frame they you're supposed to have put in multi-mount for volume So you know technically if you have this reference data you can have it that out there in the volume and all of these Instances can mount it and you can they can all sort of use that common reference point Now we get into launching workloads, you know, we tried to be pretty signal pretty pretty simple pretty pragmatic and pretty portable In all of these recover basically three three different mechanisms for starting workload The first is the easiest right a bash shell script or you know, whatever shell script of your choice python, you know, whatever you want really But most of these cloud infrastructures open stack AWS Azure Google Have API's and you can simply Write a script to launch your workload in in this it's it's quick. It's dirty. It's easy to do It's not quite as flexible, you know So you might have to do you have to do a lot of your role your own Cluster if you want to do things like schedulers or shared file system, but it's easy. It's good for a quick and easy test It's also not very dynamic. You know, pretty much you're going to Spin it up. You kind of have to know when your job is done and then you pull it back down Then we then heat heat's nice because it's built in to open stack It's pretty capable, you know, you can you can access it via CLI web UI or GUI but basically we spin up a cluster with heat using We put a cluster head not a cluster a slurm controller, right for your bat scheduler and then a bunch of slurm workers It's good because it's it takes a little bit more of the do-it-yourself off of you, right? So the the putting into a template is just a little bit easier than writing all the bascripts or the scripting Yourself it's the where it kind of is not so great. It's a really good sort of day zero and and take it down Right. It's not really heat's not really good for maintaining something So it's really about spinning up a cluster doing something and bring it down to cluster but luckily that's what most HPC jobs look like anyway and then Elasti cluster is What we what we also did your cousin example there are other tools for example star cluster Elasti cluster seems to have a lot of good capabilities in it Mostly just an example, but what but elastic cluster is a tool and this is sort of you know Being kind of infrastructure agnostic. It's a tool that you can point at different infrastructures Use the same way to spin up a cluster. It has a lot of ansible playbooks for slurm and and SGE and cluster That are kind of built in and so you can bring up a cluster launch some jobs in it and pull down that cluster So that's what I'll go over now So if we look at bash, it's pretty simple, right? I'd use my pointer, but we Establish it's not useful on these screens, but basically, you know You're you're setting up your environmental variables for things like your key and the image you want to use and the network you want to put it on and then Over here, right? We have to figure out a way to stick that job into The the node right and so we do that with cloud in it cloud in it is how you get your key in there and your name and other things Only half way through I've only lost more person so far. That's not too bad That's that's much better than last time. That was a disaster So we have to figure out a way to get workloads in there and so we have this cloud in it script I've actually been waiting for someone to leave so I can use that bit We have this cloud in it script focused even we have this cloud in it script that basically gets pushed into the The instance and then you run it to set up your environment and launch launch the workload And then we you know have here's some more bash some some more bash Here that we call the nova boot and you know your favorite control loop So you spin it up you run it the thing here is that you know over doesn't really know whether your job's finished or not It just kind of knows the stuff's there and so what you want to make sure you do is when you're done You put that result someplace where you can see and say okay Well, I've seen all 100 of my nodes kind of report back that it's done I'll bring down these this cluster, right? So it's again. It's quick and dirty, but it's it's probably you know That's not the most capable Now we've got heat, you know heat is is good It's it takes another it's another step up and complexity another step up in capability What we see here is kind of very similar when we set the environmental very environmental variables in the basket We kind of set the kind of the the the instance type We want the network to put it on the keys we're going to use and then we start set defining resources like our slurm controller resources and then our slurm Compute resources defining again basically some cloud in it that we use to bring up These nodes right to basically provision the nodes add them to slurm and and mount stuff You can leave this one hurts a little bit more because he's actually on my team, you know That actually does make me a little sad and So we define our resources We've defined the resource and these resources things that spin up like a slurm control node or a slurm computer computer Or what have you and then again more more of the same And then adding security groups right because you don't want to just kind of put this stuff out for anything for anybody to poke at asking for a floating IP for the Slurm controller that had node right and then finally we go and we capture we we launched some some compute nodes Capture the output because when you run this you want to get the IP address back of your head node So you don't have to go searching for it right and then you can log in there And you can launch jobs of slurm or whatever you whatever bat schedule you want you can also set up cluster nodes You can set up data transfer nodes you could do a lot with heat and it's there It's you know generally there in most of the open stack clusters You're going to be doing and this is this is a an example of launching the stack From the CLI but again, you know you got heat integrated into the GUI And so this is that same stack launched and you can kind of see some nice graphs And you can sort of see the resources that are up and the status and you can you know pull that cluster down when you're done Last but not least a last a cluster Again, so this is one of many tools. There's there are a couple of tools like this But but it is a piece of code You go ahead and you define some things right so in this particular case We've defined that that do we find the cloud we're going to talk to right so we're we're provider open stack It could be provider AWS For example, if you want to sort of push you want to aim it into the cloud for for more of a hybrid cloud approach You give Username and project name and and you know whether your request floating IP You're basically all the stuff that you need to to define to point it to open stacks open stack kind of you know So the infrastructure it knows how to treat that particular infrastructure and then log in for the For the nodes in in the cloud It has it comes with several playbooks. This is an ansible slurm playbook So this will go out and run that playbook on the nodes to set up slurm It actually has an ansible cluster playbook an SGE playbook a couple of other playbooks So they've kind of you put a good bit of thought into this and they use relatively new tools to do it and Then you set up the instances and whatnot, you know it's similar theme right so in bass we did this repeat to find all this I mean at the end of the day and all the tools you're gonna have to tell the infrastructure What images you want to use what flavors you're gonna want to use and whatnot in in this particular scenario What we did was or specifically where Josh did because I didn't do any of this frankly so Unless it's good. If it's good. It was all me So we define You know how many nodes we have and then this this kind of set up everything so basically does an s run to start the job and slurm and slurm goes and runs the script and all the nodes and it sets up all the nodes and Downloads whatever you need right it basically preps the nodes for this When you launch it you you simply type elastocluster start pharma This was the end of which is a pharmaceutical job. We were running for a customer And so that basically instantiates this cluster the elastocluster SF SFTP pharma is how he pushed out the script that he's going to use right that he's going to do the s run on and then This bat script is is what the job is right? So this bat script will go and define some environment variables I don't think he pulls down it I think he pushes everything out already and then he kind of runs runs the job at the end Here he's he's elastocluster SSH into that pharma node into that head node for pharma and then he fires up 256 of these workers and Sends them to slurm now the nice thing about this are if you use bash and you didn't use a scheduler You kind of you you spin up what you need right? There's no batching there with both heat and with this you can let slurm take care of that You can use a lot of the same sort of batch system mechanics that you've used for what you do today, probably right and Then you can bring it down with elastocluster stop pharma And again, it's it's not necessarily about getting the completion job of one very high Really, it's about okay This you your institution is bought has put money into these resources to do science And we want to do as much science on those resources as possible So what we do is we we load this up with as many cloud jobs as possible this this right right here We've actually got HT still turned on we've got oversubscription We were going to do run some more with HT turned off Hyperthreading right so hyperthreading is not always great for these types of jobs Over subscriptions not always great for these types of jobs So even you know it you know that little bit of tweaking that I'm talking about you could do some tweaking and have a Different aggregate of servers that are slightly more conducive to the compute workloads for example That's pretty easy different bio settings different over subscriptions such But really it's about you know 99.8% utilization on this cluster if you're going to spend money to do on this cluster you give you You know give it to as many of your scientists as possible as an asset as as a resource And so it does scale well in that respect Now a little bit about Hybrid cloud You know there's there's basically two ways to do hybrid cloud and you know we we help folks set up You know hybrid cloud infrastructure is a lot if you're an institution and you want to say okay Well, we're going to be using Amazon or some other public cloud for You know on an ongoing basis either we might put study state in there We might use it for burst, but we just kind of want to plumb that network to go into Some public cloud. There's there's a couple of easy ways to do that right? Amazon has direct connect and you can take that you can take direct connect and terminate it at some institutional border Right if you are the institution doing it if you're a user you could either spin up an NFV in Amazon or use Amazon's I mean you might not have the capabilities in the Amazon account You might share it with other folks, right? So you might not be able to bring up a direct connect but you can usually spin up an NFV of some sort in that VPC and You can spin up an NFV it might all you might art You know your your cluster might support VPN as a service for example Or you can just sort of ad hoc spin up an NFV and bring up an IP set tunnel, right? So you can bring up a path a network path between your work your your local workload and your cloud workload And you can kind of look at it as one big cluster and there's there are you know there are times when that's very useful You have to keep in mind obviously the kind of non-uniform access Your public cloud as opposed to your private cloud getting date up there might take longer it might take You know make make take shorter time if that's where your data resides. It might take longer to get locally But you know that's one way to do it right but but if you're really talking about cloud and again And you know if you're making your apps cloud ready, right? We're not talking about rewriting your apps at this point to make them truly cloud native But if you're if you're making your HPC apps cloud ready You might not needing in this extension, right? Because if you packaged it properly and you know containers Containers are also a big part of making it cloud native But just not having to install an environment every single time you want to go into another cloud If you can spin up new instances and make them available as almeas or or images on Images on your open stack, but if you can just use regular baseline images and do the least amount of possible to them Then it makes your life easier less you have to do so genomics for example genomics has like a hundred tools That are used in a hundred different ways for a hundred different reasons right and depending on what you want to do They it could take you forever to install all these dependencies and so you can you know You can solve that with Docker not not not containers for being able to run multiple workloads on the same instance, right? But really for packaging and for genomics in particular. There's a bio Docker Project and they've basically taken a lot of these tools and and and Docker eyes them with with a known sort of input and output type mechanism And what you can do is you can just call your tool You don't put your workload in the container But you just do a Docker run on that dot on that Docker container with the input and the output And you don't have to worry about doing any of your tools, right? So this takes off a lot of the burden from you about trying to make the the target environment Useful for your application you just make your application portable So look to wait look to ways to make your application as portable as possible. Don't try to tailor to the infrastructure And then storage again, we kind of talked about this before but if you use object stores like Swift for example You can make that available from your institution or I mean you can use it from s3 Or you know object storage is pretty plentiful nowadays There's lots of tools you can use interact with object storage if you make it so that you can pull that object from anywhere and Put that object to anywhere. You don't have to worry about mounting file systems And at the very you know at the very worst you you could you can fit you spin up a virtual Cluster like we talked about in that cloud to temporarily use it and then pull it down, right? Because again the public clouds that their their economics are hard to beat when you're talking about spot price And you're talking about just burqa boast burst workloads. So assuming that it's ephemeral anyway You can spin it up do what you need to do and bring it down, right? And then and then launching, you know, if you use infrastructure Infrastructure agnostic ways to launch your jobs and bash is one of those right? It's not necessarily the most flexible and and capable. I mean depending on your you know your your your level of bash awesomeness But you can call they all have CLI's right So you can call the CLI of any and you can you can choose for example to use the the EC to API for your open stack cluster and just treat everything as an EC to something and just point it to different zones whether it's or Different regions whether it's a local one or or a remote one Cloud formation you can use that instead of heat now But the problem is when you get into these specific templating languages I understand that they can kind of cross pollinate in some sense, right? But if you start using infrastructure specific template and you're kind of on that infrastructure And so it's you know, that's what you want to do That's what you want to do, but you might want to keep an eye towards infrastructure agnostic agnosticism And elastic luster right so that's one of the things we liked about elastic luster You can just sort of point it at a different back end you do the same thing every single time You don't have to worry about much of the infrastructure. It takes a lot of that worry for you And so really you know hybrid cloud is not really all that difficult you again You can pin up the infrastructure But you don't necessarily need to if you do your applications properly and then Kind of last is is are the best schedule some of the bat schedulers actually have mechanisms for hybrid cloud, right? So some of them are you spin up some instances in Amazon? For example, and you put them to sleep and then when the bat scheduler gets to some sort of threshold it'll spin those up and And launch some workloads in them and spin them down, right? So so your bat scheduler could actually help you With with private cloud That's what we have for you today Hopefully that was useful. Hopefully this is a a pragmatic approach to doing science in the cloud that either you can use For your own work or maybe that you can help some of your your customers use Love any questions any comments? Just to kind of steer our work in the future So thanks for your time