 All right, let's get started. Hi, everyone. My name is Kamesh Pamaraju. I am the Senior Product Manager at Dell. Originally, I was supposed to give this talk. And as it turns out, fortunately for us, John Paul from UAB was going to be here at the summit. And I said, what a better opportunity for us to have the customer himself right here talking about their experience. So it's great for us to have John Paul with us here today. So I wanted to give you a very quick, brief overview of how we got engaged, Dell and UAB. So this was back in, I guess, 2013, John Paul? Early 2013. When we first started talking with John Paul and UAB around some of the challenges and problems that they were facing. So a little bit of a backgrounder. And then John Paul is really going to go deep into the kind of things that they were dealing with at UAB and how they overcome some of their challenges and what benefits they're seeing. So let me first get started off with a little bit of a background. So when we first started talking, I was, in fact, I was the first guy on the phone, if you remember, John Paul. We were on the phone with UAB talking to John, just trying to understand, what is it that you're trying to accomplish? What are some of your challenges? And this is kind of what emerged. So UAB does a lot of great work in cancer and genomics research. They get tons of funding. They had over 900 researchers. And one of the biggest challenges, it all started with storage, by the way. So we had a big storage discussion. And they had obviously growing data sets. And that was challenging them. And how many of you kind of run into this kind of issue where your data is distributed across USBs, laptops, USB drives, local servers, HPC clusters? How many of you kind of run into this data distribution issue? So this is what they were dealing with. And then they had this, they were using an HPC cluster. And they had to transfer data back and forth. It was just taking too much time. It was clogging their networks. And then, of course, as you can imagine, having distributed data management was a big problem. It was putting data at risk. It was causing productivity issues. So this was the problem they came to us with and said, hey, we have this problem. Data is growing like crazy. We've got all these researchers that have data spread all over the place. We need to get control over this stuff. How do we do it? So that's kind of how the conversation started. They said, we need a centralized data repository for researchers. They have compliance things that they have to take care of. And on top of that, when we started talking, we said, hey, at Dell, we also do OpenStack. We do OpenStack. We do data stuff. And they said, great. And we're also looking at OpenStack. And we want to try OpenStack out. And so at the end of the day, what UAB was looking for was a scale out, cost-effective solution on hardware that can be used for both compute and storage. That's how it all got started. And I know you guys sort of looked at other things before you came to us. They looked at traditional sand technologies. They looked at public cloud storage. They even looked at Hadoop clusters. And at the end of the day, after having the conversations with us, they chose us because we had a platform that gave them the best costs in terms of cost per gigabyte. And it also gave them a hardware where they can run both compute and storage. So it all came around Ink Tank Ceph, which you will hear from John Paul today, which is one of the things we had been working on like seven months or eight months before we started talking to them. So we were able to deploy Ceph. We were able to make it work with OpenStack. We had networking architectures, reference architectures Randy from Dell is here. He's been working very closely with UAB on the networking architectures because you need to tie everything together and have it deployed in an easy way so they can use it very quickly. So I will let kind of hand over the baton here to John. And he will talk really all about what the journey they went through and how our partnership really benefited from getting these solutions to their users. With that, John Paul, thanks for coming again. Thanks, Kinesh. Appreciate it. Appreciate it. So as Kinesh pointed out, we have an active community of data consumers. But we didn't really get here overnight. We're in the HPC space. We help our customers, our researchers on campus really do computation and analysis on their data. And one of the communities that started growing out and which was kind of starting to bring this data problem to us was this next gen sequencing community. But we'd been engaged in high performance computing. And if folks familiar with that domain might know the distribution called ROCS. It's based on a CentOS fabric. And basically, it kind of brings an easy to build HPC fabric to your center and supports really easy scale out. You can just add servers that get to text them. And so we were familiar with this model of scaling up. And that's the model we like. We are certainly heavily invested in virtual machines. We'd been doing that for the past decade for various services. And we'd been engaged over the last decade on the theme in the high performance computing community was grids. And I'm not sure how familiar you all might be with grids. But they were basically the same kind of ideas that we talk in terms of cloud computing but kind of geared around the high performance computing community. Ultimately, they focused more on job distribution across organizational boundaries. But the basic ideas made what grids are or were, they were kind of the proto-cloud. And so we had been engaged in a variety of meta-schedulers and technology of how to spread workloads across different cluster boundaries. And one of those meta-schedulers was called Gridway. And the Gridway team ultimately kind of morphed their distribution model of jobs to actually being not just jobs but actual VMs and led to their development of OpenEvula. So we'd been having our fingers and toes in the water kind of keeping a pulse on the activity that was going on in that community. And a DevOps model was inherent in how we worked. So we were kind of primed for migrating to this environment. Nonetheless, there were still lots of questions and lots of decisions to be made. There's not just one hypervisor. There's not just one cloud technology. So there's lots of various things. And you need talent and skill and expertise. And so can we actually build from scratch an environment like this here? Of course, then we also have customers. And as we mentioned, our customers were saying, data, data. They had run into their problem, their big data problem. And we're realizing that this is kind of annoying to have to move this stuff around campus back and forth to the cluster. We didn't have any storage on our cluster that was designated for long term storage for large data sets. We had some scratch space available that was available for their compute jobs but not long term. And they didn't stop asking for computing. So we were well aware that even though we wanted to build out a data fabric, we needed to be able to also address the ongoing compute growth. So what this really is about is something called data intensive scientific computing. So with all these kind of ideas in mind, we essentially started off in August 2012 and bought another fabric for our HPC cluster that we were going to designate for some type of data intensive workflow that had computation attached to it. And so we bought a complement 10, 720 XDs. They were decked out with 36 terabytes of disk space, three terabyte disks, lots of RAM so that we could do computing and lots of cores so that we could do computing. And we were like, all right, let's do this, whatever this was going to be. And so we now had to integrate it. Well, quarter went by. We were busy doing some other stuff we had to deal with at the time but we were also realizing it's not easy to just, all right, how are we going to pass this through the organization and get adoption and get what we need in order to tie all these pieces together. And if you're like me, so Bob Cloud, Bob Cloud, I think the Cloud is a name in honor of him but he's my boss's boss or my boss, I'm not quite sure what the relationship is there but nonetheless, he went to Dell World back in December 2012 and sent me an email while he was there and telling me about some information that he was learning. And I don't know about y'all but when your bosses go to a conference it's a little bit concerning because they often come back with stuff and you're like, wait a minute, I was doing that right here, we've been talking about it. But this email was exciting because I read some words in there that were I like. He heard about OpenStack and I'd never really heard about staff but it didn't take me a little bit of Googling to learn what I needed to learn about staff and that was a little gem in this whole package. And so I said, wow, that's good to hear. I've been thinking a lot about Dell in this scenario and specifically around the fact that we need some sort of a vehicle to help bring this into place. We had the technology, we had the know-how, we just didn't have the capacity. And so that brought me to lesson one. Be sure to recognize when a partnership's gonna help you achieve your goals. And that was, it was an important lesson for me but it was really nice to be able to turn to a vendor space and see a vendor speaking the same language that we were speaking and wanted to be speaking. So we weren't running into that situation where your boss comes up with a great idea that is now 100% different from everything you've been doing for the last 10 years. It was very much in line with where we wanted to go. And so we moved forward with an implementation. It was a very easy decision to say technologically this is what we wanted. The implementation had of course the open stack components, it had a set component. It was built around an automatic deploy fabric. It had ganglion there, it had Nagia's in there. It was just, it was comfortable. It was just familiar. We knew what we were doing in that space. And so by March we had committed to the fabric and within about six weeks we had the team from Dell come on site with Randy and his team. And at the end of a week we had our fabric deployed. We had our proof of concept in place and we had a cloud on campus. It was great. It was nice to work with people who knew what they were doing to be engaged in the same kind of space and that kind of thing. So the next step was to really kind of build adoption for this. The, we had a couple of different ideas. We had the, of course, the storage model that we wanted to promote and build out for the campus researchers. And that's the CEPH component that brought that in but made it possible for us to really tie our servers together, our storage servers together. And the image component within, or let's say the RBD image component, the RADOS block device imaging component was a key component of that CEPH deploy that let us bring this stuff together. For those familiar with Hadoop, Hadoop is this nice, wonderful way to aggregate storage in kind of a similar fashion and tie in computing. What we had in our space was we had a very traditional file system based user community that really needed to still have that file system abstraction on top of their storage fabric. And so that's something that CEPH was really geared towards providing for us. We could take, manifest an image across this 36, 360 terabytes of storage and just basically pull it together, slap a file system on top of it and access it via NFS. And that's what we ultimately get to. I'll have a slide about that in a minute. More generally, we were focused on also promoting cloud adoption with an IT and with the research community and then demonstrating its utility with application. And I think that's a really important component. Once you have a cloud in place is the joy and pleasure of being able to build applications on it. Along with researchers' demands for data, they could also express a desire for a better backup solution. One of the things that we say on the cluster is that there are no backups on the cluster. If you're not managing a data backup solution, you don't have backups. The reason for it isn't a disrespect to their data. It's more about the fact that the data owner knows what's valuable and a platform provider can't guess what's really valuable to the user. Everything looks valuable. And these people are dealing with terabyte data set sizes. And if we don't have enough storage to store their primary data sets, we certainly don't have enough storage to duplicate that and just make a backup out of the blue for it. And just to maintain backups for it. So we worked with a product called Crash Plan, which is a nice user-end user-oriented solution and started deploying that. It was a really easy couple of hours to deploy the required fabrics. You just kind of read through the specs and say, okay, I need one of these, one of these, one of these, and you put it in place. And then the vendor came on site, helped us get the dual bootstrap series and had that in place within an afternoon. There was nothing like, oh well, we have to go jigger with this or mess with this or we just kind of were sitting at our web interface and clicking through and taking care of the network requirements at the long and the same way as we were deploying the fabric. Another example that I deployed somewhat recently was GitLab. I don't know if you all know GitLab, but you certainly know GitHub. GitLab's like your own GitHub built out of some of the same models for, oops, excuse me, built out of some of the same models for basically a web page or website-based repository where you can engage and do issue management and bug fixing. That was really easy. I just went to the GitLab site and said, oh okay, so a server about this size, I need an X amount of storage. And just kind of hooked that up into the VM space and was up and running. As a side effect, a beautiful side effect or just a coincidence, it also used the omnibus installer and that has a Chef component for the automation component inside of the GitLab deploy. And that was a nice synergistic thing because the underlying automation fabric for our OpenStack fabric deployed by Dell also uses Chef currently. And so that was just, it was kind of a nice little cool factor. It's like, oh okay, so now I get to actually have that layer deployed that you see in the cloud. You know, your cloud provider does some sort of magic underneath the hood to make all of this stuff plugged together and deploy quickly and then you sit on top and you really have the same problem. You're gonna have some collection of infrastructure that you need to manage. And so now you have tools available and there's potential synergies if you're the same person doing the both ends of the equation or you can go your own way as a cloud consumer. But the storage component again is really nice in this. I mentioned that on the crash plan, we just kind of needed 10 more terabytes for backup storage. And so I just went to my cloud interface, I clicked the button, said create myself a volume, attach it to this machine and then I went inside the machine and said, you know, mount it and add it to the crash plan configuration. And that's all thin provisioned right outside of the OpenStack stack which is hooked into Chef. And so I'm just pulling my storage in from across these commodity servers and making it available to applications in my fabric. So big wins from my perspective. Our bigger or more effort-filled solution was the research storage solution. And that was, I say effort-filled because it was some in-house programming that we did to develop this. But it's mainly the concept is pretty easy to understand. We create image containers much like sand lungs inside of Chef. And then we attach them to a server which is gonna NFS export them to the client and we lay down a file system on top of the container and that lets the end user consumer really deal with their native file manipulation operations on the cluster and maintain this container with their data sets inside of it. And then the benefit of course is that those containers can grow to any size. We started off with this model of your first terabyte free kind of as a teaser to get people engaged in thinking and also as out of respect of the growth of data and research that a terabyte is really a thinking man's amount of storage nowadays inside the research space. You need to give people enough room to move their elbows and do some stuff. The basic architecture is kind of familiar to those of us who do NFS servers. The experience of course is a lot of fun with Chef and these kind of components. So we did some scripting to provision our storage and it was real quick. We did the deployment very quickly. Just we had about 110, maybe 120 users that were our initial batch. So we just broke them down into batches of 10. It was really easy. We scaled it out and just quickly manifested a bunch of thin provision containers across the cluster and within maybe five, seven minutes we had all of our storage online and available to our users. So I thought that was really pretty cool. I get excited about those kind of numbers as an HPC dude. And I also get excited about pictures like this. I tend to like my pictures look a little bit less choppy than the one on the left but that was my fault. I should have taken a picture when it was live. I let my RDD database basically go towards a more condensed view. But this is a picture of what happens when you need more storage. So I was mentioning that we had 10 servers that we aggregated with Ceph. Well, we got to the point where after making this research storage available to our researchers, go figure, they go put data in it. And they started moving data into it and I was watching my numbers and I was like, ooh, okay, so they're getting kind of close to a full point and so I need to get another server online. So I went and I plugged in my server and I said, well, let me just add some more data. And I was just kind of left off the, left it all in all the defaults and just kind of provisioned it with the provisioning fabric, the crowbar provisioning fabric. And well, I watched to see what would happen. And so this on the left-hand side, I know the numbers that are scaling up on the left are a little bit small but at the top it says 500 M and it's in bytes per second. So that's 500 megabytes, five gigabits per second. And that big vertical jump there is what happened when I said become a Ceph node. And so it said, okay, and it made itself a Ceph node and told the rest of the fabric, hey, I'm here and Ceph starts to rebalance itself. And so data just started flowing to this node and we have a 10 gigabit network so it's something about half of the network that we're talking about here. But a really nice, what this looks like in my world is a great job. And so it's basically scales up and you can see this is one of the OSDs that were actually being filled by this rebalancing effort. So we were about probably 75% full at the start of this operation, kind of probably stepped down to I don't know, maybe 60% I did actually went and added another server where we're below 50% again now. But each time I did it, I basically saw the same kind of pattern. A nice steady, this is a Nagios graph over here on the right hand side, a ganglia graph on the left. And we see just a nice steady data growth pattern in here and we see it going from essentially zero bytes on the disk to its eventual balance point of a little bit over, let me get terabyte and 1.3, maybe 1.4 terabytes. Of course then I went ahead and added another one and so everything we have right now is about roughly half full. So use it, that's what it's for. That's the nice thing about this proof of concept. We got past the whole rubbing our heads and scratching and thinking, okay, so how are we gonna get this in place to having it in place and being able to actually leverage it for the things that we were interested in doing. And so what that gets you to the point of doing is thinking in terms of cloud. You're no longer thinking in terms of your infrastructure, you're thinking in terms of what you can now do with it. And that's a really important transition that you need to go through and bringing your own private cloud into the space is really helpful for that. So of course you have to make decisions up front and you don't ever know what the right one is. So this is kind of like a little bit of a grab bag of some of the things that we did early on and how we've adapted over the past year from our deployment. Mostly it's been really good. The decision that we made going forward, I think I wanna point one out in particular, which is the sizing of your network. Don't think too small. Think in terms of what your network, your own network architecture looks like. In UAB we have two class B ranges, we're a large campus. So what we did is we said, well, why not just run a whole class B off of this fabric? So we went ahead and sized it accordingly. And it's actually benefited us tremendously because it allows us to map pretty freely over to the VMs and stuff that we deploy in the class, in the VMwares, in the OpenStack space. The other, let's see here. The other thing that we noticed is that, at first you're kind of cautious. You're like, well, this is all on micro provision. But there's ranges in there that are set aside and you can work and kind of hook your own custom interfacing nodes into the space. And it's a comfortable thing to do. And that gets you access. That's how we actually do our research storage is we actually created an NFS survey that's sitting right on top of our storage network and consuming the images directly from that backend network. And another caution would be to, gotta watch out for the people who are excessively paranoid. That's a good thing. I don't wanna say it's necessarily all bad, but you bring something unfamiliar into the environment and they get extremely paranoid. And so kind of be sure to, OpenStack is not crazy or anything like that. So it's a comfortable space. So be sure to stand your ground and make sure you get your deploy in that you want because if your fabric's inaccessible, it's unusable. So I think one of the fun things that I fixed was we had kind of a range of IP addresses that we'd opened up. And well, they didn't quite match up with what we wanted. So we use the fabric to fix it with getting some reservations out of the way. And so the lesson here is use the fabric. It's flexible. Use it to your advantage and make sure to let it help you solve your problems. But problems will arise. And we uncovered a couple of ones that were a little bit of a bump in our road. One of them, as we started looking at the performance characteristics of our environment, we were noticing with iPerf, that's how we test our network performance. We were noticing some sort of really jumpy and jittery performance on our 10-giggy fabric within the Ceph space. And we eventually tracked it down to upstream packaging of the Intel driver that was embedded inside the Ubuntu distribution that we had on our fabric. It was a fairly simple fix to just then update the driver and we saw a nice clean line at 10 gigabits per second or just a hair underneath 10 gigabits per second. But an important part of that is remember that there's an upstream to these and that there's a healthy community upstream. And you need to make sure to not just look at your vendor but look at the community upstream because this particular problem happened all the way into the kernel deploy space and how the relationship between the mainline kernel and the driver database, the driver integration goes. Now, this is not an issue in modern Ubuntu boxes so this is more of kind of a historical note and an educational note. But the other thing that is sometimes you go through these processes to fix something and you coincidentally break something else maybe accidentally. But again, you have lots of moving parts that might be a little bit complicated but there's places you can go in the open source community and you can see how this fabric was built together. It's kind of like becoming familiar with how your car was manufactured and knowing what the distribution network looks like and then being able to be deeply engaged in a primary authority on what you need to essentially change inside your fabric in order to fix it or make it better or we've heard a lot of the talks talking about how making sure that things go back upstream and so don't forget that there's an upstream and solid communities upstream to work with. Just work methodically, you'll learn a lot as you go along and respect the tool boundaries as you kind of work through this. It's not some sort of a magic environment. They really is a very disciplined transition between different components. And this is just for the folks who like to read Bash. This is an example of one of the problems that I ran into. I like it in particular because it is actually problem. I mean, that was the problem. It was one of my nodes basically went into a funky state and I couldn't figure out what the issue was. And so I had to go through and kind of step through the different spaces and this particular snippet of code is something that happens during the boot sequence and it essentially is going through and running a chef client to make sure that there's a sane environment. This is part of the kind of the, and by provisioning myself where I'm in operational mode step. So I'm kind of long story short. It goes through some of these problems but I mean some of these steps and if it can't succeed, well it does what it should do. It says there's a problem. And so it says final state problem and it reports that up to the management fabric and the management fabric brings your system red. So in the end, the solution was in this particular case is that was to either reboot the node or rerun this initialization script and get myself back into a sane environment. What I had done was I had rebooted something when one of my primary components, the administrative component was offline and I had set myself in a tailspin. But it was again, not hard to fix and the important part is that the code is the documentation and that's an extremely good thing is that you're not left, I wasn't sitting there looking at a black box and saying geez, I really kind of wish that I knew what was going on here. I'm kind of up the creek here and I don't know what to do. Well, I could start reading my way through it and really understanding what is the process and it's just a very logical process. What a system administrator would do. It would say, okay, well, I'm trying to bring this thing provisioned and it's not working so that there's a problem. Well, instead of having a system admin do that and my computer did it for me and let me know. And so the code is a good thing. So where are we today? Well, as far as the research computing system, that's the environment that we provide to our researchers, OpenStack and SAF are here to stay. There's kind of a no question about that. It's provided us with a lot of flexibility with respect to storage and it's also providing us with a lot of flexibility with respect to adding new workflow opportunities to the research community, specifically around models where they just want to, they're kind of trying to have some sort of new product that go online and they say, hey, look, someone's got this working on Ubuntu and I don't have to do as quick some buttons and I can make that happen. And so we can now give them an environment in the cloud space where they can spin up these VMs and kind of think through this. One of the key components that we've been dealing with I was mentioning early on that we're kind of like a, you know, we've been thinking about DevOps for a long time and trying to operate along that model with VMs. So we had our own kind of protocloud in place that was just like KVM and different bits and tied together. We built our Galaxy Next Gen Sequencing Platforms which is a web-based interface for Next Gen Sequencers to users to basically upload data, send it through their alignment processes and do post-processing on their data. It's a really nice little tool to get researchers up and running but we're running that right now in our protocloud, our KVM fabric. So what we want to do now is move that over to our OpenStack fabric and really put the keys in the hand of the research community that manages that fabric without having to worry about their relationship with the rest of our systems in our environment. They're going to be sitting on the cloud and they'll be authoritative. We want to add some object storage services. We started off with the research storage, the imaging component, but Ceph also has an object store embedded with it. Of course, there's Swift from the OpenStack fabric. We haven't seen a lot of our use cases really come from the research community leveraging that but one example is now Galaxy is starting to work it into their own workflow so that researchers can publish objects out of the Galaxy fabric so that'll be a nice tie-in. And then making sure that the researchers can actually touch this directly. They don't have to touch it by way of our own infrastructure on hands. But the big question is how far can we really take this? And I think from my perspective, the reason why you do process automation is scaling. That's what we've known in the HPC environment. It's what's needed to be done in traditional environments as well, whether it's a little research shop that starts off with their one VM and then says, well, now I've got 25 files, I'm really like 25 VMs because it takes an hour and then I can keep my computational time the same. The manual processes and steps are a real cost and as you automate those and document those through code, you begin to save on that. And then in my mind, success is really around dual use. It's about being able to satisfy your needs internally and your customers' demands and needs because you're documenting, I mean, writing code, you're documenting your own internal processes from a research perspective. It's repeatability and reliability. And then generally, we have the talent in-house. People can do system administration self-help and that turns into future systems developers. And traditional models are just ripe for replacement. So just be aware. So I'm hoping there might be a lesson five. Can we learn from research and engage as a partner? I think that that's a lesson that's actually very apparent here in the OpenStack conference, so. Oh, let me pass this back on to you. Okay, so can you hear me? Thank you. So, well, I have a couple of questions for you before I open it up for the audience. So you know that Ceph has a file system called CephFS which kind of helps you with scalable file storage that can be shared amongst multiple users. Sounds like you're mainly using this for block storage, like RBD, and individual users kind of put their own file system on it, right? So are there plans to go forward? Yeah, I'm very interested in what CephFS has to offer. It's a very, it's got some extremely interesting models around checkpointing and being able to just let the end user touch a file that then creates you a checkpoint. So from that perspective, I'm very interested in it. We're, of course, an HBC shop, so distributed file systems are ranked high. It's mainly been an issue of time and what we can explore over the past year that we haven't brought that online. Okay, perfect. So a couple of quick note here. There is a, if you want to learn more about OpenStack and Ceph, there's a deep dive session that's happening today at 2 p.m., room number 313. We're gonna have Neil Levine from Ink Tank who is their product guy. Really go deep into what's coming next in Ceph. And we at Dell have been working on creating reference architectures that'll help you sort of scope and plan your implementation and look at the kinds of architectures you need to build out yourself cloud. So make sure you come over to 313 at 2 o'clock today. Do we have time for questions, maybe one or two? Yeah, all right, go ahead. So you spoke a lot about, can you use the mic, please? Oh, sure. So you spoke a lot about operational efficiencies and can you speak a little bit about what you looked at in terms of compliance and data security concerns as you are moving to OpenStack? Yeah, so we did treat this as a proof of concept and one of the things that we were trying to understand was how does this fabric operate so that we can essentially engage in a compliance and data security conversation around it? We've, I mean it's an active dialogue on our campus. I don't see anything inside the OpenStack fabric from what I've been looking this year that doesn't fit within those models. It's really a question of how you're deploying it and who you're letting have access to run it. From what we've been seeing, it's a lot about the compliance issues are a lot about documenting your process and following what you've documented. It's a back and forth, it's a downstream and an upstream so it's kind of got this nice pattern to it. So I'm very confident that we'll be able to get that. Now there are features of course that within even our research community on campus, we have some researchers that are looking into actual provenance issues baked right into OpenStack that would hopefully be useful in the future within that kind of space. But all of the models around the DevOps and essentially coding in, baking in your documentation and your process, I think those are good, strong foundations that we can build on to get into our compliance frameworks that we need. Okay, any other? Yeah, maybe we have time for one more question, yeah. Seth seemed to have a feature where you can actually run computation on the safe nodes. I haven't tried it personally, so I was wondering whether it is something you will be looking at, what are the use cases, et cetera. Yeah, absolutely, that's my little dream right there. We mainly separated this out from a sanity perspective. The reference architecture that we were working on with Dell, as Kamesh mentioned, they had been working with Ink Tank for about six or seven months prior to our engagement. It was about, if you will, a kind of a clean room separation, and so we wanted to just kind of respect that existing distribution. But we're very, very interested in being able to consume the computational capacity that's on those nodes. We bought them for that purpose, for data intensive computing. There's some interesting models in the CEPH community around Hadoop and being able to lay a Hadoop file system on top of CEPH. I haven't looked into that deeply, although our NGS upgrades that we wanna do this summer are probably gonna investigate that more clearly because the alignment processes that they rely on, the tuxedo tools, I believe now support distribution via Hadoop framework. So it's definitely on the model. Backfilling with Condor is something that I think is very appealing because Condor's very good at just getting the heck out of the way when you've gotta do something like rebalance your cluster because you've got a new node added. And we'll be talking a lot about that at two o'clock in terms of how do you balance your clusters, what are the considerations that are on hardware, compute, storage. So feel free to come over there. I think we're out of time now. Let's give a big round of applause to John Paul. Thank you for coming.