 Okay, now I know I'm good to go. Thank you all very much for coming along to hear about our new HPC service, Spartan, and HPC Cloud Chimera. So I'm gonna start off, my name's Bernard Mead, I'm gonna wake the computer up. From the University of Melbourne, as are my colleagues. So I'm gonna then pass over to Lev, who'll pass over to Greg and Lynn. I'd also like to acknowledge a person who's not here, Daniel Tesello, who helped us with a lot of these slides. So, I'm sure a lot of you are fairly familiar with the background to HPC and Cloud, but give you a little bit of a context as to focus the conversation. An HPC system is designed to provide an excellent performance. You're trying to get every little bit of power out of the core and every little bit of performance out of the system. Fantastic for multi-node distributed jobs. This is what HPC is about, and people who build HPC systems understand that, and they really try and make their systems work. They tune them as well as they possibly can. Cloud, on the other hand, is designed to be flexible and cost efficient, has not got the same sort of focus as HPC at all. It's there to support a lot of different workloads and a lot of different people and whatever they're trying to do. So quite a different context. Computer is sure, but different context. So, at the University of Melbourne, we had our general-purpose HPC system called Edward, and it was getting on in years, ready for retirement, so we had to look at what we were going to do to replace it. The obvious thing would be to just buy a new system. It had been very successful over its time. It's not a big system, but it had supported 886 users, 371 projects. It had done a reasonable job. From a university point of view, that's not a big system, but it wasn't bad. Now, there were some specialist queues, so some departments would buy in their own hardware that they would have attached to our system that we operated for them. So, you know, there was a lot of people counting on what we were going to do next, and we decided we were going to change things. The question is, why would we change? We had a system that worked, why not just replace it? Well, there was a big debate about the value proposition of HPC. We have a number of peak facilities in Australia, not a lot, but a number, that all of our researchers could apply for. And so, we were asked the question, why are we providing an HPC service that's really not a big HPC service? Why not just let them go to the peak facilities? Why invest in our own? And I should also point out that when we bought Edward, the Australian dollar was doing very well. When we had to replace it, the Australian dollar was doing very poorly. So, I wasn't even going to be able to replace it one-to-one because of that significant price increase, I guess, from our point of view. So, we started to have a close look at what Edward was actually doing. Okay, we can ask people what they do and what they want. We do surveys, but we looked very closely at what was actually happening on Edward, and we found that it was very heavily biased toward single-node jobs. I think it was something like 90% of the jobs were single-node and 75% were single-core. We had very little, tightly coupled workloads. It was pointed out that part of the reason for that, of course, is that we didn't provide a system that was really well-designed for that. So, we asked people, what do they want? And they all say the same thing, more cores, more RAM, bigger, better, faster. Everything is always that. It is the solution that they come to us with. But the reality is it's not the solution we need them to come with, it's the problem. The real problem is they need to get their jobs done faster. So, more is not really more. What they actually want is short-accused. They want to produce their research quicker. So, no matter what they say to us, we know that that is what they're actually trying to do. Some of them put it that way, too. The other thing is the access to the peak facilities is not trivial. It is quite a... There's a lot of restrictions on what sort of jobs you can run, and they have to have been profiled and all of this sort of stuff. And we also find that our national facilities are oversubscribed, too. So, people were waiting weeks for jobs to go through to start running. So, many researchers would put in their grants, computation resources, and they would go and buy their own system, put under their desk, put in a cupboard. I had people who were telling me they had a machine room and they had to put stuff in it or they'd lose their machine room. It's not a great justification for a machine room. The other thing is, this is why change, that we have a very big investment in our research cloud. There's the Nectar Research Cloud, it's a federated cloud for the research community in Australia. Eight nodes, which Melbourne Uni is the first and the biggest, and we've got 11,000 calls on that cloud. So, we were asked, why not just get the people who used that sort of workload to use the cloud? So, we started planning. There are a couple of ways of designing an HPC service. And I'm sure there's a lot of different ways, but I'm just focusing on these two. So, you can focus, the way we do it at the university is we find a couple of principal use cases, usually about four or five, that are important, that we can put a really good story around, and then we can convince governments and investors to back. And it's usually led by some academic champions, people who are very high up in their field. And that's how we get the funding. The trouble is, we design a system that suits those use cases very well, but 90% of the actual usage is generic use. So, those few use cases love the system, but then they've got to compete with the Roy Palloy, who are coming in and running MATLAB jobs to all that sort of stuff. So, that kind of frustrates everybody. The other approach is to design a generic system that suits the 90%. The trouble is we lose those special cases, and those special cases are actually really important, because they usually come with money. The professors who have their pet projects who decide where their money is going to go will back at the department that says, oh, we can build our own system that is designed for your job. And so they build a private cluster. And then three or four years later, that professor's moved on somewhere. His PhD students have graduated, and we get called in to find out what we can do with this aging equipment that nobody knows how to use. So, Spartan was born. What were we going to do with Spartan? We're going to build just a cluster. We're going to mix the bare metal and the virtualization aspect of the cloud. One of the biggest things I had to get across to people that we all had to try and convince people is that cloud did not automatically mean virtualization, because the first thing you say when we're going to run a cloud service for HPC is, it's going to have such a big impact on your performance that it's not going to be even remotely like an HPC system. Then you explain that a bare metal version of that can be every bit as performant as a traditional HPC system, except that you have the benefits of the cloud. And all those workloads that don't work that get in the way of a bare metal system, well, they can go on the cloud. So, we operate the service. Spartan has operated as a service on our research cloud node, which gives us a whole lot of benefits like we can migrate the VMs, we can snapshot them. We've got all of these sort of features that the cloud provides us, but these management side of things aren't needed for the tightly coupled bare metal service. So, that can perform every bit as well as it should and we have the cloud service. The other thing we're trying to point out is that all clusters have to have some base requirements as management nodes and login nodes and networking and racking and all that sort of stuff. And every time someone goes out and buys a cluster, they have to buy that stuff as well and they have to have people to operate that stuff as well. So, having a centralized service would allow us to have a better value proposition for the whole university, including those departments that want to buy their own. The other benefit of the cloud, as I mentioned before, we have 11,000 cores, not all of which are used. When you run an HPC system, if you've ever done this, you'll find that it gets saturated the day you turn it on. It's full straight away. People start queuing up. Cloud, on the other hand, people start using it, but if you notice the actual CPU utilization, usually really low. I always look at that thinking I could put HPC jobs on that and now we do because we can scale up. The actual size of our cluster, or so Spartan, is dynamic. It will grow whenever we've got the capacity and as people release their cloud resources, we can expand Spartan again. But then as the demand comes back for the cloud, Spartan just scales back and our queues get a bit longer. So it goes up and down. And that is a really good way to pitch this to management who I wonder, how do you justify that cloud resource? How do you justify the HPC resource? We don't know whether we're getting good value for money, but if you can show that there is a dynamic way of apportioning it as it's demanded, that works out really well. Management respond very well to that. We also look at how we expand one of these systems. When you build a new system, you build the whole thing, typically. That's how you would approach it. But you base this on use cases and what people are asking for. We're actually basing our investment on where the pressure points are on Spartan. If we've got a lot of people pushing on the memory side of things, we buy more memory. If we've got more people wanting GPU, we expand the GPU. So we don't actually have to just plan the whole thing ahead in three years of what it's going to look like in three years' time. We build something that we can grow organically as people use it in whatever way they use it. It also has answered this question of, why is our research cloud not getting... The usage is very low. The CPU utilization is around 5% to 10%. And we've got questions about that. The fact is that doesn't really matter, but when you're being asked to justify how much money you spend on one of these things, you kind of have to answer the question. But Spartan, because it's an HPC workload that we're putting out on here, it fills it up very quickly. So our usage goes up. It looks much better. The other thing is, with this approach, we no longer have that commitment to a physical infrastructure. We're treating this very much as cattle, not pets. I know that's a term everybody's familiar with now. Basically, we don't care about the underlying hardware. We can migrate our system to wherever is the best place for it. As hardware dies out or gets replaced, we just roll in new partitions. And as the host for Spartan runs out or dies or whatever, we just move it to another host. So theoretically, we don't have to have that big single burst of capital investiture to build up a new system. However, grunts are still the way for a lot of departments to buy their own kit. They still want to own stuff. So we provide them with exactly what they want. They can buy their own kit, but they can save on all of that infrastructure that surrounds the overhead infrastructure that surrounds an HPC system or a cloud system. They can, and it's always very nice to pitch this to the department managers, they can buy stuff that might be HPC one day and then become cloud another day. They never do that. Reality is they never actually do that, but you can tell them that that's possible. And so they feel like they're actually able to control their investment a lot better. The departments can be given their own private partition and they feel like they've got their hardware. But they can also participate in a sharing. They get a priority share. So when they're not using their hardware, and we find this happens a lot with departments, they'll buy in and then they think they're gonna use it all, all the time, but they don't. And so it sits there quite a bit at the time, not being used. Well, we can just push more jobs onto there. And then when they do want it, it becomes theirs again. It's very easy for us to do that. It's a really nice way for them to become part of the community. This is a diagram we came up with to sort of give an idea of how we were gonna put all of our bits and pieces together in under OpenStack. We've committed ourselves to OpenStack for the research cloud. And we had all of these pieces that we wanted to put together. So we had our storage. We're gonna put in this fast storage. We've got another service called Vicnode, which we have incorporated under this. We've got our HPC bare metal, our HPC virtual, which is our cloud burst partition. And we've got over in the various side of here, we've got a number of trial things that we put in GPU nodes and we're experimenting in this sort of area. But the whole thing is all managed under OpenStack. And it means that we have that same DevOps team who, well, they weren't experts in HPC, but they were experts in infrastructure. So they were able to step up and manage all of this for us. We didn't have to hire new people. So this was how we came to the idea of what Spartan should be, how it could be and how it could work for the university. So now I'm gonna pass over to Lev, who'll tell us how we actually did it. Just for a couple of slides, I will be back, however. So the plan, the thing about plans is that they very rarely survive first contact with reality and sometimes reality can be hostile. Things are against us. 2014, starting together requirements because we know that Edward, poor old Edward, incidentally Edward is named after King Edward and the system prior to that was Alfred and the systems that were supposed to come after that would have been Affelston or Afflewood and nobody wants to SSH into that. Requirements gathering, 2014. We know what's gonna happen. Spotlight search. What is this? There we go, that's better. We also had a neighbor, the Victorian Life Sciences Computation Initiative. They had a large system and they've come on board to help us out in that requirements gathering and they led a fair bit of the technological decisions or contributed quite significantly as did other research institutions such as Monash University. We got ourselves some external architects to help us contract that solution design that you just saw on the previous slide. So we moved from that particular planning, that planning phase to actually implementing it and what sort of things do we have? Well, one of the things we really need was an extensive training and transition program because that first slide that you saw there about Edward's users and projects, those are the current active users and projects. All right, so there's actually like a certain percentage which have retired over time and they're used to using PBS Talk and Moab and they would need to be trained up about how to get this into a slurm-like environment which we're using as our workload manager, scheduler and resource manager. We also find that we have a whole bunch of researchers that have been raised in a generation of being very used to graphic user interfaces and a surprising number that have got very minimal exposure to the command line which they're going to use if they're going to run HPC jobs. More advanced users like the opportunity to get some slight knowledge about shell scripting for HPC and even beyond that parallel programming. So this was a big component. We spent a day a week running classes to bring people at Unimel when up to speed on this stuff. One of the other aspects of the implementation, we started off with the campus active directory. We decided to go for a local LDAP fairly soon after that when we found there was a few issues with that particular campus local directory. Certainly the implementation which had a fully qualified email address as the username caused an issue because I would have to log in as SSHL Lafayette at unimel.hpc.au at spartan.hpc.unimel Yeah, that wasn't a good thing. So we ditched that fairly quickly and also it enumerated groups fairly poorly and fairly slowly as well and none of that was fun. So we moved away from that. There was one particular and extremely important application which was causing some noisy neighbor effects. So in that first slide, you see how there was going to be like a VHPC where you had a one-to-one commit and a whole range of more virtualized with overcommit. Sure, that works quite well in most cases but for certain HPC jobs, we found that there was a bit of a time mismatch and certain applications were not very happy. Now, to an extent, we kind of knew that there would be this issue early on because obviously when you're running a web server in the cloud, what's the CPU utilization? Usually quite low. When you're running an HPC job, what's happening? It's chugging along quite heavily. You know, you're talking about 99%. In some cases, that overcommit made a difference. So we had to turn it off and as a result, we're doing everything at a one-to-one. And as you can see, after we had our VHPC, we decided, well, hang on a minute, let's make use of Slurm's crowd-bursting facilities. Slurm, of course, has got this power-saving feature which in HPC environment, if the node's not being used, you can turn it off and then turn it back on. They've extended that so when you turn it on, it can actually access a cloud service and by default, it's an Amazon one but we've got our own and so we use that as well. So our entire cloud petition is a makes use of the bursting feature. And we've also got connections as illustrated to Vic node. That's like, you know, a larger data storage service for the state. We've also got specific IO nodes for data transfer because nobody likes being on the log-in node when people are moving huge amounts of data into their home directory. Now, I think we're going to move on to Greg now. I'll be back. Okay. My job is project manager slash infrastructure manager and I had to pull all this together. So the first thing I did was read up, not having worked on HP system, I read up what HP system HPC system was. So what should it be built from? Dedicated hardware, Finneban networking, parallel file system like Lustre or GPFS, rigid upgrade, patching policy, and lots of money. Now I didn't have any of that. So it was quite a challenge moving forward. So how do we design our system? Quickly, we had a very small budget, lots of unused cloud compute capacities as you've just heard. So what we did was, the whole HPC control plane we implemented in the cloud, we used cloud compute for low resource intensive jobs as we've just gone through, bare metal for high intensive jobs, commodity SSDs provide the high performance storage for MPI jobs and quickly, we use all our cloud CICD environment which is managers configs upgrade, patching. So we basically used everything we could on the cloud and basically just added an extra bit of bare metal and some SSDs. And that way we were able to keep the price down. And just quickly, I want to go through the hardware. Bare metal compute, our initial consultants recommended fairly fast CPU but only low cores and it's about 21 gig of memory per core and a 40 gig network. Now that does work very well. I have some performance figures in a minute but we get very few cores. So we thought that was an expensive option. So what we decided to buy on our second round of purchases, we increased the core count. Obviously we dropped a bit in the CPU speed. Memory was a big thing. So users, one or more memory, double the memory and we dropped the network speed down. 40 gig was quite expensive, 25 gig is much more, much cheaper to implement. We also bought a few specialised processing cores. The new Xeon PHY, you might have heard of, comes out with 64 cores on diRAM so we can do very fast processing of very specialised jobs. And lastly, we have a number of GPU nodes. The standard Nvidia Tesla K80. And they're used in... So we have those four queues at the moment and we prioritise jobs from users to go into those particular queues manually at the moment but in the future we hope to automate. And quickly on the cloud compute we have, we use the Dell C3, 6300, 6320 modules. And that's a very similar processor. Running a 2.1 gig, 16 cores, lots of RAM. So as you can see, not too much different from our bare metal 10 gig network connection only. So now I just want to cover what specialised networking we put in. For the bare metal network, we went the Melanox 2100. It's a very good platform in terms of price performance because it gives you 16, 100 gig ports and with breakout cables you can get up to 64 port connections for your computers. So that was a very good price and you put two in a rack and the only one you wired and you get 128 connections. So two small switches can take your whole rack at very high speeds. We might be using 50 gig for our storage and 25 gig for all our compute. So you can vary your connections depending on what the requirements are and 100 gig up. We run RDMA over ethernet and we're running Cumulus Linux OS which we're commissioning next month hopefully and we went that way because it fully automates in with our cloud and we can create a full virtual network environment without paying any more money and we're now considering routing from compute nodes no more VLANs. We go into performance just to see what kind of performance we get. We use something called the MPI ping pong test. We tested it on, we have a large HPC service using Melonox 56 gig in InfiniBand. We have our legacy Edward service, we have our cloud and then we have our specialized Melonox high speed network. So they were the four networks we tested it on and interestingly here are our results. The InfiniBand does quite well as you can see. The old Edward system is probably a factor of 20 more slower but once we look at the cloud and unfortunately the cloud is considerably slower for latency. We get to implement SRIOV but even if we get SRIOV going which is bypass on the hypervisor we will never get any lower than 19 microseconds anyway. Some jobs will suffer for that but a lot of jobs won't. Interestingly when we look at our bare metal before we turned on RDMA slight improvement with 40 gig network but RDMA at 40 gigs even beats InfiniBand. It was very impressive. The alternate speeds at 25 and 100 not much difference so we thought the difference doesn't justify the cost so we decided on the 25 gig is our next path forward. We currently run 40 and it's quite expensive because it takes up 100 gig port. You notice the numbers aren't consistent with the speed as well and that is because 56 gig is done in software on the switches and slows things down. The 100 gig uses better error correction and that slows things down so the best latency at the moment is 40 gig. It was quite interesting the results. Storage, for the bare metal nodes our scratch storage, we need some storage that was very fast. So we bought a commodity server and filled it with SSDs. It was eight terabytes of storage and we shared it out with NFS over RDMA. For single node jobs where we don't need to do two MPI we just use local PCI SSD cards. Obviously lots of performance there. For the cloud it's just the CFRBD back end and of course project home we have some of the legacy net app storage and we use a SAS aggregate there to provide the storage mainly for their project data. You'll notice we're planning PNFS money with so many nodes trying to log on to an NFS that overloads it so we're looking at people learning PNFS in the future. And finally we're trialling a Sandisk 128 terabyte SSD and that's going to give us quite a large boost in performance and more storage and we'll present that through CFFS. Interestingly I've got some figures for the performance just going again for the SSDs and these are just commodity 800 gig SSDs from Dell and we use a flexible IO test transferring about 85 giga data and as you can see the throughput results are extremely fast. Once we turned RDMA on it basically saturated a 40 gig link. So if you're ever considering putting a lot of SSDs into a commodity server and serving it out Tangi will not and running RDMA Tangi will flood in no time. You have to go for the high speeds. Latency improvements as well was a probably the big factor was latency didn't improve that much on RDMA but the consistency of latency. So there's a much bigger spread with TCP which you'd expect, you know it's windowing up and down on depending on what the workload is and the spread of latency was quite large but with RDMA turned on we got a consistent latency figure around 750 microseconds. And finally the end result we ran some benchmark tests on different jobs to see what the performance looked like. I've got a disc intensive job to start with. Once again we're talking about our traditional super computer there with all the attributes I described before which we don't have money for. It delivered quite a good before you know one hour 18 for a job. The bare metal actually outperformed it no problems at all. Surprise. Obviously we're running faster CPU but with the network and with the SSD disks we get performance better than a super computer. It was quite a surprise. Obviously we can't scale to that size but performance wise for the size we are we get a good result. The cloud, yes, is a bit slower. Particularly for disc intensive jobs but once we go into just compute intensive for a little data in and out the cloud performs just as well as a super computer. So you have to be very careful of what jobs you want to run on which platforms. Otherwise obviously our bare metal is much more expensive in our cloud and less flexible and it has a use for particular jobs but for some, for a lot of the other cloud jobs you can see the results for the Gromax it's just the same so. But unfortunately even in the same type of job running like in the dynamic simulations there's two different packages there which are Gromax and AMD and the results can still be quite different. So we basically have to come up with a catalogue of jobs that we run and we'll benchmark them all and then advise the users this is where you should be running your jobs. And obviously you all want to hear about Spartan on OpenStack and I'll just hand over to Linvue. My name is Linvue, I'm the dev of Steam Lead. One of the philosophy of our team is that we don't do premature optimization of anything we built. So with Spartan we started with very simple configuration and build it up, scale it as user needs. So the cloud bursting part we built based on a power saving hack in Sloan. Sloan doesn't really have proper support for cloud burst, it just has resumed program and suspend program scripts that you can use to do something with power savings. Fortunately you can use the same scripts to basically call OpenStack commands or API and to build VMs bursting the cloud for you. We currently using an Adcore 64 gigs custom flavor but those things can be changed based on the profiles. So we just started something really simple and based on the profiles we get from our users, we adjust. We have a golden node built and as publicized and periodically we create an image and sawing clients that we use to build all the rest of the cloud burst nodes. The build time from the cloud, we optimize over time and now we get to about 30 second build time after a sketch and currently we configured the cloud node to idle for five minutes before it gets suspended and deleted, release the resource back. This incidentally help us to roll over to new features, new kernel updates, as we're doing now today actually, the dirty cow kernel fix and other performance improvements quite easily. On other occasions like if we do a big OpenStack upgrade we can change this to say a day so that the control plane, if it goes off we can still have nodes on jobs. With bare metal, currently we're building using our custom pixie kickstart bus popup but we're investigating ironic very heavily at the moment because we like to do the rolling patch upgrade and build the same way we build our cloud nodes. And we use popup heavily to everything that's reviewed to Git and Garrett with Jenkins for automatic testing. And we have new image, basically automatically created and tested every two week cycle but that's just a default configuration. Like recently with the say the dirty cow upgrade we just did in a day. When we start building the cloud burst capability in Spartan, we were on Nova network going to just the first stage of neutron upgrade so we didn't really have any fancy features like floating IPs and so on. We just have fixed IPs and fortunately SLURM cloud bursting capability is a hack. So you have to tell SLURM exactly all the node names you're gonna have in your cloud bursting partition. So it makes our jobs fairly simple in that we just have to create a script that regenerate all the cloud config and SSS host keys for the Kubernetes going to build which takes no time to create a few hundreds of those. And once it's all created we can just put in a script to build a node for us. That does make the build time really quick and actually reduce the probabilities of things going wrong during the build. We might look into more better features of neutron once we finish the next stage of our upgrade but for now that seems to work really easily. And of course we are looking to ironic to do the same thing for bare metal side things. One thing we going into at the moment is a lot of, based on because SLURM elastic computing feature is a hack. There's a lot of bugs going on around that which we're working with SCAT MD which is the one to manage SLURM to fix. It has many weird things that can happen with your cloud VM once it gets built or gets deleted. But fortunately, SLURM is very resilient and the jobs will recover and get passed on to the next node. If the first node is failing to come up on time. Okay, so as mentioned we're using SLURM workload manager for the resource manager and job scheduler. For actually installing the whole range of software applications, I mean one of the more difficult things with I guess I suppose a traditional cloud installation is the fact that if you've got a researcher that's got to build, if you've got more than one researcher building the same application, that's lost time. HPC systems traditionally have like a big centralized repository of sysadmin built and optimized systems. We're using EasyBuild, it isn't always easy but nevertheless we're getting through with that quite well. Building from source codes, want to get passes like layers of abstraction, we've specified tool chains and dependencies, quite good actually in that regard. Once you've actually got the build script working, you've got yourself a consistent build script and you can modify that. And another thing of course is it automatically generates, by default it automatically generates Lmod files. Now Lmod files, LuaMod, have got some small benefits over the traditional TCL based modules system. The environment modules can be dynamically assigned to so I change the user's path. Lmod will provide a hierarchy of modules as well. Also it allows for you to automatically change so you don't actually end up with multiple versions of say GCC in the user's path. I gave brief mention to our training program before, as I said, one day a week is set aside. We do have these four regular courses running. These are small courses, they have about 15 people. We tend to really want to interact quite closely with the researchers and we think that other people should do this as well. We get really engaged in contemporary androgogical techniques. These are adult learners, these are advanced learners, they're smart people, even if they don't have a great deal of experience. They're post-grads and post-docs, they will pick it up and will pick it up quickly. They just need a bit of guidance and a bit of helping and a bit of hands-on. So they get through it fairly well and they get quite good at it. We also have been asked to do some application based or rather departmental based specialized courses. So the economics people want to have a specialized HPC course on R and Starter, Maths and Stats people, and of course our mechanical engineers want open foam and as mentioned, profiling courses are on the agenda as well. So that sort of gives you a bit of a bird's-eye view of the way that Spartan operates. We're really talking about a system that has profiled itself according to the users. That's a quite a big deal, rather than actually saying this is the technology, we're saying, okay, what is the technology that we can use for the users? And most recently we've said, oh look, there's a big protonomics group that have come on board and they won 1,000 cores for a couple of months. Well, we've created a partition which suits them and that works quite well. Coming up in the future is the final closure of poor old Edward after five years of faithful service. There's a relatively small HPC services going to be closed down and we suspect that we're going to get a flurry of people coming in there. By the way, it's just a complete tangent because now's a good moment. The background image is actually the ruins of Sparta. We're also looking at actually having separate new partitions according to different hardware and even login nodes. One of the things is a master's level course that runs at the university and once a year there's a sudden influx of a large number of users and they always do something wrong because they knew at this, that's okay. If we give them a separate login node, at least they only damage their part of the world for the short period of time before we catch it. And here's a really interesting bit. We actually think that Spartans architecture will serve as a model as general purpose computing in the academic and research space. We think it's really important that some people who require that high speed interconnect for distributed memory jobs have got that available to them and those sorts of jobs do not conflict with the larger number of single node jobs. So by having those separate partitions but operating through the single management system, people, we get the most effective use of resources. So because we've come from Melbourne and it's a long way from here, I came here a little bit earlier and I went into a bit of a tour which included several other institutions along the way and just to sort of find out how they do things and whether they do things differently to how we're doing things and what we're doing. And we came here with the sort of thought that, hey, we're doing something really quite unique and special here. And of course, some of these systems here are top 10 systems in the world. We're making us around about a 1% or between 1% and 10%. Some of these are national facilities. And one of the smaller groups we discovered, relatively small, and the people at Freiburg have also got a hybrid system. They too have got a system which is part HBC, part cloud but they're doing theirs a little differently and this is a different model that you might want to think of. Whereas we've created a shimmerer, a multi-headed beast where we have different partitions running according to whether the user needs to have a multi-node or single-node job. They've got themselves a cyborg which is a different model. They've actually got an admixture of the two technologies. They've got one great big Q, effectively, and they have an overclient that runs on the HPC nodes. So the HPC compute nodes can instantiate a cloud instance instead. So that's a different model that you might want to think about. Anyway, thank you, everybody, for your attention and any questions? You're all convinced? We're going to see hybrids everywhere? That's a good question. Why wouldn't they? Look, okay, there is a certain percentage... I guess there is a certain percentage based... You might know Greg Wilson's, rather than the Tories essay, high-performance computing considered harmful. And he suggested that there was some who were obsessed with the performance metrics. And performance metrics are great but they forget about the users. So what we did, which I think was quite interesting, was actually looking at the user profiles. Why pay for a high-speed interconnect if 50% of your jobs are not using it? Why not reduce your high-speed interconnect cluster size to reflect the proportion that are using that and put the rest on, say, a less expensive implementation such as cloud nodes? Yes, I think that people should and will move towards a model which incorporates multiple technologies that is appropriate to actually getting researchers' jobs done in the most highest throughput manner. Ooh, now questions, yes. Sorry. Sorry? Extend a tour to the Americas. I've asked Bernard whether he can pay for another trip. I can't say I've actually encountered that need, to be honest. I'm trying to think, yeah, you know, we have our route accounts which, you know, CIS admins have got access to. We have user accounts and we can restrict software accordingly through group permissions. But we haven't, I can't think of any circumstances where we've actually required users to have route access. Yeah, yeah, yeah. Okay, so for particular cloud, for Nectar cloud instances, yeah, go off and, yeah, they can do whatever they want to over there. Yeah. Yep. Yes. Australian Access Federation. Yep. One other thing, of course, we should mention, we should pitch Kragi as well. Kragi is a project that was developed by, initially, by the Victorian Partnership for Advanced Computing. It's an open source project and that provides a cluster and user management interface and also does a reporting tool on what percentage of the cluster is being used over time and things like that. So, you know, that's available on GitHub. If you're running a cluster, download and give it a whirl and see how you go with that. Yes, more questions. Yes, and interview. Yes, yes, if, yeah, no, if, okay, no, there's a number of ways we can do that. Of course, the most obvious way is to say, I've bought my, I've bought my, you know, small, small number, of course, and we can build that as a separate partition and slur. And, you know, it's available for you and you alone. You could also say, well, if I'm not using it, I can let other people use it as well. And, you know, basically, if I jump on and say, launch a job, I'll get a higher priority. Yes. Mm-hmm, mm-hmm. And I'll add to that. That's sort of actually that very specific requirements is really good for reproducibility of data. Yeah, that's it. Oh, look, that option is certainly there. You know, the specific, I mean, you know, one of the specific needs that we came across recently with that last, second last slide about the new Protonomics group that came on, they had a very specific need where they said, hey, we need, you know, eight gig of memory per core. And ideally, we'd have eight cores. And we said, well, guess what, we can do that for you. In fact, that's actually our preferred option at the moment. So we could actually change the configuration as required. So, yeah, look, you're right. And, you know, the option is there to actually provide very specific implementations of software which are according to the very specific implementations of hardware that they have. So, yeah, it's a good option for us. I think we did. Do we have to go? Okay, thank you very much, everyone.