 It's time to make a move. So thanks for coming, the rag-tab bunch of folks that did. I know it's probably the close to the last session for the day, so I do appreciate it. It's an unusual talk. So good evening. I'm Jake. I guess you'd say the tech boss for what is probably the largest neuroscience institute in the world now. It's called the Queensland Brain Institute. That's the URL there, if you're interested. Disappeared very quickly, don't worry, it'll be at the end. And that's us. We're part of the University of Queensland in Australia. Place called Brisbane. Shiny, sunny, lots of tropical cyclones, good stuff. So I'm pretty confident that 99% of you, maybe everybody, hasn't got a clue who I am. So I do appreciate you showed up. Especially in the open-stack space. You probably wouldn't have anything to do with me. So anyway, this talk is unusual. I hope you enjoy it. It's going to be very, very different from anything else you've seen at this summit, I daresay. OK? Right, so I'm not a vendor. I'm not somebody that is corporate. My world's really, really different. It's completely different. I would sell things to people for a start. I found it hard to write this. I think you'll understand why in a few minutes. I changed a lot of the slides just a day ago, in fact, as a consequence of something I saw just a day or so ago. So you'll get a feel for that in a minute. So as I understand, in any given year, in Canada, in particular, there's a very serious mental health and mental illness issue. And it costs the economy in this country $50 billion Canadian dollars. So I'm getting somewhere with this. And these are some photos, actually, from around Vancouver, believe it or not, of some individuals who have some serious mental illnesses. So here's where it got weird. Yesterday, I watched a guy in the street. You probably all know where that is. It's not far from here. And he was begging for some food. And I noted that. So I actually went up and had a chat to him. He was out near the waterfront. I think you probably all walked past that point. He was in real trouble. And I guess he, oh, hang on, where'd the presentation go? I blanked it. There he is. Cool. So yeah, he was in a real spot of bother. He was emaciated. He was in tears. He was shaking. And I was actually really worried about him. You probably think that's kind of strange. But did a single person stop? No. No one stopped. After 400 people walked past this guy, no one stopped. And I thought that was kind of sad. Anyway, I had a chat. And I asked what was wrong. And he said he hadn't had a job in 15 years because he had a form of paranoid schizophrenia. That's not an uncommon thing amongst the homeless in this city. So I've done a little background reading on the city itself. So depending on the study that you look at, and this is, it varies wildly, between 23 and 74% of people in Canada on the whole have reported a form of mental illness. And that's alarming, right? My own country isn't that much better in Australia. And among those, if you look at the unemployment rates, it's pretty high. So I don't know if anyone here recognizes that, but that's actually the slums of Mumbai in India. So before anyone says, oh, poor petal. He's probably just never seen a homeless person in his life. That's just life here. No, that's not quite right. I've seen some pretty bad things in my life. I've trudged through these slums in India. And I've been to Bandarache after the tsunami for other reasons, for aid reasons. I've seen some very bad things. So my point is that the world over, we've got a really big societal problem to deal with. It's not about our dashboard breaking or our glance image is not uploading properly or any of that. It's more than that, right? So you're all tech people, right? Why would you care? And you know, all you are is technology people. But no, no, you're all more than that. And I'll get to that in a minute. And I'd hazard a guess and make the assumption that there is somebody in this audience. I would almost be sure of it. I could put money on it if I were a betting man who has been touched in some way by aging or dementia or a mental illness, whether it be them, their family or otherwise, right? So it could have happened to you. It could have happened to others. But I'm pretty sure somebody will know about it. So here's some perspective, because I think this is important. You know, we've all achieved a lot in life. You know, we've all done a lot, not just stack-wise. We're technology people and we're unique and we're creative. Right? I think that's important. We should care because we actually have a lot of ability to change the world. They keep talking about it at this summit. Like, oh yeah, you know, this technology can change the world. Yeah, it can. A lot of the time it's focused on corporate money, on making better infrastructure and all that kind of thing. This is about a societal thing, way below all of that, right? It's about a real human thing. So anyway, to the point, what does QBI do? I'm gonna show you. We do neuroscience, okay? I'm fairly comfortable that everybody here can figure out what neuroscience is. But I'll go into a little bit of detail. So neuroscience is hard, right? Eureka moments are rare and things are so complex that we still don't fully understand a lot of the very, very basic things that happen in the brain, okay? It's a tough gig. So QBI is dedicated to understanding the fundamental mechanisms that regulate brain function. We do that via genomics, electrophysiology, cognitive neuroscience and world-class cellular imaging, even sub-cellular imaging. So we're getting right down there at a very, very low molecular level. We're trying to cure a lot of debilitative disorders and illnesses to try and give people a better quality of life in short. That's what we do. We're trying to help the world. Pretty altruistic, I guess. So you probably get it now, the link of why I changed my slides after seeing what I saw. I was gonna be very, very technical in this presentation, but I thought, no, I'm gonna go to the core of the problem, right? I think you get it. We're trying to stop that guy on the street from ever ending up there to begin with. But where the tech begins is here, the base of a mountain, right? So at the front here, which I'm pretty sure there's some people in this room who belong at the bottom of that frontier, you've been there before, things are hard. When you don't have a clue about how you're going to scale a big problem, things are hard. We've got big problems to solve and you'll see that in a minute and understand why. So we get stranded on the summit and I don't mean the OpenStack summit, I mean the summit of data and the wall of pain. We became a peak imaging body. We became one of the largest scientific imaging for biological organisms, entities in the world, okay? And that's good because we have some of the most advanced storage systems in the world to look after us. We use very, very, very high-end HSM technologies. We use a lot of flash technologies, things that most people, we'd be using most things, people that things people had not been using for years, we were using them years and years and years ago. So flash, we had flash 10 years ago, right? Maybe 15 even. Does anybody remember Texas Memory Systems? Yeah, so we had a lot of that stuff long before the free market knew about it. But anyway, it all fell apart when we tried to actually curate the data and start to work with it on mass. And that's a problem and you'll understand why in a minute. We went looking at our options and we could keep calm and carry on or we could panic. So do we just keep building bigger? Do we scale up and scale up and make this enormous data pit for everybody to work on locally? Do we ask for more money or do we panic? Yeah, okay, just panic, panic's fine. And we'll panic and we did, right? We worried and then economics got in the way. Our government, like the NSF, who's an American citizen here? Okay, so you probably know what the NSF is, the National Science Foundation. We have the equivalent in our country known as the ARC and the NHMRC. And at the point in time, they weren't really willing to give us any money. They said, wow, you guys have spent a lot of money and we're not cash rich at the moment as a country. So no more money for you. So I wonder what we were going to do with all our data and all our throughput needs. So after panicking, something happened. A thing called Nectar. Does anyone know what Nectar is? Well, okay, there's some obvious people in this audience who might. But anyway, the Nectar project happened, right? And the Nectar project, I'll explain what it is. There was a thing called Encriss, which was a fund set up to deal with these problems for us at a national level, which was a good thing. The government decided to deal with it. Nectar's aim was to make a federated cloud, an open-stack cloud, in fact, across the entirety of Australia in about six or seven different data centres so that we could all consume it in the research space. That was pretty much the core outline and it was pretty vague at the executive level to begin with, I'll tell you now, I was in the initial kind of talks. Tom's not here. Tom Feifeld, who some of you may know, he said he might pop along today. He was kind of the original architect of what got laid down as the Nectar open-stack environment. It was the bidding of this journey. It really was the beginning. He's now the open-stack community leader or community manager. So things started off slow and there were challenges. So why was it hard to consume? It was hard to consume because researchers didn't trust the concept to begin with. They don't like ideas like this. They like things next to them. Why isn't my stuff under my desk? Researchers don't like the storage proposition to begin with. It needs to be permanent. Why does stuff go away when my instances disappear? What's wrong with you people? I need permanent block volumes. So performance wasn't that well good at all. Things were edgy and not fast at best. And unfortunately, researchers are pretty perceptive of these things. They can smell problems. And they're people you should give a lot of credit to because they're pretty good at this kind of thing. They know what they like and they know what they don't. It took a bit of iteration. Continuous integration, that constant deployment concept we're all very, very familiar with in the way we work. And the community all pulled together. So we went very quickly from Havana and onwards to get to where we are now, which is Ice House. And we're now arguing about Juno, which is great. So at about that time, QBI got involved. There's a lot of text there. It felt mature at Ice House. It felt good. Unfortunately, QBI comes with a bit of a reputation for destroying people's infrastructure. We flatten things. We effectively eat planets of storage. If a weakness exists, we'll probably break your infrastructure. It wasn't a light decision. We've got a lot to be accountable for. It's $26 million in imaging grants every year through us. We're a bit of a gorilla riding a top-end elephant that barely fits in the room. Probably snapped your mother-in-law's sweet furniture. So it was hard for us to go down this path. But we had to have faith because the government told us we had to have faith. This is what option we had. It's a bit of a misnomer that suggests that research workloads aren't as critical. And they're not as sensitive as corporate workloads. I believe it's completely untrue and very inaccurate. If people tell you this, they probably just haven't been at the cold face of research. So what did we do with the research cloud? Well, we did this thing. Has anyone heard of the open microscopy environment? OK, a Muro is a tool that was made in collaboration between about 10 different universities, including us, with the University of Dundee in Scotland. This tool is probably the world's leading curation, processing, and image manipulation tool en masse, which lives in a web environment. And it's open source, and we all built it together. Proportionality is what we were dealing with when we were putting all this together. To give you some idea, microscopes all several, several million dollars each are the kinds of things we're using. About 50 different instruments across the institutes, generating about 100 terabytes of unstructured data per day. That's pretty big to us. That's not giant, though, because we've got genomic sequences which kick out a heap more. So we've got super resolution microscopy, multi-photon. We've got spinning disk. Some of these things might be familiar to some of you. Some of it might mean nothing to you. Either way, a lot of money, a lot of care, a lot of very sensitive instruments with a lot of big data. So the grand challenges are unstructured data, right? And when you dump that into a file share and think nothing of it, it's pretty hard to get back and it's pretty hard to curate. Also, it's pretty hard to collaborate on that piece of data there. I think that's 40,000 pixels by 7,000 or 8,000 pixels. It represents, I think, about 90 gigabytes of data in the one set of neurons. So anyway, this gives you a bit of an idea about some of the things we do with a Muro in terms of image deconvolution. So this is a single cell image of a neuron. It took, I think, I think it took about eight terabytes of RAM, 3,000 CPU cores, and about four hours to generate that. Using GPU with deconvolution, we can do it a little bit quicker. And this is the kind of thing that we're doing with a Muro. So that's, as I said, a single cell of a neuron, the thing that makes you think, the thing that makes your brain what it is. So we can do it manually locally, three or four minutes via a slidebook tool. You can do it in a couple of seconds with GPU. And we can feed it into a Muro automatically, which is quite nice. What makes our OME different, though? Well, we stacked it and we made it elastic. And this is kind of where we get to the crux of, I guess, the cool or the nicety of what we've done with a Muro. We effectively used Nova Boot and Cloudinit to make a scalable set of instances so that as things got loaded up, we could build it bigger. We use Apache ZooKeeper and GenX, Django, Java, bit of Docker here and there to keep instances and everything looking right and in shape. It's nothing you wouldn't have heard before. I guess the interesting thing is no one had ever done this before in the imaging community. So, I am, we did some other cool stuff, too. There's this thing called MuroFS, which is effectively just a posix representation of the file system underneath where the blobs get put, except it's actually a true representation of a file system. And we were worried because putting things in clouds, it has its problems. Things can go pop, stuff goes bang, so we did one special thing with it. We ended up splicing MuroFS, so it dumped to a storage technology we've got out in the background called Oracle's HSM, and it actually asynchronously writes back to a safety store in a local environment inside the University of Queensland. Think of it as a really big, WAN-replicated copy on write for really, really, really big unstructured microscopy data. So, what does a Muro look like? Well, it's a Ubuntu box up the top, several. We've got some boot from volume there because we needed larger sizes than our normal instances would provide. 64 gigs of RAM, 16 VCPUs, a bunch of Cinder volumes provided by Ceph, so that's, we use XFS on top of that. Go and die, EXT. So, a Muro lives there with our elastic extensions and cloud extensions, and then we start feeding new instruments in and out, in and out like that. And it lives across a bunch of our institutes now. And it's consumed across, actually, it's across the world. So, following on from that, this is kind of what a Muro looks like. You're seeing there, again, some single cell neuronal imaging in a web browser all puppeted by Django and Java. That can all be loaded in seconds, which is cool. So, a Muro is cool. Nobody's done what we've done in this space. We've been really fortunate. We've had really, really good ideas, great people, right time, right place. The Nectar guys have been absolutely instrumental in giving us infrastructure to work with. But that's not particularly what's most significant about this project, I don't think. There's a lot more to it. And really, like I think a lot of people and themes at this conference have suggested, the big thing is collaboration, right? So, in science, we talk about collaboration a lot. We talk about talking to each other a lot, because we tend to group think if we don't. We get stale if we don't talk to our peers. So, true power comes from users together. And I think it's interesting, this particular crowd that's here probably actually get that. Otherwise, you probably wouldn't have turned up because you read the byline and you thought, oh, this is unusual. But yeah, so we've got situations where we've got users all over the planet. One guy in Japan, another guy in Hong Kong, another guy in England, and another guy in our country in Australia, who are working on a 1.5 terabyte data set of 85,000 by 85,000 pixels that had come out of a slide scanner, right? And they're doing that all at the same time as ZStack, across a mirror. And they're doing that simultaneously. They're annotating, they're changing things, they're looking at Z sections. We've got zero friction, we've got 10 gig pipes in and out all the time, 100 gig bearers, and it's scaled on demand globally, right? So, we can do that pretty much anytime, anywhere now. We think from a workflow perspective, dumping stuff straight to web for people and their web browser is the ultimate way to get their attention, they love it, all right? I guess the biggest thing here is it's pluggable as well. We've got R, ImageJ, Matlab, Python, all that kind of thing, computational imaging tools which can really get to the crux of the problem really, really quickly. And we can scale on demand because we just know of a boot in another instance, another instance, another instance to bring it all together, right? And they're actually zebrafish. If anybody's interested, amazing little creature. So, it's not about the Swift Dream. And it's really not about how cool the SRIOV pass-through is. Further to that, it's not about the puppeteering and the orchestration. It's not about any of those things alone, right? It's more than that. It's a bigger deal. It's about what OpenStack embodies and I guess it's what it illustrates as part of its core mantra, right? I think it's what people do together that matters, right? We're trying to advance human life. We're trying to make things better for people. And I guess at the moment, at least, OpenStack is at the center of our imaging technology that does that. That's, for us, a pretty big deal. I'd like to think the community and what is OpenStack continues to do that for us. I'd like to say thanks to the boss because he lets me dream and do odd things. The Research Computing Center at UQ, who are a wonderful bunch of people. Professor David Abramson, hell of a guy. He backs all this kind of thing for us. Nectar, who funded the original deployment with the raw iron infrastructure and all the support around it, underneath all of this. For backing us, getting me over here. It was really nice of them. And all the research cloud ops guys back home. And you, the audience. You took a risk. Thanks for coming. You never heard of me. You've got no idea what I do. I know this is weird out there. Stuff compared to the normal things you'd probably look at at an OpenStack conference. So I really do appreciate it. That's my email address if you're interested. I'm happy to take questions at this point if you've got any. Oh, you might wanna talk into the mic if that's okay. Thank you. Just so they can catch it all. You mentioned the POSIX file system. Is that a front end to a Swift, OpenStack Swift? So it's sitting, it's not actually no. We've got POSIX sitting in, I guess what would look like traditional POSIX to the application. It's not like Fuse, but it's effectively exposing what looks like POSIX sitting on top of a Cinder volume, sitting on top of Ceph, okay? So it's not Swift. We were considering using Swift, but our local guys were like, hey, let's play with Ceph instead. And we didn't really need object underneath. So we thought, oh, let's use Ceph. And it performs really nicely. So yeah, good question. One other question. What about the OpenStack cloud? How do you, how has it decided who gets to use it and how much of the resources you use and just sharing in the multi-tenancy? So that's a pretty good question too. So the relationship between Nectar and us and our local institution is one of Meritus Allocation. But any researcher with a set of specific credentials that pass across the entirety of Australia can log on, boot an instance and go and do things for free. If we want something bigger, like what we've done here, we have to apply for a Meritus Allocation with one of our primary investigators or chief investigators. And they will then say, yeah, this is good, this is valuable, we're going to approve this and you can have X thousands of core hours, X amount of resources and so on and so forth. My experience so far, at least, given that it is a big environment, the Nectar project. It's thousands and thousands of cores across the country. We've always been given what we've asked for. They understand the importance of good research taking place, like world-leading research taking place. Yeah. Hi, I was wondering, is Omero specialized to any extent for microscope images and brain research? Or is it perhaps more general purpose and could be used for other things such as astronomy images? You're the third person to ask me that in the last two or three days. Omero is highly specialized towards light microscopy and optical microscopy. There are considerations to make it more geared towards the use and collaboration for MRI like microscope, MRI imaging, PET-CT imaging, and other kinds of imaging which haven't really been explored yet. So there is every likelihood that it'll turn its skills to the kind of imaging you're talking about in time. But at the moment, it was all about optical microscopy for predominantly life sciences. Good question. Any other questions? All right. Well, I think that would be about it. So, oh, yeah, go. Sean. It's an Australian research tool. Do you have a public data set or any sort of stuff that the public can get out and look at? You guys ask all the right questions. So it's almost like you've been told to. So we do, actually. We are now using Omero to publish public links which are served off the Nectar cloud so that images and citations in large journals like Nature, I'm pretty sure you've all would have heard of Nature, or Jneurosci are actually linking directly to the public database and data sets which we expose inside the cloud. So that exact thing is happening as we speak. Yes. Yeah, cool. It's all about open repositories, right? Yeah, cool. Yep, if you could speak into the mic, that'd be great. Sorry. Cheers. I guess performance is the main issue or so. Have you considered the bare metal cloud and using Ioniq or whatever? Yeah, so I was pained to use a cloud environment for this. I didn't want to. I wanted to use a bare metal environment. I wanted to run up a massive server and I wanted to run everything bare metal with a huge amount of RAM, massive solid-state caching, a huge sand, right? Frankly, I did not have the money to do it. I tried to go out of the money to do it. I did not have the money to do it. I was effectively, I guess politically pushed into a place where I needed to use a cloud environment to do it. Has it been at a detriment in terms of performance? Yes, it has. Have we mitigated a lot of it by working really, really hard from an optimization perspective? Yeah, we have. So the researchers aren't complaining. They're not talking about latency and issues anymore. The Ceph underneath is powering along from an instance perspective and a compute perspective. There was another talk earlier by one of the guys at Yahoo. He mentioned kind of that issue of bare metal versus virtualized instances. And the idea is you throw more compute at it if you don't have that luxury of bare metal underneath. We've effectively thrown more compute at it by making things elastic and it's actually worked really, really well. Is it less efficient in a way? Yes. Is it virtualized and containerized and elastic? Yes. So that's a really good question. Any other questions or considerations or comments or feedback? Cool. Well, there's nothing else. Thank you for attending. I really, really appreciate it. All of you.