 We'll start by just introducing ourselves, Carmel. Hi, so I'm Carmel Walsh, so I'm the Director of the Research and Construction Services at ARDC, Learning Research Data Commons, and basically I head up a team that manages the Nectar Research Cloud and also our storage, national storage in relation to data retention. Thanks, Steve. Thanks, Carmel. I'm Stephen Manos, I am with the Australian Biocommons. We are building national bioinformatics infrastructure for research, and I have about 10 years of history in deploying, building, operating, research computing, research cloud, research data infrastructure, primarily at the University of Melbourne since around 2010. So we'll get stuck into this. Carmel, I have about 15 minutes to pitch the idea of ARCOS to you and some of the history, and then we'll kind of get into some bit of Q&A and just feedback and really any questions you want to ask around ARCOS specifically, but I guess also the broader cloud computing HPC ecosystem in OZ. So I'll just start by saying that ARCOS is a community and ARCOS has published a paper on the community and the community's needs around container technologies and research. So we'll remind you now and remind you again at the end of the talk that there really are two resources for you to look at and to contribute to. One of them is the ARCOS website, arcos.audc.edu.au where we have our resources, we have video recordings of talks, and we also have mailing lists and other ways to access the ARCOS community, thanks Andy for posting the link. And we also have a paper that we've published which I put the bit link there for, which again will give you an idea of the challenges and priorities of the community here in OZ. So moving on to this, where did ARCOS come from? Well I guess much like Cloud came about 10 or 15 years ago past HPC, the next thing that researchers were going to start basing their research on and building workloads on much like what you just indicated, David, are containers. And there was I guess a movement and a question of how can the national research community come together around containers, what are the challenges, what are the shared opportunities, what can we do together. So the idea of kind of building a community around container and container technologies in research was born really in mid to late 2019 and culminated in a major kind of community meet up over in Perth in early 2020, just before COVID hit. And that was I guess around 40 people turned up to that meeting to really think about what are the shared challenges, what are the shared opportunities around containers over a two day workshop. Since then that community has been formalized through working groups and steering committees. We did a big survey of the community which included various organizations, universities, ARDC, the Nectar Research Cloud, ARNET and so on to say what are the challenges, what are you focusing on, what would you like to get out of a more concerted community effort. And this kind of culminated in the ARCOS paper that was first published in August 2020 and then republished in October 2020. Then late last year through the work of leadership of the ARDC and Andy and Carmel and others, we finalized the ARCOS project plan in response to those community requirements and I keep using the word community a lot and that's very deliberate because this really has been a community developed and community driven initiative. So it's worth just kind of taking a step back and thinking about what are containers and and what is Kubernetes and really containers just simply provide a lightweight execution environment for software and applications. They're basically an image that packages up pre-compiled software with all of its dependencies and libraries and even data into a single bundle that's effectively isolated from the environment that it runs in. They provide a way to share libraries, code configurations and data amongst researchers and it really does away with code compilation and library dependencies and blah, blah, and it does promise ease of use and a transportability but although that's the ambition, it's definitely not always the case. Generally speaking, there are two things we package up or do with containers. One of them is we want to containerize entire tool sets where a whole set of tools or individual software tools or pipelines are included in a single container and the other thing is a single application or a very specific function is built into a container and this is referred to as a microservice. In the container world, there's really two types of container formats that tend to be used that you would have heard of Docker and Singularity and they have their own standard runtime that they run within and the runtime is the environment that the container runs inside of and these two runtimes tend to favor or target two different use cases or two different security postures. Docker tends to favor isolation and more kind of cloud native administrative scale on our applications whereas Singularity I guess is more what you'd see in research around HPC centers where it tends to favor integration and command line applications, funky hardware access like for GP GPUs and tends to operate better in multi-tenanted environments. So of course if you want to kind of use containers then you need to be able to manage them and you need a system that you can figure containers with, you can manage their life cycle within, you maintain them, scale them out, schedule them and so on and this whole process is referred to as container orchestration. Now Kubernetes is one such orchestrator and it provides a standard I guess model of describing what a container deployment or a container orchestration system looks like. Today Kubernetes is the absolute de facto standard for container management and it's supported across a wide range of private, commercial and community cloud environments and it promises this interoperability I guess across these different environments but even with these standards and this promise of interoperability there are complexities and there are incompatibilities, differences in how compute and storage and networking is ultimately implemented by the underlying platform means that not all Kubernetes and container deployments are functionally equivalent and generally they're not very compatible. So stepping back from a research infrastructure lens or a researcher's lens, no two instances of Kubernetes deployments whether it's on the Nectar Research Cloud or Amazon Web Services or Azure Cloud or Arnett's or even you know ARDC's Magnum Service, none of these deployments are guaranteed to be functionally equivalent and be able to support the containers that you're running. But as researchers, particularly researchers who are working in national collaborations or national consortia, they need to be able to readily, more readily than ever, port their tool set to other colleagues, other infrastructures and other resources and as a result the deployment and ongoing management of Kubernetes for research is challenging as a result. So through ARDCOS we engaged as I said the community at large and really asked around their current and expected future use cases for containers and details of all of this are contained in the ARDCOS paper and based on these use cases we derived four of the main drivers that we're seeing for containers and Kubernetes. So the first driver is that they really want to use, people, operators and researchers really want to use Kubernetes containers to minimize infrastructure complexity through more isolated software environments and as said really software is contained within the container and all the dependencies are also wrapped up in that as well. So it tends to do away with these system-wide conflicting dependencies. It also allows you the second one there, easier code distribution and more portable code because you can wrap everything up into a single package. Containers really inherently support reproducibility and reusability and you kind of guarantee this coherence because all your materials are packaged in a single container. And of course it simplifies the underlying compute infrastructure because the infrastructure just needs to run containers, it doesn't have to necessarily be particularly tightly coupled to an application or other things. One of our other key findings through the community consultation is this thing called the community framework which tends to be unique I guess to the way Kubernetes and container infrastructures work. And the thing that this framework allows you to do which we discovered is that it kind of lets you think about what an infrastructure team does, what an application developer will do and what an end user will do as three very distinct and separate things. And we refer to this in the paper as the communities, the Kubernetes community framework. And the result of the separation is kind of I guess a bit profound the more you tend to think about it. The result of the separation is that very little application level knowledge is required of the infrastructure team. Little infrastructure knowledge is required of app developers and really no knowledge of the complexities of libraries and software compilation are required by the end users. The great thing about this is that each concern, it means that each of these I guess three vertical tiers can be treated and scaled and we can do work on these independently of the other. Researchers and software engineers can share their containerized tool sets globally, whereas you know infrastructure on how to operate this stuff or ideas on how to operate infrastructure can really be worked on at a national level as an example. Over to you, Kamal. Thank you, Steve. So R-Cross, why are we looking at it from a national level? Well, referring back to the paper that Steve mentioned and the community consultation what we found was that there was five main challenges that was coming across in relation to the use of containers and orchestration research. So the first challenge was that expertise and experience and practice was isolated and disconnected. So everybody was playing around doing some work with it, but they were doing it within their own institutions or areas. Challenge two was there was diverse deployments, which are leading to incompatible islands. So one of the key things going back to an earlier side was on that portability and also on that collaboration across institutions. Challenge three is Kubernetes is complex. So expertise and subject matter expertise is required. Challenge four Kubernetes is a feature rich and extremely capable. How do we take advantage of those features? And then challenge five is fair containers are rare. So findable, accessible, interoperable, and reusable containers are rare. So the opportunity was first to enable a collaborative and reproducible research environment and coherency across the different infrastructure. So that was a driver behind setting up R-Cross. Next slide. Sorry. So basically what we focus on then is what is what is it on a higher level? So basically we're looking at these four main areas. So community awareness, raising, best practice, standards, user support, and user experience for the key four areas that really came out of the paper that was pulled together from community feedback. So looking at community awareness, raising, as Steve has said, very important, this is community driven, but we also needed that national coordination to be able to pull all those elements together. So that's why ARDC became involved as well to look at it from a national part and also that international collaboration and best practice standards. So creation of reference implementations, adoption of best practice standards and training of research software engineers. So kind of looking at what we have done previously with the Nectar research crowd and how do we adopt best practice for deploying containers and orchestration across research and then user support. How do we ensure that people get the right level of support? So having that trained experts and also being able to direct people on how to use tools like Kubernetes for impact and then supporting integrated and national support services. Another key element that came out was container registry and other infrastructure. And then last but not least is obviously the user experience. So how do we ensure that there's a positive user experience that is accessible and that we have different levels of support, which aligns our user support. So we have regular national user communities and we have various events and user training. So we had a symposium last week, obviously we're at this event now and we have spoken to it at e-research Australia New Zealand. So moving to the next slide. So who is ARCOS? So ARCOS is multi-layered. First off, we have the ARCOS steering committee who makes on a monthly basis and that involves ARDC representation, POSI, Biocommons, Monash University and Arnet. So very keen for us all to work together to deliver this. And the next slide, who is the community group? Well, this community group now meets quarterly. There's over 90 members in this community group and you can see here from this slide the representation is quite broad and it is very much on a national level. And that community group is important first to understand the needs but also to put together the best practices, knowledge sharing, et cetera. And next slide. So the last layer that we have is we've set up working groups. These meet on a fortnightly basis and we have two main working groups at the moment. We have the technical working group and we have the registry working group. So looking at those two groups, the technical working group is to what I said earlier, looking at those best practices and standards, referencing implementations, determining what other shared infrastructure required, development and training and establishing champions of the program to help identify candidates and to identify opportunities to share knowledge a little bit more. And then the registry working group is looking at defining the requirements that we would need for a national registry and defining the design of the registry and building a national containers registry. And what's key about our course as well is that it's not about an individual institution providing, for example, a Kubernetes service. It's about looking at it across the e-research community and how do we provide support to move that research community forward but look at it from a national level. So that's what we've been driving from the community level. And then if we could go to the next slide. So basically, how do you get involved? So basically, you could join those community working groups and they put a link earlier to our website and also to the ARCOS paper. We would love your feedback. We had our symposium last week. So it was on Thursday and Wednesday and Thursday last week. So we will be preparing a report on that. But what we did see from this symposium is there is huge interest and demand for this. And what we want to be able to do is not just supply and support the technical development, but also supporting and that coordination. And so the coordination is a key element at the moment to be able to pull all those ideas together. And that's where we need more people involved to be able to contribute to that across across the country, basically. So that's where it's at. Please let us know if you have any questions that you would like to ask us now. We have quite a bit of time. What are your experience? What are your desired use cases?