 Alright. Hello everyone and welcome to our talk about Gen 3 and advancing biomedical research with an open source and cloud native platform. My name is Jawad Koreshi. I'm the lead platform engineer at CTDS at UChicago. And I'm Colin Griffin. I'm a founder and chief engineer at Crumbware. We develop cloud native technologies. I'm a contributing member of Tag App Delivery and the platform's working group. Alright. A little bit about the agenda for this presentation. We'll go quickly over an intro where I'll introduce CTDS and what we do. We'll talk about biomedical research, give it a little bit of background and especially in the cloud. Then we'll go over Gen 3, which is the platform that we're building to conduct research in the cloud. And just a little bit of our cloud native and Kubernetes journey. A quick plug for Crumbware. We're software developers that specialize in cloud native software solutions. We are application developers that are actively practicing platform engineering practices. And we support software vendors and their end users and help build shipable software products. And CTDS, which is Center for Translational Data Science. So as I mentioned, we're the builders and maintainers of Gen 3. We're part of the Biological Science Division at University of Chicago. Our founder and director is Dr. Robert Grossman, and he's a top leader for data sharing and data commas in particular. The CTDS vision is a world which researchers have ready access to the data needed and the tools required to make data-driven discoveries. And our mission is basically to apply data science in a translational way and apply it to problems in biology, medicine, health care, and the environment. A little bit more about our center. So it's effectively split in half, where half of our center is working on something called the genomic data commas. That's a research platform developed with National Cancer Institute. And it's used by about 100,000 researchers worldwide. It makes 6.7 petabytes of harmonized data available for about 89,000 patients. GDC runs exclusively in the UChicago data centers. But it is that technology, the pioneering technology to build data commas that we standardized and made open source and cloud native that turned into Gen 3. And Gen 3 today hosts about 2 million different total subjects. We have about 93 million files. And that totals to about 22.5 petabyte in total file size. And some more impressive stats can be found at stats.gen3.org. So let's talk about biomedical research, and especially in the cloud. So the traditional approach to doing biomedical research is you make up a data repository from all of the different research projects. And it's a centralized repository with some sort of governing body. And then you have different research institutions that basically copy the data from that repository into isolated compute environment. As you saw the numbers for Gen 3 and GDC and especially in genomic research, data can grow in size and copying that amount of data can get really expensive. You also get limited access because only a few researchers will have direct access to that repository and can utilize that data. And it's not really data sharing, it's data copying. And security as well becomes a thing that each individual research institution has to take care of. And research is essentially done in isolation. And it's hard to reproduce. And we're all familiar with the term works on my machine. Just apply that to research. So the cloud for biomedical research is a great thing. Like you can get immediate access to large data set mounted in cloud compute environments. You can have elastic compute so you can scale up and down and only pay for what you use. You can have centralized security, which I think is way more secure than isolated secure environments. And then also you can have containerized research environments. You can have reproducible research. You can have research done in a portable way. You can have isolation. And yeah, essentially it enhances the collaboration. But cloud can be a complex topic. Like let's say you've never worked in the cloud. Where do you even start as a researcher? There's a steep learning curve that we all probably know when you first start with cloud. Like how do you arrange enough compute resources for your needs? How do you, let's say you get your data in the cloud. How do you make that data findable? How do you do access control? And how do you get the tools that you want to use for research into those compute resources? And most importantly, how do you do all of that securely? That is where Gen 3 comes in into the picture. So Gen 3 is an open source data platform for building data comments and data ecosystems. I'll get back to a little bit what those are. But the goal of it is essentially to abstract away the cloud complexities for the researchers, but also to bring the open source culture to research. Gen 3 is built on cloud native principles and technologies such as Kubernetes. Also built with fair principles at its core. So all research and data that is being conducted on Gen 3 needs to be findable, accessible, interoperable, and reusable. Gen 3 is also designed to be lightweight enough to run on your laptop, but in certain use cases you can also scale it up to thousands of nodes. So I mentioned Gen 3 is a platform for spinning up data comments. So data comments, the definition of them is that it's a shared virtual space designed to collocate data, storage, and compute infrastructure, and a suite of interoperable tools. So you get basically a rich platform that you can work with diverse or rich data sets and do scientific investigation collaboratively. So with Gen 3, you deploy it, you get a cloud-based research platform and you can get your data in there. You have all your favorite containerized research tools and all the services to protect that in a single platform. Gen 3 provides a robust user authentication and authorization, so you can do controlled access to data hosted in Gen 3. We support a variety of different data types and provide a suite of data management tools. So a few highlights. Gen 3 is built on top of powerful and open APIs. We provide an authentication and authorization boundary built on top of OpenID Connect, so we can integrate with a variety of different identity providers. The Gen 3 APIs provide multiple ways of finding your data or doing data search and also a suite of data management tools as I mentioned. So you can do data access control, data ingestion, data export, but also do data quality control against a set data model. We also have Data Portal, which lets you browse and explore the data that you host in Gen 3. And Gen 3 workspaces are containerized data analysis environments, such as Jupyter notebooks. And then we also have an SDK for Python if you want to build on top of Gen 3. And I just want to mention that we're FedRAM certified for several of our Gen 3 deployments, which essentially just means that we have to do FIPS everywhere. We support ingestion of virtually any sort of data, which opens up a variety of different reuse cases that we're doing today. So that can be genomic research or DNA research or cancer research with large oncology data sets. You can do research on biomedical imaging, such as MRIs or CT scans. We're doing environmental studies where you can aggregate data for climate research or conservation efforts or pollution controls and many, many more. These are some Gen 3 instances. So we are doing genomic research for the VA. We have bi-data catalysts, which basically aggregates and shares human disease data. We have the cancer research data comments, which is just using a subset of the Gen 3 services to support their environments. We've got the metric environment, which does biomedical imaging and they're trying to build AI stuff on top of medical imaging. We have kids first, which focuses on children's with cancer and structural birth defects. And blood pack, which focuses on liquid biopsy for oncology and cancer treatments. And there are many more on gen3.org slash powered by Gen 3. So we have a lot of different Gen 3 data comments. And the next evolution of Gen 3 is to build an ecosystem that can interconnect all of these different data comments, which turns into like a federated research platform that is unified by the framework services that provide authentication and authorization, indexing and search and access to the data and also the metadata services. When you connect all of these different data comments together, you can still have all of these data resources interoperable, like operate separately and authorization to each data hosted by these comments are still controlled by themselves. The ecosystems can also integrate with any sort of repository that is fair compatible. Biomedical Research Up is an instance of the Gen 3 data ecosystem where we take data from 11 different data comments and federate them as a patient data research platform. It also was recently nominated as a driver project for the GA4GH, which is a global alliance for genomics and health. And another instance of the Gen 3 data ecosystem is the NIH HEAL initiative. The NIH HEAL ecosystem is basically a trans-NIH effort to address the national opioid crisis. The goal of the HEAL project is to get all of the research that is produced by this project to be readily available for healthcare providers and policymakers. One thing that is really cool by this HEAL project is that there is a data sharing policy that any research or data produced by this effort needs to be fair compatible. So all of the data needs to be findable, accessible, interoperable, and reusable. And that's why it's built on top of the Gen 3 data ecosystems. But that has a certain impact on research institutions. Now they either need to build APIs or product around their data to make the research fair or they can go the easy option and host Gen 3, which essentially means that they now need to start managing Kubernetes and cloud-native environments. So Gen 3 in a sense is serving as a catalyst or a gateway drug for these institutions to embrace Kubernetes. And that's how CRUM got involved. Yeah, so we are actually adopters and end users of the Gen 3 platform. We're supporting Wake Forest University and a program there called Empower. Empower means the integrative management of chronic pain and OUD for whole recovery. This is powered by an NIH grant as well, sponsored by Dr. Meredith Adams. Shout out to her. She's awesome and is in Chicago somewhere at another conference. The basic situation was Wake Forest needed a data commons but did not have Kubernetes expertise. Couldn't even spell it. And they needed with somebody with cloud native experience to repurpose the assets from Gen 3 and make it deployable into a wildly different cloud environment from Gen 3's typical. So this is a little bit of highlight of information about Empower. It's a part of a much larger network including the HEAL initiative just mentioned. So we are empowering that data sharing. In addition, we're rolling out new data commons very shortly to expand that internal network as well. All right. So let's talk a little bit of Gen 3 architecture and take a peek underneath the hood. So the Gen 3 architecture, it's a microservice architecture where all of our microservices are running on Kubernetes. The authorization and authentication is built on top of OpenID Connect and then we have our own attributes-based policy engine. I was told it also support role-based access policies. It's completely API-driven and all of our assets live in different object storage services. And then we also provide a graph-like database for clinical or structural data that is developed on top of Postgres. You can flatten that data into Elasticsearch for caching and faster queries essentially. I'm going to talk really quickly about data architecture in Gen 3. Yeah. So there's three different types of data that you can have host on Gen 3 based on their structure. So one of them is unstructured data which can be just file objects that can be, for example, biomedical imaging or genomic sequencing files. And then we also do structured data which is data that follows a strict or specific schema in Gen 3 that is called data model or data dictionary. That is the data that is usually used for clinical or phenotypical data. And then the semi-structured data is essentially a metadata to describe data sets or provide additional metadata about samples within the clinical metadata. So here you see a visual representation of a graph and this is part of our data portal. This describes how a particular data common organizes and describes this structured data. Data, when you submit it, is harmonized to fit the schema so that it'll be easier to do consistent searching across the clinical data. Then for the unstructured data, it can live in a variety of different buckets and stuff. And then within Gen 3 we meant a permanent digital ID called data GUID for each asset within Gen 3. It's made off of just a prefix that tells which comments that is hosting this data and then a UUID. Using this we can support like trillions of GUIDs across all Gen 3 powered comments. Then we also built a data format that can encompass like all of the data within a Gen 3 data comments and create a snapshot of it. With that PFB file format, you can combine like clinical and structural metadata and into like a single file and you can export it to external system to enhance the interoperability or you can do backups. But yeah, PFB can also be extended to work with like fire or healthcare data. Okay, so a little bit about our cloud native journey. So we've been doing this for a few years so I just want to go over a few of the challenges and the learnings that we've faced so far. Okay, so for context CTDS is managing about 30 different production environments and they're all on AWS and EKS. Our short-term goal is to grow that into 200 comments but those 200 is not just our comments but also community-driven data comments. Additional context, so the traditional way to deploy Gen 3 onto Kubernetes is we made a tool called cloud automation. It is essentially a monolithic repository of DevOps tooling where we have batch scripts, the Kubernetes manifest, the Terraform. The idea was to abstract away a lot of these technologies for researchers but there are some proprietary concepts in there kind of like an admin VM is where you run all of the Gen 3 commands from. This is a bastion host with a secure access to manage your data comments. It's very highly integrated to CTDS use cases and very comprehensive but it's also a barrier of entry for the community because there's a steep learning curve first of all and there's some proprietary concept that probably doesn't make sense to anyone even if you've worked with Kubernetes before. So yeah and that's how Colin also started helping us out being involved in the community and yeah. So again a little bit of background. So cloud automation was tough, Juarez right. So we were working with Wake Forest and they were working on a non-AWS cloud provider. Very favorable agreements with Google Cloud so they want to continue to maintain that environment. Couldn't run Kubernetes to save their lives. They couldn't give us any idea standardization or compliance or any way to run those environments and when we when they tried to actually run Gen 3 they couldn't use cloud automation. It just didn't work and it was proprietary to AWS at the time. So at the time Gen 3 required adopters to create the operational environment to configure, deploy and maintain that environment itself and then repurpose, configure, deploy and maintain the Gen 3 data commons environment as well. So complexity on complexity on complexity but it's not a unique problem to the industry. So when we evaluated Gen 3 we wanted to make sure it was a good fit and it is. It's a great mature open source data commons project built by researchers for researchers. Researchers know researchers best and it fits a good platform strategy for diverse build. It's an interoperable platform that we can build on top of. Go ahead. These adoption challenges are really interesting and it's a burden of responsibility for open source maintainers. You come out with a product that somebody wants to run in their infrastructure and it causes these unintended support challenges. In users we would think have the environment ready and choose our tools with the environment and the infrastructure in mind and then deploy those tools. In reality they often work backwards. They say what's a tool that can get me paid? What's an application that can do that? How do I run that? Where's the infrastructure? So Gen 3 found themselves in that situation. So how did that manifest in Gen 3? Adopters weren't ready for cloud native. They were finding when we got joined in on the project, Jawad was giving presentations and trainings on how to use Helm, how to run cloud automation, what's a Kubernetes cluster, how do you secure Kubernetes and that's not what Gen 3 should be focused on. They should be focused on improving the data commons product. So adoption challenges like that are everywhere. Again many of our adopters don't have cloud native expertise. So our considerations we need to start architecting for portability and architecting for scalability and providing those assets to the end users so they can actually run and consume our applications. And so we consider if we're thinking about platform strategy we have a new user type there. The new user type is the adopters themselves. It's the IT operators and it's the developers that need to build on top of extensible platforms and provide custom experiences. So how did Gen 3 start to tackle that? Yeah I mentioned cloud automation where like our Kubernetes manifest were hidden behind bash scripts. It was AWS specific and it was difficult to work for others that are non-CTS people. We also had a Docker compose deployment but it was cumbersome and hard to maintain. So that essentially led into the involvement of Gen 3's DevOps strategy. So we recognize that doing it in a monolithic way in cloud automation wasn't really working for everyone. It was also a barrier for us because we needed to grow our commons into 200. So we needed to adopt community standard tools to add additional automation and free ourselves from the admin VM essentially. So what we ended up doing is we adopted pure help for Kubernetes orchestration and then we took the Terraform out and made it a pure Terraform module and then we are less opinionated about how to run them and make that more ambiguous and up to the community. Our goal is to enhance the community and the developer experience but also to add more automation, as I said. And that's how we're architecting Gen 3 to run everywhere. We have our Helm charts. We have a Gen 3 umbrella Helm chart that encapsulates all of the microservice Helm charts. And then we bundled that Gen 3 umbrella chart with some development tooling as well. So it comes with Postgres, Elasticsearch, and Minio. So we can do Streamline, Dev, and CI operations. We have a common chart that does shared stuff. And a lot of the scripting that was done in cloud automation can be moved there. It can be a database set of jobs and other shared functionality. And for infrastructure management, I mentioned we're using Terraform obviously because it gives us the reusability and repeatable infrastructure deployment. And now we can also provide that as a reference architecture for the community to follow because yeah. And building a platform around Terraform added some challenges for us as well. Just keeping up with the Terraform upgrades and the state changes and the schema changes. Yeah, Ed over there has, yeah, he's amazing with Terraform and he's keeping us afloat. But ideally like we wanted to free ourselves from the admin VM so we could do pure Terraform work and not just build like a DevOps platform around it. So now we're doing Terragrunt and Atlantis with GitOps approach and hierarchical management of all of our environments, which will help us grow into 200 plus comments. GitOps is really, really important for us and we've been doing that since 2018. Argo CD wasn't a thing back then, so we built our own way with the CD's manifest and cloud automation repositories. It's very, very bespoke and we just do like cron jobs to sync whatever configurations and I think that runs once every hour or something. But now with Helm and Argo CD, we can do faster deployments and rely on the community to build these DevOps tools instead of us having to build them. And I mentioned Terraform and Atlantis and yeah, our focus should be on building research platforms, not DevOps tooling. Yeah. Yeah, another challenge for them was research compute. How do you scale to 50,000 nodes when you need to run a complex data workload? Right. So we have two different types of research compute within Gen3. We have our workspaces which can be containerized analysis environments. So they can be like notebooks such as Jupyter or RStudio or any other containerized application with a UI that we can mirror. The way it works under the hood, we have an orchestrator that basically spins up these containers either in Kubernetes or in another cloud account using the AWS SDK and then we basically use an envoy proxy to mirror the UI back in a secure way back to the researchers. Another research compute is workflows. So in Gen3, we support NextFlow and we also support running Argo workflows. And one thing that we're doing with Argo workflows which is really cool is we're doing genome-wide association studies. That is a statistical tool where you can identify genes or genomic regions that are related to health and diseases. Yeah. A little bit more about our auto-scaling strategy though. We used to use cluster auto-scaler and in our environment, we used Terraform to set up some auto-scaling groups and then cluster auto-scaler would hook into those auto-scaling groups. We had to kind of guesstimate the node sizes and types to fit all of our workloads. And we also ran into a few different issues where you want to spin up a... Anyway, we ran into a few issues. Yeah. We moved to Carpenter because that allows us to manage like all of our infrastructure and nodes directly from Kubernetes using CRDs. And with that, we could get the right size of nodes and we could choose between like a spectrum of node types. Also helped us move to spot, so helped us cut a lot of cost. And we're also using Carpenter in a cool way to do multi-tenant cost tracking. So I mentioned the GWAS workflows. That spins up about 1600 pods for each workflows and those come out to about $50 each. We need to track those costs so we can offset them and like get that back to the researchers. The way we did that is we used Carpenter in a combination with Argo events. Each time a workflow is triggered, we create specific nodes in AWS using CRDs so that it's tagged with the workflow ID and the user ID, so that we can rely on the native AWS cost and user report to generate the cost. And then that enables our Finox team to create like payment models and stuff around these workflows. Another thing that we've encountered is developer experience. And we are working hard to make that better. Traditionally, the way it worked is that we have a shared EKS cluster running cloud automation with a namespace for each developer. Developers would commit to GitHub that would build their images and then they would roll the pods in the EKS cluster. That is a slow development process and we wanted to move that a little bit more left in the development life cycle. And with the home charts, we can do that. The developers are now able to spin up Gen 3 directly on their laptops and we can build or we can leverage tooling such as scaffold or tilt or dev space around to do like hot reloading of code. And right now we're working on getting like Heroku like ephemeral environments using Argo CD's preview environments thing. And we're also looking into building like an internal development portal for our developers to have all of the dev resources in there. And we're building a new front end application or front end framework built on Next.js so external developers can also add in their own components for Gen 3 really easily. Yeah, and I can take that. Yeah. So, but what about running Gen 3 in production? We always get asked, okay, what is it going to take to run this in production? That's always the question. Go ahead. Yeah, so different organizations have different challenges like they they wanted to run it on different cloud providers or even on premise on OpenStack. We have people doing that and they have different compliance requirements and they have different cloud maturity or practices within there. And Gen 3 may or may not be deployed into a larger ecosystem, which is already part of something. So we need to like set up a shared responsibility model where Gen 3 is not responsible to manage the infrastructure for people or secure their environment. What we can do is we can provide validated and secured artifacts that are tested. We can provide recommended tools and best practices. We can provide reference architecture and specifications. We can have our APIs be interoperable and just in general provide a strong data comments experience. Yeah. And the shared responsibility model I popularized by AWS so perfect is a great communication tool if anything to communicate where the responsibility lies between the consumer and the producer. The CNCF put out a platform's working group put out a white paper. This is a diagram from that white paper. And it's a good kind of architectural mapping tool that can help communicate what are the actual tools and capabilities your platform provides and helps communicate those tools back to your adopters as well. I can tackle that too. Yeah. Next. Yeah. So if we take that and we repurpose it and turn it into an architecture or planning tool, this is essentially what the core Gen 3 platform looks like in association with the platform's working group artifacts to communicate where different components and different CNCF ecosystem products are used as a part of the encompassing platform. So this is an interoperable cooperative tool that works alongside other applications within a platform and doesn't necessarily need to function as a black box. That gives you some of that interoperability that custom customizability and lets you use the tool for more than it was originally purpose for. You'll notice that in some areas if we communicate actively that for infrastructure or for messaging, for example, they might bring your own. You might need to select something there, but just communicating that is powerful as well. So tell us a little bit more about how CTDS makes customizations to their environment to better run their commons in particular. Right. So we use AWS for all of our underlying infrastructure. We use EKS for Kubernetes. Then we have open search to offset the elastic search. We have S3 for our object storage. We use Aurora serverless for the Postgres and then Argo CD to do the deployments, DataDog for observability and Prisma Cloud and Cortex for security. And in power at Wake Forest, we mentioned we're working on Google Cloud. And in fact, we're actually deploying data commons into multiple cloud provider environments. We're not just in Google. We want supplier diversity. So again, we're running on GKE for Kubernetes, Google Cloud obviously for infrastructure. We use our own elastic search instance and support that. We're also using Google Cloud storage for S3 compatible object storage instead of using a Minio or instead of using S3 itself because we're not on AWS. And we're using Cloud SQL for Postgres. So we're showing some of that customizability and that interoperability. We're reusing those existing tools. And if we think about that overall operational platform as a whole, we need to map the additional components that we're using to supplement and to support that environment. So we're actually using some additional tools as well from the ecosystem to make sure that this runs best for Empower. We're using Rancher as a Kubernetes orchestrator so that we can run those multiple cloud environments and create new environments on the fly. It also gives us a great management plane. We're using Longhorn for storage. You can see we're actually introducing NATS and using NATS for a number of different data and ETL workload and a little bit of real time and temporal data as well. And we've actually exposed some additional interfaces for the researchers including direct access to Kibana in some cases. But because Gen 3 has developed with that core interoperable approach and is communicating how to include those other assets, it made it really, really easy for us as somebody supporting somebody else running the project to secure the environment. Like we've introduced a in-cluster security tool as well with compliance profiles to lock that down and do reporting. So we couldn't have done that if it wasn't interoperable, if it wasn't customizable. And because of that, we can build on top of it. Yeah. And as Colin mentioned, these are two very different operational environments, but they run the same kind of open data comments for difference or research. Anyway, you can also get involved and help us cure cancer with Kubernetes. We want to have more deployment experience with other cloud providers if it's not Google Cloud or AWS. When I enhance the developer experience, there's a lot of cool tools in the CNCF landscape, which my team here is here to explore now. We can do infrastructure and code enhancement and build on top of the front-end framework that I mentioned. Essentially, we want more institutions to join the Gen 3 and democratize research. We have a few community resources. We have our Slack, which you can get into by scanning that QR code or from our website. But building a community around any sort of platform is challenging. And we have limited staff, so Slack is a great tool for us to be able to connect with the community and foster a community that helps each other. And Colin is a proof of that, essentially. We're also doing bi-monthly webinars and going to start making more tutorials and stuff on YouTube to make it more known. So if you want to learn more, go to gen3.org. You can go to ctds.uchicago.edu to learn about the Center for Translational Data Science. And if you want to read the platform's white paper, you can find it on tag, appdelivery.cncf.io. Perfect. All right. A little bit over time, but we can take any sort of questions. Heck of a loss. I got two questions. One is why not the Galaxy Project like has the same kind of resources, the helm, you know, deployment for a single chart? And two, does this integrate with other NIH like data ecosystem stuff like the GA4GH DRS model or the C2M2 cross-cut metadata model? Yeah, I guess it's a supplemental thing to the Galaxy thing. I don't know what to respond to that, but I think they might overlap in a very, like a variety of different areas. For the DIRS URI, I think it's, I believe it's kind of based off of some of the technology. And we do work together with GA4GH and NIH and trying to give back to the community and build basically a open source research community. Right. Yeah. Okay, cool. Thanks. Hi. Sorry, relatively new joiner to the field of biotech. But what I noticed is that the whole ecosystem tends to have like its own set of tools for themselves. People in my company really like using a Cromwell as their workflow engine. I noticed that you used Argo CD workflows. Right. How coupled is it? It's very optional. It's very optional. Yeah. So we've just built a few workflows that work for us to do GWAS that we can run on Argo workflows. And then we did the carpenter thing on top to track the cost. If you wanted to run Cromwell, I'm sure you could do that on top of Gen3 data. As I mentioned, we have an SDK to pull in data or build on top of Gen3. So it's very possible whatever engine that you want to use to use it. Yeah. I think it's really important to highlight their open platform strategy there as well. It is designed to be interoperable and configurable. Right. Everybody, especially research institutions, you're exactly right. They have their own customized tools. They have their own proprietary tools. And so we need to be supportive of that. And that's the idea is being able to provide that core platform experience that really serves the end user as well, but allows the operators to enhance and improve those workflows by adding those additional tools. So very much taking an enhancement or an opt-in approach there as opposed to a closed ecosystem and do it our way. So those open APIs are really powerful there as well. But Argo isn't part of the core platform. It is part of maybe the recommendations and examples of ways that you can run ETL workloads or do your CI CD or rule things out that way. We personally use fleet in some of our applications as well for that, just to demonstrate there. But we are also choosing to use Argo workflows as our ETL engine of choice. One more question. Is there a chance that we'll see UK Biobank data on a Gen 3 deployment? I would love for that. I'm not sure if that's happening, but if that is happening, yeah, I would love for that to happen. Thanks. Yeah. One of the beautiful parts that hopefully they can, when we started the project and when things were a lot more manual, it was challenging. But now we have a new, we can stand up a new data commons in about a minute and a half and just replicate, replicate, replicate. And that's amazing. All right. Thank you so much.