 and pull the attendees over, get everybody to join us here. I believe with that we are in fact live. Okay, so wonderful. Okay, yes, first talk, tech support seems going okay, good. So yes, I am extremely excited to introduce our first keynote speaker of the meeting. We'll have a keynote per day. I got very lucky with very gracious keynote speakers for this meeting. I'm very happy about that. So first we'll be hearing from Dr. Katie Burner from Indiana University, just down the road from where I did my doctoral studies, Bloomington's really cool. And Professor Burner, there is the Victor H. Ingvi distinguished professor of engineering and information science and the founding director of the Cyber Infrastructure for Network Science Center, a really cool study unit. They're working on network science at IU. So without further ado, let me pass off the floor to her. Take it away, please. Thank you, Charles. Good morning everybody, or good afternoon wherever you are. It's a true pleasure to present on envisioning digital science. Many of you might be familiar with some of the exit works that we have been bringing to many public places. And it's nice to show you a little bit more about our research work as well. So this is a three-part talk. I will tell you a little bit about the exhibits that is now in the 17th year of its existence. I will introduce Spoke to you, which is a very large scale network of ontologies with three million nodes and 30 million edges. So how do you ever lay out a network as big as that? And I will also tell you a little bit more about hub maps, the ambition to map the human body at the single cell level, trillions of cells. So this is very much big science, but I think it also very much benefits from some of the works that's done in the arts and humanities. So I will try to make these connections because I absolutely believe that there needs to be a very close encounter and continuous engagement between all of the many sciences. And in German, Wissenschaft is including everything. This is a little bit different here in the US. And it's a very, very end. I will tell you how to empower yourself to map your own data, to map data that others might have in your family, in your workgroup, in the environment in which you operate. And so with those four parts, I hope you will get a better understanding of what's possible today, what are the challenges, what are the opportunities. And I'm very happy to answer a question after each one of those four parts. So please do send your questions via crowd chat. I think that's our crowd cast. I think Charles will be monitoring this. And then I will leave a little bit of a break after each part so that you can get answers to your questions. So I would love to actually have a poll of how many of you have seen the exhibit. Again, this is an effort to bring large format maps of science to many different places, to libraries, to science museums, to national academies around the globe. And we have been very lucky to work with a very diligent and devoted set of curators and exhibit advisors, but also to have more than 260 experts from around the world submit not just maps, that was the initial 10 years, but now also interactive data visualizations, which is the decade in which we are currently, which invites macroscopes. So here you see some of the very first maps, actually the very first display was in 2005 in Denver, Colorado, at the 101st annual meeting of the Association of American Geographers. And we had these maps framed and you see the proud map makers and have put the dust away, the last pieces of dust here, Kevin Boyack, which some of you might know. And then it went to many different places, including also here, the CDC Museum Atlanta, Georgia. And over time we had more and more interactive displays. So here in the lower right corner, you see a microscope display or kiosks that lets you browse through many of the interactive data visualizations. And I think Charles posted a number of links, which you can use to actually get to the interactive data visualizations if you wanna explore them. So again, in the first decade, first 10 years, we got 10 maps each year, which could ultimately, there were 30 to 40 submissions. So it's good reviewed, very much like in a normal journal review. And then we picked the best 10 maps, make them ready for display to general audiences, which is not always the original intended use. Oftentimes these were maps created for journal publication. So for more expert audiences and then bring brought them to many different places. The second decade ongoing right now looks at macroscopes, interactive data visualizations that help us see things that were never visible before. Social media data, to humanities data, how do they connect to scientific data sets? So for those of you which love visualizations, please do go to simups.org and check out the display. And of course now in the pandemic time, much of what the exhibit is doing is virtual. So it's actually easier to present the exhibit in many different areas of science and also in many different countries because I can be on one and the same day in multiple countries, in multiple continents even. And that wouldn't be possible without being virtual. For those of you which have interactive data visualizations, please do check out and try to submit in the next coming three years. We have three more years for the macroscopes and I think there will be a third decade but more on this at a later point in time, not today. I wanted to show you some of the visualizations and it looks like the slides get cut off. I don't know if you can see the acknowledgments underneath each map. So under each map here, there's also the map maker acknowledged and the original title and the original year. So maybe this is set up we have right now, this won't be visible. I can share these slides later on so that everybody is acknowledged who contributed to the exhibit. In this visualization, you see the interconnectedness of science. It's really not a local business, it's a global business. And in many cases, many of us have collaborations not only in our own countries but around the globe. And of course, these collaborations are easier if you speak the same language, they're easier if you're on the same continent but they exist nonetheless across the entire globe. You can also map funding data and in fact funders very much like these visualizations because they get to compare their funding portfolios to each other. They have an easier time finding reviews or new experts that might like to be interested in the new funding opportunities. It helps them find experts for certain new seed workshops and it also helps all of us understand how science is funded. And as you all know, there's some areas of funding of science which are very well funded and then there are areas which are not so much funded and still it all has to work together. This was one of the very first maps of all of science created by Kevin Boyack and Dick Clavins. And here you have a base map of science where scientific papers are laid out in a two-dimensional space and then this map is rotated and mirrored in a way that mathematics as the most abstract and most theoretical science is on the top. And then as you go down in a clockwise direction you get to see mechanics and physics and chemistry and then over to biology and then down to biochemistry and the medical sciences. You get over to medical treatments but also general medicine, neuroscience, psychology over to economics to socio-political and law in the top left here. And then from economics over to statistics and computer science which of course are also connected to mathematics. You can then use this reference system to overlay other data such as nanotechnology or proteomics or pharmacogenomics. And you get to see that there are some way into the disciplinary areas of science which are really highlighting papers in many different areas. Oftentimes you have sciences that have a very focused profile but then also still have some references in for instance education or in statistics or computer science because they need those areas to train the next generation but then of course also to create the computational infrastructure that is now needed to perform science. You can take patterns and you can create IP holding landscapes who is owning what intellectual property. You can also then take these reference systems here created by just listing the USPTO classification system in these neat rows only to the level of three even though the classification system actually goes down to 15 levels. So here you see only a smaller part of that classification but you can use this carpet as you wish to then overlay prior art that is cited in a certain patent or to showcase how a certain patent is cited by other patents. And here you have Gore-Tex which is highly cited on the left hand side and a patent on gold nano shells on the right hand side in the top right. And so you get to see how it all interlinks automatically. In this particular visualization you get to see how much mass science and technology is actually visible and can be read about in Wikipedia articles. So here we took the English speaking Wikipedia entries and we identified what entries actually are associated with mathematical scientific and technology terms. And you can then in size code the entries based on different attribute values. So for instance, you can look at article editing activity here on the top left or major edits in the top right or the number of bursts sudden increases of usage frequency access to certain articles here in the lower right. And you see that there are areas which are more dealing with mass science and technology. And then there are others which deal with other topics. You can also map our human disease room. Here you have a map of the human disease network. So you'll see top diseases and top genes that are known to be associated with diseases. And you also have color coding used to identifying different disorder classes. You also have in the lower left disorder class interactions. And you can also zoom into this network. There is an interactive online site for this so that you can explore this network in an interactive way. And in fact, many of the 100 maps which were submitted in the initial 10 years of the exhibit, they come with interactive websites. And many of them are still active even though time has passed now. It's non-trivial to keep those websites up to date. There are also maps by artists. Here a map of the history of science fiction. So those of you which are historians might like to try out drawing your own data sets in a more unique way. Here you get to see different science fiction novels. And they are laid out very much in the way that Amazon gives us recommendations. So if there is a book which is similar to another one they are more likely to be found together here. But you also see all kinds of other things going on in terms of clustering over time and even kind of warm halls which ultimately go back. So again, trying to represent data in new ways also engaging ways that keep the attention of viewers long enough so that they understand this new unique layout of data is very important. And then as users and viewers zoom in in this case here by getting closer to the map which might hang on a wall and then being able to read more and more defined print I think that's very much what you would like to achieve if you are in a science museum environment for instance or in a library environment. There's a lot of other things going on there are rockets going up there are cute animals and a petting so on the left hand side and in the middle as a science map or a map that tries to communicate digital science how do you have people even come over to you? It's non-trivial to create maps that attract attention keep attention and communicate and are remarkable so that when everybody goes back home or goes away from their home office goes away from their screens and laptops so that they tell their family members about it because it has changed how they see the world how they think about the world. And the exhibit is really trying to get maps and macroscopes that do exactly that. Then all these maps are online so please go to signmap.org and zoom in and explore interact with these data sets in new ways. Coming over to macroscopes here you have typically a kiosk and it's a touch panel also not possible right now in pandemic times but all of the macroscopes were made available recently online so that you can explore them from the safety and security of your own virtual setups. You can then zoom into data from climate research for instance or from national academy reports and you can explore those or you can look at news articles and how different countries report or don't report about other countries or are reported about by other countries. You can also look at 2,600 years of human history in five minutes in charting culture. Some of you might have seen this video already if you have not, you might, you are in for a treat please do come over and watch it. There are also maps which use very rich and very complex data to render for instance the states in the US in new ways based on commuting patterns and of course this data was captured before the pandemic came over us. What you see here is networks that represent how people commute from their home to their workplaces and again this might all be different and a new normal that we hope for. Before the pandemic some really had two hour commutes into New York for instance from places far away that they are calling home and then two hours back in the evening. Based on these commuting patterns you will also get to see that for instance Gary, Indiana lots of people living there are actually commuting into Chicago and then back in the evening. And so if you now draw the boundaries of Indiana as a state based on commuting patterns and Gary, Indiana actually becomes part of Illinois as you see up here. But you can then also zoom in to the areas where you live and you can understand the geography of your state and your geographic region in new ways. Here you see a smelly map. Many of you might recognize which cities this is. It's London with the River Thames going through it. You get to do the zoological garden and let me actually move our piece here. So based on Twitter activity you get to also see what kind of terms are in the tweets and so these tweets are not only analyzed for sentiment which you see down below but there's also a dictionary that helps you understand how much of a certain area is emission related and talked about this way. How much is it about food here in blue? How much is it about waste here in gray or about nature here in green? So some of you might be familiar with the beautiful gardens that exist in London. And yes, there's a lot of nature discussed there. And some of you might have walked over some of the pedestrian bridges and they are not red because there is no emission. So I think you can now redraw certain areas of a city using this smelly maps approach. By the way, it's not just London which was mapped like this but also Barcelona and many other large cities in Europe. So if you are interested to explore them other cities come over to the exhibit and check them out. Again, the exhibit has many, many different macroscopes and we are now in 2021. And the call for macroscopes typically comes out in the summertime and then submissions are due right around Christmas. Many of our clients, they are not just interested to analyze old data if you wish or analyze even today's data. They are very, very interested to understand what happens if they make a decision today, what will happen in the future? They are very interested to understand predictions. They are interested to model science and technology developments. And so we recently had an entire event dedicated to modeling and visualizing science and technology developments. There's also a special issue in PNES that followed up on that work and published some of the best work in that area. So these are predictive models, computational models that actually can forecast what might happen if certain funding happens, if certain types of teams come together and what to expect in the next five years and next 10 years so that everybody can plan for this. This is very much similar to the epidemic models that many of you are now familiar with. And however, here, papers, applications, funding and social media and other data that are used to predict how science evolves, how technology evolves in front of our eyes. There's also a new Atlas forthcoming and it will be out in August this year. It's the Adders of Forecasts that contributes different understandings of how we can model science and technology, what is possible, what's not possible. It introduces 10 different predictive models to general audiences, from agent-based models to dynamic models to cellular automata, all of which have been used, including also network models to predict science and technology developments. And I really hope that many get to explore more predictive models to help us all arrive at desirable futures, not at futures that we are stuck with because we couldn't help it, we couldn't do it in a better way, but futures that we wanted together. And I think in many cases, these models also help us agree on what futures we actually want for ourselves, for our nation, for our planet. And so in many cases, you need to visualize the modeling results, but also the models themselves in order to communicate them to a much larger audience and to make sure that everybody really has a way to contribute to our understanding of where we are going together. So watch out for the new Atlas. It actually completes the Atlas trilogy. So there are then three Atlas books for your enjoyment. The exhibit would not have been possible without the curator team and without the exhibit advisory board. So here you have major advisors listed. And there are also exhibit ambassadors that now exist around the globe, which have local copies of the exhibit in physical form. And we are happy to connect you to them so that you can explore, bringing the exhibit to your local library or to your local science museum or to other places that are open to the general public. Before I go on to the next part, I wonder if there are any questions. Charles, if you can help me understand if there's something in the crowdcast, please do. Yes, I just got one to come in now. So this is a question from Stefan Hesprigen who asks regarding the forecast visualizations. How do we visualize vagueness and uncertainty going beyond error boxes? Very good, very, very good. So as you know, the very old maps of the world, they were actually very explicit about things that are unknown. They put monsters there. They put clouds on top of Australia because they couldn't complete the outline of Australia. It was just not known. So oftentimes you see a big cloud hovering over those areas where we just don't have knowledge. And I really wish that today's map makers would be as honest, but it's hard to resist this temptation to take a data set, hit a button, have a really nice visualization that looks complete and perfect and people trust this visualization and still we know that the data was incomplete. Maybe it was only English speaking data that was used. And that can't be all of mankind's knowledge. Or it was an algorithm that has never been truly validated. That can't be good for science. So I think it's very, very important for all of us to have disclaimers, big disclaimers, whenever we use data that is new and maybe not so well understood yet or use algorithms and models that again might not have been validated completely. I think it's absolutely important. And then on the visualization side, of course, in addition to having these disclaimers as fine print, because oftentimes they get cut out by those which just wanna tweet about a new visualization, they might not go for the fine print. So on the map itself, I think it is very important to, for instance, use heat maps to indicate how certain you are about certain predictions or to use confidence intervals or to use other visual means to indicate what data is 100% is very rare but is more likely to be correct compared to other data that is less likely to be correct but still might be able to help support policy decisions or support insights that ultimately are beneficial for all of us. So very, very excellent question. And there is quite a bit of work now on uncertainty visualization. So if you put uncertainty visualization into Google, I think you will find quite a bit says not yet much on visualizing modeling results even though the epidemiologists actually have in the last year also thanks to COVID-19 made quite a bit of progress also made progress in terms of communicating uncertainty to general audiences which ultimately is very important. It's not just experts that need to understand it all. It's really everyone at the weakest links that can spread COVID to people that are old or that are in groups whereas they are especially vulnerable. So it's really important that it's not just experts understanding all of this but really also general audiences. Thank you for the question, that was a great one. Great, I'm gonna take chairs prerogative and sort of mix two questions together now that are also in the Q and A here on Crowdcast. So a bit to sort of how to put this to kind of open the black box a little bit on your side. I'm sure this could be a talk in its own right but so one person asks a rose trap assess who's making these maps? What does it mean to be a map maker these days? And then Hilal Ershan asks, so what kinds of applications and tools are you using for these kinds of visualizations in general? So is this like, are you guys using Gefi? So what's it looking like on your end? If you could maybe hard to quickly open that Pandora's box but if you can. Very good question also. So we have more than 260 map makers now. Oftentimes these are teams which work very closely together. Oftentimes also arts and humanities scholars which have deep subject matter domain expertise which for whatever reasons we're lucky enough to get to collaborate with a computer scientist and the computer scientist being lucky enough to have somebody with this very deep domain expertise. I think it really takes both. You have to understand the data very, very detailed ways and you also have to be able to manage these new tools that now are becoming available for general audiences. Really anyone can map that's the subtitle of the Atlas of Knowledge, right? Anyone can map, I truly believe in this and we have designed courses which I will show you later that really help empower anyone to do this. For the teams, oftentimes you need expertise on the data and you need expertise in the algorithms. You need expertise on how to render data visually. You might even have to have a database experts if the data is really large. You oftentimes also have to send, have kind of a writer who writes up what you're actually seeing in that visualization so that anyone can benefit from these insights. So it's oftentimes a team effort. It's very rare today that there's just one also for a map or for a microscope. For the macroscopes, you also have to have programmers because oftentimes these are using web setups where you have to have a web service running that then is available on different operating systems and a different web browsers and maybe on the phone also even though that's hard with those large format maps. Here, you ultimately need all that expertise together but you also need a team which likes to collaborate with each other. If they don't collaborate well, then you will also not get a good map or a microscope. Ultimately, many of the maps have not been developed for the exhibit but have been developed because somebody had a question or had a need to cure or to save lives or to improve health or to build better bridges or to understand the data sets that is very critically important to understand to do what better policymaking, for instance. So it's in many cases that we have submissions that come through the submission channels that we have which have made a huge difference in people's lives and then we work very closely with the map maker or macroscope maker teams so that these maps and macroscopes can be understood by general audiences which might not necessarily be the case. They might have been designed for experts because they were critically important to make certain expert decisions. So I think it's a combination of teamwork across scientific boundaries and then another steps that then translates these maps to general audiences oftentimes and also makes them available via the kiosk I mentioned which ultimately is a vessel system if you wish to then bring these different macroscopes to many different audiences. That's great, there's some more stuff in the Q&A but what I wanna do is let you keep going to the later parts of the talk and we can absolutely, they're not going anywhere we can come back to them at the end. So if your question didn't get answered yet, don't worry, we'll come back to it. And Charles, I noticed that I have one out of four parts that I have half an hour already used. So I will speed up a little bit but thanks also for watching the question list and send it to me later on. I would love to see what questions came up. Sure will. All right, so the next part is Spoke and I don't know how many of you have heard about this project. It's Spoke is the short for a scalable precision medicine knowledge engine. It's a project that tries to capture the essential structure of biomedicine and human health for discovery. It's a humongous and knowledge graph for those of you which like networks and like bepometrics approaches and like interlinking ontologies across scientific boundaries. And it's a project that's currently funded by the National Science Foundation. I think currently there are three million behind this project for the first year. And if the project does well, there will be another $2 million for the second year. So it's a very, very ambitious project. And if you go to spoke.ucsf.edu, you will find more information about it. The project is led by Sergio Baranzini. He is the PI, but Sui Huang and Sharad Israni and Mike Kaiser are absolutely essential to run this operation. And as you see, there are many different investigative teams that are also involved, including also Google, which has their own set of knowledge graphs. And we are part of this team and we are very, very interested to find ways to visualize this rather large network. And so we are in the prototyping phase for envisioning three million nodes and 30 million edges, really quite a feast in terms of just data wrangling. And so currently we are designing an interface for two user groups. One is novice users, which are interested to understand the coverage and quality of the spoke data, but also expert users interested to analyze and optimize these interlinked knowledge graphs, which make up spoke, and to decide what data sets to add in next. So here you have the different ontologies, which were all kind of interlinked and crosswalked to each other. These are all free open data sets. So spoke on purpose is available to anyone, school children, experts, industry, anyone can use that free open data. What you see here is a rendering of this network in a more cartographic way. So you can actually refer to the node at B6, which is about organisms, such as humans and mice and others. You have another node at E6, which is proteins. So the protein interaction network, for instance, is part of it. You have a link over to compounds, which might be the compounds that we eat in our food. So it's absolutely also connected to food, which is down here at F4 in yellow, which is connected to nutrients at G form. So you get to see the knowledge graph and all the many different types of data that has been interlinked in a new way in an interconnected way. See also is that, for instance, gene disease associations, there are many, many of them. That's why this linkage here is a little bit thicker than all the other ones. You also see these self links. So gene interaction networks, for instance, they would link genes to other genes. Or anatomy, part of. The glomeruli being part of the kidney, being part of the human body, that would be a part of anatomy linkage, which you see here at E9. So again, a better way of actually understanding what data is freely available now and how it can be crosswalked and interlinked. Now in the use case for patients, we decided to support a linkage that helps you understand what food is good for you if you have a certain disease. So for instance, I'm very much German, so I put in potato here. And then I might be interested how potato interlinks to coronary artery disease, which doesn't have to have heart in it, but we can actually get to this being a heart disease because that disease is linked to anatomy terms, including the heart. So by then picking coronary artery disease, which you see down here in the listing of possible diseases that are captured in this disease ontology, you then pick that particular disease and you can run a search, in which case then you get to see that food is connected via compounds to disease. So these are the compounds inside of food, but you also get to see that there's another linkage here from compounds to genes to diseases. So there is a two hop linkage, if you wish, and a three hop linkage as well. You can then go over to a much more detailed view where you get to see that coronary artery disease via ICAM-1 is linked to vitamin A, which is a compound is linked to the potato example here and is further linked to other genes and compounds. So this again is also helping us all understand what data now exists, but also very concretely, if you are interested to see a connection between a certain food item and a certain disease that your family members might have, helps you understand that landscape a little bit better. Now these large scale maps, which actually then can show, let's say 10,000 or 100,000 nodes in their interlinkages, are created in collaboration with Stephen Kuburov from University of Arizona. And you can use them to then zoom in to these maps and they look much more like a cartographic map, less like network. And you get to see also again coronary artery disease and how it is interlinked to other compounds and genes. Now this is actually very much work in progress. So these laying out a million of nodes is not possible right now on a computer screen. You can render that offsite, but we want this in real time. And so we have to collaborate with those which can parallelize algorithms. We have to collaborate with those which can identify algorithms that are scaling to that size. And so there is a Dachstuh seminar coming up very soon, April 11 to 16, which will be very much socially distanced. We have 25 experts, which I'm still planning to come in in person, including myself. There are very strict coronavirus measures at Schloss Dachstuh, which you see up here. It's a really very beautiful castle which has been a very wonderful place for computer scientists to run these kind of events. Stephen Kuborov and myself, we are the organizers for this event. And it will be virtual slash in-person events. So there will also be a possibility to attend virtually. And that causes all kinds of challenges, but I think we are up for this challenge. As part of the Schloss Dachstuh event, we also have another part of the exhibit. We are inviting maps and microscopes by those which are participating in the event. And so there will be another set of maps becoming available very soon. There's also a special issue planned on multi-level graph representations for big data and science. So this will be appearing in the July, August 2022 special issue in naturally computer graphics and applications. And so again, those are few which are planning to attend the Dachstuh or those of you which are doing work in that area, you're very welcome to plan to submit and the articles will be needed for review on December 29, 2021, this year. And I think this is the end of this particular segment. So let's see if there are any questions about mapping millions of nodes. So this is really trying to help us all get to algorithms at scale. I didn't see any new questions on this part. So why don't we keep going? And if people have them, they can add them and we'll double back. Thank you so much Charles. All right, next in line is yet more data. So if you look at the human body, our current best estimate is that we have about 30 trillion cells that make up the human body. And there's an ambition out of the human cell atlas and the NIH funded Hapna project to map these human cells to help us understand the healthy human adult body. And it's non-trivial. These single cell technologies are just becoming available. It's still quite expensive to actually run single cell analysis. And it's also quite unique that you would get to see organs and other parts of our human body at the single cell level. So we are talking about microns, right? The human blood cell is 10 to 20 micrometers. So it's very, very small scale. And the ambition is that you could actually go from the whole body, so two meter if I would be taller, down to the organ level. A kidney is about 10 centimeter. Down to the functional tissue unit level, which is, for instance, for the kidney, the chlomeruli is the functional tissue unit there, which does the exchange of blood and urine. Which is at the 300 micrometer scale. And then down to the single cell level. It's a phenomenal challenge. And I think it's wonderful to see all the many experts that are now working on solving this challenge. And the promise is that if you would know what is normal and healthy, then you would have a better way to understand what happens when disease strikes. And so the vision of HubMap is to catalyze the development of an open global framework for comprehensively mapping the human body at cellular resolution. And again, there is also a similar effort going on in Europe and actually across the world, which is called the human cell atlas, which has a very similar ambition. And actually both teams are now hosting joint conferences and events. And many of us are working across multiple projects. So here you have some of the goals. So the absolutely must accelerate the development of next generation tools, make it much faster and more robust and also cheaper ultimately to map at the single cell level. We are interested to generate foundational 3D tissue maps. So the human body, it actually matters in which context a certain cell operates. And it's not just a 2D context, it's a 3D context. And we need to capture that. We are establishing an open data platform so that many can benefit from, in this case, US taxpayer money, which is spent on making HubMap work. But we are also collaborating this effort with other funding agencies, programs, and the biomedical research community. So for instance, recently, there was a pitch event, this NIH and the Sean Zuckerberg Foundation and many other funders, where different investigators got to pitch their best ideas of how to best make progress and then got feedback from different funders and I think also got to benefit from some of the funders. And ultimately, we want to demonstrate the value of these reference maps for improving health but also for curing disease. Here is the setup for the HubMap program. So there are different HIFE components. HIFE stands for integration, visualization and engagement. There are five teams, two tools development centers. There is the infrastructure and engagement component and there are two mapping centers and I'm leading one of the mapping centers. The other one is out of the New York Genomics Center. There are quite a number of tissue mapping centers and TDDs and RTIs. We just onboarded 11 new teams that are now all starting to develop new technology and to rapidly implement this technology and to create tissue data. All this data gets collected in a big data portal out of the Epitsberg Supercomputing Center and close collaboration with CMU. And then there's assay analysis and data sets get compiled into 2D and soon-3 maps which are segmented and annotated tissue maps. Ultimately, maps get generated and that's really a big question. How do you generate these reference systems and then stored and served to the world? As you see here, we care about many, many different organs. Right now, the count is at 17 within HubMap. So there are 17 quite different organs. Some are clearly long, like the colon for instance. Some deflate if you take them out of the human body like the lung. Some are beating, some are rather small like lymph nodes. So these organs are vastly different and yet we want to find a way to register the tissue in a uniform way, but to also create a map that is capturing the 3D location and context of these single cells in an explicit way and so that other types of research can happen. Here, you see the general process of map generation and assembly across cellular and spatial assay types. So in the top left here, you see that there's genomics, epigenomics, transcriptomics, proteomics and metabolomics data and different tissue mapping centers specialized in different types of assays. You have oftentimes one tissue section for which multiple assays are run. You oftentimes also have the next inline tissue section on which different assay types are run. So you have a sandwich kind of system which can all help improve the quality and richness of the data. And ultimately landmarks or scaffolds can be used to then align different tissue sections, create a more 3D reference, but also to then get us so-called common coordinate framework. So let's look at what it actually means to have a common coordinate framework. So here you have requirements for this CCF as we call it. It must capture major anatomical structures, cell types and biomarkers and also there are interrelations across multiple levels of resolution. It should be semantically explicit meaning it has linkages to ontologies including the ontologies you just saw in the spoke knowledge graph. So for instance, in the anatomy note you have many of the human body parts named already with definitions and ontology IDs. So we are working very closely with ontology curators for uberon or cell ontology or HGNC to crosswalk anatomical structures, cell types and biomarkers over to the existing ontologies. Oftentimes also causing a need to update these ontologies or to slightly change definitions for ontological terms and it needs to be spatially explicit. Initially we are assuming a part of structure so there are cells at the cellular level so 15 microns as you see here that are part of larger functional tissue units and here the scale bar is 25 micrometers then up to the larger functional tissue units. Here for instance the nephron inside of the kidney and then the organ itself at the centimeter level and then towards the or part of the human body. And we are currently looking at two references one for male and one for female. We are using so-called ACT plus B tables to have experts agree on major anatomical structures, major cell types and also biomarkers that are used to characterize these cell types. And we can then walk over these ACT plus B tables to ontologies and to reference objects which now are making up a reference object library. So here you have a kidney with this interior anatomical structures and here you have a glomeruli this single cell resolution if you would zoom in further. These ACT plus B tables really capture AS over here and then cell types in the middle and then a subset of marker genes here on the right. They could also capture proteins or lipids or metabolic markers on the right hand side. So it's not just genes, it's also other biomarkers that we are interested in capturing. The first table was actually captured and published by the KPNP team. The kidney precision medicine team. And here you get to see their very first table which has increased since and also improved in terms of detail. Our team implemented the ACT plus B reporter interface which helps you to actually understand that these tables are not really tables. They are quite complex data structure which includes a part of potronomy tree. A typology is a tree. And then there are bimodal networks between cell types and anatomical structures and cell types and biomarkers. So it's two bimodal networks and two trees which you see here. And it's non-trivial for experts to alter those but with the user interface you can actually hover over nodes to get details, go over to the ontology terms and their definitions. You can also then see what biomarkers are connected to a certain cell type or what cells can be found across different anatomical structures. And so in order for many, many different experts to agree with each other, pathologists, anatomists, surgeons which oftentimes be quite different languages but also agree with the language that's now spoken by single cell biologists and medical experts and computational biologists. We have monthly working group meetings and in the chat you will find a link to the working group charter but also to a registration website that lets you register to receive more information on this effort but also on upcoming meetings and on publications that are relevant, et cetera. So please, if you are interested to learn more please sign up as an expert and also join us for these meetings. I think it's a quite challenging effort to bring so much expertise around the globe together to help us all understand what counts as healthy in the human adult body. It turns out that there are many, many different teams that are funded from different sources that all care about different organs. Here you have a compilation that was done at the HCA hot map meeting in last year's March event. So about a year ago, NIH and HCA got together and created a set of sessions that really brought experts from around the globe together to discuss how to best make progress on getting us a human reference at last. And as you see, there are some organs like the kidney, for instance, which are studied in many different efforts. So whenever you see a one here, that means that this particular hot map effort, for instance, cares about that particular organ and studies it has resources to get data and analyze data and use it for map making. But you also get to see that there are many, many organs that are cared for deeply. And as you know, if one of them goes wrong, you're in the deep trouble. So I think it's very important to have a more holistic understanding of many and to have a reference at this that really is not only specific to one organ but that really works across organs. And so the organs which are here in yellow, they are the organs we're currently focusing on and other organs will come next. So we have 11 organs right now, there are another 11 organs which we will have these ASTT plus B tables but also reference organs towards the end of this year. So I think it's very exciting to have 22 organs in relatively little time. But there's also a lot more work to be done. Here you see our initial collection of reference organs and those of you which have used anatomical atlases online or as a medical student might not be too impressed but the impressive part about these organs is that they are freely available, anyone can use them and they're also connected to ontologies. So you can click on any of the parts of these organs and you will get to an ontology term. And these ontology terms are also used in the ASTT plus B tables. So you have a correspondence between the 3D reference organs the ASTT plus B tables and these 3D anatomical structures. And as time progresses, these structures will become more and more detailed because there are many, many experts that deeply care and wants this kind of information out in the open for anyone to use. So here again, you would have for instance, table for the heart and that heart then connects to larger anatomical structures just ever smaller anatomical structures and then over to cell types. So if you hover over this particular vasculature item here on the note on the left-hand side then you get to see all the cell types that are commonly found in the vasculature and you also get to see the biomarkers that are commonly used to characterize these cell types. So again, if you're interested to explore this there's a link you can follow to check out your own favorite organ. Now over to the mapping part some of you might actually be in the business of creating tissue data. So what you see here is the common documentation of tissue extraction sites. So you have a kidney, it's butterflyed and you now cut out a certain segments that will undergo single cell analysis. Typically this information is captured in photos or in videos but it's very, very hard actually to get from there to an out in a unified way of capturing this data across organs. So my team as part of HUBNEP implemented the so-called registration user interface which now uses the reference organs you just saw to register tissue blocks here in yellow. And so after you extracted a tissue section you are then bringing in the width and height and depth of that particular tissue up here. You can then move over that yellow part into the area where you just extracted the tissue and you also get to rotate it so that it fits right in there. And as you do this you also get anatomical structure tags automatically assigned based on collision. So there are internal structures here of the large intestine which will collide with your tissue block and based on these collision events which you might be familiar with from computer games you automatically assign these ontology terms. So that is a very robust way to register tissue not just for one organ but really across many, many different organs. We have a carousel here which you can use to pick organs. Also soon pick left and right and male and female so that this all is done in a unified way. Here you see different extraction sites that have been identified for different organs. So for the heart for instance you have quite a number of extraction sites and if you happen to take tissue always from these extraction sites so that you have data across sex and across age groups and across ethnicity, et cetera then you can just take that extraction site which has already been registered and we already know what kind of anatomical structures collide with it and just assign it to your tissue block which makes this registration ever more faster but also optimally leads to the fact that we know that this comes from the same area. Here in the lower right you see the tissue blocks that were registered inside of HAPNAP for the very first HAPNAP portal release last summer and you see that for the spleen for instance there are these three bands where tissue is typically extracted. For the kidney it's typically in the upper pole and the lower pole where kidney tissue is extracted. And this is just helping you see the extraction sites, analysis and annotation based and collision events. So if you move your tissue block here you have quite a number of anatomical structure tags here you can add your own tags they are automatically assigned then to your tissue block but you can also just delete some tissue block because you know that anatomical structure couldn't possibly have been in your tissue. With this information in hand we can then go over to the CCF Exploration User Interface the EOI and I think you have that link also in the chat and you can start exploring that tissue in the context of the human body. So these white parts which you see shining through here the spleen and also the kidneys those are actually the registered tissue blocks and as you zoom in you get them larger and larger and you can select one of them and get to see what tissue data actually exists. You can also zoom in even farther and you get to see that three of the tissue blocks are actually biopsies which were got registered by the KBMP team. And on the right hand side you then get to see data sets that are behind these in one case it's a publication and if you click on here you would get over to that paper which documents that line of work but you can also in other cases go over to data portals which have their own access restrictions which have that particular data. So it's a very neat way to have kind of a meta portal to data that's semantically and spatially explicit and registered using this evolving human reference plot loss. Ultimately, you might like to click on a tissue section on the right hand side of the EUI which then brings you over to the Vitesse viewer which is developed by Niels Galenburg's team at Harvard Medical School and that fewer than let's you zoom into the tissue sections you get to see different stains for instance DAPI which stains for nuclei and you get to see glomeruli here in these yellow marked areas that help you understand the micro anatomical structure inside of the kidney but then also as you zoom in and get to a single cell data helps you understand really the context in which single cells operate. There's also a few which would like to learn more about hot map and about single cell analysis and assays in general. We have created visible human MOOC. It's freely available for anyone to use. I think it's also, there's a link in the chat again and so you can register for free. You can join a community of students many of which are experts in their own respective fields of research. You will get to see lots of videos, interviews but also tool demos but you also have a number of hints on exercises and you have a number of self quizzes of course. And so in the current visible human MOOC or the age MOOC as we call it you have a number of modules. You have an overview of hot map. You have information on tissue data acquisition and analysis featuring Vanderbilt University setup. You also have an introduction to Surat which helps you predict cell types when uploading genetic data that is for instance generated by 10x technology. You have information on the CCF ontology that I just introduced very briefly. You have demos and interviews with portal designers and you also have a pitch for open consenting your data. And we just added two new modules. One is on ontologies. What are they and why are they useful and how can you use them? But you also have another one on these AST plus B tables. And again, it's quite phenomenal how many experts are now contributing to these tables and we could never do this kind of work without those contributions. So thank you, expert. Thanks to all of you. This is the hub map team. I would like to point out that most of the models you just saw were created by Kristen Brown at NIAID. And then there's a number of students also and many staff members on our team here at Indiana University which are contributing to this effort. And of course, there are many patients which have agreed to volunteer healthy tissue and to this effort. It's non-trivial to give up some of yourselves to advance science. So thank you for that also. So let's see if there are questions. Yes, so just really quickly as well to say so we're right at an hour but I think that's okay because we've been slowing down for Q&A. There's two questions in the chat about this section in particular. First question coming from Stefan Lindquist who asks, it seems ambitious to start with humans. Why not pick a simpler organism? Was there advanced work in simpler model organisms or was it a bit of a moonshot project? I think there has been a lot of work on mice, on pig, on rat, on monkey but I think we are ready for the human body. And as you know, it's much transfers but not everything transfers if you go from one model organism to human. And so we believe we can actually do it for a human and it would be very, very valuable to have a human reference. And there are many other teams for instance, the spark team which absolutely have multiple species as a focal point. And then they see how much can actually be transferred for instance, say focus on the nervous system, how much can be transferred for instance from rat and pig to a human ultimately. But yeah, it's a very ambitious project but just like the human genome project couldn't have been done for mice. I think it had to be done for a human to get the human genome. Here we also believe that it has to and it can be done for human. And again, sometimes it's actually good to have a moonshot because it focuses attention. It gets everybody excited to work quite hard and quite across disciplinary and institutional boundaries. So it's not just research institutions, it's also industry and of course government labs which are here all working together. But it is a major challenge, I agree. Great, thanks. One more question on this section in particular. This is coming from Nikola Bertoldi who asks, bit of a general question, but what's the concept of health being used in the project? That's an interesting invocation. So I wonder if you have thoughts on that. Yeah, so for instance, the Kidney Precision Medicine Project, they have started with Sanjay Jain being right here to use C-HUB map data to compare this disease tissue with data. And it's really wonderful to have a reference of what is normal because then you see exactly what changes in different disease stages or changes independent on age or sex or BMI. And so having a healthy reference for you is very, very important for understanding what happens when disease strikes. Great, all right, let's keep going then and we'll come back to some of these, the questions that are left in the long Q and A, thanks. Very good. All right, last part here is how to empower yourself. And I hope you all brought your own data, I'm just kidding but I think you should bring your own data tomorrow the week after to really understand how this data can be mapped and understood and visualized in new ways. So I already mentioned that we have been very interested in understanding, measuring and improving data visualization literacy. To be honest, COVID, the pandemic has absolutely increased humans' ability to understand prediction graphs, temporal graphs, how the number of cases will increase if you don't use face masks, how in some cases the capacity of hospitals is much lower than what's actually predicted to be needed and it's very scary to see those predictions. And fortunately, many people have been able to read these maps and have been able to respond and act on them. And it's really about the acting parts that's near and dear to our hearts here. It's not just understanding. It's a way to then do something differently because many times if you understand, you could help it. You could improve the state of this planet. You could improve your own health. You could improve how people learn. You could improve many, many things if you just would understand it a little bit better. And so here in this line of work, we first of all need to define what we mean by data visualization literacy. We believe it is a combination of literacy itself, being able to read and write text, but also visual literacy, being able to compose images and legends and titles and everything else, but also mathematical literacy, being able to understand how a scatter plot works, for instance, it's a two axis has a dot, you have to project it downward and over to read off these values. So that's all a literacy that is taught in classrooms. What's not yet taught in classrooms, is how to create network layouts or how to create cartograms or other more advanced visualizations. And we believe, however, that this is really a 21st century needed skill where you absolutely have to make sense, not only of your own personal data, your health data, your bank account statements, your genealogies of what you might have inherited from your parents and grandparents, but also in your professional life, of course, you will have much data to deal with. And it's, in many cases, it's also easy to take that data and visualize it and gain true insights, actionable insights from it. So the data visualization frameworks that my team has developed in collaboration with many, many others, focus is not on reading visualizations, but on making them, on constructing data visualizations. And we believe that reading and constructing data visualizations need to benefit from what we know about human perception and cognition. So we work very closely with cognitive scientists and psychologists and others which know what the human perception and cognition can do and what it cannot do. We also built and are appreciative of much work done in cartography, in psychology and cognitive science, statistics, scientific visualization, data visualization, learning sciences. We need all of this. We could not do this just by looking at information visualization research. And it needs to be theoretically grounded as a framework and practically useful. And ideally it can be easily learned in 30 hours of time, let's say. So if you have five hours a week for six weeks, you should be able to really render data in new ways to do a temporal, topical, geospatial and network analysis and visualizations. And so we have designed courses that do exactly this. You need 30 hours and after that, you have a lot of expertise and knowledge under your belt, so to say, to make better sense of your own data. Given that there are new data sets and new algorithms becoming available all the time, it's important that this framework be modular so that you can very easily extend it. And so the framework was developed using a very extensive literature review. Some of you might have the Atlas of Knowledge. You have about 1,800 references in there, many of which are actually helping you understand how data can be visualized in meaningful ways. I then had the pleasure to teach at Indiana University and using this framework, we got to see what normal IU students can learn. And then we had the Visible Human MOOC, or here the IV MOOC, sorry, the Information Visualization MOOC, not the Visible Human MOOC. And in the IV MOOC, we had students from around the globe. So we got to test it, not just on IU students, but also on many that were already in the workforce and from high school students to AD+, if you trust the self-reported age groups. The framework then was further refined, getting feedback on many client projects that we have been running over the many years here at IU. And we also, of course, got a lot of student feedback on what works and what doesn't work for them. And the framework that's then used, is used in the Visual Analytics Certificate course and you should have a link to that in the chat as well, is then used to systematically in a very engineering, pragmatic way, construct and interpret data visualization. So it's not an art form, it's a very systematic way, just like you can cook if you have a cookbook. Here you can use that process and get from data to insights, ideally actionable insights. So how does this framework look? So it has two parts, it has the typology here on the left, which identifies different needs, different data scales, analysis, visualization types, but also different graphic symbols, graphic variables and interaction types. And all of those are detailed and defined in the PNAS papers that comes with this work, but also in the Atlas of Knowledge. And then on the right-hand side, you have a DVL workflow process that defines five steps that are required to render data into insights. So you ultimately have an interlinkage between those two, so you have these seven types and they can be overlaid over the process model. And specifically you would take stakeholders, sometimes it's just yourself, but oftentimes it's colleagues or clients you have, you identify different insight-need types together with them, you then acquire the right data and the best data you can afford or get your hands on. This data comes with different data scale types, you then analyze that data and you visualize that data. So one, two, three, analyze, acquire, analyze and visualize. And the visualization has another three steps. You pick a visualization type or a reference system, like let's say a geospatial map, you overlay your graphic symbol types, let's say for instance, all people that are infected by COVID right now. And then you use graphic variable types, size, shape and color coding, for instance, to indicate other data variables. For instance, it's a male or female or what age groups they have. And then you would deploy your visualization using different interactivity types. And then you would start interpreting your visualization, you would realize that maybe one of the data sets you used is not the highest quality, so you have to go back and do one more round or you get to see that there is something happening in a certain geospatial region. So you zoom in there in order to zoom in, you have to load in more data for that area. And you get to oftentimes do this cycle again and again and again as you generate more new questions about the data in hand or data still to be acquired. Oftentimes data visualizations inspire the acquisition of new data, data that has never been captured before in that way. So what you see here is a very general process that you can apply to temporal analysis, to geospatial analysis, to topical analysis, but also to network analysis. And so ultimately there is an interplay between picking a reference system and the data overly and actually very much depends on the deployment mechanism you pick. If you have a print out, which is very comfortable to use for many, then you might not have much interactivity besides people getting closer to your visualization, which of course is powerful because you could print in 300 DPI, which is better than 72 DPI, which you have on most screens. So here you could also pick a mobile device to ultimately deploy or a large displayable. So depending on what deployment mechanism you have, you might have to pick a different reference system just to fit it all. And then correspondingly, the data overlay has to be a little bit different. Here you have an example that I mentioned before. So you pick reference system, for instance in the app of the US, your overlay, let's say flight traffic routes, and then you might also color and size code the links and the nodes themselves. In order to teach and practice this combination of process and typology, we have developed so-called make-a-vis environment where you load in a dataset, for instance an ISI publication file, you can then pick a visualization type and then you can select graphic symbol types and graphic variable types to render different temporal bar graphs. Here, for instance, of topic analysis results for linguistic analysis. So as you see, big data had a burst of activity between 1999 and 2002. And then there are other terms which are colored in a certain color based on another attribute value. And then a computer had a humongous bird between later years. So you can very much use this to also, for instance, visualize linguistic activity of term usage over time in a quite easy and readable way. But you can also develop science map overlays or geospatial maps with network overlays. So it's quite flexible in terms of civilization that you can create. Ultimately, this typology has a lot of related work and prior work behind. So if you just zoom into inside needs, you will get to see that starting with the tongues pioneering work in 1967. We have a lot of other pioneers that have contributed to our understanding of what types of tasks actually exist and what types of inside needs exist. And on the right-hand side, you then see the one that's used in the Atlas of Knowledge. And similarly, you can zoom into visualization types and you can agree or disagree on how these are named. So there is not a general agreement among visualization experts, for instance, how to define a chart or a graph. This is the definition that is used here. And I think it's more and more accepted, but not fully. You also get to see how understanding one reference system here, a table that has a cell, which is defined by a column in a row can then be used to transfer over to a graph-based visualizations where you have an X and a Y axis, over to map-based visualizations which have a longitude and latitude axis, over to networks where X and Y is different from graphs, but still can be used to refer to a node, just like what I showed you on the spoke graph. So even so, the network is laid out and can be rotated and mirrored any which way it wouldn't change the stress in the network layout. Still, it can also have an X and a Y so that you can refer to specific nodes. And these reference systems are very important to understand before you even start plotting any kind of data. Ultimately, if you have a reference system down here, you can then start adding data overlays, sometimes just dots in a scatter plot or nodes and edges overlaid over a geospatial map or a science map or a network graph. And as you see here, you oftentimes have just the visualization reference systems and the graphic symbols and then the graphic variables. So color coding, size coding, shape coding, et cetera. And actually these graphic symbols could be much more richer than they are commonly supported by today's tools. So you have not only geometric symbols such as point line area surface volume but also linguistic symbols which you can also size code and color code, et cetera. You also have pictorial symbols, images, icons or statistical glyphs which are widely used in cartographic maps not yet so much in data visualizations. And then you have graphic symbols and graphic variables and you can create a few that really helps you understand how these two are interlinked. And so here you have some of the graphic variables. Many of you are familiar with form and color. Optics and textures are not so easily contributed and supported. Motion very much in animation but maybe not widely used yet in data visualization tools. We also know that we can be as human beings can distinguish for instance, position and lengths very easily. Angle rotation not so easy and it's not as accurate that we can distinguish them. And also area and volume again, it's harder for us. Color use we only can name a few and distinguish a few if you have a large data visualization. So I think that also helps you understand which one to use first if you just need one use position. If you need another one use position and length, et cetera. Ultimately these two variables, the graphic variable types and the graphic symbol types create a very large table of different options you have for visual encoding. And they are qualitative and quantitative attributes that can be encoded. And ultimately this table is rather large and there are some empty places just like in the periodic table of elements we just don't have any good example here yet. But I think there's a lot of works that could be done by future generations of PhD students. And again, if you would like more information on this there's a very large version of this in the Atlas of Knowledge. That's hard to reproduce here. And then for those of you which are now interested to actually learn how to empower yourself there is a course which starts May 17th. So this is a good time to register. And that brings together oftentimes industry experts which already have a full-time job. So they will take this course in the evening and over the weekend. And you see also some of the companies which we had previously take this course and we have now also entire cohorts of them go through this course. Ultimately the course is 30 hours total and you need six weeks, five hours each week to then learn in a group of other experts how to render your data visually. And you will get to use the make-a-vis to actually try this out. There's also a my project assignment where you pick your own data set and it could be a personal data set or a professional data set or another data set you can deeply about. And you start analyzing that data and visualizing it in a temporal, in a geospatial, in a topical and in a network way. And we hope that it really helps you understand how different tools support different types of reference systems, different types of data overlays but also a different types of interactivity so that you can in a meaningful way talk to other experts that might then in the future implement new functionality for you or you as a programmer yourself might now get interested to empower and to use new tools in different ways. So that's this part of the talk. Again, there's a lot of readings that you could also do. So I have some of the books listed here. There's also an entire book on models of science dynamics from the Springer. So I'm happy to answer any questions you might have but thank you for your attention and thanks Charles for organizing. I think this is very much needed in today's time and age to look at digital science and how to make it relevant for experts and general audiences but also to bring together people from across the sciences and arts and humanities to exploit together and explore together how we get to more desirable futures for all of us. Thank you. Fantastic, thank you so much. Yeah, we have a little bit of time and a little bit of questions both. So let me, I'm just now I'm just gonna work down the whole list of what we have here. So let me start with a question from Daria who asks so what are the most used metrics for your group for mapping science? So there's a broad use of citation analysis and co-authorship but in the contemporary landscape what kinds of new metrics do you see emerging? That's a very interesting question. So as you know, there are also altmetrics looking at all kinds of other metrics besides citations and funding but in addition to altmetrics and in the spirit of talking about digital science I see more and more also linkages that papers have over to datasets, over to code over to protocols on protocols.io. And ultimately I think what many of us want is a rerunable paper. So you would be able to as an also as a reviewer and as a reader of the paper you would be able to plug in your own data to run it through that new algorithms that just got proposed and to see what it does to your data or you can take a new dataset and run it through your well-known workflows and get to see what it does or going forward it's also a matter of researchers not only providing papers and publications but really providing datasets that others can use and oftentimes free data needs a little bit of free code also because you need to pass it in a certain way you need to analyze it in a certain way and open data and open code also goes extremely well with open education. So the visible human MOOC is all open and freely available to anyone. We would like to empower many to embrace these new single-cell technologies and to help us all understand what it means to be healthy or what it means to age or what it means to have a disease and how it can be cured. And again, I think there's a lot of appreciation for science right now because of the vaccines that in the US are now becoming widely available and hopefully also in Europe, soon in Germany specifically. But ultimately I think it's not just papers that we want we want to go beyond papers and go over to products of science that are immediately valuable to those which pay for science which are the taxpayer. Great, thanks. Let's see a couple of others. Let's see how far we can get in a couple of minutes. Very, I would encourage quick responses if you can. How briefly do you become a map maker? What's the five-second version of the application process for that system right now? Come learn with us. Just come into the course, learn with us. Okay, awesome. That works as a quick answer. These are amazing tools. This is another question from Stefan Lindquist. These are amazing tools. We're curious about the best practices that surround interpreting these visualizations. So how does a map maker determine whether a pattern is real as opposed to an artifact of the dataset? Are there various kinds of heuristics that you use for that? Yeah, I think you need to be honest as a map maker. Former map makers say science maps, right? And today you have to disclose the quality of your data of your QA, QC measures of your validation steps of what you saw in the data, but then also go back to experts and have some interprets of data. And oftentimes such close collaboration with subject matter domain experts, it's the most fun and most rewarding part because without that, you develop tools that maybe never ever get used. And that's no fun for any programmer. And it is challenging to work across disciplinary boundaries, but it's also highly rewarding. And in results and visualizations to truly change how people act. And I can't get to the last question. So let me just apologize to Beckett. My apologies. But for the moment, let me just thank you because of course there's the way that this disconnected symposium works. You can't see there's all kinds of thanks and appreciation flowing in in the chat right now. So a hearty round of applause from everyone on our end. Thank you so much, very much appreciated. Thank you.