 Okay. It's 11 o'clock Eastern time. So welcome everyone. Arriving to our medicine 2023. I'm Peter Higgins. I'm the chair of the organizing committee. For this year I my day job is at the University of Michigan where I'm a gastroenterologist and very interested in reproducible medical research. So to start, I just want to mention our code of content conduct. It's important just to simply be kind be respectful and share what you know with others and with the community. You can use this qr code to view the entire code of conduct. I want to thank our sponsors jumping rivers and posit. I want to thank our recruiting partners who at our job fair last night got to talk to many, many folks finishing their PhDs preparing for jobs or preparing to switch jobs looking for employment in medical data science. Many doing it remotely from across the world. I want to thank our attendees we actually set a new record this year with 707 registrations and folks from 61 unique countries. We had spectacular workshops on Monday and Tuesday. More workshops than we've ever had before and a wide range from using red cap data vis data cleaning, and a lot of really useful skills that hopefully people learned a lot from. We had demo day on Wednesday this was a first for us demonstrating specific packages or approaches. And then on Wednesday night last night we had our first late night with a poster session job fair and for the first time a live data analytics coding on Twitch with tan hoe, who saw a data set he had never seen before, and was able to create really nice interactive tables in under an hour. Very brave of him to code live in front of an audience. Today we're going to lead off with Neil Batra from applied epi doing our keynote. We'll have three dashboard theme talks, and then another Twitch session, and if you don't know how to get there. The, if you go to the schedule and click on the button adjacent to this talk it'll take you to the Twitch stream where Eric Nance will be live coding a shiny app from observational data. Then we'll have a break a few data vision reporting theme talks, then another break and finish up today with data quality and streamlining your workflow theme talks that will take us to 515 Easter. Tomorrow, look forward to a keynote from Jeff leak from Fred Hutchinson Cancer Center, a session on shiny modeling and sample size short break and then a panel, also in zoom on AI EMR and the ethics of using AI in that situation. And then five talks on gene expression mass spectrometry data, another break and then a panel on the benefits of having your own local are packages, and then five talks to finish up on applied are in medicine from dangay to heart murmurs. Really wide range of what people are doing in their medical settings and that'll take us to 530. As I mentioned, we have an incentive this year for filling out the meeting evaluations form. We have a set of heck stickers from many of the presentations and workshops that have been used this week, and you will receive all of these heck stickers by mail if you're one of the first 300 people to fill out an online our med 23 evaluation. So I think we'll be able to supervise folks we hopefully will get a lot of evaluations and feedback this year. People ask about recordings these will be posted on Friday, June 23 on the our consortium YouTube channel. So you can find it by searching within YouTube for the our consortium clicking on playlists, and currently you can find our medicine 2022 there will be a separate review from our medicine 2023 from last time we have 38 videos and these get a lot more play beyond the folks at the meeting several with over 2000 views since the meeting itself. I want to thank our sponsors and our organizing committee who met weekly to put this all together. This is actually Steve Schrager from the program committee. They did the abstract review and selection and helped organize this into a coherent whole. Our talks will be on zoom and the speaker if the talk was recorded and or a co author will often be live in the chat. So we encourage you to ask questions and engage the speaker in a live discussion, share link share related ideas and discuss the topic interaction is really the key. Now, pass this over to our moderator for our first session, Ray Belize who will be introducing Neil. Hello everybody. Welcome from sunny Miami, Florida. I'm Ray Belize some of our statistician here at UM. It's my distinct pleasure to introduce Neil Batra. If you look at his CV it is a collection of the most impressive letters of the alphabet, including CDC, USA ID. So, his work is basically in my mind been all about grassroots epidemiology, and it's given rise to the applied epidemiology project, which gives applied epidemiology and public health tools. This is a form of an online book with 50 of the most beautiful modules I've ever seen produced on topics like data management data summary, numeric and graphic summaries, as well as data dissemination. So, without further ado, Neil, take it away. Thanks so much Ray. Let me just share my screen and we can start it. Dr. Higgins, Carolyn and all of the others on the organizing committee for inviting me today. It's, it's an honor to be here and I'm very happy with to be sharing really storytelling today with you, because I think the story I wish to share today is about a group of epidemiologists who recognized a need in their community and sought to meet it to make it easier for their colleagues around the world to adopt and benefit from our. And so as I share some of the lessons that we've learned some of the approaches that we've taken to help people in our own, really our own, our own colleagues. I hope that you can take some as you think about how to make our more accessible in your own domains. So just confirming, can someone in the crowd that the screen is visible. Looks good. Thank you. So the story begins with a bit of a premise and really this was a premise but it's also sort of a visceral experience that myself and my, my colleagues experienced which is that most epidemiologists around the world are using either standard or way too expensive analytical tools. And by and large they want free open source tools like are that allow for versatile data wrangling and analysis, but there's a lack of relevant training and support and so that's what sort of underpins all of this. And I say this was an experience because many of us even before the coven 19 pandemic who knew are would receive emails and requests on a daily basis for assistance in trying to learn our so that in a way that was was relevant to sort of frontline practice. And that leads me to my next slide, which I recognize is a bit crude. So I hope you forgive me for that, or sort of simple. It's simple but in an epi there's a difference in, and there was a difference in how our was being used right so an academic epidemiology. We've widely accepted now there are many resources available. But in this applied epidemiology, or sometimes called field epidemiology although we're trying to move away from that term, where the there's always a time pressure. The objective is really operational insight for the disease control at sort of ground level, and it's really simple descriptive data that is consists of most of the work. And in this space, our adoption is lower than it could be in reproducible workflows are just less common. And so I think, even if there were our resources available in public health, you would find that they were focused on the kinds of tasks that are more common in academic epidemiology, as opposed to those that are the day to day work of those who work into at the ground level in public health. So I guess I wanted to share a short video of my colleague, Dr. Chupuma here, because he expressed sort of this sentiment of trying to learn our. Most times in the past, I try when I care about how I want to go online and look at the program and learn the program. I see that the examples that are using from the economic markets forces from predictions, some gambling and other things, but no one has because they are not epidemiologists. They don't understand the nature, the variables we work with, they don't understand the way we think and the type of output we want to produce out there. So having an epidemiologist that have experience in the field that have passed through those challenges, sit down and use those same codes to write simplified scripts or programs that can do the work we need to do as epidemiologists. And that makes learning out using applied opinion or simple. And so I think what he at the end there was saying is, you know, having someone who's relatable who understands your day to day challenges whether it's really messy data sets or the data collection that is so much of epidemiology understanding where the data are coming from and the biases that are inherent in them. And here's the, here's a, here's a story. I was working in Haiti in 2021 with a colleague, a senior epidemiologist there who had used our epidemiologist our handbook to transition his daily routine of disease surveillance to our, and the impact here is really undeniable right going to the point and click manual workflows that took the, I would say 80% of his week and switching that to being under an hour. You think of the power that that has for the workforce and really for the brain space of people working in these kinds of jobs. So here, again, just a brief quote on that. And then the data collection process is not under your control. And they have to do the cleaning over and over again every time you have to use the data set to update your reports and other things. It takes away a large amount of time that can be invested into thinking into the other aspect of the investigation. So, I think it's what I hear from there is it gives us space to be epidemiologists and not be working in Excel all the time cleaning cells manually and things like this these scripted reproducible workflows are very freeing in that way. And I know I'm speaking to people who believe similarly that the power are, but in public health. We wanted to run a survey and assess a little bit more structured way the demand for our and so what we found was that almost all at these were feeling constrained by their point and click analytical tools. That most said that our training should be a high priority, and that very few said that their agency had sufficient capacity to train their epidemiologists in our. And so to us this signaled a lack that there's a gap here that needed to be addressed. And I think it's important to step back and just mentioned that in our is not just a new tool coming in but it's also a new culture right the idea of open source. It is. It's a paradigm shifting switch right not only for the end users who actually become creators along the way, but also for the sort of more systemic shifts about employability and representation and really local problem solving and I know that you all can appreciate that. We come in applied epi is really a grassroots movement, and I say that because it's it arose organically from the ground we're not affiliated with any major institution or university, but we are now a registered nonprofit organization. And we are seasons experienced public health practitioners epidemiologists who have sort of passed through those challenges as to woman was saying of the messy public health practice. And this has resulted in us having very strong relationships and partnerships locally with national health ministries and internationally with organizations like the World Health Organization doctors without borders or msf. And also teffy net, which is one of the worldwide associations of field epidemiology training programs, which are present in most of the world. So we focus on frontline epi and public health. So training that is relevant to local practice tools, making them more accessible is accessible for agile analytics. Support, meaning sustained support, and I'll get to how we do that later and ultimately when we advance the these tools, it actually improves the methods of analysis and the standards for sort of methodological practice, we think. So here's what we've devised in our for our sort of ecosystem to support public health use of our and when I talk about public health, it's a lot of clinicians, a lot of pharmacists, a lot of disease investigators, and also researchers. The story really begins with the epidemiologist our handbook and this actually predates the existence of our of our organization. It's a book down for those of you who know the package. It's our markdown. It's free as open source and we launched it after about nine months of work back in 2021, about 120 contributors contributed to it in their spare time. And it has about 50 chapters and it has been very widely received and used in the in the in our least in our space. So I just want to touch a little bit on the at the our handbook. As I mentioned the day to day tasks are really it's a task oriented book. So of course it begins with the basics of our of which there are many resources to learn, but we wanted to have this the philosophy of this book was if this is all you have and you're going offline in the setting. This will be enough to get by. And so talking about our projects and importing data leads into the core topics of these day to day work which is cleaning the data, which is many familiar topics here. I want to highlight for a moment deduplication, which is a really common task among applied epidemiologists and so we've shared some techniques and that we found useful for deduplication. Yeah, yes, it's very common. And then we get into sort of the more epidemiological topics like modeling time series outbreak detection, and the more statistical chapters. But data visualization is also really a common task and so we do well on age pyramids and making tables, how to make transmission chains, these kinds of things. You know it's it's it's a book that's been around for almost two years now and it really needs some updating, but I'll talk about that soon. And of course the bread and butter of an epi's life is producing reports. And so talking about our markdown and making that more more easy and different kinds of dashboards. I wanted to just share this last fun video about the power of our to make workflows more simple. What about the data visualization? I think that is the main thing for me because we do a lot of analysis for me. I want to generate any map. Let me just pause it so that it offers to QGIS and come back here. So for me, I think that path that's a visualization is the main thing for you to get your code right with the plot. Amazing. So how GG plot and making maps really reduces the amount of tools that an epi has to rely on. Many, almost 100 people reached out to help translate the FDR handbook into many different languages. And so we're very happy to have these different languages up and available with more coming soon. One story I want to share finally about the handbook is recently we were excited to learn that the Ugandan Ministry of Health in their Ebola response was relying quite heavily on the FDR handbook to do their analytics for the recent Ebola response. And so this actually led to quite a strong partnership. As you can see in the timeline there, we ultimately supported their team with our training and also with our our help desk. But I think this quote from Daniel goes to say, you know, it's so nice that our art is often scattered all across the internet. And having it in one place was very helpful for their team. Some of the challenges we've encountered and some of the successes. I think the fact that we were an independent entity making this made it a lot more agile and allowed us to produce it very quickly. There were a lot of editorial choices we had to make and still have to make as we consider revisions. How much, how little to show how many solutions you all know you can do everything a dozen ways and are. And now actually maintenance and version control is becoming quite a challenge, considering that we have so many translations and we're looking to transition to using Quarto. And I think that if any of you are thinking about book downs or similar sort of texts that you're going to produce these are some of the challenges to be aware of. And then it was very quickly apparent that the handbook was never meant to be a learn our tool. It's a code sample book, right. It's a reference book and so there are many people who could benefit from a synchronous course. And so we designed was a 40 hour intro course that was really catered towards public health use cases. The philosophy we followed was that we are frontline practitioners and so we're bringing that sort of experience into the instruction itself. We really wanted people to equip people to use are on their local computer. So this affects how much time we put into the preparation of troubleshooting installations and the decision to not use a cloud environment. We wanted it to be synchronous because so much of the troubleshooting we thought we feel needs to be one on one. If we're dealing and trying to educate people for whom this is quite a scary proposition learning how to code. It's very intentional with our vocabulary. Things as simple as not using the word string right when that is someone might associate that with their shoelace when using a word or characters, at least at the beginning to introduce the topic more more in a more friendly way, building that confidence to tinker and then finishing with support after the class ends and that's where our help desk comes in. So I think that some of the curriculum choices are of interest to this crowd perhaps how early do you introduce our markdown. Some people have done this right at the beginning we elected to teach in our scripts first and then introduce that kind of workflow towards the end of the class, how much base are versus tidy verse etc. Of course we're using our projects. How do you teach people to install packages in a simple way. And so there's many sort of choices here that I can elaborate on in the questions and answers at the end if people desire, but just to say, we are losing using learn our and grade this and they're they're wonderful tools that we're very grateful for. We've been running this almost nonstop for the last year and a half. And so over 700 people have benefited from this course at, I think it's 150 agencies now around the world and again many local practitioners in the US outside the US in pretty much every single continent. And you can just see here this is one week of our training schedule, which is essentially around the clock. And the nice thing is that we have instructors all around the world and so we can actually benefit from that by having taking advantage of the evenings in one time zone being the work days in the next time zone. And I want to take a moment to thank all of our instructors who do this and who are often teaching late at night or early in the morning because they really believe in this in this objective of allowing it easy making it easier to learn our among their colleagues around the world. So thank you to them and all those who support them. This slide I have to just highlight this decision we made which was to do preparation calls with every participant before they start the course to help troubleshoot their computer setup. And this has been painstaking but absolutely critical to resolving the PN issues, one drive, syncing the package installation, having old versions. It's a bit of a nightmare but it's worth it if you want to start the class smoothly and help them understand how to do that in the future when they will undoubtedly encounter these kinds of issues again. And what this has led to is some really fun experiences. So this is one example where we ran a training for Ministry of Health and the National Institute of Statistics in Cambodia. And it was kind of a hybrid setup where they were gathered in person and we had our instructors around the world joining and this was with simultaneous translation into Khmer. And so that was one challenge we had to overcome but another class recently was with consecutive translation which again sort of impacts the curriculum and the pace of the class. And this was into Ukrainian where we several cohorts of local epidemiologists across Ukraine. Another class in Central Asia where we did consecutive translation again. And beyond the translation we also have instructors who can teach in French and in Spanish and we're building on our Portuguese team. And so we can have the materials in those languages as well and I think that that's really important to being able to reach a wide audience with with this. We've seen improvement among the among the participants in the course, although we hope to do more sort of structured evaluation of the impact in the years to come, it's just been such a rapid mobilization of our whole team that sort of we want to do more objective assessment of the our code that is coming out of the class. And I think, moving on to the next pieces here for those who can access our courses we wanted to have a free option so we built learn our tutorials that are available online that are very similar to our core, our synchronous courses and this were built with doctors without borders and we have a few are packages, but it's not really our focus. These packages are built to provide helper functions for common at the tasks and sort of work around gaps in other packages, and and also to provide templates of situation reports which are common. They produced by epidemiologists one example right age pyramids trying to produce this from scratch and gg plot can can be a little dangerous. If you're if you're if you make a mistake. Just one example. Getting to the more advanced offerings we want to, I'd say 95% of the courses we run now are intro courses but we want to build out the space for people to grow into advanced courses. These are some of the courses we have or are developing. I think there's a lot of attention on dashboards, but what we usually try and emphasize to prospective people wanting to learn dashboards is do you do you really need to learn shining for this for your purpose. And if so we can try and get you there but there are many other ways to communicate your information that don't involve the coding involved in learning shine. Practice I think is very important we hear from our participants where can I go to get more practice. And so what we've engaged in is a is a worldwide effort to partner with field epidemiology training programs who have a robust curriculum that often involves case studies using the info or SPSS or Excel and help them transition those two are and then offer them publicly to try and break down those those walls that are keeping those case studies from being used by anybody. And so we have a case study repository it's growing, but the idea here is that that that is built is built out more. And then the last piece I want to talk about in this pyramid is arguably the most important and that is mobilizing this public health community to provide sustained support. And there's two pieces here the community forum and a help desk. And so I'll start with the forum because there are many forms out there. But what we the idea here was one that is embracing the public health context that is underlying so many of these questions. And it's really friendly to begin or so many of the people we engage with are terrified of posting anything on stack exchange myself included. But the idea that this is this place where you can make mistakes where people will will help you in a friendly way arrive at how to make a reproducible example and sort of know the background context of your public health work and help you arrive at the best conclusion. And so you can see a few of the kinds of questions we get in this forum about transmission chains about simple recoding of values and duplicating again a very common topic, etc. And I think this is going to be accompanied by a concerted effort to help people make that first post and learn about reflexes because it will just help everyone the answers and the questioners so incorporating this into our courses, holding webinars and support sessions and helping people craft that first reflex it kind of breaks the ice and allows them to to feel comfortable posting. And what that grows into is an our help desk. And so this is we really wanted to be able to offer these one on one calls out out to the public. And so we have this help desk we have technicians who are staffing it in different languages. And and and at this point we're mostly offering it to agencies but we do hope to open it generally to the public soon. Here's one example where we worked with again the Uganda Ministry of Health over several months to help them analyze their their outbreak data produce communications and reports and really answer the task force that was driving the response. And so here you have, you know, the quote from our technician and from from Daniel who is largely leading the data side of this response. Another example we was my colleague, Dr. Peter at a movie and the Gambia they recently had an outbreak of kidney injuries, and he reached out and said, can, can I get some assistance with this so we were able to support him in just a matter of hours on that, on that response. And we've also worked with doctors without borders on some of their surveys and. And other sort of urgent tasks, right. And so where does this go. We have dozens of technicians who are available 24 seven but we don't yet have the capacity to offer calls the within an instantaneously. And so we want to if the demand grows to be able to offer that we want to have rapid determination scholarships and be able to offer more specialized skills. And so I think this I hope that you are excited about this as well, and that you can see it see it growing beyond our organization also addresses methods. And so this is the second book that we're looking to produce which is the applied epidemiology manual. And this was a request received by a lot of our partners, and we now have about 80 authors and reviewers working on this book as sort of a one stop shop, more on the method side so it's we're going to be tool agnostic but we'll link very closely to the our handbook and those case studies that I mentioned. And I think I want to close with some some remarks here and maybe questions for you, and we can we can talk, but I, the first one is, how do we ensure that our development growth is also led by the the practitioners and not only early adopters who are more perhaps tech savvy. You know I think that that voice needs to be part of it and one way that manifests is linking the developers with the end users and making the feedback channels between those those groups more fluid. So, for example, we partner with a number of programs and projects and universities that are building packages for epidemiology. And, and we're trying to have those ground level users maybe it's your forecasting where an academic is going, and you have an epidemiologist at a local health department, and they are trying to implement that our code package and they are not sure about the methods they are making a mistake. And they have concerns to pass back to the developer on how to make the package more user friendly or how to actually address the needs that they are experiencing in their jurisdiction. So I think having that flow be more fluid and I hope that spaces like the Community Forum will be a place for that. And I would ask you all when what other fields, whether they're sub fields within medicine or adjacent fields like public health will our adoption and the acceleration of our adoption result in tremendous real world impact. And, you know, thinking about the amount of hours and brain space that can be saved and freed up by moving those repetitive manual inefficient and error prone workflows into reproducible scripted workflows. What are the other areas like that that could perhaps benefit from the work that we're doing and that many others are doing near us. I think I touched upon this but how do we create channels between developers and front end users and I think that many groups are doing this and they're starting to do it well but there's, especially within epidemiology there's room to grow. And this gets to also how do we create beginner friendly question and answer spaces how do we bring down the walls for someone to make a post and stack exchange or in a similar space. And then I think a more perhaps provocative question is in a field where there are many tools and there will be many tools in the years to come. What is our role there. You know, we have people for whom Excel is not going away. It is a workhorse in the space that that I that we work in tools that are more point and click like at the info or SPSS, and then we have other languages coming in, whether it's Julia or use cases for Python or something like this so what is ours role there as a as a language that I think has a very robust community behind it. The public health and Epi and medicine community of developers of support is very strong and that's why when people come to us and they say how about what about Python, we say well, like let's look at our and let's look at the community and the resources and all of the innovation and rapid fire responsiveness of that community to the needs of public health. And that's, I think, a role. So, I guess I'll close with with those thoughts. There's some of our contact information, and I guess my hope is that, you know, you all work in spaces and medicine in agencies and institutions and universities, and there may be other years like we've done for ground ground level public health, but you can hopefully learn from some of the challenges we face in trying to increase our adoption, and I hope that you keep in touch as we continue to to do this together. So I'll pass it back over to Ray. And maybe we can take some questions. Very good. So I've been taking notes and watching the chat as, as you've been talking. Lots of interesting comments and dynamic things going on. The ones to comment really struck me was that a lot of the fear with learning are comes from people who start out the wrong way. So do you have any, any thoughts on that on when we talked about the tools that are made available for setting up and making sure that people can get started, but just in general, how do you prevent that. How do you prevent the initial failures. Yeah, I think this is a critical question. Having a friendly relatable face and voice at your side is so important. And that's why we've invested so much of our resources in one on one interactions, especially for people for whom have never even conceived of themselves as growing into a coater before. And being on hand when they hit those mistakes, those error messages, and otherwise they would divert back to their, their, their well practice routine with Excel. So that during their first project, they have someone that they can call. And within a few minutes, they are online with with someone who knows public health and who knows are and who knows how to speak in a, in a beginner friendly manner using vocabulary that is not intimidating. Those are the critical components and it's it's taken a lot of legwork for us to to get the staffing, and that's why we leverage this part time staffing model it's people working after day job, but we can distribute that load across the 100 plus people that we have on our team, and they're distributing few hours a month, but that has allowed us to, to tie all those people together and offer a service like that. So, yeah, I think that's, that's, that's pretty critical. Very good. It's interesting to hear your your thought. Another comment. So, Wesley Wilson dumped a big pile of resources for starters on particular that the one that I know the most about is data carpentry. How do you see your work fitting together with them I don't think I saw them called out in your slides. So, what are your plans for leveraging the other great teaching resources out there. Carpentries and many others the FBR handbook is full of links out to those kinds of resources and that's where we started was it's it was, we wanted to provide public health and FB task oriented code snippets, and then where best link out to the existing resources that teach them that show show them and so the FBR handbook is full of links to carpentries and others. Cool. Many people singing your praises in the chat. There's a few comments near and dear to my heart about tying in red cap in the future. There's comments about other useful communities that are more welcoming and stressing the importance of what you're doing. So community dot our studio calm. Someone commented that Luke Morris commented it it's it's less knee jerky than stack exchange. Let's see what else. Other folks want to throw in questions into the chat. I'd love to pass them along. And Peter you've been throwing in suggestions as we go along is there anything you want to circle back to. I wanted to ask Neil what he thought of, you know, Mina sitting Kyah Rundell who's basically posted a thesis of let them eat cake to get beginners to get from data to a visible and usually visualization output quickly so they build their confidence. And I don't know if you've seen that blog post but she's a big fan of trying to get people some early wins early successes. I think that's that's a very important methodological or pedagogical tool. And I think that our course does that to some extent we've incorporated it to some extent we we we use a very relatable example in our first session about, you know, this is teaching basic R syntax and what what can you get out of writing a code instead of doing Excel or something and it's about calculating the amount of COVID-19 tests, you might need to order this month based on a few different sites and the expected amount and it's been turning that into a reproducible workflow. And it's so it's not a visualization. Because I think it's just gets a little tough. The balance of are you going to start teaching how to make the visualization versus versus just giving them a code and saying run it. I think everyone comes into our having seen the visualizations, but once they start to say, Oh, I can write code and produce something that would have taken me otherwise half an hour or an hour, then you see the light bulb go off. And so I think that's maybe I haven't read that article but the essence of it is having the light bulb go off. And every one of our instructors can describe when that happens in a course, you know, and it may be the first module and maybe when they see the R markdown happen and they say, Oh, my goodness, I will save so much time with this or it may be the visualization that they see, or it may be the data cleaning. And that's why we start with data cleaning is because so much of our the participants that we enroll, they spend most of their time doing that. So when they see that Oh, I can press a button and have my code run and have my data clean the matter of seconds. That's a light bulb moment for them. Yeah. Yeah, I think Mina's post is essentially having run into instructors who say, Well, you have to, you have to earn that visualization by doing your data cleaning first and doing, you know, walk both ways in the snow. And she said, No, no, no, let them eat cake first, get them excited. And then, then they'll stick around. They'll keep going. And I'm massively summarizing a long post there but another thought on if you want to integrate Julia. I don't know if you know Karen deep sing. He was one of our former keynotes. He's essentially creating the tidy verse in Julia. I think it is an ongoing project, but people are pretty excited about it so that may turn out to be an entry point for folks. Wonderful. That's great. I think applied at the at the end of the day. We want our, our sort of resources to be available in all kinds of tools right so we have an at the R handbook, but we're starting at the Julia handbook right we're going to have similar resources for Excel, because so many people are still going to use Excel. And trying not to duplicate what already exists, but I think working with colleagues like that to to have to have epi and ground level public health oriented Julia resources is a great way. Super cool. As many people in the chat asking how can I help. So, how do we sign up to help you with your mission. Yeah, I would say there's a link on the screen if it's still sharing where you can apply to join our part time instructor pool. And that's sort of our gateway. So if you're interested in being in our help desk. And that's sort of, that's our first, our first line of applications is that is that link right there. Find epi.org slash join. Otherwise you can email us if you think you're you want to offer your skills in another in a different way, and we can happy to chat with you. There is one thing I wanted to share. There's a question here, Jacqueline Janis. What platform are we using for our courses. We don't use positive cloud. And so learners are using their local desktop versions of our studio but I think what we've found really powerful is actually Google meets, because it's one of the only platforms that I know where you can have multiple people sharing their screen at the same time. You flip that around, and you ask all the participants to share their screens. What you end up with is a way that instructors can watch the students as they are live coding working on the exercises, and instantly see who who's having trouble who's making an error, and then invite them into a one on one breakout room, and have that private with them about how to fix it. So that's what we've found to be an incredible virtual teaching tool for Cody just improve. You just improved my course evaluations ever going forward. Thank you. There was a couple people or someone asked about chat GPT. And so I want to open it up to a broader. I'm a big fan of Bard is a Google Bard. What are you thinking about about the rise of chat GPT in terms of the teaching tool, and also for the all parts of your mission. Yeah, I am expecting this question but I, I'll start by saying that I haven't had enough time to think about it, because our, our schedule and our growth has just been so consuming we're very small team, and we're working very hard to just kind of do what we're doing. But I think to answer your question. I'm excited. I'm also anxious. I suspect based on what I know of how technological advances are ultimately slowly distributed around the world. I anticipate we will still need to be teaching coding for quite some time. And, and, and, and, and beyond that we're teaching is not just programming. Right, we're teaching a mindset, we're teaching ways to think about tight data about data collection. And, and, and how to think in a workflow and organize workflow and in a reproducible way and no matter what the AI world comes out with those kinds of skills are really important for epidemiologists to know. So, other than that, I'd say we're looking at ways to incorporate into the curriculum, but at our core is going to be teaching people how to write the code as if they don't have that kind of support, and if they do, that's all the better. That's Danny. Are there any additional questions floating around the real world or Peter have you seen things go by that I missed. I think a lot of agreement on stack overflow being scary and harsh and looking for alternatives. Can folks who are savvy jump in and help out and answering questions on applied epi. I think it's an open forum. The account is free of course to to read it's anybody can go in and read those questions at community dot applied epi.org. And if you want to answer, please do just make an account and post or more, more than welcome. One of the website related dedicated to our code. It's definitely the most active site part of the site, but there are other parts that are more tool agnostic or more focused on methods study design surveillance methodology. These kinds of things or math modeling or upcoming events and things like that, but the our code section is definitely the most active. And there's people posting in there almost every day about working in this country and I'm trying to do this study or I'm working in this country and I have this outbreak and I'm trying to make this plot or that plot this kind of For a lot of my world is spent working with people who are trained in SAS originally and are now moving to our and summer moving to Julia. Did your platform offered tailored advice for people who are walking in the door with different skill sets. Glad you asked that so in the epi or handbook one of the first was called transition to arm and it has tailored advice for people who are coming if they have only used Excel in the past, and they need to think about transitioning from that kind of mentality into writing code for the first time. And then there's specific parts of that chapter for SAS users and for state of users and they include sort of mini mini dictionaries, if you will, and sort of how are, how are differs from the tool they're coming from. And we do see this in our classes. Some of the classes are open to anybody in the public who wants to to register a seat. And in that case you have people who have already quite substantial programming backgrounds coming in, and we have advanced materials for them if they happen to move faster in the class. And then others who are coming in and never having programmed in their life. And so we take a little bit of a different approach with them but it just depends on the person some people pick it up extremely quickly. Yeah, in my teaching experience there's always one person with a degree in computer science, and one person who has mastered both point and click with the mouse. It's, it's super challenging. I'm looking to see what else is sitting here. Peter, do you have any other, any other big picture questions to ask. You know, I ran into this in teaching the scary error messages and ended up in a book actually creating a whole chapter on what to do when you see an error message. It, what it is the best approach to get people past that just like, wow, oh, ouch. Yeah, error message stop help help. I'll start with the FBI handbook one of those chapters towards the end is common errors. And so we just, this is meant to be a brain dump where we post common errors and how we've solved them and more on the more more is so many epidemiologists work on government computers. Right. And, and there's a lot of just walls and ways that are marked out can fail. When you're trying to produce things. And so a lot of those error messages are related to that. But even the in the course we, we, you know, we encourage there's parts of the course where we have people read the documentation and the package and then try to apply it without us coaching them at all. And then we circle back to them and talk about what was difficult to read in the documentation. And share ones that are that are very difficult to read and ones that are very much easier to read and we say if you were designing a package, how would you write it. And so I think, but other than that practice and the help desk, I keep coming back to this one on one support but for people who are on the fence about this whole endeavor. And any sudden movement in the wrong direction could prevent them from continuing their our journey, having someone who's readily available and with a friendly demeanor, ready to help them solve that error message explain it and so that next time they can understand that cryptic red text in their studio console. It makes a difference. Yeah. I often find teaching people how to read documentation because it's such a very particular structure. It can be valuable and I don't, you know, in a perfect world I make the documentation a lot more friendly. But how do you approach that from for a beginner who needs to, you know, get over that hump of how do I read the documentation. I will say that the priority for us for the early days is not to turn off the person and so we don't show them the scariest documentation. Because that's a very, it's a it's a it's an opportunity for them to say this is not for me. This doesn't look like something I will ever know how to read. And that we really get into into the documentation when they're very comfortable with the terms of arguments and default settings. They've realized that, you know, writing true you don't put the quotation marks around it and all these kinds. Yeah. But I think the priority at the beginning for us is not not scaring people away. Yeah. I have a. So when I was in a great, my, my weird twist on this is I try and teach package development as early as possible. So to teach the idea of how do you document a data set an arm, you know how do you set it aside save it, build a documentation for that. And set aside how to build functions until people are much further along, but the idea of encapsulating within a package and our markdown document along with the data set. You know, we've got a fully reproducible paper. Have you put any thoughts into that like where does vignette building. Do you think about the transition from writing an arm markdown. You know paper to writing a vignette. Is that in your radar. I would say not in our curriculum yet. But I think that is as we seek to move beyond the intro scope that will be in the agenda. We, we, we tease a little bit with writing functions, sort of providing code and showing them, you could take this and put it into a function. It's people excited but it's sort of an optional part of our early curriculum, because I think it showing people that they can be creators, right is is actually a bit of a, it's a big shift for people when they realize that most of our was built by people like me, or slightly more experienced than me, and that actually warms people to the software I've found. Yep, I agree. I have nothing else on my list. Are there more questions out there. What are the top two challenges facing applied heavy. So I think the, the, we've grown incredibly fast for a small organization. The demand we're facing and the operation that just the extent of the operations were running. Having a small team that's mostly part of time has been a challenge. Right. I wish that we had a big. Under, we don't really have a big funder or a big grants to sit back and relax. And so the, the pace of work has been very, very demanding and the fact that all most of us are have other jobs is is distracting. Right. So I think that's one big challenge. The other issue is, it gets to that might slide on the NPR handbook and version control and making robust. Controlled materials now are the extent of our materials is getting very expansive and so making sure that data sets are aligned approach, you know, approaches are top are aligning across classes and that the language translation process version control is a bit of a nightmare for us. And so we're trying to reach out for assistance with with that. But we've tried a few, a few approaches, but that's difficult. Good question. Thank you. Other questions from the audience. Neil, I mean, this is kind of spitballing whole idea, but would it make sense for applied epi to get funding from something like the Ford Foundation or the Gates Foundation. I mean, it feels like your goals are largely aligned. I think there's a lot of entities that have shared our goals. Yeah. And so I think we'd be very open to something like that. I think we're, I think there's a lot of impact on global health to be had from accelerating this transition and making sure that it's led from the ground level, because if it's not it's less people will actually make the transition. And I think that there's a role for us to play there. I'm flipping through the chat. I, I am so pleased to have heard you talk. I wish I had prepared more questions. A rare opportunity to get your time. Yeah. Neil, I'm curious, what's, you know, in folks in Liberia, Nigeria, what are the most common data collection tools they actually practically use and I assume they're probably on their phones. I think in many countries around the world, there's so many, that's a variety of data collection tools. And I think that is one of the main challenges that faces public health is kind of consolidation around some of those tools. And each one has its own strengths and weaknesses for different kinds of settings. We see a lot of people using using the Kobo toolbox. And so a lot of our educational materials reference that and kind of help people use are around that. You know, a lot of agencies use the HIS to as part of their data pipeline. And so we do try and craft our materials around those common platforms, red cap was mentioned. And in the US, there's so many different platforms and, and, and, you know, sort of data data storage mechanisms that those are often kind of we have to do. Just sort of spoke solutions for each person. But what we come back to is, you know, our sort of default assumption is, you know, you're working on our locally on your computer, and more and more people are using cloud based. But that's not something that we dive, we dive into very much. You answered the question of, if someone in general comes in and says, how can I help. Let me throw it out in the other direction. There's package developers here. What do you need? What do you want us to build? Great question. Let's, let's ask a lot of people on the community forum. And then let's talk because I've been away from the ground for a year or two now doing mostly teaching. But, but I think that actually getting to Peter's question, the, the biggest pain point is the data collection coming from the ground and, and standardizing it and having it flow into analytical workflows. And so a lot of groups are working on ways to standardize what we call line lists, right, the sort of case, the list of cases and their attributes. And everybody from the humanitarian sector to the epi sector and everyone's trying to figure out what the best way forward with that is whether there's some kind of tagging hexel or standard package that assigns certain values. I think that's a space that's very fluid right now. But that is a very worthwhile place to invest time and thinking in as long as it's really well grounded in the end user. And I know a couple of years ago, Hadley Wickham was very interested in developing a data collection tool for our. And I think he's still interested in, you know, could there be a consortium of sorts of doctors without borders and, you know, interested parties to have a free downloadable open source data collection tool that would nicely export to our. Is there space for that, or is that just really hard to get everyone at the same table, virtually. Perhaps but I think that I think some good options already exist. We talked about Kobo. That's used extensively. And so I would. Yeah, there's always this balance of, is there a room for improvement or can we improve what's already there. A new thing or should we just improve what already exists. So I would be careful about that. That's just coming from my discipline, my space of public health at the sort of emergencies work. Other fields, the breadth that Hadley deals with, I'm sure there might be other ideas that most common question we get is, does our due data collection to usually say, well, there's a lot of ways you can feed data into our. And that's actually one of the positive aspects about it is the versatility. If you're not seeing in chat the the are the red cap package developers in the house are coughing and wiggling in their chairs, me included. Yeah, lots of users use red. We've worked with a number of projects and different kinds of red capped and I think there's great hard documentation there, linking red cap to our. Yeah, but we're trying. There's a, there are restrictions on, you know, it's not fully open source, but there's a lot of options available. And a lot of people use red cap. So it's very important that that kind of support is readily available. So I'm not seeing any more questions Peter you got anything else or we can scoot to the next, the next topic. I'm sharing some cobo resources because it's new to me but it looks cool. Yeah, I think we're great. Thank you so much. Thank you so much, Neil. An absolute pleasure and again, thank you for fighting the fight that you've been, you've been going after the tools that you provided my students are so grateful they've been grateful. For years you you you provide tools that make my my job easier so thank you. Glad to hear it.