 Hello everybody. So the next presentation will be by Ted Lederis talking about teaching clinicians data analytics with R. Thanks Beth and I just want to thank everyone. This has been an awesome morning and it's really been cool to see all of the communities of practice that everyone's kind of established and Cass I just want to just shout out that like I think that work that your talk was amazing. So today I'm going to be talking about our experience teaching data analytics with R. Let's see. Come on. Okay. So like we've been teaching this data analytics course for about six years. So this is just kind of what I'm going to cover today. So we'll do kind of a brief introduction to the course introduction, talking about the course and talking about the outcomes. So who am I? So I'm an assistant professor at Oregon Health and Science University. I consider myself, well I'm a bioinformatician but I also consider myself a professional collaborator. I just love working with other people enabling them to kind of go farther with analysis in R. Just a quick plug. So I'm an RStudio certified instructor. So these are kind of two kind of resources that are kind of freely available for everyone. So I did the course and all of the videos and materials available. It's called Ready for R and then there is an interactive version learning the tidy verse lessons called R boot camp. I just want not bragging but I do like this review that a clinician told me that I am a very patient man. So really the point of the course is how do you deliver actionable analytics in healthcare? So we want to deliver this kind of experience to our students. So you know part of it it's you know the part of the course and this is what I really do want to emphasize is that data science is kind of only one part of it. The other and that's kind of what I focus on at OHSU during the course but Kaiser Permanente insight and Brian's the course side also talks a lot about the organizational aspects and strategy and in the end we force our students to actually apply both of these by having a final presentation and I'll talk more about what that final presentation looks like in a second. So I want to just talk about clinicians as learners and this is no offense to clinicians. This has kind of been built up over a long time of interacting with clinicians in this course. So just saying like Mary is a clinician who wants to understand how analytics can be delivered in her healthcare organization. What are her special and thinking about what are her special needs? She has very little time she likes to learn on her own and she has a hard time asking for help and at the same time is hard on herself. So how do we meet these needs? So in terms of no time we've tried to structure the assignments to gradually increase in difficulty and we think very much like in terms of kind of just in time instruction. So what do you need to do to accomplish you need to know to accomplish a task? In terms of everything we really tried have worked on this kind of self-learning model and this kind of started kind of evolving in the beginning. You know first we kind of started having the students work in R Markdown but you know as kind of things progressed and we learned more about kind of the R and R studio ecosystem we started kind of including more and more kind of aspects kind of support this. So we started using R studio projects as a way to kind of package assignments and now finally in the last two years we've been working with R studio cloud and which is basically kind of this online platform. You can basically point a student to it and they have like a full instance of R with all of the assignments and everything in the workspace. So this has been a really kind of helpful tool and we're going to continue to use it. Also so like you know like when they have a hard time asking for help you know we do do these things like you know Cass mentioned kind of having a buddy we do do that we make we make the students team up. I try to be available as much as possible like in terms of Slack for quick questions having office hours available and now I've kind of been working on like with making making myself available via scheduled appointments so just kind of making giving support to the students when they really need it. So let's talk a little bit about what like the an overlying like problem we are trying to solve in the course. So the overall problem is we're trying to predict 30-day readmissions and this is within a simulated hospital patient cohort. And we do this by implementing a metric called LACE. So LACE is short for length of stay, acuity of admission, any comorbidities the patient has and the number of ER admissions. So this is it's nice because there's kind of a very focused task students will have to pull the data out of our simulated data warehouse. But again it's not just about implementing the metric they have to communicate how effective it is in the patient population. So talking a little bit about the data so the data is a simulated data warehouse. So these are some of the tables that are in there so you can see like you know there's a patient table there's a hospital encounter table so these are all of the encounters that patients have within the hospital and then also there's a diagnosis. Yeah and we know that this is very simplified but we've kind of kind of honed this down to kind of get to the essentials in terms of learning. And it's also so it's structured as a four-month extract of patients and it's based on real clinical data. So you know I've been trying to figure out exactly how to incorporate Brian's piece on here and this although this is just one slide I just want to emphasize that this is like Brian does kind of an amazing job at kind of talking about talking about kind of the organizational challenges of bringing analytics to like a healthcare organization. I mean there's all sorts of real real real hard hard-earned lessons that he gives to students in terms of how do you kind of get your project visibility how do you get sponsorship and how do you ask for things so these are all very kind of very very kind of social and organizational parts but you know we feel that like this is like an essential part of teaching data analytics it's not just about teaching R. So I just wanted to give you kind of an idea of what the assignments look like so like and I like we said like the assignments are very kind of they're very kind of gradual in terms of things. We start in terms with like doing exploratory data analysis so this is a great visualization tool by Nick Tierney by an R OpenSci and it's called VisVat and what it is is it's like kind of this quick kind of dashboard look at the data so you can understand whether variables what whether there's kind of missing valuables and what kinds of variables and values there are in the data. Okay why did my slide not sorry I'm having issues with I'm having issues with just okay so let's talk about so this is kind of the introductory SQL assignment and again our our clinicians are not necessarily like you know completely conversed in R and SQL so like really kind of having these kinds of great graduated kind of assignments so this first assignment is about just kind of selecting columns and data and looking at the actual data in the tables and amazingly enough like you know as like the students kind of go on and they learn they start being able to do lots more kind of sophisticated queries so here they're calculating kind of what are the number of emergency room visits from the clinical data warehouse and you can see this is not simple SQL codes so they are starting to really think about ways of summarizing and aggregating the data so we feel that this is like a really important thing like we could have started started out with like you know having all of these different scores calculated but I mean it's I think it is important for them to have ownership of the whole process so once they have those kinds of scores calculated they can build predictive models we teach them basically logistic regression and we talk about lots of concepts such as ROC and kind of and this part of this again like thinking about how to work kind of the organizational part like how do you work in organizational values into kind of deciding how a how a predictive model is applied and you know I'm I'm I was excited about the tree he trains tree heater talk because I always love to see good teaching tools this is kind of the party output but you know they can build they can build you basically do machine learning with decision trees and we have and I'm excited I was so excited for the tidy models workshop because like I'm definitely going to be work integrating more tidy models into the into the the into the lessons okay so I've been kind of stepping around this final presentation but like the thought is that they present they want to they're doing a presentation to an executive team so this kind of lace calculating this lace score is like a pilot project and part of it so this is like I just love working with Brian because you know he really kind of breaks it down what is kind of an effective presentation and it's it's all kind of building towards having a call or ask to action so like you know we want you to invest war in this in our program or we want to implement something so you know he's very good at talking about things like so what is the impact of not changing so if we just kind of have the status quo what what's going to happen and then you know really we spend a lot of time talking about like how to present results and this has been a harder lesson we've had students just basically show our model output and that has not been successful so we spend a lot of time thinking about data presentation and data storytelling so let's just want to talk about in terms of outcomes like you know some of the presentations and I think this is one of the best outcomes the students come out with different ways of looking at the data so I don't know if Kevin is online but just shout outs to Kevin and Mina so this is their way of kind of presenting the data and you can see that they they've really kind of thought through like you know in terms of highlighting you know different kind of aspects of the data and the neatest thing really is that you know the students think of different ways of presenting the data I mean here Arielle Rose and Alfonso decided to talk about kind of gender bias in the data so shout outs to Pierrette Lowe I know she's on here just so this was kind of how they kind of visualized the data and kind of talked through it Dan Slater, Laura Hickerson so they decided to kind of show you know the distribution of scores across and like highlight the highest risk group so like you know that was kind of the focus of their talk was like kind of targeting these kinds of high-risk individuals and this is so Justine, Dan and Xiao this is their presentation and they're really thinking about kind of in terms of like you know CMS penalties and of course you know implementing this has a cost but you know thinking about what the balance is and this this is this was I'm not saying all of any of these presentations are better than the others but I thought this was really effective so Megan, Wei Chen and Colleen thought about you know what are the cost savings in terms of implementing the project so I think you know this is you know it's one of these things I feel like the students have been really empowered to think about the data and they get very creative and they think about kind of the data and in different ways and it's always fun to see that so I want to just talk about I'll just kind of wrap things up talking about some of the student testimonies about the course so Kristen Stevens she said basically the course has made her a much more patient and effective collaborator so I think that's been one of the the real strengths about the course is like really kind of that camaraderie the students get with working with each other another theme students you know the students really like kind of the diverse set of learners that come into the course the other set of learners that I haven't talked about are kind of our bioinformatics students we make them take this course but it's been a very useful bridge to kind of getting them to talk to each other and I think there's been a lot of useful conversations that's come out of this so I get to embarrass Perette again but so you know it's she really feels like the course is you know it's soup to nuts I mean it really kind of covers a lot and it's just very useful to anyone who's interested in working with data in a healthcare setting and finally so Frank Logano so he's an MD clinician and he's just really talking about that this like he really thought that the course was helpful and like you know more informatics programs should have like a course like this for clinicians and other clinical informatics students not to toot my own horn but we did win an award the word itself is not that important but the fact is that we had multiple students nominate us so that was really amazing the students really loved like you know our availability like you know kind of the way the stretch of projects were planned and then you know they really felt like it really resulted in essential research skills in not in not only bioinformatics computer science but generally across biology and medicine so to me that's one of the best outcomes I could have ever hoped for so just wrapping things up so the course combines like practical and organizational skills and through that kind of final project the students are forced to combine them and the other side is like you know it's really kind of short with these real-life lessons I didn't talk too much about this but Brian's team talks about like you know the Kaiser Permanente like implementation of LACE and there's a lot of talk about kind of evaluating it for effectiveness and so again like applying the knowledge is really kind of important and I feel like it kind of cements everything that's come before it so just want to thank both the OHSU and Kaiser teams like this has been like kind of a huge kind of undertaking it's like you know not only enter departmental but it's enter like you know enter institutional so it's just been kind of testament to everybody's really hard work in making this this go so the slides are here like I hopefully people have seen the little link as we've gone on here's my information this is the github repository so if you're interested in looking at the our notebooks and everything like I'm happy you can take a look at it I need to clean it up a little bit so I didn't have enough time before this class of this talk and I just you know I I just want to kind of this is our classroom last year so I'm happy to take any questions there um the top question is that talking about the buddy coding paired programming and is there a virtual platform that facilitates paired screen sharing so like we you know I know that our our studio it's um had implemented some sort of way to do kind of uh like having two multiple people work on code at the same time we haven't looked into that so I like we really just kind of talk about pair programming strategies that work for people like for example like you know one person is kind of um the coder and the other person is kind of the driver and so kind of talking about these kinds of strategies of working together it's I will say like moving this course completely online has been a real challenge because the course itself has been it's been like some online learning and then there is this in person week and I feel like having that in person week like literally you know four and a half days of students kind of being excess and where the instructors are accessible and they can ask questions it's really great as a professional collaborator educator mentor you know how do you guard against giving tree-style burnout yeah this is a good question and I don't really have a good answer for it um I have been working with more kind of setting professional boundaries um so there's a great book um I can't remember her name but it's called on f your boundaries um and it's really about thinking about you know what is kind what are what are the things you want to do versus what are things people things asking of you and um just one have you spend any time thinking about teaching git with a GUI for a truly distributed analysis yes um so I've been having conversations with lots of people about how to best teach git and we're still working on that um you know it's it's I feel like you know the the course is really packed right now so it's trying to at least kind of show them what it's for that's kind of the level we can kind of cover it okay I think that we need to wrap things up and we'll head to the next session okay thanks everyone this was super fun