 So, my name is Garrick Rollman and I'm the Director of Learning at RStudio, RStudio Academy as it were. And I want to speak with you today because I've spent the past decade, it's officially been 10 years, trying to build a better data science training workshop. And what I hope to convince you of during this talk is that you should not have your colleagues learn to do data science or programming through a workshop, nor really through an online course. And the reason why deals with the answer to this question. How do you train a data scientist? I first started asking this question 10 years ago, like I said, when I joined RStudio, when you are a company that builds computer software, it's essential that your potential users can learn how to use your software. And that was the task I took on to teach an evangelist size free and open source software with RStudio ID, the sort of things that you use in the R ecosystem. I estimate that I've taught about 100 workshops, webinars, videos, or talks at different conferences. I've also written some books. And if you know my name, you probably know them from some of the books that I'm associated with, like R for Data Science. And one of the first things I discovered as I started to teach workshops about how to do data science is that there's nothing on the slide to know more about that, is that kind of put it charitably, teaching was not a requirement for giving a workshop. It's a lot like grad school where people are there because they're experts in doing something. And unless you're at an education conference, that something is not teaching. And so I would see these workshops that seem very painful for the students. There'd be paragraphs of text on slides that you'd have to read while the speaker is speaking, saying very important things. Or you'd go over minutes and minutes of complex statistical procedures or coding procedures. And after 45 minutes, then maybe during the workshop, you get a chance to try it out yourself. But of course, if you're brand new to this, you probably couldn't apply 45 minutes of material all at once. And I thought there must be a better way to deliver and design workshops for people trying to learn R. My father was a teacher when I was growing up, so I got to see a lot of that. And I had a bachelor's degree in psychology, so I knew that there are many principles that we know of about how humans learn when you apply in a workshop. Things like cognitive load theory, multimedia learning theory and all the rest here. If any of you are looking into training or want to be a trainer, I found that there's this resource called How Learning Happens. It's a book. You can get off of Amazon by Paul Kirshner and Paul Hendrick. It's a very concise summary of all of the most important things from cognitive science that you can use to make better training. And that's what I did. I took these things and I applied them to the workshop format. I iterated on each workshop I was teaching. As I mentioned, I did a lot of them. And I was very successful. I developed a workshop called Master the Tidyverse, where people could learn how to use the basics of the tidyverse to process data of any type. And to give you some sense of that success, here's a letter I received from the American Statistical Association after teaching this workshop for the third time. And all three times, I won an award called the Excellence in Continuing Education Award, which is based on student feedback taken at the end of each of the workshops at the National JSM Conference. And so it's very impressive for, of course, to win this award three times in a row. In fact, it's actually unprecedented. And they gave me a grant to go around to the different chapters of the ASA and teach this workshop, including Hawaii, which I was very happy to do. And there's a picture of me in Hawaii getting LA from the students of this workshop. So I felt really good. I cracked the code and I figured out how to make an excellent workshop that reliably got high student satisfaction points. And people tell me at the end of the workshop that they learned so much, or maybe that they thought they knew this was going through the workshop really helped them get it. And that seemed good to me. But I knew there was another side to the picture because in the research I was doing to figure out how to make better workshops, I came across facts like this. If you test students coming out of a lecture, college students, people you might think of as professional lecturer attenders, they only remember at best 56% of the content in that lecture. And that's for a one-hour lecture, I assume. If you put two or three or four or a whole day's worth of lectures together and then measure the content, would it be more or less than 56%? So I started to worry that people might not actually be learning what I'm teaching in the workshop. And then if you look at things like more procedural training, like how do you use this machine? Research showed that only 60% of attendees of the trainings could actually use the machine, do what the training was about right after the training. And then that percentage dropped month after month going forward. And there's psychological literature that suggests this would be the case too. This is a model really more than a plot. You may have seen these called Ebbinghaus curves or forgetting curves. One of the most reliable things in educational research, these were discovered in the 1800s and then verified since then in multiple domains. And the idea behind them or the way to understand this is if you learn something, so say that's x equals zero on this graph, you learn something, your mastery of it shoots up because you learned it. But then as time goes on, you start to forget it. And eventually you might completely forget it and that's what's on the x-axis here. But you don't have to forget things. If you practice it again, you'll learn it again. And then you'll start to forget again after that. But if you practice another time, you'll learn it again. And what happens is each time you practice a thing, the rate at which you begin to forget it decreases until finally you've practiced it so many times, it's like learning to ride the bike, you just never go forget again. It's hardwired into your brain. And in fact, that's exactly what happens. Neuroscience explains why this would be your brain's made of neurons. When you learn a new skill or retain knowledge, you do it by encoding it into a neural network. That's a group of neurons that fire together and really oversimplifying. But you may have heard neurons that fire together, wire together. And that's one of the things that underlies learning. When you do something, the neurons that allow you to do it are firing together. And when you practice it, they're firing together and when they fire together, they connect a little better in your brain. And so they're more likely to fire together in the future. Here's two hypothetical neural networks in our brain, these two circles of neurons linked. If you practice one of them over and over again, each time you practice it, those neurons will get slightly stronger connections. But if you don't practice one, so take the other one, it slowly fades away. Your brain maybe cannibalizes the materials that create these connections because it has more important connections to create. I mean, while the one you're practicing gets reinforced and reinforced and reinforced, until eventually it is permanent in your brain, you have attained that knowledge and the thing you didn't practice, you've lost that knowledge. This explains the Ebbinghaus curves. And I began to wonder if that was at play when I taught students workshops because these neural changes that happen, happen while you're sleeping. And as a general rule, I didn't like people sleeping in my workshop. So I doubt they're having the chance to really strengthen these neural connections and definitely not multiple times as would be needed to make a permanent connection. The next thing I wondered is if you think about the long-term ability of a student to do something and you compare these two curves, now this is just a model, so who knows how much area is under each of the curves, but it does seem that practice makes an outsized difference in their long-term ability compared to the initial instruction that I was providing. And when I talked to and followed up with students from the workshops, which it wasn't very easy to do, especially at conference and whatnot, but I worked for our students, so after I would see them again, I talked to them where they'd come and approached me and asked me to train their colleagues. I'd find that the ones who were doing this on their own, and who were doing it on their own before they came to my workshop, got it, retained it, and were able to use that workshop to really do something great. But those who were sort of ball and told to take my workshop, or if I went to a company and taught on premises, they didn't really have a reason to use it afterwards. And so they or their colleagues would tattle on them and tell me they really weren't using it anymore. And so that was a crisis for me. This sort of, I'm not letting people practice. These workshops aren't really helping people learn. Then also I started to think, what are they even learning in the workshop to begin with? You could ride a bike or you could attend a lecture about riding a bike. And you'll build two different sets of neural networks. One might help you memorize things like the vocabulary of bike parts. The other will help you stay balanced on a bike and ride it. And I wanted to make sure that my students were actually practicing problem solving doing data science and not just learning function names and things related to data science. I worried that in the workshop format they were doing the latter. And then I had the realization that data science is a skill. Problem solving is always a skill. You build skills through practice over time. Those are two things you can't really provide in a small window of time, such as a workshop. And to make everything easier to think about we could just think of data science as another more common skill like learning to play the piano or any musical instrument or any sport or any sort of thing that people practice over time to learn. You wouldn't feel comfortable thinking that you're gonna learn to play the piano in a workshop. Then that's it. So what about online courses? You might try to play the piano in online course. I don't know if it'd be successful but I do know that there's data about people who try to learn from online courses. If you look at massive online courses or MOOCs the data is really bad when it comes to completion. People start courses and then fail to complete them and the study at the top of Google says on median 12.6% students will complete an online course. I've helped out with Coursera and Johns Hopkins and some of the online courses and you do kind of see this. There's some wonderful hard course students who fall all the way through but most people drop off right in the beginning. I've also worked for another online course company through our studio. We helped launch a company and there's a bit of a comparison. I'll go say their name. But we built four of their first five courses and due to the payment model we got to review their data to make sure we paid correctly because we got paid when students completed one of our courses based on some definition of completion. But the company itself got paid by subscribers. It's a weird model. But what we noticed looking at the data was most people did not complete courses. Most subscribers didn't even really use the products so much or at a surprisingly low rate. It's a bit like a gym subscription model where if you have a big box gym they sell all these memberships and they could probably never fit all those people inside the gym at the same time. But luckily most of those people never show up. I mean they have good intentions of going to the gym. They don't actually go to the gym. They're just spending that money. Well that's a fine way to make money if you're a gym but that's not really what I'm after as a teacher. I want students to actually learn. And I think the reason these online courses are falling short is that as we establish day sciences as a skill being better at skill requires practice but actually doing that practice especially if you're a busy adult requires motivation. Most online courses are pretty anonymous. You're by yourself. You're watching people but they don't know if you're working or not working if you're just clicking through to the end of the course. And so students lose motivation and that course is one of the first things to go when things get tight or they run out of time on their work life. So I'm not the biggest fan of online courses either but where does that leave us? How do humans learn skills? Well, we learn skills all the time and we use things like apprenticeships, internships, residencies, coaching. All of these methods of learning to do something involve the same ingredients. There's lots of practice but there's a mentor there, a person who knows how you should do it and can give you feedback and more than that, that mentor knows what you should practice next which by the way might change based on how you're doing. Very rarely will you as a student know the most efficient thing to practice. So how can we implement this method of teaching with data science? How should your colleagues learn data science? Well, this is how we're doing it at RStudio. We have a teaching platform called Academy and if you were a student enrolling in an academy course, the first person you'd meet is your mentor. We give you a mentor, someone who knows how to do data science with R and someone who's gonna help you learn to do data science with R. You're also gonna meet your fellow classmates or as we call them group mates. It'd be nice if there's one mentor per one student but due to reality, we have one mentor for a group of students who all work together in a way I'll describe in a moment. So already you are in a social situation as you're learning to do data science and you're not anonymous in a crowd. Our groups are five to seven students. So people know you, they know when you come, they know when you're missing, they rely on you. Now what will you be doing? You and your group mates will be doing a project. This is a real life data science project from importing data to visualizing it randomly, whatever the project requires, fitting a model and then finally reporting that data. This is the sort of thing that you will do with R on your job if you learn how to do R. So immediately you can see the value of learning to do this. Essentially we've just turned this into an apprenticeship where you're gonna do this project under the guidance of a mentor. So here's an example project, analyzing click or draw data. If you think of the project as a long continuous task, we break it up into week long segments. We call milestones. So a project might be 10 milestones that build on each other and when you finish the last milestone you finish the project. We have students tackle one milestone per week to help them do the milestone. We give them a series of tutorials that we've created. These teach them the skills that they need to then apply on the milestone to succeed on the milestone. We've taken a lot of care to make these tutorials as interactive as possible with as much practice as possible. We've ran some grading software that goes way beyond unit tests, actually analyze the student code and make suggestions and tips when things go wrong. After you take those tutorials, you do your milestone. We ask students to push the milestone a little further in their own direction and personalize it and then they take what they've produced that week and they share it with the other members of their group who by the way are all doing the same sorts of things. So all along they're helping each other and asking each other questions. They don't make one project, they each make their own project but they do it in parallel with each other so everyone is sharing the same context. It turns into a very social experience. At the end of each week you get together with your group mates, you discuss what's gone on that week. I wish I could say it looked like this but actually looks like a Zoom room because that's how it happens in this world. A typical week again, you do tutorials to prepare you. This is where you learn skills and you practice skills. Then you apply them to your real-life data science project and then you extend that project which often requires you to learn to teach yourself something new an essential skill when you're working with a language like ours. And then you present to your group and also during the week there's office hours with your mentor to check in with. There's a shared Slack channel and a Teams channel for all your other team members. So there's lots of conversation, there's lots of discussion as this happens. And then this goes on for the duration of your project. Lots of practice, lots of feedback, lots of social interaction and you're doing something real life that's easy to connect to things that are valuable to you. So does this method of teaching work? The answer is yes, quite well. We've taught about 400 students so far, just shy of 400. I haven't calculated the completion rate recently but it is near 95% students finish with us. And those that don't finish are normally the ones that get promoted to a new role so they don't need this training or they leave the companies that they're at. So I guess they also don't need the training. And then, but an even better measure of success is do these students know how to use the tidyverse or R or whatever they study six months later, nine months later, afterwards. And I think the best way to answer that is to let one of our customers answer it. This is, this comes from James Wade who's a scientist at Dau Chemical. They did a lot of training with our studio academy. And I think the key thing here is he said, look, surveying students six months after they left academy, we found that 16 out of 17 of the survey respondents were still writing code at least once per month or more frequently. And that number 17 person was someone who was promoted to a manager so they didn't really have the opportunity to write code. He said they were blown away by that feedback and I was blown away by that feedback. This, this is probably what you expect when you sign up for training, but the reality is this is really unusual to achieve. We're achieving it. We're achieving it regularly. We're doing that group program called RStudio Academy. You can learn more about it at rstudio.com slash academy. But I hope that some of those ideas will be interesting to you as you enact training at your own workplaces. And if you want to sum it up to the bottom line is think of day science as a skill that you're going to help people build by practicing not as a set of facts that you go broadcasting through some sort of lecture hall environment. So thank you very much. And I could take questions if there are questions. So lots of compliments, Garrett. I don't see, let me look in the, I don't see anything in the Q and A. Compliments, this is a great accomplishment that they like the mentor and peer support practice that's put into this. Yeah. And there was a comment about, this also explains why it's hard to unlearn old or inefficient workflows. Oh, there was one question about cost, sorry. This sounds great, but also expensive. What's the sweet spot for price and outcome? Yes. Well, if you want to enroll in our studio Academy right now you do it through your company. It's right now we're only offering the service enterprises and it's about $3,500 per student. Although we do hope to have different sort of entry points for Academy in the future. Any other questions? It's not, thank you Garrett. Great presentation, really appreciate it. Thank you so much, Denise.