 Sorry, your mic's muted. When you're ready. No, when I'm ready, I suppose, right? OK, welcome everybody here in the Wikimedia Foundation headquarters and anybody listening live streaming. My name is Amin Azam, and I'm really delighted to present this work that my team and I have been doing over the last several years. I think it's important to start with some disclosures. I want to acknowledge explicitly that I have accepted travel funds to go to conferences, academic conferences, associated presenting my work. And then also, I am currently a consultant on a Hewlett Foundation grant that's designed to foster what we call open educational pedagogy, which I may describe a little bit more in detail later. OK, first I want to start with Jimmy Wales's inspirational quote that many people within the Wikipedia community have heard, imagine a world. Just imagine it. It's really remarkable and amazing when you think about what the Wikipedia movement has really initiated. OK, a lot of Wikipedians like to ask the question how big is Wikipedia for my health student community. They don't always know how to find the answer to this. Like many of us, I just Google search. And so if you Google search Wikipedia statistics, you'll get this hit list. This is from when I searched on the 12th of February. And if you click on that Wikipedia statistics link, it'll bring you to this Wikipedia statistics page where we can see that Wikipedia averages 10 new edits per second and that there were over 5,333,000 articles. You scroll up on this page a bit, you'll see where I'm getting those numbers from. There is a little table here on the side where you can see the auto-updated count on the number of articles. So when I last did these screenshots, it was 5 million. When I last updated this as of the 12th of February, we were up to 5,334,000. That number I'm sure is bigger than now. I'm sure it's bigger. I just haven't updated or searched recently. The other thing that Wikipedians like to do is to talk about the size of Wikipedia compared to traditional print encyclopedia volumes. So here's a picture where you have the average six foot man or woman. Each of these blue bars represents a volume of an encyclopedia. And you can see that Wikipedia is currently 22,217 volumes or the equivalent of 12 library stacks. If you zoom out a little bit on it, I like to use this image to point out here that there's still room for more volumes. We're not yet done producing all the sum of human knowledge on Wikipedia. Okay, who runs Wikipedia? We're here in the place where it happens. The Wikimedia Foundation is a nonprofit based here in San Francisco, my hometown. And what I think is really wonderful is that it employs fewer than 300 people, yet despite that number, it consistently ranks amongst the top 10 most heavily trafficked websites on the planet, which is really remarkable when you think about the movement, the power of a movement. Okay, so people sometimes wonder whether Wikipedia is used for health. So I have a journal article here from the Journal of Medical Internet Research, a relatively new journal, and a colleague of mine, James Heilman from Canada, has produced this article looking at that exact question. Within the abstract of the article, I wanna show you that at the end of 2013, the medical content was 155,000 articles, one billion bytes of text across 255 languages. And they argue that that makes it the most viewed source of health information globally. In talking with James, because this study hasn't been updated since 2013, his personal communication with me is that the current, as the end of 2015, we were up to on 1.4 million references supporting this content. Okay, within the Wikipedia community, Wiki projects are fairly prevalent. A lot of non-Wikipedia's or Wikipedia's to be have not heard of Wiki projects. If you Google Wiki project, you'll get this Wikipedia page. And basically, a Wiki project is a collection of individuals who are dedicated to improving a topic domain in Wikipedia. So, I wanna show you some examples of some Wiki projects. Here's the Wiki project food and drink everages task force. And you can see they divide up the beverages into different subdomains. Here's the Wiki projects mountain community who are dedicated to improving the quality of information about mountains on Wikipedia. Here's the beekeeping task force from the Wiki project agricultural group. Wiki project mathematics. Wiki project physics. And then, which leads to my community, Wiki project medicine. That organization of volunteer individuals, here's their Wiki project page. And their commitment is to provide high quality health information on Wikipedia. They do something really wonderful to help the people who choose to receive information on Wikipedia that it's health related. They rank order the 35,000 English language Wikipedia health topics by importance. And that rank ordering takes into account at least two important factors. One is the number of eyeballs reading those pages, but so too is the global burden of disease. So, human sexuality is top importance because everybody reads about that on Wikipedia, but so too is dengue fever even though it's not endemic to the United States. Here's an example of that rank ordering sort of structure for Wiki project medicine. You can see it's top high, mid, low priority. I wanna acknowledge that all diseases are top importance to people who either have them who have loved ones who have them. So I don't at all mean to imply that they are less important, but when you factor in global burden of disease and number of eyeballs reading, the Wiki project medicine volunteer community puts about 1% of those 37,000 articles in the topic priority. And you can see they are things like tuberculosis or cancer. Broad global topics of global relevance. Number of people don't realize that all Wikipedia articles are graded for quality, and that grading scale is represented here. This is true across all of Wikipedia, but I'm showing you the grading scale as it relates to the health related topics on Wiki project medicine. All articles start as a stub quality article, and as participants and volunteers add more text, it becomes a start quality, then a C quality, B quality on up the scale. The highest part of the quality scale is called a featured article, and featured articles appear on the Wikipedia splash page, and they rotate through different featured articles. So less than 1% of the featured articles are health related topics, but it gives you a sense of how and where we need to go with health related information on Wikipedia. When you combine the Wikipedia grading scale with the Wiki project medicine importance scale, you have this wonderful array that's available on the Wiki project medicine page, the Wiki page that I like to think of as a roadmap for our work as health professionals who wanna join and embrace the Wikipedia movement. So if you will, the top importance lowest quality articles are the place that Wikipedia needs our help the most. Let me give you an example. So here's an example for my own field, I'm a psychiatrist, so schizophrenia is an important topic within mental health globally, and if you click on the top tab of the schizophrenia Wikipedia page, you will see the behind the scenes where the Wiki project medicine community has labeled this article top importance, and it's also sufficiently high enough quality to have been graded as a featured article, quality article. Here's another example. The American Rotkin Ray Society is listed here. This has been a while ago since I printed this slide, so it may be longer. I don't mean to pick on my radiology colleagues, it's just that it all fits on one screenshot, and if you click on the talk page, you will see here that this is listed as a stub of an article, but also low quality, I'm sorry, low importance on their importance scale. Okay, so just to show you what those 18 articles are, if you click from Wiki project medicine, you will get the listing of the articles, and when they were listed as top importance, when they were ranked as a C quality, and see that important work that needs to be done. So we at UCSF decided to get on board by creating an elective for fourth year med students to edit Wikipedia for academic credit. It turned out to be very interesting to the press, and so I had a lot of opportunities to share what we're trying to pioneer here at UCSF with other health professional schools and with general people who are curious. Because everything we do within Wikipedia should be globally available for anyone that wants to see it, we had to put all of our work and effort on Wikipedia. So rather than making it behind a course firewall or within a learning management platform, we created everything we did on Wikipedia. So if you're interested, you can look up at UCSF School of Medicine Wikipedia elective and you'll find this webpage, this Wikipedia page, excuse me, and you will see all the history of our elective that all the students that have done all this work during the last four years. We have now offered the course six times over the last three years, and I'm happy to share that the seventh cycle has just begun, so I am thrilled to have eight students in this cycle that are continuing the work of their 72 predecessors as part of our work. The question then becomes, if you've never done this work before, how do you go about evaluating the impact of an elective course like this? I'm an educator, so I think very much about the role of the elective in the lives of my students. So in partnership with my colleagues who you'll see listed at the end of this presentation, we designed, we used principles of emergent design to think of three broad content domains to evaluate the impact of the elective. The first domain is the impact on the students themselves. For that domain, we chose to use a variety of education standard practices in education research or scholarship, mid-course one-on-one interviews with all the students, with some structured questions I asked the students. We did an end-of-course evaluation as a group so that the students could weigh in on each other's opinions and influence each other. And then we took all that data, we transcribed the interviews, we coded them for themes, we conferred across interviewers to make sure that those themes were accurate, and I'll share that data with you shortly. Second broad domain we thought we'd look at is the impact on Wikipedia readers themselves. For that, Wikipedia does the work for us already. We can just look at the traffic statistics on those articles during only the months that my students were actively editing. And the third broad domain is the impact on the quality of Wikipedia itself. I'll go in a greater detail about all of these domains, but we used a variety of strategies to assess the impact my med students were having on the quality of Wikipedia's health content. So I'd like to share that the results I'm sharing with you now are from only the first three cycles of the course, the first 28 students that did the course. And I need to share with you that we did ask for permission to do this research from the UCSF Human Subjects Committee who deemed us exempt from formal review because the risk to our students as study subjects was sufficiently low that we didn't need to ask for consent from the students. All of my students knew they were being researched for purposes of talks like this one. Okay, so let me share with you what we found. In the first domain, let's talk about the impact on the students themselves. What I'm gonna share with you is the themes that emerged from our qualitative data, and then I'll share with you some paraphrase or some examples for why and how this came up. The first thing was that our students learned that they needed to write for what we and they called the lay audience. They're not writing it for fellow medical students or fellow health professionals. And one of the things that I think is important here is that the average Wikipedia is going to, or the average consumer Wikipedia is going there for health related information is not as highly trained as as my students are themselves. So they rightly grappled with this task. I'd like to believe that this work also makes them better at communicating with their patient populations when they actually work with patients in real clinical encounters, but we did not assess this formally. So this is a hand-waving opinion on my part. Second domain, they love giving back to Wikipedia. They will all acknowledge at least privately to you and to their friends that they use Wikipedia. They tend not to admit it to me as their faculty member, but it's one of those things where they love specifically giving back to Wikipedia, which is good. I think it's wonderful that they're giving back to that instead of to other closed access resources. So they use it all the time and now they can give back to other medical students wanting to use Wikipedia going forward as well. Okay, third domain, it's hard to delete other people's work. So our students, even though they have a lot of knowledge as four-tier medical students, deferentially defer to perceived authority sources, and so I would offer that my students at least have a hard time deleting that other work, even though that other work may have been people who, written by people who know less than they do about the body. They find it hard that another domain is the notion of editing live on Wikipedia. All Wikipedians can edit in a sandbox before they edit live if they want, and my students have a hard time switching from what they've learned in their entire academic careers, which is to buff and polish their work until perfection before they hand it over to me as their faculty member. The problem is that Wikipedia's ethos is to be bold, and so one of the tasks that I've had to learn to do with my students is to unlearn the notion of perfectionism before you edit live on Wikipedia. Another domain that emerges is they actually really love it. They love the opportunity to provide information to a huge readership of Wikipedia, and I'll show you that in a minute. And this, I've Drunk the Kool-Aid is a literal exact quote from one of my early students. I'm quite, and I have to say I agree to. I've Drunk the Kool-Aid as well. One way to frame that is you think about the breadth and duties of physicians going forward is this theme that my students now begin to realize that contributing to the digital healthcare space is perhaps a broadening of their sense of what it means to be a physician in society from what they thought it was when they started medical school. And I agree too. This is easily the biggest thing I'm doing, helping future generations of physicians become Wikipedian contributors beyond only the patients they choose to see and beyond only the students I teach. Okay, so that's the results on the students. Let me tell you about the results on Wikipedia readers themselves. Those first 28 students edited 28 different Wikipedia health related articles, and those, sorry, I went too fast. Those articles were viewed during only the 30 days each of my students were editing 974,000 times. Remarkable number when you think about it. I didn't realize how high it was gonna be. It's, I'm not proving anything we don't already know, which is that a lot of people go to Wikipedia, but the number of people that go to Wikipedia for health related topics was just striking for me to actually calculate. Okay, let's switch to talking about the impact on the article quality. I told you I'd go over all of these different domains, so let me go over them systematically for you. I wanna acknowledge it was wide variation across these 28 students, but I'll share with you some results in aggregate. Okay, in total, those 28 students made 1,084 edits to Wikipedia on average of 39 per student. However, I wanna point out to you that if they convert a comma to a period and hit save, that counts as one edit. But if they spend a week working on a whole new section and then hit save, that counts as one edit. So it all depends on how many times they hit save. So that edit's number is kinda hard to interpret. So a different strategy would then therefore be to look at the number of bytes of text they've added to Wikipedia. So in total, my students added 369,000 bytes of text. However, they were also removing text that they deemed to be inappropriate or inaccurate or low quality. So they also removed 82,000 bytes of text. In total, they added 287,000 bytes of text, an average of 10,000 bytes per student. But unless you are our computer science colleagues and friends, you don't really know what bytes mean. So a third strategy we decided to look at was how many references are they adding to the text they're putting on Wikipedia. So one student added as many as 66 new references to the article she was working on. Another student removed as many as 64 references that were either low quality or that the student felt didn't accurately reflect the sentences that they were citing. In total, those 28 students added 274 references. And you can also look at not just the number of references but the number of references per article. On average, when they started those 28 pages, 29.4 references, and after they were done 37.6 references per article. Okay, I wanna add another domain or measure of quality. And for this, I'm gonna talk about looking at Wikipedia's quality scale. So of course, we could easily measure the articles before my students touched it and then after and compare them. However, because I wasn't convinced that the academic health community was gonna believe this scale, I had a pair of UCSF affiliated physicians that graded the articles pre and post to see if we could compare the quality according to physician eyeballs. And that scale, I think I'm gonna show you next. No, let's talk about the Wikipedia quality scale first. So I told you we had 28 students. The first of those 28 articles that they worked on, eight of them were either sub quality or start quality when they started the work. By the end of the month, the students moved those articles according to Wikipedia scale as shown here. 15 of the articles started at C quality and they moved up as you see here. 24 of them moved up to B quality. 11 stubbornly stated C quality. And five of the articles started as a B quality article and all five stayed as B quality. What I think is interesting is when you look at the UCSF physician sort of quality scale, we, I wanna be transparent, made up the scale here as unimproved, improved and very improved. So totally made up, but at least using the physician's sort of wisdom and experience to grade the articles. I had two physicians, each grade the articles pre-post, they then compared their grades to achieve consensus about that before telling me what their final grades were on this. So on that scale, unimproved, improved and very improved, of these 28 articles that my students worked on, the physician graders felt that 14 were very improved, 12 were improved and two were unimproved. What I think is here fascinating and unintuitive here is that I fully expected it to be easier to move up the Wikipedia grading scale than it was to impress the physicians. And in fact, it's the physicians that are seeing the articles as being higher quality, faster than Wikipedia. And what it speaks to, frankly, is the rigor of the Wikipedia grading scale and what it is necessary in order to move up that scale based on number of sentence referenced, et cetera, et cetera. So it was a counter-intuitive findings from our work. Okay. One more domain of quality I wanna share with you. There's a nonprofit that we collaborate with that produces a report called an AcroLynx report. And this is a, I like to refer to it as a spell checker, grammar checker on steroids, if you will, because it's a text simplification software that produces reports for my students about the Wikipedia pages before they start editing. Here's an example of the report. So you can see a number of highlights in the back. Blue might be grammar errors, green might be spelling errors, and red might be style errors. But in addition to this Wikipedia page with those reports, the students also get a summary of what those mean. So you can see in this example, a sentence that's 34 words is deemed too long to be able to be translated or to be easily understood by the Wikipedia average Wikipedia reader. So the reports also generate a scorecard that looks something like this. And you can see that there are scores for spelling, grammar, style, readability, translatability. These scores are on a 100 point scale, 100 is good, zero is bad, and all the reports had baseline scores in all these domains. I told my students, of course, that you have your own brains here just like a spell checker. You don't just automatically accept all the changes. So I asked them to use their own best judgment as they did this work. But we also then could look at the baseline acrolink scores and the end of rotation scores and compare. Acknowledging that there were modest gains, there were consistent gains across most of my students who work across all of the articles in all of these five domains. Style, readability, translatability, overall writing, and grammar scores. Okay, so in sum, when you converge all these lines of evidence, I would like to make the argument for you that effectively my medical students are improving the quality of Wikipedia through their work. Okay, I want to remind you, everything we've done is on Wikipedia. You can go on the Wikipedia page. You see SF School of Medicine, elective, Wikipedia Project Medicine elective. You'll find it a variety of ways. All of it's there for all the world to see. Okay, I want to share with you a couple of postscripts now from that initial research, which has to do with longevity and cumulative impact. Again, on those same three domains. When we talk about the impact on the students themselves, to our knowledge, no more than one of them has gone on to actively contribute to Wikipedia after the course ended. To be fair to my students, they're all busy being doctors right now. So they may not choose to prioritize this work even if they do believe in the importance of it. From my perspective, I still have more students coming in. So I'm not trying to convince medical students to become lifelong Wikipedians. I need to just get other med students joining them in the ranks afterwards. Second point, in terms of the impact on the article quality, to our knowledge, none of the first 28 students' edits have been vandalized or reversed, in some cases as much as two years after they left the work. Now we haven't looked at this again in a couple of years now. This was the early sort of work that we've done. It would be interesting to look to see whether that work has been reversed or vandalized. And importantly, I think also whether it's been approved upon since they left my course. If we look at the number of people reading the work my students have done, those first 28 articles have been viewed 22 million times subsequent to my students' work. And in some case, and I should say this is the last time I checked just October of 2015. So it's been a while. But the number is just at some point, it gets to be astronomically big. We know it's a big number of people reading their work. I don't think we need to go above 22 million. Okay, I want to share with you the latest in the work that we've done subsequent to the early work. To share with you now, the subsequent groups of students that have done the work. So I've been offering this course up to two times a year for the last four years, including the current cycle with eight students. In the fall, last fall I had 22 students enrolled and you can see that they edited 14 pages. My course is always about a month long. In total, as we add up the work that UCSF medical students have done, effectively our students have made over 2,700 improvements to Wikipedia pages that have been viewed on average 1,650 times a day. It's really remarkable when you start to add up how much impact our students can have on Wikipedia readers. Okay, I come back to the classic quote from Jimmy Wales just because I've drunk Kool-Aid as well. Let me then just broaden to talk now to talk about the Wiki Education Foundation, a wonderful organization. If you're not aware of it, it's a spin-off from the Wikimedia Foundation. It's a separate nonprofit, also based here in San Francisco. But their sole mission is to partner with faculty and university members across North America to replace a traditional classroom assignment with a Wikipedia editing assignment. And they ask the audacious claim, can editing Wikipedia change a student's life? I'd like to share you some of the quotes from students who have done Wikipedia editing work as part of the Wiki Education Foundation. Here's some examples of some quotes from some students. I won't read them all because they're all on the webpage. One that I like here, I was encouraged by more than the grade. I wanted to contribute something long lasting, something bigger than myself. In total, the Wiki Education Foundation since 2010 has partnered with students and faculty and universities in North America, and I'd like to share some of their aggregate totals. There have been over 1,000 courses that have embraced Wikipedia editing as part of their courses. 22,000 students have taken on Wikipedia editing assignments as part of a course assignment since 2010. Those 22,000 students have edited 37,000 Wikipedia articles. 97% of faculty who embrace Wikipedia editing as a course assignment would continue doing it again. In 2016 was labeled the Year of Science for Wikipedia to encourage promotion of active editing about Wikipedia content that was science related. And in 2016, in April 2016, 6% of all the science content created on Wikipedia was content created through the Wiki Education Program or content created by students. Those students have created work that's been viewed on Wikipedia 147 million times. And those students have produced 24.4 million words on Wikipedia donating that work for free or for academic credit, however you wanna define the work. Okay, I'd like to share a video with you now from the Wiki Ed folks just to share that they're doing some wonderful work and if anybody out there is interested in joining them, here's why. Hi there, I'm Samantha Erickson and I'm the Outreach Manager for the Wiki Education Foundation. We at Wiki Ed are here to support you and your students as they learn to write for Wikipedia. An assignment on Wikipedia looks very much like a literature review. Students are asked to select a topic, compile a bibliography, draft their work, and complete a peer review. The biggest difference is during the assignment, students move their work live onto Wikipedia and there are so many benefits. Not only do students find an audience for their work outside the classroom, but they are doing a public service providing insight into paywalled academic research. Other learning outcomes include critical research skills, information literacy, and writing skills. As a non-profit, Wiki Ed is committed to creating free teaching tools and resources to support you through this process. Over the last few years, we've done just that. We've provided support for over 1,000 university and college courses in the United States and Canada. In spring of 2016, we supported 215 courses and over 4,000 students as they worked to update nearly 5,000 articles on Wikipedia. Together, they added 3.73 million words to improve the site. We offer assignment design guidance, staff support for students, online trainings, and print resources as students update articles about topics like the Arab Spring, dinosaurs, or women in science. Yes, our students have improved those exact articles. On top of these resources, we have a custom-built course dashboard that will track your student work, provide access to our assignment template and online trainings, and track live page views your students receive on their work. So when a student improves the article about vaccination policies on Wikipedia, we can see that this article has been viewed over 5,000 times. If you've ever looked at Wikipedia and thought, that could be better, but haven't known how to start, I encourage you to think outside the box and replace or compliment an assignment in your upcoming course with a Wikipedia project. To get started, visit teach.wikii.org or contact us at contact.wikii.org. I want to acknowledge that that video was produced in collaboration with the WikiEd community and with UCSF in our efforts at embracing change. Let me share with you my WikiEd sort of course dashboards just to show you the work that my students have been doing. So these are the previous cycles in my course, and if you click on any one of them, this was the one in fall of 2016, I think you can see here, I structure my course by having two days of what I call a sort of immersion boot camp about learning about Wikipedia editing, and we have some guest faculty members and folks from other communities like the WikiProject Medicine community join us. I use the WikiEd training modules to have my students train up on all of those general principles, and then when my students create Wikipedia user names, I can see what articles, they choose an article each to work on for a month, and then I ask them to peer review each other's work. So this is the screenshot where I can assign those articles and see how they're doing. Track adherence on the training modules, and then so here's tracking adherence so I can see that they've done all the modules that I've asked them to do, and then WikiEd's sort of dashboard allows me to track every single edit my students make very conveniently so I can see how many characters they're adding or taking away, and there's a nice side-by-side comparison of the pages pre-post so I can see the work that they're doing. Makes it extremely easy for me as a faculty member to do this work, and the best part is the aggregate totals here. So you can see that in my last cycle of the course, I had 22 students, they added 21 articles, they made a total of 850 edits, they added 35,000 words to Wikipedia, and those pages were viewed 513,000 times. This is a nice way to motivate the students to continue to do as much as possible during the course. The fruits of their efforts I wanna share, brag a little bit about my students. Last spring I had seven students enroll in the course, they decided collectively to work on one article, Hepatitis. Why? Because it was a top-importance article and it was relatively low quality. So the seven of them together worked on that. If you click on the talk page of the Hepatitis page, you will see up at the top that this shares with you that this was a course assignment in the fall, sorry, spring of 2016, and you can also see, thankfully, that they were able to move it all the way up to a beat quality article. On the talk page, you will see some examples of where my students are declaring the work that they're doing and the sections they intend to work on. So those group of students decided to divide it up into different sections, and one of my students was working on signs and symptoms, a different one on the prevention section. Here's a different example. This fall I had a student work on Bacterinia. If you click on the talk page, you will see again that this is an assignment for our course this last fall, and I had three students reviewing it and three students contributing to it. And again, in the talk page, you will see the examples of their work where they declare how and what they're gonna do during the course of the course, and they also review each other's work and they post their peer reviews on the talk pages. And part of the rationale for that is that even if my students don't respond to all of the good recommendations for how to improve Wikipedia, that other Wikipedians can have some suggestions here from UCSF med students to improve it, and they can also tackle that work as well. Let me share with you that on my end, as an academic medical educator, I am thrilled to have brought this work forward to sort of the coin of the realm in my world, an academic peer review journal. So academic medicine is the top peer journal in my field, and I'm thrilled to share that in February, earlier this month, we had an article published in that journal, why medical students should, medical schools should embrace rather than shun Wikipedia. And I of course have to put my money where my mouth is, we have made this an open access, no issues accessing the article within the journal. So you can find this on our Wikipedia page, you can also go to academic medicine to find it and download and read it for free. Goes into much greater detail about our work. Okay, I wanna move then to talking about like how many people are reading our work, my students, wonderful work. So the Wikimedia Foundation has a tool that allows you to search any given Wikipedia page and just see what the traffic statistics are. So here's the hepatitis page I shared with you earlier. And those are the dates from the, you can change your date range. What I wanna point out here is that there's this interesting pattern here, basically because people don't read about hepatitis on weekends. So this pattern is really representing the traffic associated with weekdays, probably if people have been diagnosed with hepatitis or have heard about it or a family member has just been told the news about hepatitis. So I just keep track of all my students' work and how many people have been viewing their pages. So this is just one of the sort of spreadsheets in my computer. But I wanted to share with you all that I last updated the search in February 1st. And the current total as of February 1st was 34 million views of pages that my 72 UCSF medical students have touched. And that's subsequent to the moment my students stopped doing the work. Again, it's remarkable. Okay, in terms of good scientists will tell you the limitations of their work and next steps that I wanna do the same. Limitations here, we've only had 72 med students do the work. There are early adapters in this domain so it may be that they are more atypically enthusiastic about Wikipedia editing. I doubt that but I haven't systematically studied that. I have also not systematically been analyzing the subsequent work beyond the first 28 students. And this is a one institution study so results may not be generalizable to medical students at other health professional schools elsewhere in the world. In terms of next steps going forward, we have a total of 31 students enrolled in the elected this year. And UCSF School of Pharmacy now has pharmacy students editing Wikipedia health related pages as they relate to medicines topics on Wikipedia. And it might be interesting to collaborate across our schools interprofessionally. It would also be wonderful to see other schools jumping on board this bandwagon and having their students contribute to Wikipedia as well. Lastly, I think it'd be interesting to think about the right size of student teams. I told you about a time when I had seven students editing one article, maybe it would be better to distribute the student population into smaller teams rather than large teams to see how can we optimize how much they can improve Wikipedia. Okay, I wanna share with you the work at your UCSF with my colleague at the UCSF School of Pharmacy as well. The variation to show you for those of you that are out there listening who might be considering doing this work, other strategies for embracing Wikipedia editing and health professional schools. So in our course, it's a one month long elective course. Students are enrolled only in this course and no other course. And the only work I'm asking them to do is to improve Wikipedia. It's not like they're doing other tasks or seeing patients during the elective as well. In contrast in the School of Pharmacy, my colleague Tina Brock has a 10 week course. The Wikipedia editing within the course is only four of the 10 weeks. And the assignment is about 15% of the student's final grade. It's not all of their work, obviously. And the students are also enrolled in other courses. There's nothing magic about the two ways we're doing this here. These are just two strategies to embrace Wikipedia within health professional schools. But the work that we've done so far here, you can see that on the pharmacy side, there are 360 students that have done the work. And what I like to do in pointing out the slide is that when you make it a required element of a required course, even though they've been at it less than we have, they've made a bigger impact than my students. Here's Tina Brock, my colleague from the School of Pharmacy sharing why she embraces this work and why she thinks it's worth it for students to do the work. Hi, I'm Tina Brock, a pharmacist and educator at the University of California, San Francisco. And I'm here to talk about motivation. For me, what's been so exciting about editing medicines related Wikipedia pages in our pharmacy course is that the activity hits what the author, Daniel Pink, has described as the motivation trifecta, supporting students and instructors with autonomy, mastery, and purpose. Let me explain. Autonomy is our desire to be self-directed. With this type of assignment, you have the option to have students select which drugs they edit and which parts of each page they edit. This can stimulate some powerful conversations and even debates about need, what pages get the most hits, what parts of the page are most critical to get right and so forth. Mastery is our urge to get better at what we're doing. For sure, the students' first few edits are slow to come and even a bit scary for them. The vastness of the Wikipedia landscape can be intimidating. The style guidelines take some getting used to. We encourage them all to start small, building their confidence along with their competence. But since their progress is visible, they can watch themselves get better at this right there on the Wikipedia page. By the end of the four weeks, they are not only better editors, they are better at giving feedback to other editors, students or otherwise. This is a hard one, but transferable skill. And finally, purpose. Purpose is believing that you're making a contribution. Today's learners can smell inauthenticity from a mile away. Probably most pharmacy programs in the world already have their students prepare a drug monograph of some sort. And at the end of term, once these are graded, we pitch them in the recycle bin and get ready for the next year. But by working within Wikipedia, they can see the actual impact of their hard work immediately. They are actually helping others while they're learning. That realization that this type of assignment isn't busy work is transformative for most of them. And frankly, it motivates me too. So incorporating Wikipedia assignments into pharmacy courses can activate our natural motivators, autonomy, mastery and purpose. And because motivation is linked to effort and persistence, this activity may show great benefits for students, instructors and society. I wanna talk about that article that I showed you earlier in this talk because if that isn't big enough, there are some other reasons why embracing Wikipedia and health professional schools and in universities should be the norm. In this article that my colleague James had published in the Journal of Medical Internet Research, he looked at the internet by language and he looked at the languages spoken by the world's population and looked at Wikipedia by language. And what you can see in this circle here are things that we kind of intuitively know. Most of the internet's in English. The problem is that most of the world doesn't speak English. And Wikipedia is closer to that circle up here than the rest of the internet. Why? Because anyone can edit, anyone can contribute. We have all these different language Wikipedia's. So James makes the argument if we wanna provide high quality health information to the world's population in their languages they speak and read, Wikipedia may be the most viable mechanism to do so. To make that part of a more broad collaboration, an additional community that's joining that movement are translators without borders. Which is a volunteer community that partners with Wikiproject Medicine to take English language Wikipedia health related articles after they reach a minimum quality of B or higher and then translate them to other language Wikipedia's. So as of the time of this talk they have translated over 1,900 articles in more than 90 languages. I wanna show you their work. Here is that they use sort of Google Sheets to sort of monitor and track the work. So you can see an example here of some of the European languages and the health related articles that are being translated to other languages, other language Wikipedia's. Here's another screenshot of the Asian and Middle Eastern languages. Of course it scrolls down and across but you just get the idea that there's more work here to do. And then even more work to be done in African and South American regions of the world. The point is that that work is ready to be done once these articles get to a high quality threshold. Lastly, I wanna tell you about Wikipedia Zero if you're not familiar with it. It's an effort to provide access to Wikipedia in the developing world without charging data access charges. The Wikipedia Zero Wikipedia page shows you a map of the countries in the world that now have access to Wikipedia for free. And as of the time of this talk 600 million people in 64 countries across 82 mobile operators now have access to Wikipedia for free. When you put all that together I like to tell my students that they have an opportunity in collaborating with all these different communities to provide high quality health information to the world's population in the language of their preference in the way they already access the internet and for free. And when I tell them that, I say, get busy. And then they do amazing things over the course of the time we partner together. I wanna show you in closing a fantasy idea of mine. The Association of American Medical Colleges publishes a report annually on the number of graduates from each medical school in the US and they break it down by men and women and total number of learners. And I wanna show you our school, UCSF. So in 2014, we graduated 160 new medical students. Of those 160, 12 of them did my elective. In 2017, we had, sorry, in 2015 we graduated 177 new doctors and 16 of them did my elective. Last year, we graduated 179 students of which 22 did this course. This current academic year, we have 31 students enrolled and expected to complete the course. If you add up a total number of doctors in the US and Canada that graduate every year, that total number at the bottom is 18,990 or 38. The number goes up as new schools increase their enrollment and so that will go up above 19,000 soon enough. And I would like to propose to you a fantasy idea. If every single one of those newly minted physicians on the day they put the MD initials after their name, simply as an active service made one edit to Wikipedia's health related content. Can you imagine how quickly we've been improved the health related content on English language Wikipedia? To me, it's just really a call to action for all health professionals to embrace being a part of the digital community that we intend to serve as physicians. I want to close by thanking all of the colleagues who are involved in this work because I could not have done it alone. I happened to be the one person standing up at the podium. But these are the faces of some of my colleagues who were involved in the initial work and where they're from in evaluating all the work that my students have done. And I really want to acknowledge more importantly than anything else, my students themselves. So this video is me talking and since you've heard me talk, I don't need to play it. It's a little clip about why this is the biggest thing I'm doing in my life. Oh, sorry, one thing. Okay, so before I show you my students in the work they've done, I will show you that I think that we're at the cusp of an emerging movement in embracing Wikipedia editing as part of health professional schools. So we at UCSF has done this since 2013. There's a medical school in Israel that started this work in 2014. You can see here that there's a medical school that started it to this year in Australia. And there's a couple of other schools that are exploring the possibility of embracing Wikipedia editing in their schools going forward. Okay, this is the slide I meant to show you. I want to close by just acknowledging that it's not me doing the work, it's my students. And I'm really proud of them. So I always like to close by sharing with you some of those early pioneering students that did this work, their Wikipedia handles and the articles they worked on. And by closing and honoring their work by leaving it up at the end of any of my talks, I'd like to acknowledge that I'm really proud of them and believe in the work they're doing making the world a better place. Thank you guys for your attention. And for joining me on this presentation. All right, off. I don't know if we are, Nicole, are we? I don't think we plan to, do we?