 So, there's an interesting news story floating around the internet at the moment that claims that a German scientific study proves that playing Super Mario Odyssey can treat depression. The strength of this assertion varies between outlets, with some being a bit more, shall we say, careful than others in the way they phrase their headlines, but this makes a lot of sense, right? Playing Mario is fun. Of course, playing Mario is a solid treatment for depression. Well, hold your horses or yoshies or what have you, because with academic studies like this, things are often not quite as simple as they seem. And indeed, in some ways, things are even simpler than they might appear. It just so happens that in the days before a witch cursed me with the ironic punishment of making YouTube videos for a living, it was my job to analyse scientific studies. And from the moment I actually took a look at this paper about the benefits of playing 3D games for helping with depression, I saw a big red flag that meant that the whole thing should absolutely not be taken at face value. I'm not saying the study is worthless, far from it, but the problem with academic or scientific studies is, when the media gets hold of them, you can never tell how a story might end up bouncing around from news site to news site. While a scientific study has found some limited mental health benefits to playing Mario in very specific circumstances, there's more going on here than you'll get from a news headline. It's important to talk about this, because if all you take away from an article is that playing Mario is more effective at treating depression than traditional medicines or counselling, well-meaning media coverage of a scientific study could inadvertently end up causing more harm than good. This being the case, let's get into it. But first, a quick disclaimer about headlines in general, and indeed YouTube video titles. There's more at play here than just clickbait as you might understand it. Let me explain. If you've never worked as an online journalist, you might find it frustrating the way some articles have headlines that are written in a sensationalist way that make them seem more exciting or groundbreaking than they actually are. As someone who's had an article featured on Saved You a Click, I can confirm this isn't usually deliberately The fact is, the 24-hour news cycle is harsh. You've got to get relevant articles up online as quickly as possible and simply by virtue of the low-pay and constant time crunch, a lot of journalists end up taking the path of least resistance. So when you see another publication covering a story about a Nintendo game treating depression, boom, that's an easy win of an article. You can just copy the basic facts from other coverage and you've just solved one of the big challenges of your day. This is exacerbated over the quiet Christmas period, for example, which is why a study that was published several months ago has suddenly become newsworthy in the absence of anything more attention-grabbing. The problem is, hard-working but time crunch journalists don't always do due diligence in checking the original sources from these stories. That takes time and a well-research story doesn't get any more clicks than the copy-paste rephrase approach. There's a reason after all that this video is coming several days after GameXplain did their video on the same subject. Research takes time, speed gets more eyeballs. So a headline in a scientific publication might read, cognitive trainings using video games might increase subjective well-being of individuals with depression. But that story gets picked up by a more mainstream website, one that goes with a more affecting headline. Something like, scientists used Mario game to treat depression, here's what happened. Or a new study shows that playing Super Mario Odyssey can combat depression. Or this gem of a headline, can Super Mario Odyssey take away depression? A study gave the answer. Right up top, the answer to that one is no. A Mario game may be able to help with the symptoms of depression, but it won't take away depression. In the same way that regular exercise can reduce symptoms of depression but won't necessarily cure depression outright. Broadly speaking, the further away from the academic side of the internet you get, the less helpful it is to just read the headline. Kudos to several Nintendo-specific news sites that went with something less clickbaity and reported this as something like, study finds playing Super Mario Odyssey cuts depression symptoms by nearly 50%. But things aren't that simple. That nearly 50% is pretty dubious, for reasons I'll get into a little later. The problem with this is that on the internet, most people either only read the first couple of paragraphs or they just read the headline. And people might walk away with the idea that, A, there's something specifically potent about Mario games, which is not what the study was attempting to prove, and B, that the Mario game was the only factor at play here. Unfortunately, lengthy academic texts don't always translate well into a social media friendly bullet point list. A lot of the nuance gets lost when the story jumps out of scientific circles. With that in mind, let's look at the study itself. Scientists use Mario to treat depression. Here's what happened next. I don't think that anyone involved with this study will be too upset with me calling their methods and findings into question. I think this because of a quote from the papers abstract. Given the mixed and unblinded nature of this study, the results should be interpreted with caution. So the authors of the paper themselves, Bergman et al, fully admit that their research is flawed. This is not meant to be a perfect study definitively proving that Mario can cure depression. The study didn't even set out to test games as a means of treating depression at all, but rather about treating one of the symptoms of depression, a decreased cognitive function. Essentially, and I apologize for oversimplifying things to the point that it will probably really upset actual mental health experts watching, but if you are experiencing depression, it's harder to think clearly. The example that I like to use is that you've got too many tabs open in your browser, and depression is a 4k YouTube video upload that's just slowing down other processes. So with this in mind, and based on prior research showing the cognitive benefits of exercising your brain with video games, Bergman et al set out to conduct a brief preliminary study to see how well video games can help with, well I suppose the most apt expression would be brain training for people with depression. To quote the study, the aim of this study was to investigate whether a six-week video game intervention leads to improvements in depressed mood, training motivation, and visuospatial working memory functions in patients with major depressive disorder. So the study was to test participants to see if, in addition to all other standard treatments for depression, playing 3D games regularly improved patients' mood, motivation for engaging with treatment, and visuospatial memory, the three M's. Mood, Motivation, Memory. Spoiler alert, Mario did indeed help with all three, but doing absolutely nothing was apparently also pretty effective all things considered. I'm getting ahead of myself. Let's talk about the study's methodology. Let's see if you can spot any problems with the way this study was conducted. 46 patients from the Department of Psychiatry and Psychotherapy at the University Hospital Bonn were recruited to take part. Patients had to meet certain criteria. They could only have a diagnosis of major depressive disorder, no other mental illnesses such as schizophrenia, bipolar disorder, or substance abuse disorder. They also couldn't be too familiar with 3D games, in particular Super Mario Odyssey. Participants scored themselves on the Bex Depression Inventory test, giving themselves a score of 0 to 4 in the following criteria. 0. I do not feel sad. 1. I feel sad. 2. I am sad all the time and I cannot snap out of it. 3. I am so sad and unhappy that I cannot stand it. They were also tested for memory recall. The participants were split into three groups. One group was told they were to play Mario Odyssey to cure their depression. One group instead trained with a different software, a more clinical offering called Compac, and a third group didn't do any kind of brain training at all. All participants continued to receive standard treatment for major mood disorder. At the end of a six-week study, all patients completed the Bex Depression Inventory questionnaire again. Those who had been given some memory training were asked about their motivation to complete their assignments, and their memory recall was tested again. Did you spot any problems with the study? There are several. 1. The first red flag that gave me cause for concern, the first thing that I check any time I look at a scientific study, there were 46 participants. This is a painfully small number, and you can usually tell a great deal about the accuracy of a study's findings by how many participants are involved. The more people who take part, the more generalizable the results are to a wider population. And the fewer people who take part, the easier it is for any statistical anomaly to look more important than it is. With a sample size of 46 people, if one person happens to find their depression is reduced during the trial period because of a different factor, a change in living circumstances, for example, or simply a shift in seasons, it counts for over 2% of all participants in the study. Because there were three groups in this study, only 15 or 16 people played Mario, and that means we're looking at each participant counting for around 6% of the total results from the Mario group. The numbers can easily be given false relevance, just because one participant is having a particularly good or bad day. So, as a rule, a small sample size makes a study very difficult to rely on, and indeed the academic paper reporting this study, written by the scientists who conducted it, admits that the sample size is too small and that further testing is required. Preliminary studies like these are very common in science, and they serve an important role. They're good for testing out a working hypothesis in a quick and dirty fashion, the scientific equivalent of running some ballpark maths in your head rather than sitting down with a calculator and getting things worked out clearly. This is useful if you're looking to commit years of your life to a study, and if you're looking to secure funding, because scientific research isn't cheap and sometimes you need an initial smaller study to prove that your hypothesis is even worth exploring further. It's also worth pointing out that with this kind of study, if you're looking to acquire funding to pursue things further, you have a vested interest in interpreting your numbers to make the case for your theory as strong as possible. Nobody wants to do a study like this and come to the conclusion that their hypothesis simply wasn't born out in the data. Quite aside from anything else, that's a bit of a career dead end. Similar to the sample size issue, I would argue that a six-week test is really not enough to get a long-term sense of the effects of any treatment. The human body is actually very good at doing incredible things in relatively short bursts when given the right motivation, even if it doesn't actually solve the problem long-term, which is widely believed to be the science behind so-called faith healings, shh, you didn't hear it from me. My other concern with this study is twofold, but I'm lumping these together. The fact that this is not a blind study, meaning that participants and scientists alike were very well aware of who was receiving treatment, and the use of a self-assessment questionnaire for judging levels of depression. Blind studies are important in medicine to avoid bias. If a patient knows that they're receiving treatment, it can often cause their body to respond differently even if the treatment is useless. Partially for this reason, studies generally have a control group, a group that don't receive any treatment or receive a placebo so that the results can be compared against a normal baseline. In this case, because everyone knew what treatment they were receiving, it likely impacted how effective their treatment was. Sometimes, all it takes to improve mood is knowing that you're taking a positive step to fix the problems in your life. That brings hope, which in turn raises your mood. The problem with self-assessment then is that depression is relative. How depressed are you? How depressed is the person next to you? How can we get a meaningful comparison between how two different people feel? Everyone interprets their own feelings in different ways, and while there's a huge gap between zero on this scale, I do not feel sad, and one, I feel sad, there's a much smaller gap between three. I am sad all the time and cannot snap out of it, and four, I am so sad and unhappy that I cannot stand it. Beyond the issue of relative feelings of depression, participants also feel a desire to be helpful, and this skews their responses, whether intentionally or subconsciously. For example, another study found that participants who returned Bex depression inventory scores by post scored themselves higher, more depressed, than those filling them in in person in a laboratory environment. Something about being with a doctor at the end of a trial makes people score themselves lower, less depressed, and we see that in these findings. That almost 50% drop in depression? That's a loaded number. It should not be trusted, or at least not taken at face value. There's more going on here. First off, all groups saw a drop in self-assessment scores, even the control group that had no other treatment. The Mario Odyssey group had a pre-trial BDI score of 100. Afterwards, it was 57. That's a 43% drop. Nice, which is where that almost 50% statistic comes from. Meanwhile, the control group had a pre-trial score of 75. Afterwards, it was 63. That's a 12% drop for doing nothing. And in real terms, the Mario group ended the trial with a score that was just 5% lower than the control group. Note how even to start with the scores weren't even. The Mario group started at 100, and the control group started at 75, so their relative levels of depression weren't on an even footing to begin with. This is a typical symptom of, you guessed it, a small sample size. Then see how much depression dropped even when no additional treatment was given. Simply being involved with the study caused people to report being less depressed. Forget Mario, the real breakthrough of this study is the finding that doing absolutely nothing can relieve the symptoms of depression by 12%. Unless, of course, we go with the simpler explanation. External factors caused many patients to report a change in mood that was unrelated to whether or not someone had played a computer game. And look, I'm doing the YouTube version of a report on this. I'm not giving you all of the specific details on the methodology, or how the study deliberately waited its results to try and trim out some of the obvious bias in said results. I'm not covering everything here. It's very important that you know this, because YouTube absolutely definitely should not be taken as a source of accurate medical knowledge under any circumstances, even from someone like me who has professional experience doing this. Once more with feeling, do not trust YouTube for accurate medical knowledge, even this video. You're not getting the whole picture. I hope that's clear. I'm just covering the points to show how the gaming news websites can't be taken at face value either. Now, let's talk about motivation. This, in my opinion, is perhaps the strongest finding in the study, the result that I feel the most confident in believing without questioning the methodology. Hey, guess what? Participants who got to play Mario reported that they were more motivated to take their medicine, if you can call it that, than participants who had the less fun brain training compact exercises. Mario is more fun than a literal brain exercise. Surprise, surprise. There are things to mention here around how the Mario exercises were conducted. It wasn't free play or anything, but I feel I don't need to add extra nuance to this point because if you've watched this far into this video, you probably also agree that playing Mario is more fun than doing homework. I also imagine that if you've ever played a computer game at the end of a stressful day, you'll appreciate how this cathartic experience can help you relax, help you recharge, help you function a little bit better afterwards. This is, ostensibly, part of what the study set out to explore. And lest you think I'm completely opposed to all of these findings, I want to make it clear. I do indeed agree that playing games can help someone with depression and low mood, whether that be severe enough to require additional treatment, or simply the kind of malaise you get from a rough day. So, that's mood and motivation, and Mario is out on top in a very fast and loose study that lacks plenty of rigor. Last is memory, the actual point of the study, and Mario does not come out on top here. Scientists wanted to know whether playing 3D games improved working memory for people with depression. I'm not going to get into all of the details here, but basically, there were two memory tests, and looking at both of them together, the compact groups saw a statistically significant improvement in both tests, while the Mario group only improved in one of them. The point of the study was to show whether playing 3D games helps with cognitive function for people with depression, and the finding is, yeah, sure, a bit, but it's not better than doing actual brain training exercises. Mario is not a better exercise for memory improvement than existing medical treatments. But as said, these findings are not by any means concrete, because the sample size is too small and the study's methodology is too leaky to hold water when applied to a more general population. And indeed, the point of this study is not to be a solid definitive proof of the power of Mario. It's supposed to serve as the justification for a more far-reaching study that would have a larger sample size and greater scientific rigor. Just to reiterate, this isn't me having a dig at the study. The scientists involved would literally be the first to tell you that this study's findings are highly questionable, because that's exactly what the study's academic paper actually says. Here are some of the quotes from the paper. A limitation of this study is the small sample size which may have reduced the power of the present design to detect effects. Another critical point is the non-blind nature of the intervention regarding participants and the experimenter likewise. Therefore, one has to be cautious when generalising the results. The problem is, this kind of nuanced approach to a scientific study doesn't always come across in a headline on a gaming news website. Reading that playing Mario made depression symptoms drop by nearly 50% doesn't give you the full picture, because, hey, turns out the inner workings of the human brain can't be summed up in a single sentence. Mental health issues are complicated, and the reason this study is even necessary is because there's still a lot that we don't know about how the brain works and how it responds to different stimuli. This study is an excellent first step to gaining better knowledge of how we can treat depression. But if you come away from reading an article, thinking that playing Mario is more effective than taking medication or counselling or any other more scientifically proven treatment for depression, there's a danger that media spin on this study could end up genuinely hurting someone. To that end, the moral of the story is actually far simpler than the headlines might suggest. You can't believe everything you read on the internet.