 Welcome once again, welcome everyone. And please, if you don't speak, mute yourself. And otherwise, I would like to start with a very quick introduction of Bettina Rill, who's our speaker today. Bettina has been working as a patient advocate in the melanoma field for several years now. And before that, she was a medical doctor and a scientific researcher. She also holds a PhD in medicine research. And Bettina has been doing a lot of research and work around the theoretical framework of patient involvement in scientific work and biomedical research. And she also does a lot of very practical stuff with the melanoma community and beyond. And without any further ado, I would suggest that you take it away, Bettina. I start the slides. There you go. And you just let me know whenever you want me to advance the slides. Thank you, perfect. I hope everyone can hear me well now. And I would say we just get started. So what I would like to share with you is about the next hour or so is why I believe that thinking like a scientist is something that is of enormous value to us as patient advocates. And in the next 45 minutes, if you could jump to the next slide, please, I would like to give a brief introduction where we should care about science in the first place. Why is it worth our time? Why thinking like a scientist gives us the power to support our friends with evidence and apply that relevant to our field? As one part of one field of science, because science is obviously a very, very broad arena that is particularly relevant to us is medical science, of course, because we are all in cancer. So we will for sure spend a lot of time today on the medical science. And I will go into detail about how one can actually get started in reading original scientific literature or some medical scientific literature. There will be a very, very, very snapshot, like a very brief introduction to the Esmall magnitude of clinical benefits scale, because I think that is a useful tool for us as advocates. At the time, we're not allowed to really go into depth, but the link will be on there. So I just would like to give you an impression of what it is about. And then, of course, we're going to sum up the next slide, please. If you could move to the next slide, please. Okay. So why should we care? Why should we care about science? Now, there are two aspects of why we should care. One is about individual patients. We all, many of us are patients themselves, or we do help single patients to cope better with their disease, to get better care. So if you really truly want to help people, it is often that it is the latest scientific developments that make the biggest difference for patients, especially when they're in difficult situations. And this is usually when they come to us, when they're out of treatment options, when they tried everything that was out there, or, however, on the opposite side, they've just been diagnosed with a disease and don't know where to go. So understanding what the latest scientific understanding is allows us to give good help and provide a context that helps patients make better decisions because they are able to discuss with their medical team. We should support evidence, not eminence-based science. This is something I actually had the time to be able to add another slide, but I already had too many slides. But this is something that I have actually borrowed from the trial workshop that I teach every summer, where also, junior oncologists are told to place their decisions based on evidence and not on the eminence of other clinicians. True patient empowerment is, in my mind, impossible without certain understanding of the scientific area we are moving in. If we really want to have impact and impact the decisions around us, we have to first understand what is going on. And obviously, it is a protection from quackery and form hopes. And someone in my network said knowledge is protection, and that is something that really resonates with me. So it is important to help a single person. Next slide, please. Next slide. Okay, super. No, no, that's as Advocates. This is perfect, thank you. So now we're here as Advocates. You are in a special course, you're already a part of a particular training program. So we're not just patients, we're Advocates. So why does it matter particularly for us as Advocates? Now, if we want to push for progress in science, and this is the ultimately where our cues will come from, hopefully, one day, it is the medicine that will bring us there. We can only influence what we understand and what we know. The science, how we currently conduct science affects us. Anyone who has ever been on a clinical trial, we know that it is the trial design that defines your experience on the clinical trial. And so most, yeah, I know I've seen from the names that some of you know me already. So you know that clinical trial design is a topic I'm passionate about. So the way a scientist conducts it, so the methodology affects us as patients. We cannot change them if we don't understand what science is about and by which rules it is conducted. It is impossible to provide accurate understanding or accurate health care information to patients without understanding what is going on. And if we are able to support our advocacy plans with evidence, the chances that they will be successful is way harder, a way higher, because it is harder to discard evidence-based claims than if it was just an opinion. So, and apart from that, I believe that as advocates, we have to question ourselves in what we're doing and we have to ensure that what we are doing is really truly in patients' best interest. It is not just our personal experience, because as a single person, we can never, ever be representative. We are just one person. We have a certain view of the world. So we have to kind of, like I think of it as an insurance for ourselves and back insurance, that what we do is truly for the benefit of the community. Next slide, please. So to sum up, scientific thinking is the basis for evidence-based advocacy, and that is why I believe that we have to care. Now, next slide, please. The good news is that you do not have to have a scientific degree or wear a white coat as Ben Golderka in this Guardian article said and he will share the slides afterwards so you have the link to the article and it's definitely a really worthy one. Science is not about authority. It is about the method, and that basically means that it is a matter of being willing and able to learn about the method and it's not about getting a scientific degree in order to be able to follow. Next slide, please. Next slide, please. Perfect. So something that for us is important and that I think is a fundamental kind of way of thinking about people, about knowledge and about learning. This is something I'm very passionate about. Knowledge is infinite and no one needs permission to learn. So it is not about what should patients know. In my mind, it should be what do patients want to know and what is sufficient and what is too much. It's up to the learner to decide. Of course, we as advocates might have an opinion that we know from experience and unidiagnosed patients will fare better if they know this much. But ultimately, what people spend their free time on, that should be up to them. And I think as advocates, it should be our ambition to provide and encourage people because these people will go and talk to their medical teams and they're surrounding in a very different way. They will be able to argue at a very different level and they will be able to enforce their positions to a way higher degree than if we just give them the single leaflet that was spetted by someone else who has never been in their shoes. Something that we have to really, really read, however we be aware of, and I like this Albert Einstein quote from there, everything should be made as simple as possible but not simpler. There is a certain level up to which we have to bring our people because certain things cannot be simplified without losing accuracy. So it is about finding that in that interface to make it as simple as possible. And that is actually extremely hard. Simple is hard. Putting complicated context, complicated content in a simple accessible way is an art. So there's a real challenge for us because our audience tends to be diverse, they come from different backgrounds but at the same time, our audience is highly motivated. Next slide please. I believe that it is what makes us different from many other learning audiences, it's the motivation and the effort that people are willing to put in. In my network is people who are willing to spend their weekends over extended periods of time to learn and we have had clinicians coming to our events. We always have educational sessions and the feedback we always always getting is that they're amazed how well educated the audience is and how evidently they learn. So I think that is something that makes us unique and something we should be proud of and encourage. And this is something that I actually send this slide and other Albert Einstein is something that I send to our faculty. It's like, if you can't explain it simply you don't understand it well enough and that's the challenge I put them under in order to deliver speeches, talks that are meaningful to the audience. This is actually that it's not on the slide but something for you to remember the moment you go into organizing events and trying to share and spread scientific knowledge it will be up to you to moderate the discussion and to breathe your faculty and that is a true challenge because people are usually especially medics and not you can be bossed around by anyone and they usually do worse presentations you can imagine. So a classic example is you give them a 15 minute slot to talk until they send you 45 slides. You know from the beginning that it will never work. I count 1.5 to two minutes per slide which means 15 minutes is 10 slides. So I will cut them and I tell them exactly this is I will cut you after slide 11. Please put all the relevant stuff in the first 11 slide. So this is something to consider it is about knowledge but it is also how we make this knowledge accessible and share it and it will not be possible without putting considerable effort into the educational aspect of knowledge. And to add to the bad news next slide please. It is possible but it does require effort and it is a skill. So without effort without input it will not be possible to sail on the forefront of science. So it is possible, you do need a degree but it does require effort and time to put into it. So without I would like to thank you for being here in the first place because this is a pretty technical and I call it slightly nerdy topic. So I really appreciate your interest at this late time and so close to Christmas where I'm sure we would all be or would like to bake cookies or something similar help. Next slide please. So I would like to now start with some principles of scientific thinking that I think are absolutely crucial and fundamental. For me, this is the essence of modern science and I would like to give you a bit of context why I'm because I believe that this is really, really important. Modern science is based on the principle of falsification, which I call the art of nitpicking. The fundamental, and of course this is kind of a bit tongue in cheek, but I would like you to understand because it is crucial to understand how modern science operates next slide please. It's, oh my God, on the right it says like I blown it up too much. The underlying principle of everything we're doing now and the underlying principle of falsification is the realization that we only ever see a part of the total truth. And you might be aware of this Arabian fairy tale about the wise men who are arguing about who knows the absolute truth. And there's a wise blind man who makes them actually meet an elephant. And usually the original story is that the elephant is in a tent and every one of the other people gets to touch a part of the elephant. So someone as you can see on the right, someone touches like a leg, someone the tail, someone the nose, and everyone comes back and has a totally different story to tell. And that's a bit summarized on the left. That some, like I think the one who touched the leg like compared it to a column, there's someone who touched the tail, compared it to a tree or like a branch or something like this. And in the end, while everyone was right, no one had the full picture, so no one came back and said, I have touched an elephant. So this is a very, very powerful, in my mind, powerful image and something that we should be aware of. Our understanding of the truth is only ever incomplete and partial and we have to treat it as such. Next slide. In the Middle Ages, when there were the first movements towards scientific approaches, what people did were they tried to verify their claims. So verifying means you look for evidence that supports your beliefs, while falsification is just the opposite. You are trying for reasons that that's basically true that you are not wrong. This might seem very similar, but it is the difference between right but not being wrong. The verification part of the story is that people will only consider evidence that supports them and will ignore evidence that basically shows them that they are wrong. We can also call that cherry picking. Falsification, on the other hand, will go looking for evidence to prove that one is wrong. At the point where we no longer find evidence that we are wrong, we assume that we are kind of right for now because tomorrow there might be something coming along that shows us that we were not right or not entirely right or that we got aspects wrong. In a way, it's a kind of insurance for ourselves that we are not overstating what we believe. So when you now, and that's the next bit we will go into, is how to read actually an introduction, how to read scientific literature. It is an approach and it sounds quite negative, but I think it's a healthy critical thinking. It basically means that we are looking for reasons not to believe the authors of anything we're reading. If we cannot find anything, we will go, yeah, that looks kind of all right for now. And that is the basis of modern science, of modern scientific thinking. This is also the difference between a typical quackery or fake like pseudo scientific claim for be it green tea or any type of snake oil. If you go through there, these people usually just, if they cite evidence at all, and usually that is like not really true either, they will only cite support in claims. Never things that disprove them. And that's the difference today between true science and pseudo science. Now, I always say science, next slide please. Science however is a huge term and it bridges a vast majority of different areas. And then two big differences in there. And I really like this quote that not everything that matters can be measured. Not everything that we measure actually matters. And out in a way is the difference between quantitative and qualitative research. Next slide please. Quantitative research, as I said, is anything that can be measured where you can attach numbers to it? Well, qualitative research is non-numbers. If you want it would be things like surveys, interviews, focus groups where you just cry your results and the result is rather a text and not a number. Next slide please. If you start, if I start to think, I started thinking about which fields of research actually purchase as patient advocates. And it's a huge number and I'm sure this list is not complete. If you start very early on, fundamental biological research, basic science is relevant to us. Maybe not the most relevant, but it is relevant to us. The most relevant probably for us is clinical medical research. So these are clinical studies and the publications about clinical studies. However, also epidemiological research. So anything that is about population-based research. How like for example, the frequency of cancer in certain populations, general health issues in populations. This is epidemiology. Have systems research. How well do our health systems perform? What are waiting times for our patients? How long does it take to receive a diagnosis? Equally affects us. Have economic research. I'm sure that everyone of us are now is aware of the economic implications and the challenges, especially in oncology. So we are affected more and more with health economic research outcomes. Sociological research. Anything that comes in the field of behavior, anything that goes more into like how people perceive things is if you want to support a sociological research with its own methodology. Political research is yet another one which all falls under like the big umbrella. And these like sociological and political research, they do a lot of quantitative research. They don't do so much like qualitative or they have a mixture of it. And sometimes what one tries to do is try to quantify aspects that are actually at the base through qualitative. Now, if we want to continue now to the next slide, I would just like to give you a brief introduction on how we started in my own network. So as Tamasha really said at the introduction, I found it on a number of patients in Europe and because I'm a medic by training and then work years as a scientist in the fundamental biological research lab. Science is close to my heart and I've done a lot about it. And as a PhD student, you get tortured at a sense. So how to properly read scientific papers here. That was a bit of motivation or well a bit of background that helped me to design this workshop on how to read scientific papers because this was a demand that came up on our forums all the time. People were asking all the time, how do I get started? How do I do this? Where do I find this paper? So that we decided to run an entire workshop and this was like two and a half days on a single study on a single publication. And we took obviously something that was relevant to us in melanoma and really took it apart. Normally as a PhD student, you will have one hour a week where you just have some paper. But of course what we did is we took it into parts and then provided the background for people to understand what each of the figures meant and to understand the statistics behind them. And it turned out to be a really useful exercise for our network and has helped enormously to raise the level of the discussion on our forum. So this is something that I would highly recommend. And I'm really happy to find that you are under the link below there. You'll find the program of the workshop. So feel free to copy. It's not, you know, it's better for everyone. Now, next slide please. Why would we actually care to review original scientific medical literature? Why go through the pain? Why don't we just rely on the guardian or anyone else to write it up in a more digestible form? For me, there are three aspects of it that are relevant. The first one is a time aspect. Scientific publications are the first thing that appear before anyone shares it, translates it, it goes all over the place. So this also means that this is English. Scientific publications and medicine, the important ones today are in English, whether we like it or not. Also, whether we like it or not, not all physicians are up to date on the latest progress in our specific disease. And that matters especially when it is a rare condition or it is a condition where patients simply don't have time. So if it is a time sensitive issue, you cannot hope or you should not hope for your physicians to be able to speak, you're way better off to read it yourself. The next point is correctness. You cannot rely on public media like the Guardian reporting correctly on medical findings. Often what happens is, and Ben Goldberg has worked on that and published on that, is what happens is that scientific publications often end with a statement about the future, like a forward-looking statement as it is called in any investment context. So it's like the vaguest, most speculative, biggest hope at the end of any paper. It goes something like that and then we're gonna save humanity. And this tends to be unfortunately what the general media picks up and turns into the headline of the report about the paper. So the most speculative and weakest part of a scientific publication turns out to be the headline of a general media report and that of course is an issue as such. In addition, by relaying our lay versions, these often often make crucial details. It does make a huge difference whether a result was found in a person or whether it was a fish or a cell in a dish. You would be surprised what lay summaries often omit. So by reading the original publication, you can protect yourself from that. And the latest one is empowerment and great power comes with great responsibility. Knowing what is going on is part of it and now knowledge is protection, but knowledge is also power. So in order to be truly empowered, we have to understand the science and empowerment in the sense of having an educated communication with your medical treatment team. Only by understanding what is truly going on, you'll be able to negotiate as equals. So it is for me definitely worth investing the time to get there. Now, we will go now in the next part a bit through the practical aspect about how to read actually these papers. Because first before you can read them, you have to find them. Now, where does one find scientific papers? Scientific medical papers. I call it the Google of medical literature because if you want to find a paper, you don't go to Google, you go to PubMed. The link is on there. PubMed is a publicly available database. It's based on Medline, which is a repository, but what we all use is the service. You've got a layer, you've got the search field and actually PubMed has produced extensive how-to guides on their own website. I put the link up there. Please have a look. Better you know how to search the more you're going to find out. But this is the reliable source to go looking for scientific papers. However, next slide please. Even PubMed, which will not give you ads about dubious products or snake oil, which Google will still return for you. Still, even PubMed is only as good as you know what to look for. So in order to be successful in your search, you should look for a specific about three points. Whom you're looking for, what you're looking for and where you're looking for. And I'm going to go through each of them now in turns. Next slide please. So the first one is whom are you looking for? If you look for scientific publications, you will find the authors of that publication listed straight under the title. I should have potentially a screenshot here. Medical research has a code how articles, like how authorship works for articles. How it usually works is that the one who did most of the work is the first author. The last author is usually the one who runs the group and who got in the funding for it all. In different stages of your career in clinical research or medical research, you need different things. If you do your PhD, you need first-author publication. If you want to show that you're an established researcher and able to run a lab, you'll need last-author publication. So you get a feeling for where people are in their career and how important they are by looking at the authorship position on a paper. Now, which name do you want to put in PubMed? The first one I would recommend is you look for the established authorities in your research. If you don't know who those are, go to the conference program that is most relevant to your cancer. So you can go to ASCO, ESMO, ECHO, or I think for you, for in myeloma, ASH is probably even more important for myeloma SMR would be an interesting one. Go through the program. Don't just take one, take two conferences, three conferences, and you would see like conferences from different years and you would see that the names start repeating themselves. Usually the groups are not that large. You maybe end up with 10, 15 names if at all. Take one of them, start looking. If you want to do a quality check on your treating oncologist with that in department as well, then you know where they end up, they're in the hacking order. It doesn't mean that the ones who don't publish are bad, it just means that they're not, they don't have the influence because it is in the end by a publication that people are influenced on what is considered the best standard of care. Now, you have found that person and this is the person you wanna put into your PubMed search. Now, still, lots of these people publish lots of different things. Next slide, please. So you wanna look for what and something very specific. What you wanna look for are the big unanswered questions and also going through the conference programs, as I said in the earlier one, we'll help you identify those if you don't know that already because usually we don't start reading medical papers because it's so much fun. It is because we need answers to a very concrete, very pressing situation. So for us, and I know I'm gonna give you an example, it would be resistance after immune therapy. Patients who do not respond to immune therapy become desperate and the next options are not so clear. So this would be an area where we are really, looking for answers. Also too, if you still think they find it hard to define, it is very, very, it's worth going to medical meetings and see whether they don't have bookmarks or articles about heart meetings because they usually pick up the highlights of the conference which would always be a reflection of the big unanswered questions and the most relevant questions. Now, for a standard stream, this is like for the big new unanswered questions. If you are like in a scenario where there are already a standard treatment, guidelines are really good source of information and you can go to the societies of the disease, like ASCO, like ASCO or like hematological societies that will have published guidelines. Those would be a useful place to look up once the treatment has been established. In order to make it into a guideline, however, there needs to be enough evidence of a certain treatment. So this does not help you if you have a reason or like what I call the edge program. So the really challenging bits usually don't have enough evidence that it may be a guideline. Has someone put the, there's a background noise? Is that okay? Tamash? Yeah, I can hear the background noise but I don't know where it's coming from, so. How it's gone? Yeah, I don't know what it was, I'm sorry. Just once again, please, everybody mute yourselves if you're not speaking. Everybody but Bettina should mute themselves. Perhaps the two people are not muted. So. It's better now. Yeah, much better. Thank you for that. And now the next thing is what I call where next slide please. PubMed is already a really great repository. It's not as bad as Google but even in PubMed you find endless papers and it is neither recommended nor possible nor even advisable to read all of them. It's a typical beginner's mistake for PhD students and I can tell you I read a lot of rubbish when I started. If you don't wanna go through the pain of doing that I think it is a smart approach to start with the highly regarded established journals and in medicine that would be the New England Journal of Medicine and the Lancet. Basic research you wanna find in nature or science and then there will be usually respected the electricity specific journals like blood for hematological diseases. We have some Malanoma journals that where we would go and what is constantly highly regarded. If you limit now you put in the person who believe is relevant in your field you have found the topics you care about then start with the highest ranking publications and start reading them. What the help stand is once you have found a paper there will be references to other publications and then it is wise to follow those references and basically build a tree of knowledge if you want. Now I said highly regarded and how do you kind of know what is highly regarded or not. Next slide please. There's a way how we can measure everything and even remembering that I said not everything we measure matters and vice versa there actually is a factor that is supposed to measure the quality of a scientific journal and this is called the impact factor. Now it is a combined factor between how like it's supposed to measure or capture the quality of a journal and it is based on how often how often articles published in this journal get cited by others and how long that happens. There's something that is just hot for like a month or two and then disappears is not as important as something that will be cited for 20 years. Now there's a huge ongoing debate and there's a link I made you can read up on it. There's a huge ongoing debate whether impact factors are the right thing or the wrong thing and you always hear this kind of oh this is not important we shouldn't be measuring this but then the next thing is that today scientific careers are made on impact factor publications. So for professorship in Germany one nature paper will be sufficient. If you wanna go to Harvard or other places you need more than one nature paper. So while there are discussions around it the thing is so the impact factor is not without yeah without issues it is still a good rule of thumb to find things like journals that are trustworthy and discard ones that are less trustworthy. So just for I think something like Lancet or New England are now in the order of 50. So if you have a journal that has a 1.5 you know that it is probably not worth your time. So that is more if you wanna read in addition to New England, Lancet, Nature Science and your Journal of Interest. So if you are really not sure you've read something it sounds too good to be true and it's in like a low impact journal it is wise to be cautious. It's also I think a good exercise to generally look at publications in publications in low impact journals and then compare them with high impact journals probably publications and then see what the typists look like because you will find that oddly enough it is in the lower impact journals that they will promise you to save the world while high impact implications often have very modest rather boring sounding titles because they're way more specific, way more well they're concrete in what they claim and that is why they're high impact in the end. So if it sounds too good to be true with insurances it most likely isn't true either. So a word of warning to that. Next slide please. Now once you found this article and you now know that you want to read this unfortunately what happens now is that often these articles are not just free for download so you just don't click on the link and you have the full thing. Unfortunately many of our high ranking journals as they pay world so you pay for access to the journal. That might not look fair to you but unfortunately this is still the way it is. Now there are always ways around any access issue also to papers. We listed some the open access button is a bit of a pain but you can like if you look up the link it will explain how it works it is a way to get your hands on a paper. I actually prefer I think it's easier if you look up the first author usually the affiliation of the author is listed there together with an email address. Drop them an e-mail say you read you would love to read their article because and ask them to send you a PDF that usually works. They tend to be flattered about the interest you usually get the article very very fast. Another way would be research gauge you can register your organization under research paper free and then you can upload your own publications that you will hopefully produce in your network and also ask authors about a copy of their paper. Authors can upload copies of their paper for free and anyone else can then download them for free. These even the paywall journals give authors the right to share an earlier version of the published article for free so that you will always get your hands on. Usually the only difference between that and the final published article is that the layout is not perfect yet and it doesn't look as pretty but the text is the same. Some journals offer free access for patients like patient access. Read up on there and then you can only have to sign up and then you get the papers for free. It's also worth to have a good connection to your oncologist and just send them an email. I do a lot of that. They often have access to other institutions where the university pays for access and they get the PDFs and can send them to you which tends to be the easiest way to get access. At the same time it's also a nice hint that they know what you're reading and might read it themselves so that could have a double function. Now let's assume you have your PDF in front of you. Now we go about how to actually read it and read it that you get most out of it because I assume we all read well enough. It's not about just reading it. It is about dragging out out of this entire information what truly, truly, truly matters. Now the most important point of any scientific paper is the results section. The results in a medical paper are usually presented in figures. So you should focus your attention on the figures and the results section before doing anything else. Because if you start with the conclusions a typical beginner's mistake is to read the abstract whether that you have to do in order to know what this is about, the introduction and then the summary and the discussion. That's the point where you fall for the author's narrative. And not everything that authors claim is also correct because they obviously have a vested interest to look as good as possible. So if you don't want to fall for that you have to make up your own mind first before reading what they think and then you will see whether you agree or not. And you might well find that you missed important bits and you got it wrong but then in itself it has been a valuable exercise. And often it actually happens that we do not entirely agree with the authors and that's where we have these valuable points in progress. So it's important not to be blinded to the beliefs of others. Now, next slide please. Scientific papers are not random pieces of literature. A scientific paper has what I think of an anatomy, a backbone and it will always be structured the same way. There's slight variations between journals but as a base you can say that the paper has an abstract which is the short bit of short paragraph summarizing everything that's in the paper. An introduction that gives you the background and the motivation for the study. Materials and methods that describe is like how the results were generated basically the cooking recipe. The results section which I already said was the most important point that's the meat of the paper. Then there's a discussion, what the results mean according to the authors and a conclusion. Materials and methods are the section that can move around. Some papers have both materials and methods to the very end because it does break the reading flow. If you're just like after reading the introduction I'm really excited about this material like these results or this story and then the first thing you start reading is and then we use five microliters of this or that that is not very much. So materials and methods are often now the last final section. But this is usually what differs. Some papers also have like a discussion conclusion as in one section but the base is always the same. Next slide please. Now we will just one word of warning. Results are important and especially in medicines these are statistical results. There's always this quip among scientists. Don't trust statistics you happen to fake yourself. In order to really truly understand whether or not you can trust statistical results in any given paper you have to understand how they were generated. So while materials and methods might not be found to read there are actually a crucial part of any scientific publication because only if the methods that were used in order to generate the data you were looking at the results can be valid. A wrong methodology will mess up the results and then you will not be able to trust the results. So the devil is in the detail materials and methods are dry reading material but there are important. And I always joke that having a statistician as a friend is a very, very useful. It's worth a lot. Having someone who can help you go through the statistics and explain them is really, really helpful especially for people like us. And we have now started having regular statistics workshops at our conference for example. Now my personal recipe would be the next slide please. This is how I tend to read the paper. I start with the extra what is this about. Then I go through the figures to see what the data actually really looks like and can I really understand from the figures what this is supposed to be all about? And if I don't, that's already not a good sign because usually people put a lot of effort into the figures and the figures will tell you what the bones or what the meat is on those. Then I go through the results and see whether there's something I missed in the figures and whether I have already actually seen everything. If anything in there is unclear, I will go to the methods and the materials and see how it was generated and whether this can explain to me why things looked the way they looked. If there's something I wasn't clear or that I still wonder about, I will then go back and read the introduction. Sometimes I jump back in the middle because you might look at the figure and go, why did they look at this? And then the introduction will hopefully provide you with a reason why they did that. Then only I will read the discussion and the conclusion and then come to the conclusion whether I agree or I do not agree. And then, as I said earlier on, I was sigh at the last sentence and I will worry that this will be the one that will end up at the next general media or mass media publication and which patients will then fall for. Now, next slide please. One quick word and we don't have the time to really go into detail about this is the ESMO MCBS on Magnitude of Clinical Benefit Scale. I put it here because I believe that it is actually a very useful tool to help systematically about scientific publications and this is a very specific and important trial. Funding the mayor where there's a lot of material on the ESMO website about why this scale was constructed and I will briefly give you an introduction what it is and how it was and why I think it should be already detailed. The Magnitude of Clinical Benefit Scale is a way to evaluate findings from clinical trials. We have the background on again. Did someone unmute themselves? No, actually not. I don't understand what this might be, but it's gone again. Okay, nice. So what this scale is is basically it looks only at very hard data endpoints. So every clinical trial that runs has an endpoint, a primary endpoint and a secondary and there can be several secondary endpoints. The primary endpoint is basically the reason why we're running this clinical trial in the first place and a hard endpoint would be overall survival because that you can measure without any doubt something like progression-free survival which is already a bit trickier and severe toxicity, which is great for toxicity for anyone who's familiar with the toxicity grade scale. What this scale does is it puts it into a systematic system and then ends up with a grading that tells you whether this is a very good drug, a movement drug or rubbish drug. You do not have to agree with a tool but I believe the thing is that any tool is only as good as it is appropriately applied. You will only use a hammer for certain things but not others and you should think of this scale in the same way. What it does help, it gives you a framework how to look at a paper and unfortunately I don't have the sheet here and they have made kind of a helper sheet where they can, that helps you to go through the paper and it's very specifically asked you, what is the primary endpoint? What is the difference between the new treatment and the old treatment? And you do this very, very systematically. If you then look at two different treatments, you end up with two recommendations and that then allows you to say this is better than that and that is the full point of the exercise. And while it is not easy to get started in it, I believe it is worth the time reading up on it because we're definitely gonna do this at our next annual conference. We're gonna go through the melanoma papers that are relevant to us and try to understand whether we, well first to understand what they have done and then see whether we agree or not. A word of warning because you're in a hematological disease, this scale has been designed for solitumers, so not for hematological tumours. But it is now, EHA is now currently adapting it for hematological diseases and they have done first pilots and it also seems to perform quite well there. I know that people from us are involved in there. I think Jan is involved in there. So please check out what they're doing and then use the appropriate one for your disease area. So that's another reason why I don't wanna waste too much time on this now, but please look out for it. It helps to go systematically through different studies and it gives you in addition to that, an argument if you find that something that is ranked very highly on such a scale and for example, it's not available in your country for reimbursement. This is the perfect bit of borrowed authority that can help you to support your claims in terms of evidence-based advocacy. Now, I would like to, we're coming to slowly coming to an end. I would like to summarize that a bit and have a few words on how to get started because I'm aware that that's the daunting or the most daunting bit. Next slide please. You just have to start. It's about start reading the original publications in your specific field of disease. Don't be too worried if the first time is our heart, it does get easier with practice and usually when I started like in the research, I started in youth field, I would spend like a week or two do nothing but reading. Of course I was lucky that I had access to all these publications, but I would start with one point and then would go through the references, mark them, then look all of them up, reading them and continue that until I had read a few kilos of paper. I personally quite like and I must admit that I like them in hard copies so I still do print out my favorites, at least the relevant ones and I mark the bits that are important and I put notes on them and I keep them. I have a folder for the really important things. I have a folder like I'm resistant to targeted therapy because that's something I was interested in and I keep them and I go back to them. If you just throw them away or you just read them online without annotation, you will have to read the same thing over and over again. If you go back to see your own notes and you have a structure in place that will really, really save your time and your own. It does help to find like-minded group of people for discussions. More I see more, so it is helpful to discuss the paper and see what other people believe. Form a journal club. That's actually an established concept for anyone in science. As I said earlier, we had one every week. It was an hour every week where you get together to discuss one piece of like one paper. Everyone has to prepare for it and then you have a round of discussion around this paper. So it might be an idea to start a journal club in your advocacy group. You also want to think about finding the filing system very, very early on because we're totally guilty of it ourselves. I had a professional filing system at university and we didn't have that. We just saved them in a folder. It's already a shared folder between several ones of us so anyone who wants access can have access, but we haven't saved the papers clearly so they're very difficult to find back. So which means you have to open lots of different papers and it's a pain. So it's worth to think about some filing system either investing in the library program and these are like databases for literature. Some of them are quite nice. We're currently investigating a few ones. I have something I'm happy to share that but it's definitely something you want to consider and ideally something that you can also share in your community that everyone else can benefit and profit from the knowledge you have recorded. As a general rule, I believe it is very, very important to curate your patient forums. Patient forums should not just be a group like a forum for support where people send each other hugs and hold thumbs or hold their fingers when scanning time is coming up. We have started establishing on our forums a very, very strict rule that any claim with regard to health, especially when it's about new drugs needs to be supported by evidence. That doesn't mean that we say, you know it has to be the clinical study but we want people to put where they got it from. If it is not coming from a source that we identified as reliable as specified earlier on but let's say it is the daily mirror we use that as an example or as an opportunity to explain why this is not reliable health information. I would warn against just kicking these things off a forum because that means that people will just leave your group and will go somewhere else where you will have no influence whatsoever. So you have to be kind but it is good to have clear rules on a forum and we have seen that these rules are now spreading from our central forum to other groups that copy it and people then come back because they know that this is a trusted like this is a place where you get trustworthy information. And I think that's the most, the single most important thing you can do to create a better scientific understanding in your community and keep people safe from false information and sort of science. Start with a pinning your advocacy claims with data, however simple, however basic get it from somewhere else, borrow it but start thinking about any claim in terms of what do I need in terms of evidence so that people cannot refute what I want. And then I think last but not least I would like to warn you that in any type that also research, there's the good, the bad and the ugly. So it is about understanding what good research looks like. What is good in medical research is not good in psychological research or political research but there are measures of what good clinical research looks like and what a good qualitative research should look like. For example, just to give you a concrete example if you have surveys, surveys have to look a certain way in order to have questions that are not leading because if you ask in question that basically pushes people already in the direction of a certain answer you have a bias here, a survey on what you're getting out is usually a confirmation of what you wanted to hear but it's not valid science. So it is important to spend some time thinking about quality control. Next slide please, two useful tools things that we are using. On the left side, this is a Google form. On the right side, this is serving monkey towards on that from our own experience and that's not representative and you might have very, very other experiences. We've used Google forms, they're free. They integrate very well with any Google drive and if you have shared Google drives with your input in your organization this is like seamless integration. They're quite pretty. You can choose the photo, you can choose the background. You can have them integrated into the email so you send them out and everyone has this in their email so it's nearly impossible to miss. However, I have limited data analysis and we have been using that. As you can see now, I use a concrete example. This is just a workshop. We just ran like three, two, three weeks ago in through the bed and people had to choose between projects and we found that actually it was not able to give us correct numbers for the people who had chosen each given workshop. So I would be careful with that. On the right side, this is our survey monkey. We call them monkey surveys actually and we use them for nearly anything because it turned out that this really helps to streamline around conferences. So we use this for anything like applications for conferences, registrations for conferences but also researchers. So we're using them actually quite a lot. However, as you can already see it's not quite as pretty as the Google form. You have also have to pay for relevant features. For example, in order to be able to export the data. So get the data out from the survey and what we use a lot is export as extra cheats. You have to pay for it. However, it has very powerful functions. You can easily, for example, set up anyone who has like a reminder image or everyone who hasn't filled in the survey and the analysis are really good and useful. And I forgot to add this. You can get a charity discount which is really, really interesting. So before you sign up and pay make sure that you have found out about the discount and applied it. Now, next slide please. I would like to summarize that we as advocates have to be aware that argumentation based on evidence is extremely powerful. And that for me is the single most important reason for us to get engaged. Scientific thinking is not got given. It's not bestowed upon us. It is learned and requires training and training makes perfect. One note, the MCBS is a useful tool for a systematic way of thinking and judgment. It's not the end of all of our worries but it's a good starting point to get into judging publications. With that last slide, I would like to thank you for your attention at this late time of the evening. And then now we have about half an hour left for questions and I'm happy to take questions. Thank you, Bettina. Thank you very much. I don't see any questions here in the list but now is the time for you to raise your hands or to type your questions into the chat box on the bottom of your control panel. And while you do that, I would like to ask a couple of questions to you, Bettina, if you don't mind. Some of them came up during your presentation and some others are ones that I have been parking for you for a while. And one of those is that you set about the paywall problem so that it's quite difficult to access papers which are behind the paywall or in any protected system. I wonder if you have ever tried working with any of the open science initiatives like what the frontiers publishing house is doing now that they give free access to all papers published in their journals. Do you think that it makes sense for us patient advocates to approach these new initiatives that are about open science? Listen, I have my daughter sitting in front of the door and it's really cold here, it's minus five. I have to let her in, I'll be back in two minutes. Go ahead, go ahead, sure, yes. I'd like to have questions, I'd be back. Yeah. While we are waiting for Bettina to return, please go ahead and add your questions or just raise your hands if there's anything that you would like to ask from either Bettina or myself or also we have Anna Vajejo from ILMO Patients Europe. So if you would have any questions to her, then don't hold yourselves back and just ask those questions and go ahead. But then I will have some other points or some other questions that I would want to bring up with Bettina when she's back. Hello everyone, I'm back. Sorry for that. Hello, no problem. So you have a question? Are they open science questions? Yeah. Okay, so they're two sides to this story. We've been generally advocating in VINBO Collect, they're on Twitter about open access and supportive that. At the same time, the reason we got into it is to help our patients to argue with their medical team to get the treatment they need. For that, I go where the knowledge is and we don't have the time to fix the world before we can give our people the answer they need. So it's a bit about having a pragmatic approach to go and find this access to stuff that you need today. Why at the same time taking a more long-term approach? For example, in research projects, we have now stipulated that if we engage in a research project, the publications afterwards have to be openly accessible. So that's something to do as a, if you as a patient are engaged in a research collaboration, stipulate that the results have to be open access. That works. So now my other question is that how do you handle at least in your own communities or community, how do you handle this problem that scientific questions or inputs from patient representatives or patient communities are often looked down as something less valuable or less meaningful than what is done by so-called proper researchers? Have you ever been confronted with this issue? I mean, not really. I mean, just like they're different level states. So first I'm personally reviewing research protocols for others. So they send it to me because yeah, well, they know by now what they get back. The next thing is that especially in the Netherlands, foundations now need the support of a patient organization in order to get funding. And we have now started using that as a tool of basically gaining influence. If we don't agree with the project, no support, that's it. And we've gotten pretty like, I mean, at the beginning, it happened to me as well that people would send you like, could you support our research project and then send you a pre-written letter of support, but not the project. It's like, you must be kidding me, right? I mean, I must support a project I've never written. I've never read it. I have also worked as reviewer and that was anonymously. And then I got actually a thank you for the constructive remarks. I'm not sure I would have gotten them hit the no go home course. But you know, I think there's a lot of about us. I think we shouldn't spend so much time worrying about it. I think we should work on our quality and then just go with it and be like cutthroat. So if you have no one prevents you to publish comments on papers on the websites, for example, make use of it. Shouldn't be stupid, of course, do your own work. But once you have reputation to your name, I think that's totally fine. So I do this now with Tini. It takes time, unfortunately, and you have to keep yourself to do it. But I think that's the way to go. Some people are actually looking for patient reviewers. Yeah, I mean, building reputation is something. I mean, this is what you've been speaking about in terms of systematic and scientific thinking and structuring your work is actually, this is also the way how you build your reputation for any future work. At least that's my understanding. But tell me one more thing. What about unpublished negative results? Because we get a lot of questions about, okay, so there was this trial and we know that it failed, but the results are never published. How can we get access to, because this can also be quite relevant for future understanding of whatever should not be done, actually, in clinical trials. Have you found any way around this? So what's your view on that point? So I actually have two points on this. So first one, something that people are not aware is that to publish negative results costs as much effort as publishing positive results. People have this naive idea that they have result and then it's kind of published. And that's unfortunately simply not true. Writing up a publication is huge effort in addition to having produced the results. So they're having the results sitting in front of you is even not a third of the publication. It's a huge effort to write a publication. Publishing negative stuff takes you as much time as publishing positive stuff, but publishing negative results will never go into high impact journals unless you flop the huge study. It will never go high. So if you have a pile of positive results and a pile of negative results and you know that your next 10 year review is underway, what are you gonna do? You're gonna publish the positive result because that will go in a higher impact journal and you will keep your job. So that's a problem we have to think about when we just argue we wanna see everything published. Point two being in my opinion, if we conduct research on humans, the publication should not be optional. A publication of any research conducted in humans should be published end of story, whatever the result that should become like a compulsory like just coming from the fact that people have been exposed to research and potentially harmed. And the least we can do is to publish it. So these are the two points I would like to share. So in this case, we don't have to insist on high impact journals, but we have to insist on getting the data published somehow. Yeah, and of course we can whine about how Lented in New England is not accepting negative results. There's enough about out there, but I think that's a bit pragmatic, right? I mean, I wanna keep our people safe and we can find progress as fast as possible. If we fix society and humanity and understand at the same time, fine, but let's be realistic. It's not how the system works because the incentives for people to keep their jobs are different. Yes, no, but I agree. I mean, and I'm happy to hear that you also represent this point that no matter what the results are, as long as it impacts the lives of people, it should be published. And then we can of course argue about where and how it should be published, but it should be published. Now, also the other thing is, I wonder how you structure this time wise because often we find that at least in my field where I work in HIV and AIDS and also to some extent in hepatitis, people simply get exhausted and energy depletes and then they don't join the journal club anymore and then they don't want to talk about it. Also, there's oftentimes policy matters take over. So we just suddenly find ourselves in the middle of yet another access and policy discussion and science is somehow pushed into the background. How can you keep this scientific engagement and interest on the right level? No, I won't read until the essay said science is like, I mean, science is just an overarching area, right? I mean, we have contact with different areas of science and my background is obviously medical and biological. So that's where I'm at home. At the same time, we should run policy based on evidence. We shouldn't run policy because it kind of feels right today because our feelings might change tomorrow. And even when we support policy, it should be evidence-based and there are actually no think tanks out working on evidence-based policy. So I think that is for us to say really actually, I think it's like for our own network. This is a very important new area to get into. Health systems research, for example, is very, very powerful. If you can measure how long it takes to get a patient from diagnosis to treatment and you can correlate that with outcomes, that is very, very hard to refuse. If you just go there and say, oh, this is all taking too long, how likely are they going to listen to you? So I think this is like, we should look at it as an ally for us, a powerful tool for us. This is at least how I see it. And this is also science. A different type of denavid is that it runs differently, but it is the science and it still operates according to classification principles. It's not the basic understanding is the same. It's just a different area, a different methodology. Now, actually, of course, you also know when most of us on this webinar know about you, Patty, and this is one of the efforts of one of the initiatives that try to make sure that there's patient education and the higher, more professional level of patient involvement. Are you aware of any other similar initiatives that could be that you recommend that you could perhaps look at in terms of education or educating ourselves? Well, I actually don't think that there's anything of the scale that your party was. I think that's a unique and one-off and maybe there are country-specific networks. I met someone from, I don't even know how to pronounce it properly, I posted it, like the Irish, like an Irish person. Yeah, not the E, but the I. So the Irish one, not the European one. And they seem to have a really quite impressive program in place also about educating their community. But I think the thing about science is you can't teach it. It's not like, it's very, our interests are very disease-specific, very focused on our own problems, our own disease, our own, even within our disease, they're different issues. And it requires a lot of personal skills. So you cannot just say, you can't put people in a room and talk at them for a day and then they will know how to do science. That's not how it works. So I think we have to find a different format, like encouraging these journal clubs and let it become the standard of, our standard of interaction, our standard of argumentation. Because what I often find is that people think they become advocates and they go like, I haven't studied this. And they are highly still professionals and they go like, oh my God, I have no idea. And they think that anything then that was useless. I mean, if you look at our audiences, we people who do lots of statistics for their jobs all the time to think that they use this is just stupid. So I think we have to kind of, I think you see it as a culture change. We can help people and we can establish certain kind of, you know, like forums, like journal clubs or formats and share formats that work and keep doing that. And I think that will increase awareness to do too much higher degree. And I mean, as you know that, it's like in our, in my own network, my ambition is like, we want top education for our people and we want them like technical education at a level that really moves the needle. I alluded to that earlier on. I think that's above all a matter of how we educate about scientific education. How do we make science accessible to people who haven't studied biology or medicine? So I think this is more how I think about it right now. Okay, so one last round, if anyone has any questions that you would like to ask Bettina or anyone else from the MPE team, then it's your turn now. And I will count back from three, like three, two, one. And if nobody else has any questions anymore, then I would like to thank you very much, especially you Bettina for the time and the effort. And it was fascinating as always. And please remember that the webinar has been recorded and will be uploaded to the MPE website. And the MPE YouTube channel. So if you go to the Melanoma Patient Europe website, then you can watch the webinar within a couple of days. It's going to be there in an edited version. And thank you very much for your attention and see you real soon. And Merry Christmas, Happy New Year everyone. Thank you. Thank you. Bye bye. Thank you everyone. Goodbye. Bye. Bye.