 So gyddoch chi'n wrthno'n gweithio ar gael ac yn rai cwrs, yw'r tyfu yna yma, mae'n ymddangos i'r hwn ar y hwn hefyd y dyfodol o dylunio yma ar y hwn mae, ac yn ymddangos i'n ddylunio ar gwn hefyd. Mor fwy rôl efallai â'r ddweud o'r llwyddon a oedd ers i hyn i'r Marshal yn y cyfnod, a'r David Lewis, maen nhw'nichi'n chennau ein chymryd ymddangos arond oedd ydy hwn i fynd i'n ddwy'r Williamson. So, rwy'n ffollow up, I'm sure what's done today in those discussions that you have very regularly. Welcome to this workshop. You were at a very key point in your two year programme, and I'm delighted this evening that we're able to have Professor John West Burnham with us. First of all West Burnham works closely with us in the School of Education, and you will know him, I'm sure, through his range of writing, and I won't test you on that, and I'm sure nobody will have read all of it, but a significant body of writing that we use in many of our seminars, and his professional prowess in the field of education, in particular leading innovative practice, innovation and change in the context of ethical and moral practice. And so it's very much appropriate that this evening he guides us through works with us in this workshop. The purpose of which is to provide a springboard for you for your knowledge and understanding of the requirements of undertaking qualitative research, research inquiry in an educational setting as a practitioner. And so I'm sure this evening there'll be an opportunity for you to ask questions, to explore points, issues, things perhaps at the moment that you are beginning to clarify in your own mind. I know many of you are at the stage of having a research area in mind, but not necessarily a clear focus yet in the form of a question, or perhaps having worked through your methodological issues, etc. So we will be recording this so that this can be shared with colleagues who couldn't be here this evening. That means that when you are asking questions or making comments, your voices may be recorded, so if for any reason that's a problem for you, please let me know. And so the workshop will end at 7 o'clock, and over to you, John. Thank you very much indeed. Good evening. Thank you. For those of you who didn't say anything to me, hello, and it's a pleasure to be with you. Now, what we're going to do is really have an extended conversation. And at any time you're welcome to question, to interrogate, to explore possibilities, and crucially where appropriate, please do provide examples if you've been involved in this area of academic activity before. And all we can do this evening, really, is to map out certain basic aspects of the territory that you're moving into. So it's not definitive, and it really is essential that you engage with your supervisors, with your tutors on this area, because it really is a partnership in terms of producing good quality research. Now, in essence what I want to try and do, as I say, is to give you, if you like, the headlines, and then to start the process of thinking through. And I can't stress just how important it is to keep going back to these issues, to keep visiting them, to keep testing and checking, because for some of you it may be that you've never, or it's been a long time since you've written this many words, and you're doing so under severe professional pressure, and this must be done in such a way that you build in your own capacity and sustainability. And it really is important that you are very, very clear and confident as you go through the various stages of your research to know exactly what's going on and feeling controlled with the process. And I can't, as I say, overemphasise that. So I'm going to take you through a series of stages which are fairly logical, I hope, and which hopefully you will be able to relate to your own thinking and experience. I'll pause quite regularly so you have a chance to have a conversation with your neighbours in order to clarify and just make it personal to yourselves, and as I say, don't hesitate to join in. So the first thing, and it does sound painfully obvious this, and I think quite a few of these issues are very, very basic, but at the same time that successful completion of any kind of research activity does have to start with this basic notion of why am I doing this. And that I think is fundamental to the shape of your research, to the sustainability of your engagement with your research, and also in really making clear the shape and nature of the final research that you produce. So is it, for example, as I've said on the sheet, is it about your contribution to your school, your institution's improvement? And I would guess that, for many of you, that's really at the heart. This is a way of contributing towards school improvement, and it's probably one of the most underused and potentially most powerful ways. But if it is about school improvement, then it does need to be very clear and very robust in terms of how this is going to work and so on. It may be that you are engaged in some kind of evaluative project, and that again has got enormous potential power and use. It may be that you're simply seeing it as an opportunity to extend your own personal knowledge. But what I would suggest to you, ladies and gentlemen, is that if you do not have a very clear, very focused and very robust reason as to why you are doing this, then it's going to be very hard work indeed. And you also need to see the relationship between your motivation and the outcomes that you're going to be producing. So, therefore, the big issue is why are you doing this? If it's to get a degree, that's fine, but there might be another dimension to the process. And also, it is very, very important, and I can't stress this too strongly, that your motivation is going to sustain you through the process. Basically, your topic's got to have legs. It's got to actually carry you through. And I think there are so many people who start out, this is a brilliant idea. It looks really interesting, and then as you unpack it, then either it hasn't got the potential that you hope for, or alternatively it simply doesn't engage you enough. And so you do need to be pragmatic. You do need to be realistic and say, does this have the potential to really sustain my motivation and engagement right the way through at the end? And pivotly, is it a topic that really can be sustained? Does it have enough potential capacity? So, your understanding of why you're doing this is absolutely fundamental, and also the need for your own personal engagement and commitment to be very robust. If you get that sorted, then crucially, and I think this is so fundamental that, again, it can get lost, is what's the context of your study, please? And this is essentially the biography. This is the contextual description. This is where we are coming from. Whatever your study is, because it's going to be essentially empirically based, therefore we need to know the variables that are significant and interesting as far as your study is concerned. And you need to recognise the fact that the assessment of your project will be a significant extent driven by the internal logical coherence of your study. And therefore the contextual issues are really important. Now it's not a matter of painting a huge portrait of your school or institution. It is, is to say, this is the context in which my work is taking place, and then you provide the information that informs the reader. And in many ways this is just courtesy to the reader, but it's also to help you justify the particular choices that you've made and the particular issues that you're focusing in on. And then I think it's something that we often don't really have enough of, which is clarity and clear explanation. This is why this is significant. This is why it's important. And I always want at the end of some kind of introduction saying, I have chosen this topic because of the context I'm working in and because this is something that is perceived to be very important within that context. And that again, it sounds embarrassingly straightforward, but in reality it is something that is not often made explicit enough. And then thirdly, and here's the big one. This is the one that really is pivotal in every sense, which is what are you looking at in terms of the big picture, the generic issue, and then how are you going to focus it down. So for example, if your study is looking at the whole notion of school improvement, then as you know, obviously, one of the key driving forces in school improvement at the moment is this notion of closing the gap. And that notion of closing the gap, so you have school improvement as the major contextual issue, you have closing the gap as a more specific manifestation of that, and then again within the context within this country, you have the focus on teaching and learning as the key practical manifestation of school improvement, closing the gap, teaching and learning. And in that way you begin to show how your chosen topic has a clear contextual focus and you are identifying very clearly where it resides in order to allow you to be very confident about all the elements that you need to cover in your study. Is that example reasonably coherent? Just nod or smile in my direction or if it's not, shout at me. You're okay. And therefore it might be something like closing the gap. I mean hopefully, and this is the area where there's the greatest impact of research on schools, often it's on classroom practice. And therefore one of the most challenging areas, but one of the most rewarding areas potentially is classroom practice. It's an enormous area. And therefore a former colleague of mine said that there are three basic rules about successful research at Master's level. The first issue is to have a clear focus, the second issue is to have an even clearer focus, and the third factor is to be really focused. And therefore to talk about classroom practice just isn't enough, it will not sustain you. You have to get into the detail, you have to get into the specifics, and you have to test out the feasibility of your studies, don't you? You have to make sure that this is going to work in a way that really does, as I say, carry you through at an appropriate level of academic sophistication and professional relevance. And getting that balance right is crucial all the way through. And then the third example I've given you there, change, innovation, it's massive. So it is so big it's almost impossible to contemplate. But most of your schools, most of your institutions are going through varying degrees of very significant change. In fact, it's probably true to say that the English education system in every aspect is probably going through more change at the moment than has ever been seen before. Therefore, that's too big. You know, it is too much to contemplate. And it has to be focused down. And that focusing down is in response to the contextual issues identified in my second component. So it really is important from the outset that you avoid being too general. You begin to clarify and you identify the very clear and very real issues that are of most significance and most likely, as I say, to produce useful and relevant outcomes. Good evening, ma'am. And innovative choice of classroom practice. Absolutely. Because it's always going to be permutations. I mean, those three there are purely for illustrative purposes. But in many ways, what you have to do is to say, right, what's my core theme, i.e. classroom practice? And then what aspect of classroom practice am I going to investigate? So it might be, for example, impact on pupil achievement. Or it might be helping colleagues to change that classroom practice. Now, they're both related to classroom practice and they're both equally valid and significant. But they're going to take you down very different routes. And that's really helpful, thank you, because it really does highlight the notion that you can have one generic topic, but then identifying your specific topic is the really important part of this. And then people go astray because it's too big and it's too much to handle. And so having a range of permutations around varying themes and how they interact. So, for example, if we take classroom practice, then one issue is does this approach help to enhance pupil achievement? Another one is what is the problem of getting old and crotchety members of staff to change their classroom practice? And another one might be the role of middle leaders in changing classroom practice. And another one is the extent to which teachers are comfortable with being creative and innovative with regard to classroom practice. So you take the one broad theme and decide which of the satellite themes is going to be most helpful. And that's where you can be, that's when your work is original, that's when it's distinctive, and that's when, hopefully, it has the potential to make impact. And it really is important that you've got to think of your research, please, as being essentially you're going into the professional domain. And we hope that your research will be useful in applied practical terms, but also in terms of the academic contribution that you make. And we really want to encourage you to contribute to the dialogue around what works professionally. Thank you very much for that, very helpful. Any other thoughts at this stage? Any other themes, topics to be explored? Too early in the evening, and not enough cake, okay. So you need to, as I said, I think we've covered number four partly, but the notion is it's the focusing down and getting it right, and that, I think, is the very challenging area, and that's when you've got to do the exploration. That's when you've got to test it out, that's when you've got to have a look at other examples of research in your area, that's when you've got to go to the journals, and that's when you've got to really try and find as many parallels to your own interests as possible to see just where they took you. So just for a couple of minutes, please, just with your neighbour, or if you prefer the people sitting behind you turn round, but basically what sorts of themes and issues emerge in terms of your current interests and crucially also in terms of relevance and significance in your school, and then thirdly what I suppose we might call the doability of it. You're not going to find out in your master's degree about the attitudes of 450,000 teachers towards Mr Goe's view on the curriculum. Well, if you do, congratulations. I think that may probably count as about five PhDs. What's doable, what's valid, what's significant, what's interesting, what are the sorts of topics and the components of those topics which are most likely to engage you. Okay, just have a couple of minutes just to reflect on that. Thank you. This is John O'Neill. I just thought we often refer to, I know you've talked about the conceptual framework, and that is in the documentation that we give them out. So I wonder if you could just touch on that because that might just kick. Oh, right, okay, coming down there. Thank you very much. Thank you. Thank you. Thank you. Thank you. Okay, everybody, can I move us on? Any observations at this stage? Any comments, thoughts, questions? Any issues emerging? Okay, yes, please. I'm actually collecting data, for example. Oh, I see it. Developing Six Formers as Mentals, lovely. Great stuff because, as you know, there's very, very strong evidence base to say that peer-supported learning is very powerful. It comes number three in John Hattie's study and it comes number three or four in the University of Durham study for certain trust. So peer-supported learning has got a very strong research pedigree. So that means it is worth looking at. That's the first thing to be said. The second thing to be said is that it's very much tied into raising achievement. Therefore it's a real school improvement project. So that makes it, there's another big tick for it. And then the use of Six Formers in terms of supporting learning, there's not, just at the top of my head, I'm not aware of very much that's been done, which makes it interesting and worth exploring. But certainly there's very strong anecdotal evidence that, A, for example, year seven, year eight students absolutely love it because their heroes are available to them. And that's actually, and certainly from my own experience, when I was head of a Six Former Centre, part of their social duty was to be involved with younger students and supporting them and so on. And, for example, with problem solving in year 11 coming up to GCSEs, that sort of area is very powerful. So, therefore, there's a very strong rationale for saying there is research evidence which says that this type of approach to learning is very powerful. It fits certain criteria that we want in terms of developing social responsibility across the school. It's about student leadership as well, so it takes that box, and so on. I'm doing a dissertation for you at the moment, I'm sorry. And then the issue is, and here's really the point that's on the sheet, is what is your research question? So, for example, you might say, Hattie says, and the University of Durham say, that it works, does it work in this school? And so what you're doing is taking an existing piece of generic research and testing its transferability into your own school context. And that is good research practice. That's testing a hypothesis generated by somebody else. Yeah? And then, for example, there are some ethical issues here, aren't there? And we need to be very clear about those because everybody needs to know that these six forms are taking part in a piece of research which has potential benefits, but actually everybody needs to be informed and consent to being part of your research project. You can't mask it as a school policy when in fact you're researching it. That would be somewhat inappropriate, yeah? And then the notion is you have to focus in on, for example, how about starting, the starting point is student perceptions. So you gather data from the beneficiaries of the mentoring approach, say. And you have seven and you have eight students or you have 11 students. What about the mentors themselves? How do they feel? So you've got some very rich qualitative data coming through. And you then have got a problem, haven't you, of a lot of data to handle. So you're going to have to design it very carefully so you get some significant responses, but at the same time you don't overwhelm yourself with masses of data generated by interviewing every seven that was involved and every year 12 that was involved. And those are the sorts of practical considerations. But in terms of a school improvement project, it's lovely. It really is because you say in this school, assuming the foreign criteria are met, then we have the following evidence in terms of A, pupil satisfaction with the experience, but also we have evidence that students unease at certain aspects of, with regard to literacy or numeracy, have been overcome by working with other students rather than with staff. That's a really nice project and you can put the check in the post, OK? Anybody else want their project doing now? But I think that's a lovely example, thank you, because it really does capture the issues. The notion is that you see all the way through, you've got to make really robust decisions and you've really got to make sure they're consistent because in examining a dissertation, what we're looking for is the internal logic and consistency that really does mean that it's holding together and you are aware of the implications of the choices that you've made. For example, you might decide in your example that actually you don't need to interview staff because you're more concerned with the particular focus on the relationships between the students as mentors and I refuse to use the word mentease. I hate mentease, don't you? Because they're large aquatic mammals, aren't they, mentease? Yes, is that right? Just keep up. But that again sees part of your research design and it all comes down to your title eventually and as you all know, your title is the very last thing you write to make sure that your title reflects what you've written rather than what you intended to write. That makes sense, doesn't it? Sir. Right. Oh, yes. I think the crucial thing is as long as you define your boundaries and as long as you say this piece of research is interrogating X, Y and Z, I know that there's A, B and C but the following reasons in this context are what I'm concerned about. And then, again, thank you, because your example is far more helpful than mine. You see, the motivation of teachers is such a massive topic and therefore, for example, you could get into incredible detail with all the competing examples of motivation theory there are and actually the motivation of teachers, well, I would immediately pick up on your title and say, one human being can't motivate another human being, all you can do is lock into what that person's drives are and support them. And so, immediately, we're into a semantic debate that has to be clarified and then you've got so many variables because of all the different factors that we know go into engagement and motivation. So, it's a real health warning on that one. Really interesting, very significant but conceptually very complex and also, and here I speak with, I didn't do motivation. What I did was school culture as part of my master's degree a long time ago and quite a few of my colleagues refused to participate because they said, I really don't want to respond to this. I don't want this information going around. I trust you personally, John, but if the head gets hold of my sheet, it's going to be held, you know. And so, therefore, the sensitivity can compromise the potential of your study. And that, I think, is again, that's where you were to do the hard thinking earlier on. And that's when you've got to test the viability of your project because there's nothing worse than getting down the road and then suddenly finding that the majority of your colleagues say we're really interested and very sympathetic but it's not a good time for us to think about this. Yes, ma'am. Right. Yeah, that's lovely. Thank you. That's really a pivotal issue because that, you see, long before you get into actually saying, can we do this? Then there is the issue first and foremost of informed consent. And I suppose that most ethical issues in school-based research come down to informed consent and it's who's informed, but the answer is children, their parents and so on. There's one university which has, like this one, has a standard research ethics sheet. And in this one university, the first question is, does this research involve experimentation on animals? And everybody in education can just write, just children. And it's much easier to experiment on children than it is on earth. The problem you have is that if you are testing out an unknown strategy, then you are to some extent experimenting. And I think there's a real issue around experimentation in education in the same way there's an issue around experimentation in health in the sense that, you know, we really say that we're going to have a double-blind model and the answer is we can't. And in medicine, of course, if there's any sign at all that a particular technique or medication is working, then immediately the research has to stop because it has to be applied to everybody. Morally you can't say, several of you will die, but you would have contributed towards the integrity of the research for this particular treatment. And so the problem you have, therefore, is to say, I'm going to do an investigation with this particular class to test the hypothesis that this particular approach is more appropriate, and that's acceptable, I think. But you're making it very clear that it's very limited research because of the class that you're working with, why have you chosen that class, and then we go back to the context argument and also recognizing the fact that if you've done it with another class, then the outcomes might be different. That's perfect because that's an evaluative study, isn't it? You're saying that we've got evidence that this particular approach has been tried and tested in other schools, but we're different and we're not simply going to bring it in, and therefore we're going to actually work it through in terms of its implications and so on. And that is far more safe. Far safer, sorry. But, oh yes. The new knowledge argument is very simply resolved. You have never done this study before, therefore it's new knowledge. Across the whole of academia now, if you think of all the education students around the world who are doing master's degrees at the moment, it can be very different or impossible to be unique. And therefore it's you in your context in investigating this topic, that's the uniqueness. But that's why your research methodology has got to be so precise and so robust in order to allow other people to learn from it. John, I think there's another issue pertaining to what you're referring to. I think there's the issue about if you're testing the impact. I think it works with schools, but there's possibly an even more important question about why it works as well. So you might look for causation, but is it causation that has impact or are you looking at the process of that impact? And I think there you need to unpick that a bit further as well. Thank you. Absolutely. That's when you are defining your research question in terms of saying what am I looking for here. Is it simply to see what the variables are or is it to say this is working, why, and is it replicable? And you see, in your example, replicability is one of the key criteria because it might be that you have got a particular relationship with this class, they love you, and the way you introduce it to them is so exciting, but it was a me, your next-door neighbour on the corridor, crumpy old miserable son, so he hasn't changed his teaching techniques for 30 years, and it doesn't work and I blame you. That's the issue, isn't it? Is it replicable? Is it transferable? Crucially, is it trustworthy? Yes? Please. You've got to mark out your territory, haven't you? You've got to define your boundaries. Again, I would always say that in this kind of... when you are working full-time and some of you may even have domestic lives and some of you have lives outside education, do you? We know who you are and where you live, then the crucial thing is to focus on depth rather than breadth because the scholarship and the academic integrity is more likely to come from the depth than from the breadth, if you follow me. Therefore, you just have to define, in terms of, for example, there's some interesting work being done, as I'm sure you're aware, on the whole issue of student engagement at Key Stage 4. Now, I think it's perfectly proper to look across the spectrum of the variables that inform student engagement across a whole Key Stage because that's a survey approach and you're not going to go into detail of why this student behaves like this because what you're doing is saying, broadly speaking, what are the variables that influence student engagement across this particular school's Key Stage 4? Now, you will find some comparative studies available and that'll give you your triangulation and that'll give you your means of compare and contrast and that's perfectly acceptable but what you're not doing there is to go into massive amounts of detail. How big is your Key Stage? 10 million students? It's almost unmanagable, isn't it? If you go for a lot of qualitative data, you can't handle it. If you go for a sample size, you've got to be so careful but if you do a survey on the generic issues, then that's probably okay. So, again, it comes back to design. What's the purpose, you see? Go back to where we started. One of the key principles in good design in industry is form follows function. So what you want your research to do and that determines the form. So it might be a small-scale case study with one class or it might be a wider survey depending upon whether it's teaching techniques or whether it's student attitudes. You're number two, sir. Implementing the project itself, I can see how I'm going to see it in my head. But then what I'm going to do with or how I'm going to collect data that's worthwhile because you've got issues you've just touched on popular teacher versus unpopular. You've got the issues of do you do questionnaires which are one to five or a leading question that makes your research redundant because it's a leading question anyway. But if you do interviews, then your history with that person or with the people who will be leading and inspecting them, I know what I feel that question is. I lie on questionnaires and often I'm way too nice. They're outstanding, outstanding. Half of the time they actually ask me genuinely to a cynical friend out in the pub, I'm very quite cynical about it. So how do you ensure feedback that is really useful and what I mean by useful, I mean true? I wasn't employed to do this this evening. That's next week, isn't it, this one? I mean, no, no, it's totally right and proper. And again, the first thing I would say is that it goes right the way back to your design. What are you trying to find out? What sort of knowledge are you creating? And that's the first big issue. And the second thing is then to really begin and we'll come on to this in more detail in just a moment to really say, look, what is the most appropriate form of knowledge that I'm creating here? In other words, is this going to be objective and incontrovertible and highly reliable? Or is it going to be impressionistic, et cetera, et cetera? So what you want to find out, whether it's a small-scale case study of the attitudes of a certain number of students, all their success or whatever, that will determine different approaches in terms of methods and different methods of data collection. If it's a big survey, it's going to be statistically robust. And that's where you need to use the appropriate techniques. I think I've examined something like 25 or 30 PhDs and EDD dissertations. And I've only failed one. And that was because it was based on bad statistics. And it was totally unreliable. And the person who had done a huge amount of data collection had used the statistics, if I can say it properly. She had used them arithmetically rather than statistically. Does that make sense? And therefore, she was coming up... Basically, she had just done... I've got a sample size of 500. I'm going to divide it by 10. That gives me an idea. She was doing the simple sums rather than using statistical techniques. And therefore, her data was crude. And therefore, it was unreliable. And because... And here, again, in the vaver, she denied that it was unreliable. And the three of us who were examining it became more and more hostile because she was not acknowledging that she had made a mistake. Had she done so, then we would have been quite satisfied to say, go away and rework it. We'd get some expert help. But she was just digging a deeper and deeper, deeper hole. We said, look, there's just no way that we can pass this. Oh, absolutely. That's part of being actively engaged in creating knowledge. But the notion is that what you're looking for is trustworthiness and reliability. Now, if you go quantitative, and there may be a case sometimes to do that, then I would say you need to have very, very robust management of your quantitative data. If you go qualitative, then the challenges are much more difficult in some respects. But there, again, you see what you do there is triangulate. You look at alternative perspectives, and you build into your investigation, your data collection design, a recognition that you're dealing with subjectivity here, and therefore you need to acknowledge that, firstly, and then to provide a reassurance to the reader that what you're writing is eminently trustworthy. The most famous example of this, and it is contentious, is a book that I'm sure most of you have engaged with at some point in your lives, which is Margaret Mead's Coming of Age in Samoa. One of the 1930s classical pieces of ethnography. What Margaret Mead did as a young American researcher was to go and live amongst the Indigenous peoples of Samoa, and she produced this amazingly detailed account of their lives, their loves, all the different aspects of being a young person in Samoan society. About 10 years ago, an Australian anthropologist went to Samoa and said to the group of the now very elderly ladies, do any of you remember a young American woman coming here in the 1930s? Oh yeah, we remember, she was a lovely girl, we really liked her, and she asked us a lot of questions about her sex lives. Yes, we remember her, and we told her things that we thought would please her. And it's a huge debate within cultural anthropology now, is Margaret Mead's work reliable simply because apparently, and here's the debate, that she was being somewhat naive in terms of listening because what she did not do was triangulate, she reported verbatim. Now there's nothing wrong with reporting verbatim as long as you say, is it just conceivable that being a participant observer so clouded the responses that it invalidated the usefulness of the data I produced? And that's the area where you've got to be so careful. Yeah, who was it? Hi, is that okay? Sorry. It depends what it is, doesn't it? Right. That's a whole month away, yeah. Are you in a primary or secondary school? Secondary. Now, my major response to this is simply to say that what year 7s would regard as rewards, year 11s would not. Right? So there's no, as you say, I'm looking at reward systems and everything we know about adolescent psychology tells us that as students get more mature, so what they value changes. So that immediately tells you that a small sample of one particular year group is not going to be appropriate because you're only going to get your 7s opinions or your 9s opinions. If your 9s ever have opinions, I doubt it myself, I think your 9s are very strange people indeed. Year and so on. So you raised then the really important issue is how many students in your school? 1,200. 1,200. And so you say, right, I'm going to have to sample. Yep, and that means that you can't survey 1,200 students on the basis of some fairly detailed information that you need, so you're going to have to identify a sample from that 1,200. And that's when it gets interesting. And there is a very strong literature, which I'm not going to go into now about what makes a valid sample size. But that's what you need to explore and to say if you want to get statistically reliable figures in terms of a sample based on a population of 1,200, then there are mathematical formulas available which will tell you exactly how many you need. Now sometimes you can just do it speculative and say, being pragmatic, how many people can I actually handle here? And so it might be, for example, if it's a fairly short questionnaire and without too much data, but lots of tick boxes and the classic Leichert scale on a scale of 1 to 4, never 1 to 5, sir. Because if it's 1 to 5, then being British, we go for number 3. And so it's either positive or negative. You don't allow people to sit in the middle because that's what the British will always do. You design your research instruments on the basis of the sample size and crucially that you can manage in the three and a half weeks before Easter. Yes? And I mean my own view would be immediately to say it's going to be around 10%. So you take 10% of each year group and you very carefully say this is based on a sample of 10% and you may decide then, right, is it boys' girls' school? Boys. So that makes life easier immediately, right, because you don't have to differentiate. So you're going to go for 10% of each year group at random and so what you might do, for example, is ask somebody if there are... How many have you got? If there are 200 students in each year, you want 20 students from each year, so therefore please go through the register and pick out every 20th student and then you've got a random sample that gives immediate a higher credence to your research. John, can we have a simple review from it? Because I think it's probably worth mentioning and just contextualising this evening's presentation in terms of where our students are at and it's a research module which, as you all know, we're putting into place what John's talking about, that focus for a main research inquiry and it also involves trialling one of the methodological instruments that you'll be using. On stage, we wouldn't be looking and indeed we would positively not want you to undertake something on the scale of such that has just been described. So I think it's just worth keeping in mind where we're at with regards to our inquiry whilst obviously we're going to be moving on to one of these issues. No, it's fine because it's all part of the same process but then if you've got to produce this data before Easter then what you might do is simply may point out the methodological issues conserved and then say right, well in that case, what I'm going to do is because I've got a limited amount of time available is simply do a focus group of say 8 or 10 students per year and sit them round a table for half an hour and ask them and then you've got a different technique at work much simpler, much lower key and easier to manage because you're working then on your recording of their opinions but you'll have some data but it'll be a totally different type of data. Anything else? Right, we move on then. Doesn't time fly when you're having fun. The point below point 4 again, this is for conversation with your tutors, your supervisors and so on but it's, I personally think that one of the really powerful formative things you can engage in is saying what are the questions I'm looking to answer and what's my one big question? Which is to what extent will students be motivated by a reward scheme that's the big question and then what are the subordinate questions related to that and as I said there you in many ways the subordinate questions actually outline the structure of your study but I do think it's a very very useful discipline to actually explore how best can I design a question which really really focuses in on the central theme that I'm investigating and that's a really good piece of academic discipline. Now it may well be and I personally welcome this that your research question will change. It starts out as essentially a hypothesis which you then test against the context and then test against the literature you then test again and so on and so on and in that way you refine and refine and refine it until eventually it becomes your title but all the way through you are working to clarify and ensure that you are responding to all the different items that are coming into your study so really make the research questions your guide as you go through the study and it doesn't matter how many times you change them because actually that shows that you're actively involved in thinking about is my focus shifting are my priorities shifting and so on. Okay. Yes ma'am. Perfectly timed that's the next topic. Thank you. Lovely segue. Yes. Then you change your topic. I'm just trying to think of an example. I mean there will be areas that are significantly under research shall we say and where it gets very esoteric and again I would be very cautious about your life is too short to find out what's not out there you know and I think there's a real caution that this is part of the practicality this is part of you know being you really need to complete this you've come so far in getting your degrees that don't be stopped now by an elusive of tasting a will of the wisp. So therefore be pragmatic at the outset this looks interesting but actually a very quick survey of the journals shows virtually nothing available and therefore they may be that is simply unfeasible. I'm just trying to think of a purely hypothetical case and I can't at the moment sorry. Yeah yeah. That's a useful example you see because the chances are if you're interested in cognitive intervention strategies then the research team at King's is probably your starting point you know the work of AD and share and so on you know that the kid AD died four nights ago it's really sad we've lost one of the great researchers in education's country and in essence you follow the cues it is detective work the clues it is detective work and there's a logic to it and you begin to map which journals and then which writers and so on but if you don't have a really rich scene to investigate that I think you really have to be very cautious indeed because you know with great respect you are doing a master's dissertation it's not a doctorate it's not a design to go into every permutation so therefore keep it real keep it controllable by all means be intrigued and fascinated about going off a tangent and so for example I did some work a few years ago on children's spirituality now there's a huge literature on children's spirituality but when you get into actually and it goes back to this issue about cognitive aspects of children's understanding of what it means to be a spiritual being then the literature begins to get a little bit drier you know there's not as much and you don't get that sort of interactions of different types of spirituality and different cultural systems and so on so therefore you come up against a bit of a brick wall and I would suggest again with great respect that at master's level you leave that brick wall alone because there's so much out there it's massive and therefore there's plenty where you can get a lot of data straightforwardly martyrdom is not part of the course requirements of this particular programme as far as I'm aware is and so make it real thank you for that it's a very helpful point okay everybody keep going then so the the literature review there is a real issue for me around literature review on a number of occasions and I'm not joking here or being facetious people have come to me and said I've done my research now can you please give me some advice on doing the literature review yeah I've got the data and can I just say that in the logic of these things you do a literature review in order to identify the issues that you're going to research right it sounds sorry I apologise but actually you get the right questions by looking at the existing research and so on you also indicate to the reader that you're aware of the territory and so if we go back to cognitive intervention there's a huge literature which you can identify in order to show the reader that you understand where you are and then you can move into your own particular area but the notion is in using the literature you basically are doing two things one is you're building up your own confidence in understanding exactly what is required in terms of the components of your research and secondly you're demonstrating to the reader that you have gone beyond your own particular situation and you're aware of the wider and broader issues and that's when you get the interesting debates that's when you begin to see where the arguments lie and so on but it is it's probably I think it's one of the two make or break elements of most master's dissertations we do not need and then he said and then she said what we need please is a synthesis what are the key themes and ideas and how do you pull them together for example let's supposing that you're looking at this whole issue of feedback to students in the classroom yep now the key people here are obviously John Hattie Carol Dweck and the University of Durham report for the people premium and they all agree this but then if you go into the literature there are numerous permutations of what feedback looks like how it works, the psychology of it Carol Dweck's perspective is a very distinctive one John Hattie provides a very limited perspective in my view on what how it might work then this whole issue of assessment for learning and how that ties in with feedback and so what you're doing essentially is clarifying refining synthesizing and defining that work and if it comes down to the crunch you don't actually do this but at the end of your section on the literature you tell the reader in this study these are the terms I'm using and they mean this and so you filled in the gaps in the plan and therefore you have got control of the knowledge one of my favorite moments was a master's in the north west and it was largely action-based research and people would do real-time team-based projects in schools and one of them was developing teams in schools and one of them the groups came in and they sat down and I was the external examiner and I was introduced and one of them, if you'll pardon the phrase she went oh my god because their entire project was based around one of my books and they said are we really going to sit with you for an hour now and tell you what you've written and the answer is yes you are because you've got to make it yours you've got to make it understandable so you work through the dissertation so you work through the literature identifying that which is pertinent that which is relevant and then crucially you make it your own and you really do demonstrate you're in control, you understand and you've done the hard work in terms of surveying what is and is not available so you are confident that you are you know where your particular topic is in terms of the evidence and the research having established that you can then go back to what is the study you've decided to find out and you then have to define colleagues exactly what the methodological issues are and we get there in the end and again there is I think an area where just to offer you a health warning to say basically that you need to write about methodological issues as well as well as about methods and that one of the things people tend to do is to say two sentences I'm going to do a qualitative piece of research and then I'm going to about five pages on interviews not good enough what we need to know is what sort of knowledge are you actually creating and so for example on page three of the handout the second paragraph there in fact no the first paragraph there are many forms of knowledge which we use often subliminally in our everyday life so for example knowing that God exists knowing that there are black swans in Australia that the sun will explode in 5.5 billion years have you all done a risk assessment about that by the way and that your partner loves you are all different forms of knowledge aren't they and the crucial thing is that each of those has got a deep epistemological basis of saying this is the basis on which we know this is true are you with me at this stage on the evening after a long day and you need to be able to demonstrate to the reader of your research that you understand that there are different types of knowledge with different levels of confidence and the example I always use which I hope helps is that in a court case different types of evidence carry different levels of significance and trust, yes so for example sometimes spuriously but usually correctly forensic DNA evidence is given the highest possible credibility whereas circumstantial evidence by a friend is often treated with great caution and your your research has got a fit into that spectrum and therefore you've got to be able to argue that this type of evidence is to be accepted in my study because but you need to know what sort of evidence it is that you're producing you need to argue that it is the most appropriate form of evidence in the context of your study and then you need to be able to demonstrate on what basis it should be accepted is that reason we go here and for everybody and on pages three and four and five there are a range of activities which we we're not going to do this evening but each of them have different issues and hopefully you'll find them helpful but it's the diagram in the middle of page three or not of the diagram the figure there which really does capture the debate this is what you've got to argue in a doctorate there's usually a whole chapter about this in a master's dissertation I would say that you need a significant amount of discussion before you get into your methods how we actually got the data you've got to say what sort of data are we looking to collect and so for example those different continuums there are you going to be empirical in other words this is what the science tells us this is the hard objective data or are you saying basically that everything's relative because that again is a perfectly legitimate academic stance are you going to be objective or subjective if you're going to be subjective if you're going to claim to be objective what basis are you being objective in other words how trustworthy is your work sometimes actually for example if you're asking people to discuss their lives as educators where's the objectivity in that there's none is there because each person's personal life history their narrative is of itself valuable and one of the things we've learnt mainly from feminist theories of research is we have to respect people's life stories so even though they're totally subjective they have an authenticity and a value simply because they are reflecting the integrity of the person so therefore you really have got to say how confident am I about this data and how do I argue for its validity in this context is that making sense for everybody I know it's a tough time of day to keep going and it's getting quite warm in here but you're not nodding off yet as far as I can see so I'll keep going basically the issue is to be explicit, robust and confident and say even what I'm investigating for example if you really are looking at the motivation of teachers then you're going into deep subjectivity aren't you there's no objectivity about something talking about motivation therefore how can I trust and that's when you do the hard work in terms of justifying your approach because of what it is you're trying to find out and then showing how the methods you've chosen are appropriate in order to maximise confidence and that's when you really earn your degree right because that's the really tough academic, when people talk about a piece of research being scholarly what they're actually saying is is this person able to argue the case for the integrity and the trustworthiness of their research is that reasonably clear lots of little nods around the room yep so now for example if you go over to page 4 just to reinforce the point at the top of page 4 there's a quotation from a book written a long time ago now by a very distinguished head teacher and just look at that first sentence there as far as management styles go it seems to me right that immediately tells you something about the status of the knowledge there doesn't it and then three lines down four lines down it seems clear to me next line down including myself yep that's neither right nor wrong that's how she has written but it tells you something about the status of her assertions doesn't it whereas look at the the second one down which is from one of the great books as far as I'm concerned about educational research again quite old in this chapter we postulated that so immediately you've got a different language at work yeah and and so on and I think it's a really powerful piece right because what Susan Rosenholz did was to really get this balance of evidence and interpretation and conclusion right and so on and then number three very detailed research Kenneth Leith would one of the great educational researchers at the moment and then some action research action research you see falls outside the normal categories and yet it is immensely powerful if we are clearly managing the process I've got no problems at all with biography I've got no problems at all with journals for example I supervised one PhD that was based largely on journals and that was a very powerful document very difficult because all the time I was asking the writer to say explain why that's the case justify that argument what are the alternative perceptions there so I mean and she got really angry with me but the notion is that it's all valid as long as it is justified and demonstrably appropriate to the sort of study that you're carrying out yep and having gathered your data sorry having identified the methodological context the epistemological framework of your study you can then say in the light of what I want to find out in the light of the conceptual framework of this topic in the light of the methodological issues these are my chosen methods and you've got a nice logic working through your study and that's perfect that's absolutely wonderful and all the time you're saying does this section link to that section link to that section link to that section when it does you're writing a great dissertation when you've got your data and again one of the great failings at this stage is here's my data and what people don't do is go back to the literature they don't go back to the context and therefore what they do not do is engage the reader in a conversation around what this shows what it means interpreting it explaining it and so on and then depending upon what your study shows you are then able to move into a situation of saying here's what this study has meant to me here are its implications this is where I now feel I am and so on and it's therefore very much about the what do I do with my findings essentially point number eight I think we've mentioned you must as part of your methods be absolutely explicit that you are aware of the potential ethical dilemmas and that you have responded to them in an appropriate way and that you've observed the university's requirements it's all about showing that we can depend upon what you've written number ten it's an academic exercise that's a good thing it means that you are going to observe certain conventions but at the same time how are you going to use it how are you going to apply it and what's really wonderful to see with master's dissertations is when they have their own internal integrity but then they actually they make a difference in school and that's when it gets interesting that's when you should be writing that's when you should be publishing sending articles to the TES getting into the professional journals and so on and then finally can I just and this again is a personal thing but I'd love to see whether in masters or doctorate of work at the end of the whole thing you've said I'd be really worried if somebody did a whole dissertation and at the end of it said and I have not changed and if you talk to anybody who has completed their master's dissertation or doctorate I think it's pretty much a life changing process and therefore it's something that you need to reflect on because it just corroborates the significance and status of your work and again as I said the big caution at the bottom there is just going to keep it going so on page three and page four there's an activity which sort of helps you perhaps to explore this real issue of methodological concerns what sort of knowledge are you writing and then on page five what I've done there is to identify a range of topics and I've actually jumped the gun because I've put in the methods the strategies to collect the appropriate data what you might wish to do consider the methodological implications of that topic and those strategies purely as a way simply of reflecting on the sorts of topics that you might research in order to be confident that you are going to adopt the appropriate methodologies and all the time validity, reliability and trustworthiness now if you want to know the difference between validity and reliability it's all to do with bathroom scales for way yeah have you heard this one if you use your bathroom scales and you weigh yourself week on week then and you see that you're losing weight which is the main purpose for weighing yourself then it is reliable yes if you go to the doctors though and their scales are different which they almost invariably are then your scales are not valid in other words if you don't test your attendance against an external source how do you know whether it's valid or not it might be reliable because it's internally self-justifying but unless you test it it's not going to be trustworthy and it's trustworthiness that really does seem to make the difference now it's been a real rush but thank you very much to those of you who actually came in with ideas and so on are there any final points before we go things that you say I need to ask about that fine the easiest way and probably the best way is that if you think of all the academic journals that there are somewhere somebody has done what you've done and that's when you begin to find the comparisons they won't always be precise the triangulation absolutely and the best form of triangulation is to find a piece of research like yours which you can then compare and contrast it'll never be exactly the same but that often is necessary or you build an internal triangulation because what you're saying is I'm going to do a survey here but I'm also going to check the perceptions of some of the people in the survey using a different method and then you build confidence that this is not just one perspective it's multiple perspectives it's all about getting different frames on the same topic thank you thank you very much so if you thought you were going out with all the answers I think you're probably going out with more questions than answers which is exactly the right place to be at this point I think so yes I'm going to no worries about that just to encourage you if I can please to visit the Sims Capital sites and to keep an eye on that because the application form for ethical approval is on there and we'll be asking you to submit those and you will need to submit them before you collect any data so do keep an eye on that and there will be more information coming through to you thank you very much for coming along tonight and for your contributions tremendous range of contributions and good night safe home and I'm sure we'll see you again shortly thank you all very much good night