 Fodd bwyddiolch. Rydw i'r gweithio i gael dweud i gael 12 gai meddwl ar Fygedeithasol cwmwyntydd ac rydyn ni'n fbytdo i'n bwysig i eu gwiswärddion i ddefnyddio dyn nhw i gael y cystafol ond ry解'r gweithio. Rydyn ni'n ffysgdd iawn i gael ddefnyddio'r agendafydd ar Fygedeithasol Cymru, a'r Rhyw Llywodraeth Cymru ar hynny ymddur. Ar gyfer y rhan fyddai, rydyn ni fyddai'n ddefnyddio'r agendafydd ar gyfer y gwaith fyddiadau hefyd. Agenda item 2 is the third of four sessions on the committee's pre-budget scrutiny. Earlier this month we had heard from Skills Development Scotland and the Scottish Funding Council. This week we have the Scottish Qualifications Authority and next week we will hear from Education Scotland. I welcome to the meeting Dr Janet Brown, chief executive and Linda Ellison, director of finance of the Scottish Qualifications Authority. Before we start, I would like to put on record the committee's thanks for the SQA, hosting a visit for Fulton MacGregor and Ross Thomson last week. I understand Dr Brown wishes to make a short opening statement. Thank you. Good morning everyone. As the national awarding and accreditation body for Scotland, SQA is responsible for the quality, validity and maintenance of the credibility of qualifications offered to learners in Scotland. Given the size of the accreditation function compared to the awarding body, we fully understand that the focus of this committee is on SQA's role as an awarding body. In that capacity, the SQA develops qualifications to support the education and skills system in Scotland. Our remit is to ensure the ability for learners to progress successfully through learning with the qualifications by building upon the previous learning and preparing the individual to be successful in the next phase of learning or into work. SQA qualifications must therefore be set at the correct level and the course content must reflect the knowledge, understanding and skills necessary for achievement of a successful destination for that learner. In the case of curriculum for excellence qualifications, SQA was asked to develop a suite of qualifications that build upon the learning and level of achievement of candidates that they would have received during broad general education. The CFE management board approved the design and structure of those qualifications. The course content and associated guidance were developed in consultation with stakeholders from across the sector, teachers, colleges, universities and employers, in addition to professional associations and particularly subject specialists. It should be noted that the nature of the assessments reflected the desire of CFE to provide opportunities for personalisation and choice for candidates and teachers to be able to set the assessments within those associated personalisation areas. SQA does indeed play a significant and key role in Scotland. However, it is important to understand that role in the education system as a whole. The structure of curriculum models, the nature and number of subjects undertaken by an individual learner or group of learners is determined within a school or college and the qualification any individual candidate undertakes is a matter for those centres in consultation with learners and parents and carers to support the best interest of that young person. That does not fall within the remit of SQA. The development and delivery of qualifications is a complex process and SQA takes the approach of trying to ensure inclusion of both professionals in terms of teaching and learning but also subject specialists and those who will be receiving the learners after they have undertaken and achieved the qualification, namely colleges, universities and employers. We have a strong working relationship with teachers and lecturers across Scotland. Indeed over 15,000 of them work with us in partnership every year to develop and deliver the qualification system as a whole. As the national awarding body, SQA is responsible for ensuring the standards, credibility and sustainability over time and in doing so has to balance the needs of a variety of stakeholders. The introduction of CFE has been one of the biggest educational changes in Scotland and indeed that change continues with the recently announced decision by the DFM when he asked the SQA to redesign the assessments associated with national qualifications. As the new nationals have been implemented, SQA has provided additional support and listened and responded to the needs of teachers. In the past few years it has taken significant time in terms of implementation and adjustment for that significant change to become embedded. During the course of the development, there was absolutely an essential need for SQA to communicate as much as possible with those involved in the delivery and learning and the teaching of candidates who have completed a qualification and this engagement has continued as the qualifications have been put in place over the last three to four years. Our responsibility to understand and address issues is well understood. We solicit regular feedback and receive feedback from teachers and other interested stakeholders but the nature of our work in some cases means that we sometimes receive different advice from different sectors and different people reflecting the different approaches of respondents to their subject and the different opinions of the nature of the course content and the approach that they would like us to take. SQA seizes its responsibility, the need to fully understand how our qualifications and assessments are operating in schools and identify any issues that need to be addressed. It was in this vein that at the end of 2015 SQA undertook both a review of the qualification design and detailed field studies to develop a research base and an evidence base that allowed us to understand what was needed to be strengthened and what had worked well as the qualifications had been implemented. That field study involved talking to teachers, senior management teams within schools and learners themselves to get a full understanding of how the qualifications were operating. We also believe that it is our responsibility to publish our findings so that others within the system can understand the nature of these results. The committee may be aware that in May of this year SQA published two research reports, one of the field work and one on the detailed study of the nature and design of the assessments. As a result of this evidence SQA made changes to unit assessments for the current session and communicated these and planned future changes through the subject review reports for each subject again in May 2016. At the current time SQA is undertaking further study and further field work again with senior management teams, teachers and pupils to understand both how these changes have worked to improve aspects of assessment but also to take further the discussion to understand the nature and experience of implementation of particularly national 4 and national 5 within the school and college sector. We are fully aware of our responsibility to provide value for money to the public purse and we are focused on the safe and secure delivery of our remit within a decreasing public purse. We regularly review our processes and procedures to identify ways to improve and provide value for that public purse. We welcome the opportunity today to discuss our activities with the committee. Thank you. Thank you very much for that full short statement. Do you mention in your statement that you have a strong working relationship with teachers? The online survey that we received and a meeting that I had with teachers would suggest that that was not quite the case. Can you explain to me why there has been so many negative comments? I normally would not take anonymous online comments seriously, except for the fact that pretty much everything that they said was backed up by the teachers that we spoke to face to face, who were quite very strong in their criticism of certain aspects of the SQA. Can you tell us what has been done over the past year to build this relationship, to continue this strong working relationship with teachers, and why is it that the teachers do not seem to feel that they have that strong working relationship? Two issues, I think. The education and the qualification system in Scotland is run in partnership between SQA and teachers, so we have 15,000 of them that work with us every year. Those people are involved in developing the nature of the qualifications themselves, the nature of the assessments that are undertaken every summer, and those people are very supportive and very engaged in ensuring that what we are trying to do is do the right thing for the learners of Scotland. I think that that is a significant number of people. As in any case, when you introduce any major change such as curriculum for excellence, there are things that people agree with, things that people do not agree with. The approach that was taken with CFE was very much to move away from a prescriptive NAB-based assessment to something that was very much more teacher driven. That, I think, has proved to be a challenge for some teachers and not something that some teachers wish to do. As we move into the new situation where there will be no unit assessments, that aspect of the discomfort that teachers have found with the approach that was agreed by the CFE Management Board will actually be removed. We also do regularly get feedback. We have received feedback from teachers about what they did like and what they do not like in the courses, how the assessment has worked, and that was partly the reason that we went out last year to do the fieldwork, to try and get a better understanding of what that meant so that we could go away and address it. The subject review reports for every single subject were published that told everyone what we were planning to do to change and address any issues that we had identified. The group of teachers that Ross Greer and I met covered the primary secondary, but mainly the secondary teachers who were taught at the SQA, but they also covered varying degrees of experience. They all, I think, without fail, had the same complaints about the relationship between the SQA and the teaching profession, and they did not seem to think that it was working. They did not seem to think that there was an awful lot of… Your response touches on communication, and they seem to think that communication between the SQA and the teaching profession was non-existent, or if it was anything, it was negative, as opposed to a positive communication, that they were not getting anything from it, that it seemed to be more to obstruct them than to help them, and that a lot of the things that the SQA seemed to be putting in place for teachers were done for no apparent purpose as far as the teaching profession was going. Surely, either you are doing it wrong or you are communicating badly, because that is the only one of two reasons that they could possibly be unanimously telling us that they did not feel that the SQA were getting it right for them. Communication, as we all know, is an extremely complicated and challenging area, and it can always be done better, and I will not disagree with that. It should not be, but communication should not be. If you use simple language, the most simple language to describe the most complicated reasons, the most complex reasons, then it should not be. I accept that you are talking to lots of different audiences, but communication should be fairly straightforward. I think that some of the feedback that we have got in terms of the complexity of our documentation is something that we are looking at, and one of the actions that we took from the assessment of the national qualifications group was to go back and look at that. Sorry, Dr Brown, but I know that Liz Smith is going to come in shortly on us, but I saw that the previous convener, Stuart Maxwell, had this discussion with you last year. We are having this discussion with you this year. What has happened in the interim? If it is a complicated issue to speak simply, then surely there has been a lot of work being done between last year and this year? There has been a lot of work done between last year and this year. We have a specific liaison team that is targeted to every single part of Scotland that we are getting very strong positive feedback on. They go into individual schools and they work with those schools to understand the nature of their concerns, and we very much act upon that. We get very strong positive feedback from schools on that. I think that the other thing that we have done is we have very much focused on trying to simplify what we are doing and trying to make sure that teachers are aware of the changes. Unfortunately, there have been significant changes again because of our findings. We made changes to the nature of the assessments. That is something that needs to be conveyed to teachers. A lot of the complaints that we have had—and I recognise that we do have complaints—are around the number of changes that have been made during the course of the introduction of the qualifications. We are about to enter another phase of that and that will not be comfortable for some teachers as well. It really is keen that we not only communicate to teachers but we also do to parents and to employers and to the learners themselves. It is interesting that you said that teachers will not be comfortable with it. However, as I said, there was a wide range of experiences in the teachers. I am sure that many of these teachers have been through a number of changes over the years and have adapted to them very quickly. However, I am glad that you mentioned parents because the national parent forum was quite scaring of the communication and saying that they could not take part in the survey because they did not understand it. It is not just teachers that are affected by this. Obviously, parents are the most important adults in this process because it is their children that the SQA is responsible for getting to them. We again meet on a regular basis with MPFS and with the SPS. One of the things that we make sure that we do is try to understand the nature of the concerns of parents and how to make them simple. We worked with the parents council to make sure that we supported them in putting out nationals in a nutshell, which was very much focused around the language that parents would understand. We take feedback and we try to address that as much as possible. We have a parent representative on our advisory council, and that voice is there very much to give us the feedback as to what is working and what is not working. The advisory council is a key component of that whole feedback mechanism. That is my last comment before I pass on to my colleague Will Smith. The submissions from the two organisations in Education Scotland are totally inaccessible to average parents. That is clearly not acceptable, even if you have a parent on your advisory group. It is clearly not acceptable. There is a lot more work to be done. I would hope that, when we come back here next year, we are not having the same discussion about the lack of communication or the inability to communicate with others. Before I go to Will Smith, I have had my turn. Can we keep the questions as brief as possible, please? We have a lot to go through today. Dr Brown, in my 10 years in this Parliament, I have come across a set of evidence that is so compelling in its concerns about SQA. I say that with regret, but the messages that we have received in the submissions are deeply worrying, and I am sure that it is a worry to pupils, staff and parents. You cannot be satisfied with what we are being told here. Can I ask you whether you think that the system is fit for purpose, and, most specifically, whether you think that the delivery of the curriculum for excellence is fit for purpose? First and foremost, any feedback that we get that is expressing the level of concern that we have seen in the submissions is something that SQA needs to take seriously, and that is the reason why we are engaging at this point in time across the country to try to understand what is going on. I fully accept that. In terms of curriculum for excellence as a whole, as I have said several times before, we have discussed that the findings that we identified during the course of our research last year identified multiple reasons why there were issues associated with qualifications, one component of which was the responsibility of SQA, and that is what we are trying to address and deal with. That is part of the reason why the units have been removed and why we are redesigning the assessment. The way in which the qualifications are perceived by those who receive candidates with SQA qualifications is a testimony to the value that is put upon them. That is something that we will continue to guard and continue to work to make sure that it is maintained. I am not sure that I have much reassurance from it. One of the consistent themes that is coming through those papers is the fact that people are questioning the process of setting exams and the process of marking them. We have teachers who are making a strong comment that they feel that there is a lack of effective scrutiny over this and transparency. I went back to the committee session of 22 November 2015, when you were asked about the production of the minutes relating to the grade boundaries and the way in which the assessment takes place, and you said that you would produce those minutes. They were not asking for what the decisions were. They were asking for how they were made. In other words, where is the transparency and the scrutiny over the setting and the marking of the exams? That has not happened as far as I can make out. I have been on the SQA website and it is clear about what some of the decisions have been, but there are people who are asking about whether we need much greater transparency in that respect. Can I ask you what your reaction is to that and whether you feel that an independent body of scrutiny would be helpful? The way that we undertake the establishment of the grade boundaries in the grade boundary meetings, we have changed the way in which we have reported that. We have added more information in terms of the background and the reasonings behind the decisions that were taken. That is something that we have published and that is available for every subject at every level. In terms of the scrutiny that SQA has put under, one of the things that we have within the organisation in the governance structure is the qualifications committee. The role of that qualifications committee is to ensure that SQA's approach to qualifications development, assessment development and establishment of maintenance and standards, is appropriate. That group is not only made up of SQA board members but also has representatives from teachers, colleges, professional bodies and assessment experts from universities both north and south of the border. That group has to approve all aspects of the way in which SQA undertakes its work in terms of assessment for qualifications. I think that that scrutiny is a very challenging scrutiny and is there to be able to make sure that we are doing the right things. John Swinney said in the chamber and he replied in a question for me on 22 November in this committee that he said that it was absolutely intolerable to have mistakes in any exam paper and that those mistakes should they persist. That draws into question SQA. Do you agree with the cabinet secretary? I believe that we should not have errors in our exam papers and I have said that before in this committee. Why are they happening, Dr Brown? I think that they are happening because people are working extremely hard. One of the things that we have done is that, as a result of the errors that have occurred, we have added additional quality assurance steps and we have fresh eyes looking at the nature of the assessments. The other aspect that we need to do is that we need to make sure that everything that is required to make questions within the question papers valid and appropriate, we control and we make sure that we have appropriate engagements with institutions that are developing those particular aspects. My last point, Dr Brown, on this section. You have replied in a letter to me that there is additional scrutiny for STEM subjects and you have explained that the reason for that is because some of them have an increasing amount of technical issues. Could you just tell us which subjects are included in the definition of STEM for this purpose and why is it that other subjects are not receiving the same degree of additional scrutiny? I think that part of the issues that we have seen historically in the STEM maths, computing science and science subjects in general, is the fact that there are technical aspects to the questions themselves that it is important that you have a separate set of eyes to look at. I will bring up national 5 computing. If you are looking at national 5 computing, there is a particular programme language that is used that we need to make sure that we are engaged with and that is written specifically for SQA. It is important that that complexity of the nature of that question as opposed to the nature of a verbal question that might be set in a social science subject, that that level of technical detail is looked at by more than one person who is of that expertise. That is why we are focusing specifically on those areas. Just to come up with a point of clarification, geography, which has obviously been very much in the news, and one or two geographers in those papers have suggested that they are more akin to a science discipline than they are social sciences. Is geography included in this? The whole issue associated with when you are putting in complex questions such as statistical questions or equations, etc. That is what I am talking about. That happens in economics, for example. Is that included? There are very specific actions and they vary by subject, depending on what the nature of the questions is. There is an overarching quality assurance process that is in place, and I want to make sure that you do that. Can I get some clarification there, Dr Brown? I am not really sure if the answer to the last question is geography and economics, part of that extra quality assurance. Economics is definitely not. Geography, I am not aware of at the moment. You will get back to us on that. Yes, so we will get back to you on that. I am back, convener. I do think that Dr Brown is extremely important for the point of view of scrutiny. I think that this is a question that parents want to know, because we have had issues in which you have admitted this morning about specific exams, and we cannot go on like this. We cannot have an on-going issue about the nature of some presentations. It matters about where that scrutiny is and about which subjects it is. Parents have a right to ask if there are some subjects that have this additional level of scrutiny. They want to know why, and they want to know exactly which subjects are involved and which subjects are not involved. Would you accept that? I think that that is a fair point, yes, definitely. Can I just say to the committee, though, that we develop a significant number of question papers every year? No one question paper should have any errors in it, I totally agree, but there are a significant number of question papers that are developed every single year. Thank you, Fulton. I say thanks for having us over last week. It was a very worthwhile and interesting visit, and it was good to see the scale of the work that you are involved in. Certainly, I think that I reflected after the meeting that I felt that I learned a lot about what you do, but a lot more. You do a lot more than what I initially thought. I think that the convener and Liz Smith have already picked up on the line of questioning that I was going down as well. There cannot be any escaping that the submissions and what not are very damning for you, and I think that you have reflected that. I suppose that what I would like to hear is, rather than going over the facts again, is something more emotional. Can you convince me, can you convince this committee that you will seek to change the nature of the relationship between the SQA and teachers? I would like to get an answer that I can feel will be back here next year, and that will change. I think that you could do it. You are capable of doing it in the team that we met last week. It is a fantastic team. I match your people in the word to the emotional side, because I think that you have covered your statement and what you have already said in the last two answers. It has covered the facts, but I want to feel convinced. That is the focus of what the SQA is trying to do. The challenge for us is to reach every single teacher. One of the things that we very much focus on is that we run sessions for teachers. We try and provide specific requested support. If a local authority has aspects of particular qualifications that they feel their teachers need support on, we will go and engage with that. We meet with the teachers unions on a regular basis, and we have very productive conversations about that. Sometimes very challenging conversations, but definitely strong communication and strong engagement. Similarly, we have the same thing with the Head Teachers Association SLS. We are in the process of revising and reviewing how our messages get out. There is feedback, as you have seen in your responses, about our website. We are very aware of that. One of the challenges that we need to move towards is taking advantage of new technology to be able to, instead of providing long documents that, by their nature, have to include all aspects of a particular subject. We break that down and customise the responses that a particular teacher will get, depending on the nature of what their inquiry is. That is not a simple and easy thing to do, but it is something that we are absolutely focused on and working on at the moment. The nature of our engagement and communication to teachers is also very critical. I think that one of the pieces of feedback in your submissions was also the fact that a lot of information goes through the SQA coordinators within a school. That is a challenging role for a teacher to undertake. One of the things that we need to make sure is that we make it possible for us to communicate with each and every individual teacher at this point in time. That is not possible for not just our reasons but also reasons within the schools. Again, it is about customising what information is received by an individual to make sure that they get what they need in the format that they need it, as opposed to a very long PDF document, which is the current status. That is a major focus for us, but it is a complex thing to do. I am glad that you picked up on that, Dr Rowan, because that is where I was going with it. When I read the submissions and I have heard about the group of teachers that met the convener in Mr Rear, I also took the opportunity last night to contact by text, just a quick text, teachers are not, because I think that, like everybody we all know, many teachers are friends. In the kind of responses I have got back where, maybe not as critical, I would say, but they were kind of down the same line. I think that the impression that I get is that teachers feel that the SQA is something that is done to them. I might be wrong, but it is something that they have not really got a real say in or they are not really part of the process. I am encouraged by what you said there in what I heard last week. I think that, if you are able to drive that forward, as you have said, and it is more individualised, you could really change that perception. Therefore, I am quite happy that that sense that you have given. Do you think that local authorities and teachers in general can be a bigger part of the Government's accountability, so that they are feeling more part of the process? We currently have people who are teachers on our board and we have representatives of teachers and teachers unions on our advisory council. That is one of the things that we very much take a focus on. When we are making changes, for instance, we will take those changes to the advisory council and get their feedback on them. Again, part of the reason why we meet on a regular basis, individually with the unions, is to get their feedback on what we are trying to do. The local authorities in terms of the Association of the Rights and Education of Scotland, SQA meets them and engages with them. Again, they get strong feedback on what is working and what is not working and the different approaches that have been taken across the country in the different local authorities as to how they utilise the people that SQA has trained. SQA has trained a significant number of people over and above our requirement in terms of standards and in terms of internal assessment. The expectation was that those people would be used by the local authorities. We need to look at how we can improve that across the piece. That is closer engagement with the director of education to understand what they need specifically from us and the needs may differ from the different regions of the country. For instance, Shetland finds it challenging to be able to send teachers to the events that we run. We have started to do webinars, we have started to do twilight sessions so that people who cannot travel or who cannot get out of school for other reasons are able to engage with us, give us their feedback but also learn and be able to respond when we need to respond to them in different ways as opposed to just assuming that everyone can come to an event even if it is in Aberdeen. I remind witnesses that we have got enough lot to go through today so that we can have shorter responses. Tavis Scott. Thank you. I just want to make Dr Grant back to the performance of the SQA. In one of the submissions today from a physics teacher with regard to a higher physics assignment, they say—please correct the record if this is wrong—that several of those documents are already on their third version despite it being the third occasion that the courses run. The submission goes on to say a total of 81 pages of guidance across five different documents, three accessible on the main SQA website but two on the SQA secure website. Is that all true? I cannot tell you whether the exact numbers are true. There is a significant amount of documentation associated with this and that is what I was trying to say. If you write a document—is it true that it is a third version of the documents on the third occasion that this course has been run? I do not know the answer specifically to that but what I would say is that it is not unlikely because one of the things that we have been doing is trying to respond to the feedback that teachers have been giving us. In responding to that feedback we have to modify our documentation and that is a double-edged sword for a teacher because one of the things that they wish is that we change the way that we are approaching something and therefore we have to change the documentation. You made some good points about the format and the point about simplification. Is it simplified to have five different documents, three accessible on one website and two available on another website? Is that accessible to teachers? One of the things that we try and do is that we try and make information accessible to not just teachers and part of that challenge again in terms of the way in which it is worded is a significant issue. We try and make that available to other interested stakeholders who wish to see that. My son is doing physics. I want his physics teacher to know what the stuff is. The primary responsibility that you have is to get this information easily to teachers, is not it? Yes, it is. Is three different websites and 81 pages across five different documents the easiest way to do that? No, absolutely not. What are you doing about fixing it then? We are improving our systems. Can I say that it is a real challenge for a public sector body to renew its systems? You mean IT systems? Yes. Oh my gosh. Please do not scare me any more. No, we are not doing it in a very large, big bang way. What we are looking at is— We have heard that before. I am quite happy to go through our approach to this if you would like me to. No, I am not too bothered about the systems. Please write to the committee on the systems, because some of us have been in the audit committee and seen these from different organisations of public sector and have seen scare story after scare story, so maybe you can write to us on that. Can I just answer your point about the different websites being accessed? One of the things that we are currently in the process of doing is giving individual teachers individual access. As I said, that is not going to happen tomorrow, but the approach that we have is that a teacher will have associated with them what they can see. They will go through one website and one website will give them access to the secure information that they need to see and the generic information. If I, as a parent, wish to go on to an SQA website, they will not be allowed to go into the secure website, but it will be transparent to that teacher as to where that documentation is held. When will that be available? That is the process that we are currently undertaking and that will not be in the next session, because it is not an easy thing to do. Forgive me, but when will it be available? Right now, I do not know when we will be able to deliver that, but we are planning and we have a detailed planning process in place to understand when we can deliver that, given the complexity of the other activities that we are undertaking. Okay, and the other question that I wanted to supplement, which I wanted to ask further to the convener's questions, was that you will be familiar because you have obviously seen all the submissions that the committee has had. The Scottish Association of Geography Teachers claimed that this year's higher exam was, quote, the worst ever. Why is that? We continue to have regular dialogue with the Royal Society of Geographers. In fact, they were in to see us last week and we have another meeting set up in January. We have not received significant feedback other than what we have just seen in the submissions to the committee as to the nature of the exam paper during the session. On the day that there was very little feedback, we got very little feedback afterwards. We have had conversations in the nature of the content. If you were to look at our subject review reports on geography, you will see that part of the actions was to address some of the findings that we had found associated with our fieldwork, and also the conversation and input that we have had from people like the Royal Society of Geographers. Some of the changes that they are looking to see are already documented in that May document that went out. The committee has been told that more of the half of the respondents to the survey said that the paper was, quote, poor, shocking, terrible, worst ever and nothing like the specimen or previous papers. Putting aside all the adjectives, nothing like the specimen or previous paper, is that true? No. That is not true. You have been able to furnish the committee with evidence to the country about that. Yes, we will. Could you furnish the committee with a real, detailed answer to why so many geography teachers think that it is the worst ever? On to finance, 75 per cent of the SQA's cost base comes from P-Wall. A significant part of that is from appointees, ie markers and invigilators. Concerning the reports in the Guardian that the SQA, a recent meeting with the poverty lines, couldn't confirm the actual hourly rate paid to invigilators. Clearly, that has an impact on your ability to comply both with the living wage and also national living wage, the minimum wage legislation. Given that invigilates aren't paid hourly, can I confirm that the SQA has no way currently of knowing the effect of hourly rate that each invigilator has paid? In other words, you do not know if you are paying the living wage or not to invigilators, do you? Thank you for your question on that, because that is something that we have been working on with internally and with the poverty lines recently. We have a process in place that takes account of the way invigilators and chief invigilators wish to work, particularly chief invigilators, on how they manage invigilators over the exam diet period. We pay a session fee of £27.15 per session, which is either a half-day or a half-time session. Which could be as much as four hours or more. No, no, sorry, it's not. At least about 47 per cent of our diets is under two and a half hours. That's but you're not including extra time there, are you? A preparation time? There's half an hour for a preparation time. It's on top of that, I accept that. So it should take it over three and a half hours? No, it's not to be done. Your own guidance says half an hour either side? No, it's 15 minutes either side, sorry. Up to 45 minutes is what I've seen written down. The bottom line is that we have looked at the diet, the number of exam sessions, the majority of them, sorry, not the majority, at least half of them are under two and a half hours. Where those happen, obviously what we are paying is significantly more than the minimum wage and the living wage. There are other days when, or other sessions, which might, yes, be longer, but what the chief invigilators are responsible for doing is making sure that there's a fair allocation of sessions across the payment period for the diet to ensure that we are satisfying the living wage. What we've accepted though, and what we recognise, is that because we pay by session and because the chief invigilators are able to look at the workload an invigilator has taken on in a particular centre. If they think that that invigilator has worked more hours, they have the right to add another session fee for that invigilator. Equally, chief invigilators or invigilators can speak to us if they think that they've done more time than we have allocated. That's the process as it has been. The process going forward is that we will now record the hourly rate, so that we have added now for the forms that we use to collect the information on the payment for invigilators so that we will now have the hours worked so that we can absolutely be sure and demonstrate the evidence that we are paying. In effect, you're saying to me that you don't know the effect of hourly rate that invigilators are being paid. That's the long and the short of what you've just said. What we have done is, at the moment, we are asking the chief invigilator to highlight to us when that has not been the case. What we are doing is formalising that mechanism. Based on the information that you have provided, £27.15 for a session that can last as many as far as when it is four hours for that session, the effect of hourly rate is below £7 when the national living wage rate is £8.45. That's correct, isn't it? If it may be in that one session, when you are calculating the minimum wage or the living wage, you look over the payment period and how much work has been done on that. You don't know if that's happening or not, do you? Yes. We pay the chief invigilators to manage the invigilators and to keep those records, and therefore they have the right and the ability to add an additional session fee if they think that that has not happened. I have statements from chief invigilators who are saying that the only way they can ensure not only that their invigilators are paid above the living wage but above the national living wage is by adding them on to sessions where they know that they are surplus to requirements and they will be sent home. Surely that is a wholly inadequate way to be paying people. Indeed, the sort of practice is surely that when we see them in the private sector, i.e. extensions to people's working day through additional requirements and duties which we condemn as poverty pay practices, we should condemn the private sector, but it's something that just frankly shouldn't be taking place in the public sector whatsoever. If you look over the pay period, some of those invigilators will be working less than the full session time but will be paid for the full session time. That is the point of paying them over the entire pay period. It's as if you are working a working week and you get paid at the end of that week and on some days you work three hours and on other days you work eight hours. The pay period is over the week. That is the conversation that we've been having with the poverty alliance that is the diet period. We pay monthly, but the diet period is six weeks, so there's essentially a payment at the end of probably June and then July based on the fact that exams are up to me. Sorry, it may be me and June. Can I ask why you're not just paying an hourly rate? Should that be the straightforward, transparent way to it and what good employment practice would dictate? Indeed, it's what happens in other exam boards in the UK. On 2014, we reviewed the whole approach to invigilation payment and talked to chief invigilators about that. There were two reasons why we continued with this session for you. One was that chief invigilators felt it worked better for them in terms of how they managed the process, but equally, if we only paid the hourly rate, many of our invigilators would be paid significantly less than the session fee. You've already admitted that you don't know what the effect of hourly rate is. We do know the session times and we know that 3% of our session times in 2015 were three or 3.5 hours before additional admin, so the four hours that you're talking about is about 3% of the diet. The rest of the time, people are being paid either £9.00, £5 an hour, or significantly more than £9.00, £5 an hour. Do you accept that if invigilators are working four hours or indeed more, once you've taken both prep time and extra time, that the effect of hourly rate for that session would be well below £7, and that is significantly less than £8.00? Do you accept that? In very limited circumstances where it is, because it's those parts. Do you only have to look at the exam timetable and see that there's a very significant number of exams that would take you past that four-hour period in the exam diet? We have done that analysis, which I can share with you, of the 2015 diet, and, as I say, 3% are in the three hours to three and a half hour band, which, when you add the half hour for administration, would take you over the four hours. Where do you have special circumstances? There's extra time, but that is a small part of the diet. A lot of the exams are half an hour, some of them are an hour and a half, some of them are two hours, some are two and a half. When you add in the admin element, which is 30 minutes, most people, the vast majority, are being paid £9.05 an hour or more. Over the four-hour period, they are all paid the living wage. With respect, you don't know that, and you have no way of knowing that. You have admitted to the programme that you have no mechanism for measuring or tracking that. I struggle to understand how you can say that with such confidence. I say that because one of the responsibilities of our chief invigilators is to ask us, is to highlight to us when we need extra, when they need to pay extra for people going over their time. What you have highlighted is the fact that we don't have that reporting mechanism, and that is what we're putting in place. I do agree with you that there is a reporting structure that needs to be put in place to address that. You'll send us the information that you've got to show that that's been done. I want to return to the previous theme of complexity and some of the issues that we heard back from the teachers. No one wants to undermine the SQA or calling to question your credibility in terms of the impact that you have on Scottish education, but some of the feedback that we have had does cause concern from teachers in terms of the unclear marking instructions, unclear complex course documentation. I know some previous questions that have alluded to these concerns expressed by teachers. How does it get to that stage? I just want to know how we get to the stage where teachers find what has been issued by the SQA very complex, vague and difficult to understand. In introducing any new qualification that has a significant change associated with it, we are required to provide information. One of the challenges that we've had and I think has been discussed at previous committees, both just for myself and others around the table, is the fact that teachers rightly requested more and more information. As you put more and more information out into the system, it becomes more and more complex, especially when you are putting it out there in individual documents rather than, as we have described, the good way that technology would allow us to be able to do that in the future. The whole issue that has been raised earlier around which version are we talking about is a real challenge. We have tried to respond to the way in which teachers have told us that they would prefer things to be seen differently. We have done that. In some cases, that has increased the level of complexity because that is another document out there. One of the things that we committed to do was to review our documentation and to streamline it. As we move into the revised assessment for the national qualifications that we are currently at the start of, that documentation will be completely revised and will be much simpler. It is an opportunity for us to address that issue. It has got there because of the timing at which the qualifications were introduced during the same sessions that they were being implemented, and that produced an additional challenge. Two more brief questions. First, do you think that we are in danger of sinking in a sea of jargon, and is there anything that the SQA can do about that? In your opening remarks, you said the phrase associated personalisation models, whichever that means. I know that the leaders of the SQA will be wrapped up in their day-to-day jobs of dealing with education and all the professional aspects of your job. In earth, do we demystify Scottish education and perhaps address the issue that communication could actually be improved? I think that there is a danger that we all sink in jargon. In fact, one of the things that happened when the committee visited SQA is that I kept reminding people not to spell things out, because by the nature of any business or any organisation, you use your own shorthand. I think that that is a very, very key point, and I think that the point that was made about how do we communicate with parents in a different way is another key point. It is about demystifying it. I think that it is about going back and looking at the original full set of documentation associated with CFE, not just about the qualifications and looking at what does that say, and making sure that we say it in a clear and concise way. The whole issue of personalisation and choice, just to explain that, is about allowing teachers to be able to teach a particular topic, like angular momentum, in a context that is of interest to that particular class that is sitting in front of them, as opposed to a particular context that SQA defines. That is a long-winded way of saying personalisation, but personalisation, if it is not understood, is not valuable. One of the big themes just now in education is teacher workload. How do you measure teacher workload and the impact of what the SQA issues has on teacher workload? Do you feel that, since you have been imposed, that the teacher workload has increased or decreased in Scotland? Part of the reasoning for us to go out and do fieldwork was to try and understand the nature and the causes behind what we were hearing from not just teacher unions, but teachers across the country of the workload that they were experiencing. That is what we did. We did that specifically to understand what part SQA's role played in that workload. As part of that research, we also identified other things that were playing a part in that workload that manifested itself in assessments. It is our responsibility to not only address the component that we are responsible for, but to continuously raise the issues that need to be addressed to reduce the workload associated with assessment as a result of other aspects from across the system. I take that role very seriously. It is not just for us to just sit and say, we have done our bit. It is actually our responsibility to keep that on the forefront. I think that the nature of the way in which curriculum for excellence qualifications, the idea that teachers could develop their own assessments to allow them to teach a particular subject in a particular context, that workload was significant. That is our contribution to that. I think that that is a balance against what we are trying to achieve with curriculum for excellence. I suppose that, for me, the way that the old qualifications were developed, where there were off-the-shelf assessments and there were off-the-shelf books and there were off-the-shelf everything else, was probably less of a workload for teachers. The challenge for Scotland is to understand whether that is the type of education that we want or that we want the ability to teach in, not to that straight jacket, because there is a negative to that straight jacket of providing the detailed assessments and the detailed information. Thank you very much. Maybe I should declare an interest as an ex-school teacher. I do not know if I ever used you, but I was a school teacher for 20 years when standard grades were introduced. I know that when there is change, some people in their profession might be a bit reluctant. They might be a bit concerned and might be a bit anxious. I have to say to you that having read the evidence that was given to us by a whole range of organisations and individual teachers, it is of a different level. Do you recognise that it is of a different level to simply saying, we know what teachers are like, they do not like change, they would rather have a book off a shelf rather than do their job, which frankly is the defence that you are presenting today? I did not mean to say that, that is not what I meant. Teachers have a very difficult job, teachers are there and they feel passionately about educating the young people in front of them. What I was trying to indicate was that by the nature of what curriculum for excellence was trying to do, there was a decision taken that there would not be detailed off-the-shelf assessments provided to teachers. It was very much around trying to move towards a use of evidence that was naturally occurring within the classroom. That is something that has not worked for multiple reasons, and that is the reason why we need to address that issue. As a result of that approach, it has been more complicated and more challenging for teachers. If your mindset is that the problem is that school teachers do not want to change, you end up in a position where you come to this committee, have not read, presumed with the same papers that I have read and still say that you have a strong working relationship with teachers. In what parallel universe do you have a strong working relationship with school teachers? I am not saying that teachers do not want to change. What I am saying is that the nature of the information and the nature of the structured provision by SQA was different as a result of the philosophy of CFE. I do not know if you have looked in any details what the themes are that are coming out from the evidence, which is not really about curriculum for excellence, but about not being able to access information, getting different answers to the same question at different times, and constant change. I would have thought if your core business was presenting a document that you should not just be waiting for feedback and then having seven different iterations of it. Perhaps there should have been something pretty solid at the very beginning. Unless you simply throw something out and then see what feedback comes back, I am in. I am genuinely astonished at the fact that there are so many criticisms here and what we are getting back is that it seems to be about process. In future, we can use the technology. Technology has existed for quite a long time. Can you explain it as an example of it? What motivates geography teachers not to tell you that there is a problem with the paper but for the society of geography teachers to present a document that is scathing about the geography paper? I would like to refer back to what I said earlier about the subject review reports. One of the things that we have done is actively engaged with each particular subject area. We have something called a national qualification support team, which is made up of teachers and of teacher unions and of other stakeholders involved in that particular subject. Every single one of those was engaged in trying to understand what was working and was not working on a subject by subject level. As a result of that conversation in geography, there was a set of actions that were identified that was published in the subject review reports. We do actively engage with teachers and we continue to engage with the Royal Society of Geographers and we will continue to move forward on making sure that we address the issues. However, as I said earlier, in some subjects, you will find two different teachers saying that there should be two different sorts of content in a particular course and that it is SQA's job to try to mediate between them. No, it is the same as you would talk to two different academics in a university. They will have a different view of the nature of what is important in their subject. It is your job to make that decision. That sounds again. That is why we pull together a group of people from across the piece to try and get a consensus view of what should be in that particular qualification. You have singularly failed to get that consensus. We have a consensus, but on a consensus you will always have people who are not happy with that consensus. All of the people who have written to the committee to express their concerns are people who, for whatever reason, are not prepared to fall in with the consensus that exists. The consensus that I think exists is that people do not think that SQA is working properly, that it is in the road of them doing their job. Not the people who are dragging their heels and don't want change. People who clearly are trying to navigate themselves around a system in the best interests of their subject that they care about and the young people that they are teaching and they regard the SQA as a block to that. What is your answer to that? It can't simply be that over here we have built a consensus and those people are just going to well, they don't get it. No, absolutely not. What I've been trying to say is that what we do is that when we get feedback such as the feedback that you have got, we look at it and we make changes. Unfortunately, when we make those changes we end up changing our documentation and that has a knock on impact on people's confusions and on these. People's concerns are not just about the documentation. If it were just simply that you put the wrong word in a document and there was a debate about what that word meant, fair enough that that can get sorted. There's three hours' worth of reading of documents and I took time to read last night so I'm sure my colleagues did. It's not people nitpicking. It's people saying there's a fundamental problem and what I get today is you don't accept there's a fundamental problem. When you were asked about the geography teachers you said you didn't know they had a problem with the paper. You hadn't been aware. Despite the fact they produce a survey, which I have never had anything like it in all my life. We have had a full-on conversation with the geographers and we fully understand the position and at the point of delivery of that examination we did not get feedback. That is what I was trying to indicate. We did not get feedback that there was an issue. That paper was to standard and that paper was aligned with the previous question papers that have been put out there. The geography teachers are wrong then. All of them, 75 per cent of them who said that it wasn't adequate, not a group on a committee, but 75 per cent of those surveyed who care deeply about their subject are saying there's a problem. Do you accept there's a problem or not? I have said that, as part of our research and our taking of information, we have published a subject review report on geography, which indicates the changes that we are making as a result of that feedback, some of which has come from the Royal Society of Geographers. I move on to Ross Greer. We have mentioned a number of times that the first question should be relatively short. There are clearly problems in the way that assessments are put together. We saw it last year with the higher maths paper this year with the national 5 computing. I am confused. A lot of the discussion so far has revolved around quality control measures, around resource allocation. You mentioned extra resources being allocated to stem subjects, and yet this year the most high profile issue was with the national 5 paper. Is it the case that it doesn't matter much extra resources that you are putting into your quality control structures are simply not working and there needs to be a more fundamental re-evaluation of them? First and foremost, we evaluate everything we do every year after we finish a diet. We go through and we look at what's worked, what hasn't worked, and we try to identify ways in which we improve. That is what happens every single year after the diet has completed. That includes the question papers and the procedures and processes that we undertake to deliver that 140,000 candidate certification. One of the things that we, particularly with the national 5 computing, is that one of the aspects associated with that was the computing language that was involved with Haggis. We have had subsequent conversations with the developer of that particular language, and we have reached a process improvement associated with that that makes sure that that issue will not occur again. Each individual problem is addressed at that particular point in time. Addressing each individual problem, though, as you point out, is not sufficient, we do need to make sure that we look at our quality control procedures across the piece. One of the reasons that we have introduced demarking is to improve the quality assurance of the marking procedures that we undertake. We are actively looking at how we continue to improve the quality. As I have said before, the Scottish Qualifications and the SQA Qualifications are seen as high-quality qualifications that are regarded highly by not only those candidates taking them but also the receiving organisations that take the candidates. When the issues with the national 5 computing paper came up, the SQA engaged in what I would describe as defensive PR and indeed said that the anecdotal evidence that you would receive from teachers was positive. I cannot understand how that was the case. The anecdotal evidence that I was receiving at that point was teachers contacting myself and I am sure colleagues as well, because they were so concerned by the paper, so concerned that the SQA wasn't taking their concerns seriously, that we were being asked to raise it in the chamber, which I did, I believe, as Liz Smith did as well. Why does the SQA believe that, at a time where there are clearly issues of teacher trust in yourselves, a defensive PR exercise that publicly dismisses those concerns is helpful? I think that if you remember back to that time, one of the things that I specifically said is that we had made errors in that paper and that we needed to address that. The initial response that I have got, the response is here, an initial statement from the SQA said that the paper met our course and assessment specifications that allowed candidates to demonstrate knowledge and understanding and anecdotal feedback was positive. You later altered that line and released a statement saying that the exam paper contained a small number of typographical errors. The initial statement was not that. The initial statement was very defensive. I take that point. However, I remember because I sat through that particular grey boundary that the way in which that paper was set allowed candidates to be able to demonstrate what they could do and we subsequently have had meetings with the Computer Society of Scotland and we have discussed whether there were any issues associated with the candidates not being able to do the question paper and the issues associated with that were felt by all of us that candidates were able to demonstrate what they needed to do. Although the errors were there, we were absolutely able to certificate appropriately with that examination. Taking aside the issues with the paper itself, what I fail to understand is why you were publicly dismissive of concerns initially. The SQA in your submission to the committee site a survey that you conducted, the results of which are entirely contradictory to the survey results that this committee received, to the overwhelming weight of the evidence in the submissions that we received. I cannot understand that. There are clearly issues of teacher trust in the SQA. Your public statements initially, when those issues come up, are dismissive, and the survey response that you have submitted is part of your evidence, I just do not understand where it comes from. The survey that we undertake is an independent survey, it is not an SQA survey. It is an independent survey that randomly samples our customer base and we do that on a biannual basis. Why do you think that it is so radically contradictory to all the other evidence that we have received? I think that one of the things that we do, because it is a random sample, you get a random set of views. Rightly, people who have concerns see the committee as a way to address their concerns, and I think that that is a very important role that the committee has. All I can say is that we will be going out again. We are going out for our field work and we will get the feedback from them, but we will also be going out again with an independent survey of our customers to get feedback on that. Is this random survey unrepresentative or is our evidence unrepresentative? I do not know the answer to that. I do know that there are teachers that have concerns about what we do, and I do know that we continue to try and engage with as many of those as possible, and try and address their issues and try and change what we do to make sure that we continue to address those issues. I move on to resource pressures and Ross Thompson. Thank you for your patience. Thank you very much, convener. I would also like to extend my thanks to you for welcoming us to your HQ in Glasgow and finding the visit there. It is really helpful, so again, thanks for your time last week. During that particular visit, one of the key themes that came through for me run about resources was how SQA is going through that intense period again of assessment redesign. That being on top of what business is usual in the commercial side of that as well, which is on top of your programme of transformation, touching on the IT issues, and I know that that is part of that. It has been so clear during this committee and you have acknowledged it yourself that, in relation to getting the new assessments in place for day 18, it is absolutely crucial that it is right and that there are no mistakes. I am aware that you do not have the same resources that you did with the curriculum for excellence because you had that development team and this is all now being done in house. The question is, in relation to pre-budget scrutiny side of things, what resources do you need, where should they be coming from, and do you have enough to make sure that we do get this right going into day 18? The decision was made to redesign the assessments in September. One of the things that SQA has to do because the complexity of the job that we do is that we have to have a detailed plan to understand exactly what we have to do when to make the deadline. We are currently in the process of doing that planning. That planning includes understanding what we are going to do, how we are going to identify what we are going to do, what resources we are going to require, and how long we are going to have to have those resources for when we need them. We are currently in the midst of that planning process and we anticipate that planning process to be finished by the end of November, beginning of December. We are expecting and fully anticipate that we will require additional resources because, if we remember, the people who help develop and deliver the qualifications are the teachers of Scotland, so we will be asking teachers to engage with us on that. That is going to be a challenge. As I said earlier, we engage with the directors of education, the head teachers association and the unions on the fact that, in order to develop qualifications in Scotland, you have to have the participation of teachers, and that is why we get so many of them working with us. The identification of the nature of the resources is currently being scoped. We are looking at making sure that we minimise that as much as possible, but you are absolutely right. We have to get it right because it is not appropriate to have any errors. It is a hugely challenging timetable because we all know that the session does not start in August as many people think it does. It starts straight after the exam cycle, so it starts in May. However, the nature and the content of the course is not changing, so teachers do not have to worry about the course changing. It is the method of assessment that is changing, and that is a difference from where we were with curriculum for instance. I cannot tell you exactly what we are going to need, but I can tell you that we will know exactly what we need. We will then be working very closely with our partners to identify where that is going to come from. Thank you very much, Dr Brown. I know that it has been highlighted by other members, but when it comes to mistakes that have been mentioned, particularly the typographical errors that we saw in the national computing exam, we know that those are mistakes that the SQA is responsible for. In that particular instance, it ranges from grammatical mistakes to questions that could not be answered. My question is, how much of those mistakes are related to the resource issues that you have identified? Are you confident that the executive management team has the skills in the leadership too? The executive management team is a very well positioned and a very strong level of expertise across the piece in order to deliver that. If you look at what SQA was asked to deliver in terms of curriculum for excellence, in terms of the qualifications, every single milestone, whilst they might not have been what the teachers had wanted but they were the milestones that were agreed by the CFE management board, all of those were met. In terms of the ability, the issue associated with errors is that associated with a resource issue. One of the things that we have done in terms of the work that we have identified as a result of our lessons learned exercise is that we have increased the amount of resource that has gone into the STEM subjects in terms of that space. Again, that is a very exam-heavy and question paper-heavy area, so we have increased the resources that we have put into that. I am not going to hide behind the fact that it is a resource issue that resulted in that error, because I do not think that it is appropriate to do so. Following on from that in relation to resources, one of the themes that came out from the meeting in the presentation was that an area of real cost pressure seems to be that. It was portrayed as a significantly higher level of support that you have to give to the system in relation to CPD and teacher training. It is something that we were advised during the visit with something that would previously have been carried out by local authorities. We know that local authorities are obviously going through their own cost pressures and reducing budgets. I would like to get an idea from you in relation to where the public money is coming from. Would it be better that local authorities take on that role if that is a significant cost pressure for you, or is it something that the SQA should continue to do but would obviously necessitate a significant grant from the Scottish Government? There are aspects of the support that we provide that need to be provided by the SQA given the assessment expertise. There are other areas that I will be better delivered elsewhere, and that is something that we need to discuss. That is an area for discussion that is occurring at the assessment and national qualifications group, to try to understand where is the best place. One of the reasons why we trained more nominees than we needed in terms of understanding standards was because we wanted to make sure that that resource was available in the local authorities. We are not precious about retaining that level of support that we have put in place rightly, because teachers have a right to expect a lot of support during the early implementation of new qualifications. However, where that is totally appropriate for the SQA to do that, we believe that we should continue to do it. It is something that we are doing in terms of subject-specific support, for instance, whereby we will offer every local authority if they wish us to come in and specifically talk about a particular subject. We will go in and run sessions for their schools. We ask, through our liaison officers who visit every school across Scotland, if there are particular issues in any school, we will go into that school and work with them on that. That is not necessarily best done by SQA. It is a question that we need to think about as a system. I have a supplementary following on from the question of the convener. As you know, we have always conducted an online survey as part of the committee, and 71 per cent of those respondents were teachers. 67 per cent of all of the respondents expressed distrust in the SQA by strongly disagreeing with the statement that our customers and users trust us to get it right for them. I have to ask Dr Brown why do not your customers trust you? The feedback that we get varies significantly. We have customers that are frustrated by how we engage with them, but we have other customers who are very happy. I think that you will also have seen a submission from one of the colleges indicating the level of support that we undertook, and another submission that cited SQA as one of the best awarding bodies in the UK. I think that there are a variety of views, and it is important that we do not only focus on the good news, but on the issues that are identified. That is one of the things that we try to do. I will ask a question on the quality assurance site. I know that questions have been raised in relation to geography and computing science. In my own area, I have had a constituent who has contacted me, whose son was predicted to get five A's. He ended up getting four A's in failed geography, despite achieving 92 per cent in their geography prelim. The appeal was rejected, and I know that his parents in particular are worried that some of that work may have been lost in transit to the marker. I am looking for reassurances from you on what you do to ensure that you have the quality of markers that is needed, especially given some of the submissions that we have seen already to the committee, in which markers feel that there is not enough information, where they feel that the information is there that is confusing. In relation to art and design, there was some real criticism that the SQA had not communicated well enough to teachers or pupils about what was needed or required. One of the things that we do every year is run markers meetings to specifically have the opportunity to meet markers face to face to make sure that, if they have concerns or if there is confusion or if they have questions about the nature of the marking that they are able to do that. In terms of e-marking, one of the other things that happens is that we have team leaders for the e-marking group. If we see a marker having difficulty in marking, that team leader will contact specifically that marker and have a conversation about what it is that you are finding difficult or do you need clarification on any of those things. When we originally set out with CFE and e-marking, it was anticipated that, at this point in time, we would not need to have as many face to face events. However, we have continued those face to face events because we believe that teachers get benefit from them and like them, the feedback that we get from those markers meetings is hugely positive. That is why we continue to run them because teachers run the system. It is very important that we make sure that we have clarity in that. Turning to the budget, looking at the variable factors that come into the budget, is that a sustainable financial model? That is something that we discuss regularly with our board and with our external auditors. The view is that we are sustainable in terms of the going concern principle, if you like, from the accounting perspective. However, it is quite difficult to predict acuratly what our costs will be, our budgets need to be, because we have not had the stability that I thought we would have coming into this particular year. In 1617, I had hoped that it would be the start of the end of the implementation of CFE, which has been the biggest change in the education sector for a generation. I had expected that 1617 would be the first year in which we would start to bed down and have business as usual. The announcement around the revision of national qualifications has meant that it is harder again to predict what our costs will be. We are working through that just now, but we will determine what our budget will need to be by December and have those discussions with the Scottish Government. In terms of sustainability, where we are trying to get to is to reduce the pressure on the public purse. We want to be able to minimise the amount of grants that we need to deliver the business, but we are trying to balance that with ensuring the safe delivery of the diet each year while we are making further changes. It is quite difficult to get to a sustainable position. The other component that was in our submission is the fact that we are aware that the fees that we charge for the qualifications in Scotland do not cover the cost of delivery, and therefore there is a funding gap associated with that. The fees have been in place since 2010, but there has been a fixed rate charged based on 2013-14 candidate numbers to local authorities. We are getting more candidates than was anticipated sitting our qualifications, so our costs are going up, but our income is remaining the same. On the component parts, you mentioned the question of fixed charge arrangement for local authorities. That presumably does not cover the cost at this time, so it is in effect subsidised. Local authorities have some certainty in their budgeting that that is what will be charged, but the uncertainty is then passed to you. If you like, the cost is passed up the line. It is essentially that the 2010 prices did not even cover costs at that point, so they have been held at 2010 prices. The 12-13 candidate entry levels were true for that year, but we have taken the risk of those varying, and what has happened is that there has been more of those. Is this a fixed fee per head or a fixed fee as a global level? It was what the local authorities were charged for their 12-13 candidate entries. We have charged them exactly the same thing for the last three years. So the candidates can vary. What sort of percentage increase has there been in the candidates? If we had charged on 2010 fees by candidate this year, we would have had an additional 1.2 million. I see that in your papers. What sort of percentage increase in the numbers of candidates have there been? I think that the approach that is not a responsibility of SQA, but the approach of two-year qualification structures would have resulted in candidates bypassing and candidates bypassing national 5, so those were rolled into our assumptions. We are seeing a significant if you look at the candidate entries at this diet versus last diet, I think that there was a very slight decrease in the number of candidates but not significant. Look at one specific area. There is reference here on page 5 of your document that you submitted. You are saying about non-commercial products to support niche sectors in the economy. Can you give us a little more information about that? As the awarding body for Scotland does an NDPB, we are asked and expected to develop and maintain qualifications that are not commercially viable. Because we charge on candidate entries, there are some that have very low uptake. Can you give us an example? Aquaculture, for example, is quite a small niche sector and niche part of the economy that we are supporting. There are other areas that we are supporting. The business that we should be doing, we are not in any way saying that we shouldn't be doing this. We are a national awarding body. It is our responsibility to provide qualifications for the industry sectors that exist in Scotland. Some of those industry sectors are quite small or require only a small number of candidates per year to go through a particular qualification. Things such as fish farming, things such as furniture making, we have qualifications in kilt making, we have qualifications in stone wall building that are a part of the fabric of Scotland. What it costs us to develop and deliver that is absolutely not covered by the small number of candidates taking those subjects every year. So, is it a significant cost that you are bearing there? How much? For us? It is significant. For us to break even on national qualifications and the cost of national... Which is a different piece. We can give you an estimate of what that means. It varies on a topic by topic basis. We can tell you whether Latin, advanced to our Latin, probably does not wash its face. English definitely does. I am not looking for a particular subject, so I am just looking to see what the cost your budget is in supporting those. Can we move on? Just looking again at how you prepare your budget, if I recall correctly in this document it says that 31 March is when you know how many students you are going to have coming through. Would that be correct? Yes, that is the final number. That is the final number. Yes, finalisation of the number. Of course, you have to prepare your budget in advance of that. Yes, yes. How do you do it? We do it based on estimates. We get initial indications of candidate entries in November and we need that to ensure that we can start to plan. However, teachers or centres can continue to change and move candidates as going forward for a final assessment or not right up until 31 March. We do not know the final numbers until 31 March. Put your error margin. Put your error margin. You must have some margin that you work to. What we do is we predict it in November. It is tended to be reasonably close, but I do not have an error margin as such. We know when we expect to come through and then we confirm that at the end of March with the Scottish Government. There is a fluctuation but it is not hugely significant. It is not hugely significant. The more years under our belt that we have of the new qualifications, the more clear that will be. Yes, that is true. There has been a changing pattern of presentation and that is starting to stabilise. Although we are not sure what will happen next year with the new assessments being introduced. We are still not in a steady state. We have an estimate. We can look back at what happened last year and get an estimate of what we think might happen in March. That is why when we talked to the Scottish Government about our budget requirements, we said that this is the draft as at January when we submit with our corporate plan. We estimate what we think it will be, but we continue to refine that after the 31st of March when we see what the actual costs are going to be. So your budget only becomes firm with the Government after the 31st of March? Yes, we continue to have discussions with the Government throughout the year. We tell them what we believe we will need for the full year. However, we firm that up with them in the course of the year. At the budget revision stages, spring budget revision or autumn budget revision, they will allocate additional funding to us in relation to what we need. I would say from what you are saying there that there is no prospect of SQA becoming self-sustaining. One of the things that we actively chose to do is we actively chose, given the focus that we absolutely have to have on curriculum fractions, a lot of the work that we were doing in that space was definitely put on the back burner. The issue for us now is to take advantage of where we have that opportunity to bring in surplus that reduces our dependence on the public purse, and it is a surplus, not the income that we need for us. Thank you very much. We are going to move on. Can I remind both members and witnesses to speak through the chair? Thank you very much, Gillian. I am sorry. Lash, there is quite a lot of that that I was going to ask you. You have already been asked, but it leaves me on to maybe coming off the back of Colin's questioning about your income. What is your income from appeals from schools? The post-result service is fee while Linda is looking for the number. I told you that it was specific. Yes, it absolutely does not cover the cost of doing the post-result service. Talking about money from the public purse, if you have a situation where there is maybe some inconsistency that has been mentioned, some of the responses around the assessments and the marking that inevitably leads on to more schools wanting to appeal decisions on behalf of those pupils. On the one hand, that has affected the public purse quite significantly because it has affected schools' budgets. There are obviously limitations on how much they can actually spend, and it is also, if you say that there is actually a shortfall in the fees that you get from schools and does not cover your efforts, how do you see that that could be solved? I think that the issue with post-result services is that post-result services are there to address anything that teachers feel is inappropriate whether a candidate has not got the qualification that they wanted. It is not an appeals process. It is actually going back and looking at how that candidate has undertaken and has done in their assessments. I think that the point of how those fees are paid, I think, is, as we have said before, is not a matter for SQA. It is a matter for local authorities and for schools themselves. One of the things that we see as we increase our policy assurance processes is that the success rate is declining. We have yet to publish this year's figures. That will be in the coming month. But one of the things is that we need to make sure that we continue to improve our quality assurance processes so that candidates get the right result the first time. That is not like the old appeal system. The old appeal system would look at, for example, pre-lim marks. It does not any more. Am I right? No. Why not? Because there was a review undertaken, a very extensive consultation undertaken, and there was a feeling across teachers as well as across many other stakeholders that the old appeal system was unfair. That is a much fairer system. It is a system that operates in every... The old appeal system was the only one in the world and no one else did it that way. One of the challenges that we have in the post-result services is that we do have special circumstances. I think that that is a huge benefit and a huge positive of the change away from appeals. Because if a candidate is truly, truly disadvantaged at the time of their exam, if they suffered a bereavement or if they are ill, we now, because of the nature of what we do, can look at the evidence from whatever part that school wishes to give us and we are able to certificate that candidate on results day. In the old system, we were unable to do that. We could only look at their pre-lim. If that pre-lim was a poor pre-lim, we could do nothing for that candidate. We have focused the requirements of the exceptional circumstances on those candidates that truly, truly need that specific support for their personal circumstances. One other thing that I'd like to ask about, and I know that it's not remarking really, isn't it, instead of appeals remarking of it? No, it is not remarking. So can you talk me through how it's done then? Basically, it is a debate that's had across the world as to whether or not the second marker is better than the first marker. This is about checking that the marks are correct and that means that you have a qualified marker go back and look at making sure that that marker has done a good job. If that marker has done a good job, there is no change. At that point, the candidate will be charged. If that marker has not done a good job and the candidate result is changed, there is no charge. There is no charge for exceptional circumstances either. So how do you ensure the quality of markers? So you're looking ahead at your plan for the amount of markers that you're going to need for next year. And there have been some criticism about the quality and quantity of markers in order to get these results out to the standards that you'd expect. What are you undertaking to ensure the quality of markers for next year? The aspect that I talked about earlier and I won't go over again is the way in which we train our markers and we have markers meetings. But the recruitment of our markers is critical. We ask our markers to give us an understanding of what their experience base is, and we have a set of criteria that says that you have to be a practising teacher or very recently retired and that you have to be teaching at this level and have taught at this level for a period of time. So that is the criteria that we use. So that's the initial recruitment. We do anticipate that we will need more markers in the coming session because we're now extending the amount of external course assessment, whether it's coursework or whether it's the exam itself as a result of the removal of units. So we are anticipating an increased requirement. One of the other aspects of quality assurance that we do is during the course of e-marking, we not only put markers through a set of scripts that they have to pass, so they have to effectively prove that they can mark to standard, but during the course of their marking procedure we will also seed marked scripts in there that they don't know are marked and we will monitor their quality. So that's happened over the last few years so there's a significant increase in the quality monitoring. Of markers now than there was before. Just my final question that this is picking up on one of the comments that was made is that the SQA has an annoying habit of making changes to assessments and examinations in mid-session. That's obviously going to have an impact on results. And also for making changes in mid-session through halfway through the teaching where I said here it was actually the higher history, important changes have been made to the way essays are marked in the final exam. For example, where essays were previously awarded for marks blah blah blah. So they had to advise students to the essays that they had written which had previously been a pass. They would now lose marks for. So that's a fairly bold criticism of the... One of the things we learned very early on is that we should absolutely limit the number of changes that we made during the session and that was one of the things. That's that balance between responding to what teachers say is an issue and making changes. So we try and make all changes now well before the start of the session so teachers know what's going on. As we're moving into next session the information around what the assessments will look like will be available to teachers in April, the end of April. That is our anticipated date. We haven't finished the planning yet but that's what we're hoping for. That will be just before the start of the next session. So we very much focus on making sure that we don't make changes. And I guarantee that they will not change beyond that point halfway through a teaching session. One of the things... The only time that that would happen is if there's something that makes it completely and utterly invalid. And that's not something that we are anticipating because, as I've said, the course is not changing. Thank you. Okay, can I just come in in the back of that then? The... Not just the quality of markers but do you sometimes have a shortage of markers and then are seeking markers nearer of the time where they're needed? Yes. In fact, the last session we saw that. Can I ask, given that you would have a rough idea of how many students would be sitting in exams, why you don't have a pool earlier so that that situation doesn't arise? We do have a pool. The issue that we have is the fact that we are seeing an increase in the requirement for markers. So whilst we have 15,000, we might need 16,000 next year or 16,500. So that's about recruiting and training new markers in that particular space. So we're already in the process of recruiting markers now for next year. So it is a year-round activity and a lot of people are very actively wanting to get engaged with SQA in this space. But you couldn't have been doing that last year if you were towards the end of the session, you were still finding yourself struggling for markers and you must have had a rough idea of the complexity and the need for more markers. Yes, we did. There were some markers that pulled out last year, a small number of markers pulled out and if that's a challenge in a particular subject, you will see specific subjects which are particularly small subjects or niche subjects where we are struggling for markers in certain places and those are the spaces that we looked at. And what we do there is we work with schools, we work with adders and we actively look at how we can make sure that everything is covered and that's what we did last year. Okay, thank you. I move on now to Joanne. Thank you very much. Can I just say first of all that I was very relieved when in response to Ross Greer you said that you were not just a focus on the good news. And I would ask that you, if it's possible, following this session, you actually look again at the submissions that have come in and respond in detail to some, I think, to some pretty serious questions there. I want to ask a bit about the equality of the current system and, you know, talk about the budget pressures on, you think you've identified some of those. You've made a number of decisions that are presumably partly education and partly because of budget pressures. Can I ask you how many state schools have taken the opportunity to put in what's not longer called an appeal, whatever term you used? How many state schools have used that? For last year, sorry, for 1314, we had 6,901 state schools, requests from state schools. And independent schools? 1,369. We provided this information, I think, to the committee at the end of the last session. Again, we'll be very happy to provide that when we result... So you've tested this proposal against issues of equity and justice and any call and impact assessment? We have because it is for the schools and the local authorities to decide how they allocate the fees. Right. So it's not an issue for SQA that local authority schools may not have the means to access the justice or the confidence in the system that independent schools have. That's not an issue for the SQA. The SQA is concerned that what we provide is an equitable, valid system. The way in which it is accessed is a matter for other people. And you have no view on whether there is a difference between access of resources of an independent school and a local authority which has no money. We talk to local authorities on a regular basis as to whether or not they believe that they should be. They are limited in the post-result services that they're putting in. I think one of the things that will be interesting to look at this session is the relative performance of the requests that have come in from private and independent schools and the state sector. On the question of national force and the decision that there should be simply a pass or a fail, I said to you earlier that I was a school teacher during the implementation of the standard grades and the most powerful thing that was decided in my view was certification for all. And part of my working life was getting young people from foundation to general. Do you accept that the new proposal undermines entirely the valuing of that group of young people? As I said at the beginning in my opening statement, the design and nature of the qualifications was agreed with the CFE management team. The philosophy of national force was very much that that would be a progression pathway for candidates who would ultimately leave and go potentially to college. The use of internal assessment and a pass-fail was associated with that philosophy. What we are doing at this point in time is we are going out and actively soliciting feedback from SMT, from senior management teams in schools, from teachers themselves and from pupils and from employers as to the value of national force and whether that should be a pass-fail, whether it should be internally assessed, etc. Because it is a conversation that absolutely needs to be had and the assessment and national qualifications group that the Deputy First Minister leads are also looking at that and we will be discussing that at its meeting in January. It is a very important area that I totally agree with you. You would agree that Professor Lindsay Patterson, for example, has given evidence to the committee who has expressed grave concerns about the implications of what now is expected around national force, around inclusion, opportunity, closing the attainment gap. I think that that is an area that we absolutely, as a system, definitely need to look at and discuss and decide what we want to do with. The two other areas that I want to just highlight, presumably around budget decisions, decision to remove as EIS highlights human scribes and support for people with additional support needs. Again, I worked with young people who were exceptionally bright, who were able but needed support through scribing, either a person with them writing what they said or reading the paper to them. You no longer do that. We do that for all subjects with the exception of where literacy component is included. That was a particular... So the scribes are there for all other subjects. Absolutely. The issue here was if you have a qualification that includes a literacy certification, it was important that we were able to do that. What we did in that particular case associated with the literacy edition was to go round to each of the individual schools and understand what their concerns were. Because the assessments are internally assessed, the candidates have a lot more flexibility in the nature of the assessment that they undertake and the time in which it is undertaken. We worked very hard to make sure that we absolutely did not have issues associated with that for special needs candidates. That is the initial issue that is being highlighted by the EIS. So it would be useful, perhaps, if you could provide us more detail with what the technicality is trying to do, because we would be gravely concerned, I think, that young people weren't able to access examinations because of a budgeting decision to remove the script. It absolutely was not a budgetary decision and the scribes are there for other subjects. I think that we would all agree that the last period of transition was really important in terms of education. Can you explain why, according to the EIS's figures, there has been a 500 per cent increase in certification beyond Scotland since 2010? One of the things that we talked about a little bit earlier was that if we can generate a surplus associated with the work outside of Scotland and if we are supporting Scottish Government's international agenda with the positioning of education on the international stage, then one of the things that we should be doing is doing international work and that is associated with that. With respect to positioning ourselves on the international stage by diluting the support and capacity of the organisation to deliver in Scotland, I wouldn't have thought that that would be terrible for our reputational damage, but would you accept, at least, as the EIS suggests and I find very compelling that there must be a dilution of your concentration on your main job around Scottish education if there is that level of certification externally? No, there is not. We look at this very specifically and we allocate resources associated with international work that are not resources that could be put in place in Scotland, but the committee needs to remember also that we undertake vocational qualifications in colleges, we deliver qualifications in training providers, in industry sectors, in private companies across Scotland for the benefit of the learners of Scotland. What we are doing is using that expertise and that knowledge to be able to provide an income-based, profit-based that allows us to do more in Scotland. It is not about using resources that could be applied to Scottish activities. I am very well aware of the range of areas where the SQA has a role, but I go back to the point that the resources, at least to explain where does this resource come from to deliver 500% increase in 2010 in external qualifications if it is not coming from your core provision? We entirely add additional provision and that provision is paid for by the income that we generate and the profit is also generated. Do you recognise that there is a concern among some people that the SQA has become an organisation that has a business model which creates incentive to sell abroad or beyond Scotland and the danger is that you are not focusing on the day job? All I can tell you is that that is not the case. Our major focus, the major activity that we are currently undertaking is curriculum for excellence. The other thing that we absolutely need to make sure that we do not lose sight of is our requirement by statute to support the vocational space in Scotland as well. Those are our two major priorities. The other aspect is there to try and ensure that we can continue to do that over the long term. And final, final point on this question of national forward and what I think is an utter injustice towards a whole range of young people who no longer have external certification. There's a real fear we'll go back to days of non-certification certificate of classes I taught and no resources followed it because there was no external examination. When will there be a conclusion to the work that you're talking about in terms of the impact or an assessment of actually how that's playing out now in terms of national fore and the aspirations of young people? As I said, that is part of the discussion that Deputy First Minister is leading at the Assessment and National Qualification Group and I'm not aware of that timetable. Okay, thank you. Thank you, Janet. I've got a smith forward by Tavys Scott. Dr Brown, could I ask you about some further comments that came through the Royal Society papers Mark Priestley's papers and also Lindsay Paterson's papers? I think that if I can sum it up, I think that there are questions as to whether the current national qualification structure at S4 is compromising subject choice and therefore compromising the ability of pupils to study the core subjects that require university entrance and the quality that goes round that. Professor Priestley makes that Mark that is a lack of dialogue about what is learned in schools and why. Official documents have tended to focus on skills rather than on knowledge and that our research suggests an absence of this sort of dialogue in schools. I think that they are all drawing a conclusion that there is a serious issue about the curriculum for excellence in its delivery that subject choice is being compromised and therefore that is leading to further problems in colleges and universities. Do you agree with those comments? I think that there are two aspects to that. One is what is SQA's role in that space. SQA, as I said at the beginning, our job is to ensure that we provide a course and a subsequent qualification that builds on what that learner has achieved and takes them to the point that they can move to the next level. So there is a fixed amount of content about both knowledge and skill and the application and the understanding that needs to go into that course. That creates a certain size. The nature of how that is applied within a school is a matter for that school, for that local authority in consultation with parents and learners. The way in which broad general education prepares a candidate for that qualification has an impact on that as well. As we talked at the beginning, part of the philosophy of curriculum for excellence was that we would get away with the two-term dash. So, if a candidate was absolutely going to get a higher, that they would work straight to a higher. And if they were a candidate that was better suited to a national five or a national four, they would work towards those qualifications. And I think that that is an area that is still a topic for debate across the country. No, I was standing in the fact that it's not your decision to say what courses are taught in schools. That's not for SQA to decide that. Have you had discussions, however, about the concerns which you have raised not just today but before, that there is a squeeze on the number of subjects as a result of the curriculum for excellence, which pupils can opt into. And that is seen by many parents and indeed by some local authorities and by schools themselves that it is compromising the choices that they make in terms of university entrance. Is that a concern that you have as chief executive of SQA and have you had discussions about that with Education Scotland with the other educational bodies? I think that's one of the conversations that Scotland as a whole needs to have. And yes, there has been many conversations about what are we trying to achieve with the outcomes at the end of the senior phase. One of the issues that has been tried to be addressed by insight, for instance, the measurement tool that the Scottish Government has put in place is that we should not be thinking about the number of qualifications at the end of a particular year but at the end of the candidates' time at school. And I think that the practicalities of that is something that needs to be thought through. Just to add the points that Joanne Lamont made about youngsters who might end up with nothing. And on top of that, there seem to be concerns about those who are doing the qualifications that their subject choice, which is so crucial to what they're going to do after school, is being compromised because the way the system is being run. Is that, when you say it's a conversation that ought to be had? I would hope that it has already been had because parents, pupils and staff are asking about it now because of the seriousness of it. I think that it's being brought home by constituents who tell us that in one school it might be 10 miles away from another school that they're offering fewer subjects than they are in the other school. Yet there's no rhyme nor reason for that. That surely is a concern that SQA must have, even if you're not entirely responsible for it happening. I think that we should all be concerned and we are concerned about positive destinations for students. One of the things that we... I'm not privy to all of the conversations that occur in this space because as I sound like I'm copping out, sorry, but it is not part of our remit but we do input into those conversations when we are available. I accept what you're saying. I think that that's true that it's not for you to dictate that. However, are you not concerned that there is this disjointed approach because it is something that needs a joined-up approach in terms of the ability of youngsters to choose their courses and be examined in them, which obviously is your remit. Those professors in education, highly experienced people, are making the point quite rightly in my opinion that the subject choice approach is not comparable with the statement about broad general education, that it's condensing the ability of those students to have a well-rounded education, particularly in what they go on to choose. That's something that I think is hitting right at the very strong traditions, let's be honest, of what Scottish education has been able to deliver in the past. The subject choice issue is becoming a big problem for curriculum for excellence. Is that correct? I think that it's a topic that is increasingly being discussed. Is it a problem, Dr Breen? Some candidates need a lot of subjects. Other candidates are ill-served by a lot of subjects. One of the philosophies of curriculum for excellence was that schools should be the best place to discuss that. All I can say is that the teachers should be the ones. Now the challenge is in a school sector are the teachers able to do that? Are they given the flexibility to do that? Or what are the other constraints on the system? The point there is that again what the submissions see on the very same point that Liz Smith is raising is that the SQA recommends 160 hours teaching and assessment time for their courses which, as one of the submissions says, which is impossible to achieve in a single year. Isn't that the concern? The building block that you've been answering is your requirement, forgive me, not your requirement, but your requirement for 160 hours in one year? That's absolutely true. What we're trying to do is take someone from the position that they're at at the end of broad general education and take them to the point that they can get entry into their next level. That has a specific requirement associated with the amount of knowledge, the amount of understanding, the amount of skills development. That is what defines the size of the course. Yes, we could have a half-sized course and I think if you think in England you have the A level and you have the AS, actually no longer in England, but you have something halfway between. If you want to get the full course you have to have that amount of knowledge, learning and understanding. And our anticipation in those notional learning hours is that an average candidate would take approximately that amount of time to do it. It is therefore appropriate that schools allow that amount of time to do it. If you're trying to do all of that in one year you are limiting the number of subjects that you can do. That's the reality of it, isn't it? That's what's happening right across every school that you've talked about. Or you are actually giving courses with as little as 90 hours of learning. That is a real challenge both in terms of work code for teachers because teachers are now not just teaching one 90 hour, they're teaching multiple 90 hours. But there are also huge issues for learners associated with that and the amount of information they're trying to... And at the other end... No, I agree with that. And the other end of that surely is that you'll tell me but I don't think there's a school in Scotland teaching three sciences in one year, is there? So all those kids who are good enough to do this because they want to go and do a science degree at university and that's... I appreciate your earlier point. It's only a small percentage. They can't do it. I mean, tell me if I'm wrong, right? I think there are some schools teaching three sciences in one year. So you still can do it? Yes. And how do we know... Who keeps a record of these things? How do we find this kind of thing out? That, again, we don't collect that information. I don't know whether Education Scotland does. No, they don't. I've asked them that question too. And we don't know as my... I think the challenge here is what we need to be doing is making sure that candidates are given the appropriate level and the appropriate time for learning, teaching and the requirement for assessment that they need to be able to be successful in that course of qualification and be successful in terms of the amount of learning they've got to make them successful in their destination. It's not just passing the hurdle of getting the qualification. Okay, can I ask... I'm not sure I fully answered that, but anyway, can I ask one final question? In all your evidence today about work load, the committee has cited the papers to you. You said that... I'm sure this is true that the SQA meet the Education Scotland and all these other organisations regularly. Do you ever get together and say, look together, this is what we are doing to schools? Yes. Do you? Really? Yes. Well, why then have we had all these submissions like this? And why, comparably, when we see Education Scotland, there's a similar number of vast submissions saying going through 1800 E's and O's and the changes of benchmarks which have just come out again. Do you not get together and say, right, you're doing this, you're doing that? How do we ease this pressure on schools and teachers? Yes, and that is why we put out the subject review reports because prior to being asked to remove the units, we recognise that our units were having a very detrimental effect in schools and we tried to address it. In the years time when we won't be receiving 142 pages or whatever, Joanne Lamont was citing at us on all this again and similarly with Education Scotland, it'll all be smoothed out for next year. No, we don't know because we now are on a path of removing units and we need to... So we're not going to follow through with what the subject review reports requested because those are very heavily focused on units. We are now no longer doing units so we will have, I think, feedback on the changes that we're implementing. Okay, thank you. Feedback, yeah. Good, thank you very much. We had two other subject matters, quality assurance and accountability in governance, but they seem to have been covered in the previous questions. Does anybody else have any contributions that they'd like to make before I round it up? Daniel, very briefly. I just really want to come back on... You said to Ross Thompson's question that it's important not, and Joanne Lamont mentioned this as well, it's important not to just focus on the positives, but then you didn't really provide any explanation on the negatives. And I think that this tab is just pointing out it's not usual for us to have this volume. And just without reference to your current processes, I think that most of your answers as you've talked about your feedback and current processes, in just broad simple terms, why do you think there's this very substantial and significant, regardless whether it's a majority or not, of teachers who seem to have lost confidence to you? And what, again, in broad simple terms are you going to do to fix that? Because I think that's what this committee needs to hear today. I think a lot of the negative views are associated with the way in which the qualifications have been designed, implemented and have worked. What we have done is try and we continue to try and understand why they have not worked the way we anticipated they would work, and we need to make sure that we remove the problems that we have created as a result of those designs. With respect, is that not a pretty damning statement? No, it's a non... The way that the exams are designed and delivered, isn't that what the SQA is for? And if you failed in that, isn't that a pretty significant statement? Well, again, one of the things that we tried to do with our research was understand the root causes, and I use the word causes as to why they haven't worked. There are aspects that are in SQA's remit and in our responsibility to address my responsibility, I believe, to Scotland is to tell people when we have not got something right, because I think that's the way that we should improve. We are introducing any new qualification from any awarding body anywhere. You learn what is practical and how things are actually operating in schools. We've had a lot of discussion about the nature of the curriculum and the nature of how things are operating in schools. There were other reasons why the assessments were not working properly. I cannot address them. All I can do is highlight them. All I can do is address the issues that the assessments themselves have caused. And in my opinion, that is something that SQA should be sharing, because we should learn from the things that we had assumed would work. Yes, we had conversations with teachers. Yes, we thought the feedback that we got would be that this would work. It has not, in certain cases, not across the board. The qualifications are absolutely not a problem in general. There were issues associated with aspects of the unit assessment. The course assessments were all fine. That was the feedback that we got from our fieldwork. I really just wanted to reflect that this has been evidently a very challenging evidence session for yourselves. I think that you will respect that it has to be that way, because, as a committee, we have a job to scrutinise and to make sure that things are as based as they can be. I do not think that I was lost when following on from the visit last week, which is why I wanted to come back in, because I did visit and I would say to other members as well that to take the opportunity to visit if you get a chance to see the amount of work that you are doing. What I have seen last week was a group of people who were dedicated, who are proud of where Scotland sits educationally and who are wanting to make a difference. I think that we need to hold on to that. What I would ask you to do is to take away what you have heard today around the table, take away the submissions that have been put in, and when you are doing your fieldwork and your other assessments, it is to be honest about it, and come back next year or whenever, and hopefully we will have another more positive discussion around the table if you are able to do that. I just wanted to kind of reflect on that. Okay, thank you very much. Do you want to respond to that? Can I just say thank you for that, because the one thing that you notice about the SQA when you walk in it, sorry, is everybody cares passionately about the learners of Scotland, and we do not like it when we don't do it right. Absolutely don't. Okay, thank you very much for that. I would like to thank you for your time, evidence happens over two hours. One thing that I would say is, and it's clear from the evidence that we've had today, no matter what the changes are next year, and you're right, there will be feedback, there's going to be changes, that's the way it should be, but there's two things that we shouldn't be getting the same response on, and that is your relationship with teachers and the communication that you have with teachers and parents. You should at least be able to make sure that you're communicating whatever changes are required and the need for them to the organisations in a manner that everybody can understand, because that does still seem this year to be an on-going problem. But once again, thank you very much for your time, thank you. That closes the public session.