 Buenos días a todos. Soy Jorge Berné, trabajo en el Instituto Nacional de Valuación Educativa. Ya me daba a conocer en el foro de nuestro Moodle, así que ahora me podéis poner cara. Bienvenidos a este curso y a esta segunda parte de nuestra primera jornada en este curso sobre la competencia lectora a través de la evaluación y, después de esta potentísima primera ponencia por parte del doctor Mata, vamos a entrarnos ya en una temática relativa de la evaluación que tiene que ver directamente con los cometidos y con el trabajo del Instituto Nacional de Valuación Educativa, que, como bien indica nuestro nombre y apellido, es la evaluación. Entonces, para ello tenemos a dos ponentes internacionales, que son Juliet Mendelovitz y Liz Twist, que son expertas de la evaluación, de la evaluación de la lectura y de la lectura en general. Una de ya se ha estado involucrada durante muchísimos años en PISA desde antes de sus comienzos, Juliet Mendelovitz, y la otra de nuestros ponentes de esta segunda mitad de la mañana, Liz Twist, está muy implicada en la evaluación PILS, que en España también se aplica y que evalúa competencia lectora en alumnos de cuarto de educación primaria. Entonces, le hemos dado este título genérico a ambas ponencias, la competencia lectora en la evaluación internacional. Vamos a comenzar con la que corresponde a Juliet Mendelovitz, la elaboración de evaluaciones a gran escala de la lectura impresa a la lectura digital. Juliet es directora de investigación y directora general del Consejo australiano para la investigación educativa en el Reino Unido, ASER, en el Reino Unido, como digo, y está especializada en el desarrollo de los marcos y de las evaluaciones a gran escala nacionales e internacionales, específicamente en lectura. Juliet ha jugado un papel fundamental y ha desarrollado un gran liderazgo en el desarrollo de la competencia lectora del dominio lectura en PISA, en el programa para la evaluación internacional de los alumnos de la OCDE desde el principio de los tiempos. Desde que empezó PISA hasta la edición de 2012, formó parte en el desarrollo de ese marco teórico y del dominio de la competencia lectura. Su trabajo más reciente incluye la conceptualización y el desarrollo del indicador de competencia lectura que se ha desarrollado en el trabajo conjunto entre la UNESCO y ASER, que tiene relación con los objetivos de desarrollo sostenible de la UNESCO para el año 2030, específicamente el objetivo 4.1, que bien conoceréis, que dice así que se asegurará que todos los chicos y chicas, todos los alumnos y alumnos completarán una educación gratuita, equitativa y de calidad que conduzca a resultados de aprendizaje relevantes y eficaces. En la actualidad, Julian es directora del proyecto de ASER en el Reino Unido que trabaja con el Gobierno de Escocia y que está dirigido a la evaluación online de la evaluación de la competencia lectura y matemática en primaria y secundaria, de la que también hablará en su ponencia. Sin más retraso, te cedo la palabra, Julian. Muchísimas gracias. Y Liz y yo hemos hablado de cómo esta idea de tener unos cursos para los estudiantes en la evaluación, en el asesino, sería muy útil en nuestros países también, ¿verdad? Sí. Ok. Así que voy a hablar con ustedes hoy sobre desarrollar la escala larga de asesinos en general internacional, un asesino internacional, PISA, que he trabajado en unos 15 años, y también algunas más recientes desarrollaciones en las asesinas largas que espero que sea de interés. Voy a hablar un poco de por qué hacemos asesinos, ¿qué es el punto de ellos? Voy a darles algunos ejámenes de asesinos internacionales y nacionales. Entonces voy a hablar en más detalle sobre PISA, el desarrollo de su escala y las asesinas en general, y finalmente, un poco de palabras sobre lo que está sucediendo ahora en las escala largas, una de las nuevas cosas que están sucediendo. Así que, primero de todo, ¿por qué asesamos el aprendizaje? ¿Por qué los estudiantes asesamos el aprendizaje? ¿Qué es el punto? Estoy seguro de que hay ideas sobre eso en tu mente. Hay probablemente dos propósitos principales para los estudiantes para hacer asesinos. Primero de todo, para coger información para el uso en reportar, para los padres, para los estudiantes a los estudiantes, para la escuela larga y para las autoridades locales. Y la segunda razón, una muy importante, para los estudiantes en prácticas, es para coger información sobre lo que los niños han aprendido, lo que están aprendiendo, donde están en su progresión en leer o matemáticas, o lo que sea el sujeto. Coletar información para ayudarte a los estudiantes a trabajar lo que los niños necesitan. Así que eso es realmente para los estudiantes. Creo que mi organización cree que es la principal propuesta de asesino en la clase. ¿Por qué los regiones y los países hacen asesinos? Bueno, también quieren reportar, para coger información de los diferentes niveles, para la provincia o la nación, para ver cómo el sistema está indo. Para, por ejemplo, dar recursos más fuertemente a un lugar o a otro, donde se ve una necesidad, o para hacer más desarrollo profesional, o para mantener un trato de cómo las cosas están indo a nivel. También quieren darles a los estudiantes, a lo mejor, información sobre las escuelas y cómo los niños están indo en las escuelas. Y, finalmente, otra razón que a veces escuelas o naciones, o, perdón, naciones o pequeños regiones quieren hacer asesinos, es para dar un cambio, para ayudar a las personas a expandir la noticia de dónde la educación puede ir. Hay un mensaje en inglés, un pejorativo del mensaje sobre el asesino y el asesino. El asesino, el asesino de asesino, es visto como una mala cosa, porque los estudiantes piensan que los asesinos son un perro, y que hacen escuelas y estudiantes hacer cosas que no quieren hacer y que no son buenos para los niños. Pero hay otro lado de eso. Si los asesinos son buenos y usados, ellos ayudan, no son un perro, son un beneficio. Y, a veces, las innovaciones en la educación pueden venir, porque los asesinos muestran lo que es posible. Así que, creo que todos deberíamos tener asesinos para hacer ese tipo de trabajo. ¿Qué es la participación de los países en los asesinos internacionales? ¿Por qué participan los países? A veces, los países no tienen asesinos nacionales, y creo que eso es el caso en España. No hay una asesina nacional de los niños, los niños de la escuela en primer y segunda edad. Es un modo de un país que gana información a través del país para que puedan obtener un sentido de cómo los niños van en una forma comparable. Hay también la razón obvia para hacer un asesino internacional, que es comparar a uno de sus países con otros países. Así que sabemos lo que estamos haciendo, pero ¿qué están haciendo? ¿Están haciendo algo mejor que nosotros? ¿Qué pudimos aprender de ellos? Así que esos motivos son los que hacen asesinos internacionales atractivos a los países. Y, de nuevo, hay una razón de que, a veces, promueven cosas en la educación, y, de nuevo, han producido algunas de las desarrollaciones en PISA para introducir nuevas ideas en las comunidades de la educación alrededor del mundo. Aquí hay un camino de hacer asesinos, aquí hay un camino de aprender que es importante. ¿Pueden asesinarlo? Y algunas de las áreas innovativas de PISA han sido diseñadas con esa idea en mente. Por ejemplo, la educación de los países nacionales en esta última edad fue una nueva área para PISA. En los años previos, hay una solución de problemas. ¿Cómo podemos asesinar la solución de problemas? Esta es una competencia muy importante para la competencia curricular que los niños necesitan. Así que veamos si podemos asesinarla y eso lo llevará a la mente de las personas. Así que esa es otra cosa que asesina a los asesinos internacionales y por qué países están interesados en ellos. Así que estas son todas las razones que justizé para por qué diferentes grupos hacen asesinos, asesinos de aprendizaje. ¿Cuáles son las conexiones entre ellos? Bueno, ese propósito de asesino para diferentes niveles es común a todos tres. Teachos, grupos nacionales y grupos internacionales. Así que cuando hay un asesino, hay siempre algún tipo de asesino y esas son razones comunes, aunque son para diferentes audiencias. ¿Pero qué es eso? Para dar información a las escuelas y a los asesinos o a los asesinos, es otro factor común, al menos en el nivel de la escuela y en el nivel regional. ¿Está bien? ¿Puedes stillar aquí, por ahí? ¿Puedes escuchar? Así que hay algunos propósitos comunes entre asesinos de escuelas y asesinos nacionales y asesinos internacionales. Y luego hay un paro que son, probablemente, más removidos de la escuela, esos son los comparos, hacer comparos entre diferentes regiones o escuelas o con otros países, no es tan preocupante para los estudiantes de escuelas y puede ser, a veces, un poco aprecioso. Y luego, el último que mencioné, introducir nuevas ideas. De nuevo, eso puede ser un componente de asesinos de escuelas, pero probablemente no el mejor componente. Como estudiante, quieres saber donde tus niños son, lo que han aprendido, no podrías usar asesinos para introducir nuevas ideas porque tendrías muchas otras formas de hacer eso en tus prácticas de escuelas. Entonces, algunos ejámenes de asesinos nacionales y asesinos internacionales, algunos de los que podrías saber y algunos no. En el nivel nacional, el primer asesino de gran escuelo fue implementado en el latino 60, un tiempo pasado, en los Estados Unidos, un programa llamado NAPE, Valuación Nacional de Progreso Educativo. No en el latino, es en realidad americano. Entonces, empezaron como asesinos de monitoración donde pudieron examinar a los niños de alrededor del país para ver cómo aprendieron y cómo estuvieron. Y eso ha estado continuamente por, qué es, casi 50 años, sí. Otro asesino nacional muy, muy diferente, mucho más pequeño y mucho más reciente, es una evaluación de progreso educativo en Afganistán, que mi compañera ACER ha estado trabajando desde 2012. Voy a hablar un poco más sobre eso más tarde. Ellos son dos asesinos de sample, que significa que no todos los niños en el país están asesinados a un pequeño grupo para dar una buena representación al país. Y la última en ese lado, el asesino nacional escondido de los escondidos, que fue solo introducido el año pasado, es un asesino de población. Entonces, todos los niños en cuatro niveles en el país tienen un asesino de una vez al año, y eso sirve de varias propuestas. De nuevo, voy a hablar un poco más sobre eso más tarde. En el escenario internacional, hay un número de asesinos regionales, no internacionales, pero países dentro de las regiones, así que hay uno que ha estado trabajando por mucho tiempo en África, para los países fríos, otro en África para los países de inglés, hay uno en el sur de Asia, y este es el laboratorio de Latinoamérica de la Calidad de la Educación, yese, tal vez has escuchado eso. Eso ha estado desde 1994, así que es bastante un asesino long-standing, y muchos de los países en África de Latinoamérica, países de español, tienen parte en ese asesino, cada tres años, creo. Y se usan para comparar, para dar un feedback al país. Y luego, el que probablemente hayas escuchado, PISA, Programa de Asesino Internacional, que empezó en el año 2000, para la primera colección de datos, y ha estado trabajando cada tres años, y el más reciente fue este año, en 2018. Y la última en esta lista es PIRLS, que Liz estará hablando de, que es para los 9 años, PISA es para los 15 años, PIRLS es para los 9 años, PISA es, tiene áreas coreas de leer, matemáticas y science, PIRLS es solo sobre leer. Así que voy a hablar en más detalle sobre PISA, y más sobre la leer que las otras partes, porque es un curso sobre leer. Y voy a hablar, en particular, sobre el desarrollo de la frámida, y las tascas de asesino, los ítimos. Entonces, este es un diálogo, que mostrae los años en los que PISA ha estado trabajando, fue el primero en 2000, con 32 países, y cada tres años, ahora, en 2018, creo que hay 70 países. Así que es todo los países de OECD, y obviamente, hay 35 países de OECD, creo, sobre ellos. Obviamente, muchos otros países han participado en los que no son países de OECD. El asesino fue originalmente diseñado para países desarrollados, aunque tenía una particular focuses. Muchos otros países han participado en los que están desarrollando países, con menos desarrollo de sistemas de educación, y el asesino ha sido modificado un poco, para dar cuenta de eso, de esa población también. Una vez cada 9 años, los tres centros centrales, de la lectura, de la matemática y de la ciencia, tienen un gran fóclico, ellos llaman el fóclico de los grandes domainos. Y en ese año, cada 9º año, hay una revisión de la frámida, así que los expertos van a la frámida y vean si es todavía clara y relevante, lo que debería haber añadido, lo que debería haber caído. Hay nuevos ítimos incluidos. En los otros años, ese mismo tema es asesino, pero solo con un poco de los ítimos. Así que suficiente para dar información a los países, pero no un reporte mayor. Entonces, para la revisión, había tres goas. Era el primer, el primer cabo de la frámida, como diríamos en inglés. Y luego, en 2009, hay otro gran review de la revisión, y un montón de nuevos test de desarrollo. Y luego, en 2018, hay una gran revisión. Entonces, puedes ver que el área principal de asesino cambia cada tres años, pero siempre hay un poco de examen que hace cada tres años. Mi compañero, ACER, de la Universidad de Educación de Australia, fue el PISA para el OECD para los primeros cinco ciclos, de 2000 a 2012, y eso es donde sé nada de PISA. Desde entonces, los últimos dos ciclos, un consorcio diferente ha estado construyendo PISA. Entonces, mi hablar es sobre los primeros 12 años de desarrollo, en realidad, los primeros dos grandes ciclos de la revisión. En el primer año, yo era un test de desarrollo, escribí preguntas en 2000. En 2009, yo estaba en contra del desarrollo de la revisión para el proyecto internacional. Y en 2012, no lo recuerdo más, pero yo he manejado todo el desarrollo de la revisión para PISA. Entonces, yo tenía bastante involvement en los primeros 12 años. Entonces, ¿qué es una reforma de revisión? Empecé con una reforma de revisión y desarrolló los ejemplos de test en el mismo tiempo, que era bastante un rápido turno. Una reforma de revisión es un argumento explícito y una discusión sobre lo que una reforma intenta acceder. Entonces, es importante empezar con una teoría sobre lo que estás intentando hacer. Y sus razones son para ayudar a los desarrolladores a crear una asesina, que es significativa y útil, para dar una discusión en común sobre el tema de la revisión, en este caso, para asegurar que hay continuidad de un año hasta el próximo, para que no te acese una cosa en 2000, algo diferente en 2009, que sería comparación difícil o imposible. Entonces, tener la reforma da un tipo de acercamiento. Y para comunicar la propuesta y los features de la asesina al público, a los estudiantes, a los escuelos, a los sistemas, a los padres y a los estudiantes. Entonces, la reforma nos permite hablar de la asesina en una manera que nos da sentido. La reforma de la reforma empieza con una definición del tema. Entonces, ¿qué es la revisión? No creo que yo realmente ha tried a articular lo que la revisión era hasta que empezé a trabajar en PISA. Y la definición de la revisión ha cambiado un poco desde 2000, pero no es muy diferente, así que ha tenido continuidad en el tiempo. Y aquí es la definición de PISA para 2018. La literatura de la revisión es entender, usar, evaluar, reflejar y engañar con textos para lograr sus objetivos, desarrollar su conocimiento y potencial y participar en la sociedad. Como se sabe, el OECD es la organización de la Corporación Económica y de la Developación. Así que ha tenido una mantra económica. Así que participando en la sociedad es una cosa importante para el OECD y para las naciones también. Entonces, el desarrollo personal, tener un trabajo y contribuir a la sociedad, son todos importantes. Hay mucha discusión, siempre hay mucha discusión sobre la definición, horas y horas de debate que van a eso. Y es muy cuidadosamente crecido y cada parte de ello se convierte en parte de la revisión. Así que, si veas estos bites en diferentes colores, la utilidad de usar, reflejar y tener interés en la revisión es parte de lo que fue. Y eso se refleja en diferentes partes de la construcción de la frontera. Los textos, los procesos cognitivos y las escenas, los contextos son todos tomados de la frontera para los ítimos actuales. Entonces, para los textos, tenemos la base del texto, la organización y la navegación del texto. Esto es especialmente importante para los asesos digitales. El formato del texto, si es continuo, no continuo o mezclado, y el tipo de texto, narrativa, discursiva, exponente o tal. Los procesos, que aparecen en la definición, han sido definidos como información ubicada, entendiendo y afectando y evaluando. Y los contextos, personal, público, educativo y profesional. Entonces, todos esos elementos son instanciados, representados en todos los ítimos. No todos, pero uno de cada de esos categorías es representado en cada ítimo para dar buena coverage de lo que es la leería. Por cierto, creo que este PowerPoint estará disponible para vosotros online, después del curso. Por lo tanto, no hay que dar notas, pero tienes que dar notas. Entonces, pensando en los textos en particular, como veis, hay tres formatos diferentes, continuos, discontinuos y mezclados. ¿Estás familiarizados con estos conceptos? Sí, algunos son. Entonces, continuos significa un texto que está en los parágrafos. Puede ser una historia o un artículo, pero son pros continuos y sentencias. Un texto noncontinuoso es un texto en el formato de tablas o grafas o diágrimas donde hay palabras, por lo tanto, no es lo que es la leería, pero también puede ser imágenes. Entonces, puede ser una combinación. Y un texto mezclado es un texto que tiene continuos y noncontinuos features. Entonces, puede ser un artículo con un reporte pro y un tablo, o algo de ese tipo. Entonces, cada texto en PISA se pide en uno de esos tres categorías. Y hay un balance de texto. Entonces, used to be, no sé qué es ahora, used to be, I think, 60% continuos, 30% noncontinuos y 10% mixed. Puede haber cambiado. Pero hay una tarjeta de prescripción para el desarrollo. Entonces, tipos de textos, estos son probably terms that you are quite familiar with. Descriptive, narrative, expository, argumentative, instructive, and transactional, which is probably a little less familiar. It is in English-speaking countries anyway. Transactional means things like letters or blogs where you are writing to get a response. And again, every text in PISA is classified according to one of those categories. I think sometimes in traditional language education systems there is a lot of emphasis on narrative when reading is being taught and not much emphasis on anything else and certainly not on noncontinuous texts. And one of the aims of PISA was to expand the notion of reading so that there was a more, a fuller account of what reading is to prepare young people for life in the world in which most adults, in fact, read more noncontinuous texts than continuous text. Here is an example of a text from PISA which you may know from outside PISA. Macondo, from 100 years of solitude. And that was one of the texts that was used in the very first assessment of PISA in 2000, this particular piece, and again in 2009 because some of the material is used again so that we can track trend over time. So if you look at this text, which text format do you think this one is? Remember the three were continuous, noncontinuous and mixed. Continuous, yep. Okay, and what about text type? Is it narrative, expository, descriptive? Not at evil, see. Bueno, okay. Okay, here's another text. Here's the answer. Here's another text. This one came from PISA 2000 and increasingly in PISA countries have been invited to contribute text material or stimulus for mathematics and science and sometimes questions too. But we ask countries to send in to the center things that they think 15 year olds would be interested in that they would find in their country was useful as a reading assessment and this one was sent in from Chile and it was turned into, actually it wasn't sent in like this, I'll show you what it was like originally. This is how it appeared in PISA in 2009. So thinking again about the text formats, continuous, noncontinuous and mixed, which one do you think this one is? This continuado, see, okay. And what about the text type? Narrative, expository, descriptive, transactional, hands up those who think it's narrative. Narrative, no, okay. Instructional, descriptive, feel like it right, yeah, it is descriptive. Yes, it's just a description of a system. There's no elaboration which would make it exposition, it's just a description, yeah. So it's not always clear cut though. It's sometimes it's not really straightforward to decide what a text is. So something we're thinking about. Now turning to the cognitive processes, every item in PISA has an alignment with one of these three broad areas and as you can see they're divided into subcategories as well. So the first one is locating information which means going into a text and finding something which is usually, literally there, you can find the words, exactly. Often associated with information texts or descriptive texts, not so often with narratives. The second category, the broad one, second broad category, understanding. This is a very broad category and in fact about half of the items in PISA fit into this category. So it's finding links between parts of the text, so integrating different parts of the text, making inferences from what is literally stated. And sometimes when there's more than one text, making comparisons and contrasts across the text or even within a text. So a lot of different mental processes are involved in that one. And the last category, which I think again was quite an innovative category for some countries, education systems, evaluate and reflect. So evaluating can be about judging the quality and the credibility of a text. Reflecting on a text can be again about thinking about the quality of the text but also about thinking about your own experience and what you know about the world in relation to a text. So it's moving outside the text itself to thinking about how it relates to your world. So again, for people who love reading and value reading, that's a really important part of the process, isn't it? So thinking about what in the text relates to your life, how your life relates to the text. And so that was also included in PISA. Not a large proportion of the items but some of the items address that cognitive process. So here's Metro Transit again, old friend. Here's one of the questions that was included in the PISA assessment in 2009. You don't have to find anything here. I'm not asking you to do the question but which of the cognitive processes do you think this one addresses? Is it locating information? Is it understanding or is it evaluating and reflecting in your head? Okay, so that's how it was categorized. Again, these ones can be subject to a lot of debate but it's accessing and retrieving information from a very straightforward text. Here's another one from Macondo. At the end of the passage, why did the people of Macondo decide not to return to the cinema? Now, I'm not asking you to read the text although some of you may know it already. Do you think that's a locating information question or an understanding question or a reflecting and evaluating question? Is all the information there in the text or do you have to think about your own experience or your judgment? That's how we classified it. So drawing an inference from and within the text. And here's one more example again from Macondo. Do you agree with the final judgment of the people of Macondo about the value of the cinema? Or explain your response by comparing your views with those of the people? So what do you think that one is? Which of the cognitive processes do you think that one addresses? Locating information, understanding or reflecting and evaluating? Third one, yes. Okay, so just to give you a bit of a feel for the kind of thinking that goes into creating these items, we needed to address all of those different areas of text and processes. So in the first year of PISA, year 2000, the focus was entirely on print reading. There was already thinking about digital reading and the definition in 2000 took into account there might be digital reading assessed later on, but we didn't try to do that in the first round. But by 2009, it seemed very important to have digital reading because it is such a large part of people's reading. Now, then, and even more so now and into the future, it's really taking over. So why assess digital reading? ¿Isn't it just the same as print reading? Only it's on the screen? Well, no, it has very special features. And here are some of them. So of course it's still reading text and children need to develop the basic skills of reading, decoding and making sense of language to read in the digital format as well as in the print format. But there are a lot of differences. So some of them are just tabulated here. So in the one hand you have the page of a book, web page is different. You have a book, a website. You have bibliographies and references to find out more. But on a website, you can just click on and go to your references straight away. If you know how to use the system. And one big difference between print and digital texts is that when you read a book, say, you usually read linearly. You turn the page, you read the next page. In a digital text, very often, you're going all over the place. You're sort of creating your own text depending on how you navigate. There's one area that I didn't put up here for the comparison and contrast and that is about the source of the text and the publication of the text. Generally speaking, in printed text, published texts in the print world, there's an author, there's an agent, there's a publisher, there's a bookseller and so on. They're all these different people. The publication has to go through all of those filters before it gets to the reader. And so there's sort of a bit of a process of evaluation, censorship, if you like, filtering that goes on. In the digital world, oftentimes, people can put up anything they like. There's no filter between the author and the reader. And that creates its own set of challenges. And I think tomorrow, Lalo Salmaron es going to be talking a lot about that area of digital reading. I won't touch on that again, but it is an important area. So in 2009, we introduced digital text as an option in PISA and 19 countries took part out of the 60 or 70 that were able to do it. Spain was one of them. UK was not one of them. Australia was one of them. So countries decided whether or not they wanted to do digital reading. By 2015, all countries were expected to do a digital reading component. And in 2018, I think there's only digital reading. So printers more or less disappeared. So this is what we said about the importance of navigation in PISA in the report that came out of 2009 study that navigation is a key component in digital reading. People who navigate well find the reading task much more straightforward and find information quickly. People who don't know how to navigate well struggle to make sense of the reading. So a lot of the focus on the assessment in PISA 2009 was on assessing whether children could navigate in the digital framework. So we were thinking about two elements, processing text, which is the same sort of skills as in print reading and navigation. And we designed items so that they assess different degrees of those things. So there were some items that had low text processing, just a few words maybe, and not very much navigation, maybe only one or two pages. Then there were texts that had a lot of words and not very much navigation. Text that had a lot of navigation and not many words, like buying a ticket for a cinema occasion. So there's not just the names of the films and the times and so on, but you had to go to a lot of pages to get that information. And then there were texts and questions where there was a lot of text and a lot of navigation. They were the most challenging ones. So we kind of designed the items so that they'd test all those different areas. I'm just gonna show you one example of a digital reading text from 2009. By the way, all the texts that I'm showing you and all the items I'm showing you are in the public domain. So there's a whole lot of material that is held secure so it can be validly assessed for trend. But these ones have all come out into the world. This was a unit about a 15-year-old girl who wanted to be a volunteer and she writes a blog. So the first part is about her blog. I would really like to contribute something to the world. So I'm thinking about what kind of volunteer organization I should join. These are my interests. And then her friend tells her that she should go to this page called I Want to Help, which has a whole lot of different volunteer organizations and allows people to find out more about them and select one that they think is suitable. So there's a whole lot of web pages attached to this, to this facing page, different website. And then eventually the student is asked to choose a volunteer organization on behalf of Micah and they send her an email to tell her why it's a good one for her. So that's the task is choosing for Micah, the girl, and telling her why she should go for it. So the last page is an email site with a key in their answer and that is what's assessed. But we also collected information about how they navigated. So the data collection included time stamps on every page that the student went to. So we could analyze afterwards without contributing to the score because it was a research exercise, analyze how efficiently they went from one page to another. And then we looked at the relationship between their scored answer, which was how they answered the question, which entailed going to all these different web pages, the relationship between that and which web pages they visited. And we found that there was a lot of diversity in their navigation skills. Some of the very best performers on the whole assessment went straight through and found exactly what was needed. Some of the worst also went straight through and probably didn't find what they needed, but they were also very quick and only looked at two or three web pages. In the middle, children had all sorts of different pathways and there's ongoing research to look at the relationship between the pathways that people take and what sorts of success they have in terms of finding a good answer to a question. So that was the kind of model that we used for digital reading in 2009 and that has continued, that kind of model is being used with some modifications still in PISA 2018. And I'm sure that there are more things happening that will ensue in later years. There's the question that I was just talking about. Okay, so what's happening in reading assessment lately in large-scale assessments? So this is back to the page I showed you near the beginning of some different assessments. And I'm going to talk about a couple of them in very briefly. The assessment of learning in Afghanistan since 2012. And this is a program, there are lots of, a lot of programs going on in Afghanistan, as you probably can imagine, to help in various ways with health, with the military of course, with housing and with education. Many organizations are working in Afghanistan and ACR was contracted by the Ministry of Education in Afghanistan to develop a learning assessment so they could monitor what was happening in different parts of Afghanistan. In fact, only 13 of the 30 provinces in Afghanistan were able to take part because of security issues, but that was still quite a useful collection. And it was a collection of year six assessments in 2012 and an assessment of year three children in 2014. So this is part of the year three assessment which was delivered on tablets on iPads. That was quite an adventure. So the reason for doing that was because it was easier to capture the information digitally than in print, carrying the papers around through the countryside was quite a challenge. And this was a challenge too, but this was what was decided on. And this is a little bit of the assessment of reading for year three children who are seven or eight years old. So they have a storybook, a digital storybook and they tab on these pages. As they click on each page, they get the continuation of the story. It's quite short, but it's got pages like a real storybook. And this little icon down here with the mouth on it, if they click on that, they hear the instructions. So sometimes they have to read the text, but sometimes they just have to listen and it's to see whether they understand the language as well. So here's a few pages of the story. They can click through the story, but they can hear it read to them, or they can read it themselves if they're able to do that. And then here's a question. Again, the answers are voiced. So by clicking on the mouth button, they can hear the question, the multiple choice question, and then they click on the one they think is correct to enter their answer. And that's all captured on the tablet taken back to Kabul and sent to ACR for analysis. So it's quite an interesting program that has got some legs. This is a photo of the pilot of this year three reading assessment. So the invigilator went into the village. He had children sitting around him. You can see his group of girls here sitting around having a look at how it works, having the instruction. And then here they are sitting and doing the assessment with their own tablets that were brought in. So that is quite an exciting part of the work that ACR is doing now. And we're hoping to develop that further over the next few years. A very different assessment is one that I'm working on myself in the UK for ACR. And that's the national assessment of children in primary one, which is four and five year old children. Primary four, which is nine year olds. Primary seven, the last year of primary school. And then secondary three, which is 13, 14 year olds and third year of secondary school. So all children in government schools in Scotland, and that's most children, are doing this assessment. They do it in reading, mathematics and writing. And it was introduced last year and it has some interesting features. As I just said, it's all the children in those year groups that are doing it. The main purpose of this assessment is to give useful information to schools and teachers about how their children are going. The teachers have to report nationally on attainment of their children, but most of the information that they use is their own classroom assessments. But they can use this assessment to help them to make the judgment. So it's not the only evaluation, it's a part of the evaluation. Two interesting technical features of this that we have helped them to develop are that it's online and adaptive. Do you know that expression adaptive assessment? It means that depending on how well the child is answering the questions, they move to different questions in the assessment. So this is a diagram of how it works. So in our model, the first 10 items are the same for everybody. They're actually different sets, but they are about all about the same level. Then if they do well, they go from A to B. And then if they do well on B, they go from B to D. So 10 questions each time. So each child is getting about 30 questions. On the contrary, if they don't do very well in A, they go to an easier set of questions, which we expect will be more at their level. And then if they do well on these ones, they go to a harder set here or to an easier set here. So it means that it's a different kind of experience for each child in getting the assessment. And because it's digital, it can be done instantly. All the questions are answered automatically and so it's scored automatically. So there's instant feedback within the system to send them on the appropriate path. The advantages of adaptive model are that, well, a possible advantage is that you can have fewer questions to get a precise measure of where the child's attainment is. However, if you've got a curriculum, a broad curriculum, and you want to actually sample from the curriculum, you need more than a minimum number of questions. To get an accurate notation, you can have 10 questions only, but to get curriculum coverage, you need more than that. And 30 is about the number that we think is reasonable. But what it does do is can give a better description of what the child can do, because we can use the items that the child has seen to say what they are doing well and what they're not doing so well. And if all the items are too hard for a child, all you can say is they can't do this, they can't do that, they can't do the other. Not very helpful or encouraging. And if the child is very clever and they can do everything, then you can't say, well, what should they be doing next if they can answer everything. So our aim is to have an assessment that pitches right for each child, according to their attainment so far. And another important ingredient is that it's a better experience for the child. So even the brightest kids, we hope, will be seeing things that challenge them and even the children who are struggling will be seeing questions that they can do. So this is another development that we are excited about and hoping to do more work with as we go on. So that's all I wanted to say for now. Thank you very much for your attention. Bueno, pues antes de pasar a la presentación de Liz Twist, si queréis plantear preguntas a Juliet sobre su presentación, sobre Pisa, sobre el trabajo que desempeña en este momento, ahora sería la oportunidad. So, do I ask in Spanish or in English? Well, what I think is that you trained the children at schools in order to do those exams. I mean, those international exams, they have got a specific structure. A specific structure. And I understand that the children at school are trained in order to face those exams in a proper way. I mean that they use the computers at schools in order to get the digital competence that they need to do those exams. I understand that. And to succeed. And on the other hand, I understand that especially they are reading text. They need to know strategies in order to find the answers more easily. And in the frame of time, because there is a frame of time, maybe in an hour they need to do three or four readings. I'm talking about especially the I ask exams. I think in a spot on, you have to hear, you know. And I have to read, I think it was three or four readings in an hour and to find the answers very, very, very quickly, you know. So, I think those international exams are okay if the teachers are aware that they have to teach in a certain way for the children when they have to take them, they succeed, you know. And they don't feel anxious in some way. And on the other hand, I just wanted to ask if you give the same meaning to assessment and evaluation. I'm not very sure about that. I don't know if the assessment is the process that you evaluate and evaluation is the final result. Maybe it's not exactly like that, but I'd like you to explain a little bit how you differentiate those two terms, evaluation and assessment. Thank you. Thanks. So, the first question was about preparation for exams or assessments. I think there's a clear distinction between something like IELTS, which you are doing for your own career, personal, and your result matters very much to you, and between assessments like PISA or PURLS, where the individual result is not important, it's not high stakes for the individual. PISA and PURLS, I'm sure, are designed so that children have a chance to practice the basic item formats and techniques of using the technology, whether it's digital or print, but we do not want people to rehearse or to practice doing the assessment. The assessments are supposed to be designed so that if somebody has the basic equipment of knowing how to click on a button or turn a page, they can do it without practice, and we really want to find out whether children are doing the kinds of things that are assessed in those kinds of assessments in their classroom across the nation, not whether the individual can answer this particular question. So I think there's quite a difference, and the aim is not for children to rehearse or for countries to coach their children for these assessments at all. Does that answer your question? I mean, I know that sometimes that happens even though that's not the aim, but it certainly is not the aim and there are little practice assessments within the assessment session to give people confidence about how to answer the questions. The other question about assessment and evaluation, I have to confess, I have never understood the distinction. I think people use those terms differently, interchangeably. Some people are very kind of fierce about their definition, and some people, for instance, in Scotland, we are never allowed to use the word test, it's assessment, fine, but I don't think it's universally a definition between assessment, evaluation, and test that one can apply. Do you have a different view? Yeah. We keep assessment to tests or exams and evaluation to measure the impact of something that's happened. So the outcomes and assessment would contribute to part of an evaluation rather than be the evaluation of itself. So we have a whole section of people who do work on evaluation, so we wouldn't use the terms interchangeably at all. There you see. Any other questions for now? If not, I will just ask them back to you. Oh, you're going to do it, yeah, yeah. Well, thank you, thank you very much. Juliet, it's been a pleasure to have you present in our program and thank you for your valuable insight and for making it so easy for us. All right, a continuación. Bueno, un aplauso, por favor, muchas gracias. Bueno, pues a continuación, seguiremos con la presentación del list twist. Os voy a dar unos detalles sobre su biografía y su trayectoria. Entonces, bueno, ella, su formación inicial es como psicóloga y también como maestra de educación primaria a lo que ha dedicado parte de su carrera, también ha trabajado como vicedirectora en centros de educación primaria y desde el año 1997 ha trabajado para la Fundación Nacional de Investigación Educativa de Inglaterra, donde ha coordinado las evaluaciones nacionales de lectura y escritura que cada año llevan a cabo alumnos de 11 años de edad. Desde el año 2000 trabaja para la evaluación PILS, Progress International Reading Literacy Study, la evaluación de progreso internacional en competencia lectora, cuyo último informe de 2016, Internacional y Nacional, se publicaron el pasado diciembre y es parte del equipo de desarrollo. Hay un equipo de expertos, es un equipo muy reducido que trabaja en la selección de textos, de las preguntas y todo el proceso. Entonces, Liz trabaja directamente con este equipo y en la actualidad ella es la jefa de evaluación, de desarrollo y de investigación de la Fundación Nacional para la Investigación Educativa Inglesa. Bueno, sin más retraso, I'll give you the floor list so that you can present about la lectura en los centros de Educación Primaria y la evaluación PILS. Muchas gracias. Thank you. I'm very happy to be here. My focus will be on PILS and there are some differences, I think, between PILS and PISA that it will be useful to bring out as well. And I'm sure, well I hope you've heard of PILS, but just a little bit of information about what PILS, the PILS survey is. It stands for Progress in International Reading Literacy Study, that's how we got the name. Although when it started in about 1998, 1999, we would see it written down with the letters P-E-A-R-L-S, which is the little, what's used in jewellery, the little pearl, because people hadn't seen where the name came from. So, it's the baby of the international surveys. PISA is probably the biggest one. TIMS, Trends in International Mathematics and Science Study, is, I think, the oldest. PILS is the youngest, but it's a beautiful survey, perfectly formed. And it takes place every five years, and it has this extra year. TIMS is every four years. PISA, every three, as Juliet said, although reading every third cycle. PILS has this extra year, because in reading development, you have to work on the texts as well. Y, then, once you have your texts, you can then develop your questions, your items. So, TIMS, every four years, Maths and Science, reading every five years. And it assesses just nine in 10-year-olds, and I'll talk about why it's that age group a little bit later. So, the most recent results were published last December. 50 countries took part. So, it's creeping up, slowly creeping up the numbers, and 11 benchmarking countries or jurisdictions, and they participate separately. Their results are published, but they're published separately from the main participating countries. And in Spain, there were 629 schools, 14,500 students, y, give or take, 670-something teachers as well. So, a big sample, in fact, of students in Spain took part last year. No, 2016 was actually when they did the survey, and then 2017 was when it was published. There's a gap between the assessment being completed by the children and the results being published, because you could imagine the amount of work que hay que hacer en el futuro. Y los findinges de la línea siempre son el máximo de los países de los más altos. Y nosotros tratamos de encontrar diferentes formas de presentar los resultados, así que eso no es solo la única cosa, sino la única cosa que se trata. Y esto es muy difícil, porque el press, el medio, siempre me gusta decir, como si fuera un table de fútbol, ¿cuál es lo mejor? Pero hay mucho más, para todas las surveys, debajo de esa lista de el país más alto y el más alto, hay mucho más información. Pero en la lectura, el ámbito, uno de los ámbitos es la profecación de la lectura, y eso es en una escala, una escala numérica que puede ser comparada de una survey a la próxima en cinco años. Los países individuos pueden ver el tren, la dirección, si su performances mejoran, si el 2016 su performances mejoran el 2011, por ejemplo, mejoran el 2006. Entonces, un país individuos puede encontrar que fue en la segunda posición en una surveya y en la tercera, en la próxima surveya, porque más o menos diferentes países han sido expuestos, pero su actual performances pueden still be improving. Así que es por eso que las tablas de la liga están limitadas en lo que dicen. Y nos dividimos en dos propuestas, y por qué puede leer, literar, leer, que es el narrativo, que se conecta con PISA, y información, leer. Y también tenemos cuatro propuestas diferentes. Entonces, hay algunas similares con la estructura de PISA, pero también algunas diferencias, también. Y en muchas formas, para reflejar mejor lo que estamos hablando de estudiantes de la primaria edad, que tienen, en todo, una balanza un poco diferente de la material que ellos read comparado a los 15 años. Y, solo para hablar con los pérdidos digitales, fue diseñada como una asesoría de papel. Estos son los libros que fueron anunciados a estudiantes en su escritura, y se completaron en el pérdido o el pérdido. Durante el tiempo, como la literatura digital de los niños, y como el acceso a la revisión online ha cambiado, así como con PISA, también ha cambiado. Y, en el siguiente survey, en 2021, así que cada 5 años, en 2016, entonces, la dirección muy mucho será que los países completarán las principales asesoraciones. No estoy hablando ahora sobre la asesoría digital. La asesoría de la asesoría digital. La asesoría principal que está ahora en el pérdido será en la escritura. No será asesorando las escrituras digitales, eso es, pero será asesorando la comprensión, pero será asesorada, será la desgracia digital. Y cuando veas algunos de los pasajeros y los ítimos, puedes imaginar cómo eso será. Y hay también lo que llamamos E-PURLS, que es un asesoría separado de la competencia digital. Y eso es cómo efectivamente ellos readen websites y pueden navegar. Y algunas de las escuelas que Julieta ha hablado. Y, eso fue asesorado en 2011. Y, si solo google E-PURLS 2016, E-PURLS 2016, podrás ver que hay dos bloques de E-PURLS, uno llamado sobre el mar de la tierra y uno sobre el primer doctor médico, el doctor Elizabeth Blackwell. Y ellos muestran cómo conceban de E-PURLS que son las habilidades de niños para navegar websites y por un avatar, un pequeño símbolo. No voy a hablar sobre E-PURLS hoy, pero es la dirección, como con PISA, la dirección de viaje es tener a los estudiantes completando E-PURLS a través de un tablero o un PC. Entonces, por qué este grupo de la edad de 9 a 10 años, bueno, la razón de enfocar en esto es que en muchos países, y, por supuesto, siempre tendremos que hablar de la adverigencia, lo que es overall, en muchos países, por esta edad, es sobre el cuarto año de la escuela. Los niños han, por la mayoría, aprendido a leer. Se han aprendido a decodir. Obviamente, continúan practicando y siendo más competente como llegas más viejos. Pero por la edad de 9 años, más, no todos, pero más niños han aprendido a leer textos apropiados de edad. Podríamos encontrar cosas que no podríamos leer, pero cosas que esperas y que los niños pueden decodir y tener un entendimiento. Y ahora, comenzan a usar ese entendimiento para leer, usar el entendimiento para ayudar a aprender a través del currículum para que puedan ver los libros de información en la geografía. Pueden ser instalados para hacer una ciencia y la maestra puede enfocar en la ciencia en vez de preocupar sobre su habilidad para leer. Así que, por eso, este es el punto en el que no es todo, pero más niños han aprendido a leer y ahora utilizan esa habilidad para accesorizar el currículum. Puedo mencionar lo que usamos para llamar el currículum y ahora lo llamamos la literatura de Pearls. Entonces, lo que sucedió en los primeros años de Pearls, 2001 y 2006 era que fue muy claro que algunos países estaban muy cansados de participar, pero sus estudiantes realmente, a la edad de 9, no estuvieron reading muy bien, a la mayoría de ellos. South Africa fue un muy buen ejemplo. Morocco es otro. Y así, la información, la información que ellos pudieron obtener de Pearls fue muy límica porque no era muy preciso. No lo contaba muy bien, porque muchos de esos niños realmente no pudieron responder ninguna de las preguntas o suficiente de las preguntas para obtener una mesura, que es, lo que Juliette decía, la razón por que el test adaptivo potencialmente tiene mucho valor. Entonces, introducimos un asesino que fue, esencialmente, más fácil para 9 años. Y, en más países desarrollados, seguramente en España, en el UK, es un poco más fácil para 9 años. Ellos tendrán casi todo lo que es correcto, así que no te dió suficiente sobre sus habilidades de reading. Pero en algunos países, tiene el nivel correcto. Entonces, tenemos la literatura de Pearls y creo que es dos años debajo de la Main Pearl, así que probablemente trabaja por 7 años en España y en Inglaterra. Pero, es... En algunos países, es una mejor medida de la performances típicas de los... los niños que están haciendo Pearls son 9, 10, y, en realidad, en South Africa, son 11 y 12 años. Entonces, hay Pearls y entonces hay una confusión que se llama Pearls literatura. Como Juliette mencionó, el... cómo cada de las surveys describen lo que es más importante. Es un framework para Pearls, lo que tengo en mi bag, pero es publicado de nuevo, y lo revisamos para el 2021 en el momento. Pero, lo definimos lo que piensamos, lo que piensamos es para el grupo de la edad. Entonces, esta es la definición que tenemos para Pearls. Y, al momento, creo que no nos parece cambiar, especialmente para el 2021, pero para la discusión. Entonces, es muy similar a Pisa. Yo creo que esta es la comprensión y la usación de la comprensión de por qué la comprensión es importante. La una gran diferencia que voy a hablar un poco más tarde, es... es la única que menciona que la comprensión también es para la disfrutura. Y... creo que esta es el centro. Esta es realmente lo que esperamos que los niños desarrollarán las habilidades, pero también verán que la comprensión es no solo como parte de su habilidad para lograr una vivienda, para desarrollar y para creer, pero también porque ellos decidieron hacerla. Y eso es realmente critical. Y en las escuelas primeras, cuando mucho, cuando están aprendiendo a leer, no... no alguien siente aparte de leer con ellos, pero niños practicando leer y leer en casa, entonces queremos que se motiven y disfruten leer, porque esos los niños que se convertirán los readers fluentes cuando están motivados a leer. Entonces, eso es cómo percibimos, Pearl. Entonces, sí, la comprensión es de valor para el individuo, para participar en la sociedad, pero también para su propia disfrutación. Entonces, estos son las tres cosas que voy a hablar de hoy, cómo vamos a leer y leer en Pearl. Y entonces, solo una de las información que tenemos desde el survey de 2016, que fue publicado en en diciembre de 2017 sobre la experiencia de España en ese survey. Y también, una de mis intereses sobre qué, desde el survey, lo que sabemos sobre los niños, lo que se cree sobre leer cómo se siente sobre ello. Entonces, debemos explicar cómo se conoce. Una de las cosas es que tenemos cuestioneros, tenemos datos de obtenimiento, tenemos las asesuaciones de ellos, pero luego tenemos una serie de cuestioneros que se han hecho a los estudiantes, ponen el asesuamiento y luego hay algunas preguntas, todo tipo de cosas, lo que llamamos información background, que incluyen obviamente su edad, pero también incluye su género, incluye si les gusta leer y hay una serie de cuestioneros sobre eso. Nos preguntamos, en Perles, nos preguntamos cuántos libros tienen en su casa, porque eso es lo que llamamos una buena proxie, una medida de prevención para su estatus socioeconómicos, la advertencia o la desadvertencia, la circunstancia de su familia. Entonces, hay más información que tenemos, que nos hace background. Entonces, estos son justos dos ejámenes de pequeños libros, son extractos de Perles, de nuevo, estos son publicados, así que no están en el momento. No son en 2021, Perles, son open para las personas para usar y para tener una idea de lo que la investigación parece. Y Perles, lo que los estudiantes, los estudiantes no se ve bastante en esta forma, se ve, se ve un booklet en el papel o se ve, se ve en la pantalla, pero se ve el texto primero y luego, se ve el texto y luego se ve las preguntas. No se ve una página con una cuestión, una página y una cuestión. Se ve eso en Perles Literal, Perles Literal, la diferencia es se ve una página y una cuestión, una o dos preguntas. Entonces, en este caso, un poco sobre los insectos en la ropa, en la ropa de ropa, es información sobre insectos y luego, se ve tres o cuatro páginas de información, se ve en los diagramas y luego, responden las preguntas. Entonces, esto es uno de los pasajeros de información. Hay cuatro pasajeros de información y hay cuatro pasajeras literarias o narrativas. Y el uno en la derecha, el top, el extra, una historia de fantasía sobre una chica que fallece a dormir y, bueno, el final, ¿es un sueño o no? Nunca he decidido, pero creo que el Crocodile está intentando romperse en su edificio, como tú lo haces. Y llaman la historia y luego, tienen una serie de preguntas sobre eso. Y los estudiantes tocan dos pasajeros, así que pueden tomar una pasajera de información y una pasajera literaria, dos pasajeras de información o dos pasajeras literarias. Es randomizado en un clasificador. Entonces, los estudiantes que estén en la mesa estarán haciendo un nuevo asesoramiento a su vecino. Y cada asesoramiento, cada bloco, entonces, el bloco de inserción llevará aproximadamente 40 minutos. Entonces, alguien viene a la escuela, está con este clasificador de 9 o 10 años, distribuye los booklets y tienen 40 minutos para hacer uno de los asesoramientos. Y hay sobre 13 preguntas, 17 marcas, más o menos. Y entonces, tienen un pequeño asesoramiento porque sólo 9 o 10 años necesitan un poco de energía y luego, van a hacer el segundo asesoramiento, que es diferente en un diferente sujeto. Y luego, tienen un pequeño asesoramiento, a la noche, y luego, van a hacer información de ellos en el día que han hecho su asesoramiento. Entonces, sólo para dar una explicación de los tipos de asesoramiento, tenemos tipos de preguntas. Entonces, el uno en la izquierda, los insectos, lo que es el propósito del artículo. Entonces, esto es intentando ver a uno de nuestros propósitos integrantes, muy típicos para información. Y el uno de nuestros propósitos integrantes, es preguntar sobre el carácter de Ana, la niña, y te describas lo que es como y dar ejemplos de lo que hiciste. Eso es muy típico de los tipos de preguntas que tenemos, algunas de las preguntas más difíciles que tenemos en Pearls y un poco más detrás un poco más tarde. Entonces, por la mitad de las preguntas hay varias opciones. Entonces, cuatro opciones y los estudiantes piden una de ellas, no hay necesidad para ellos escribirla. La otra mitad es alguna forma de respuesta constructiva. Entonces, esto puede ser muy corto solo preguntar una palabra o una frase. O puede ser como el que viste. O sea, tiene cinco opciones. Entonces, eso requiere estudiantes para dar una respuesta. Y, por supuesto, estás en este territorio de, estamos asesinando la receta, pero preguntamos en escribirla. Así que, siempre tenemos que leerlo en mente. Es un asesino en la receta, pero la respuesta para algunos de estos es en escribirla. Y cuando está escondida, es importante que entendernos que no estamos escondiendo cómo correctamente correcto la respuesta es. Es, nos da credición para su intento de expresarse y estamos buscando ver qué es lo que están intentando decir, porque no es un asesino en escribirla, pero estudiantes responden en escribirla. Entonces, uno, dos o tres marcas, las respuestas, incluyendo, como digo, las respuestas, pero no estamos buscando bien construyentes parágrimas con capitaletras y puestos y puntuaciones. No es el punto, es las palabras que utilizan, es lo que están intentando decir que vamos a esconder. Entonces, por unos 40 minutos para cada de los blocos, y luego por unos 30 minutos para las preguntas. Ahora, esta es, para Pearls. Entonces, hay... At the top, tenemos Pearls, la principal surveya, solo el primero de esos tres labios. Entonces, tenemos Pearls Literacy, que es la material que es un poco más más fácil para permitirme tener esta buena medida de algunos países donde la educación es aún desarrollada. Y luego, tenemos Pearls. Entonces, hay algunas diferencias. Entonces, Pearls y Pearls Literacy tienen about half the material on literary texts, which in Pearls is essentially story. We don't have any poetry because it's translated into 40 languages and you really can't do justice to poetry if you translate it into 40 languages and then ask questions. So there's no poetry. So predominantly narrative in the literary, purpose and acquiring use information. So about half and half and that's for Pearls and for Pearls Literacy. For e-pearls, which is our digital assessment, it's all information. So we're assuming, I know people read on kindles and so on, but that's still a continuous text. It's just replacing a paper, page with an electronic page, but it's not digital reading as we see it. It's just translating one to the other. Digital reading involves much more than that. So, for e-pearls we only have information text and I said the two that were available to look at are one about the planet Mars and one about Elizabeth Blackwell. And then we've got the reading processes. So first one, which has about half the Pearls Literacy items are retrieving information that is explicitly stated in the text. So you're looking at you're going to the right place in the text and you're extracting a particular word, phrase or bit of information and that's about half of the easier part of Pearls and 20% roughly. This is overall any one particular block may be slightly different but it's about 20% of e-pearls and Pearls, the main Pearls. The second one making straight forward inferences. So this is what competent readers are doing all the time without even being aware of it. They're making associations as they're reading they're linking bits of information together that help build up their picture of what it is that they're reading. So that's about a third of Pearls and e-pearls and about a quarter of Pearls Literacy overall. And then the big difference is that the other two interpret an integrate and the evaluation and critique elements which are much more to do with the whole text, the whole block of information or the narrative read there's rather less in Pearls Literacy. It's not although there is a hierarchy of skills I'm very wary of saying these get harder because it does depend on the text that you're reading. And I use it as an example someone gave me an advanced chemistry textbook and asked me to extract a bit of information I probably couldn't do it because I'm not a chemist I might be able to read the words technically read them decode them but I wouldn't understand them and I probably wouldn't be able to answer a question but to a chemist it would be probably straightforward they would retrieve the information so it does depend on what you're what you're reading but that's the overall shape of Pearls so we expect the children to be demonstrating quite different skills depending on what they're reading but each student will demonstrate different different skills as they go through and I should say that we are participating countries to submit texts for Pearls so we have a lot it's quite an iterative process we have a lot of reviewing of the texts at the moment I think we've got texts really that are in Pearls are about to be used from all over the world very different they get reviewed and developed they're translated into English but into American English and then that the review takes place in English and then for use they get translated into 40 different languages so as just to say the types of texts we have as I've said in literary ones we've got fantasy texts myths and fables contemporary narrative we've been trying to look for a play script not found one that we like at the moment in information texts all sorts of different information texts I think we haven't got one as sparse anything as sparse as the metro map that was shown in the PISA I think because we probably don't that's not the sort of text that you would expect a 9, 10 year old really to be reading and coming across in school we've used maps but they're set in a context we've had a map I think in the very first one Pearls 2001 we had a map that was a family wanted to go on a cycle ride and they had to hire cycles and then there was something about that we've had a map about going walking so they're putting a more familiar context for 9 and 10 year olds because that's the context in which they're likely to have come across those sorts of texts I just wanted to show now an example of what we call the standards in Pearls the benchmark so the surveys don't have passing and failing a student can't fail Pearls that notion doesn't exist and we're not trying to assess an individual student it's not designed to give a score for a student it's designed to give an overall measure of a country's competence rather than an individual student but the only way to get that is to assess individuals but they do want a 14 different booklets so we want to know the competence of all these students and put that together but what we can do is look at this overall scale that we have of reading competence and say there are certain at certain points on the scale this is the sort of thing we can say about what students can do at that sort of point on the scale so what's called the intermediate benchmark which is a a point of four seven, five I should say that for those of you who aren't aware Spain's national scale score the achievement score was 528 in 2016 on average overall score with all the analyses was 528 I think so this is rather below what is typical so this is what students scale looking at four seven, five on the scale this is what we expect are the skills being demonstrated and you can see there are some limits beginning to beginning to recognize language choices so thinking about why a writer did one thing only the start of that sort of skill which you can imagine it's a higher order skill it is more challenging and in 2016 about 80% of students were achieving this could do this achieving this benchmark but to give you an idea of the range that we get and in fact I think it does indicate why we needed pearls literacy the easier pearls so in South Africa there were just 8% of students who reached this benchmark so if you imagine you're teaching students of this age and just 8% had reached this benchmark compared to 94% in the Russian Federation so that's the enormous breadth that you get when we talk about primary reading towards the end of primary school what students can do and 8% in South Africa 94% in the Russian Federation and of course a lot around the 80% mark which is most students can do this and this is not the lowest I should say there's a low benchmark lower than this one which and the students who who are not reaching that benchmark those are the students who are really going to be struggling who have not really at that point really learnt to read independently and it was international average about 4% of students didn't reach that Spain about 3% but those are your struggling readers and I haven't got here the number for South Africa but it or some other it is worryingly high the number of children who have not learnt to read at the age of 9, 10 so this item is pretty typical of the sort of item that children can do at this so in Spain 82% of children could do at this sort of item and the reason for having this format, this is quite an interesting one so I think tomorrow you're looking at Macy and the Red Hen, it's one of the texts with some items that we used but this is the sort of question the question in English how does the author show you what the Red Hen is like the 9, 10 year olds struggle to explain this in their own words it's quite hard how does the author show you quite difficult to begin to construct an answer and we don't want to stop those who know it in the head but can't write it down can't get the words together so multiple choice means that they're provided they have to recognise that and it makes it much more accessible it is easy for them to do it now because we've given them abilities and they're recognising it so the type of question that they couldn't give an answer to in their own words they can indicate that they understand it now because they're recognising the answer and that's one of the while people sometimes moan about multiple choice because children can't express themselves what it also does is give them a step up to that type of question that they very few would be able to articulate the answer and obviously it would have to be scored accurately as well so in this case giving them a choice and enabling them to recognise something they couldn't express themselves is particularly useful so that is quite a typical intermediate type of item so internationally 79% students could do this Russian Federation 96% Morocco was the lowest 34% so the next benchmark is at scale .550 so quite a bit higher so this is a more challenging more accomplished set of skills that students have at this age so when I said that they previously we had that they were beginning to recognise the authors language choices here we've got recognise the use of some language features metaphor tone and imagery so figurative language is now becoming something that they can begin to recognise so in this case 39% of Spanish students reached this benchmark in South Africa it was 2% Egypt 3 these are the countries that are struggling with the pulse standard international average is 47% Russian Federation 70% Singapore 66% Hong Kong 65% so some countries really have most of their students competent at this level many are very similar to Spain around about the middle and some are really struggling very few students are working at this level and then there is an advanced benchmark which you are really very very accomplished readers sort of the top 5% to 10% so typical of the high benchmark so this is one where actually what we are going to do is just find some information so in and of itself just finding information not commenting on what the author intends to do not evaluating it not telling about the character but actually just finding information the cues that the children have are we say they see the little oval with the pencil in and that telling them how many points there are for this item and it's got number 1 and number 2 so it's saying 2 things the sort of clues that we want the students to be aware of 2 things to find and we put in bold what our 2 things Macy does that do not work so we use bold in the text and the numbers to help the students be aware of the sort of thing that we are asking them to do these particular things and this is where I wonder if some countries where children are just not familiar with written tests are disinvantaged they have practice material but it's nevertheless in some countries this is much more familiar than in others and those clues about what we're looking for so although it's straightforward retrieving information it's not the easiest because they have to work out where they'll find that information and there's 2 things they're looking for and it's a things that don't work so those are a quick run around a sort of assessments that there are in PURLS there's a lot more information that's published as with PISA a lot of the information is made freely available if you just go to the PURLS 2016 website you can download a lot of it what PURLS also has as well as the student achievement it has the student questionnaire which I mentioned earlier it also has questionnaires completed by teachers and by school principals and by parents or carers as well so there is a real wealth of information about the context in which children have learnt to read as their parents remember and the home environment which of course reading is so critically important as their learning the class teachers can talk about the strategies that they're using to teach reading and the school principals can talk about the school context and the resources and so on so there's a lot of useful information but one of the things just before we leave reading achievement and we move to engagement is this this is the bit about the teaching strategies that Spanish teachers these 70 teachers that completed the questionnaire for PURLS 2016 this is what they said they did so this is the proportion of students whose teachers said that they do these things at least weekly so the first set of data is the proportion of Spanish students whose teachers said this and then the right hand is the international average proportion of students internationally whose teachers said they did this and you can see that there is a very steady very typical relationship between the Spanish data and the international data the last one is slightly according to this sample less common in Spain this is the author's perspective or intention but everything else is very close to the international average so that's just a sample of the sort of information that's from the what we call the teacher questionnaire and then as I say similar information from the principals questionnaire as well and if you're interested in early reading so early years beginning reading then the home questionnaire is quite interesting but this is what parents think they did so this is four years on y la diabilitación I think is less sure so I want to just to move now to the other focus which is on students engagement so reading is just so central to students success in life and it starts obviously starts in the home before they ever get to school children's love of books there is a very old study in the UK published many years ago and it's never got referred to now but it found that the best predictor of later reading achievement at that time at that time was where a three year old could tell you what their favourite book was and if a three year old didn't read it not to read it but they could tell you the title of their favourite book think about it if they can name a favourite book at three they've been exposed to lots of reading at home or at nursery and it was a really good predictor of later achievement which I think is just such an interesting statistic this is one of the things that has worried us for many years in England especially so we have this engagement this wish to read so we have this great group here with the halo these are the children who can read and who love reading and we've all got those in the classroom and they're great because they're the ones that are like sponges offer the books, they'll sit in the library they'll read, they love reading they quite possibly have got books at home they quite possibly heard nursery rhymes when they were one you know, they've had this teaching reading has been fine they've learned to read and they love it you know, if you had a class of those life would be easy although other issues no doubt then we've got the children who they love reading but they find reading hard and I've written can't read it I can't read but they don't find it the easiest of things they're quite possibly the children that will get extra help but they haven't yet got that fluency of reading that will enable them to build on their enthusiasm for it we've then got a group that certainly we worry about who they've learned to read in the sense they've learned to decode but they just don't do it because they don't like it they've got other things they do so they are not practicing in their own time they're not reading when they're not being told to read they're reluctant to read and the critical thing is they are not practicing reading and then we've got those children who have not yet learnt to read and who really have built up all the resistance because they don't now want to read because why would you demonstrate something you can't do so those are the troubled children in the reading world because they can neither read nor are they wanting to learn they've got all the barriers there for reading and it could be there's all sorts of reasons that may have led to this but certainly our focus has may well be getting some extra help but our focus at one point was in this top left because they weren't practicing and the evidence which I'll talk later on is so clear about those that not only learn to read but continue developing their reading are the ones who will have the advanced skills ultimately so this is a if you're interested in that there's work by Stanowicz which talks about the Matthew effect which you may have come across which is from the Christian Bible saying essentially the rich get richer so if you love reading you choose to read you're motivated to read you read more it's a virtuous circle you read more you get better because you're good you read more and it also works you know can't read don't read and the gap gets bigger so the poor get poorer poor readers get relatively poorer the good readers get better because they practice more and it's this reciprocal relation is fundamental and I think it applies in reading more than anything else and reading if you think about reading in primary schools it's the one thing where we engage parents and we say read with your child home matters massively and there are some children the Matthew effect is working from age 4 you know even before then and there are other children where the opposite applies they don't have a book in the home and there's nobody there who says have you brought your book home can you read with me and the gap gets bigger so just quickly go through these pearls has the survey of do students like reading so they ask a series of questions about whether they like reading or not and I won't go through all of these because you'll get the slides but basically they give an answer and it divides them into three groups very simply those who very much like reading those who we say somewhat like reading a bit in the middle and those who do not like reading and it also includes information about whether they read for fun as well which is this bit about the other part of the scale but basically do they choose to read reading for fun and do they like reading and as you'd expect do they like reading well half the children in Spain above the international average say yes they like very much like reading about a third of them some of the time and then 10% no so at 9% 10% don't like reading and you can see achievement wise as you'd expect the 534 534 is the highest is the score scale score the better readers say yes I like reading very much in terms of country groupings at the top of this reading enjoyment scale are Southern Europe including Spain Middle East and North Africa at the top of the scale very roughly English speaking are around about the middle and the Nordic countries are at the bottom of this reading enjoyment scale so that's interesting when there's a lot of talk around how competent students in Finland are but actually enjoyment how they respond to the questions tend to be low another aspect is the confidence in reading so reading is one of these questions reading is easy for me I am just not good at reading isn't it awful a 9 year old just has to say yeah I'm just not good at reading I have no trouble reading stories with difficult words there's a series of questions and they say whether they agree a lot a little disagree a little disagree a lot with that and again that's produced a scale of very confident, somewhat confident and not confident readers and this was the data from 2016 so just under a half 10% you can see they are the strongest readers scale score 554 again another 40% somewhat confident but about 20% fifth both internationally and in Spain are not confident in their reading so they judge themselves to be basically other people are better than me at this a scale score looks as if they on average they are the weaker students in contrast to the reading enjoyment where the Nordic countries Finland, Denmark Norway and Sweden in pearls were not didn't enjoy it they now are the confident readers so don't enjoy it but they're confident English speaking countries are in the middle again and Southern Europe including Spain are at the bottom of this one so relative to other countries not as confident last one of these is on their engagement in the lessons so are they engaged in their reading lessons so I like what I read about in school I know what my teacher expects me to do my teacher tells me how to do better when I make a mistake etc goes from very engaged somewhat engaged less than engaged and you really want the students to be interested and engaged in their reading lessons so this is the students saying themselves whether they feel they are or not so 70% come out as in the very engaged they should be higher than the international average and only 3% not engaged so most students in primary I suspect it might be different you couldn't do quite the same question in secondary it might be different by the time they're getting older but most students still are relatively interested in and enjoying their reading lessons might not translate into reading voluntarily but in school, in class most of them quite enjoy that and Spain Southern and Eastern Europe Middle East, North Africa at the top English speaking drifting around the middle and Nordic and Far East at the bottom again so a lot of cultural issues as to what children will say I think in response to these and finally just the final one is why engagement matters in reading so this table is from the very first PISA survey in 2000 that Juliet will recognise that's been recast but essentially it's the one that looks at now we've got to 15 year olds it's the I can never understand why it's not as you talked about in the UK along the bottom on the horizontal axis you've got low reading engagement and high reading engagement so the sorts of things it's a different measure but it's the same idea we talked about interest in an amount of voluntary reading the type of reading and how diverse different types of texts that students read but it's that how much and how interested they are in reading low, medium and high on the horizontal vertical we've just got reading attainment, the measure of reading and the lines are the socio-economic status, the background of these children, these students now based on parental occupation and that's the PISA measure, parental occupation as a proxy for socio-economic status and essentially what it's showing us is that no, no, no, no, so the purple line at the bottom is students from a lower socio-economic status the green line is the middle socio-economic status and the blue line is students from high socio-economic status and we know that disadvantage impacts on attainment so it's not a surprise the gap between the purple line the green line and the blue line because we know SES disadvantage impact from early days in schooling right through and everyone I think all countries aim to narrow that gap to reduce the gap but you can see where the gap reduces which is on the far right at the high engagement and those highly engaged students from low the purple right hand purple diamond, the low SES high engagement students performed better than the high SES low engagement students the top blue on the left engagement matters engagement makes that difference and it's not the cause and effect it's an association because it's reciprocal they get engaged they want to read, they read more they read more, they get better they want to read and this massively shows you what's happening from the early stage preschool and then at pearls at nine where you see the engagement and by and large we know most primary children for the most part you can infuse about just about anything if they get in the hands of the enthusiastic teacher and then this shows what an impact it can have with high levels of engagement so essentially to summarize highly engaged students the high engagement can compensate for the rest of that disadvantage engagement really matters and that's the end of it thank you very much thank you very much Liz we'll see if there are any questions tenéis algunas preguntas en relación con la ponencia de Liz con la evaluación a pearls bueno mañana en la segunda jornada del curso durante la primera sesión expondremos los resultados de España en pearls y en PISA esto ha sido una primera aproximación muy exhaustiva y muy detallada pero bueno mañana entraremos más en la parte de los españoles y por la tarde tendrán lugar dos talleres uno centrado en pearls más dirigido a aquellos de vosotros que sois de primaria otro sobre PISA más dirigido a aquellos de vosotros que sois de secundaria bueno no hemos hecho mención pero nos ha sorprendido gratamente la cantidad de especialidades y de cuerpos de los que venís y que representáis en este curso profundizaremos por la tarde en un sentido muy práctico en estas evaluaciones esto es un inicio pero bueno cualquier pregunta que se os plante en este momento bueno pues si no la hay solo quedaría agradecer a las dos ponentes Liz y Juliet sus intervenciones su tiempo, lo fácil que han hecho articular y coordinar todo esto así que un último aplauso para ellas antes de cuestiones logísticas muchas gracias