Webinar - The use of learning assessment data: what have we learnt so far?

Description

In spite of the increasing number of countries participating in large-scale student assessments and important human and financial resources that are invested, related efforts do not necessarily lead to the effective use of this data in policy and planning. Although much has been published on the importance of learning data for improved practices, a more systematic understanding of why it remains underused is still missing. This is why UNESCO with its partners organized a Webinar to share the results of their recent work on the theme. 

This Webinar addressed three main questions building on the findings of recent research and capacity building activities:

  1. What are the main facilitators and barriers for the efficient use of learning data?
  2. What are the key elements of a broader conducive environment for the use of learning data?
  3. What are the risks entailed in ‘misusing’ the assessment data? 

Ieva Raudonyte from IIEP presented findings from a literature review on the use of learning assessment data in education policy-making, emphasising the importance of understanding how countries use learning assessment data and possible reasons as to why this usage may be ineffective. It is not enough to focus only on learning assessments and processes directly linked to them; there is a necessity to have a broader perspective considering the culture of the use of evidence and planning as well as the inherent policy-making dynamics. Finally, although learning data has a lot of potential, the possible misuses of this information should also be considered, such as using a single test score for decision-making or using data to justify pre-defined agendas.

Davide Ruscelli from the UNESCO Office in Dakar identified a number of barriers that make it hard for countries to use efficiently large-scale assessments. These include a lack of ownership of the assessments, slow release of results, a lack of integration between large-scale and other assessments, and poor institutional collaboration. In addition, it is important to ensure that learning data respond to policy concerns if decision-makers have and that capacities of assessment units are strengthened.

Guy Roger Kaba from the Conférence des Ministres de l’Education des Etats et Gouvernements de la Francophonie (CONFEMEN) concentrated on the role of PASEC data in education policy-making. He presented the preliminary results of a survey on how participating counties have used the Programme d’Analyse des Systèmes Educatifs (PASEC) data to improve the quality of learning. Results identified the extensive use of the PASEC 2014 data in promoting pre-primary education, improving the quality, availability and allocation of school equipment and pedagogical resources, as well as strengthening teacher training and upgrading teacher status. He also highlighted the barriers for the efficient use of learning assessment data due to ineffective dissemination and appropriation of data, financial and material constraints as well as issues linked to the governance of the education systems.

Maya Prince from the UNESCO Education Research and Foresight team noted that the rationale for undertaking a large-scale learning assessment might go beyond its potential to inform policy and practice. Participation in an assessment often involves a combination of rationales that cut across technical, economic, socio-cultural and political reasons for engagement. She also drew attention to large-scale assessments’ limitations including a lack of breadth, and exclusion of the most vulnerable and marginalised, as well as potential risks of over-use and over-interpretation of the data. In addition, coupling assessments and accountability may imply certain risks such as teachers teaching to the test, narrowing of the curricula, focusing only on students who they believe will perform well – although the evidence of such unintended consequences remains quite limited. She noted that despite large-scale learning assessments being ill-suited for high-accountability frameworks, the recent attention to overall rank in cross-national initiatives as an indicator of national progress may be creating new incentives for education authorities to reshape curricula or adapt teaching practices to the content or style of these cross-national initiatives in an effort to improve student performance. 

 

Below is a selection from the Q and A session both on and off line

How does one ensure the results of large-scale learning assessments (LSLA) answer decision-making needs and identify the root causes of low learning outcomes when many factors may be operating?

Policy-makers need to be involved in the development stages of the assessment so that its results respond to their needs. In the case of the PASEC assessment, a standard questionnaire identifies most of the countries' concerns, in this way paying attention to country-specific aspects. Moreover, the development of items is based on the proposals coming from countries in order to take into account what is actually taught in the classrooms. One of the important SEACMEQ assessment features is the ‘policy thread’ that runs throughout the entire assessment cycle that starts with the identification of policy concerns, and ends with research-based suggestions about how to address the initial policy issues (Saito, 1999).

Large-scale assessments cannot possibly answer all the questions decision-makers may have, but they must be able to understand the various dimensions related to the learner, the teacher, the school, the education system and economic, social and political contexts. Learning assessment data should therefore be considered and analyzed in combination with a range of other evidence generated at national and international levels to grasp the root causes of low learning outcomes. Inclusive discussions with different stakeholders are also key. 

It appears that one of the reasons governments do not use the data is that its interpretation is very technical and therefore not easily understood. Is there a framework for simplifying the presentation of assessment data, one that will make the data easily understood and usable?

Policy-makers need clear messages and actionable recommendations that can inform their decision-making. It is important that those who produce and analyze learning data communicate clear and concise policy messages that will have higher chances of being considered by decision-makers. CONFEMEN is currently working on making its communication products more reader-friendly. In addition, its teams are working together with countries when translating the PASEC data into avenues for reflection.

Practical ways of simplifying policy messages coming from assessment analysis include the development of short policy-briefs, dissemination seminars targeted at policy-makers, reports that focus on key messages, but which also provide access to more detailed methodology for those interested in the technical elements. 

TALENT has been working on dissemination with planning directors and national large-scale learning assessment managers from different countries in Sub-Saharan Africa. It is key to identify the different information needs of the key stakeholders through consultations. It is also important to include different communication methods into a dissemination strategy that cater for different intended uses and anticipate potential unintended consequences (Robertson S., presentation at TALENT regional workshop, 2018). Various products or approaches can help meet the needs of different stakeholders and  help them understand the results of LSLA and what they mean vis-à-vis the situation of the education system. Products vary from highly technical reports intended for a specialized audience of researchers to relatively ‘lighter’ products from a technical perspective such as media reports, press releases and blogs. 

It would be good to hear of cases of countries in different contexts making effective use of large-scale learning assessment (LSLA) data.

The upcoming CONFEMEN study on the use of PASEC 2014 results will highlight good practices from participating countries, as well as a variety of ways in which countries use learning data. The study will be available towards the end of 2019 on the CONFEMEN website.  

UNESCO Dakar, as the TALENT Secretariat is working on a policy brief, gathering best practices in the use of LSLA that will be published in the last quarter of 2019. Some positive examples in the Sub-Saharan Africa region include the introduction of mother tongue teaching in early grades in Uganda and Mali; improving the training system for teaching staff at all levels in Senegal; use of LSLA results in the development of the State Report on the National Education System, supported by the IIEP-Pôle de Dakar. 

Are there countries where data is being used to inform and bridge the gap between learners-teachers-education officers-policy makers and various stakeholders?

The ongoing IIEP-UNESCO research project is looking at the positive examples where learning data bridges the gap between the work and needs of different stakeholders. For instance, the Gambia assessment system is trying to work in this direction by creating an institutional setting that would make sure schools, regional officials as well as the key departments in the Ministry use learning data effectively.

Learning assessments are seriously affected by political issues. How best can we as planners overcome this challenge?

Learning assessments, as other system elements, are not insulated from political dynamics, which might sometimes negatively influence their functioning as well as the uptake of their data. One strategy consists of providing an assessment agency with a legal mandate that is long-term, and ensure that it is well-funded so as to be less vulnerable to political instability and regime change (Tobin et al., 2015). The assessment agency in Chile is an illustrative example. This well-established body within the Ministry of Education remained insulated from direct political influence while allowing the assessment to respond to policy concerns and priorities of the government (Ferrer, 2006). 

What about the technical capacity of teachers and other education staff to actually perform learning assessments themselves and use the results in the classroom?

Large-scale learning assessments are sample-based, system level assessments with the primary purpose of providing a snapshot of learning achievement for a group of learners in a given year and in a limited number of domains. The wealth of information they gather plays a key role in guiding and orienting policy actions. They are not intended primarily for use by the teachers and there are many other types of assessments that teachers use in their day-to-day teaching. Nevertheless, there are examples of how large-scale learning assessment data have been used with teachers (see the example from Oxford University). In addition, certain systems encourage the use of national large-scale assessments by school heads and teachers at a school level planning (e.g. the Gambia) and the assessment design might considerably contribute to the effectiveness of this.

It was mentioned earlier that one of the limitations of LSLA is the narrow focus on literacy and numeracy skills. As countries are moving toward broadening their educational goals to include a wide range of skills and competencies, beyond subject-based areas, can data from LSLAs be effectively used to meet the changing educational goals? If so, what would this look like?

The limited number of targeted domains and abilities can be considered intrinsic to the design of LSLAs. LSLAs necessarily focus on a limited number of subjects and abilities – more specifically, easily measurable areas of learning that lend themselves to system-level or cross-country comparability, or those  considered foundational skills that enable further learning in other areas. However, even within the commonly assessed domains, they are insufficiently attentive to the breadth of knowledge available. For example, in assessing language, subdomains such as appreciation of literature, writing, and listening comprehension are often missing.

In addition, they do not adequately assess the broad range of knowledge across a wide spectrum of domains such as the arts or media and information literacy, or the broad competencies that schools are meant to develop such as citizenship, environmental responsibility or collaborating thinking. As a result, this may lead to a narrowing of the multiple purposes of education and dimensions of learning.

Nevertheless, there are increasing efforts to integrate a broader range of knowledge, skills, attitudes and values into large-scale assessments – both in national and cross-national learning assessments. See for example SEA-PLM (measuring global citizenship attitudes and behaviours); or ICCS (how young people are undertaking their roles as citizens). At the national and sub-national level, there is also the Brookings-UNESCO Optimizing Assessment for All initiative to support countries to improve assessment, teaching and learning of 21st century skills. However, this pilot mainly focuses on other types of assessments, such as classroom-based formative assessments, that seem to be a more appropriate entry point for capturing such skills.

It will be a long time before such national and cross-national assessments systematically capture at large-scale data elements linked to healthy lifestyles, values and attitudes that support democracy, the protection of human rights, respect for cultural diversity, active responsible citizenship and environmental sustainability, or other principles put forward by the 2030 Agenda. It is even debatable whether we would want large-scale learning assessments to encompass these areas.

Further resources are available on the TALENT website.