Developing a monitoring framework

The main purpose of any education system is to provide students with the necessary knowledge, skills, and values for effective participation in society. Most countries keep records that focus on counting schools, students, resource inputs, and outputs in terms of examination results. However, many countries have no systematic information on learning outcomes. A framework that monitors learning outcomes can provide the basis of comprehensive monitoring and evaluation (M&E) plan for assessing how an education system performs over time.

The framework can be used to monitor learning at national and international levels. The latter enable comparisons between countries and may be used to monitor progress towards Sustainable Development Goal (SDG) 4. One of the main purposes of M&E in education is to ensure that equitable and good-quality education is being provided to all of the population and at all levels (UNESCO, 2016: 116).

A monitoring framework should be designed to monitor quality at different levels (country, region, schools) and for different groups (school authorities, teachers, students). Monitoring frameworks draw on a number of different tools and components to collect and organize data needed to monitor a system’s performance (UNESCO, 2016). This includes tools at the school level for data on students and teachers, national education management information systems (EMIS), human resource or teacher management information systems for teacher recruitment and deployment, inspection and appraisal systems, and financial management systems. The framework may also include system-level student evaluations.

What should the framework monitor?

Monitoring should occur at both individual and system level. Teachers, parents, and head teachers need information relating to their own school. Education ministry officials require data aggregated at various administrative levels. Ministries of finance need information on system resource inputs, use, and cost to inform their cost-effectiveness estimates.

The monitoring of quality of education in terms of inputs, processes, and outputs can be classified in a three-part typology:

• Compliance monitoring, intended to ensure educational institutions comply with standards and norms, focuses on educational inputs such as teachers, textbooks, classrooms, and equipment.
• Diagnostic monitoring focusing on instructional processes and whether students are learning. It provides information on the quality of education provided by schools.
• Performance monitoring focusing on monitoring academic achievement of students through testing. This provides information on the results of investments made in education.
  (Adapted from Richards, 1988)

These three types of monitoring, using different monitoring tools, may co-exist within an education system. For example, school record-keeping systems include input, process, and output data that may be used to support school-level management.

What are the components of a monitoring framework?

An effective monitoring plan should provide answers to several questions.

• Which indicators should be measured in order to determine what was done, how well it was done, and what was achieved? The indicator should be directly related to the expected outcome or goal. It should be measurable and well defined. Baseline and target values are calculable.
• Who will collect these data and how will they collect them? This will indicate the institution or personnel responsible for collection and the tools to be used, which include checklists, forms, or surveys.
• How will the data flow from the original collection locations to technical staff and to management and policy-making levels? This may be through existing data-sharing networks like EMIS or through specially tailored computer programs.
• Who will check the data quality, conduct the data analysis, draft reports, and make decisions based on the data? Experienced technical knowledge and expertise will be needed.
• How will the data be managed to ensure privacy, enable access to those who need it, and guarantee safe storage over time? Storing the data on a dedicated computer system with monitored access would probably ensure all of these requirements.

What tools for monitoring are available?

Monitoring frameworks generally rely on a number of different data collection tools and practices.

School census

Most countries use an annual school census as the primary way of collecting information from schools around the country. The school census usually takes the form of a questionnaire, and includes questions on school infrastructure, furniture and equipment, teaching and learning materials, school income and expenditure, teacher characteristics, and student characteristics. School census data can also be used to provide feedback to school managers and teachers.

School audits

Most school systems have an institution in charge of auditing financial issues, materials and infrastructure, human resources, and other aspects of school census reports and school accountability mechanisms.

Education Management Information Systems (EMIS)

An EMIS is a database centralizing school-based data collected from an annual school census, transactional data about education stakeholders’ operations, and other data sources (such as population census data). It is used by education ministries, NGOs, researchers, donors, and other education stakeholders as a reliable source of educational data for planning, monitoring, and policy decision-making. There is often also a separate human resources or teacher management system (TMIS), which is used for decisions about school head and teacher recruitment and deployment.

Principal, teacher, and student surveys

Large-scale international assessment programmes use surveys of students, teachers, and school heads for background and contextual information. Such information includes perceptions of education quality processes, including teaching practices, the school climate, and school leadership and management practices. Surveys provide one way for education leaders to gather system-level information about education quality processes, even where direct pedagogical observation on a large scale is not feasible.

School inspections

School inspections can contribute to the quality of schools and education systems. Terminology varies from one country to another; common names for the process are school accreditation, inspection, or supervision. These processes generally have two interwoven objectives: public accountability and school development.

School report cards (SRC)

School report cards aggregate information on schools including enrolment, teacher and student attendance, and student academic performance, in a form accessible to the public. Several models exist, from well-institutionalized models where information is provided for each school and published regularly on the Internet, to one-time report card surveys conducted with the support of civil society organizations. By publishing school-based data, SRCs promote transparency and accountability.

Public Expenditure Tracking Surveys (PETS) and Quantitative Service Delivery Surveys (QSDS)

PETS track funds from the central down to the school level. Using sampling methods, they seek to measure the leakage or diversion of funds in the education system, giving insight into whether resources are being used in the ways intended for general operational needs and for efforts to improve education quality. PETS can be used to check whether flows of key resources that have a direct impact on learning reach their intended beneficiaries such as school funds, or are used to purchase textbooks, learning materials, equipment, and so on. A PETS is often combined with a Quantitative Service Delivery Survey (QSDS), which focuses on other dimensions such as ghost teachers or teacher absenteeism, and which are also key to improve the quality of learning.

Mapping

Geographic Information System (GIS) technology can be used to map data onto geographical areas, exposing regional patterns and relationships that may be less obvious when represented only in numeric tables and databases. The visual mapping of complex data can help decision-making on such factors as resource distribution, teacher deployment, and planning for education in conflict or emergencies.

Pedagogical and classroom observation protocols

The core process that matters most for improving learning outcomes is the interaction between teachers and students. Yet many educational monitoring systems do not collect data on this process because it requires direct observation and reliable coding of primarily qualitative information – both of which are difficult to achieve on a large scale. Many systems use pedagogical observation as part of a teacher appraisal and re-certification process. Pedagogical observation can also be used in a peer-to-peer format as part of teachers’ continuing professional development.

Graduate tracer studies

Graduate tracer studies are surveys that aim to determine the percentage of graduates gainfully employed, self-employed or admitted to post-secondary studies after completing secondary education, analysed in relationship to other important background variables. Tracer studies have typically been conducted on graduates of technical, vocational, or post-secondary institutions, but can also be employed after graduation from the basic education system—particularly in contexts where the majority of secondary-education graduates do not go on to higher education.

Employer satisfaction and skills gap surveys

This approach surveys employers to assess whether graduates possess the skills and qualities needed to be employable in different sectors of the economy. In many countries, certain industries consistently hire workers from other countries because local graduates are perceived as not having the technical skills they need for the job. Employer satisfaction surveys or skills gap surveys may also reveal problems at a more basic level, such as insufficient abilities to communicate and collaborate. Education planners can use such data to reconsider the subjects that are being taught in school, as well as the overall pedagogical approach.

Expenditure surveys

These may be used to monitor expenditures among different groups such as students or households. Financial Management Information Services (FMIS) collect and integrate public financial management processes including budget formulation, execution, accounting, and reporting.

Learning assessments

Assessment data can play a key role in a monitoring framework as part of the analysis of issues in an education system and to monitor plans for improvement. National examination data may also be incorporated into a monitoring system through an EMIS or school report cards.

Monitoring progress towards SDG 4

There is a growing recognition of the need for internationally comparable data to measure progress towards the SDG 4 targets. The UIS-led Global Alliance to Monitor Learning supports national strategies for learning assessments and the development of internationally comparable indicators and methodological tools to measure progress towards key targets of SDG 4.