Program evaluation: large-scale and small-scale studies

It is increasingly incumbent upon ministries of education to build evaluation into new programs – especially those programs where substantial amounts of money are being spent. Each new program will usually be accompanied by questions about the impact and effectiveness of the program. For example, ministries may direct increased resources to classrooms and schools. Question: “Have these resources gone where desired and have the increased resources had an effect on student achievement?” Or, a new curriculum may have been introduced. Question: “How was the curriculum introduced and what problems occurred with its implementation?” Or, a new teacher in-service program may have been developed. Question: “Did the teachers learn what they were meant to learn? And, if so, did what the teachers learn have an effect on what students learned in terms of achievement, attitudes, and/or behaviour?” These are some examples for the case of general school education, but the same is true for new programs in pre-schools, in schools for the handicapped, in vocational education, and so on. All education programs need to include an evaluation component if their success is to be determined, and if weaknesses in the programs are to be identified and corrected. When introducing new education programs it is not easy to assess whether they have had an effect on student learning. The kind of research design needed to get at the true cause of changes in student learning will vary according to the type of learning specified in the program goals. At the same time, however, there are some similarities in terms of sound and defensible evaluation designs. For example, it is always important that some measure of student learning be made at the beginning of a program. Education programs cannot be said to be effective if there are no measurable improvements in student learning over time. Similarly, some comparison group, or groups, of teachers and students should be included in the study. If there are measurable changes in student learning over time, but the magnitude of the changes is not different from changes that occur in non-program students, then the program cannot be said to be effective. This booklet is about small-scale and large-scale program evaluation studies. In many cases, both small- and large-scale studies are needed within any one evaluation project. In this booklet the authors argue that there is a need for both kinds of studies – provided that they are conducted according to scientific standards. This booklet is the eighth in the Educational Policy Series developed by the International Academy of Education and the International Institute for Educational Planning. Each booklet seeks to bring research evidence to bear on important topics in educational policy.