Since the introduction of the 1994 Education and Training Policy, the Ethiopian government has initiated and implemented five successive longitudinal education sector development programs (ESDPs) with the intention to achieve improved access, efficiency, equity, quality, and relevance. Reform efforts implemented through the programs have brought about rapid education system expansion. Despite the achievements in improving equity and access to education, the quality of the education system continues to struggle as demonstrated in the National Learning Assessments and Early Grade Reading Assessment results. To redress the challenges of education quality and to sustain improved access across regions, gender and income levels, the Ministry of Education and development partners have begun to prioritize education quality, and subsequently, planned and began to implement the General Education Quality Improvement Program (GEQIP) in 2009. So far, the program was implemented in two phases spread over eight years period: GEQIP I (2008-2012) and GEQIP II (2013-2018). GEQIP II was introduced to scale up the coverage and scope of education quality interventions and programs and strengthen the general education quality reforms on the basis of lessons drawn from GEQIP I. It was generally intended to achieve improved quality of general education (grade 1–12) throughout the country and improved learning conditions in schools, and strengthening institutions at different levels of educational administration.
The main objective of this study was thus to conduct a comprehensive evaluation of GEQIP II to assess its contribution to the improvement of Ethiopia’s General Education Quality, to generate evidence of the program’s results and impacts towards improving learning conditions and strengthening capacity of education management at different levels, and to assess how key stakeholders view the relevance, effectiveness and impact of the sequencing and mix of key inputs, and to use this information to adjust the program over time.
The methodology for this evaluation study was mainly guided by the purpose of the study and the underlying contexts of the program. Hence, the study included both the process evaluation (which aims to evaluate whether GEQIP II operational mechanisms support the achievement of the objectives of the program), and the impact evaluations (as measured by the standard measurement indicators of the program, such as reduced dropout rate, improved teaching effectiveness, improved utilization of school grant for teaching and learning, improved utilization of textbooks, improved learning conditions of schools, etc.). The final updated PDO indicators (after the mid-term evaluation) pertaining to the components of the project (curriculum implementation and teaching learning materials, teacher and leader development school improvement program, system management and capacity building program, and program building) were used as bases for undertaking the evaluation study.
Out of 1023 Woredas in the country, 130 were randomly selected after giving proportional sampling weights to the 9 regional states and two city administrations. However, we set the minimum number of Woredas to be included in the sample as three. By considering school location (urban vs. rural based on EMIS data), school distance from woreda town, and school size (measured in terms of enrolment) as stratification variables, 615 (85 ABE, 462 primary, and 68 secondary) schools were included in the study. Both primary and secondary data were collected at school, woreda, region, and federal levels. To obtain primary data, survey questionnaires were used for school principals, teachers, cluster supervisors, and students. In addition, school and classroom observation checklist, Key informant interview (with experts at federal, Woreda level, regional level, and CTE and University), and Focus Group Discussion with PTSAs were used. Secondary data were obtained from documents at sample schools and reports from the MOE.
Before collecting the final data, a five-day intensive training, accompanied by a pilot test of data collection, was given to data collectors. Every item of the survey questionnaires, FGD and interview protocols, and observation checklists were discussed so as to ensure the data collectors have a thorough and common understanding about the data needed for the study.
Both quantitative and qualitative data analyses methods were applied. Descriptive and inferential statistics were used to explore both the primary (survey) and quantitative data obtained from documents. On the other hand, qualitative data collected through FGDs, interviews and open-ended questions were analyzed according to themes based on the evaluation questions and the study objectives. Results from the quantitative and qualitative data were combined according to major topics and assessed within the context of the GEQIP II components outlined in program impact pathways.
Reduced dropout rate of grade 1 students – The study revealed that the target of lowering dropout rate to 17% by the end of GEQIP II is achieved at national level as it is found to be 12.64%. This result is based on field data from the sample schools. EMIS data also show that grade 1 dropout rate is 17%. Field data, however, showed considerable regional differences in grade 1 dropout rates; Benishangul (42.39%) and Somali (33.16%) regions still showing very high dropout rate. Comparison of schools based on their levels revealed that high dropout rate is observed in Level I and Level III schools. But no differences were observed between urban and rural schools.
Printing, production and distribution of textbooks and supplementary materials: The federal level data show that more than 133 million textbooks and supplementary materials were re-printed, printed and distributed. The target set for GEQIP II was 120 million. However, as the following findings show the textbook-to-student ratio for the selected MTs has not reached the desired level.
Textbook ratio for mother tongue subjects: Considering GEQIP II’s indicator to achieve 1:1 student to textbook ratio for selected mother tongue subjects, the results in the present study revealed that the target was not met. The textbook-student ratio for Afan Oromo is lower than the other groups due to the change in the prepared/printed textbooks and the sampled schools may not have the newly prepared textbooks. Data to calculate textbook ratio was collected from each of the schools. The school principals were asked to report the number of books received and distributed, and the number of students at each grade level. The ratio was computed by dividing the number of textbooks distributed to the number of students.
Textbook utilization: Classroom observation results revealed that the targets for textbook utilization are well below the targets set for GEQIP II. The PDO indicator state that 90% of students bring mathematics textbooks to class and 70% of students bring science and social studies textbooks to class. Textbook utilization was calculated by counting students who have a textbook of a certain subject and dividing by the number of those students who brought their textbook to class during the period the subject is taught. Since the student textbook ratio may not be one-to-one in all the classrooms observed, we found dividing the number of textbooks brought to class to the total number of students in the class may not exactly indicate textbook utilization. Hence, data were recorded for the total number of textbooks distributed for the class and the number of students who brought textbooks into the class (which indicates the number of textbooks brought to class).
Percentage of teachers adopting active teaching methods: – Cluster supervisors’ ranking of teaching methods at schools revealed explanation or lecturing as the most frequently used, followed by 1 to 5 group work and discussion. On the other hand, laboratory work, case studies/debates, and students presenting assignments were ranked as least frequently used teaching methods in the schools. Correspondingly, most instructional time (17.44 minutes on average or 43.6% of the 40 minutes instructional time) is spent for lecturing/explanation and note taking from a blackboard (8 minutes or 20% of the 40 minutes instructional time) and the least instructional time (2.26. minutes on average or 5.56% of the 40 minutes instructional time) is spent for students’ presentation.
GEQIP II’s indicator to assess the outcome of the project on adoption of active teaching methods is related to the change in the performance of Level I and Level II schools on four inspection standards. The baseline’s scores of Level and Level II schools on the four standards during the first-round inspection were 45.4 % and 59.4% respectively. Re-inspection of Level I and Level II schools in the four inspection standards indicate very marginal improvements in the scores: 46.42 % and 59.53 % for Level I and Level II schools, respectively.
Teacher development: The targets set regarding the number of teachers nationwide who would participate in various capacity development activities are achieved though inconsistent figures are observed between data obtained through field work and documents from the MOE.
In relation to licensing exam, according to the MOE report, only 23.44% of those teachers who took licensing exams met the requirement for certification.
School leaders development: The target was to train 1,600 school leaders with Post Graduate Diploma in School Leadership (PGDSL). While the survey result from REBs revealed that 13,869 school leaders attended the program from 2006 E.C. to 2010 E. C., a report from the MOE raised the number to 14,299 from 2007 E.C. to 2010 E. C. Regardless of the inconsistency in number, one can observe that the target is achieved.
School grant arrival date: The PDO target for the arrival of school grant is October 31 (Tikimt 21) of every year. By the end of the program, it is assumed that 80% of schools will be able to get school grant before the deadline. However, school grant arrival letter and reports from the sample schools principals indicated that most schools receive school grant after October 31. Difference is observed between urban and rural schools with relatively more urban schools able to get the grant before October 31. School grant arrival date was recorded by looking at the school grant arrival letter and reports from principals when the letter was not available.
Utilization of SG for teaching and learning: All schools are required to use 50% of SG for teaching and learning activities. The results indicated that, overall, around 72% of the schools utilize more than 50% of the SG for teaching and learning activities. This was calculated after the schools reported the total grant received and the expenses for various teaching learning activities. The percentage of the amount spent for the teaching and learning activities was divided by the total grant received for the year.
Percentage of Level 1 and Level 2 schools that have move up to the next level: Data collected from the Ministry of Education’s Inspection Directorate revealed that the intended target of moving 10% of Level 1 schools into Level 2 have been met; however, the target of moving 5% of Level 2 schools into Level 3 was not achieved as the number of schools that moved from Level 2 to Level 3 is nearly the same as the number of schools that moved down from Level 2 to Level 1. Interestingly, although more than half of Level 1 schools (3470 of 6099 or 56.8%) inspected during baseline have moved to Level 2 during re-inspection, the performance or ratings of Level I schools on the four indicators (used to assess teaching effectiveness) during re-inspection did not improve much compared to the first-round inspection (as presented in earlier section). About 7.5% of Level 2 schools (1097) have also moved up to Level 3 while a similar percentage of Level 2 schools (1106) have regressed from Level 2 to Level 1 during re-inspection. The target of inspecting 95% of the schools is reported to have been met by the Ministry of Education as more than 35,000 schools were inspected during GEQIP II period.
Regarding stakeholders’ (principals, teachers, and cluster supervisors) judgment of contribution of GEQIP II in improving various school related activities, its implementation, and the relevance of some activities:-
- they believed that the program has helped in improving textbook availability, accessibility of supplementary materials, teachers’ teaching practices, teachers’ assessment practices, teachers’ subject area knowledge, teachers’ language skills, student dropout, school leadership practices, students’ performance, participation of school community in the and decision-making process. The only doubt school principals have is on GEQIP II’s contribution to improve learning conditions of special need students.
- They rated the level of implementation of GEQIP II as ‘Good” and better than ‘Good” and judged the various activities of the program as important in improving quality of education.
Students’ views on learning conditions: Students reported that teachers do not often correct homework, do not give feedback to classwork; students do not often bring their textbooks, and that they don’t ask questions when they don’t understand a topic or presentation. Students’ ratings on the above issues were compared with expected mean value of 4 (“Often”) and the observed means were statistically lower than the expected mean value. In addition, students’ mean rating on teacher absenteeism (2.5 out of a five-point scale) was found to be significantly higher than the expected mean value of 2 (representing seldom in a five-point rating scale), showing teacher absenteeism is a concern in Ethiopian schools.
School improvement planning practices: Responses from FGDs and interviews showed that schools have improved their capacity to develop school improvement plans. However, supervisors and woreda education offices mentioned that school improvement planning has become a routine activity with less concern with using data in setting priorities, monitoring and implementation of the plans. School principals also reported that they have problem in implementing school improvement plans. Document analysis of school improvement plans indicate that some school improvement plans set very ambitious targets of improving learning performance of students for all subjects by 10 or 15%; do not have academic focus; are very similar in terms of priorities across different years, and lack clear focus in terms of priorities.
CONCLUSIONS AND RECOMMENDATIONS
- Reducing the dropout rate in grade 1 has been met during the end of the project. To sustain this, it is essential that reasons that cause students and parents to lack interest towards education is properly identified and addressed in the long term.
- Different studies conducted on textbook-student ratio indicate that there is overall one-to-one student-textbook ratio in primary grades for most subjects, except MTs and Civics, and almost one textbook to one student for almost all subjects except for Amharic, History, and Mother tongue in lower secondary education (PRIN (MoE), 2018). Quantitative data in this evaluation showed that student-textbook ratio in the selected MTs has not reached 1 to 1 ratio except for Tigrinya. Some principals identified textbook shortage as one of the most serious problems schools faced. In most rural schools, textbook shortage was mentioned as one of the three problems that hindered the teaching and learning process. It appears that some textbooks are stored either at woreda education offices or are poorly distributed to reach fairly to all schools. The distribution and management of textbooks must be revisited so that schools can have fair access to textbooks.
- GEQIP II aimed to improve the baseline data of 58% of the students bringing their textbooks to the classroom to 90% for mathematics and 70% for science or social science subjects. A Ministry of Education sponsored study reported that the average proportion of students carrying textbooks to classrooms was found to be just 30.58% (PRIN, 2018). The above study appears to have assumed that all students in a class have their own textbooks and reached to the above finding by dividing the number of students who brought textbooks to the number of total students in the class. In the GEQIP II exit evaluation study, number of students who have textbooks were collected and divided by the number of students who brought textbooks to the class, taking into consideration that all students in a class may not have their own textbooks. The findings indicated that 51.77% of the students brought their Mathematics textbooks, 60.95 % of the students brought their environmental science, and 56.56% brought their social studies textbooks. The two studies demonstrated that textbook utilization targets for mathematics and social studies/ science subjects have not been met during GEQIP II. Developing a textbook use guideline for teachers and students or keeping textbook in the students’ desk, when available, may help to improve the textbook utilization. Teachers’ lecturing or explanation and note-writing and copying teaching techniques may also contribute to the low textbook utilization. Aligning teachers’ practice to the planned curriculum can also contribute to improve textbook utilization.
- GEQIP II identified four standards to serve as indicators of improvement in teaching effectiveness. Re-inspection of Level 1 and Level 2 schools in the last two years showed that Level 1 (45.74 %) and Level 2(59.1 %) schools’ scores in four standards did not improve from the baseline scores of Level l (4 %) and Level 2 (59.4%) schools. Surprisingly, 49.8% of Level 1 schools moved up to Level 2 during re-inspection, indicating that many of these Level 1 schools have improved in many inspection standards. Classroom observation results of the GEQIP II exit evaluation study may provide supporting evidence why Level 1 and Level 2 schools did not improve in some of the indicators used to assess teaching effectiveness. As presented in the findings section, students and teachers were spending majority of their time on lecturing or explanation and writing and copying notes from a blackboard. In addition, supervisors reported that case studies, laboratories and student presentation were the three least often used methods in schools. This finding supports the absence of improvement in teaching effectiveness scores as two of the indicators used to assess teaching effectiveness of teachers (teacher use modern teaching methods to increase student participation and teaching is well planned with appropriate resources) cannot be met when teaching is lecture oriented and laboratories and case studies/debates are two of the least used methods. Re-conceptualizing CPD to be relevant to teacher’s classroom practices and curriculum contents and activities and improving resource allocation to school appears to be the appropriate strategies to improve teachers’ practices.
- The findings of this evaluation indicated that 70-72% of the sampled school use 50% and above of the SG to the teaching and learning domain of school improvement. This was done by identifying the number of schools that use half of the SG (based on dividing reported expenditure on teaching and learning to the total amount of received school grant). The GEQIP II target is that 100% of the schools use 50% of the school grant for teaching and learning domain. This indicator was revised based on a mid-term study (PRIN (MoE), 2016), which reported that 100% of the schools use 50% of the school grant for teaching and learning domain. The study’s finding was based on survey response of school managers and teachers if schools are using 50% of the SG for the teaching and learning domain. All respondents in the study responded affirmatively. This, however, shows the perception of the respondents rather than the actual reality. Very encouraging in the SG is that most schools expend most of their SG for purchasing reference materials, improving library and laboratory facilities and supporting CPD and students’ extracurricular activities. Yet, some schools insisted that there should be flexibility in the school grant guideline as some schools in rural areas have serious shortages on basic facilities, such as student chairs and toilets.
- School grant arrives to most schools (more than 80% in 2009 and 2010 EFY) after October 31. A mid-term evaluation on school improvement and school grant assessed school grant arrival date by asking school managers on whether school grant reaches to school by October 31. The mean rating on a five-point scale was 3.42, indicating that some of the respondents did not agree that it reached schools by October 31 (Ministry of Education, 2016, 27). Despite this finding and the serious limitation such reported evidence may entail on the exact arrival date, the study concluded that “this timetable [October 31] has been met in almost all cases” (Ministry of Education, 2016, p. 26). Contrary to this study, another study that examined arrival of school grant by tracing school grant arrival letter found that school grant reached to most schools after October 31 (BDS-CDR (MoE), 2018), indicated that school grant stays for some weeks in BOFED and BOFEC offices. It is necessary to learn from the experience of some regions, like Tigray, in which school grant reaches from BOFEC to schools in a few weeks.
- School improvement planning and implementation practices need fundamental re-thinking as it appears that the completion of the school improvement guideline is equated with school improvement planning. SIP plans are very similar, have sometimes non-academic matters, depend on similar strategies for achieving different objectives, do not show specific interventions, and are saturated with too many lists of activities rather than selected priorities. Some principals stated that school improvement planning is becoming a routine requirement for schools without serious consideration to its implementation. Part of this could be that the planning guideline is very big and schools keep on identifying generic goals and actions that do not lend themselves to measurable assignments to teachers and other concerned bodies. Supervisors indicated that school improvement planning is not data or evidence based and school improvement plans are copied from previous years. In the absence of identifying specific problems in teachers’ classroom practices and schools, school improvement plans will not bring the desired changes. SIP planning practices must be revisited and revised to identify causes and interventions in the teaching of different subjects and how these problems can be addressed through proper interventions by school leaders, teachers, and students.
- FG discussions in some schools revealed that teachers are marginally involved in preparing school improvement plans as planning is assumed to be the responsibility of school leaders. Most school improvement plans have general strategies for all school teachers and subjects and do not specifically show the expected changes in classroom practices and identify what needs to be changed and how it can be changed. Supporting schools to develop SIP based on organic problems inherent in a school in a simplified manner and state explicit and specific interventions and expectations could enhance the practicality of school improvement plans.
- Schools reported that they have begun using GEQIP II funds for professional development activities. Studies by the Ministry of Education and local researchers indicate that current CPD practices have serious conceptual and practical gaps. Schools need to be supported to design CPD activities that enhance teachers’ subject knowledge and pedagogical competence and classroom practices. One useful suggestion in this regard could be that teachers plan lessons together, observe each other’s class and discuss the content and pedagogy used in the classroom. This can be facilitated by a teacher selected from each department. Organizing a few days training or workshops may not bring about the intended impact on teacher’s practices and students’ learning outcomes.
- The SG guideline offers the same guideline to all schools in Ethiopia. Rural and remote schools indicated that they could not identify proper bidders in their surroundings to buy laboratory items and other supplies. School support system that enables rural schools to procure necessary items by capping additional grant so that the purchasers can go and purchase from cities or ways of procuring these items by the cluster school should be arranged.
- Principals, supervisors, teachers and students provided very positive and high ratings for the relevance and contribution of GEQIP II in almost all GEQIP related interventions. Contrary to supervisors’ and teachers’ ratings, principals’ rating on the contribution of GEQIP in improving learning conditions of special need education students was significantly below the expected mean.
- When students were asked to rate the frequency in which major teaching and learning activities occur in class, teachers’ readiness to support students learning, writing notes on a black board and taking notes, and teacher’s coming to class on time recorded the highest mean ratings. On the other hand, teachers’ correcting of students’ homework in class, giving feedback to classwork, students’ practice of bringing textbooks to class, and asking questions to teachers when they don’t understand a concept or topic are significantly rated below expectations. This concords with the other finding that textbook utilization is low, classrooms are lecture or explanation dominated, and feedback is not usually given to students. Students also reported that teacher absenteeism is more often than “seldom”.
- There are many problems in the education systems that are out of the scope of GEQIP II. Teachers’ motivation and commitment, parents’ and students’ awareness on the importance of education, cheating and disciplinary problems, and macro socio-political problems are some to mention. GEQIP II can help to address some of these problems through enhancing the institutional capacity of schools and interventions on curbing some of the student related problems.