Guatemala 2017 Teacher Support Results

By: Jonathan Tan, Senior Impact Data Coordinator

As Pencils of Promise (PoP) continues to scale our impact in Guatemala, support more teachers and provide high-quality education to more students, we rely on rigorous program evaluation to inform future decisions. This includes using both quantitative and qualitative results that, when taken as a whole, paint a full picture of our true impact and help us make strategic decisions moving forward.

2017 marked the fourth year of PoP’s Teacher Support (TS) programming in Guatemala. In total, we supported 37 teachers in 6 schools across the Boca Costa region of Guatemala, impacting a total of 831 students.

As always, we believe in making programmatic decisions by looking at the evidence. Many new features of our 2017 program were directly motivated by our learnings from the prior year:

In evaluating the effectiveness of our TS program, we look primarily at students’ learning gains in three literacy indicators:

In a surprise turn, we found that students in PoP schools showed smaller learning gains in Phonemic Awareness than those in Control schools. We observed the same phenomenon happening in Passage Reading scores. Lastly, we found merely comparable learning gains in Passage Comprehension between PoP and Control students.

Given our new focus on Independent readers and reading comprehension strategies, these results came as a surprise. We immediately began looking for reasons behind this, and eventually identified two primary factors:

Our 2017 program was — in one word — ambitious. In response to the prior year’s results, we attempted to cover a wide range of new teaching strategies and techniques that, as later indicated by feedback from our coaches, we now believe to be too broad. Furthermore, the focus on advanced comprehension strategies in 2017 left inadequate time for reinforcement of the foundational skills that every teacher needs. Without mastery of skills such as, say, classroom management, we believe that our teachers weren’t able to focus on and productively employ more advanced techniques in the classroom.

In 2018, we’ll incorporate training on those fundamental skills for all teachers, which should empower them to effectively use all the other techniques in their toolkit. We’ll also restructure training to encourage teachers to deepen one specific skill over time, which should prove more effective than attempting to slightly improve many skills at once.

A review of our outcomes from 2017, as well as that of past years’ results in Guatemala, have highlighted the need to refine how we can best tailor our coaching to the local context. In 2018, we’ve addressed this by hiring a Guatemalan Content Specialist, based right out of our Quetzaltenango (i.e. Xela) office. The specialist, Carlos Mendez, is longtime educator and professor of pedagogy in Guatemala, and he has already been making great progress with both the Programs and Monitoring & Evaluation (M&E) teams in-country. We believe that he’ll bring to the table a greater expertise in working with native Spanish speakers and revamp our content to the specifics of Spanish language acquisition.

Welcome to the PoP family, Carlos!

We’ve worked hard to identify flaws and develop hypotheses for potential solutions. Now, we’re taking action to gather data, test those hypotheses and adapt our work accordingly. The improvements we’re making in our approach to monitoring and evaluating our work in Guatemala include:

Who better to explain what teachers need than the teachers themselves? To obtain further insight into how we should adapt our program, we’ll conduct regular interviews with a sample of teachers in our program throughout 2018. The interviews will focus on gathering qualitative data on teachers’ experiences in and out of the classroom, their involvement with our program, suggestions for improvement and other insights. We believe that the interviews will help us to better interpret results at the end of the year, as well as influence the direction that future iterations of the TS program will take.

Our M&E approach has typically consisted of two rounds of teacher observations, with the purpose of objectively recording teacher performance and reporting results to the Programs team. By increasing the frequency of our observations to five times throughout the school year, we can report more data points to Programs and give them more opportunities to make adjustments if necessary.

There are three predominant language groups in Guatemala:

Considering the diversity in language around the country, it is challenging to tailor literacy programs to each group, especially when considering all Guatemalan public schools conduct lessons in Spanish.

Currently, our TS program exists only in schools in Mayan monolingual communities. Our hypothesis is that each community’s linguistic context has a strong impact on the ease with which our students learn Spanish and, therefore, an impact on the effectiveness of our program. To test that, we plan to expand our program to 18 schools in 2019, equally spread across Mayan monolingual, Spanish monolingual and bilingual communities respectively.

Needless to say, it’s important to PoP that we provide a quality education for all students. If we find that our program is most effective in Spanish monolingual communities, we plan to scale our TS program in those communities in the short term while simultaneously developing a version that effectively reaches those in Mayan monolingual communities. In that way, our evaluation strategy will enable us to understand what’s working now and what must be adapted to fit the needs of others.

Coaching logs are used by the TS team as a method to record outcomes of coaching sessions throughout the year and create plans for moving forward. Starting in 2018, a quantitative evaluation component (i.e., ranking scale) has been added to the coaching log that will help the coach and the teacher have a better understanding of what needs to improve and by how much. We’ve also given more attention to the feedback portion of the TS program by asking coaches to spend more time on providing critical feedback, in partnership with developing detailed plans for how improvements can be made, during each coaching session.

Correction: Feb 20, 2018
An earlier version of this blogpost stated that we’d be implementing teacher-specific training phases in Guatemala in 2018. Teacher phases have currently only been implemented in Ghana.

Related posts:

Beer lovers in Paris have a wide range of brews to choose from if only they know where to look. Craft beers and IPA’s have been on the rise in America bringing with it a host of microbreweries for…

Aerospike Connect for Presto was released earlier this year to address the need to convert data stored in Aerospike into valuable insights quickly using the Trino (formerly known as PrestoSQL)…

Whilst no one can definitively predict what the education system will look like in 20 years from now, Steve Jobs once said during his graduation speech: “You can’t connect the dots looking forward…