How do we reliably measure what distance-learning methods actually work for children?

Source: Save The Children

Save the Children’s education program in Somaliland distributed 8,000 solar radios with MP3 players to students across the regions of Somaliland. This intervention helped students to continue learning and cover the missed lessons due to COVID19 school closures. Mustafa Hassan Abdillahi / Save the Children

This is a question that Save the Children, Noam Angrist (co-founder of Youth Impact) and Angelica Ponguta (researcher in the Child Study Centre at Yale University), discussed in a recent webinar: Remote Assessment of Learning: Filling a Critical Gap in our Assessment Toolbox.

In early April 2020, to halt the spread of COVID-19, an estimated 1.6 billion learners globally were impacted by school closures. For the first time in human history, an entire generation of children had their education disrupted.

It was clear that when schools closed, far too many national education systems were ill-prepared to deal with the challenges of providing good quality learning opportunities remotely in a timely manner, and in a way that could be effectively measured.

However, all was not lost, this was also a time of major innovation. Save the Children and other organisations distributed paper materials and broadcasted lessons through radio and television as well as through mobile phones and online, in contexts where resources allowed.

Like 26 million other children in Ethiopia, Mahadiya, 13, was out of school because of the coronavirus, but thanks to Save the Children’s camel library, she was able to continue reading and learning at home. Save the Children / Seifu Asseged

Almost two years’ later, we can view those strategies as a lifeline in an unprecedented global crisis, but looking forward, we should strive to be prepared for the next time. How do we know which distance-learning methods actually work for children?

Save the Children’s answer to this problem – the ReAL project

Over the last year, we have been testing The Remote Assessment of Learning (ReAL) tookit. This aims to provide the international development sector with a valid and reliable way to evaluate the most effective remote learning approaches, specifically based on children’s ability to read, complete basic math, and demonstrate social-emotional skills. This is the first of its kind.

In our webinar, the key note speaker, Noam Angrist explained how his organisation, Youth Impact, explored how to stay connected to children and their families during the pandemic. He noted that 80-90% of households had mobile phones (including simple feature phones, not only smart phones) across low, lower middle, and upper middle income families. Because of this, Youth Impact decided to design their approach to learning and assessments utilising the devices that households were already using. The results have been very enlightening and have helped inform Save the Children’s approach.

So how does ReAL work & what are the results?

For children’s learning to be tested by ReAL, the only requirement is a basic phone. Of course, we recognise that accessibility varies between contexts and that is why the toolkit comes in three ‘sizes’: High Access (for students with smart phones), Low Access (for students with standard feature phones), and Caregiver Access (for caregivers that have phones and we cannot speak to their child).

The first phase of the ReAL project concluded in early 2022. It was run in four countries: Bangladesh, Guatemala, the Philippines, and Zambia. The data from this phase shows that testing with ReAL produces similar results to other tools that are designed to be administered in-person. This provides promising evidence that we can reliably assess children’s progress, even in situations where we cannot meet them face-to-face: whether due to COVID-19 restrictions, lack of access within a conflict zone, or other types of emergencies.

Amal, 7, sharing day’s homework with her cousin Amina who is living in an area where is no school. Save the Children

Another key finding was that the uptake of testing was relatively high. Youth Impact’s findings also bolster this point, with Noam explaining that their randomised control trials in Botswana and Nepal found response rates to be 70-90%, which is “extraordinary”. We attribute this to the fact that phones are easily accessible – they already belong to the household, and air time is relatively cheap.

In addition to ReAL’s flexibility and its low tech requirements, phone-based assessments in general, are extremely cost-efficient. In Bangladesh, we found remote assessments to cost around $4.93 per child and in Zambia around $15 per child. Likewise, Youth Impact’s study in Botswana cost $4.40 per child. In contrast, PIRLS, an in-person assessment, costs approximately $62.50 per child. That said, PIRLS is more intensive, has more questions and serves a critical purpose. But this puts in perspective the potential role of phone based assessments, especially in frequent testing.

What’s next for the ReAL project and how your organisation can get involved

For phase two of this project, Save the Children has partnered with Yale University. We will work together to coordinate and analyse data from nine test sites (countries), in both humanitarian and development contexts. Excitingly, our partnership with Yale also allows us to bring in local universities and research institutes, whose input for each site is critical.

The ReAL project brings us one step closer to understanding what remote learning methods actually work for children. If children can receive the best education possible, feel confident in what they’re learning and be placed at the right level on returning to school, this will have a subsequent effect on their future attainment and chances of breaking the poverty cycle. The impact, to put it simply, can be intergenerational. 

Rammilan teaches children in his village for free and is now supporting 18 children with their studies. He has also provided writing materials to children who couldn’t afford them.

MIL OSI