AppLab’s Initial Social-Impact Measurement Efforts Pay Off


Eric Cantor has led Grameen Foundation’s AppLab efforts in Uganda for the past three years, and continues to serve as an advisor on the project.

Grameen Foundation takes outcome measurement seriously.  We want to make sure that our programs and services are effective, and that we can demonstrate their benefits before implementing programs or practices on a wider scale or urging others to replicate them.

With this in mind, we recently completed one of the first randomized control trials designed to assess the impact of a mobile phone-driven health service aimed at improving the lives of the poor.  The service we sought to measure was Health Tips, part of the Google SMS suite launched throughout Uganda in 2009 with our partners Google and MTN Uganda.  Our social impact partner Innovations for Poverty Action (IPA) performed the study.

Preliminary findings from the study are substantial, supporting some of our initial hypotheses and refuting others, and informing our approach to building pro-poor, mobile phone-driven solutions going forward. In short, findings indicated that when people learn of such services, they use them. People also seem to learn from this particular text-message query-based product.  But we also found that, because of the limitations of human motivation and barriers like language and literacy, we have a lot more work to do.

The Health Tips study was conducted in Uganda over an 18-month period. Before the launch of Google SMS in June 2009, IPA conducted a baseline survey of 1,800 people in 60 rural communities, assessing demographic profiles, attitudes, and knowledge and behavior regarding sexual and reproductive health, and collecting data from local clinics.  When we launched the service, we initiated a marketing campaign that randomly targeted half of those communities (the “treatment” areas) and did not reach the other half (the “control” areas).

A Mobile Midwife counselor talks with a client

Our studies have shown the value of "trusted intermediaries" -- such as the Mobile Midwife counselor in the photo above -- as a way to make mobile phone-based communications to the poor more effective.

Through randomization, IPA chose two sets of communities that were uniform in every relevant respect – except that one was exposed to the product through targeted marketing campaigns, while the other was not.  Nine months later, they began a follow-up survey of 2,400 people to detect changes.  They looked at data from surrounding clinics, conducted qualitative interviews and assessed the information provided to the communities. Because the targeted marketing in treatment villages was effective – we saw more than four times as much usage in the treatment areas as in the control – we were able to assess the effect of the service on attitudes, knowledge and behavior relating to sexual and reproductive health.

The first thing that is clear from this study and the user feedback we have collected over time is that, with our partners in Uganda, we built an innovative telecom product that addresses an area of high interest where few avenues for reliable and timely information previously existed. Google SMS enables people in villages in Uganda to look up information about sexual and reproductive health, as well as about agriculture and trading partners, from completely local sources, in a language and an interaction mechanism that – based on our rapid prototyping and local partnerships – conforms to the way people want to communicate. The service was well received, generating since its launch more than 4 million queries to Health Tips from more than 250,000 distinct phone numbers. People also used the service repeatedly, with more than a quarter trying it on more than 10 occasions.  We believe that by including local actors like Straight Talk Foundation in the design of the product, we were able to reach people more effectively while aligning the incentives of important stakeholders to sustain the overall initiative.  Most rewarding were the personal conversations we had, such as the woman at a rural clinic who revealed in deep detail how our service had given her the courage to get tested and treated for the sexually transmitted disease she’d contracted, resulting in better health for herself and her family.

We believe that Health Tips is among the most widely used opt-in mobile services targeting the poor.  Though today’s usage falls short of our goals – our ambitions remain much higher for uptake, impact and sustainability – our adoption numbers are still an achievement in a sector where the majority of related projects are still in a pilot phase.  Pundits praise the telecom sector’s ability to bring valuable services to large numbers of people, but other than the mobile money services just beginning to pop up across Africa we don’t know of any mobile phone-related initiatives that are reaching as many people. This may have contributed to why GSMA awarded us last year’s Global Mobile Award for the Best Use of Mobile for Social and Economic Development at the GSM Mobile World Congress for the AppLab concept.

Preliminary findings of the study indicate that improvements in knowledge about HIV and contraceptives, the main topics of inquiry, were limited. Findings also clarify that, like many technology tools, the Health Tips service is capable of reinforcing both positive and negative behaviors. Respondents who fit a risk-averse profile (especially women) became both less sexually active and less likely to engage in sex with casual partners. Others who were more prone to risky behaviors in this area were further emboldened, reporting more partners, more unfaithfulness and less use of contraceptives.  It is logical to infer that phones have the potential to reinforce existing tendencies, rather than change behavior, because they make it easier to seek and find information.

In a recent critique of the information and communication technologies for development (ICT4D) sector, Kentaro Toyama labels technology “a magnifier of human intent and capacity,” suggesting that the fundamental drivers of behavior are not altered simply because of the phone. He adds that if the phone does not address each of those drivers, behavior change is unlikely.  Because our content in this case was not particularly ideological, focusing instead on simply providing information and letting people make their own decisions, it follows that their behavior – good or bad – would not significantly change as a result of using the service.

Though we haven’t yet seen all the benefits we aimed for, we learned valuable lessons in how to improve lives through mobile phone-driven social enterprises, and are implementing what we’ve learned.  As discussed in this forum, information is most effectively delivered in a context where behavior change and learning are already emphasized, such as agriculture extension programs and maternal-health outreach programs.

We’ve also learned to focus on the way content is delivered. For example, a survey in Ghana showed that most recipients of maternal health information preferred it to be delivered by voice rather than text, so we adjusted accordingly. We’re also examining whether the method of search that resonates so well with an audience familiar with the Internet is appropriate for those who have never used it – or, indeed, for those who are unable to read and write.  Of course, we’re going to continue to explore the concepts behind how and why people change behavior based on what they hear, see and learn, and apply those lessons to real life.

Google SMS, and the AppLab concept in general, was our first foray into mobile phones for social change. We are taking what we learned and applying it to an agriculture-focused project in Uganda called the Community Knowledge Worker initiative and to a mobile phone-based livelihoods and entrepreneurship initiative known as Indonesia AppLab.  We hope that others in the field benefit from our measurement efforts and feel the same urgency to rigorously understand and document the effects of their work in this sector.  Stay tuned to this space for more information about this, including the upcoming report from IPA and more feedback from our own social measurement.

Tags: , , ,

2 Responses to “AppLab’s Initial Social-Impact Measurement Efforts Pay Off”

  1. Lessons learned: Reflections on M4D Projects | Think! blog Says:

    […] post on the Grameen Foundation blog discusses a control trial to asses the impact of a mobile for health project in […]

  2. AppLab’s Initial Social-Impact Measurement Efforts Pay Off | AppLab blog Says:

    […] the rest of Eric’s blog post by following this link. Share and […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: