Microfinance Practice: Re: [MFP] Paper available: "How Do the Poverty Scorecard and the PAT Differ?"

Thanks Julie and Margaret for an update on some practical ways in which these poverty tools are being used. And it's good to hear the tools are being employed outside of microfinance. I recognize that our understanding of poverty (or happiness for that matter!) is virtually always incomplete. The original intention of the poverty scorecard was to create something short, simple and relatively easy to provide MFI staffs a snapshot on which to reflect and make decisions (e.g., are we reaching the right people? If not, why? What do these changes suggest for our services or the changing context? What else do people need? Etc.).

And thanks Malcolm and Mark. Back when Mark, Michal Matul and I worked through what seemed like a conceptually elegant scorecard approach to assess likely poverty status and change in status over time, we weren't sure how long it would be before anyone would genuinely report on change in status. While it's probably cost-effective for most organizations to sample, we envisioned the scorecard being used by at least some MFIs with sophisticated enough MIS to track and calculate the scorecard's indicators on a census basis for all clients (we were happy to find the developer of Kredits MIS willing to incorporate scoring). At the same time, we developed a dropout and credit scoring system (both also incorporated into Kredits MIS). In my wonky experiment, I was interested in understanding what it would mean for an institution to be able target those who were (a) likely to be very poor, (b) likely not to drop out and (c) likely not to default. 10 or so years on, I'm curious to understand to what extent organizations (or individuals) can report on the poverty scoring experience (at least) and what it is telling us, if anything.

Cheers,
Sean

On Mar 3, 2014, at 6:55 AM, <margaretelise@gmail.com> <margaretelise@gmail.com> wrote:

Hi Sean,
I think this paper offers a great example of how PPI was used in combination with PWR and another household survey to target services of BASIX (India). Fonkoze also has a number of documents describing how it has used PPI data (in conjunction with other data) for social performance management, notably to manage impacts of the earthquake.

But point-in-time and change-over-time analysis is essentially what these tools are designed to do (as are most M&E activities). As Julie said, the tools are being used in contexts outside of microfinance (most PAT users are not MF): FHI360's STRIVE project has used both tools (documentation forthcoming) in the context of VCD; PAT is being used as part of a larger tool to assess impacts of a women's empowerment program in multiple countries. Another project was working on integrating GPS data to track changes by location more accurately. The LIFT project is using PPI & PAT (modified) as part of a larger survey to develop a referral system to help health clinic patients link with economic strengthening services.

I've also helped people analyze their PAT data more deeply to learn from specific indicators, client data and disaggregated results (many people use the rural/urban and regional PAT indicators to compare results).

I will point out that a number of organizations have found that these tools are not gener ally sensitive enough to track changes over short periods of time. I'm working on a new project with Abt Associates that attempts to address the issue of change over time sensitivity; we are piloting tools in Uganda & Tanzania - if anyone has particular interest in these countries please contact me directly as we are looking for input on their design.

For this and other reasons (like getting a more holistic view of poverty), I generally recommend people use more than one tool: PAT (and PPI) users will combine a tool with food security measures, a housing index, asset indicators, self-reported income data, etc (see these videos I produced last year; essentially poverty tools 101). Though even doing so people will often see inconsistent results. We did an e-consultation through POWG (now STEP UP) a few years ago to discuss issues Freedom From Hunger and Trickle Up had in interpreting PPI & food security data (basically, the results were inconsistent: PPI score going down while food security increasing). FFH has since developed an Impact Stories methodology that really does a lot to highlight the complexity of poverty and how household decisions are made (I'm a big fan of this and anything that uses qualitative info to explain quantitative findings). And everything we know about impacts of MF (summarized in Goldberg 2005, Odell 2010, EPPI Centre 2010) points to inconsistent client outcomes related to poverty and its many dimensions (food security, housing, education, empowerment, etc.)

So back to your question about how the tools are being used, Sean, I brought up the points in I did abo ve to emphasize that they are being used to tell a piece of the story of poverty outreach and changes over time but that the story is incomplete without deeper measures.

Thanks,
Margaret Richards
PAT Help Desk Director (helpdesk@povertytools.org)
STEP UP Facilitator & consultant (margaretelise@gmail.com)

On Mar 2, 2014, at 7:57 PM, peacheyj@gmail.com wrote:

Hi Sean,

Great question! Grameen Foundation has a case study on the PPI website about Grameen Koota's use of the PPI to track poverty movement over time - http://www.progressoutofpoverty.org/sites/default/files/Grameen%20Koota%20PPI%20Mini%20Case%20Study.pdf. We are also currently in the process of conducting analysis for a few other organizations who have multiple years of PPI data, with a report coming out in about a month. We'll be coming out with guidelines for conducting that type of analysis also.

In addition to movement over time, there are more and more organizations using the PPI now in multiple ways and in many sectors including agriculture and health. See more case studies - http://www.progressoutofpoverty.org/case-studies-reports. We are working on a 'PPI Global Use Report' which will highlight the many users and uses of the PPI globally. If your organization uses the PPI and you aren't sure if we know about it, please contact Lindsey Alexander at lalexander@grameenfoundation.org.

Best regards
Julie Peachey
Director, Social Performance Management
Grameen Foundation
jpeachey@grameenfoundation.org

On Mar 2, 2014, at 9:52 AM, Mark Schreiner <mark@microfinance.com> wrote:

Dear Malcolm:

Good to know that your organiz/sation is finding the PPI useful. I think we had some discussion in the past about whether the all-India PPI is appropriate for all of India, which as you point out is huge and widely varied. As we discussed, I have found that people have more or less the same basic goals/needs at the household level everywhere, and thus there are many indicators that "work" in most of India as well as in most "very poor" countries. I don't doubt Assam is very different from Kerala, but I would be surprised if there are more than one or two questions in the scorecard that don't apply in one or the other. (And just having all the answers be always "yes" or always "no" does not necessarily mean that the indicator is not appropriate.)

I'd be happy to do scorecards for each Indian state, Somaliland, etc., the constraint is usually data and funding. In most countries, the national expenditure survey just does not have the sample size required to make anything beyond (perhaps) urban and rural scorecards possible.

For Somaliland, it can be done, there just has to be data (that the government is willing to share) and funding.

In the case of India, the national expenditure survey is a lot larger than just about any country except maybe Indonesia, so the sample sizes (about 8,000 HHs in Assam, about 11,000 in Kerala) would actually permit state-specific scorecards in those states (and in several of the other larger states), and regional scorecards in others. I did test doing separate scorecards for 4 regions in India (SW urban, SW rural, Northcentral urban, Northcentral rural), but accuracy improved in 2, and got worse in 2, compared to just applying a single all-India scorecard everywhere.

Finally, applying the PPI based on recall is something I understand, but it is an off-label use. Accuracy surely suffers, mostly, of course, from recall bias and other non-statistical things. Nevertheless, I would still expect the results to be "directionally correct". The challenge is then to compare that with some benchmark to then judge if there is any impact. After all, real decisions proceed on the basis of judged impact, not just "change". Of course, we are not talking about "academic" or 100%-certain impact, but the normal we-have-got-to-decide-something-based-on-what-we-have-as-best-as-we-can impact. And that process should still be "rigorous" (explicit, quantitative, well-discussed and open to critique) even though it would not satisfy academia's false standard of rigor=100% certainty. The point is, with real decisions, you don't want to be wrong or fool yourself, even though you have imperfect information etc. and you recognize that sometimes you will end up not making the best decision. I think this is more or less what you have also been saying in many contexts and ways, and also fits with Kim Wilson's recent critique of academic research.

Kind regards,

Mark

From: Malcolm Harper

Sent: Saturday, March 01, 2014 4:36 AM

To: Microfinance Practice

Cc: John Belt ; Rajeev Roy

Subject: Re: [MFP] Paper available: "How Do the Poverty Scorecard and the PAT Differ?"

Thanks Sean, I and my two co-authors (John Belt of KIT and Rajeev Roy of IIM Raipur) of a book/study of what we call 'for profit inclusive value chains' (VCs that substantially benefit large numbers of poor people and have not and are not aided, subsidised and so on) hardly qualify as 'an organisation'. But we have been using the PPI to get an idea of how participating in the VCs benefit people. Our case writers ask a small sample of them to answer the questions today, and five years ago or whenever they first joined the VC. Not very systematic, for sure not an RCT, but it's useful.

Recalling the answers does not seem to be very difficult, people do remember more or less when they acquired this or that important asset, or household members. The main problems have been the inapplicability of the questions (Assam in North East India is far more different from Kerala in South West India than say Malawi is from Zambia, but there's one PPI for all of India) and the lack of any PPI for places such as Somaliland.

Malcolm

From: Sean Kline

Sent: Saturday, March 01, 2014 1:02 AM

To: Microfinance Practice

Subject: Re: [MFP] Paper available: "How Do the Poverty Scorecard and the PAT Differ?"

Thanks Mark.

Can you (or others) share any recent examples of organizations that are using one or both of these tools?

And for anyone out there using a poverty scorecard, have you used it beyond determining the likely poverty status of people you serve to measure changes in poverty status over time?

Cheers,
Sean

On Feb 28, 2014, at 2:57 PM, Mark Schreiner <mark@microfinance.com> wrote:

Abstract: "The poverty scorecard and the Poverty Assessment Tool (PAT) are simple, low-cost ways to measure households' poverty. How do the two differ? For estimating a group's poverty rate, both are unbiased, and the scorecard has smaller standard errors. For targeting individual households, the PAT correctly classifies about one more household per 100. The scorecard has an edge in availability, recentness, and transparency."

Download at: http://www.microfinance.com/English/Papers/Scorecard_versus_PAT.pdf

Mark Schreiner