Measuring Nonprofit Outcomes: Choosing Indicators and Handling Missing Responses
On SIMI indicator examples, JANPIA evaluation materials and a hypothetical case: measurement conditions, missing responses and the distinction between change and project effects
Outcome indicator databases offer examples rather than proof of results. Drawing on SIMI and JANPIA primary sources, this article examines populations, methods and missing responses. A clearly hypothetical example distinguishes observed changes from program effects and asks how evaluation informs subsequent decisions.
TL;DR
- SIMI provides examples; selecting an indicator does not establish that an organization can measure it appropriately.
- KPIs can track outcomes. Define populations, timing and methods, and read the original instrument’s tested scope.
- Respondent percentages differ from whole-population findings. Keep missing responses distinct from no change.
The cover is an AI-generated conceptual image, not a photograph of an actual survey or collected responses.
What Is Happening
SIMI provides examples; selecting an indicator does not establish that an organization can measure it appropriately.
Finding an indicator and being ready to measure it
Nonprofits considering outcome measurement can look for candidate indicators in a database. The provides a database searchable by field, beneficiary population and keywords. Beneficiaries are people a program seeks to benefit.
The page presents indicators as examples and cautions against treating them as a standardized or comprehensive set. Putting a discovered name in a report does not establish that participants’ changes have been measured. Organizations still need to examine its source, what its questions investigate and the population for which the method is intended.
An absence of matching items also does not establish that an activity produces no outcomes. Conversely, a similar item name does not confirm a method’s suitability. Describe the population and purpose before searching, then return from a candidate’s name to its original measurement method. Search results should not become judgments of a program’s value.
Here, an outcome means a change in a population’s circumstances or behavior. Counting workshops and examining whether participants use what they learned in everyday life answer different questions. Activity records also help check whether planned support reached people. What was counted matters more than the number’s size.
Reading the scope of the latest guidance
The publishes revised May 2024 evaluation guidelines and July 2026 guidance for fund distributing organizations in investment programs. The latter addresses post-investment impact measurement and management, including impact KPIs linked to short- and medium-term outcomes. A KPI is an important indicator tracked in a program; it is not an opposing concept to an outcome.
These materials concern the Dormant Deposits Utilization Program. Investment-program guidance does not establish mandatory procedures for every nonprofit. Check whether the materials concern grants or investments, which version applies and which application or contractual conditions govern the organization. Distinguish shared measurement ideas from institutional requirements.
Grant recipients and investees may also face different reporting audiences and management decisions. Publication of new guidance alone does not establish that an existing evaluation plan must be replaced entirely. First identify the applicable materials and explain the purpose of consulting additional ideas.
Background & Context
KPIs can track outcomes. Define populations, timing and methods, and read the original instrument’s tested scope.
Distinguishing purpose, indicator, target and method
A purpose such as reducing isolation does not itself determine what to examine. If an indicator asks whether someone has a person to consult, first define whose circumstances are examined and when. Whether having such a person is equivalent to being able to consult them when needed may require discussion with participants. This illustrates indicator design; it is not a finding from a particular program.
JANPIA’s reference materials distinguish intended outcomes, , target values and measurement methods. An indicator examines change; a target specifies a desired level. Setting a target and observing that value are different events. Changed questions, response methods or populations can alter comparison conditions even when an indicator’s name stays the same.
People a program seeks to support, people it actually reaches and people answering its survey may differ. Describing participants’ responses as change across an entire community exceeds the records’ scope. Specify the program and survey populations separately, including whether people leaving support are included, so readers can follow who was counted.
What to examine
- Outcome
- Change
- Indicator
- Measure
- Target
- Desired level
How to examine it
- Population
- Included
- Timing and method
- Conditions
- Missing responses
- Keep unknowns
The article’s schematic synthesis of JANPIA’s evaluation guidelines and reference materials. It is neither a particular program’s indicators nor a compulsory scoring sheet.
Sharing the purpose first can help avoid selecting only readily available data as evidence of outcomes. More records do not necessarily reveal the changes that matter. Deciding who will use the records for which decisions before adding indicators can keep collection from ending with reporting alone.
Original instruments and the populations in which they were tested
SIMI advises users to read original papers and understand an indicator’s underlying theory and tested population. An instrument uses questions or other observations to examine a condition. Another organization’s instrument does not automatically have the same meaning for people with different ages, languages, circumstances or ways of answering.
Answering a staff member in an interview and completing a form alone may differ in how comfortable someone feels responding. If questions are shortened or translated differently, record the changes from the original method. Both reducing burden and preserving suitable comparison conditions need consideration.
Response rates alone cannot establish whether people understood a question. JANPIA’s reference materials also discuss pilot surveys. Checking unclear wording or difficult items before distribution and retaining revised questions helps prepare for asking under the same conditions next time. Complete answers and measurement of the intended concept need separate checks.
SIMI distinguishes citing original literature when reporting its research findings from crediting the database when reporting an application of it. It also asks users to contact it about commercial use. Access to a public page does not establish unrestricted republication of every item or freedom to use the material for every purpose.
Reading the Structure
Respondent percentages differ from whole-population findings. Keep missing responses distinct from no change.
People who responded and people in the population
Consider a hypothetical example: 20 people are supported, 12 answer the same follow-up question and nine report a positive change. These are explanatory figures, not program findings or public statistics. Nine divided by 12 gives 75%, the percentage of respondents reporting positive change. It does not establish a 75% outcome across all 20 people.
Nine divided by 20 gives 45%. That describes people with confirmed positive responses as a share of the full population; it does not mean the eight nonrespondents experienced no change. Their reasons for not answering and their circumstances are unknown in this example. Presenting respondent percentages alongside whole-population confirmation makes those unknowns visible.
Respondent results
- People responding
- 12 people
- Positive responses
- 9 people
- Among respondents
- 75% (9/12)
Full population
- People targeted
- 20 people
- Positive confirmed
- 45% (9/20)
- No response
- 8 unknown
All figures are hypothetical, not actual results or statistics. Nonrespondents are not counted as negative responses. Neither percentage represents effects caused by the program alone.
Before-and-after comparisons also require checking whether populations changed and whether the same people answered under comparable conditions. A higher average need not represent individual changes if the respondent mix differs. Keep observed records separate from records unsuitable for comparison rather than filling gaps by assumption.
Where ascertainable, distinguish unanswered questions, departure from support and loss of contact after a move. Unknown reasons should remain unknown. Avoid treating nonresponse as no change or assuming that nonrespondents improved at the same rate. Staff impressions should not be counted as participants’ answers.
Avoid constructing outcomes from only easy-to-measure people
JANPIA’s 2024 guidelines warn against selecting only people for whom outcomes are easy to demonstrate. Retaining only convenient responses risks omitting experiences of people who find it harder to be heard. If a survey relies on online responses, consider how to include people who cannot use that method or find reading and writing burdensome.
The reference materials address question burden, consent, the right to refuse and personal information. Distinguish access to support from cooperation with research and explain that refusal will not disadvantage someone. Decide how to explain the survey, who can access records and what will be retained before aggregating results.
When preparing a survey, also consider whether names or detailed personal circumstances are needed. Matching records internally and reporting results externally may require different levels of detail. In a small program, deleting names alone need not prevent identification. Compare the uses explained to participants with the scope of published results and cases.
More follow-up attempts are not necessarily better. Decide who may contact someone, for what purpose and through which methods. Organizations with limited staff should also share why they selected fewer items and what they could not examine. Concealing unknowns differs from setting a proportionate survey scope.
Returning measured changes to subsequent decisions
The investment-program guidance describes data collection, impact verification, reporting and feedback into the program as stages. Moving from observed changes to a judgment about the program’s contribution requires considering other influences and comparisons with what might have happened without it. A before-and-after difference alone cannot identify a single cause.
Before concluding that a missed target means failure, examine whether planned activities reached people, questions matched their purpose or participants’ circumstances changed. Simply rewriting targets to match results does not create a record of learning. Retaining reasons, consultation partners and dates of changes helps explain conditions for interpreting subsequent results.
Improvement discussions should also consider burdens participants experienced and unexpected changes, alongside favorable figures. If measurement takes so much time that support time falls, that can justify reviewing the indicator’s operation itself. Keeping or reducing indicators involves comparing purpose with available resources. Connect the numbers required for reporting with the conditions for continuing support.
Remaining Questions
Ask whose experiences are missing, how methods change and how participants receive results.
How will organizations using indicator databases examine experiences absent from their records? Alongside respondents’ changes, explain who was targeted, who did not answer and how far the reasons could be established. Additional contact can burden nonrespondents. Can an organization prepare options that participants can choose, allowing continued support and freedom to decline the survey?
When using common indicators, how far can differences in measurement conditions be disclosed? Identical questions may yield results unsuitable for direct comparison if ages, languages, timing or response methods differ. Avoid sacrificing people’s circumstances to comparison or ending examination with the assertion that circumstances differ. Share the aggregation scope, unknowns and survey costs, and assign responsibility for deciding whether comparison is appropriate.
What changes after measurement, and who receives the findings? If results differ from targets, can the organization record which activities, targets or questions it revised? Alongside reporting to funders, can it explain the findings to participants in an understandable form? Selecting an indicator is not itself an outcome. Evaluation must continue through decisions about subsequent support using both confirmed changes and unrecorded experiences. When a successor uses the same indicator, can they trace why questions changed and which responses were excluded from comparison? Can participants’ questions about the findings be retained, without confusing refusal to join a survey with ending support, and used to discuss how the next assessment will be conducted?
References
Outcome Indicator Database — Social Impact Management Initiative (SIMI). SIMI
Evaluation Guidelines and Related Materials — Japan Network for Public Interest Activities (JANPIA). JANPIA
Evaluation Guidelines for Fund Distributing Organizations, Activity Support Organizations and Implementing Organizations — Japan Network for Public Interest Activities (JANPIA) (2024). JANPIA
Evaluation Guidelines for Fund Distributing Organizations in Investment Programs under the Dormant Deposits Utilization Program — Japan Network for Public Interest Activities (JANPIA) (2026). JANPIA
Reference Materials — Japan Network for Public Interest Activities (JANPIA) (2022). JANPIA
For the overall design process, the outcome indicator design guide（このサイトの記事） examines activity counts, participant changes, starting records and evaluation questions.
Further Reading
ソーシャルセクターのための事業マネジメント実践ガイド――混沌から「チームの道」を見出し、社会的インパクトを創出する（外部サイト、新しいタブで開きます） is related reading on managing social-sector programs, described in the publisher’s product introduction（外部サイト、新しいタブで開きます）. Only the public introduction, contents and bibliography were consulted; the full book has not been read. The book is not used as evidence for measurement methods or current institutional conditions, which are examined using the primary sources above.
An English rendering of the full Japanese title is Practical Guide to Business Management for the Social Sector—Finding Your Team’s Path Through Complexity and Creating Social Impact. This is an unofficial translation, not an English edition.
Amazon links in this article are affiliate advertising links.
Questions to Reflect On
- Are changes reported by respondents being described as outcomes for everyone supported?
- If results differ from the plan, can the organization explain which activities, targets or questions it changed?
Key Terms in This Article
- Outcome Indicator
- A measure intended to examine changes or benefits relevant to the problem a program seeks to address. Its design distinguishes outcomes from immediate activity outputs; improvement in a measure alone does not establish the program's causal effect.
Related Content
Designing Outcome Indicators — Evaluation Thinking for Social Projects Beyond KPIs and KGIs
Participation counts and event tallies fill the report, yet no one can answer 'So what changed?' A guide to designing meaningful outcome indicators.
Series: Evaluation & Measurement Frameworks
Designing Collective Impact — How to Move Intractable Problems Through Cross-Sector Collaboration
What does it take for multiple organizations to tackle structural challenges no single entity can solve? An introduction to designing collaboration as a system.
Series: Evaluation & Measurement Frameworks
Writing a Grant Application — A Practical Checklist to Improve Your Approval Rate
Writing each grant application by trial and error, never understanding why the last was rejected? Rebuild your proposals from the reviewer's perspective.
Series: Funding & Grants
Cash Flow Design for Nonprofits — A Practical Guide to Solving Funding Gaps Through Structure
Your grant was approved, yet you still cannot cover expenses. The cause may not be a lack of effort but a structural flaw in your cash flow.
Series: NPO Management & Operations
Get new columns by email
1-2 social structure analysis columns per week. Free to subscribe.
Join ISVD's activities?
Sign up to receive the latest research and activity reports. Feel free to reach out about collaboration or project participation.
Support ISVD’s research and activities
Support ISVD’s research and publications with JPY5,000 per year. There is no automatic renewal.