Can Vocational Training Be Measured? — EBPM and the Evaluation Design of Workforce Development Policy

Introduction

The Japanese government has committed to a "1 trillion yen over five years" reskilling investment, expanding public spending on vocational training through enhanced education and training benefits and human resource development subsidies. In fiscal year 2023, approximately 44,700 individuals enrolled in job seeker support training, and the employment placement rate for displaced worker training was reported at roughly 70–80%.

However, these employment rates lack comparison against the counterfactual — what would have happened without the training. Can figures contaminated by selection bias truly be called "effects"? The structural gap between investment scale and impact evaluation defines the current state of Japan's vocational training from the perspective of EBPM (Evidence-Based Policy Making).

---

The Gap Between Investment and Evaluation

International Comparison of ALMP Spending

Examining active labor market policy (ALMP) expenditure as a percentage of GDP reveals Japan's position starkly.

Denmark

1.3%

Sweden

0.9%

Germany

0.6%

OECD Avg.

0.41%

Japan

0.15%

Source: OECD (2023)

Japan's ALMP spending stands at approximately 0.1–0.2% of GDP — less than half the OECD average of 0.41% and an order of magnitude below Denmark's 1.3%. The problem is not merely the scale of spending. The absence of mechanisms to evaluate even this limited investment constitutes a "double lag" that defines Japan's structural challenge.

Where Is the Rigorous Evidence?

Studies that employ counterfactual analysis to evaluate Japan's public vocational training can be counted on one hand. The most notable is Hara's (2021) propensity score matching study at RIETI, which used microdata from the Employment Status Survey to assess the effects of displaced worker training.

Metric	Effect	Source
Male employment rate gain	+15.4pp	RIETI (2021)
Female employment rate gain	+17.4pp	RIETI (2021)
Displaced worker training placement	70–80%	MHLW
Gyoda City SROI	3.78	SIB pilot
Denmark ALMP mandatory	High transition	OECD
Germany apprenticeship retention	74%	OECD

* pp = percentage points. RIETI study uses propensity score matching for counterfactual comparison

The finding that displaced worker training increased employment rates by 15.4 percentage points for men and 17.4 percentage points for women represents a significant contribution. However, these results are limited to short-term effects within one year of training completion. Wage effects and long-term career impacts remain unexamined.

---

Why Rigorous Evaluation Has Not Advanced

While RCTs (Randomized Controlled Trials) are considered the gold standard for policy evaluation, structural barriers impede their implementation in Japan.

1. Tension with equity principles: Deep administrative and ethical resistance exists toward randomly assigning policy beneficiaries into treatment and control groups
2. Personnel rotation cycles: Analysts transfer before studies are completed, preventing knowledge accumulation
3. Fragmented data infrastructure: Training enrollment data from Hello Work, employment insurance data, and tax data cannot be linked across ministries
4. Absence of longitudinal tracking: No panel data exists for medium- to long-term employment and wage outcomes after training completion

These barriers point not to isolated technical problems but to the absence of a culture that embeds policy evaluation into institutional design.

---

Evaluation Framework — Kirkpatrick + SROI

The Kirkpatrick four-level model is widely referenced for training evaluation, yet systematic implementation in Japan's public training is limited to Level 1 (participant satisfaction) and a partial application of Level 3 (employment rate tracking).

LevelEvaluation focusJapan's status

Level 1ReactionParticipant satisfactionImplemented

Level 2LearningKnowledge & skill acquisitionPartial

Level 3BehaviorEmployment rate & behavior changeEmployment only

Level 4ResultsWage change, productivity, tax revenueMostly unmeasured

SROISocial Return on InvestmentMonetization of social valueLimited cases

For Level 4 (societal outcomes), applying SROI (Social Return on Investment) offers a viable approach. The employment support program in Gyoda City achieved an SROI of 3.78 through a Social Impact Bond (SIB) evaluation model — meaning that each yen invested generated 3.78 yen in social value. Combined with participant career determination rates exceeding the national average by 5.9 percentage points, this case demonstrates that outcome-linked evaluation models can be applied to public training programs.

---

Lessons from International Evaluation Design

Denmark: Evaluation Culture Embedded in Flexicurity

Denmark's flexicurity model integrates a flexible labor market, generous social security, and active labor market policies into a unified system. Recipients of unemployment benefits are required to participate in ALMP programs, and program evaluation is embedded within the policy cycle. The decisive difference from Japan lies in treating evaluation not as an afterthought but as a structural component of the system itself.

Singapore: Quality Control in SkillsFuture

Singapore's SkillsFuture Career Transition Programme publicly reports that 55% of completers secure employment within six months. In 2024, approximately 15% of audited training providers were removed for failing quality standards. The willingness to apply rigorous evaluation to the supply side of training represents a model Japan should study.

Germany: Tracking Outcomes in the Dual System

Germany's dual vocational training system achieves a retention rate of approximately 74% — trainees who remain with the company where they trained. The structural integration of in-company training with vocational schools ensures a strong link between training and employment outcomes.

---

Evaluation Design for NPO Practitioners — Three Steps

For NPOs and social sector organizations conducting employment support and vocational training programs, the following steps provide a practical starting point for impact measurement.

Step 1: Make the Causal Hypothesis Explicit Through a Logic Model

Structure the causal chain — "Provide training → Participants' skills improve → Employment rates rise" — as a logic model. This transforms vague "support" into a measurable intervention. See also Introduction to Social Impact Evaluation.

Step 2: Select Outcome Indicators Carefully

Choose one or two indicators corresponding to Kirkpatrick Levels 3–4. Specify the timing and measurement method concretely — for example, "employment retention rate six months after training completion" or "hourly wage change rate before and after training." For detailed guidance on indicator design, refer to Outcome Indicator Design.

Step 3: Secure Baseline Data and a Comparison Group

Post-program surveys alone cannot establish causality. Always conduct baseline measurement before the program begins, and where possible, attempt comparison with a wait-list group or non-participants of similar attributes. Even without a full RCT, accumulating pre-post comparison data serves as the departure point for EBPM practice.

---

Conclusion

Japan lags significantly behind OECD peers in both the scale of vocational training investment and the rigor of impact evaluation. The reality that rigorous evidence for a trillion-yen-scale public investment is limited to a handful of RIETI research papers represents a fragile foundation for policymaking.

Establishing evaluation designs that extend to Kirkpatrick Level 4 and SROI — in both public training and NPO programs — is an urgent priority. Cross-ministerial data linkage and medium-term tracking systems must advance in parallel. Denmark, Singapore, and Germany each demonstrate the same lesson: evaluation is not an afterthought but a structural component of the system itself. Japan is only now approaching that starting line.

References

Effects of Public Job Training in Japan

Hara, H.. RIETI Discussion Paper 21-E-027

Read source

Active Labour Market Policies in OECD Countries: Why Renewed Investment Matters

OECD. OECD Blog

Read source

Social Impact Evaluation Toolkit

Cabinet Office, Japan. NPO Homepage

Read source

Analysis of Public Vocational Training Effectiveness

Ministry of Health, Labour and Welfare. Employment Insurance System Study Group (3rd Meeting)

Read source