Strategy evaluation 2017–2021

September 2017 – December 2021: 54 Integrity Clubs, 810 students trained and accounts audited every year, for US$251,077 received of US$575,000 planned. A sound strategy, but targets too far from resources.

Translated from the original French edition.

Download the report (PDF)

1

About this report

AuthorsCERC’s monitoring, evaluation and learning team

The report was prepared by CERC’s monitoring, evaluation and learning team, with the executive management and the finance team.

The opinions, conclusions and recommendations in this report are those of the evaluation team. They do not necessarily reflect the views of the donors and partners that funded the projects evaluated.

AcknowledgementsThank you

CERC thanks the students, teachers, head teachers and facilitators of the Integrity Clubs of Uvira and Bukavu, who carried this strategy on the ground, as well as its partners:

  • Integrity Action, with NORAD and Sida
  • The US Department of State (INL), through Accountability Lab
  • Twilio
  • Microsoft
  • The UNCAC Coalition
  • USAID, through IREX
  • The Global Partnership for Education, through Oxfam IBIS
  • The education authorities of South Kivu and Kinshasa, and Group BEAG, which audited our accounts

CERCCentre de Recherche sur l’Anti-Corruption

CERC is a registered non-governmental organisation (NGO) (Registration number: F.92/42579) and given Legal Personality by the Ministry of Justice: JUST/SG/20/21311/2021. CERC holds Special Consultative Status with the United Nations Economic and Social Council (ECOSOC) since 2021.

info@cerc.cd · www.cerc.cd

LicenceCC BY-NC-ND 4.0

This work is licensed under Attribution-NonCommercial-NoDerivatives 4.0 International. To view a copy of this licence, visit creativecommons.org/licenses/by-nc-nd/4.0/

CitationHow to cite this report

CERC (2022). Evaluation of CERC’s 2017–2021 strategy: More inclusive youth engagement in improving public services in South Kivu. Centre de Recherche sur l’Anti-Corruption, Kinshasa. https://cerc.cd/en/strategy-evaluation-2017-2021/

Cover photo: training of Integrity Club teachers and head teachers, Uvira, 2019. CERC.

2

Acronyms

AcronymMeaning
CADOECD Development Assistance Committee (DAC)
CERCCentre de Recherche sur l’Anti-Corruption
CNUCCUnited Nations Convention against Corruption (UNCAC)
CODESAHealth area development committee
CLDLocal development committee
ECOSOCUnited Nations Economic and Social Council
INLBureau of International Narcotics and Law Enforcement Affairs (United States)
IREXInternational Research & Exchanges Board
ODDSustainable Development Goals (SDG)
OSCCivil society organisation (CSO)
PMEGlobal Partnership for Education (GPE)
RDCDemocratic Republic of the Congo (DRC)
SHINEStudents Acting for Honesty, Integrity and Equality
USAIDUnited States Agency for International Development
USDUnited States dollar
3

Executive summary

Background. CERC’s 2017–2021 strategy, More inclusive youth engagement in improving public services in South Kivu, was adopted on 15 July 2017, three months after CERC was founded, and came with an operational plan in September 2017. It aimed for strong links between the users and the providers of education and health services, with young people, girls and women at the heart of decisions, in the territories of Uvira and Walungu and the city of Bukavu.

Implementation in a difficult context. The 2018 elections, the Ebola epidemic in the East, insecurity in South Kivu and the Covid-19 pandemic in 2020 and 2021 – which closed schools and cut funding – weighed on implementation. CERC had to close its Uvira and Bukavu offices during the pandemic, and it began working in Kinshasa.

Why this report. The evaluation aims both to prove and improve: to account to our partners for what we did with their trust, and to learn the lessons of our first strategy to make the 2022–2026 strategy, adopted in January 2022, succeed.

Methodology. The evaluation, an internal one, is anchored in the six OECD-DAC criteria – relevance, coherence, effectiveness, efficiency, impact and sustainability – organised in three pillars: the strategy and its plan (pillar 1), the effectiveness and efficiency of our programmes, through three case studies (pillar 2), and the value, impact and sustainability of our work (pillar 3). It rests on a review of 14 documents, our audited accounts and the archives of our website; it includes no new interviews.

Key figures

US$251,077 received from 2017 to 2021, for US$575,000 planned (44%) · 54 Integrity Clubs and 810 students trained · 32% of the problems reported by students fixed · 4 of 4 financial years audited · 5 of 24 targets met, 7 partly met, 8 not met and 4 not measured.

Overall performance. The strategy laid CERC’s foundations: a method that works – Integrity Clubs –, accounts audited every year and a voice recognised up to the United Nations. But it reached only part of its targets: with 44% of the planned funds, clubs and students trained stayed five times fewer than planned.

Pillar 1: the strategy and its plan. The diagnosis was right and the choice of young people relevant, as confirmed by our 2017 survey of 980 households. The health component was hardly implemented, and the plan’s targets were set without regard to the funding secured. Our annual reports followed four objectives that differ from the plan’s, which made monitoring difficult.

Pillar 2: our programmes. SHINE trained 810 students in 54 Integrity Clubs in Uvira and Bukavu and fixed 32% of the problems reported, for about US$163 per student. The small 2020 projects were managed rigorously, to the point of returning unspent funds to the donor. In 2021, Youth Excel and the year zero of Students Acting prepared our work in Kinshasa.

Pillar 3: value, impact and sustainability. Club students know how to monitor and report, schools were repaired, and CERC became a partner of the education authorities and an NGO in special consultative status with ECOSOC. But 53% of funds came from a single donor, and CERC had neither a reserve nor own funds before 2021.

What should change. Set targets that match the funding secured, and review them every year; diversify donors and build a reserve; report on the plan’s targets in every annual report; and embed the clubs in school rules so that they live without funding.

Overall conclusion. A sound first strategy, carried by a small team, that learned faster than it grew. The 2022–2026 strategy must set targets that match our means, and raise means that match our ambitions.

Organisation of the report. Chapter 1 presents the methodology, its questions, pillars, sources and limits. Chapter 2 presents the findings, pillar by pillar, and a synthesis against the DAC criteria. Chapter 3 presents the recommendations, lessons and suggestions in a table, with their category and criticality. The annexes present the evaluation requirements, the choice of case studies, the team, the work plan, the documents reviewed and the funds by donor.

4

1. Evaluation methodology

The evaluation follows an approach anchored in the six OECD-DAC criteria – relevance, coherence, effectiveness, efficiency, impact and sustainability –, attentive to the context of eastern Congo, where public services, conflict and corruption are linked. It treats CERC as one actor among others in a local system: schools, education authorities, donors and civil society.

1.1Evaluation questions and pillars

Table 1: evaluation questions, pillars and weights

Evaluation questionPillarDAC criteriaWeight
1. Did the strategy and its operational plan meet users’ needs, and were they coherent with policies, partners and our own monitoring tools?Pillar 1: the strategy and its planRelevance, coherence30 %
2. Did our programmes reach the plan’s targets, and at what cost?Pillar 2: programme effectiveness and efficiencyEffectiveness, efficiency45 %
3. What changed for students, schools and institutions, and will these gains last?Pillar 3: value, impact and sustainabilityImpact, sustainability25 %

1.2Instruments and expanded questions

Table 2: expanded questions and instruments

PillarExpanded questionsInstruments
Pillar 1Was the diagnosis right? Were the targets realistic? Did our reports follow the plan?Review of the strategy, the plan and the annual reports; comparison of targets and results
Pillar 2What results, for what funds? What helped or hindered?Three case studies; analysis of the audited accounts, by year, donor and spending category
Pillar 3What changed? Do the gains depend on funding?CERC surveys (2017, 2017–2018, 2021); SHINE evaluation (2018); donor sources; website archives

1.3Sources, data management and ethical safeguards

The evaluation rests on the documents in table 3, on the audited accounts of each financial year and on our “Financial accountability” page. Amounts are those of the audited accounts, in US dollars. No personal data was collected; students are never named. Data follow quality principles inspired by Integrity Action’s PICTURE principles: precise, inclusive, credible, triangulated, usable, results-focused and ethically collected.

Table 3: main sources of the evaluation

DocumentDate
Stratégie 2017–202115 juillet 2017
Plan opérationnel 2017–2021septembre 2017
Rapports annuels 2018, 2019 et 2020–20212019–2022
Group BEAG audit reports2018–2022
SHINE project report2017–2020
SHINE internal evaluationOctober 2018
Survey on perceptions of corruption in South Kivu2017
Learning report on Integrity Clubs2021
Civil society parallel report on the UNCAC2021
CERC website articles2017–2021

1.4Conflict sensitivity and “do no harm”

The evaluation publishes no information that could expose a student, a teacher or a monitor to reprisals. The problems reported are presented without naming the schools or the people concerned.

1.5Limitations and risks

Table 4: risks and limitations of the evaluation

Risk or limitationPotential impact on the evaluationReal impact and mitigation
Internal evaluationA less critical view of our own work.Findings rest on verifiable figures – audited accounts, written targets – and the report also publishes what did not work.
No new interviewsThe voice of students and partners is missing.We use the existing surveys (2017, 2018, 2021); an external evaluation is recommended for the next strategy.
Missing dataSeveral targets cannot be assessed.They are classed as “not measured”; measuring them is part of the recommendations.
Diverging figuresGaps from one document to another (54 or 55 clubs in 2019).We use the best-documented figure – the project report, the audited accounts.
Insecurity and Covid-19No field visits.A document review is sufficient for a strategy evaluation; no personal data was collected.
5

2.1 Pillar 1 – The strategy and its operational plan

A relevant strategy, well connected to its partners, but a plan whose targets were far from resources and monitoring that did not speak the plan’s language.

2.1.1Relevance: a sound diagnosis

Our May 2017 survey of 980 households, in the nine entities of South Kivu, confirmed the diagnosis: 56.9% of respondents consider corruption serious, but only 20.9% know which institution to report it to. Training young people to monitor their services and report problems answered that gap. So did the attention to girls: they made up 51% of first-year monitors.

Two limits: the health component was not funded – only a Covid-19 vaccine education project (US$6,500, 2020) touched on health –, and Walungu did not host a programme. From 2020, working in Kinshasa, where national decisions are made, was an expansion the strategy allowed.

Rating: satisfactory. Sound in its diagnosis and audience; too ambitious in its targets.

2.1.2Coherence: strong partners, monitoring to unify

Integrity Action funded and supported SHINE with its approach and the DevelopmentCheck app; South Kivu’s education authorities worked with CERC from 2018; the UNCAC Coalition (2020), Catalyst 2030 (2020) and ECOSOC (July 2021) connected our work to the UNCAC and SDG 16; the Ministry of Justice invited CERC as an expert to the national anti-corruption conference in October 2021.

On the other hand, our annual reports from 2018 to 2021 report on four objectives – diagnose, build coalitions, engage citizens and young people, build the organisation – that are neither the strategy’s objective nor the plan’s areas. No report tracks the plan’s targets: this evaluation brings them together for the first time.

Rating: moderately satisfactory. Good external coherence; internal monitoring to unify.

2.1.3The plan’s targets and the results

Table 5 compares the operational plan’s forecasts, totalled over 2017–2021, with the results achieved.

Table 5: plan forecasts and results, 2017–2021

TargetPlanned (total 2017–2021)ResultStatus
Programme
National trainings with government and civil society2 trainings, 20 participantsBukavu exchange workshop (2018): 13 people from 6 organisations and 2 ministry representatives; 4 consultation workshops (2021) with 54 senior education officials and 19 expertsMet
Integrity Clubs set up28554, in Uvira and Bukavu (19%)Not met
Students in the clubs, 50% of them girls4,350810 trained (19%); 51% girls among first-year monitorsNot met
Schools and projects monitored, 50% of them improved57041 services and projects in year one; 55 schools monitored with DevelopmentCheck; 32% of problems fixedNot met
Learning and impact
Annual impact and learning surveys43: households (2017), SHINE baseline and endline (2017–2018), 534 young people consulted (2021)Partly met
Independent surveys and evaluations1 semi-independent, 4 independent1 internal evaluation of SHINE (2018); no independent evaluationPartly met
External learning events5Bukavu (2018), Integrity Action annual workshop (2019), 4 consultation workshops (2021), national anti-corruption conference (2021)Met
Fundraising
BudgetUS$575,000 over the periodUS$251,077 received (44%)Not met
Unrestricted funding30% in 202113.5% in 2021; 0% beforeNot met
Large institutional donors1 to 2 a year1 from 2017 to 2019; 3 in 2020 and in 2021Met
Individual donations15% of budget in 2021No donation recordedNot met
Management and finance
Annual audits44, by Group BEAG (2017–2018, 2018–2019, 2020, 2021)Met
Strategy reviewAnnual reviews; next strategyRevised in 2021 with staff and board; 2022–2026 strategyMet
Board of directors7 active members, 50% womenMeets every quarter; gender composition not publishedPartly met
Risk management100% of teams, quarterly reviewOffices closed in 2020–2021 without a published continuity planPartly met
Staff
Women among staff and volunteers50%Not trackedNot measured
Staff fellowships and online courses12Staff training in 2021 (research, public integrity, gender, UNCAC); number not trackedPartly met
Congo Integrity Leadership Fellowship3 universities, 10 fellowsNot launchedNot met
Communication
Social media followers25,000 in 20215,000 new Facebook likes (2018); no annual trackingNot measured
Website visitors170,000Not trackedNot measured
People reached by campaigns8.5 millionMore than 5 million media impressions (2018), which are not peopleNot measured
Posts on our website5014 articles (28%)Not met
Conferences, forums and events14At least 4 documented, including the Integrity Action workshop (2019) and the national conference (2021)Partly met
Media mentions25, 11 of them internationalMore than 16 articles on Integrity Clubs (2018); no tracking afterwardsPartly met

Overall: 5 of 24 targets met, 7 partly met, 8 not met and 4 not measured. Amounts from the audited accounts.

6

2.2 Pillar 2 – Programme effectiveness and efficiency

What was done was done well, at a reasonable cost; but on a much smaller scale than planned.

2.2.1Summary of findings

Three case studies, chosen to cover the period, both provinces and the main donors (annex 2), illustrate what our programmes achieved. Together they received US$251,077 from 2017 to 2021; US$237,823 was spent or returned. Of US$231,399 spent, 37% went to programme activities, 33% to staff, 22% to rent, running costs and equipment – including US$24,000 of donated Microsoft licences –, 6% to administration and 1.5% to fees, training and audit.

Table 6: funds received and spent, by year

YearReceived (US$)Spent (US$)Donors
2017–201857,888.9256,272.27Integrity Action (SHINE)
2018–201944,425.4044,786.65Integrity Action (SHINE)
202079,458.0570,526.36Integrity Action, INL / Accountability Lab, Twilio, Microsoft (in kind)
202169,304.9666,237.48USAID / IREX, GPE / Oxfam IBIS, UNCAC Coalition, own resources
Total251,077.33237,822.76

2021 spending: US$59,813.25 and US$6,424.23 returned to the donor of Building Integrity in the DRC.

2.2.2SHINE: the first Integrity Clubs, in Uvira and Bukavu

Funded by NORAD and Sida through Integrity Action from October 2017 to 2020, Students Acting for Honesty, Integrity and Equality created CERC’s first Integrity Clubs: 20 in Uvira in year one, 54 in Uvira and Bukavu in 2019. Students were trained to monitor their schools and their communities’ projects with the DevelopmentCheck app, then to talk with their head teachers and the authorities.

From 2017 to 2020, 810 students were trained and 32% of reported problems were fixed. In 2018, 645 people were involved – 525 students, 35 head teachers and 70 teachers –, including 277 girls and 47 people living with a disability, and 1,780 students study in safer conditions thanks to the clubs’ monitoring. In 2019, 202 teachers and head teachers from 35 schools were trained in Uvira and Luvungi, a club got toilets and classrooms built, and 100 club members handed a memorandum to the mayor of Uvira.

SHINE received US$132,621, about US$163 per student trained, including teacher training and school monitoring; every period was audited. The October 2018 internal evaluation found the project relevant, effective and efficient, but too small.

2.2.32020: small projects in the middle of a pandemic

In 2020, CERC ran Building Integrity in the DRC (US Department of State, through Accountability Lab) and a Covid-19 vaccine education project (Twilio, US$6,500), and received US$24,000 of Microsoft licences. The pandemic slowed activities: the first project spent only US$12,228 of the US$18,652 received, and CERC returned the remaining US$6,424 to the donor in 2021 – a sign of accountability, but also of limited delivery capacity.

2.2.42021: preparing Kinshasa

Youth Excel, funded by USAID through IREX (US$32,507 in 2021), was launched on 19 July 2021 in Kinshasa to support 10 youth-led initiatives. The year zero of Students Acting (Global Partnership for Education, US$23,986) brought together 534 young people in 39 workshops and 89 teachers and head teachers from 24 schools, ran baseline surveys in 41 schools and consulted 54 senior education officials and 19 experts. Funds arrived in mid-June 2021, three months after the planned start; the work plan was adjusted. CERC also wrote the civil society parallel report on the DRC’s implementation of the UNCAC (US$3,432), with about 100 recommendations.

Rating: moderately unsatisfactory (effectiveness); satisfactory (efficiency). Real results at a reasonable cost, but most scale targets were not met; a heavy structure for small budgets.

7

2.3 Pillar 3 – Value, impact and sustainability

Real impact on students and schools, a recognised voice; an organisation that still depends on a few grants.

2.3.1Impact: students who know how to act, schools repaired

According to the SHINE evaluation, 93% of club members knew how to monitor a project or a public service at the end of the year, against 55% at the start; only 11% would give a teacher a gift to improve their mark, against 64%. The first-year Fix Score reached 43%, for a target of 15%. Toilets and classrooms were built, and 1,780 students in Uvira study in safer conditions.

In four years, CERC became a partner of the education authorities, an expert invited to the national anti-corruption conference, a member of the UNCAC Coalition and an NGO in special consultative status with ECOSOC. The Global Partnership for Education chose our Integrity Club model for a project from 2022 to 2024. These results, however, come from our own surveys: no independent impact evaluation was carried out.

Rating: moderately satisfactory. Real, promising impact, still little measured independently.

2.3.2Sustainability: a fragile financial model

Integrity Action provided 53% of the period’s funds, and was the only donor until 2019. No unrestricted funding and no individual donations were received before 2021, and CERC could not build a reserve: the Covid-19 crisis forced it to close its Uvira and Bukavu offices. The facilitators appointed by the schools and the manuals remain in the schools, but after SHINE no funding supported the clubs in 2021.

CERC enters its second strategy with new funding – the Global Partnership for Education, through Oxfam IBIS, and USAID, through IREX –, but still concentrated on two donors.

Rating: moderately unsatisfactory. The gains are real; the financial model remains fragile.

8

2.4 Synthesis: assessment against the DAC criteria

Table 7: assessment against the six OECD-DAC criteria

CriterionRatingMain findings
RelevanceSatisfactoryReal needs, a sound bet on young people; a health component almost absent and targets unrelated to resources.
CoherenceModerately satisfactoryStrong partnerships; annual reports that do not track the plan.
EffectivenessModerately unsatisfactory5 of 24 targets met, 7 partly met, 8 not met and 4 not measured; 44% of the planned funds, five times fewer clubs and students than planned.
EfficiencySatisfactoryEvery year audited; US$163 per student; unspent funds returned; heavy fixed costs.
ImpactModerately satisfactoryStudents who know how to act, schools repaired, a recognised voice; few independent measurements.
SustainabilityModerately unsatisfactory53% of funds from one donor, no reserve, offices closed in 2020–2021.

Scale: highly satisfactory, satisfactory, moderately satisfactory, moderately unsatisfactory, unsatisfactory (annex 7).

9

3. Recommendations, lessons and suggestions

The recommendations address the implementation of the 2022–2026 strategy. They do not say that the 2017–2021 strategy failed: like the whole evaluation, they aim to prove and improve. Each is classed by category – start, stop, continue, grow, consider – and by criticality.

Table 8: recommendations, lessons and suggestions

№CategoryRecommendation, lesson or suggestionCriticalityPillar
1StartSet targets in two scenarios – funded and to be funded – and review them every year against the funding secured.HighPillar 1
2StopStop annual reports organised around objectives that differ from the plan: report every year on the plan’s targets, in a single results framework.HighPillar 1
3ConsiderStart health with a pilot project with a few CODESA, or take it out of the targets.MediumPillar 1
4GrowAim for at least three donors a year, none above half of the funds.HighPillar 3
5StartBuild a reserve of at least three months of running costs, from a share of each grant and from donations.HighPillar 3
6GrowEmbed the clubs in the rules of schools and education authorities, so that they live without funding.HighPillar 3
7StartPlan an external evaluation of the project funded by the Global Partnership for Education.MediumPillar 2
8StartMeasure followers, visitors and people reached every year, from a baseline.LowPillar 2
9ContinueContinue to audit every financial year and to return unspent funds.SuggestionPillar 2
10ContinueContinue the Integrity Club model with the facilitators appointed by the schools.SuggestionPillar 2
10

Annexes

Annex 1Evaluation requirements

  • Assess the 2017–2021 strategy and its operational plan, from September 2017 to December 2021, against the six OECD-DAC criteria.
  • Compare each target of the plan with the results achieved, and the funds planned with the funds received and audited.
  • Draw lessons and make recommendations for the 2022–2026 strategy.
  • Publish the report online and in PDF, in French and English.

Annex 2The choice of case studies

ProjectPeriodDonorFundsWhy this choice
SHINE2017–2020Integrity Action (NORAD, Sida)US$132,621CERC’s first programme and most of its funds; the Integrity Club model
Building Integrity in the DRC; Covid-19 vaccine education2020INL / Accountability Lab; TwilioUS$25,152Small projects during the pandemic; funds returned
Youth Excel; Students Acting year zero2021USAID / IREX; GPE / Oxfam IBISUS$56,493The opening of Kinshasa and the preparation of the next strategy

Annex 3The team and its roles

RoleResponsibilities
Evaluation leadThe monitoring, evaluation and learning team: questions, document review, drafting
Financial analysisThe finance team: reconciliation of the audited accounts, by year and donor
ValidationThe executive management: review of findings and recommendations

Annex 4The work plan

StagePeriod
Document reviewJanuary 2022
Financial analysis, after the audit of the 2021 accounts (28 January 2022)February 2022
Drafting and validationFebruary – early March 2022
Publication7 March 2022

Annex 5Documents reviewed

Strategy 2017–2021 and operational plan; annual reports 2018, 2019 and 2020–2021; Group BEAG audit reports (2017–2018, 2018–2019, 2020, 2021); SHINE project report and internal evaluation; household survey on perceptions of corruption in South Kivu (2017); learning report on Integrity Clubs (2021); civil society parallel report on the UNCAC (2021); CERC website articles, 2017–2021.

Annex 6Funds received by donor, 2017–2021

DonorUS$Share
Integrity Action (NORAD, Sida)132,620.6252.8%
USAID, through IREX32,506.7112.9%
Microsoft (in kind)24,000.009.6%
Global Partnership for Education, through Oxfam IBIS23,986.009.6%
US Department of State (INL), through Accountability Lab18,651.757.4%
CERC own resources9,380.753.7%
Twilio6,500.002.6%
UNCAC Coalition3,431.501.4%
Total251,077.33100 %

Annex 7The rating scales

  • Highly satisfactory: expectations exceeded.
  • Satisfactory: expectations mostly met.
  • Moderately satisfactory: real results, with notable gaps.
  • Moderately unsatisfactory: major gaps outweigh the results.
  • Unsatisfactory: expectations not met.

Each target of the plan is classed as met, partly met (real results, but below the target), not met or not measured (no data).