October 2017 – September 2018: 300 students trained to monitor their schools in 20 Integrity Clubs in Uvira. A relevant, effective and efficient project – but still too small.
Translated from the original French edition.
A relevant, effective and efficient project, managed in a professional and transparent way – but still on too small a scale.
monitors aged 15 to 19 trained, 152 of them girls, in 20 Integrity Clubs
young people in other schools reached by club members
services and infrastructure projects monitored: 20 services, 21 projects
impact measured by the Fix Score among young people, against a 15% target
In October 2018, CERC evaluated the first year of its first project: 20 Integrity Clubs in secondary schools in Uvira territory.
Students Acting for Honesty, Integrity and Equality (SHINE) ran from 1 October 2017 to 30 September 2018 in 20 secondary schools in Uvira territory, South Kivu, with a budget of £50,000, funded by NORAD and Sida through Integrity Action.
The internal evaluation assessed the project against World Bank evaluation criteria, adapted to CERC’s own. It draws on the project documents, a field mission from 15 September to 4 October 2018, and a survey of club members at the start and end of the year.
“Project management was professional and transparent.”
The results of the evaluation, criterion by criterion.
| Criterion | Evaluation question | Rating | Main findings |
|---|---|---|---|
| Relevance | Did the project’s objectives meet the needs of building integrity in its context? | Satisfactory | An acute need for integrity in education, confirmed by students and head teachers; low critical mass (20 schools) and clubs that rely on volunteering. |
| Effectiveness | To what extent were the objectives achieved? | Satisfactory | Activities and results achieved; quality training with detailed manuals; project monitoring and dialogue carried out as planned and relayed on radio. Results framed in very general terms. |
| Efficiency | Were the results worth the money spent? | Satisfactory | Budget kept, minimal reallocations, full reports to Integrity Action; US$184 per person trained, US$1,609 per service or project monitored. |
| Impact | Did the project change attitudes and behaviour? | Moderately satisfactory | A clear impact on students’ knowledge and attitudes; a multiplier effect on more than 1,600 students in other schools. Data still partial. |
| Sustainability | Will the effects last after funding ends? | Moderately satisfactory | Trained facilitators, teaching manuals and clubs with elected representatives; no explicit exit strategy yet. |
Ratings taken from the conclusions of the internal evaluation report of October 2018.
SHINE responded to the lack of transparency and poor service delivery in the DRC, which the project put down first to the lack of a culture of integrity.
The DRC’s vast natural wealth does not translate into better development for its citizens. Decades of war have devastated school infrastructure. Free education is a constitutional right, but the education system is largely funded by parents; most schools fall short of minimum standards: overcrowded classes without equipment, sanitation, running water or teaching materials, and underpaid staff.
Because young people have little power or influence in education, CERC chose to build strong links between service users and providers – the State, faith-based bodies, head teachers and teachers – and to make gender a tool for fair and inclusive participation.
Each result had a priority action: Integrity Clubs and training facilitators in community integrity building; community monitoring of schools’ education services and of ongoing projects in communities; and dialogue with providers, local authorities and implementers, through joint discussion groups, radio debates and advocacy campaigns.
Year one indicators, from target to result achieved.
| Indicator | Target | Result achieved |
|---|---|---|
| Secondary school monitors (aged 15 to 19) | 300 | 300, of whom 152 girls and 148 boys |
| Young people still active in the clubs at the end of the project | 300 | 277 |
| Girls among club members | 50% | 52% |
| Girls among club leaders | 30% (40% in the project document) | 48% |
| Members from groups at risk of exclusion | 15% | 4.6% (14 young people) |
| Integrity Clubs set up and running | 20 | 20, of which 19 active at the end |
| Young people reached by the clubs | 2,000 | 1,623, of whom 61 from groups at risk of exclusion |
| Impact measured by the Fix Score among young people reached | 15% | 43% |
The only target missed is the share of members from groups at risk of exclusion: 4.6% against a 15% target.
To understand what makes a project successful, to inform CERC’s future strategies in education – and to check that the expected results were achieved.
Under its terms of reference, the evaluation was to “carry out an in-depth analysis of the project’s implementation in order to better understand what makes a successful project” and help CERC set its future strategies for promoting transparency and integrity in education in the DRC.
The evaluation was led by Douce Kanyeba Songolo, Monitoring, Evaluation and Learning Officer, and Heri Bitamala, Programme Officer, who wrote the report.
The evaluation questions follow World Bank criteria, adapted to CERC’s: relevance, effectiveness, efficiency, impact, sustainability and CERC’s added value (annex 1).
The project was unquestionably relevant: it met an acute need to build integrity and fight corruption in education.
The project’s diagnosis was sound: many young people struggle to access education, and government, providers and implementing agencies are not truly accountable to citizens. Club members and head teachers interviewed confirmed “the poor state of school infrastructure, mismanagement, young people’s exclusion from decision-making and the lack of accountability”.
The project reached only 20 of the 1,478 schools identified in South Kivu, and of the 303 secondary schools in Uvira territory: on its own, it could not build a culture of integrity across its area.
The clubs depend on their members’ voluntary commitment, while few Congolese can give much time to unpaid activities – a limit eased by the commitment of head teachers and teachers.
All planned activities and results were achieved – although the results were framed in very general terms.
In schools: class sizes, administrative and financial management, quality of supervision, materials and infrastructure, teacher performance, quality of the curriculum and relations with parents. In communities: water supply, electricity, and school construction and rehabilitation projects.
Most results were achieved within the budget agreed with Integrity Action – itself an appropriate budget.
| Budget line | US$ | Share of budget |
|---|---|---|
| Meetings and training sessions (including training 5 staff in Tanzania) | 12,496.50 | 19% |
| Salaries of 7 staff, 12 months | 20,325 | 31% |
| Reimbursements to Integrity Clubs for community monitoring and advocacy | 9,451 | 15% |
| Rent, premises and office equipment | 5,810 | 10% |
| Printing of student and teacher manuals, publications | 1,500 | 2.3% |
Setting up the Integrity Clubs took only modest sums. The salaries of seven staff – a programme director, two coordinators, a monitoring and evaluation officer, a communications officer, a finance officer and an advocacy officer – stayed below the time they actually gave the project. Reallocations and overspends were minimal, and reports to Integrity Action were full and detailed.
Efficiency was strengthened by a competent and committed team, trained by Integrity Action in Dar es Salaam, Tanzania, in November 2017, and by exchanges with other partners in the Integrity Clubs network.
Students who know how to monitor their schools, report problems and refuse corruption: the baseline and endline surveys show a clear change.
In a qualitative project, changes cannot be attributed to the project with certainty, but their signs can be traced. Share of club members who “strongly agree” or “agree”:
| Statement | Start | End | Change (points) |
|---|---|---|---|
| Q1. I know how to monitor a public project or service | 55% | 93% | +38 |
| Q2. I know how to make a recommendation to improve a project or service | 42% | 66% | +24 |
| Q3. I know how to use DevelopmentCheck to monitor construction projects in my community | 20% | 41% | +21 |
| Q4. I know where to report a problem I find in a public service | 44% | 76% | +32 |
| Q5. I know how to organise a group of citizens to monitor a project or service | 42% | 68% | +26 |
| Q7. Job opportunities should be the same for women and men | 55% | 90% | +35 |
| Q8. People like me can make a difference in fighting corruption | 59% | 94% | +35 |
| Q9. I would happily give my teacher a gift to improve my grade | 64% | 11% | −53 |
| Q10. An honest person has less chance of success than one willing to be corrupt at times | 35% | 39% | +4 |
| Q11. I know how to work with others to solve problems in a project or service | 44% | 93% | +49 |
| Question | Start | End | Change (points) |
|---|---|---|---|
| Q12. A contractor uses poor-quality materials: “I report my suspicion to the authorities” | 41% | 93% | +52 |
| Q13. Integrity, “the qualities that make a person reliable, honest and trustworthy” | 51% | 94% | +43 |
| Q15. Democracy, “power held or controlled by the people” | 68% | 88% | +20 |
| Q16. Gender equality, “equal opportunities for women and men in all areas” | 30% | 61% | +31 |
Trained teachers and students played their roles as facilitators and monitors better, taking communities’ concerns to providers and implementers. The knowledge shared with more than 1,600 students in other schools points to a multiplier effect.
The project built in elements of sustainability; an explicit exit strategy would strengthen them.
A project that achieved its results thanks to the quality of its trainers and the motivation of the CERC team.
Activities and results were largely achieved, with the support of many of the head teachers and teachers targeted.
CERC managed the project effectively, combining collegiality – consulting the whole team – with good governance.
Relevant topics and high-level trainers; repeating the training, set aside to save money, would have better embedded what was learned.
The attitudes of head teachers, teachers and students changed; the students interviewed firmly resist corruption in their schools.
The work of community monitors must be complemented by civil society and the State, especially for advocacy and accountability to the population.
The evaluation’s recommendations to CERC – and what CERC did with them.
| Finding | Recommendation | Follow-up | Criterion |
|---|---|---|---|
| The project only began an agenda for reducing corruption at local level | Develop a project that builds on SHINE’s achievements | SHINE continued until 2020: 54 Integrity Clubs in Uvira and Bukavu, 810 students trained. | Relevance, sustainability |
| Many civil society initiatives for transparency | Strengthen collaboration and synergies with other NGOs, including for training | Partnerships with Accountability Lab, the UNCAC Coalition, Oxfam, IREX and RCN Justice & Démocratie. | Coherence, efficiency |
| Low critical mass: 20 schools | Extend the clubs beyond Uvira | 204 Integrity Clubs in South Kivu and Kinshasa by 2022. | Relevance, impact |
| No explicit exit strategy | Plan an exit strategy in partner schools | The education authorities agreed to extend the clubs across South Kivu province. | Sustainability |
| Training not repeated | Repeat training throughout the project | Facilitators trained on the anti-corruption legal framework in 2019. | Effectiveness |
| 4.6% of members from groups at risk of exclusion, against 15% targeted | Reach young people at risk of exclusion better | CERC’s gender equality and social inclusion strategy, 2021. | Relevance |
The first two recommendations are the report’s; the others follow from its findings. The Follow-up column is taken from CERC’s reports published since.
The evaluation questions, the survey’s mixed results and the documents reviewed.
Relevance: were the objectives consistent with the needs and priorities of building integrity? are they still valid? does the project meet stakeholders’ expectations? would another strategy have worked better? were risks identified and addressed?
Effectiveness: to what extent were the objectives achieved? what were the outputs? why were any results missed?
Impact: did the project affect the problem it set out to solve, have tangible effects on beneficiaries, bring planned or unplanned change? can it have a catalytic effect?
Efficiency: was the clubs’ budget reasonable? were results worth the spending? were partnership requirements met?
Sustainability: will processes extend the impact after funding? do stakeholders want to continue?
CERC’s added value: did CERC achieve results the clubs could not have achieved alone?
Q6. “I know how to collect evidence on a serious problem”: 73% agree at the end of the year; the published baseline is inconsistent (it totals more than 100%).
Q14. Social audit: the answer “all of the above” rises from 12% to 47%.
Q17. The legal provisions on corruption: scattered answers with no clear progress.
Q18. An example of corruption: “reporting a teacher who asks for a gift” chosen by 58% at the end, up from 50%, and “cheating in an exam” by 13%, up from 2% – a misunderstood question.
Integrity Clubs user guide (Integrity Action); results framework; project document; monitoring and evaluation tool; mid-term report; final narrative report; project budget; final financial report.
CERC thanks Douce Kanyeba Songolo for organising the evaluation, project coordinators Evariste Loketo and Bernardin Bitundwa Ruhindisha, and everyone we met, especially the facilitators and members of the Integrity Clubs.
Report what you see in your school or health centre. Your feedback helps make education and health services in the DRC better and more transparent.
