Evaluation of Multi-Country WASH Project (Bhutan, Nepal, Lao PDR), Extension Phase

Introduction

SNV will undertake a final evaluation with three project teams and partners of the 2-year extension phase multi-country project 2023-2024 as a summative assessment of project achievements, learning and contribution to outcomes. These terms of reference present the objective, the methodology, the deliverables and responsibilities of this process.   As a final evaluation it will address both project impact and effectiveness to understand the difference the multi-country project has made and how well it has contributed to outcomes in each setting. It is also important that the evaluation consider the sustainability of outcomes - including in relation to its previous phase, along with equity considerations.

For SNV, the purpose of the evaluation is for accountability as well as program improvement and knowledge generation.  As the project extension focused on developing program approaches to strengthening climate resilience in WASH programming, the learning from this evaluation will inform wider approaches in future climate resilient programming in SNVs Water sector and beyond.

To enable this, SNV are seeking applications for a Lead Evaluator, or team that draws on upon relevant WASH, Gender Equality, Disability and Social Inclusion (GEDSI) and climate technical advice with combined demonstrated experience in undertaking WASH programme evaluations, tailored to the projects focus on capacity building, system strengthening and transformative approaches.

Objectives

The objectives of the evaluation are to:

1. Evaluate the impact, effectiveness, equity and sustainability of the three projects’ outcomes
2. Support learning and improvement,
3. Document and share knowledge from implementation[1].

Methodology

This extension phase evaluation is intended to be conducted as a hybrid evaluation[2] which draws on a mix of both internal and external resources that engages project teams, partners and external evaluators working together, through defined roles, sense-making processes and facilitated reflection workshops. 

The evaluation is expected to use suitable qualitative research methods, including review of project documentation and literature. In consultation with SNV, the consultants will develop a detailed evaluation plan[3] that further refines the evaluation questions, data collection methods and the division of roles and tasks between the evaluators, SNV and partners.

The methodology proposed will need to be in line with the project’s principles, partnership approaches and m&e plan, ensuring they are fit-for purpose and contextually appropriate and meet the requirements of the Water for Women Fund Guidance.  The evaluator/team will design the approach to respond to the evaluation questions and then work with SNVs in-country teams (Bhutan, Nepal, Lao PDR) to undertake further data collection as necessary and to facilitate joint sense making. The external evaluator/team will then provide the final review and assessment of findings to meet the evaluation objectives. It is envisioned that support is provided as a combination of in-country and remote/online to maximise efficiencies, but that each project will be visited.

The evaluation is expected to draw on existing performance monitoring data which has been collected in line with SNVs Performance Monitoring Guidelines[4] by SNV teams end 2022 and October 24 and is not intended to duplicate survey processes. This includes existing household and institution level surveys (baseline, endline), annual capacity assessments with government line agencies and SMEs, FGDs with key groups (eg people with disabilities, women, households living in poverty), WASH GEM analysis[5] as well as project deliverables and knowledge and learning products.

Evaluations should address quality of evidence factors such as triangulation (using multiple data sources rather than relying on one data source such as interviews or a survey), contribution (establishing a basis for the projects role in causing the changes identified), uniqueness (ruling out alternative explanations that might have caused changes), transparency (being open about how data was collected, analysed, and reported) and independence (explicitly addressing potential biases). Evaluators might find it useful to use a tool such as the QoE Rubric (Aston, 2020), or the Bond Evidence Checklist as part of their evaluation design and review process.

Evaluation questions (from the M&E Plans)

Impact: How has the project implemented in 3 countries contributed to climate resilience and increased equitable access to and use of WASH services? What JMP service levels were achieved?   Where outcomes as expected? Why? Why not? What project strategies were most effective?

Sustainability: To what extent did national or local authorities, institutions or communities demonstrate an increased capacity and commitment to inclusive climate resilient WASH? What evidence is there that systems and services have integrated considerations of climate risks and resilience?

Equity: How has the project implemented in three countries furthered gender equity, disability and social inclusion/or transformation? Is there any evidence of uptake of practice, innovation To what extent did the project reach and meaningfully involve the most marginalised and socially excluded people? What project strategies were most effective?

Contributing to this, the evaluation should consider the four end of programme outcomes, which are aligned to the overall Water for Women Fund Theory of Change.

End of Programme Outcomes are to be evaluated against each projects agreed results. The following are indicative guidance questions from the project m&e plans.

Systems Strengthening: 1 What changes have occurred in the capacity, agency and performance of governments, private sector, community-based organisations to better respond to climate risks and resilience when planning, investing and delivering inclusive WASH services?

Access to WASH Services 2. To what extent have changes occurred in equitable, universal access to and use of climate resilient WASH services, particularly for marginalised communities and community members?

Gender, disability and social inclusion 3. What broader social inclusion changes have occurred in households, community and institutions as a result of gender and socially inclusive WASH approaches and outcomes? How has the project understood the linkages between inclusion, transformation and climate-resilient WASH?

Knowledge and Learning 4: How have the projects commonly understand climate risk and resilience? How can this be further developed to better inform the design of WASH interventions? In what ways have other CSOs, governments, or other organisations (within and beyond Fund participants) taken up and used project generated documentation, evidence and/or demonstrated practice? How has this informed, influenced or changed policy or practice? 

The evaluation will involve 4 stages – design, data collection, synthesis and results sharing. It includes both individual project level processes within country workshops, and the overarching multi-country programme level.  These stages and their roles are described as follows.

The following are the key indicative tasks aligned to the different stages

Design

• Consultation with Multi-Country Project Manager (MPM), Country team leads, partners to determine the overall evaluation approach, roles and responsibilities.
• Online workshop and follow up - At the workshop, and subsequently via email, agree/refine key evaluation questions, methodology, roles and plans
• Refined methodology for evaluation and supporting tools developed to guide in-country data collection supported by evaluation plans (3)
• Quality assurance review - Feedback on methodology, tools and evaluation plans with input from the WfW Fund Coordinator M&E Advisor.

Project level data collection

• Local teams/partners orientated to conduct/support data collection and collate existing data to respond to the evaluation methodology, questions and aligned to evaluation plans by the Lead Evaluator.
• SNV Project teams compile all available data, including from the performance monitoring which includes surveys, FGDs, capacity assessments and KIIs to provide evidence for evaluation questions.
• Lead external evaluator/team conducts data collection with key stakeholders at multi-country level (eg ISF-UTS, CBM) and WfW
• Quality-assurance check-in.
• Local teams/partners/evaluator conduct any additional data collection with key stakeholders in Nepal, Bhutan and Lao PDR with oversight by lead evaluator and logistically supported by SNV country offices.

Interpretation

• Data synthesis - The country evaluation teams, together with the evaluation lead/team synthesize and analyze data collected.
• Sense-making workshops (3) – engaging each project at the country level together with key stakeholders, partners, to share and reflect on the findings.
• Combined (remote) multi-country programme level workshop (1) brings all actors together, to present findings, reflect on and agree any overall review findings and recommendations.

Results Sharing

• Presentation of findings – senior management team
• Drafting the (3) country level reports and recommendations for review
• The MPM, partners, peer reviewers provide two rounds of substantive comments.
• The quality assurer writes a separate report on the appropriateness and robustness of the process and findings. This is included in the evaluation reports.
• Communication Outputs - The evaluation lead/team finalizes the report and produces supporting presentation per project (3) using infographics were suitable.
• Final report and supporting outputs to be completed by January 20th 2025 for donor submission.

Follow up

• Follow up actions agreed.

Key Roles aligned to tasks

Evaluation Lead/team (LOE 58 Days) and travel.

Design Phase (8 Days)

• Prepare for assignment (reading of relevant documents such as project document, baseline survey report, annual reports, etc.) 
• Determine overall evaluation approach, with country team leads, MPM, partners.
• Facilitate the design workshop (online 0.5 day)
• Finalise evaluation approach, revised evaluation questions and scope of additional data collection requirements  
• Orientate Bhutan, Lao PDR and Nepal local teams on data collection processes and tools.

Project data collection (5 Days Remote + 5 Days in country for each country– 20 days in total) plus travel   

• Provide advice, oversight and backup support on demand to in country teams
• Collect data at multi-country level (eg ISF-UTS)    
• Collect data in country with support from local translators/team.

Interpretation (15 Days)    

• Lead data synthesis and data analysis (3)
• Lead preparation of 3 country level sense-making workshops  
• Facilitate 3 x 1 Day sense-making workshops – in country 
• Lead preparation for programme level workshop (0.5 day online)
• Facilitate combined programme level workshop (0.5)

Results Sharing (15 Days)  

• Lead the drafting of the 3 country evaluation report and recommendations
• Finalise the 3 country evaluation reports  
• Produce supporting summary presentation per project
• Overall project management and consultations with country teams and MPM  
• Final presentation to management team (2 hours)

Tasks for SNV Country Teams– 10 days per country project (Nepal, Bhutan, Lao PDR)

Support from SNV Country Teams is planned indicatively as 10 days per country project. This includes time for preparing for the assignment, participating in the design work and providing input into the development of the evaluation plan, methodology and tools and final reports. The teams will also provide support in completing agreed data collection in country in coordination and with oversight from evaluation lead, supporting data synthesis and analysis in line with the evaluation plans.  Active roles are foreseen in collaboratively identifying key findings and participating in country level sense making workshops.

Key Deliverables

The main deliverables of the Evaluation are:

• Design Workshop (1)
• Evaluation Methodology (1), supporting data collection tools  and three country level evaluation plans
• Sense making workshops (3 minimum) and combined Programme Level Workshop (1)
• Draft Evaluation Reports (3)  
• Summary presentations (3)
• Final Review reports (3) including follow up actions.

SNV teams will provide logistical support throughout the process and facilitate meetings with key government partners, stakeholders and partners as required.

Further resources

To help inform and guide evaluation methods and approaches there are many resources available including the recent IDS M&E Framework for rural sanitation and hygiene and the DFAT M&E standards.  Additional guidance is available from the Fund on disability measurement, knowledge updated as well as  ISF-UTSs on qualitative approaches for understanding GESI changes. For technical WASH monitoring the Monitoring and Evaluation for Rural Sanitation and Hygiene: Framework (IDS, 2021) may be of interest. Projects should refer to the Gender Equality in Monitoring and Evaluation – Good Practice Note (DFAT, 2020) for integrating GEDSI into their evaluations. Projects should also refer to Monitoring and evaluation for climate resilient WASH (GWP/UNICEF, 2017) for guidance on M&E for climate resilient WASH.

Timeline and Duration

The assignment is estimated to cover 58 days for an Evaluation Lead or team approach, plus additional days for international travel.

The entire assignment is anticipated to commence at the start of October with the final deliverables to be submitted to SNV on or before 20 January 2025.

Payment Terms and Conditions

The payment schedule shall be as follows:

-               25% upon signing of contract

-               45% upon completion of the interpretation phase

-               30% upon submission of final approved deliverables.

SNV are seeking applications for a lead evaluator or team with demonstrated experience in undertaking WASH programme evaluations, with a focus on capacity building, system strengthening and transformative approaches.  The lead evaluator/team must have availability throughout the duration of the project and bring excellent evaluation, coordination and communication skills and a demonstrated ability to work across culturally diverse settings and teams. Experience in facilitating remote sessions and processes preferred.

[1] Water for Women Fund: Evaluation Guidance, June 2024

[2] https://www.betterevaluation.org/methods-approaches/methods/hybrid-internal-external-staff

[3] Refer to Standard 8 > https://www.dfat.gov.au/about-us/publications/dfat-design-monitoring-evaluation-learning-standards

[4] These are available online at https://snv.org/assets/explore/download/2019-1-impact-indicators-ssh4a-performance-monitoring-review.pdf and https://snv.org/assets/explore/download/2019-2-outcome-indicators-ssh4a-performance-monitoring-review.pdf

[5] Projects have collected baseline and endline data collection supported by ISF using the WASH Gender Equality Monitoring Tool. More info > https://waterforwomen.uts.edu.au/wash-gem/