Policy Assessment Toolset
for equity and learning

Participatory Impact Pathways Analysis (PIPA)
In the context of school leadership policy evaluation, the Participatory Impact Pathways Analysis (PIPA) approach can be used to carry out an evaluation of likely programme/project impacts (ex-ante impact assessment), help the people involved in related policy programmes better understand what each other are doing, identify common interests and foster programmatic integration, provide a framework and design for monitoring and evaluation, and provide the impact hypotheses required for impact assessment after the programme has finished (ex-post evaluation).

PIPA is a programme/project planning, monitoring and evaluation approach (Alvarez et al, 2010; Douthwaite et al. 2007). It draws from program theory evaluation (Chen, 2005), social network analysis (Cross & Parker, 2004) and organizational learning (Argyris & Schön, 1974), to understand and foster innovation. It is designed to help the people involved in a project, program or organization make explicit their theories of change, in other words how they see themselves achieving their goals and having impact.

Scenario for a PIPA exercise: A school leadership capacity building programme on drop-out prevention

Implementing PIPA on the leadership projects in schools

The process begins with an Impact Pathways Workshop.

The Impact Pathways Workshop

In the Impact Pathways Workshop representatives from each school leadership project work to develop the inputs required to build their project's Impact Pathways (IP) logic model and network maps. The workshop is facilitated by members of the Impact Evaluation Team.

The workshop participants are project implementers, participating 'next users' (which are the school leaders involved in capacity building activities), 'end users' (or beneficiaries, in our scenario students at risk of dropping out of school and their families), and politically important actors (such as local school authorities).

The PIPA Workshop flow (Alvarez et al, 2010)

The workshop is aimed to help participants to do the following:
Clarify, reach mutual understanding, and communicate their project's intervention logic and its potential for preventing at-risk students in high-poverty urban neighbourhoods from dropping out of school;
Understand other projects working in the same programme, and identify areas for collaboration;
Generate a feeling of common purpose and better programmatic integration (when more than one project is represented in the workshop);
Produce a narrative describing the project's intervention logic and possible future impacts (a form of ex-ante impact assessment);
Produce a framework for subsequent monitoring and evaluation.

Developing a cause-and-effect logic
The workshop begins with participants developing a problem tree which links the problem of high student drop-out in high-poverty urban neighbourhoods that the project is directly addressing with the leadership capacities and other school-based conditions that it wishes to improve. The branches of a problem tree stop when the problems that the project will directly address has been identified. These 'determinant' problems help to define the outputs.

Developing a network perspective
To connect the linear problem tree model with the network perspective, participants (i.e. school leaders, teachers, at-risk students and their families, local authorities, etc.) construct a vision of success in which they imagine what they will do differently after the project.

Next, participants draw a 'now' network map, showing current key relationships between stakeholders, and a 'future' network map, showing how stakeholders should link together to achieve the vision. Participants then devise strategies to bring about the main changes. The influence and attitude of actors are explicitly considered during the drawing of the maps.

Developing the outcomes logic model and a monitoring and evaluation plan
In the final part of the workshop, participants distil and integrate their cause–effect descriptions of project-impact pathways from the problem tree and the network view into an outcomes logic model. This model describes, how stakeholders (that is, school leaders, teachers, at-risk students and their families, local authorities, and project implementers) should act differently if the project is to achieve its vision, i.e. prevent at-risk students in high-poverty urban neighbourhoods from dropping out of school.

From this model there are outlined the main knowledge, attitudes, and skills that school leaders should be able to put in practice to promote effectively drop-out prevention. This, in turn, will inform the design of school leadership capacity building activities that school leaders will engage in.

The outcomes logic model is the foundation for monitoring and evaluation (M&E), because it provides the outcome hypotheses, in the form of predictions, that M&E sets out to test. M&E requires that the predictions made in the outcomes logic model be made SMART (specific, measurable, attributable, realistic, and time-bound), so that project staff and stakeholders can know whether or not predictions are being realised.

After the workshop. Monitoring and evaluation (M&E)
After the workshop, participants complete their M&E plan, ideally with key staff and stakeholders who could not attend. Projects periodically hold reflection-and-adjustment workshops with their key stakeholders. These reflection workshops can be seen as the culmination of one set of experiential learning cycles and the beginning of others. The vision is changed to some extent, based on what has been learned, the outcome hypotheses are revised when necessary, and corresponding changes are made to project activities and strategies. New milestones are set for the next workshop.

Ex-post impact assessment
Ex-post impact assessment, which generally occurs 1-2 or more years after a project has finished, seeks to:
verify the direct benefits of the project, and then
trace how further adoption and use of project outputs contributed to drop-out prevention impacts, such as increasing the school completion and grade progression rates, reduced unjustified absences from school, etc.

Back to top

© 2014 EPNoSL Project