Evaluation Model dimension and questions - Nyoman Sutarsa - MIKM 3

Program Evaluation – Theory and Models I Nyoman Sutarsa, MD., MPH., PhD., FHEA [email protected] [email protected]

Learning objective: • To re-visit logic model (e.g. definition, key components) • To explain various program evaluation theories • To describe various program evaluation approaches • Designing evaluation questions

Evaluation and Program Planning dissemination Life Cycle • Problem identification • Summative • Prioritisation of problem • Program development evaluation (outcome • Evaluation plan and dissemination and impact) • Economic evaluation plan Needs assessment, Formative Evaluation, Program Logic •ImpPleromceesnstation evaluation, evaluability assessment



The Basic Logic Model The W.K. Kellog Foundation Inputs (Processes) Outputs Outcomes Impact Activities Program level Population/system level 5

Developing Logic Model • Deductive approach – based on evidence from literature • Inductive approach – based on field study (talking to program developer, program manager, implementers, and program beneficiaries) • Combination – the most common practice!

Logic model and evaluation • Describing program in lay language and specific to facilitate solid and sound evaluation • Guiding tool in formulating evaluation questions, indicators, and performance level • Again, improving engagement and participation of stakeholders in the planning, implementation and utilisation of evaluation data



Evaluation Theory

What is program evaluation? Systematic activities or methods for collecting, analysing and using the information to: • Assessing if the program is implemented as planned, or if the program has achieved its intended objectives • Establishing program mechanisms by linking contexts, program, and outcomes • Identifying any unintended consequences • Assessing sustainability, replicability, and effectiveness of the program

Why learn evaluation theory? Program evaluations are embedded in societal • Facilitate assumptions and dynamics, and highly program logic subject to political forces, • Guide the formulation of and sound program evaluation questions evaluation theories should account for • Guide the most appropriate cultural and socio- evaluation approach political contexts. • Facilitate the design of a sound and rigour evaluation study

Evaluation models: • Goal oriented evaluation model – measuring whether if the intended objectives has been achieved or not • Goal free evaluation model – identification of all effects (positive/negative, intended/unintended) • Formative and summative evaluation model – implementation and effects evaluation • Contingency-based evaluation model – emphasis on generalisation of finding • Responsive evaluation model – process evaluation to measure how well the program is implemented • Holistic evaluation model – evaluating the whole project cycle • CIPP evaluation model – context is important (context, input, process, product) • Realist evaluation model – identify mechanisms C-M-O

Three main evaluations: Implementation (process) evaluation • Assessing program implementation, evaluating if the program was implemented as planned, or if any variations were observed • Describing reasons or explanations for any variations Effects (outcomes-impacts) evaluation • Describing range of the effects of the program, and assessing either efficacy, effectiveness or efficiency • Can include only effects on target population and/or beyond target population Contexts evaluation • Describing all external factors that could influence implementation of the program, as well as external factors that might produce similar outcomes/impacts • Assessing how these factors relate to the produced outcomes and implementation

Process evaluation - example

Process evaluation – implementation failure

Effects evaluation (1) Effects documentation (to what extent were the outcome indicators met) (2) Effects assessment (to what extent is any changes attributed by the program) (3) Effects evaluation (to what extent is the changes is associated with the program or something else)

Dimensions of program evaluation: REEIS: • Reach and relevance • Effectiveness • Efficiency • Impact and sustainability

#1: Reach and Relevance (R): The extent to which the program is suited to the identified needs of the primary target group • Is the project relevant to the health of the population? How important? • Is the project responding to identified needs of the community?

#2: Effectiveness (E): Alignment between the stated goals and objectives of the program and the actual reported results • Is the project achieving its intended goals and objectives? • What is the effectiveness or impact of the program compared to the intended objectives (comparing outcomes/impacts on planning vs actualisation)

#3: Efficiency (E): Measures the outputs in relation to the inputs • Does the program use at least costly resources possible to achieve the desired results? • How the resource used to achieve the desired results? • What is the efficiency or utilisation ratio of the resource used?

#4: Impact (I): The positive or negative changes, or intended and unintended consequences produced by the program • Does the program contribute to reaching higher level of effects for and beyond its target population?

#5: Sustainability (S): Measures whether the positive effects of the program are likely to continue after the support has been withdrawn • Long term ability and operational capacity of the program to continue delivering against its goal • Are the positive effects or impacts sustainable and scalable?

Evaluation questions: criteria • Feasible, ethical, useful, accurate and relevant • Lead to more than one possible answer (inclusive and flexible) • Could produce information needed or wanted by decision makers and stakeholders Designing appropriate and specific evaluation question will guide the design, scope, focus, data collection and analysis of data in program evaluation

Process evaluation questions - examples • How is the program being implemented? • How appropriate are the processes compared to quality standards? • Is the program being implemented correctly? • Are participants being reached as intended? • How satisfied are program clients? For which clients? • What has been done in an innovative way?

Outcome evaluation questions - examples • How well did the program work? • Did the program produced or contribute to the intended outcomes in the short, medium or long term? • For whom and in what ways and under what circumstances did the program work? What are the intended and unintended consequences? • To what extent can changes be attributed to the program? • What were the particular features of the program and contexts that made a difference? • What was the influence of other factors?

Economic evaluation questions - examples • What has been the ratio of costs and benefits? • What is the most cost effective option? • Has the intervention been cost-effective compared to alternative? • Is the program the best use of resources?

Appropriateness questions - examples • To what extent does the program address the identified needs? • How well does the program align with government and agency priorities? • Does the program represent a legitimate role of government?

Effectiveness and efficiency questions - examples Effectiveness: • To what extent is the program achieving the intended outcomes in the short, medium and long term? • To what extent is the program producing positive results and/or meeting each of its objective Efficacy: • Do the outcomes of the program represent value for money? • To what extent is the relationship between inputs and outputs timely, cost-effective, and to expected standards?

Questions? Contact: [email protected] [email protected]