TEACHER TRAINING
CITIZEN SCIENCE
University Level, community & Intern students in the One Earth Education Innovation program complete the ISE4GEMs Inclusive Systemic Evaluation course. Data collection and analysis will incorporate the 'Most Significant Change' MSC evaluation system. The MSC technique is a form of participatory measurement and evaluation.
The following documents provide an outline of the research techniques that research students will master in the Inclusive & participatory measurement and evaluation course and utilize in the Global Experiences program. Students, as part of a research team, will contribute to the design of the REC Living Lab research project and collect data at the Global Experiences.
PRACTICAL MEASUREMENTS
The REC 5 year project will develop K-12 courses and camps that are inclusive, diverse, globally efficaceous, and local and community driven. The K-12 courses & camps goals are:
-
to act as a story and aspirational endeavor, inspiring youth with knowledge and positive experiences of systemic alternatives for ecosocial regeneration through nature and community connection practices
-
to provide stories and experiences relating to the reintegration and protection of ecosocial core values, and cultural practices
-
To develop and teach knowledge, skills and practices for personal, community and global Anthropocentric action, mitigation and resilience.
What practical measurements can be made toward monitoring and evaluating this 5 year program?
-
The ‘Inclusive Systemic Evaluation for Gender Equality, Environments, and Marginalized voices’ (ISE4GEM) is an evaluation system initiated by the UN Women Independent Evaluation Service and designed by Stephens, Reddy, & Lewis (2018) that can be applied to the REC project.
-
The ISE4GEM approach as described by Stephens et al. (2018): draws upon the knowledge created by methodologists from the systems thinking and complexity sciences and builds on best practice for systemic evaluation using critical systems thinking theory and tools to analyze interrelationships, understand multiple perspectives and conduct continuous boundary analysis. (p. 6)
Inclusive Systemic Evaluation for Gender Equality, Environments, and Marginalized voices’ (ISE4GEM)
Figure 1 lays out the four stages of the ISE4GEMs evaluation learning and action cycles.
The evaluation cycle will completed after each regional camp, at the end of each Intern training cycle and at the end of each of the five years of the project.
Each ISE4GEMcycle constitutes a PDSA cycle.
Figure 1 The ISE4GEM’s learning and action phases. Adapted from ‘Evaluation Guidance Series Inclusive Systemic Evaluation for Gender Equality, Environments and Marginalized Voices. ISE4GEMs: A new approach for the SDG era’, by Stephens, A., Lewis, E. D., & Reddy, S. M., 2018, p. 58
Phase 1. Preparation & Design (Fig 2)
This stage consists primarily of defining evaluation boundaries and is important as boundaries are at the heart of systems thinking. “Having a clear picture of what is being evaluated is essential” (Stephens et al., 2018, p. 19). The boundary story for this evaluation will be based on the project vision, mission, and goals
Figure 2 The actual boundary of evaluation. Adapted from ‘Evaluation Guidance Series Inclusive Systemic Evaluation for Gender Equality, Environments and Marginalized Voices. ISE4GEMs: A new approach for the SDG era’ by Stephens, A., Lewis, E. D., & Reddy, S. M., 2018, p. 68.
Phase 2. Data Collection
“The idea of educational evaluation is deceptively simple. It involves the systematic collection and analysis of data needed to make decisions and identify effects of educational initiatives” (Earl & Timperley, 2015, p. 10).
However, as Gamble (2008) says:
Initiatives that are innovative are often in a state of continuous development and adaptation, and they frequently unfold in a changing and unpredictable environment. The destination is often a notion rather than a crisp image, and the path forward may be unclear. (p. 13)
All stakeholders should be involved in the evaluative thinking process, including communities, parents, and students themselves as key participants and decision-makers. When all the groups who have a commitment to and interest in the innovation bring their diverse perspectives and intentions to the evaluation, the evaluation is likely to be more authentic and all stakeholders are more likely to understand, share, and support decisions (Cousins & Earl, 1992).
The methods used for collecting information from stakeholders will include “document analysis; narrative, stories, and vignettes; surveys, focus groups, and interviews, just-in-time responses using digital technologies and social media” (Earl & Timperley, 2015, p. 24). As a part of their data collecting protocol for evaluation, the team will also make and use videos of teacher training practicum student programs of both teaching moments and practices as well as personal evaluation interviews with participants as feedback. For final project evaluation, information will be collected of continued ecosocial education community participation of alumni beyond individual course completion
Most Significant Change Technique (MSC)
Evaluating emergent programs and systems change approaches
Some programs are by nature emergent. By this, we mean it is hard to predict at the start what the
results will be, and there are likely to be unexpected outcomes. When this is your situation, it is good to
include methods that are able to search for impacts and make sense of them. Emergence is a feature of
place-based approaches and systems–change approaches, so these types of approaches are also
relevant here.
MSC technique is a form of participatory measurement and evaluation. It involves stakeholders collecting
stories about significant change directly from families and individuals. It is participatory because many
project stakeholders are involved both in deciding the sorts of change to be recorded and in analysing
the data.
MSC can be used to help understand the impact on individuals in an authentic and powerful manner.
Phase 3. Data Analysis, interpretation, and reporting
The evaluation report will be built considering the facts, values, and a Boundary Analysis as represented as three sides of an analysis triangle shown in Figure 3.
Facts and data collected will be analyzed through gender equality, environmental, and marginalized voices themes. The insights and observations need to be converted into knowledge that is both insightful, useful, and relevant.
Figure 3 Systemic Triangulation. Adapted from Reynolds (2015). Adapted from ‘Evaluation Guidance Series Inclusive Systemic Evaluation for Gender Equality, Environments and Marginalized Voices. ISE4GEMs: A new approach for the SDG era’, by Stephens, A., Lewis, E. D., & Reddy, S. M., 2018, p. 110
The goal of the analysis is to arrive at knowledge, “the kind of knowledge that can be transferred and further developed across contexts (Earl & Timperley, 2015, p. 32). The contexts in the case of each program will be preparation for the following program leading ultimately at the end of the year cycle to building capacity for future years' programs.
Figure 4 Communicating evaluation results by conceptualizing systems change using the forest ecocycle analogy. Adapted from Zimmerman, Lindberg, & Plsek (2001). Adapted From ‘Evaluation Guidance Series Inclusive Systemic Evaluation for Gender Equality, Environments
A forest eco-cycle practice model
Here, aspects of the programs being evaluated can be analyzed as belonging to one of four quadrants: Birth, Maturation, Creative Destruction, and Renewal (Figure 4). This model allows for an iterative series of PDSA evaluation led improvements.
In this model, the biological ecocycle metaphor is shown as being an infinity loop. “The infinity loop depicts a living systems scenario with no beginning or end. The movement from the lower-left Quadrant I to the upper right Quadrant II follows an ‘S’ curve” (Stephens et al., 2018, p. 140). It is on this ‘S curve that a focus of strategic planning to improve the efficiency of programs and interventions leading to mature and improved outcomes would be best applied.
Phase 4. Capacity Development
UNDP defines capacity development as “the process through which individuals,organizations and societies obtain, strengthen and maintain the capabilities to set and achieve their development objectives over time” (UNDP Partnership with Global Fund, 2017).
Capacity development goals for this project involve working as a networked improvement community to build and develop theories of knowledge and theories of change and ascertaining how the theories of knowledge and change can be used to scale up mutual program success at more locations through further lateral capacity building with communities and educational organizations.
