Sometimes the obvious way to aim towards networked evaluation is to start a conversation with a network of interested organizations.

Example: The Alliance for Watershed Education in the Delaware River Watershed

[text to come!]

Example: Colorado Collective Outcomes

[text to come!]

Sometimes a different approach is effective, which is to first recruit organizations with the same goals and outcomes and that may be using or are interested in using the same evaluation tool(s) or approach(es).

Example: Improving EE distance learning

In the midst of the pandemic in 2020, faculty at Virginia Tech and Clemson recruited 43 EE youth-serving organizations, who were all unexpectedly pivoting from in-person to online programming, to create a networked community with a culture of continuous improvement and learning. The organizations had various levels of prior collaboration with each other from none to a significant amount. All shared similar desired outcomes resulting from their programs and agreed to use ​​a consistent outcome measure (a survey that assesses a broad range of EE outcomes: Environmental Education for the 21st Century, EE21) to evaluate programs. This allowed these organizations to draw conclusions about their programs, identify potential areas for improvement, and share ideas about what works best both within and across organizations. 

Learning cycles: The program included two learning cycles. In the first cycle, educators were coached and guided through evaluation and data collection processes. After collecting data on their programs they received an evaluation report that summarized their results. The educators were then asked to reflect on their programs and identify areas for improvement based on the evaluation results. The second cycle was an opportunity to evaluate programs after these adjustments were implemented and to reflect as a community on what was learned about these adjustments to foster collective learning. This model supports educator autonomy and decision-making, and takes advantage of the broader community of peers to develop new insights and innovations.

Other activities: The facilitators used a number of techniques to foster evidence-based learning across the network.

• In network meetings, participants were provided with the

  Referring to ability diversity and a variety of abilities and disabilities. These variations include differences in cognitive, social-emotional, and physical abilities. Differences in abilities may need more individualized approaches to instruction and evaluation (PennState Extension, 2017).

  to choose breakout rooms in order to network with specific organizations/people with commonalities.
• Examples and facilitated discussions were provided about programs that attained high outcomes.
• Specific topics for spring meetings were chosen based on member requests and interests (e.g. attracting participation in new online programs) and facilitated presentations and small group discussions on those topics.
• One-on-one consultations with each organization were added to interpret and reflect on data from spring reports.

Extending to collective evaluation: 

The EE21 team has a number of initiatives that extend their evaluative learning to the field of environmental education. 

1. Results from these networks were widely shared to inform the field about promising practices.
2. Results about the network and what worked and what could be improved will be shared widely so that other organizations and evaluation specialists can replicate or develop similar models.
3. Survey instrument (EE21) is provided here in its entirety. 
4. Examples of evidence-based practices for online environmental education programs, designed by participants in our network, can be found on this website. 

And of course, networked evaluation is never quite as simple as the first two paths indicate; combinations of paths are common! For example, some collaboration might pre-exist across a network, and subsequent use of a common tool simultaneously accomplishes networked evaluation and increased connection and collaborative activity across the network.

Example: Oregon Outdoor Schools

Oregon Outdoor School (ODS) is located in the United States, and is a Statewide outdoor formal educational program, primarily for 5th and 6th grade students. Outdoor School is a hands-on week of experiential science education in the field, with a more than 60-year history. Originally, the reach of the program was not consistent throughout the State, but nonetheless, the originally active sites have a long history of familiarity and collaboration, so the network and sense of community already existed to a large degree. At this time, some networked evaluation existed in the program, including a formal summative evaluation, a

A logic model shows the rationale behind your program. It is a graphic representation of the linkages between program goals, resources (inputs), activities, and expected outcomes. Logic models illustrate the ways in which program inputs and activities are thought to lead to outputs and outcomes in both the short and long term. Logic models often include diagrams or pictures that illustrate these relationships. In program evaluation, logic models provide a basis for developing evaluation strategies. Depending on the scale of your project, you might need more than one logic model. Click here for what a logic model can look like. (Zint, n.d.).

style evaluation, and an overall comprehensive analysis of any evaluation activities over a four-year period. These activities appear to have been primarily conducted to help programs clarify their goals, and seldom to have resulted in evidence-based programming updates.

New funding from the State legislature allowed expansion Statewide in the 2017-2018 academic year, and by the end of the 2018-2019 school year 84% of eligible 5th or 6th grade students had attended. With this expansion, evaluation needed to be established that would use a common measurement system, and that would enable programs to track their outcomes, looking at change over time within their individual programs, among various programs, and overall statewide trends. Evaluation started with a small project strategically funded, adapting and using the EE21 outcome measure (a survey that assesses a broad range of EE outcomes: Environmental Education for the 21st Century, EE21). Upon completion of the pilot study, the evaluative process/system was reconsidered, making efforts to center cultural responsiveness and to increase stakeholder participation.  Methods and thinking were updated to be more culturally responsive, which included modifying our

A standardized method of collecting data that consists of questions and responses. Questions can be "open-ended", but often take the form of being "close-ended" where respondents choose from predetermined answers (Frechtling, 2010).

and administration scripts, focusing our data analysis and engaging in

The guarantee of fair treatment, access, opportunity, and advancement while at the same time striving to identify and eliminate barriers that have prevented the full participation of some groups. The principle of equity acknowledges that there are historically underserved and underrepresented populations and that providing equal or identical treatment to all does not improve the fairness of these unbalanced conditions (Oregon State University, 2021).

,

The unique differences among individuals in a group based on which we may be treated differently in society. Ethnicity is not the only way in which we are diverse as a group. There are countless visible and invisible facets of diversity. This may be race, socioeconomic background, gender (especially in the sciences), physical ability, age and how people come to environmental education at different points of their life, or sexual orientation. It includes any sort of “-ism” that brings people to the conversation, especially at different places of power. Power differentials may invite or limit the conversation. This is sometimes known as a "dimensions of difference" (The Avarna Group, 2020).

and

Authentically bringing traditionally excluded individuals/ groups into processes, activities and/ or decisions in a way that share power (Racial Equity Tools, 2020).

training. The evaluation was implemented statewide at 39 outdoor schools in 2020 (Braun 2020).  

All outdoor school programs receive private and anonymized final reports with data for their specific program in relation to the statewide sample. Subsequently, outdoor school providers are encouraged to attend an evidence-based workshop where they are supported in understanding evaluation results for their specific program, for all ODS programs statewide, and in the context of the field of EE. Each program then translates these learnings into discrete action steps to improve their respective programs, typically related to culturally responsive instruction and pedagogy. 

One result of the evaluation activities was that analysis of findings at the statewide

A set of numbers or symbols used to represent characteristics of a variable in measurement (Vogt, 2005).

revealed assets and inequities related to

Refers to recreation access to green space in the environmental literature; access can mean seamlessly including people with varying identities and abilities in all the ways that visitors recreate in green space, while feeling safe, secure, and a sense of belonging.

and ability, gender and racial

Identity includes all the labels one uses to represent themselves, particulalry referring to social groups, including but not limited to sex, gender, ethnicity, race, nationality, ability, religion, spirituality, age, socio-economic status, political affiliation (SAIC, 2020).

. These findings strongly informed the nature of professional development opportunities stakeholders engaged with, or provided. Equity specific analyses of findings from the outcome-based evaluation led to expansion of evaluation in two particular areas. Process level evaluation tools and trainings were developed, and targeted inquiry/trainings were created - both of which centered on equity.  

Extending to collective evaluation

The Oregon Outdoor School has a number of initiatives that extend their evaluative learning to the field of environmental education. 

1. Suite of self-evaluation tools and learning resources available free online
   • Instructional Resources Self-evaluation Tool (Backe, Braun and O’Neil, 2021) and learning resources
   • Culturally Responsive Self-Evaluation Tool (Brooks, Braun, Backe and Jones, 2020) and learning resources
   • Special Education and

     The extent to which people are excluded or not from an activity, program, or experience on the basis of experiencing a disability. In an accessible activity, program, or experience, people with disabilities are able to do what they need to do in a similar amount of time and with similar effort as someone that does not have a disability. The concept of accessible design ensures both direct (i.e. unassisted) access and indirect access, meaning the activity, program, or experience is compatible with a person's assistive technology (for example, computer screen readers). See also universal design. (Henry et al., 2014). 

     Self-Evaluation Tool (Brooks, Braun, Backe, Arbuckle and Jones, 2021) and learning resources 
2. Dr. Spirit Brooks from Oregon State University and Dr. Leilani Sabzalian from the University of Oregon created a free online class on Critical Orientations: Indigenous Perspectives in Outdoor Education. The course is designed to surface some of the common assumptions and misinformation about Indigenous people and cultures, particularly as they relate to outdoor education. While the course centers the 5th and 6th grade Outdoor School Program in Oregon, it is applicable to all levels of outdoor education and can be applied outside of Oregon.