1. Evaluation is supported by relationships. Learn about the context, the program, and the people it serves. For example, you might speak with

   Can be geographically based (neighborhoods, towns, cities, etc.) or based on interest, identity, culture, belief, profession, etc. In environmental education community is often thought of as a system of systems including both natural, social, and economic systems (NAAEE, 2017).

   members and read about the history of the program, community, and the people that the program wants to serve. This is a process that takes time and draws on many sources, such as historical records and conversations with stakeholders.
   •  Many tools are available to help you build relationships and understanding. For example, this Toolkit for Engaging People in Conservation provides insight to defining your vision and identifying your audiences. 
2. Be thoughtful about who leads the program

   A systematic process that critically examines a program's merit, worth or significance by collecting and analyzing information about a program’s activities, characteristics, and outcomes. The goal is to improve the program's effectiveness, and/or to inform programming decisions (Patton, 2001).

   . Ideally,

   "Anyone who cares about and/ or is affected by a decision or outcome of a program, project, or initiative."

   (persons impacted by the program directly and indirectly) are involved in evaluations as team members, designers, decision-makers, and implementers of the evaluation from the very beginning. Someone from the organization might be accountable for the project, yet not the sole decision-maker.
3. Take stock of available

   Refers to time, money, and other in-kind resources.

   , both human and capital, such as time staff can devote, project partners, funding, technology, materials/supplies, community-based office space, civic capacity, in-kind donations, and other capital.
4. Determine how to allocate resources to compensate individuals who co-create and participate in the evaluation process. Compensation shows you value their time and contributions and is especially important if you are engaging individuals from marginalized identities and communities in co-creation. Compensation can include monetary and other benefits. Ask individuals what they would value or prefer in exchange for their time.

1. Identify and understand the purpose for conducting an evaluation from various perspectives, including program participants and partners.
   • Theories of Change and Logic Models are tools you might use to communicate and build consensus for evaluation. A

     A Theory of Change is the place to link what you do and how you do it. A theory of change explicitly exposes the assumptions you have for why what you do will lead to the outcomes you want. Often, practitioners and programs have a Theory of Change and don’t acknowledge or describe it. That can lead to putting in a lot of effort on activities without achieving your desired outcomes. To give an example for, this eeVAL website, an underlying Theory of Change is that infusing a culturally responsive and equitable focus to online evaluation resources will lead to three outcomes: (a) environmental educators who use the site will improve their evaluation practice, (b) organizations will increase their capacity for quality evaluation, and (c) environmental education as a field will benefit from these improved evaluation efforts.

     and logic model explicitly expose the assumptions you have for why what you do will lead to the outcomes you want.
2. Identify the

   Evaluation questions guide your evaluation. After deciding on your evaluation goals, you are ready to identify questions for the evaluation to answer. Evaluation questions may ask about a single element of your program, or the questions may focus on relationships between two or more elements of a program (e.g., to what extent does a particular output result in a desired outcome). They may also ask: To what extent were the planned activities completed? Why? To what extent are we achieving our outcomes? How well are we managing our program? What additional staff and resources are needed to meet our objectives? There are three types of questions you might ask about your program, process, outcomes, and impact questions.

   of importance to program participants and partners.
   • Program participants might include persons impacted by the program directly and indirectly (e.g., program providers, program participants, non-participants). Ideally, program participants are involved in evaluations as team members, designers, decision-makers and implementers of the evaluation (rather than as data sources only).
   • Project partners might include program staff, community members at-large, business owners, funders, and policymakers. Partners may have different questions about an EE program. Working closely with project partners at every stage of your process can help meet the needs of your project partner.
   • Through the process of engaging diverse voices, you will likely generate many potential evaluation questions, often more than you have resources to pursue. This means you might need to prioritize your evaluation questions.
3. Design an evaluation that is appropriate for the program and context. Think about how the evaluation will balance technical, political, and ethical considerations.
   • Technical aspects could include how prior knowledge and experience (

     The average level of a variable before a measure of the effect of the educational experience is taken (Vogt, 2005).

     ) are assessed or whether the size of your

     A group of subjects or cases selected from a larger group in the hope that this smaller group (the sample) will reveal important things about the larger group (the population) (Vogt, 2005).

     is appropriate. Avoid

     A sample that consists only of individuals who volunteer to participate in the evaluation (referred to as a "convenience" sample) can be problematic. The self-selected group may be different from the rest of the program participants, making it difficult to know whether results are truly representative of the larger group. This problem can even arise when random sampling strategies are used, if a large percent of those selected refuse to participate in the evaluation. You can overcome this problem through a non-respondent survey. Remember the quality of your sample can have a great deal of influence over whether evaluation results truly represent the experiences of all your participants.

     . 
   • Political aspects might include how to negotiate differences in evaluator and program partner perspectives. Fluid logic models are a great tool.
   • Ethics includes how to gather

     A legal condition entered into by a person who agrees or “gives consent” to participate in a study based on learning about and understanding the facts and implications of participation. Before being asked to agree to participate, participants should be provided with information concerning the purpose of the program evaluation, what they will be asked to do as participants, and how the information they provide will be used. Individuals should also be informed that they can withdraw their consent and stop participating in the evaluation at any time. In some cases involving adults, verbal consent may be adequate, but in general, it is preferable to provide written information about the evaluation that individuals can take with them and to have people who agree to participate sign and return a written consent form that is kept by the evaluator. When minors are participating in an evaluation, consent from a parent or guardian is required and additional safeguards (e.g., written consent from school district) are usually required. For more information on this topic, refer to Evaluation Consent and the Institutional Review Board Process (Zint, n.d.).

     to participate, who owns and will profit from the data that is collected, and who gets credit for the knowledge generated by the study. It is important to not only reflect on these questions, but to explicitly state how communities will have ownership over the data and numbers that impact them.
   • When you design your evaluation plan, remember to include program partners in the process. This builds trust and

     The way in which individuals and groups hold themselves to goals and actions as well as the values to which they hold themselves responsible. Can be internally or externally imposed (ICMA, 2021).

     , while attending to everyone's needs. In addition, this process can help ensure the quality of your data. 
4. Explicitly ask program participants and partners about the

   Both a term used to point to social inequalities that limit advancement, typically due to individual, structural or system discrimination based on an identity character (e.g., race, SES, gender), that operate outside of an individual and a term that can point to barriers internal to an individual (e.g., motivation, values conflicts, etc.).

   they may face to participation. Barriers vary by community, and it is important not to assume these include time, transportation, cost, or language only. 

Are you wondering how to select evaluation participants?

Initially, you may be tempted to get feedback from everyone who participates in your program. This may be an appropriate strategy, if the program serves a very small audience. With a large group of participants, however, this approach strains limited resources, such as time and money. Instead of collecting information from everyone in your

A group of persons (or other subjects of study) that one wants to describe in the evaluation process (Vogt, 2005).

(i.e., all of your participants), you can learn just as much about your program by collecting information from a sample of participants.
Sampling can apply to more than individual participants! Sampling also applies to other types of data such as documents (teacher logs, student exams, etc.), artifacts, photographs, and even environmental quality

Indicators are measures that demonstrate whether a goal has been achieved. We rely on indicators in everyday life. For example, if you have a goal to lose weight and improve your overall health, you may measure success using indicators such as the number of pounds lost, a change in your body mass index, lowered cholesterol, or even an increase in your perceived energy level. In an evaluation context, indicators provide the information needed to answer the evaluation questions. 

. You can also draw samples from different units of analysis such as classrooms, schools, or different program sites.

1. Consider (re)purposing data that is already generated (e.g., registrations, applications, attendance records, participant projects) and using embedded assessments (assessments that are directly built into your programming).
   • Do not forget the easy data! As a part of your evaluation work plan, remember some of the "simple" indicators such as counting the participants at events, the bags of litter collected on river clean-up day. It can be easy to forget to collect this information when you focus on the more intensive tasks of developing and administering evaluation instruments. Because simple indicators may provide critical evidence of success and may be impossible to collect later on, do not forget about them!
2. Choose tools,

   When you adapt it means you first pilot-test instruments to ensure that directions are clear and that measures are appropriate for the audience. After adapting, if time allows, you might "pilot" it with a few members of your study sample to see how it works. 

   , or design data collection tools that are appropriate to cultures, histories, and traditions within the local context, as well as the format and setting of the programming (in-person, virtual, outdoors, indoors). Methods might include

   Uses narrative forms, such as thoughts or feelings to describe what is being evaluated. Can provide rich context for examining participants’ experiences and how a program operates. Allows for questions to be investigated in-depth.

   tools like storytelling, interviews, chronicles, memoirs, observations, and focus groups, as well as participatory visual methods— photovoice, filmmaking, digital storytelling, drawings, painting, fabric art, fashion shows; and

   Uses numerical data to make sense of information. Analysis is perceived to be less open to interpretation and thus typically considered more objective. Allows collection and analysis of large amounts of data relatively quickly.

   tools such as surveys, questionnaires, and polls. Because there are advantages and disadvantages to both quantitative and qualitative data, many evaluations rely on a mix of the two (which can be practiced as a

   Uses a combination of both quantitative and qualitative data in an evaluation. It allows quantitative data to be collected from a large number of participants (increasing the likelihood that results can be applied to all program participants). It also allows in-depth qualitative investigation of evaluation questions with a smaller number of participants. It requires an evaluator who is able to collect data using a variety of methods and analyses (Zint, n.d.).

   evaluation approach). 

   Some additional resources that might be helpful as you do this work include:

   • User-Friendly Handbook for Mixed Methods Evaluations. Frechtling, J. and Sharp, L. (1997). National Science Foundation.
3. Engage program participants as partners in designing the instrument and data collection procedures. Before you do this, remember to consider the site and location where the data collection will take place and review the process of administering the evaluation instrument or approach thoughtfully and with attention to project partner needs.
4. Be thoughtful about who collects the data. For example, in a youth program, does it make sense for youth to help collect the data? If you’re conducting a

   The acronym stands for "Black, Indigenous, and People of Color" and is pronounced as "by-pock," rather than saying each letter individually. This updates the term “People of Color” and its acronym “POC” which has been criticized as erasing Black and Indigenous lives and experiences. The “B” in the acronym stands for “Black” and refers to people who have African or Caribbean ancestry. Its addition highlights the specific forms of racism and oppression that Black Americans face. The “I” in the acronym stands for “Indigenous” and refers to groups native to the Americas who were here before colonization by Europeans. Its addition refers to the discrimination and mistreatment Indigenous people have and continue to endure from official policies and practices as well as erasure of their culture and identity. The “POC” in the acronym stands for “People of Color” and is used as an umbrella term to refer to non-White individuals, including but not limited to those who hold Asian, Middle Eastern, Indian, and Pacific Island heritage, who often face discrimination. (Ansai, 2020; Garcia, 2020)

   focus group, does it make sense for a BIPOC person to lead that focus group? Does it make sense to have an environmental educator other than the one who taught the session collect feedback on the educational experience? It matters who does the asking. 
5. If you’re interested in understanding differences across subgroups, you have to find a way to respectfully ask participants how they identify along those dimensions of difference. For example, organizations that want to broaden the cultures and communities with whom they engage may want to know about the demographics of those who engage with their activities, and of those in their communities who are not

   “Engagement refers to the degree of attention, curiosity, interest, optimism, and passion that [learners] show when they are learning or being taught, which extends to the level of motivation they have to learn and progress in their education” (Great Schools Partnership, 2016). Engagement can be with students, with the teaching, with learning, in research, with educators, and/or with the community (Groccia & Hunter, 2012).

   . In these cases, you will have to think about how to ask culturally relevant demographic questions in an ethical way. You can get help on this topic here.  

6. Analyze the data (review this document for brief tips for ensuring that data are ready to be analyzed) in a way that includes cultural and contextual factors to increase the accuracy and relevance of interpretations. For example, examine outcome patterns across dimensions of difference (e.g. race,

   A social construct that divides people into smaller groups based on characteristics such as a shared sense of group membership, values, behavioral, patterns, language, history, and ancestral geography (Racial Equity Tools, 2020).

   , education, family composition,

   Referring to ability diversity and a variety of abilities and disabilities. These variations include differences in cognitive, social-emotional, and physical abilities. Differences in abilities may need more individualized approaches to instruction and evaluation (PennState Extension, 2017).

   status, etc.). You can’t clearly see what you don’t name or measure. Analyzing data in partnership with program partners and participants increases the accuracy and relevance of interpretations.

   Quantitative data can be analyzed in a variety of different ways including

   Descriptive statistics are numerical summaries of features of the data, such as correlations and averages.

   and

   Inferential statistics allows you to make predictions from the data.

   . Fortunately, there are many software programs to choose from including inexpensive or free ones, such as Statpages.org and Open Stat.

   Qualitative data can be analyzed using a variety of lenses or

   Qualitative traditions might include phenomenological approaches, case studies, or ethnographies. 

   . Many of these traditions rely on methods of reviewing data, coding it by theme, reviewing and revising the coding, and presenting the themes and support for the themes. This technique does allow for analysis by hand. A sophisticated software program may be more than you actually need or may simply not be cost-effective, although free programs include AnSWR and CDC EZ-Text. Analysis by hand or with a widely available program is most accessible when working with program partners. 

1. Communicate findings in

   Accessible language is language that accommodates people of all ages and abilities, including those with cognitive disabilities, people with low literacy skills, and speakers of English as another language (Bureau of Internet Accessibility, 2019).

   , written reports, and other forms in partnership with program partners and participants. Use reporting formats that honor cultures, histories, and traditions within the local context (see C2).

   Also, don't forget to plan ahead - reporting may take longer than you think! Don't forget to allow time to collect feedback from program partners and for partner-informed revision.

2. Share results with a wide range of program and community partners, including program participants and

   Collective evaluation extends evaluation accomplished in networked improvement communities, adding that the networked organizations share those components beyond the network for learning and improvement in the field at large (Clark et al., in prep).

   A group of people who come together to fulfill individual and group goals. They may share a common concern, a set of problems or an interest in a topic (Cambridge & Suter, 2005).

   . Internal and external program partners can help act on the findings and provide insights that can be used for use and continuous program improvement.

   Remember, different audiences may need different types of reports. The needs of the community partner are likely different than the needs of a funder. Carefully consider the interests, backgrounds, and expectations of your program partners and participants.

1. Use the evaluation results, and strengthened partnerships that come as a result of the evaluation, to enhance and implement programming with relevance.
   • Verify that evaluation results are being used as originally intended and stated to honor the

     Collaborative development of concepts, solutions, or processes with experts and project partners (Fronteer, n.d.).

     and build trust among evaluation participants.
   • Find ways to make evaluation an ongoing and regular part of your programming.
   • Incorporate evaluation processes and findings into strategic plans to stay responsive and relevant. Contexts and communities that are always changing.
2. Create opportunities to celebrate, learn about, and learn from culturally responsive,

   Aims to better understand evaluation efforts and their effects, both intended and unintended, by expanding the definition of validity and rigor. Equitable evaluation works to name implicit biases in order to make more informed decisions and to embrace the complexity of many lived experiences of practitioners and participants (Equitable Evaluation Initiative, 2020).

   .
   • Creating opportunities to celebrate and learn from evaluation might include events that bring together program or project teams to learn about and reflect on evaluation findings and processes.
   • Opportunities to learn about culturally responsive, equitable processes might include attending workshops that explicitly center equitable evaluation principles, as well as identifying or creating evaluation

     A space for people to align a shared goal or purpose. Effective communities are both aspirational and practical, allowing participants to learn from each other (Center on the Developing Child, n.d.).

     with partner organizations and individual colleagues.
   • Create opportunities to learn from your evaluation by sharing your process and outcomes within and beyond your organization to contribute to collective evaluation.