EE21

Background 

This tool was developed by Dr. Marc J. Stern, Dr. Robert B. Powell, and their team in collaboration with practitioners, organizations, and academics in an effort to identify and develop crosscutting outcomes for environmental education in the twenty-first century. The EE21 questionnaire measures outcomes that many environmental education programs seek to address and influence in youth participants. They initially tested the tool at six STEM-related environmental education programs in the United States and have since used the tool with over 400 EE organizations.

Format 

The EE21

A standardized method of collecting data that consists of questions and responses. Questions can be "open-ended", but often take the form of being "close-ended" where respondents choose from predetermined answers (Frechtling, 2010).

include two single item questions and 10 scales, collectively measuring 12 outcomes. The questions ask youth to respond to a series of statements on a

A set of numbers or symbols used to represent characteristics of a variable in measurement (Vogt, 2005).

of 0 to 10. 

Audience 

The EE21 surveys are designed for program participants ages 11 and up. 

When and how to use the tool 

There are two ways to use these surveys.

• Post-experience only retrospective survey. This survey should typically be administered immediately following a program and can be used to understand how the program impacted each outcome area. The post-experience only method is best used in a comparative sense. That is, to compare outcomes across multiple programs. Pre-experience and post-experience surveys are more appropriate for

  A systematic process that critically examines a program's merit, worth or significance by collecting and analyzing information about a program’s activities, characteristics, and outcomes. The goal is to improve the program's effectiveness, and/or to inform programming decisions (Patton, 2001).

  purposes, but only for longer-duration programs.
• Pre-experience and post-experience surveys. These surveys were designed to be given to students prior to the start of the environmental education program and at the end. These surveys are not appropriate for programs that only last a few hours, rather, they are ideal for longer-term programs or overnight experiences.

Further guidance on how to use these surveys can be found here. 

How to analyze 

We recommend entering survey responses into a spreadsheet using a program such as Microsoft Excel. Create a spreadsheet with a column for each of the statements and a row for each individual. Assign each survey a

Unique number assigned to survey participants to allow for confidentiality and ease of data analysis. Similar to Medical Record Number used by doctors (Law Insider Inc., 2021).

, and enter each individual’s responses (ranging from 0 to 10) across the corresponding row. Enter a dot if the response was skipped.

Create an average score for each of the outcome areas across all individuals. The authors have provided a table (EE21 outcomes), which displays the survey

A type of question or statement used as part of a survey or interview to understand a respondents' reactions or feelings. Each item measures a particular idea (Salazar et al., 2020).

that can be combined to collectively measure each key outcome area (such as self-

Efficacy can be divided into self efficacy and collective efficacy. In self efficacy the focus is on the indidivuals’ belief in their ability to succeed or accomplish a task, whereas in collective efficacy, the focus is on the groups’ belief that they can succeed and accomplish a task, or that the group has the capacity to succeed (Krasny, 2020).

or connection to place). To do this, add up all of their responses for the set of questions in an outcome area and divide by the number of questions to get an average score. Do not include skipped questions for which you entered a dot. If a student (a case) skipped a question that is part of a set of questions to measure a key outcome (for example, three questions are used to measure “connection to place”), then leave the average for this case blank. Averages will be between 0 and 10. 

When administering the pre-experience and post-experience surveys, you can conduct higher-level statistics on your data to understand if participants had significant changes in the outcome areas after their participation in the program. 

What to do next 

Once you’ve administered your survey and analyzed the data, consider the following suggestions about what to do next: 

• If initial results show that your audience has lower scores for certain outcomes vs. others, you might want to determine how or if your organization wants to improve programming for those outcomes. 
• You could compare the results across different groups of participants (urban youth vs. rural youth, different grades, or gender identities) to determine if members have different scores than other groups and to see if your program is serving participants equitably. This could also provide justification for program revision or development, marketing, or funding proposals. 
• If you used the EE21 to measure changes in outcomes by administering it to participants before and after programming, do you see a change in scores between the

  A test or measurement administered before the program or activities begin. The results of a pretest can later be compared with the results of a posttest to show evidence of the effects of the program being evaluated (Zint, n.d.).

  and

  In program evaluation, a posttest is a test or measurement administered after services or activities have ended. Posttest results are often compared with pretest results to examine the effects of the program being evaluated (Zint, n.d.).

  ? Keep in mind that you may not see a change, particularly if your program is short in duration or is not designed to influence a particular outcome.
• Invite program staff or other partners to look over the data. Together you might consider:
  • What do these results tell us about our programming? Why do we think we got these results?
  • What results did we think we would get? And did these data support our goals?
  • If our results did not support our goals, can we brainstorm on areas within the programming or delivery to influence the specific outcome area? What changes should be made to programming, or how should new programs be designed?
  • What stakeholders should we reach out to for collaboratively discussing program design?
  • Who or what organizations can we share our learning with?

How to see if this tool would work with your program 

The EE21 can be used to measure the impact of an environmental education program across 12 outcomes (enjoyment, place connection, learning, interest/motivation,

A broad set of knowledge, skills, work habits, and character traits that are believed to be critically important to success in today’s world. 21st century skills reflect a broad and ever changing list, but include skills such as critical thinking, research skills, creativity, self-direction, communication, collaboration, technological knowledge, social-justice literacy, and more. Learn more at this site. (Great Schools Partnership, 2013)

, meaning/self-

Identity includes all the labels one uses to represent themselves, particulalry referring to social groups, including but not limited to sex, gender, ethnicity, race, nationality, ability, religion, spirituality, age, socio-economic status, political affiliation (SAIC, 2020).

, self-efficacy, environmental attitudes,

Behaviours with an intention to benefit the environment or to minimize negative impacts on the environment. Environmental behaviours often refer to personal and individual actions. For information related to collective action on environmental behaviours see the Environmental Action outcome definition (Stern, 2020; Kollmus & Agyeman, 2002).

, school behaviors, cooperation, and

An action towards something. Environmental educators are often interested in human behaviors that have direct or indirect impacts on the environment (Marcinkowski, 1993).

change.) If you seek to measure these outcomes in your program participants, we recommend reviewing the EE21 Survey Guidance and EE21 Outcomes. Are the outcome relevant and appropriate for your program? If so, then review each statement to further determine if it is relevant. Discuss with staff to decide if the results will be useful to you, and

An initial trial of a program, instrument, or other activity intended to test out procedures and discover and correct potential problems before proceeding to full scale implementation. Pilot tests can be conducted either for a program (i.e., testing out the program with a small group of participants) or for an evaluation (i.e., testing out instruments and data collection procedures with a small group of people similar to program participants). When possible, a pilot test, or trial run, is conducted with a sample group that is representative of the target population. Based on the results of the pilot test, revisions and improvements can be made before wider implementation of the program, instrument, or activity (Zint, n.d.).

the survey with youth ages 11 years and older who represent your audience. 

Tool Tips 

Here is some additional information about the EE21:

• You may want to customize the survey. For example, you may wish to insert the word “field trip” in place of “experience” or “program” or include a somewhat different introduction. You might also consider adding an open-ended question or two. If you decide to add additional items, these should come at the end of the survey, not before. 
• If you do not measure all of the outcomes in the EE21, you can pull out only those outcomes which you wish to measure. Be sure to utilize all of the appropriate questions for each subscale. 
• The authors of the survey are also working on developing a shorter EE21 survey and will be publishing this in the future.