VISIONING:

MEASURING IMPACT

This module will guide you to assess the impact of your program. You will identify the goals of your program and outline ways you will collect data to help you understand and evaluate your program’s impact.

OBJECTIVE: Connect the parts of your program that will help you achieve your goals

OBJECTIVE: Create a plan for gathering data to measure the success of your program goals

What do we mean by data?

Data are pieces of evidence we collect in order to understand and measure change over time. Data can be quantitative, measured by numbers, like a test score or the number of participants, or they can be qualitative, which is any data that is not quantitative. Qualitative data can be made quantitative by assigning numerical values. One example of this is applying a rubric to homework and assigning a score out of 100. Data can be collected through observations, reflections, surveys, measuring or counting specific types of engagement. They can be created by participants, like artifacts or homework, or can come from facilitators or observers.

What do we mean by impact?

Impact is a measurement of the change over time of your program or a new part of your program. The greater amount of change you see over time, the greater the impact. The goals of the program determine what kind of impact you’re hoping to achieve.

What do we mean by program evaluation?

“Program evaluation is a systematic method for collecting, analyzing, and using information to answer questions about projects, policies and programs, particularly about their effectiveness and efficiency. In both the public and private sectors, stakeholders often want to know whether the programs they are funding, implementing, voting for, receiving or objecting to are producing the intended effect. ... Important considerations often include how much the program costs per participant, how the program could be improved, whether the program is worthwhile, whether there are better alternatives, if there are unintended outcomes, and whether the program goals are appropriate and useful.” (Wikipedia)

GETTING STARTED

This module can be completed on your own; however, you’ll have a much richer discussion if this is a collaborative effort with colleagues and partners.

WHO YOU'LL NEED

WHAT YOU'LL NEED

Your program system map with identified parts of your program

COLLABORATIVE VISIONING JOURNAL

VIRTUAL

Miro, Padlet, or Jamboard

PHYSICAL

whiteboard, post-its, chart paper, and markers

TIME TO COMPLETE

The program values module will take you and your team at least two hours to work through the first time. Remember this is an iterative process - you may need to come back and revise your thinking after you work through other modules.

LET'S PLAN!

PART ONE: Identify Program Goals

REFLECT INDIVIDUALLY

What are the top three to five long-term goals of your program? For reference, look back at your program development goals (module 3, part 2.4) and stakeholder goals (module 2, part 2).

Break these large, long-term goals into smaller pieces and identify any preconditions necessary for these goals to be met. What are some assumptions or preconditions that will need to be met to make these goals achievable? These might be things you will be able to address as a smaller step to the larger goal or they could be things you need to be true of participants before starting the program.

DISCUSS AND SYNTHESIZE WITH COLLEAGUES

Share your top long-term goals with your colleagues and collectively pick three to five long term goals. Document these in your collaborative visioning journal.

Continue to break these large goals down into smaller pieces with your colleagues.

What are the biggest goals for your program?

What are preconditions that need to be met for these goals to be successful?

What assumptions have you made?

What are some of the subgoals for these big program goals? What are steps you’ll need to take before you can achieve these larger goals?

PART TWO: Connect Goals with your Program

PLAN WITH COLLEAGUES

For each goal, identify the parts of your program that will lead to that goal/outcome.

Reference your thinking from Module 3: System Map (part 1) and Program Elements cards (part 2).

You might identify multiple parts of your program that you believe will lead to each goal. You might also uncover missing components you want to add to your program plan.

Iterate and document your thinking in your collaborative visioning journal.

For example

If one of your goals is to increase educators’ knowledge and use of maker-centered learning experiences, one of the parts of your program might be to provide workshops to experience different maker activities, or it could be providing time and structure to modify lesson plans to incorporate maker-centered learning, or perhaps coaching to reflect on a maker experience with learners and discuss how it went.

PART THREE: Identify Measures of Success

PLAN WITH COLLEAGUES

For each goal, identify three ways you can collect data to measure the success of that goal.

Consider how you can collect both quantitative and qualitative data.

Quantitative data are represented by numbers, for example, how many people signed up for a class, how many times was this video watched, or the average score on a test. These data are useful for understanding the breadth and scope of your program.

Qualitative data are represented by written descriptions, for example, a blog post about a lesson with pictures of students’ work in progress. These data are useful for understanding the quality of your program. Quantitative data provides an overall picture and is usually easier to collect and analyze while qualitative data gives you a deeper understanding of the details..

For example

If one of your goals is to increase educators’ knowledge and use of maker-centered learning, you could ask educators to:

share their lesson plans or artifacts created by learners.
reflect on how they think they’ve grown or incorporated maker-centered learning.
share ways in which they are using maker-centered activities in their learning environments.

As you collect these data over time, you will be able to see shifts in language, activities, thinking, etc.

PART FOUR: Synthesize and Document your Thinking

SYNTHESIZE WITH COLLEAGUES

Keep track of your goals and how they connect to your outcomes and measures of success using the table we provided in your collaborative visioning journal.