Developing your impact measurement framework

Measuring and Managing Your Impact

Gauging success in a traditional, profit-centric business can be done easily using established and readily understood financial measures. Impact performance is more difficult to identify and quantify. How do you track changes in attitude, knowledge, and behaviour? Collecting and assessing this data requires a structured approach. You need an impact measurement framework (IMF)!

An IMF puts your results flow theory to the test. It answers the questions: How achievable are our outcomes? Does our timeline make sense? Are our strategies working? And lets you adjust your activities as you go to focus on what gets you closer to your goals.

It’s also a really handy resource for developing your business plan, setting your roadmap, and determining more challenging future targets.

We’ve previously covered setting goals, defining impact, and creating a results flow, and now we are learning how to make a plan to achieve our desired outcomes and measure our progress toward them. We are layering tools on top of each other to create a framework for turning our dreams into measurable impact.

Together, these tools will make it possible and easier for you to:

• Pitch for social finance (and traditional) investment
• Align your activities with your goals

We will walk you through the elements of an IMF, dig into how to create and evaluate indicators and get you started on designing a custom framework in this article. Let’s get to it!

Why use an IMF?

Can you get away with not having an IMF? Sure – pretty much every indie studio does. But they’re not deliberately working towards positive social impact like you are. There are three core reasons you should invest in creating and maintaining an IMF if you are impact-focused:

Accountability

Your funders – especially social finance investors – will expect tracking and reporting. Your stakeholders (employees, workers, community of players/supporters) expect you to operate and create games that reflect your stated values and goals. Your IMF allows you to identify, assess, and involve these stakeholders.

Resistance to this is understandable! Isn’t this just a capitalist, colonial tool for assigning value? It can be, but you can use it as a tool for yourself and your community instead. This is great to acknowledge and critique as you are developing your IMF. Be explicit about who is centred in your work – it will help you align with the right investors whose priorities are the same as yours. Think about who your stakeholders are. You need to measure different things depending on who they are and what they need. What is important to them?

Building on your results flow

The results flow identified the impact you want to have in the world and the activities you need to do to achieve that change. But it’s a fairly general tool. Each outcome describes whom you wish to impact and what outcome you hope for. But how do you move from wishes and dreams to making all this happen?

You need to build accountability into your plans. Accountability leads to a deeper analysis of what you are doing to make change, and how your activities actually lead to the changes you expect. Your results flow forms the basis of this accountability.

Evidence

An IMF allows you to gather evidence that tells you if you are living up to your intentions at any moment – so you can determine if the work you are doing, the audience you are serving, and the games you are designing truly put you on the path to fulfilling your mission. This evidence allows you to see what you’re doing that has a positive impact, and what is ineffective or harmful.

Improved activities

This evidence makes it possible to assess how effective specific strategies are at helping you meet your goals so you can adjust your activities, seek new resources, and better define your expected outputs. It allows you to refine the design of your games and make decisions that directly support your studio’s ultimate outcome.

Why not use an existing tool?

While there are out-of-the-box applications out there (Sopact, B Impact Assessment, Unit of Impact, Cigarbox, CSI Impact Dashboard), there’s no universal standard framework or set of metrics for social impact. Systems such as [IRIS+](https://iris.thegiin.org(https://iris.thegiin.org/) (which provides a taxonomy of standardized definitions) can be used for inspiration but can be overwhelming and hard to implement for a small studio. We’re going to go the custom route instead.

We believe your framework – just like your outcomes – should be carefully designed around your needs and desired outcomes, and heavily informed by the full spectrum of your stakeholders and community (including your employees!).

Understanding the IMF

What exactly does an IMF look like? They take many forms, especially web-based applications like those mentioned above. We will look at a spreadsheet version and an Airtable version today, but you can use any set of tools and technology that is comfortable for you. The main thing is to define your elements carefully and collect data in a structured, sustainable way.

At a glance, your IMF will show:

1. Progress towards change and impact in measurable terms
2. Documentation of that change

Here is what (a portion) of two real IMFs looks like:

When designing your framework, the goal is to:

• Identify the right indicators to measure your outcomes: What change are you measuring?
• Describe a plan for how to collect data: What are your measurement tools? Who collects data, and how often? What is automated?

Elements

Here’s a quick overview of the core elements of an IMF:

Building Your Own IMF

Before getting into more detail about each of the above elements, let’s set you up with a template so you can start roughing your information.

If you want to use a spreadsheet, you can access our template here: Impact Measurement Framework - Blank Template (Google Sheets). Select “Make a copy” in the File menu to save a version to your Google Drive. If you prefer Numbers or Excel, you can download your copy and import/open it in one of those applications.

At Weird Ghosts, we use Airtable to manage our IMF. Airtable is similar to a spreadsheet application but works more like a database and includes powerful features like data modeling, automations, and custom views and interfaces. You can use our template here: Impact Measurement Framework. Click “Use template” at the top, or for a sneak peek first, click “Explore the base” then “Copy base” in the bottom left corner.

Let’s start adding some info!

Outcomes

The first thing you’ll want to do is populate your template with the outcomes you’ve identified as those you’re ready to be held accountable for. We suggest looking through the outcomes in your results flow and selecting one or two per timeframe (short-, medium-, long-term). Just copy them over into the appropriate spot. (In the Airtable template, add them in the Outcomes table.)

Indicators

Indicators are next. You’ll be spending some time brainstorming, refining, and reviewing these, but you can start by jotting down ideas that seem doable in the near future. Let’s get into more detail about what makes an effective indicator.

Aim to identify two or three indicators per outcome, with a mix of qualitative and quantitative indicators, to get a wide view. Remember, an indicator is a neutral measure (quantitative) or descriptor (qualitative) of a change. It specifies the value to be measured to track the expected change. They allow you to understand how well you are moving toward change or where you are on the “thermometer.” It describes progress.

Indicators can come from internal or external sources – e.g., data collected through your game, user surveys, and community engagement, such as Discord or Twitter. You don’t have to focus on traditional marketing metrics like clicks, likes, predatory interactions and addiction-based mechanics.

Privacy and consent are essential considerations in all these methods. Remember to consider what is ethical and legal when collecting data from your users. Always be transparent about what data you’re collecting and why, and give users the option to opt-out.

In addition, different types of games may require different data collection approaches: a mobile game might collect different data than a console game, and a single-player game will have different considerations than a multiplayer game.

Evaluating your indicators

As you draft some indicators, evaluate them based on the following questions.

• Validity: Does the indicator actually measure progress toward the expected result?
• Reliability: Will the data be consistent over time and readily available?
• Sensitivity: When the result changes, will the indicator also change accordingly?
• Simplicity: How easy will it be to collect the data?
• Utility: Will the information collected be useful for your programming and decision-making?
• Affordability: Do we as a studio (along with partners) have the resources to collect the data?

We also recommend you use the SMART goals framework to test them:

• Specific - narrow and accurately describes what will be measured.
• Measurable - can be counted and observed the same way every time.
• Attainable - achievable and affordable.
• Relevant - aligned with your ultimate outcome.
• Time-bound - attached to a timeframe.

Ask yourself:

1. Does this indicator directly measure the outcome?
2. How easy is it to collect this indicator?
3. When the outcome changes, will the indicator change as well?

Spend time brainstorming, editing, and sharing your indicators with your team. Focus on the outcomes you said you’d hold yourself accountable for achieving – not your whole results flow.

A good indicator makes it easy to know at a glance that changes have occurred.

Examples

Here are several short-term outcome–related indicators for a fictional game studio whose ultimate outcome is “People feel connected to their devices in a way that supports their health and wellbeing”:

Here are some for a studio whose ultimate outcome is “LGBTQIA+ identities are understood and accepted in games, mirroring a society that celebrates diversity and inclusion”:

Here are some more ideas about what you could measure based on the topic area of your outcomes:

Player experience and satisfaction:

• Average user-reported satisfaction score.
• Ratio of positive to negative feedback.
• Trend in sentiment scores from feedback, reviews, and social media.

Community engagement and health:

• Total active community members.
• Average engagement in community activities (Discord messages, event participation, newsletter opens).
• Measures of community cohesion versus toxicity.

Inclusivity and diversity:

• Average inclusivity rating from user feedback.
• Diversity of user base demographics.

Environmental and sustainability impact:

• Total energy consumption (servers and game usage).
• Amount of waste from physical game production.
• Impact of environmental offset actions.

Access:

• Proportion of players requiring accessibility features.
• Number of scholarships or grants for underrepresented developers.

Revenue and financial metrics:

• Total revenue, in-app purchase income, subscription income.
• Total costs (development, marketing, support, server maintenance).
• Profit margin and return on investment.

Gameplay metrics:

• Total, daily, and monthly active users.
• Average session length and frequency of play.
• Player retention and churn rates.

Challenges and limitations

Be aware of potential problems that may influence the accuracy and reliability of your indicators. Here are some areas to consider:

Bias

Bias can creep into your measurements. It might be introduced during the data collection process, especially if the data is self-reported by users. Consider methods to mitigate the impact of bias, like random sampling or anonymizing feedback.

Quantifying impact

Quantifying the impact of your game, particularly in terms of social outcomes, can be challenging. There’s a reason it’s not mainstream! Metrics like player satisfaction and community engagement can be measured relatively directly. Still, others, such as the influence of your game on players’ perspectives or attitudes, can be more difficult to capture.

Using a variety of qualitative and quantitative measures helps address this, as could using proxy measures. For instance, if you want to track increased empathy in players due to exposure to diverse narratives, you might look at changes in the way players discuss these themes on social media as a proxy.

Poorly articulated outcomes

Poorly articulated outcomes can lead to indicators that don’t accurately track progress or contribute to misunderstanding the impact. When defining outcomes in your results flow, put SMART goals to work!

Responsibility

Who will collect the data or stories? This person is responsible for collecting data for reporting and analysis; it is not the person responsible for achieving the stated targets. Be as specific as possible – a name is best, but a job title, role, or department can work too. When it is time to collate and analyze your data, you need to know whom to go to. And they need to know the full scope of what they need to collect well before deploying tools like surveys.

Frequency

How often will you collect the data: monthly, quarterly, semi-annually, or annually? Each indicator warrants a different cadence. It can take time for information to accumulate and for discernible changes to happen, particularly for those medium and long-term outcomes. But don’t do it too infrequently, or you will miss opportunities to adapt or course-correct.

Data Sources and Tool

Where will you get the data?

As noted earlier, data sources can be varied. Try to triangulate your sources, such as:

• User playtesting/focus groups: Feedback from users in structured sessions to understand user experience, game mechanics, and narrative effectiveness.
• Anonymous user data: Data gathered from user interactions with your games for quantitative insights into gameplay mechanics, user preferences, and behavioural patterns.
• Community members: Online communities related to your games (e.g., forums, Discord servers, social media platforms) tell you about game experience, character engagement, storylines, and community health and engagement.
• Employees: Experiences and observations of your wonderful team members highlight studio culture, development process, and diversity and inclusion initiatives.
• Platforms: Sales data, user reviews, and other metadata from Steam, Epic, Apple’s App Store, Google Play, Switch, etc., shows you game performance and player engagement.
• Existing databases: Research databases, industry reports, and demographic data from industry associations such as the ESA, Interactive Ontario, and DigiBC give you context and comparison points for your indicators.

How will you collect the data?

The data collection method depends on the nature of the indicator you’re tracking and the available data sources. Some examples include:

• Monthly automated reports: Some analytics platforms can generate regular reports providing detailed user activity, in-game events, and monetization statistics.
• Surveys: Online surveys can gather specific data directly from players or employees and can be designed to gather quantitative data (e.g., ratings) or qualitative data (e.g., open-ended responses).
• Interviews: Interviews with users, employees, or other stakeholders can provide rich, in-depth qualitative data.
• Focus groups: Gathering a group of users or other stakeholders for a discussion can provide diverse perspectives on a specific topic or issue.
• Observation: Behavioural data gathered from user interactions with your game or online communities can provide insights into play patterns, user engagement, and community health.
• Research: Secondary research (e.g., reviewing industry reports) can provide contextual information, trends, and benchmarks.

Baseline and Target

A baseline and target are needed for each indicator to actually make use of your IMF.

Your baseline describes the situation at the beginning (e.g. when you start a new project or start collecting data). For example, say you’ve just launched a new moon journey feature in your app. You start to collect data, and in one week, you have a baseline – 10% of users have found and used that feature.

Once you have your baseline, you can set a target by stating a value or figure you aim for. Describe the desired situation if the change is realized, in terms of the indicator. It must be realistic given your capacity and resources! Arriving at a doable target will take some trial and error.

Your target describes the desired situation if the expected change is realized, expressed in terms of the indicator. For example, “90% of users discover and use the moon journey feature in Q2 2021.”

Setting targets and collecting data

Here are a few hot tips for setting your targets and collecting data!

1. Be kind to yourself: establish realistic targets and baselines. They set the direction for your work and form the basis for measuring success. Take into account factors like budget, available time, people-power, and tech resources. Overly ambitious goals will lead to frustration – and setting the bar too low means you never reach your goals or make a real impact. (The opposite of what we’re trying to do here!)
2. Consider the cost associated with collecting and reviewing data, the time it takes to gather it, the humans required for these tasks, and the software or technology needed to process and store the information.
3. Don’t forget about administrative tasks like cleaning data to remove inaccuracies or inconsistencies, safely storing data to prevent loss and unauthorized access, and taking steps to protect data to ensure compliance. Read up on federal and provincial/territorial privacy laws.
4. Think strategically about success and failure. Generally, success means meeting or exceeding your targets. Failure, on the other hand, could mean falling short of these targets or running into negative impacts you didn’t think of. But failure is not a dead-end – it’s an opportunity to learn, adapt, and improve your approach to impact measurement. It’s part of your journey with us toward creating a more impactful, inclusive, and sustainable video game ecosystem in Canada. 😀

Moving forward with your IMF

Creating an IMF is far from a “set-and-forget” thing; it’s a living document with a complex interplay of elements that need to be kept top of mind, tracked, and updated regularly. Your operations will evolve as your studio matures, and so should your IMF. Regular reviews and revisions will keep it responsive and relevant to your goals.

Transparency is a huge opportunity once you’ve started measuring your impact. Don’t just sweat over numbers behind closed doors – share your impact. Report the good, the terrible, and the unexpected – they’re all part of your unique story and will resonate with those who care about what you’re doing. Like us!

So, where are you at? Have some indicators you’d like some feedback on? We’d love to hear about it and help you on your path to impact measurement.

Resources

• Airtable IMF template
• Google Sheet template