Indicators

Indicators are next. You’ll be spending some time brainstorming, refining, and reviewing these, but you can start by jotting down ideas that seem doable in the near future. Let’s get into more detail about what makes an effective indicator.

Aim to identify two or three indicators per outcome, with a mix of qualitative and quantitative indicators, to get a wide view. Remember, an indicator is a neutral measure (quantitative) or descriptor (qualitative) of a change. It specifies the value to be measured to track the expected change. They allow you to understand how well you are moving toward change or where you are on the “thermometer.” It describes progress.

QuantitativeQualitative
Statistical measuresPerception, opinion, or quality
Number, frequency, percentile, ratio, varianceStories, presence/absence of certain conditions, quality of participation, level of user satisfaction
e.g.: Percentage of core funders with an adequate internal action plan for addressing the diversity of applicantse.g., Average value of the sense of relevance to users’ lives

Indicators can come from internal or external sources – e.g., data collected through your game, user surveys, and community engagement, such as Discord or Twitter. You don’t have to focus on traditional marketing metrics like clicks, likes, predatory interactions and addiction-based mechanics.

Privacy and consent are essential considerations in all these methods. Remember to consider what is ethical and legal when collecting data from your users. Always be transparent about what data you’re collecting and why, and give users the option to opt-out.

In addition, different types of games may require different data collection approaches: a mobile game might collect different data than a console game, and a single-player game will have different considerations than a multiplayer game.

Evaluating your indicators

As you draft some indicators, evaluate them based on the following questions.

  • Validity: Does the indicator actually measure progress toward the expected result?
  • Reliability: Will the data be consistent over time and readily available?
  • Sensitivity: When the result changes, will the indicator also change accordingly?
  • Simplicity: How easy will it be to collect the data?
  • Utility: Will the information collected be useful for your programming and decision-making?
  • Affordability: Do we as a studio (along with partners) have the resources to collect the data?

We also recommend you use the SMART goals framework to test them:

  • Specific - narrow and accurately describes what will be measured.
  • Measurable - can be counted and observed the same way every time.
  • Attainable - achievable and affordable.
  • Relevant - aligned with your ultimate outcome.
  • Time-bound - attached to a timeframe.

Ask yourself:

  1. Does this indicator directly measure the outcome?
  2. How easy is it to collect this indicator?
  3. When the outcome changes, will the indicator change as well?

Spend time brainstorming, editing, and sharing your indicators with your team. Focus on the outcomes you said you’d hold yourself accountable for achieving – not your whole results flow.

A good indicator makes it easy to know at a glance that changes have occurred.

Examples

Here are several short-term outcome–related indicators for a fictional game studio whose ultimate outcome is “People feel connected to their devices in a way that supports their health and wellbeing”:

OutcomeIndicators
Coworkers feel critically engaged with the collective creation of games within the company- Percentage of coworkers who actively contribute to the design process
- Employee sentiment around their involvement in collaborative game creation processes
Players are aware of apps that encourage and allow healthy interactions with their phone- Percentage of self-described gamers in the sample who can name at least one app that promotes healthy phone interaction
- Frequency of health-focused apps being featured in popular app storefronts
- Volume and tone of media coverage of apps that encourage healthy phone interaction

Here are some for a studio whose ultimate outcome is “LGBTQIA+ identities are understood and accepted in games, mirroring a society that celebrates diversity and inclusion”:

OutcomeIndicators
Players gain exposure to diverse characters and narratives- Percentage of players who can recall diverse characters and narratives in the game
- The number of diverse characters and narratives represented in the game
Community members feel safe and accepted, fostering a sense of belonging- Percentage of community members expressing a sense of safety and belonging
- Number of reports of harassment or toxic behaviour
- Frequency and quality of positive interactions in community spaces (forums, social media, etc.)

Here are some more ideas about what you could measure based on the topic area of your outcomes:

Player experience and satisfaction:

  • Average user-reported satisfaction score.
  • Ratio of positive to negative feedback.
  • Trend in sentiment scores from feedback, reviews, and social media.

Community engagement and health:

  • Total active community members.
  • Average engagement in community activities (Discord messages, event participation, newsletter opens).
  • Measures of community cohesion versus toxicity.

Inclusivity and diversity:

  • Average inclusivity rating from user feedback.
  • Diversity of user base demographics.

Environmental and sustainability impact:

  • Total energy consumption (servers and game usage).
  • Amount of waste from physical game production.
  • Impact of environmental offset actions.

Access:

  • Proportion of players requiring accessibility features.
  • Number of scholarships or grants for underrepresented developers.

Revenue and financial metrics:

  • Total revenue, in-app purchase income, subscription income.
  • Total costs (development, marketing, support, server maintenance).
  • Profit margin and return on investment.

Gameplay metrics:

  • Total, daily, and monthly active users.
  • Average session length and frequency of play.
  • Player retention and churn rates.

Challenges and limitations

Be aware of potential problems that may influence the accuracy and reliability of your indicators. Here are some areas to consider:

Bias

Bias can creep into your measurements. It might be introduced during the data collection process, especially if the data is self-reported by users. Consider methods to mitigate the impact of bias, like random sampling or anonymizing feedback.

Quantifying impact

Quantifying the impact of your game, particularly in terms of social outcomes, can be challenging. There’s a reason it’s not mainstream! Metrics like player satisfaction and community engagement can be measured relatively directly. Still, others, such as the influence of your game on players’ perspectives or attitudes, can be more difficult to capture.

Using a variety of qualitative and quantitative measures helps address this, as could using proxy measures. For instance, if you want to track increased empathy in players due to exposure to diverse narratives, you might look at changes in the way players discuss these themes on social media as a proxy.

Poorly articulated outcomes

Poorly articulated outcomes can lead to indicators that don’t accurately track progress or contribute to misunderstanding the impact. When defining outcomes in your results flow, put SMART goals to work!