Scorecards: Creating Alignment Throughout the Process

So in March last year, just as we were entering our WFH period, our Principal Designers in Premium R&D updated their product development toolbox. The goal: to help teams in Premium R&D increase the quality of what they ship, and create greater efficiency and alignment within their larger teams. A key part of this release was a hive of resources, templates, tools and activities, which teams could use throughout each of the development phases.

One of the tools in the box was Scorecards, a tool to guide the process of defining clear success metrics for a project. Scorecards are a transparent and collaborative artefact, which get everybody on a project aligned, and prevent ambiguity.

The ability to visualise thoughts, beliefs, and success is a Designer’s superpower, which can ultimately get everybody on the same page. Which is why we felt compelled to share this internal tool to the design community, in the hope that designers can bring this tool to their teams to set them up for success. We’ve found that by capturing a team’s success criteria early in a project, you can give all partners that superpower, making it easier for everyone to evaluate the work as they go.

Before you start using Scorecards

Before explaining how Scorecards work in more detail, it’s important to highlight the product development process we’ve adopted at Spotify. 

In this process, an idea goes through five phases of development: Understand it, Think it, Build it, Ship it, and Tweak it. You can learn more about each phase in this article, which includes a rough example blueprint of the Spotify product development process. Another crucial step that needs to happen before a Scorecard is created is aligning everyone on the knowledge surrounding the project. Without this, it will be very difficult to have a productive Scorecard discussion. 

There are several ways to achieve this alignment, and different teams have their own approach, but here are a few steps we tend to follow when working on our Premium product:

1. Leverage what you know about the problem

Surface existing data and insights about the problem space and share them with the team ahead of the project kick-off. A good place to start is with a Thoughtful Execution tree (find out more about how this works here).

2. Kick-off the project with an Understand It Brief

As part of the kick-off for a project we fill in a project brief (the success criteria in a project’s Scorecard will most likely be a distilled version of this). The Understand It Brief can serve as a collaborative document, which allows all partners the opportunity to align on the core pieces of the project (problem, outcomes, success, etc.). 

3. Acknowledge what you don’t know

In many instances, you won’t have all the information you need at the start of the project, and that’s okay. However, it’s important to acknowledge that and define a plan for how you’ll gain those insights.

After following these steps you should be ready to start creating a Scorecard.

Our success factors explained

We’ve defined five key success factors within a project: Desirable, Viable, Feasible, Usable, and Unified. These factors embody the values we look to achieve within each phase of our development process, and they’re the foundation of our Scorecard tool — you’ll learn more about how these factors apply to a Scorecard in the next section.

The illustration above shows at which step of the process we refer back to each of the success factors. 

Understand It: What are our success factors?

The Understand It phase is where we build out our definitions for each of the five key success factors. We then use them as gates to check-in as we advance the project through each phase of our process.

Think It: Is it Desirable?

During the Think It phase, when we’re ideating on potential solutions for addressing the problem we’re looking to solve, we focus on Desirable as the core success factor. We use it to evaluate the potential of an idea to address real user needs. If it isn’t solving a real user need, we’ll start exploring other ideas that will meet the criteria for desirability.

Build It: Is it Viable and Feasible?

Once we agree that an idea meets our Desirable success metric, the project moves into the Build It phase, where we introduce Viable and Feasible as success factors. Viable is an important factor at this stage as it helps us understand if an idea will contribute to our product and business goals. Feasible helps us understand the complexity of building this idea within a given time frame. Having met these three success factors, we can then start building our idea and discussing how we’ll ship it to our users.

Ship It: It is Usable? Is it Unified?

In the Ship It phase, we introduce the last two success factors, Usable and Unified. Usable helps us evaluate if people can successfully use the product we’re building, and Unified helps us understand if an idea aligns to our design system and design principles. It’s worth noting here that your team may have different success factors already in place, and if that’s the case, you should have no problem substituting in those factors for the ones we use at Spotify.

Creating a Scorecard

Every project has its own timeline which can be anything from straightforward, to something very ambiguous and strategic. Regardless, Scorecards and their embedded success factors can still be used to keep any project on track. Creating a Scorecard — defining the key success factors within your project — is designed to be a team activity because it’s important to give everyone an opportunity to contribute, in order to achieve overall alignment within the team. However, within each success factor, there should be one person listed as the primary evaluator. 

Here’s a breakdown of each success criteria, what they evaluate, and their recommended primary evaluator:

Desirable

Evaluate an idea against user needs.

Primary Evaluator: User Researcher

Viable

Evaluate if the idea will help us meet our product and business goals.

Primary Evaluator: Product Manager

Feasible

Evaluate the complexity of building and shipping an idea.

Primary Evaluator: Engineering Manager

Usable

Evaluate if the idea we’re building is usable (e.g. can users find the feature, is it easy to use, do they want to use it?)

Primary Evaluator: User Researcher

Unified

Evaluate if the solution is aligned with our Design Principles and Design system. 

Primary Evaluator: Product Designer

To understand Scorecards in practice, let’s have a look at Discover Weekly as an example. Discover Weekly was one of many attempts Spotify made at tackling the need coming from users to have a convenient way of discovering new music.

The Desirable portion of that project’s Scorecard looked like this*: 

You can see in the example above that we’ve used not just one definition of success for Desirable, we also included what it meant for an idea to fail or to surpass our key metric. We’ve found that the clearer you are with your success criteria, the more useful your Scorecard will be when it’s time to evaluate your project against them. It’s important to be ambitious and set lofty goals for your surpass criteria. In the Discover Weekly example, the goal was to make music discovery effortless, delivering a playlist that simply works like magic. To get a full picture of the project, this is how the complete Scorecard shaped up for the Discover Weekly project*:

Tools for creating a Scorecard

At Spotify, Scorecards have been super helpful in assisting many of our teams throughout the development process for projects. And over time we’ve learned how to use them more efficiently. We want to share our learnings in creating Scorecards with you, so we’ve created a brief guide with some tips on how to run a productive Scorecard session with your project partners as well as a Scorecard document template (with a Discover Weekly example included). We hope Scorecards will be a helpful tool for you and your project partners and allow you to develop products with deeper clarity and efficiency. Enjoy :)

Shout-outs to Antonio Rezende and Andrew Bennett for their help putting this post together!