Integrating Data Scientists and User Researchers at Spotify

Integrating Data Scientists and User Researchers at Spotify

Sara Belt and Peter Gilks respectively lead the Creator and Free Revenue Product Insights teams at Spotify. In this article, Sara will explore the practice of User Research at Spotify, and Peter will lay out how Data Science and User Research work together to drive product decisions.

This piece was originally published by the awesome folks at Epic People:

Enjoy our companion playlist for this article:

Part 1. User research at Spotify

Sara Belt, Head of Creator Product Insights

When I say I work in user research at Spotify, folks' minds tend to travel in two directions: they think I research the kinds of music people listen to or I research the music itself: melodies, harmonies, rhythms, and how they impact people. Because, you know, what else is there to research with the world’s largest streaming music platform?

Over the past few years, Spotify has experienced incredible growth and the research scope has grown with it. My team focuses on how Spotify can help artists grow an audience, express their creativity, and thrive. We research fandom and how it manifests on and outside of Spotify and study the creative process and the daily hustle of being a musical artist. Spotify’s Product Insights team has over 100 people strong, exploring topics ranging from intuitive music interfaces for kids to differences in musical traditions of India and Brazil. We also study successful strategies for brands to leverage music in ads and how personal taste should be reflected in music recommendation algorithms.

The evolution of Spotify from a music player to a creative platform mirrors a trend we see throughout technology. As the concept of the product grows and the way we build them changes, we encounter a set of challenges and opportunities for user research as a practice that requires new tools and ways of working. Let’s take a look at six themes that define research at Spotify.

  1. Designing interactions between people. At Spotify, we design experiences for music lovers, artists, small businesses, record label employees, and internal playlist editors who all cross paths within a single product. Beyond deeply understanding “the user,” research is faced with complex optimization questions across different types of users with diverse incentives, needs, and abilities. We work on understanding and enabling interactions between people.
  2. Moving beyond transactional products. Products are growing outside the transactional box: tools for a job that can be perfected for easy task completion. Technology is increasingly designed for lingering, shaping our culture and society as it creates new behaviors and new economies. The palette of traditional design research methodologies is optimal for building tools that automate a known task but can fall short when designing experiences that are intended to facilitate a wider sphere of life or culture.
  3. Studying implicit interactions. The practice of user research has shifted from a core focus of understanding explicit user interactions to understanding implicit motivation and contextual expectations. Technology is increasingly experienced in the background without explicit attention or intention. This, of course, is nothing new, considering the pervasiveness of technology. What is new is that those ideas have traveled from academia to the everyday jobs of user researchers. Spotify is a background experience: consumed mostly through a phone or on the speakers. This means the only signal we get from the user through logs is what is being played. Research in the lab, or even the field, requires creativity and investment in longitudinal and naturalistic techniques, among others.
  4. Personalizing experiences. Through the expansion of artificial intelligence, a sea of new opportunities opens up not only for experience design but user research alike. We are beginning to abandon the idea that a product is singular and static. Instead, the research question becomes: “what are the dimensions that make our users different?” One of the distinctive experiences that Spotify is known for is its personalized playlists like Discover Weekly and Daily Mixes. While machine learning and algorithms play central roles in creating such playlists, what ultimately matters is whether or not that playlist resonates with listeners in a given moment of time. Achieving resonance requires an understanding of why people listen to music, how they intuit a good listening experience, and how this understanding can translate into ML models that provide personalized experiences at scale.
  5. Rapid and incremental cycles. Development of digital products, nowadays, is characterized by nonlinearity and speed. Instead of long development cycles resulting in the release of complete products, atomic components of products are released quickly to accumulate learnings about the intended and unintended outcomes. Having the opportunity to conduct extensive field research prior to a release is rare, but the need for understanding and representing people comprehensively throughout ‘piecemeal’ product launches is more necessary than ever.
  6. Accountability through measurable outcomes. Measurability of outcomes, and the immediacy of the feedback loop have a significant impact on our research practice. Researchers not only have to become comfortable with quantitative data and learn to triangulate that with qualitative insight but also are now empowered and accountable for producing hypotheses that can be tested at scale within weeks or months.

Given this context, what are the strategies we employ in building a research practice at Spotify?

We believe in mixed methods and diverse teams. Not only have we taken the step to merge our data science and user research teams into one , we are investing in heterogeneity within the insights disciplines as well as literacy and collaboration across them. The most important part of an insights practice is our ability to identify the right research questions and then lean on the community for collaboration and methodological expertise. We believe that mixed methods yield comprehensive answers: blind spots and caveats in specific approaches can be tackled through mixing methods. Triangulation allows us to have greater confidence and richer insights than possible to achieve through a single method alone. We aspire to form a comprehensive narrative of what we know about the current and future users of our products rather than methodologically siloed insights.

While maintaining the standard for validity and reliability, we encourage and celebrate creativity and experimentation with research designs and tools. We frequently try out novel approaches to gain insight into questions we haven’t explored before, or shed light into parts of the experience we haven’t studied in the past. We accept failure and expect inconclusive results from time to time, a valuable part of maturing and learning as an insights organization.

We double down on frameworks that describe our users and on storytelling around the insights. Producing high-quality insights is one thing, dispersing that into an organization in a way that evokes empathy, changes minds, and sets direction is another. Although writing and reading reports is expensive, we invest in mixed media storytelling, interactive insights, and employ typologies and illustrations that attempt to encapsulate knowledge beyond isolated findings.

Members of the Spotify Product Insights team at EPIC 2018

Part 2. The Power of Cross-Disciplinary Teams

Peter Gilk, Product Insights Director

Our approach to insights at Spotify is centered on our belief in triangulation, in mixing methods and in cross-pollinating ideas by bringing together Spotifiers of different backgrounds and expertise. We are highly invested in this approach and reflect it in our organizational structure.

At Spotify, we have two important, yet differentiating approaches to how we approach insights for product development; our Data Scientists and User Researchers form a single discipline that we call Product Insights, which is not a centralized function. Our insights teams are embedded with product teams to work alongside product managers, designers and engineers seamlessly, bringing plenty of benefits.

Building a cross-disciplinary insights team

A typical Product Insights team at Spotify contains people who are User Researchers and Data Scientists. In our case, we make no title distinctions between people with different areas of expertise within these groups. Some of our user researchers are experts in evaluative research methods, some informative methods; some more quality, and some more quantity.

However, most will use a variety of tools that will define that the right questions come before selecting how they will be answered. Similarly, our data scientists wear a number of hats and can be found running A/B tests, building data pipelines, conducting exploratory analysis, building inferential models, and designing visualizations.

There are a couple of very important things that are consistent across these roles. The primary job of everyone in Product Insights is to drive evidence-based decision making. Second, all insight teams have both data scientists and user researchers working together and reporting lines, not split by discipline. We have managers who came up through a data science route leading user researchers, and vice versa.

Generating hypotheses through qualitative data

Of course, organizing people from different disciplines into a single team is a good first step. But what does working together look like in practice? What benefits do we see? One type of common collaboration is the use of qualitative findings to generate and clearly express hypotheses that we can then test quantitatively.

Testing hypotheses are one of the most fundamental functions of our Product Insights teams. We don’t want to invest in developing a new product or feature without testing it out first, and we certainly don’t want to deploy it to production without thorough A/B testing. A good hypothesis is backed by evidence.

Of course, you can achieve this with separate teams, but there are many advantages to working together. As the qualitative findings develop, data scientists can begin searching for data points that reflect the human activities we are witnessing. This leads to four subsequent advantages:

  1. The time it takes to start any A/B testing is reduced as we are already preparing data sets and building pipelines.
  2. If the data can be used to measure the impact of an issue we witness from a handful of users in person in terms of millions of users, we can make the argument to act much more compelling.
  3. Our data scientists and user researchers don’t have to ‘stay in their lane’. If they have ideas that might improve the impact of each other’s work, they will share these ideas and debate them,.
  4. We develop a shared language for discussing insights focused on outcomes (e.g., does this feature drive product satisfaction?) rather than methods (e.g., run a logistic regression on this satisfaction data).

Explaining data through rich qualitative analysis

Another example of cross-disciplinary work is when we need user research to investigate the data we don’t really understand. For example, we might be doing some exploratory data analysis and spot a certain group of users behaving in a way that differentiates from the majority. We may have some idea as to why this is happening, or we simply may not. However, we identify and target users whose behavior displays these traits and conduct research with them—perhaps through a survey. Our findings can have a profound effect on what we decide to do with the product and how we expect it to perform.

Now that we’ve done this a number of times and have some idea of what to expect, we often preempt the need for follow up research and plan ahead as best we can. We recently conducted a large-scale diary study on how our Austrailian users would adopt and discover skippable ads, a test feature that we recently rolled out to our entire Australian user base. We sampled a range of users whom we knew had differing behaviors, ensuring that we asked participants to talk about some of the more interesting points we could see from their log data as the study progressed.

Representing users through storytelling

In order to ensure evidence-based decision making, we need to present our insights and the importance of each to our user base, in a compelling way.

When working with our colleagues in design, engineering, and product management roles, our usual approach to knowledge sharing is to give the whole picture whenever possible. Charts are accompanied by videos, survey results and A/B test results. Quotes alongside metrics and key takeaways are built from a triangulation of sources. This helps to engender user empathy and also means that our findings appeal to a broader audience who may among them have different learning styles or be most interested in different aspects of the findings.

This has been one of the most enjoyable aspects of managing a cross-disciplinary team. Seeing the creativity used in bringing these insights to life--with video, audio, interactive web design, data visualization, illustrations and more--is a delight and enables all of our team members to bring their own personality to what they deliver.

A Spotify data scientist and user researcher conducting an in-home study

Illustrating for Spotify with Jarom Vogel

Illustrating for Spotify with Jarom Vogel

In the Spotlight - Daniela Fernandez, Senior Product Designer

In the Spotlight - Daniela Fernandez, Senior Product Designer