Machine Learning Times
Machine Learning Times
BizML: Bridging the Gap Between Data Science and Business
  Eric Siegel, author of The AI Playbook, was...
The AI Hype Cycle Is Distracting Companies
 Originally published in Harvard Business Review. Machine learning has...
HR Analytics: Measuring Acquisition, Retention & Satisfaction
 Your firm is growing rapidly, and to maintain pace,...
To Avoid Wasting Money on Artificial Intelligence, Business Leaders Need More AI Acumen
 Originally published in Analytics Magazine, July 19, 2023. Business...

5 months ago
Experimenting with Machine Learning to Target In-App Messaging

Originally published in Spotify R&D Engineering, June 28, 2023.

Messaging at Spotify

At Spotify, we use messaging to communicate with our listeners all over the world. Our Messaging team powers and creates delightful foreground and background communications across the Spotify experience, experimenting with and tailoring the perfect journey for each user across our platform. Today we are able to send messages through WhatsApp, SMS, email, push notifications, contextual in-line formats, and other in-app messaging formats, like modals and full-screen takeovers.

In-app messaging at Spotify

What we refer to as in-app messaging covers a range of different message formats, with the unifying trait that they all appear when the user is using the app. See below for some example in-app messages.

This is possibly the most direct method we have for communicating with our users, which also means we need to be careful not to interrupt their listening experiences unnecessarily. We deliberately withhold in-app messaging for some users so that we can measure its overall effectiveness. These holdouts show that in-app messages have a mixed effect on user behavior when looking at the whole population, demonstrating that we would benefit from targeting users more selectively for in-app messaging.

We believed that we could use ML to decide in-app messaging eligibility and that by doing so we could improve user experience without harming business metrics. We discussed a number of possible solutions to this problem but decided to focus on uplift modeling, where we try to directly model the effect of in-app messaging on user behavior.

Heterogeneous treatment effect & uplift modeling

It’s clear that in-app messages have a different effect on different users, which is known as a heterogeneous treatment effect. We have some users that might enjoy Spotify Premium and would benefit from receiving a message prompting them to subscribe. We also have users that are happy with their current product offering, where messaging wouldn’t benefit either the user or Spotify. Our task, then, is to predict the effect of in-app messages on users. In particular, we wanted to understand the causal effect of sending in-app messages.

To continue reading this article, click here.

Leave a Reply