Unlock this video + Thousands more !

Start your free 30 day trial today!

Join Free Sign in

Moving a Metric at Scale

Speakers

Jason Schwartz , Spotify

Jason is the Director of Growth Product at Spotify, where he is focused on fueling the companies rapid growth. He has worked with dozens of top startups to optimize the core funnel to move key metrics. He has written a forthcoming book on the subject, and has taught hundreds of product managers how to achieve these results through his class at General Assembly.... Read More

What you will learn?

  • Understand the ingredients and steps Spotify uses to grow
  • Understand what a lightbulb moment is and why it is important for your users to have it
  • Understand what a habit path is and how to push users to go down this path
Video Transcript
How's everyone doing? Good?

So I’m Jason, I'm the Director of Growth Product at Spotify. And so, you know, we do things that are probably very similar things that you guys do in your roles. But just so I understand who I'm talking to, so a show of hands. How many you are product managers? Okay, just growth marketers? Designers? Engineers? Okay, cool.

Alright. So the first thing I did when I got into Spotify was that I assumed that there was going to be a bunch of people inside of there that knew all these secret tricks about like how they managed to get Spotify to grow as much as it did and I was going to be like super outclassed and you know, they're going to find out that like I didn't know as much as they did.

The fact of the matter is I got in there and realized that that wasn't true, right? And then I started talking to a bunch of people, bunch of other companies kind of like Spotify, same level, wasn't true there either, right? There is no secret trick. They don't know some secret growth hack that suddenly made any of these apps blow up the way that they did, alright? And so that was kind of that was a bit of a relief was like, okay, there's no secret trick and then I had to figure out, okay. So like what are we going to do now, now that there's no secret trick?

When I really dug into it, like why was Spotify growing so much, there's really two things. One, that they had product market fit. And you really can't underestimate how much power that has and how difficult it is to screw up once you have product market fit. Number two is that people are executing well. And I say well intentionally, I don't mean like executing like at the absolute most extreme level of perfect, because companies are not perfect. They're executing well, and they're executing consistently well. And that's kind of what I saw. It's like, okay, like everyone around me is executing on a really good level all the time as opposed to a lot of companies where you see like someone's executing like really great and there's a bunch of people whp really aren't executing that great at all, and that creates these kind of like peaks and valleys and then the company doesn't really move forward.

So those were the two things that at least that I could identify but like why is this thing working so well. And what was interesting is that you might have access to more tools than Spotify does. Because what happens is there's a huge amount of tooling for growth marketers, people who do the types of work that we do, that work all the way up from I have a small start-up to I have a medium-sized company and maybe even have a million users or so and you can utilize all of them, right? You can plug them in and you know, there's some integration pain, but, you know, you're up and running and you're working and you're focusing on something else.

At Spotify and in companies of that scale, you can't plug those things in anymore. They just don't scale. They weren't built for any of that. Not only do they not scale, there's all sorts of issues about data privacy, the risk of a leak is absolutely astronomical and it's astronomical from the trust of your users and astronomical in terms of the fines you would have to pay if something like that were to happen. And so on many many levels you just can't plug in these vendors anymore and you're kind of stuck in this situation, the scenario where you know, you had all these tools when you were a smaller company and now all the sudden you don't have these tools anymore because you have to build them all for yourself, right? And this is this is a process that many companies like Spotify have gone through and had to figure out for themselves. So it should be heartening to some extent that you actually probably have more tools than maybe even Spotify does that at your disposal.

So I want to talk a little bit about this lightbulb moment because this as kind of growth practitioners inside of Spotify is what we really think about the most.

So there are millions of people who are convinced that Spotify is valuable, right? Like we know that. We know that something happened and they convinced themselves like this is a really great thing. And so growth in terms of what we do is figuring out that light bulb moment and getting more people to experience it. What happened to all of those millions of people were there like, oh, this is a pretty valuable thing, I should do more of this, right? Figuring out what that thing is is what we define is as figuring out growth.

So, how do we get there? What do we what do we do? So we have this model where we learn and we get some signal and then we feed that signal back into what we learn and we kind of use data to tie the two together and that's kind of how we figure out what drives growth and I'm going to break down break down that model for you.

So on the learn side, we need to discover the light bulb moment. What happened in all those users’ minds that said, wow Spotify is great. And on the signal side, we need to lead more users towards that light bulb moment, right? If all of you or a big majority of users had this lightbulb moment because they realized X then we got to get more people to realize X. And that's pretty much what we do.

So let's say we want to increase conversion. We'd say what user actions lead to conversion, we call this ‘finding the habit path’. I don't know if you caught Nir’s talk earlier, but I actually stole that from him. Oh, there you are. Hey, stole it from you. They call it the habit path, like, right how do we, how do we get more users to take that?

And then on the signal side, how do we increase the number of users who take that habit path? If we teach users how to do these steps, will it actually increase activation? And that's the big if, right? Because many of the things we're going to try actually won't increase activation or conversion. So within this model, first, we identify the behavior. We’re going to surface user behaviors that might lead to that habit path. And so for example, let's say the majority of activated users use the playlist feature within the first three days. Now that's not true. I'm completely making this up. Please don't like tweet this or put this somewhere and I don't want to see this in TechCrunch anywhere. But let's just say, hypothetically, majority of activated users use the playlist feature within the first three days and if we get more users to play that to playlists in the first three days, will activation go up?

Alright, that's our hypothesis. So we use external users to bring users back into the application. So in this case, we're talking about activation. That means someone has gone from a registered state to I'm now using Spotify. Chances are, if you’re worrying about that, they’re, might not be in the app any longer. So we're going to use our external signal to try to try to pull them back in. And let's say that it took, on day two, users have not created a playlist. We're going to send them a push notification and that pulls them back in.

Now we have our in-app signals. And so let's say we need to now educate user on that identified behavior. So we sent them the push, they came back into the site, and upon opening the app, we're going to show a tutorial. That tutorial teaches them, how do you playlist more?

And then we're going to measure for causality does increasingly identified behavior actually increase activation, right? And that's the question, right? Coz we have two things we have to measure, like did we get more people to playlist? And by getting those people to playlist, did our metrics on this case activation go up?

So let's say we do all this. We increase the number of play listing that people do but we didn't increase activation. And that's actually the normal state. That's what will happen most of the time. Most Loops will end in correlation without any causality. Right? And that's the trick and this is where people often get discouraged. This is where people on my own team have gotten discouraged like, they do this work, and they run the test and they got the number of playlists to go up, and then it didn't move the number and they’re like, this doesn't work.

Well the truth is that you're gonna have to do this many many many times. There are companies that have taken, big companies that we all know the names of that has taken them two years of doing this type of work in order to get to this sort of answer. So this is a long sustained effort and is not something that happens very quickly.

All right, so that's kind of the model, the model that that we follow. So let me, let me talk about the platform that we actually built or are in the process of building to support work of that kind.

So in the first phase, you need to be able to identify behaviors that lead to the lightbulb moment. Now, this is actually tricky, right? Like I've tried to throw like data scientists at this and do it sort of in like a one-off way, and it's actually a bit too slow. So you actually need to build some tooling out in order to make that make that easy.

In kind of phase two, you need to be able to take the learnings from the test that you're running and feed that automatically back into your system so that your data is getting better and better and better and it doesn't require like a manual person to go in and like update the data and reupdate all their models and like figure out what to do with that because again, it's going to take too much time. Right? And we're always, you'll start to notice everything I'm talking about we're optimizing for time here, because we have to run this cycle again and again and again a hundred, hundreds of times. Really the only thing that matters is how fast can you run it. Right? And so that's everything that we're optimizing for. How fast can we get those cycles going.

So on the signal side, phase one is just the ability to trigger an email or a push notification based off of user behavior. Now, there's all sorts of vendor tools that will allow you to do this. This is a bigger deal at a company like Spotify where a lot of times those vendors don't work and so like we have to build something like that.

And then in-app, and there's less of a vendor solution here. Most people have to build this themselves regardless of size is how do you talk to the user inside of the application? How do you dynamically serve in at messages from the back end where you're not having to do a client release in order to talk to the user just because you had a hypothesis?
In phase two is really getting to that speed where it's like, all right, there's no feature release required, we can talk to the user inside of an application. We could trigger on any number of things. And we can do that pretty quickly. We can do that within the day as opposed to it's going to take us two weeks of engineering, we're going to submit to Apple, and Apple’s going to review and then it's going to roll out, and then it's going to take a couple weeks in order for everyone to upgrade on and on and on and on and now it's like two months later and you finally have your result and you were wrong, right? That's never going to get you to the type of growth that we're looking for.

And there's also omni-channel messaging. You might send an email and a push. You might use a Facebook ad, you might want to talk to the user in and outside of the app. How do you do that in a way that, that's actually all connected to each other and not just a bunch of separate systems?

And phase three is machine learning. And so at some point what's going to happen is that you have the ability to talk to the user outside of the app and in the app. You have the ability to get many hypotheses coming at you from the data and at some point, it's going to be beyond people's ability to manage all that right? You're going to have all sorts of people running experiments and they're going to start to collide. What any given user would be good for them is different from a different type of user and how you optimize and manage all that is basically impossible at some point. And so you need to use machine learning in order to, in order to handle that. And so the question is like how do we still deliver the right message to the right user at the right time given all these incredible capabilities? Given the fact that we've enabled ourselves to move fast? And that's what we, that's what we use machine learning for.

And so growth is only achieved if it's personable, personal and valuable and not creepy. And so when I talk about this stuff, it often, I'm sure it's come up in conversations when you guys have told people like what you do for a living, they’re like, oh like that sounds like manipulative or that sounds weird. But the truth is that if you do this well, what you're doing is you're creating communication that's personalized to that user that they truly see as valuable. They don't see it as a marketing message. They're like, oh like that was great, that just made this experience better. And if you can get down to that and it doesn't feel creepy, it feels great, and this stuff works phenomenally well. You're on the other side of the spectrum and you're just like spamming users and you're basically like kind of like forcing it down their throat, none of the things that I just talked about work particularly well.

So people who do this in a fantastic way. So this is an email from Mint, says I got hit with a finance charge, right? This is extremely personalized to me. It's completely different, you would get a completely different email. This is Quora. There are, they’ve basically taken out the news feed of their application and embedded that into email. Right? So every person's email is completely different. And this is medium where they've done a very similar thing. They've embedded your feed inside of inside of inside of an email.

So in each of these cases, the signals are actually becoming an extension of the product and that's what makes it actually work. Right? It's not just that we're emailing people or that were behaviorally tricking emails. It's like we've managed to take our product experience and bring it into a different channel and enhance the user's experience.
And this doesn't work with Phase 1 signals. So just the ability to say, okay, well if the user does X, Y and Z things, I'm going to trigger an email. That really doesn't get us to the type of individual personalization right message to the right user at the right time that I'm talking about here. And so you really have to go through the other steps of those phases in order to get to sort of this Holy Grail.

And those right signals, that's what makes the lightbulb moment into a habit. Because you don't just need the user to say like, oh that was great and then forget about it. Like they got to do it a couple times and it has to become habituated in order for you to have a metric that lasts over the long haul.

So if we do this, this turns all of our users into our best users and that's kind of the mental model of this. You look at the best users. You see what makes them your best users and then you do a bunch of things to try to make those users, try to guide their experience, so that they become what looks like basically your best user and that's how you increase numbers like activation and conversion and retention.

So this is what we're working on. This what we're building at Spotify. We're not all the way there yet, and you know, we're actively working and looking for designers and engineers and experienced product people to help us do that. So if you fall into any of those categories, feel free to speak with me afterwards. Thank you.

Related Videos

Premium How to Build a Growth Roadmap for Your B2B Product Cover Image
How to Build a Growth Roadmap for Your B2B Product
Dan Olsen
Conference Logo
Premium The Ultimate B2B Growth Stack Cover Image
The Ultimate B2B Growth Stack
Dan McGraw
Conference Logo
Premium How to Drive Optimization, Personalization & Explosive Growth with... Cover Image
How to Drive Optimization, Personalization & Explosive Growth with...
Krista Seiden
Conference Logo
Premium A/B Testing: Why You're Probably Doing it Wrong Cover Image
A/B Testing: Why You're Probably Doing it Wrong
Martijn Schiejbeler
Conference Logo
This website uses cookies to optimise your online user experience. Some of the cookies we use are essential for the site to work.
By continuing to use our site you agree to us using cookies in accordance with our Cookie Policy.
Accept