Improving Retention with Audience Surveys
Welcome to a fresh edition of Growth Croissant! 🚀 🥐
I’m Reid, your host on this journey. I’ve been lucky to be part of awesome teams that launched and grew Hulu, Crunchyroll, and HBO Max. A few years ago, I founded Yem to help individuals and small teams build their own media businesses. Yem was acquired by Substack, where I currently work on growth.
Growth Croissant will be an evolving home for our learnings, painful lessons, and frameworks for making hard decisions. My goal is to deliver you a comprehensive and actionable guidebook on how to grow your business.
Hello friends,
We’re continuing our multi-post series on improving retention — one of the most meaningful levers we have to improve customer lifetime value (“CLV”) and grow revenue. In our first post, we went through a few growth tactics that can help chip away at cancels; in our last post, we discussed improving retention by connecting new subscribers with your value proposition through better onboarding.
Another way to improve retention is to significantly increase the degree to which you’re solving your customer’s problem. It’s not easy to improve retention this way: minor tweaks or improvements are unlikely to have a noticeable impact on retention. At the risk of being abstract, we need to aim for a 10x improvement in how much we’re solving our customer’s problem.1
There won’t be any obvious or realistic ways to 10x the value you provide for all your subscribers. We must focus on our core audience. It can be uncomfortable to neglect parts of your audience or turn your attention away from expanding your audience. But both are essential if we want to have a noticeable impact on retention through better problem-solving.
If you don’t see a pathway toward solving your customer’s problem in a much deeper way, you may need to accept the zone of retention you’re in. As long as your retention is roughly around your peers, it may be better to focus on expanding your audience and bringing in new subscribers.
If your retention is lower than your peers, and you’re seeing challenges with audience growth, it might be time to go back to the drawing board and rethink your product. I’ve been down this road several times — it sucks. BUT, it’s much better to find this kind of thing out sooner than later. The last thing you want to do is waste your most precious resource — your time — working on the wrong thing.
I won’t be able to tell you who your core customer is, the problem you’re solving for them, and how to 10x the degree to which you’re solving that problem. But let’s talk through one of the ways we may be able to chip away at answering these questions: getting feedback from your customers through surveys.
Audience surveys
At Yem, one of the things we did right after starting to work with newsletters was run an audience survey. There’s no better way to get an understanding of your audience and why they use your product than reading through tons of survey responses. Let’s go through how we ran audience surveys in case it sparks any ideas for you.
We would send a dedicated one-time email asking subscribers to complete a survey, split into one version for free subscribers and another for paid subscribers.2 We would also apply a few filters, ensuring recipients had engaged with the newsletter recently and had been subscribed for enough time to experience the product (e.g., signed up more than 30 days ago).
With a dedicated email and these filters applied, we would see 5% to 10% of email recipients finish the survey (20% for paid subs and 5% for free subs). To encourage a higher rate of responses, we would frame the request as a way for us to improve the product for the recipient. Here’s an example of an email we would send to ask subscribers to fill out a survey:
For the survey, we would use Google Forms or Typeform. Here’s a sample survey — please feel free to duplicate it and make it your own. Each survey was different, but we always asked some version of these questions:
The first question was helpful in generally (but certainly not perfectly) segmenting responses into your core audience vs. non-core audience. The “main benefit” question drills into the value subscribers get from your product — the essential question of the survey and, for the purpose of improving retention, where you should spend the most time.
For paid publications, we drill further by asking paid subscribers what encouraged them to upgrade to paid and asking free subscribers what would get them to upgrade to paid. If you’re considering certain benefits (e.g., community features, live events, courses, paid-only posts), the survey is an excellent opportunity to get a signal on which ideas are most enticing.
Here’s an example from a finance newsletter, showing a summary of responses to questions on primary benefits, reasons for paying, and how we could improve the product them. To improve retention by 10x’ing the value for existing subscribers, we would need to focus on the bottom section. As you can see, these ideas often rub against new subscriber growth, like increasing price or introducing a more expensive tier.
Other Tidbits and Tactics
It can also be valuable to ask demographic and work-related questions, especially if you’re considering sponsorships.3 This qualitative info helps you tell a story around your metrics (e.g., email list size, open rate, click-through rate), making your audience more compelling to the right partner. Here’s an excellent example from
, emphasizing his audience is affluent, highly educated, and in leadership roles responsible for purchase decisions (ideal for B2B/SaaS products).
Another question we usually asked was a flavor of “What other newsletters or podcasts do you enjoy that are like Growth Croissant?”. Some responses gave us a sense of who we were competing against and how we may be able to differentiate our product for the core audience. Other times, we found complementary products representing partnership opportunities (e.g., a newsletter writer being a guest on a relevant podcast).
For your first time running the survey, except for the first question, I recommend open-ended responses the first time around. Once you have a good handle on how your audience will likely respond, you can transition to multiple-choice, though I’d always include an “Other” option to hear what people want to tell you.
While we mostly did one-off surveys, having a survey email toward the end of an onboarding series can be valuable. Seeing how responses shift over time can help you gauge how your audience evolves and track efforts aimed at delivering more value.
If you’re doing a one-off survey, it could be worth thinking about how to share the results with the audience. Below are a few examples: WaitButWhy sharing results from the audience taking the Myers–Briggs personality test, and another from Shaan Puri sharing survey results with subscribers (subject: “your survey results”). Sharing results can strengthen the sense of community and encourage higher participation in future surveys (or similar activities).
Cancel surveys
Cancel surveys can be a treasure chest of feedback and a vital ingredient in our effort to improve retention. Responses are usually quite candid and come from folks that have paid for your product but weren’t quite satisfied.
From an earlier post, it’s usually best to avoid requiring subs to complete a survey before they cancel. As an optional survey after a subscriber cancels, you will still get plenty of responses, and the quality of feedback will be much better. Moreover, you won’t harm your relationship with your subscribers, increasing the likelihood that they’ll return as paid subscribers.
The core question of any cancel survey is “What’s the primary reason you’re canceling?” Respondents usually choose from a list of potential reasons. Here are a few examples from Netflix and Hulu:
In the streaming world, we saw up to 50% of respondents choose some flavor of “price” as the reason they canceled, which always felt like an easy-out response. Our services were usually priced below most competitors (even HBO Max was roughly in-line with Netflix’s average revenue per membership). Price alone usually wasn’t the issue — folks weren’t getting enough value relative to what they were paying.
While we weren’t considering lowering our retail price4, we were certainly interested in delivering more value. At Crunchyroll, we eventually removed “price” as a cancel reason and saw responses shift to more actionable cancellation reasons. Here are a few examples of more actionable cancel reasons:
“Didn’t have a certain series or movie” — buying and making content was our largest expense, and this feedback helped guide programming-related decisions.
“Switching to free, ad-supported service” — when this would trend upward, we would consider whether we were overly generous in the value given to free subscribers and whether we should shift more value behind the paywall.
“Video quality issues” — a valuable proxy for the severity of any issues with site or service stability and how those issues impacted retention.
Compared to generally more casual free subscribers, cancel surveys allow you to get feedback from people that have a solid understanding of your product. As such, we would ask more detailed questions on how we could deliver more value, specifically:
What features would you like to see added?
What movies, series, or genres would you like to see added?
For features, rather than just having “device availability” as the cancel reason, we’d know most folks want us to be on Nintendo Switch or PS Vita. It would also help us prioritize exploratory features (e.g., co-viewing with friends, offline viewing) or features unique to our service (e.g., making it easy to read manga or switching between dubbed and subbed in the video player).
On the programming side, many responses would be the usual suspects of classic anime titles or the hit show of that quarter. But many Crunchyroll subscribers wanted more non-anime shows, specifically adult animation (think Family Guy, Rick and Morty, and BoJack Horseman). We used these survey responses as a key part of our pitch to invest $100 million and a ton of effort into building VRV.
Getting more detailed responses can also help future reacquisition efforts. If we launched on Nintendo Switch, we could message everyone who mentioned Switch in their cancel survey. If we picked up a frequently-requested show (e.g., Dragon Ball), we could let them know it’s now available. These loops helped strengthen bringing back former subscribers as a key growth channel.
Summary
Surveys are an excellent way to better understand your audience and the value they get from your product. With that knowledge, you can explore ways to improve retention by 10x’ing the degree to which you’re solving their problem. But beware: retention is incredibly difficult to improve, and this pathway is no exception — it will likely take ongoing effort over a long period to drive any noticeable impact.
Please share your own stories! Have you ever run an audience survey? What surprised you? How did it help you?
As always, thanks for reading,
Reid
The “10x improvement” can be ambiguous and more of an art than science.
To make it more concrete, let’s imagine we’re Netflix, and our mission is to “entertain the world.” Earlier on, let’s imagine our north star metric is hours watched per paid account. If we’re currently at 100 hours watched per paid account per month, plotting a pathway to 110 hours won’t do much to retention. We need a plan to get to over 200 hours or something where we’ve made profound progress toward solving our customer’s problem.
It’s hard to have a blanket statement on what a 10x improvement means, but just trying to convey that we need to go way beyond incremental changes to impact retention.
If you don’t want to send a dedicated email asking subscribers to fill out a survey, we’ve seen folks prompt subscribers to fill out surveys in the header or opening of regular posts.
We always made the surveys anonymous and avoided hyper-detailed questions.
We thought about offering lower-priced subscription tiers (ad-supported or with less content) to keep or win back folks that were more price-sensitive.
thanks for your insights, Reid 🙏🏼
This is so interesting. Until now I was very survey shy as I felt that they can often not really make a difference, your Substack has started a rethink, thank you