If this topic fills you with dread, your tracking infrastructure is likely broken somewhere

Dan Layfield

·

Follow

Published in

Entrepreneurship Handbook

·

9 min read

·

Jan 11, 2024

--

8

Listen

Share

I would argue that 99% of companies that are good at developing tech products do these three things:

1. They have clearly defined metrics that they are trying to move
2. They review those metrics regularly
3. They try to determine the link between the work that they release and how those metrics move.

Let me also acknowledge that I have never seen a company with 100% of its tracking set up correctly.

Something is always broken, out of date, or missing. It’s just the reality of tech products, unfortunately.

If this topic fills you with dread, your tracking infrastructure is likely broken somewhere. That’s fine; literally, everyone’s is to some degree.

That said, every subscription company should have a basic hierarchy of metrics, understand what those metrics mean and try to improve them.

Why are Metrics Important

Regardless of the type of product that you run, there are 4 core things that happen in product development:

1. Deciding where to focus
2. Deciding what exactly to ship
3. Measuring the impact
4. Using this to inform what to do next

If you aren’t measuring the impact of your work, you can’t move to step 3 or 4 and play whack-a-mole with new ideas and bugs. Very early in a startup’s development, product development is guided by the founders’ instincts.

The founders have some unique insight into the market, and they use this insight to guide product development and ship a product. They are probably only looking at user sign-ups and revenue and trying to move as fast as they can.

Over time, however, as the company grows, the founders need to spend more of their focus building the company, and the market evolves. This instinctive style of development stops being as effective.

Companies at this point need to switch to a more structured approach, where they have product teams who set goal metrics and then try to ship projects to move these goals.

If you don’t have a clear hierarchy of the metrics that govern your product, it’s too easy to change your goals every quarter or let teams set goals that don’t scale up to the company’s goals.

The biggest danger I see in growing $1M — $5M ARR companies is that they have something that works; they mistakenly break it and don’t realize it for many months.

The best-case scenario is that they figure it out and just lose out on many months of growth. The Worst case is that this kills them.

What Subscription Products Should Be Measuring

This is a basic hierarchy of metrics that the majority of subscription products can use, which breaks into 3 core categories.

1. How well the product is attracting new users
2. How long they stay in the product
3. How they leave the product

Note here that I will only lightly touch on the core product metrics (activation, retention, etc) as those are highly dependent on the product you’re creating.

I will focus instead on the “business” metrics and the revenue the product produces.

I am going to break each group into:

• Primary metrics — these are your top-level metrics that are likely reported across your company and/or to your board. These are things like “Revenue” or “New User Sign Ups.”
• Secondary metrics — sometimes called “operational metrics,” these are the metrics that ladder up to drive your primary metrics. So, if you were trying to improve your “New User Sign Ups,” you might be working on your home page’s conversion rate.

1. How Well the Product is Attracting New Users

This bucket is effectively attracting new users to your product and getting them in the door.

Primary Metrics:

• Volume of New Sign-Ups — e.g., how many people/companies are paying you for the first time this day/week/month, etc.
• Revenue from New Sign-Ups — how much money you made from these sign-ups this day/week/month.
• Conversion Rate to Paid — How effectively does the non-paid stuff convert people to paying customers?
• If this is a direct purchase product, this will increase the conversion rate of people landing on your site to paying users.
• If you have a free product, this is the conversion rate between your free and paid products.
• If you have a trial — This is the end-to-end conversion rate, meaning everyone on your site, how many both start a trial and convert from that trial.

Secondary/Operational Metrics:

They start to vary from product to product, but I’ll list some ones here that are commonly useful.

• Trial Start Rate — this is the number of users that you have registered who see a trial and start it. It’s helpful to cut this metric by time window, so how many do it within 7, 14, 30, or 60+ days of seeing your products?
• Trial Conversion Rate — this is the conversion rate to paying within the trial experience. I would also cut this by the same time window as the trial start rate.
• Checkout Page Conversion Rate — Of everyone who lands on your checkout page, how many successfully pay you? You should segment this data by device size, country, and payment method.
• Payment Button Conversion Rate — of all users who click “pay” on your checkout page, what % of these users actually successfully pay you? You want the highest conversion rate you can here. Segment this by payment method (credit cards, PayPal, etc.)
• Plan Mix — If you have monthly and annual plans, what % of users choose each? Monitor this per month and try to increase the number of people in longer-term plans.

2. How Long Users Stay With the Product

This bucket is how well the product is retaining users. As you’re a subscription product, you always want to increase these numbers.

As noted in other ​articles​, many products have a natural “cap” on retention based on the problem they are solving for users. That said, you should figure out that limit and get retention to that point.

Primary Metrics:

• Lifetime Value — As noted in other ​posts​, this is a very important metric to know; however, it is a very hard metric to use operationally. You want to be moving this number up, but you’ll likely have to define operation metrics to try to move this metric.
• Length of Usage Retention — How long are users returning to the product to perform the core action, typically measured in weeks or months?
• Length of Payments Retention — How long are users paying for the product, typically measured in weeks, months, or (if you’re lucky) years?

Ideally, you are looking at this data on a cohort level. So, for every month of new paying users, you should see how many months they stay around.

Usage retention will likely drive payment retention, so when someone stops using something they will eventually stop paying.

It’s your job to figure out how to keep driving usage of the product.

You should look at both forms of retention both as a table and as a chart. They’ll look something like this.

Operational Metrics

• Payment retention — this is pretty simple: what % of the people who should pay you, do pay you?
• Usage retention — is a lot more complex, and I’ll go into it below.

Most of these metrics are governed by the core product that your problem is solving. This is a framework that I learned in Regorge’s “​Engagement and Retention​” course, which is great.

I can’t go nearly into the depth that this course can, but on a high level, you’re trying to measure the path that users have to “retained”, which means they will keep coming back to your product.

This path goes like this:

• Signed up: User joins the product
• Setup moment: User is prepared to experience the core value prop
• Aha moment: User has experienced the core value prop for the first time
• Habit moment: User has established the habit around the core value prop
• Retained moment: When the user is reliably coming back to the product

Improving engagement and retention is the core job of product teams and is essentially a never-ending process.

3. How They Leave the Product

Users will leave your product in a few ways, and it’s important to measure cancellation according to these groups.

Before we get into metrics, there are two ways that users will leave your product;

• Happily — I bought your product to solve a problem. That problem is now solved, and I don’t need this anymore.
• Unhappily — This product didn’t solve my problem, and I am leaving.

In the example of language learning, if I was paying for a product like Babble to learn Spanish, I either

• I learned enough Spanish that I am happy and no longer need it.
• I didn’t learn Spanish and am giving up.

Within the “Unhappy” bucket, I would break this again into two additional buckets;

• Intentional Cancellation — I went into settings and canceled my account.
• Unintentional Cancellation — Something went wrong with payments, tech or another blocker that prevented me from using this service.

Primary Metrics

• Month Over Month Churn Rate — Of the users that you had last month, how many canceled? When most people talk about “churn,” this is what they are talking about, and this allows you to compare to benchmarks.
• Time Boxed Churn Rate — Of the users who signed up in x period, how many are left months later? Typically, this is 3, 6, or 12 months.

Operational Metrics

• Payment Processing Failure Rate — Of all users with recurring transactions, what % are failing their first payment attempt and reaching your retry process?
• Payment Failure Win Back Rate — Of all users that fail their initial payment, how many can you recover with revenue recovery strategies?
• Churn Appeasement Rate — Of all users that enter your cancellation flow, how many of them take an offer that is presented and don’t churn?
• “Happy” Cancellation Rate — How many users did this product work for, and they no longer require this product? Note that to do this, you’ll need to add this option to your cancellation flow.

So, what do you do with this information?

There is a lot in this post, so don’t get overwhelmed and try to implement everything at once.

1. Define Your Core Sources of Truth

As metrics always break, they help anchor you to the sources of truth for what you care about the most. These are going to be the key lifecycle events that are captured by other systems.

For the average subscription company, these are:

• Page views: for the core page of your site, like home, onboarding, checkout, etc
• Sign Ups: The free (if that exists) sign-ups, the best source of truth is typically email data
• Payments: Typically, your payment system, such as Stripe/Recurly/Braintree/Paddle, etc
• Cancellations: Either the app itself or your subscription-manager

This is especially useful when your tracking data breaks or is suspect. It lets you quickly hand calculate numbers like your free to paid conversion rate by taking your email sign-ups/ your new payments.

2. Validate Data for Primary Metrics

Pick the primary metrics you want to track and ensure that whatever feeds your reporting systems roughly equals the SOT data.

These won’t be exactly equal, but they should be close. So, when you look at your free-to-paid conversion data, it should be roughly equal to a calculated conversion rate.

Then, pick 1–2 operation metrics and do the same gut checks. You will likely find things that are broken, and it’s time to start prioritizing things to fix.

3. Build Basic Dashboards & Monitor Daily

Even though most of these numbers won’t move daily, you’ll be able to draw a relationship between the projects that you see shipping and how numbers move.

You’ll also be able to build your intuition for the seasonality of your business.

The End

Typically, when a company is not monitoring metrics, it is driven by one or more of these reasons

1. You have no tracking set up and/or it is too broken to be usable
2. You don’t have a self-service analytics tool
3. Your team doesn’t have the skills to look at numbers

Figure out which of these are your core problems and work from there.