page updated on July 18, 2015

Measure, Experiment, and Iterate Based on User Engagement

Eric Ries's book The Lean Startup suggests an experiment-driven, iterative approach to discovering the right customers, products, and business models for startups. His oft-stated suggestion is to measure the right things, then analyze those measurements to devise experiments to improve the business strategy. (This is the scientific method and empirical reasoning applied to business processes.)

One such experiment is cohort analysis.

Cohort Analysis

The technique of cohort analysis divides users into several distinct groups (cohorts) based on specific and measurable actions. Each cohort represents a portion of the group as a whole. A cohort graph plots those cohorts and those percentages over time, so that you can see how those cohorts grow and shrink relative to each other over the course of your measurements.

Good analysis requires you to identify these cohorts carefully. By definition, a cohort is a group; by implication, everyone in a cohort has something in common. A thoughtful analysis will resemble a sales funnel. Your goal is to improve the percentage of users who travel through each stage of this funnel, because each stage represents greater user engagement and, presumably, more revenue.

Common examples of cohorts include users reached through a specific campaign, users who exhibit a specific threshold of use, and users with distinct demographic characteristics.

Graphing these cohorts over time allows you to see the effect of changes and specific experiments intended to improve user engagement. Does simplifying the sign-in process help convert visitors into registered users? Does changing the text of the pricing plan lead to more paying customers?

Using Cohort Analysis Effectively

Two obstacles may present difficulties to using cohort analysis effectively. First, you must instrument your processes to measure user engagement and conversion. The more data you collect, the better. If you overlook a customer segment of any kind, you've lost actionable data—clearly a substantial problem to overcome! Choose each metric to measure with care.

Second, you must choose your cohorts well. Dividing data by the level of engagement is essential to producing effective reports, even if the line between "this customer has spent $90 this month" and "this customer has spent $100 this month" may be thin. The better you define your cohorts, the better you can analyze them.

The results may sometimes be horrifying; the change you expected to have a positive effect may have had the opposite effect. Worse, it may have had no effect at all. This is a victory for cohort analysis—if you have the discipline and the courage to re-examine your assumptions, make a new business hypothesis, and repeat the cycle of change, measure, and review.

Only when you can describe the questions you're trying to answer and form a statistical model of how user behavior supports or negates your hypothesis do you have the framework with which to make sensible decisions.

Success is no guarantee, especially when your business is searching for a business model, but cohort analysis offers the opportunity to measure your progress toward your goals. The better your organization of thoughts and applications, the stronger your feedback cycle and the more you can learn from cohort analysis.