Measure, Experiment, and Iterate Based on User Engagement

Eric Ries's book The Lean Startup suggests an experiment-driven, iterative approach to discovering the right customers, products, and business models for startups. His oft-stated suggestion is to measure the right things, then analyze those measurements to devise experiments to improve the business strategy.

One such experiment is cohort analysis.

Cohort Analysis

This technique divides users into several distinct groups (cohorts) based on specific and measurable actions. Each cohort represents a portion of the group as a whole. A cohort graph plots those cohorts and those percentages over time, so that you can see how those cohorts grow and shrink relative to each other over the course of your measurements.

Good analysis requires you to identify these cohorts carefully. A thoughtful analysis will resemble a sales funnel; your goal is to improve the percentage of users who travel through each stage of this funnel, because each stage represents greater user engagement and, presumably, more revenue.

Graphing these cohorts over time allows you to see the effect of changes and specific experiments intended to improve user engagement. Does simplifying the sign-in process help convert visitors into registered users? Does changing the text of the pricing plan lead to more paying customers?

Using Cohort Analysis Effectively

Two obstacles may present difficulties to using cohort analysis effectively. First, you must instrument your processes to measure user engagement and conversion. The more data you collect, the better. If you overlook a customer segment of any kind, you've lost actionable data. Second, you must choose your cohorts well. Dividing data by the level of engagement is essential to producing effective reports, even if the line between "this customer has spent $90 this month" and "this customer has spent $100 this month" may be thin.

The results may sometimes be horrifying; the change you expected to have a positive effect may have had the opposite effect. Worse, it may have had no effect at all. This is a victory for cohort analysis—if you have the discipline and the courage to re-examine your assumptions, make a new business hypothesis, and repeat the cycle of change, measure, and review.

Success is no guarantee, especially when your business is searching for a business model, but cohort analysis offers the opportunity to measure your progress toward your goals.