Measuring MVP's

Measuring MVP

source: Photo by Artur Tumasjan on Unsplash

The question frequently raised when a team is experimenting with a new idea or MVP (Minimum Viable Product) is "what will the returns be and how soon"? In a start-up, both founders and investors ask this question, but they are equally satisfied by trusting the team to do what's right, even if it means not having achievable goals. In larger, more traditional organisations, the question is about being precise about goals and objectives with as little margin for error as possible. We need something in-between.

MVP = Experiments = Innovation

Almost any question can be answered cheaply, quickly and finally, by a test campaign. And that's the way to answer them – not by arguments around a table., Claude Hopkins, "Scientific Advertising" (source: UNHCR)

We all agree that Innovation will cease if we don't experiment. The very definition (Merriam-Webster) of Innovation is "The act or process of introducing new ideas, devices, or methods." This process is the iterative experimentation as established in Agile, Design Thinking and even Lean Start-Up.

Measuring Innovation Experiments

Source: Photo by Girl with red hat on Unsplash

Measuring Experiments

"Length" takes the podium as the most useful in the history of scientific measurements. It was created to maintain a single idea of surveying land for property rights purposes, which led to establishing standards. So the point of any measurement standard is to ensure everyone measuring something can discuss findings in a common language. Experiments in new Products, Processes, or Services are no different today; there must be a way to establish our conclusions in a manner that's both universal and widely accepted. Once measurements are selected, you have to decide how far you want to take them. Measurement depth is where start-ups and traditional organisations disagree; the latter is mainly risk-averse and will require everything measurable to be measured, tracked and reported. On the other hand, start-ups can take a "much ado about nothing" attitude and wing it on the metrics. So what's the better middle ground for Innovation?

Innovation Accounting

"Innovation is a bottoms-up, decentralised, and unpredictable thing, but that doesn't mean it cannot be managed", Eric Ries, author of Lean-Startup.

In his book, The Startupway (2017), Eric Ries dedicates an entire chapter to something called "Innovation Accounting", which evaluates progress when all the metrics typically used in an established company are effectively zero.

I'll try to explain the concept using an analogy of Innovation in a bank; the idea was for a new homeownership product, where you rent from the bank with an option to buy the home in 5 years. Unfortunately, this idea went through the mill of analysis-paralysis on Risk models and ROI's (Return on Investment) before anyone even asked the customer if they would buy the product. Logic would dictate that running multiple experiments with customers to prove the hypothesis early on is a sound idea. Sadly, customer experiments were delayed because of lengthy measurement requirements. Alternatively, what if an MVP with experiments validated an initial assumption that a customer would like the idea? An example of this is Customer Conversion Rate, which measures the percentage of customers who try the product's trial version.

Essentially, Innovation Metrics is about measuring the success of experiments at various stages. Broken down into three levels;

Level 1: Per Customer - Frequently measured metrics per customer like Customer Conversation Rate, Cost per Customer and Referral Rate. You would test various assumptions with a small group of customers at an early stage. As Eric puts it, "its metrics that view customers as a flow through the experiment factory."

Level 2: Growth - Level 1 will tell you that customers like the product but level 2 goes deeper to ask if we can grow the customer base. Its metrics reflecting engines of growth like "stickiness" and "virality". Examples include Site Visits, Orders per Customer and Customer Conversion Rate.

Level 3: Present Value - Done at a later stage in the experimental journey, it's about translating learnings into revenue at present. Net Present Value (NPV)'s are measured for every new experiment performed, such as additional features or tweaking of existing features. It's measuring the magnitude of success in revenue!

Scaling Innovation Accounting

Scaling Innovation

At any given time in an Agile organisation (if such a thing truly exists), there will be multiple initiatives executed by various Scrum teams. These ideas or MVP's probably get implemented over a few Sprint iterations, and in the end, a decision must be made to Persevere or Pivot. The Transformation or Strategy teams would like to compare initiatives on equal terms at a portfolio level.

The comparison is possible by having a clear set of metrics as shown in the three levels of Innovation Metrics. For example, to know if an experiment is going well, you can compare initiatives with Level 1 metrics like Customer Conversion Rates. Once measured often enough, the organisation can establish minimum goals to achieve.

Conclusion

An organisation will have to run experiments to test new ideas. The virtue of scientific experimentation, which is to hypothesise, test, learn and repeat, is no different for Innovation and Agile Teams. So start small and build internal standards on these measurements, but don't overdo it; otherwise, it will kill the very spirit of trying new ideas.