source: Photo by Artur Tumasjan on Unsplash
The question frequently raised when a team is experimenting with a new idea or MVP (Minimum Viable Product) is "what will the returns be and how soon"? In a start-up, both founders and investors ask this question, but they are equally satisfied by trusting the team to do what's right, even if it means not having achievable goals. In larger, more traditional organisations, the question is about being precise about goals and objectives with as little margin for error as possible. We need something in-between.
MVP = Experiments = Innovation
Almost any question can be answered cheaply, quickly and finally, by a test campaign. And that's the way to answer them – not by arguments around a table., Claude Hopkins, "Scientific Advertising" (source: UNHCR)
We all agree that Innovation will cease if we don't experiment. The very definition (Merriam-Webster) of Innovation is "The act or process of introducing new ideas, devices, or methods." This process is the iterative experimentation as established in Agile, Design Thinking and even Lean Start-Up.
Measuring Innovation Experiments
Source: Photo by Girl with red hat on Unsplash
"Length" takes the podium as the most useful in the history of scientific measurements. It was created to maintain a single idea of surveying land for property rights purposes, which led to establishing standards. So the point of any measurement standard is to ensure everyone measuring something can discuss findings in a common language. Experiments in new Products, Processes, or Services are no different today; there must be a way to establish our conclusions in a manner that's both universal and widely accepted. Once measurements are selected, you have to decide how far you want to take them. Measurement depth is where start-ups and traditional organisations disagree; the latter is mainly risk-averse and will require everything measurable to be measured, tracked and reported. On the other hand, start-ups can take a "much ado about nothing" attitude and wing it on the metrics. So what's the better middle ground for Innovation?
"Innovation is a bottoms-up, decentralised, and unpredictable thing, but that doesn't mean it cannot be managed", Eric Ries, author of Lean-Startup.
In his book, The Startupway (2017), Eric Ries dedicates an entire chapter to something called "Innovation Accounting", which evaluates progress when all the metrics typically used in an established company are effectively zero.
I'll try to explain the concept using an analogy of Innovation in a bank; the idea was for a new homeownership product, where you rent from the bank with an option to buy the home in 5 years. Unfortunately, this idea went through the mill of analysis-paralysis on Risk models and ROI's (Return on Investment) before anyone even asked the customer if they would buy the product. Logic would dictate that running multiple experiments with customers to prove the hypothesis early on is a sound idea. Sadly, customer experiments were delayed because of lengthy measurement requirements. Alternatively, what if an MVP with experiments validated an initial assumption that a customer would like the idea? An example of this is Customer Conversion Rate, which measures the percentage of customers who try the product's trial version.
Essentially, Innovation Metrics is about measuring the success of experiments at various stages. Broken down into three levels;
Level 1: Per Customer - Frequently measured metrics per customer like Customer Conversation Rate, Cost per Customer and Referral Rate. You would test various assumptions with a small group of customers at an early stage. As Eric puts it, "its metrics that view customers as a flow through the experiment factory."
Level 2: Growth - Level 1 will tell you that customers like the product but level 2 goes deeper to ask if we can grow the customer base. Its metrics reflecting engines of growth like "stickiness" and "virality". Examples include Site Visits, Orders per Customer and Customer Conversion Rate.
Level 3: Present Value - Done at a later stage in the experimental journey, it's about translating learnings into revenue at present. Net Present Value (NPV)'s are measured for every new experiment performed, such as additional features or tweaking of existing features. It's measuring the magnitude of success in revenue!
Scaling Innovation Accounting