The Trouble With Metrics

The trouble with metrics is simple. They are quantitative and they measure results. Taken alone, they cannot measure true transformation over temporal change. In general, metrics, measuring change from one point in time, to another point in time, do not, and cannot, measure the kind of change that leaders institute today, and be vigilant in sustaining for tomorrow.

Result-based metrics are used to measure success or failure. This is a good thing; it helps us set expectations. It quantifies goals that otherwise may be subjective. However, in the effort to drive transformation, true transformation from one state to another, metrics that measure only change, can be misleading.

Transformation and Change

To fully understand the trouble with metrics, compare change with transformation. Change is a delta, or in mathematical terms, it is the incremental change in a variable. A “variable,” by definition can change up or down over time. In business, we use spreadsheets to measure change. We compare a quantity before, say at the beginning of a quarter, with a quantity at the end of a quarter. The difference between the comparison and the expected result tells if we have achieved our goal. The problem is that the following quarter, change can be reversed putting us right back where we started.

Transformation, on the other hand, is a complete state change, a metamorphosis. There is no turning back. The factors that were causing one habit, behavior, or outcome, must be completely replaced with processes that no longer allow the former state to exist. The strategies that drive a transformation involve change, to be sure. In total, along the transformation path, we cannot always easily measure the change with quantitative metrics. Consider that some business goals require combining change in terms of culture, attitude, satisfaction, delight, customer-centricity, brand recognition, user experience, and other service-levels that are largely subjective to measure.

Statistics and Distributions

Right about now, some are saying that “yes, we can measure those; we just do surveys of people giving them a 1 to 5 option to answer their perception on those factors.” However, that makes the point that they are subjective in terms of the perception of the individual being surveyed, at the time. And then, subjective to the interpretation of the analyst compiling the data. OK, I will agree that given statistical methods, we can develop a distribution and standard deviation and come close to a good metric for these factors. However, when distributions, or populations, shift, they generally shift more permanently like a metamorphosis or transformation.

Statistical methods help us know the degree to which we have driven sustained transformation, or just temporal change.

The Trouble with Metrics

The real trouble with metrics is that leaders often erroneously measure transformations in terms of single points of change, and in too short of a timeframe. Without context of those other factors, like culture, attitude, delight, and so on, we reward results, not true and complete transformations. We like doing this, as it is human nature and it makes leaders feel good. People in our organizations also learn how to game the system, and will tend to find ways to measure, and be rewarded for temporal change, over sustained transformation. This is understandable as the latter is harder to achieve and generally more complex to manage.

True transformation also takes longer. Strong leaders know that sustained transformation starts with the factors that are the most difficult to achieve and measure, like culture, attitude, delight and so on. They also require that we measure these factors throughout our business ecosystem. That means we must measure it with our employees, our customers, analysts and press, and even our competitors, and more based on the complexity of the ecosystem.

Leadership and Metrics

The next time you develop metrics, remember the trouble with metrics. Consider the vision you have of the future state. Develop a strategy of compounding steps to get there. Whether it is for annual company goals, or a project goal, ask first if it’s just a milestone achievement, or if you are after a sustained transformation. If your goal is the latter, then as you develop your metrics, consider the less measurable, and perceived aspects that exist. You will want to work on those areas much more than the numeric change that shows up in the spreadsheet.

The measure of sustained transformation is best done with statistics, and over periods of time. We’re moving toward recent rolling real-time metrics that some call sensitivity analytics, mood analytics, or tone. Either path you choose, recognize the trouble with metrics, and take caution, accordingly.

Leave a comment and share your experience with transformation and change.

Image Credit: by Pauk, from Wikimedia Commons, via Flickr upload bot, 2010.