How to avoid KPI psychosis in your organization?
A practical guide to a good measurement culture where numbers don’t replace common sense
We live in a world where we collect data about everything. Think of the data we track on our navigation, our customer behavior, our health, and our company/team/individual performance. Unfortunately, this abundance of data led to a growing prevalence of KPI* psychosis in technology companies.
What do I mean by this? Let’s take the definition of Merriam-Webster for psychosis: “a serious mental illness characterized by defective or lost contact with reality”. This means KPI psychosis is a state of mind where a company has dysfunctional contact with its reality and makes decisions only based on numbers.
Human subjectivity
So why are companies so obsessed with collecting data to guide decisions? Data is seen as a way to fight biases, and when it comes to biases humans lead the way. We are so much biased that the people who collected the sorts of biases that feature human cognition received a Nobel prize for their work.
Just to give a glimpse: we take information that is easiest to gather (availability bias), often that is heavily affected by the latest info we got (recency bias) and when we make conclusions we tend to overgeneralize (halo/horn effect) with more confidence than it was justified by our lintels (overconfidence bias). On top, we are typically stubborn and seldom change our opinions and worldview (anchoring bias, confirmation bias). This is especially problematic when changes happen slowly and in small steps. For instance, the way software development productivity changes in our team.
KPI objectivity
Enter KPIs as the objective truth. Free of subjectivity, perfect, right?
Not so fast. In fact, often our data collection and measurement are also biased by us (e.g. algorithmic bias). And even if that is not the case, unfortunately, KPIs suffer from tunnel vision: they measure what is measurable, while not necessarily all aspects of the situation are. Albert Einstein put it brilliantly:
“Everything that can be counted does not necessarily count; everything that counts cannot necessarily be counted.”
This results in perverse motivation in many organizations, where people have to choose between doing their job well (broader reality) or getting promoted for meeting the KPIs (tunnel vision). And that’s exactly the KPI psychosis I described above.
The trivial solution
It takes a different mental framing to fix this situation: KPIs should be used in combination with human intuition to enable optimal decision-making. So not just intuition or data, but a constant back and forth of making (i.e. intuition) and testing (i.e. data) hypotheses.
This leads us to the importance of explaining what a KPI actually indicates — and not only what it measures. This is often underestimated. Let’s take the example of Time To Last Byte (TTLB), a common metric of service quality that we also use at Promaton. It measures the time between the request sent by the client and the last byte of the related response from the server. In business terms, this is how much time your customer has to wait to get what they want. It’s easy to calculate and you can easily see how optimizing the runtime of server-side processing would result in improvement in the TTLB. However, you never actually want to improve TTLB per se, what you want to optimize is user experience. And understanding this is essential.
If you know that you care about user experience, it can easily happen that you start exploring first the user segment that suffers most from high TTLBs. Maybe you are going to see that these users have something in common, e.g. all those requests are from an area with poor internet connection. Now your course of action will not be optimization of server-side processing, but something else (e.g. deploying in that region or on-premise for those customers). And most importantly maybe this action could easily lead to a smaller overall TTLB improvement than optimizing the most time-consuming process in the backend, yet you clearly improved the user experience more.
Getting KPI optimization right
KPIs are not born perfect: they often require lots of conceptualization, trial-and-error learning, or redefinition before you find comfort in relying on them. So you work on reaching your objective and while doing so you constantly check both what your KPI shows and also how much you can rely on it. Let’s say you want your product to be useful for your users in the long term. You start measuring retention, but there are so many conceptual challenges that you’ll have to resolve before your KPI would be really correlated with the long-term usefulness of the product; e.g. things like who a user is and what long-term means. This is the process called KPI optimization.
KPI optimization is not about selecting one such value, telling them it’s bad, and telling your people to optimize it (e.g. our churn is 5%, it should be 2%). The problem with that is that it’s going to discourage people from asking up to which level that number represents reality. Reality both from the validity (it measures actual user loss) and from the reliability (it reliably changes when user loss happens) point of view.
And finally, people are pretty good at number hacking: when I was about ten years old, I learned that secretly rubbing the mercury container of the thermometer helps to increase my “measured” body temperature. This was the KPI my mother used for deciding if I was (too) sick to go to school. Learning this hack helped me to score quite some extra holidays, but surely didn’t help our family’s objective of keeping kids at home (only) when they are sick.
To summarize, here are my recommendations:
- Use KPIs in combination with intuition for optimal decision making
- Always keep in mind your actual objective instead of only defining the KPIs
- Continuously reflect on the reliability of your KPIs and revise them if necessary
Btw, do you want to work in a company that does not suffer from KPI psychosis? Check out our open positions here: https://careers.promaton.com/
*: I use the acronym KPI (key performance indicator) instead of “data” because I consider it advantageous to have the word “indicator” always in mind but think of all metrics, numbers, indices, key results etc.
