ITIL strongly recommends that organizations establish a measurement framework with relevant metrics at the technology, process and service level that provide a comprehensive and overall view of organizational performance. There are two types of metrics, quantitative and qualitative.
Quantitative metrics are often counts of things. Some examples of quantitative metrics at each of the levels that ITIL recommends include:
- Technology — The percentage of disk space that is used at a point in time on a disk array
- Process — The number of failed changes per week
- Service — The number of new customers that purchase a service per month
These examples are purposefully simple, but they illustrate the point; quantitative metrics are objective.
Qualitative metrics, on the other hand, seek to measure subjective things. Some examples of qualitative metrics at each of the levels that ITIL recommends include:
- Technology — Whether or not the IT staff prefers one disk type over another
- Process — Customer perception of the effectiveness of the change management process
- Service — Customer satisfaction with the outputs of a service
Qualitative metrics have a subjective aspect.
ITIL recommends that service providers establish a measurement framework that includes an effective mix of both quantitative and qualitative metrics.
Qualitative metrics are often difficult to collect and even if they are collected are often impacted by things irrelevant to the metric itself. For example, in every class that we deliver, we offer students a qualitative survey. However, one never truly knows whether the results of that survey are purely attributable to overall class performance. Rather, the qualitative metric could have been affected by something else that happened unrelated to the class.
One of the worst experiences I’ve ever had with a qualitative metric dates back to the mid to late-1990’s. I worked for an organization that would annually assess the performance of measurement using a very subjective list of 5 questions. In this survey, we were asked to rate the performance of management on a scale of 1 to 5, with 1 low performance and 5 indicating high performance. Of the 5 questions, 4 of them were fairly useful in gauging the performance of management. However, one of the 5 questions was the worst possible subjective question. As I recall, the question was:
“Do you have a best friend at work?”
Every time I find someone who is aware of this survey they have exactly the same feeling about the question that I have and roughly the same experience.
In my case, my employer received typically low scores for this question. Employees thought that the question was nonsensical and responded as such. Once the survey results were returned, we were required to attend a number of meetings and sessions to explain why we were wrong about how we had responded to the question. This explanation, review, and follow-up occurred at significant expense in terms of time and money to my employer.
Then, when the survey ran the next year, despite all of the pointless explanations and follow-up, the scores were just as low as they were previously. Rather than change the question, more time and effort was spent explaining why employees didn’t understand the question and what they were actually trying to assess.
I believe there’s a quote from Einstein that’s relevant here. Something about doing the same thing and expecting different results.
As it was explained (repeatedly), what they were attempting to assess was whether or not there was someone at work that employees felt they could confide in. That is a very different question than whether or not someone has a best friend at work.
If you have to spend time explaining what your question really means and what you were actually trying to measure, guess what? Your question isn’t measuring what it intends to. That’s one of the issues with qualitative metrics; it’s sometimes difficult to ensure that they are actually relevant to what we’re attempting to measure.
Qualitative metrics are both necessary and useful. However, sometimes low performance of a qualitative metric simply means that your method of collecting that metric is weak.
The funny thing about the “best friend at work” question is that I still encounter this from time-to-time. In the past year I worked with no less than 3 organizations that use a similar question to assess management performance. According to individuals in those organizations, it continues to be at best a puzzling question.