Tag Archives: metrics

The worship of the yardstick

I just read this article (the text of a presentation, really): James L. McConaughy, The worship of the yardstick, Educational Review, Vol. 55, No. 3, March 1918, pp. 191-200

(Yes, that is: nineteen eighteen.)

How did I get there? First, I read The Tyranny of Metrics, which cited Education and the Cult of Efficiency, which cited this article. (I like to follow the citations because I'm a Super-Fun Guy.) All three of them, for me, have ineluctable titles—tyranny, cult, worship—that leave little doubt about which direction they're approaching from. Moreover, they're all related to a problem that drives me crazy nearly every day: the unquenchable, unquestioning love for measurement at work.

It's out of balance. On a recent project, metrics (the word that means "measurement" at work) accounted for nearly 50% of our group's time—so much so that it was necessary at the culmination to bring in extra people to actually do the work. Here's a natural question to ask: did the work get done? Of course not. Mission Accomplished.

Here's where I drive people crazy: I push back. But I think I'm misunderstood. Because I'm such a smooth talker when I get worked up. I'm not against metrics or measurement—I use measurements unprompted in my own life—but rather the mindless way in which they're applied. I get upset when the project bureaucrat asks why a document is only 84% done when the plan clearly states that it should be 87% done. It's like someone knocking on the bathroom door, upset that you're only 84% done taking a dump when you've clearly been in there long enough to be 87% done. What? Anyway. It's not a real measurement. There is no such thing as an 84% done document. It's a heuristic. It's not real. But measurements become very real when everyone pretends they're real. And measurements become the standard against which rewards and punishments are meted out. After a while people forget that the numbers aren't real, and they don't think very deeply about what the measurements mean. And they don't have any feeling in their hearts for the product itself, which is just some sort of abstract byproduct of the process.

I will die on this hill.

Here's one other worry that I have about the cult of metrics. Whenever I get saddled with some new measurement, I try to find a way before the end of the day to automate the calculation. I get to do a little bit of thinking while setting up the algorithm, and it costs me less time over the long haul. Win-win. But what becomes of the people who every day, every week, etc., devote non-negligible amounts of their time to manually calculating things that are easy to replace with a program? I guess we'll find out.

Data fictions

Those who gather and interpret such aggregate data understand that there is a certain fictional and arbitrary quality to their categories and that they hide a wealth of problematic variation. Once set, however, these thing categories operate unavoidably as if all similarly classified cases were in fact homogeneous and uniform.

—James C. Scott, Seeing Like a State: How Certain Schemes to Improve the Human Condition Have Failed, Chapter 2: Cities, People, and Language (notes)

This is one of the aspects of work that I really try to press on younger engineers: metrics are just metrics. Especially in organizations that live and die by earned value management, it's easy to forget: the map is not the territory. Metrics are just metrics. People get hung up on the difference between, e.g., a specification being 82% done and 83% done. It doesn't matter. It's the trend that matters. The specific number is just a heuristic. It has no meaning as a precise measurement like the outside temperature or your tire pressure. It's just a formula that I applied to a database based on a tailoring of a company-wide process instruction with weights that I applied based on experiences (read: bias).

But the real problem is that once you put that heuristic into place—that definition or interpretation of the data that everyone knows at the onset is a useful fiction—it tends to become Law. The old people forget or move on, and the new people never knew. Whether out of fear or custom, one thing is certain: the Law must be upheld. What happens to the person who tries to remind everyone that, hey, wait, that's not a law, that's just a helpful fiction? Enjoy wearing that big scarlet A on your chest, Asshole, it matches your eyes.

Metrics are like your speedometer reading: glance at it when you need to know, but don't stare at it or you'll really be in trouble.

Anyway, Matthew Mcconaughey does it better:

Fugazi, Fugazzi. It's a wazzy, it's a woozy. It's [whistles] fairy dust. It doesn't exist. It's Neverlanded. It is no matter. It's not on the elemental chart. It's not fucking real.