Lies, Damned Lies, and KPI – Part 3: Countermeasures

Kid with Glasses — How to measure good KPIs

There is an inflation of key performance indicators (KPIs) in industry. In my last posts I have explained how KPIs are often wrong, and why bad and fudged KPIs are a huge waste. Yet, you cannot really run a larger corporation without KPI. In this post I will finally give some advice on (1) what you need to do to measure good KPI, and (2) how to avoid fudged KPI.

How to Create Useful KPI

Before we get on how to avoid KPI fudging, let’s first discuss what you need for a good KPI.

Measure Only What You Absolutely Need

The first suggestion is to measure only the KPI that you really need. This sounds like a no-brainer, but unfortunately, in many organizations it is not. Just count for one moment how many KPIs there are in your organization, or how many you measure yourself or with your team. If you still think this is a good number, then ask yourself, How many did I use to initiate improvement projects or warned you about upcoming trouble?

Hence, make sure any KPI you measure is actually relevant for the success of the organization. In industry parlance, these are also often called critical success factors (CSF). A plant measuring 45 different logistics KPIs (actual example) cannot possibly have all of them relevant. In any case, nobody really looks at 45 different KPIs. While it sounds counter intuitive, less is more for KPI.

Use Temporary Measurements for Projects

Vector Computer Repair with Barrier — Temporary only

Sometimes you need more detailed KPI for projects. Depending on what you want to improve, you would need to measure the current state and also the situation after the project is completed. For this some additional KPI may be useful. There is, however, a tendency to keep these KPIs around even after the project is completed. Make sure that once the project is completed, the measurement of the KPI also stops, or at least their necessity is reviewed. Please note that, for me, completion follows the PDCA (Plan-Do-Check-Act) approach, and the completion of a project should include some time after the changes to see if your system actually improved.

Occasionally Clean Out No Longer Needed KPI

Good housekeeping

Despite good efforts to prevent an accumulation of KPI, chances are that over time new KPIs are added. Please make sure that the existing KPIs are regularly checked and old KPIs are thrown out.

Usually, when you want to throw out a KPI, you will see some resistance. People dislike change, and there is probably at least one person who wants to keep this KPI (especially if that person is not the one doing the work of measuring it). Depending on the person resisting, it is worth eliminating the KPI anyway.

Cascading KPI Structure

Ideally, the KPI should be part of a cascading structure throughout the organization. It should be possible to aggregate KPI at the bottom-level groups into one KPI for the department. Department KPI should be aggregated into one for the plant, which then are aggregated into a business unit and eventually into a KPI for top-level executives.

Of course, this point clashes somewhat with the suggestions above. If you throw out a KPI that is part of a cascade, then the entire cascade may collapse, which will not please the higher-ups, and hence may be unpleasant for you. Here you have to find a trade-off between the cascade and cleaning out KPI.

SMART KPI

Term coined by management guru Peter Drucker. In his view good KPI have to be:

Specific
Measurable
Attainable
Relevant
Time-Bound

Especially “4: Relevant” is in my view important to avoid waste, although the other four are also highly recommended.

Countermeasures against KPI Fudging

The above discussion helps you on establishing useful KPI. However, even with good KPI there is always the temptation to fudge the numbers rather than do the real work of improving the system. Some will fall into that temptation. However, a false KPI is even worse than no KPI. The best situation of course is to know (having a good valid KPI). Worse is not knowing, but being aware of not knowing (having no KPI). Worst, however, is not knowing, but believing that you know (believing a KPI is true, while it is actually wrong). Hence, if you go through all the trouble of creating and measuring a KPI, make sure that it is correct.

Keep Measurements as Simple as Possible

The simpler the measurement is, the harder it is to fudge. Although, in general it is not hard to fudge KPI. Even relatively simple KPI like OEE or delivery performance are regularly fudged (see one of my favorite posts on the Top Three Methods on How to Fudge Your OEE). However, if the measurement is simple, it is easier to debunk such fudging.

An example for a measurement that is complex is usually anything related to product cost, or money in general. If the product is too expensive, simply assume lower warranty-related expenses, or tweak some other assumptions, and the very same product is miraculously getting cheaper, at least on paper. The warranty cost has the additional benefit that the person in charge will likely be somewhere else when these costs pop up a few years later.

Another example: Do you know a big government project like building an airport (Berlin, Germany), buying a new aircraft carrier (USA), or anything similar? The final price tag is usually between twice and ten times the original price tag due to … unforeseen … cost overruns. Same things happen in industry if you do not take care. However, you probably need cost estimators. This leads us to the next point:

Verify, Verify, and Verify Again

Verify your KPI. Depending on the number of KPIs, you of course will be unable to verify them all every time. But you should make every effort to check them every now and then. There are different ways on doing that.

Do the KPIs match other KPIs? For example, if you are being told delivery performances, check occasionally with your larger customers if they also measure your delivery performance. If your guys claim that they are 98% on time in full, and your customer sees only 63%, then one side must be wrong. Even if your customer does not measure your delivery performance, you could tell them the percentages and ask for the customer’s comment. (Note: Laughter means that you are probably not as good as someone wants you to believe.) Such cross-checks are often possible.

Can you check in person? Visit plants or sites, and try to see if the claimed KPIs seem to be possible. If a machine is claimed to have an OEE of 93%, stop by to see if it is actually running. If they are proud of their low inventory, look at the inventory. It is easy to estimate the number of pallets and combine this with the observed material consumption to estimate the inventory reach? If they claim a wonderful Just-in-Time delivery, ask the operators. If they are telling you about their wonderful leveling, get the leveling pattern and compare afterward what really was produced (see for example The Folly of EPEI Leveling in Practice). In any case, do not announce your visit or check, but just pop in and check. Oh, and read my article on How to Misguide Your Visitor – or What Not to Pay Attention to During a Plant Visit!

Get help by a third party reporting directly to you! There are probably too many KPIs for you to check, especially if you are higher up in the hierarchy. In this case you can outsource these checks. Maybe an assistant can see if the reality matches the numbers. External consultants are also popular for especially this task, even though it is usually not the prime reason they are hired.

Use common sense! Definitely use common sense! If the numbers sound fishy, they probably are. If the OEE exceeds 100%, this means that they produced more parts than the theoretical maximum. Same for delivery performances above 100%. Or one example from history: During the great famine in China (1958–1962), local leaders greatly exaggerated the amount of food produced on a field, eventually claiming quantities that would have covered the field 4 feet deep in grain. At that point even Mao stopped believing them, but as a son of a farmer he should have known much earlier.

By the way, people new in a position are less likely to fudge, since they do not yet know the fudging process, or at least do not control it. Additionally, it is a well-known strategy to remove all number fudging when starting a new position, in which case the previous occupant takes the blame for the fudging, whereas the current occupant comes across as honest. Over time, however, fudging may increase again, leading to a perceived improvement, until the next occupant of the position resets the fudging again. Over time this may result in a sawtooth pattern, although this is usually lost in the comings and goings of management in different positions.

Overall, these checks will tell you which KPIs are (probably) valid, and which KPIs are (probably) not. Therefore this will also tell you which of your people you can (probably) trust, and which you (probably) cannot, which brings us to the next point:

Punish Liars

If you have found that you have been fooled, take actions or you will be the fool! If you’re sure that one of your people feeds you bogus information, you need to issue a stern warning. If it is a repeated offender, you may have to exchange the person. This behavior has to stop one way or another. If you permit such skullduggery, you hurt the company, you hurt your reputation, and you foster even more skullduggery.

Summary

Overall, meaningful, correct, and not too many KPIs are part of an strong organization. If you can take action by removing unneeded KPIs or by checking the validity of KPI, please do so. It does not take much effort, but it can simplify life quite a bit. In any case, now go out and organize your industry!