Improve your statistical reasoning: Commonality between Covid-19 & inspection systems

“Look! Vaccination is not working: there are as many vaccinated as non-vaccinated persons in the hospital.”

Disclaimer: this article does not intend to make any statements about the best approach to deal with Covid-19. It just uses Covid-19 as a tangible example and aims to create insight into a common misconception.

I took the inspiration to write this article after having read a similar post and the reactions to that post. It seems a hard concept to grasp for many of us. I will try to simplify it even more for you & explain step by step why above conclusion is incorrect.

https://www.linkedin.com/embeds/publishingEmbed.html?articleId=8137801172930178130

Let’s start with a population that is vaccinated for 90%, which is roughly the case in several areas around the globe.

I have depicted 500 men & women of which 450 are fully vaccinated & 50 are not vaccinated at all.

Then corona strikes, and a certain number of people becomes ill. Alas, as we have all witnessed, a vaccine is not perfect. And hence some of the seriously ill people are people who are fully vaccinated. In our example, the split of people becoming seriously ill is equally distributed between vaccinated & non-vaccinated people.

For the ease of calculations reason, I have taken 10 + 10 men & women in red respectively purple to represent illness because of Covid-19. This is exactly the amount from the observation in the beginning, now placed in the right context. This situation (50-50 split) will always happen somewhere in time if a population goes from 0% vaccination to 100% vaccination. In the very beginning, no-one is vaccinated, so everybody becoming sick is not vaccinated. When a population reaches 100% vaccination, everybody becoming sick is vaccinated. And somewhere along that journey is a point at which the split is 50-50.

Hence, 10 out of 450 vaccinated persons have become ill (which is 2,22%) and 10 out of 50 non-vaccinated persons (20%). That is roughly a factor 10 difference if we compare relative numbers (instead of the absolute numbers used in the starting observation).

If vaccination would not have an impact, we would expect that 20% of vaccinated people become ill. In our example, that is 20% out of 450 or 90 persons. If there would be no vaccination or the vaccination would not be effective in protecting from illness, the total number of ill people would be 100 out of 500 (exactly the 20% we would expect). That is 5x more illness and hence 5x the amount of pressure on resources in healthcare.

When you observe – in absolute terms – a 50-50 split when one group is multiple times larger than the other, that shows a statistical difference. In our case, this implies vaccination has a significant & large impact, exactly the opposite of what many would conclude from the absolute numbers.

That concludes the Covid-19 example.

Let’s have a look at another situation in which this – at first sight counterintuitive – effect plays a role: inspection systems.

It is not uncommon for people in manufacturing to state: “How come our inspection system is only rejecting good quality product? Is it working correctly?” The observation that it is rejecting almost only good quality product is correct (and I will explain why that is logical in this situation). The conclusion that there is an issue with the inspection system is premature.

Like vaccines, inspection systems are never perfect. There are 2 types of errors: false positive & false negative (aka type I resp type II error). Think about the Tour the France & doping. Some have used doping and – luckily for them – slipped through the net. While others have not used doping, but got a positive test because of a lab error or medication.

Testing is the norm in manufacturing environments to check for the quality of products being produced. Among other industries, that is the case for food & vaccines. When talking about how to set-up such testing, naturally, everyone finds it important not to allow inferior products to go to the consumer. “We need to capture 99,9% of defective products.” No-one would disagree. Alas. The higher you choose that percentage (which represents the false accept risk), the higher your chance to reject good quality products (false reject risk). Under the motto better-safe-than-sorry, we accept a 1% false reject rate (in many environments that assessment is not done formally).

Say we are working in mass manufacturing environment that produces 1 000 000 products per day. And its quality performance is not too bad with 1000 ppm (parts per million) defects. In our example, we produce 999 000 of good quality products & 1000 defective products. That is what we produce. Now the products arrive at inspection. From the 1000 defective products, we correctly identify 99,9%. This means 1 defective product goes to an unhappy customer. From the 999 000 good quality products, our inspection system incorrectly classifies 9990 products as defects (corresponding to our agreed 1% false reject rate). You read that correctly: nine thousand nine hundred ninety excellent products go to waste. In pharmaceutical manufacturing, any doubt results in a decision to scrap the product, meaning an even larger waste.

Sometimes, there is a second issue: the inspection system is by-passed. Referring to an initial observation about testing, an operator of the morning shift sorts through the scrap (a common practice for analysis used in improvement projects). That operator observes that all the rejected products are of excellent quality. She talks to a colleague who had the same observation. They talk to the team leader, who cannot explain the situation either. The same happens in the afternoon shift. It seems as if the inspection system is failing. It would only be during the night shift that someone captures that one defective product in the reject bin. In a competitive market with high pressure on cost, this is a tough situation, especially when an extra security measure is implemented: the line is stopped in case there are too many consecutive rejects. This is a very tempting situation to cheat and by-pass the system. After all, in many environments, operators are rated on output. And so, a system designed to protect consumers has turned into a system where we feel in control because we have a great inspection system. Only a few know the system is by-passed.

I have witnessed such a situation and when discussing the issue; it was not a rare exception. I learned not to blame operators or team leaders: they act because of the incentives given. Design your systems meticulously, ensure managers give the right incentives (and not just on paper) & train your people. If you do so, you will get excellent results.

As a conclusion, both the type I & type II error should be considered. I hope that next time you are confronted with a similar situation; you do the simple math and draw the right conclusion.

(Originally published on LinkedIn on Nov 21st 2021)