My wife was diagnosed with Shingles. The first question the doctor had asked was: “Did you recently got your Booster vaccination?” The answer was “yes”. “You are the 5th patient in a short time whom I diagnose with Shingles. And usually, patients are older. That is not normal. I will pass your case to the official body for further investigation.
I searched Google to check what was known. One article caught my attention: it was written 5 months before and referred to the known studies about the topic, concluding that there is a presumption of correlation, although it has not been possible to prove a causal relationship.
Whether one can prove a causal relationship, in reality, that relationship is there or not. Statistics won’t change that reality. When you develop a vaccine, it either has a sufficient effectiveness or not. The side effects are what they are. After a time, it will be clear, but in the beginning, it is foggy. What statistics try to achieve is getting a faster and more accurate image of that reality. And whether we like it or not, there (still) is a cost (and time) related to that understanding. In the meantime, the clock is ticking.
The same challenge is omnipresent in manufacturing environments.
In food processing, you want to know whether there is a risk of contamination by some bacteria like Salmonella. So manufacturers have to do sampling. This does not guarantee there is or there is no contamination. It just helps to manage the risk. And if the risk is large, you are likely to detect it.
When you are manufacturing self-tests, you want to ensure that the capsule contains the right amount of liquid. We had one that was empty with the seal intact. Sure, the manufacturer did quality inspection. I have no doubt. They probably did not do 100% inspection as I got feedback that is a common problem (It is even possible they did 100% human inspection: because the people cannot concentrate a 100%, it is possible for defects to pass through). Somehow, this manufacturer concluded the batch was ok for shipment. Regardless of the inspection, the statistics & the decision, the defects are what they are and we have received an empty capsule. As a consumer, you suffer the consequences.
The less you (can) test, the less likely it is to find a deviation. Statistically, you make a correct (statistical) conclusion such as there are less than 1% defects with 95% confidence. In this case, it is possible there are 0,01%, 0,5%, 1% or even 1,2% defects. Statistics won’t change that reality. Without getting the full data, it is hard to interpret the findings.
When conclusions are being presented to you, always check the sample size and the conditions used to conclude this. This reminds me of an article I read once, stating: “Consumer research with 23 consumers of Coca Cola has revealed that …” It can’t have been too hard to find some more consumers.
When you are responsible for the analysis yourself, ensure you use the proper tools. For the case of Shingles, using statistical process control (even simply plotting the data) will be the most effective way to detect something is going in. In the case of, of the empty capsule, when starting up the production, sequential testing is the most economical (for the same quality). In production, statistical process control will yield better results than AQL sampling (despite the latter one being the norm). None of these tools are complex at all to use. You might need some help to set them up.
See also: Improve your statistical reasoning: Commonality between Covid-19 & inspection systems
Dear Maarten, the statement “Whether one can prove a relationship, in reality, the relationship is there or not.” deserves perhaps some nuance. A relationship can be direct or indirect. And in fact there are different kinds of relationships. You also state that “when you develop a vaccine, it either works or not.” But in reality it is possible that the vaccine only works if certain conditions are met or that it works better for certain groups in the population. I suggest to take these factors into account. Maybe you can draw a DAG’s (Directed acyclic graph) to make the assumed relationships and potentially confounding factors visible.
Thank you Joris for your feedback. You are right. Hence I have adjusted those sentences to be more accurate.
I have tried to do so by keeping the text as simple as possible. That is why I didn’t include the DAG’s: it won’t enhance readability or improve the primary message I am trying to convey.