The Illusion of Validity was first introduced by Amos Tversky and Daniel Kahneman. It is described as the tendency for one to overestimate his ability to accurately predict outcomes when analyzing a set of data which shows a consistent pattern. As humans, we’re natural pattern seekers or pattern detectors. The problem is we assume meaning in these patterns or believe that it “tells a story”, when there is simply none.
Let’s explain this with an example. There’s a common phenomenon known as the Gambler’s Fallacy. In a casino when we walk up to a roulette table we see all the numbers that recently came up on the board. Why is it so? When people see that a number has come up quite a few times they think it’s a lucky nu
mber to bet on. Some believe that it’s a wise decision to not bet on that number because of its frequency of occurrence. In reality, each spin is independent of the other. So, if a number has come up in a streak, it’s purely based on probability. It doesn’t verify whether the number will come up in the next spin or not.
We can’t help but think that the world is more predictable than it really is. A similar fallacy is the Hot Hands Fallacy.
Imagine if it’s Poker, and you’re playing on straight Aces for maybe 3-4 times in a row, you fall under the illusion that this hot hand will continue and that you’re doing exceptionally well on purpose. The truth is that this hot hand is purely a statistical probability. We find it difficult to believe that in fact, anything could happen.
Daniel Kahneman discovered the Illusion of Validity while he was working as a psychologist for the Israeli Army. He was required to put the candidates through certain challenges in order to test their leadership potentials. Based on how these candidates completed the challenges, he would predict their performance at the Officer Candidate School. Kahneman later discovered when the actual performance of recruits was compared with the ones predicted by the tests, the correlation between the two was
Zero.
This was an example of how human beings think “what we see is all there is”. We made up a story from the little we knew based on the candidates’ performances which had no way to tell us about the individuals’ future.
As Kahneman said,
"We were required to predict a soldier's performance in officer training and combat, but we did so by evaluating his behavior over one hour in an artificial situation.”
Moreover, illusion of validity blinds our overconfidence. This discovery should have made us alter our predictions about these candidates but it did not. When faced with events and evidences which are incompatible with our “stories”, we’re simply unable to absorb them. This is even more with statistical studies of performance, which provide general facts which people will ignore if they conflict with their very own personal experience.
The Illusion of Validity can easily be seen a concept of “mistaking luck for being a skill”.
Scientist Freeman Dyson, was a statistician for the Bomber Command of the British Army during World War 2. The bombers were unable to fly high to avoid getting shot down because of the heavy gun turrets they were carrying. The logical solution is to remove the turrets, right? Not according to the commander in chief. He was disillusioned that their chances of survival would improve if the team became more skilled with time and experience. The reality however was that the death of the crews was random, unaffected by experience.
Based on these examples, we can say that the Illusion of Validity leads us to understand three basic facts;
Firstly, human beings like making up “stories” when they see consistent data patterns. Secondly, they're incredibly overconfident in their abilities to make predictions from these “stories”, and lastly, they disregard evidences which do not support these stories and assumptions.
What this means for us is that we consider how frequently an event occurs rather than the probability of that event actually happening. Our predictions thus, are only slightly more accurate than complete guesses.
WRITTEN BY
Comments