📈 Regression To The Mean
Good morning, friends!☀️
I am writing this on the train to Tunbridge Wells to celebrate my Mum’s birthday. I hope the grammar and spelling are reasonable, as I am using an iPhone!
I learned about this interesting statistical phenomenon a few years ago when reading Thinking Fast and Slow by Daniel Kahneman (affiliate link). It's called regression to the mean, and it influences our lives almost daily. It explains why normal is normal.
In the 70s, Kahneman taught Israeli Air Force pilots the psychology of effective training. The pilots claimed that negative feedback is better than positive feedback for getting the pilots to perform better.
They backed this up by saying that a pilot's subsequent performance is often better after they perform poorly and receive negative feedback. However, if they do very well and get praise, their performance will worsen.
This goes against many studies that say positive feedback is better than negative feedback. So what's going on here?
More than people realise, chance and luck play a huge role in our lives, and this Israeli Air Force pilot example showcases it.
Kahneman told the pilots they couldn't conclude that negative feedback was better than positive feedback, as this was an example of regression to the mean.
When pilots perform abnormally worse, they will perform better next time. Likewise, if they perform well, their next performance will likely worsen. This is just pure probability and statistics. They slowly regress back to average performance.
This is because scoring well in their flight tests doesn’t correlate perfectly with talent and hard work. Other random and lucky variables, like weather, tiredness, aircraft age, etc., are often out of their control. This adds inherited variability to their performance, which makes it challenging to sustain outlier events.
This phenomenon is seen everywhere; sports are a typical example. When a football player does well in one game, they are doubtful to play to the same level the next game as many factors are out of their control, so they will regress down to their average performance.
I personally experience this with my YouTube videos. No matter how much time and effort I put in, I have no control over whether the audience will like the video. Sure, there is some correlation with hard work, but much luck is involved.
Whether when we perform well or poorly, we should expect to be closer to our average next time.
This doesn't mean we shouldn't try to improve, but we should realise that it takes time to be good at something and slowly raise one's average.
Weekly Favourites ❤️
🎙️ Podcast — How To Build A Multi-Million Dollar Empire In Your 20s - Grace Beverley. Really enjoyed listening to this conversation between Ali Abdaal and Grace Beverley. I found Grace’s transition from content creator/influencer to business owner interesting and gave me some ideas.
🎬 Video — xLSTM: Extended Long Short-Term Memory. Been learning about RNNs recently, and this popped into my feed. Fascinating research paper on improving upon the classic LSTM models that initially came out in the 90s.
🔨 Tool — iTerm2. If you use the command line, you know the basic terminal on Mac isn’t the greatest. I have been using iTerm2 for the past 2 years as its customisability is great.
(PS: Some links are affiliate links that I get a kickback from with no extra cost to you 😎)
My Latest Content 🎬
You can reach me on:
LinkedIn, X (Twitter), or Instagram.
My YouTube Channel and Medium Blog to learn technical data science and machine learning concepts!
💡 If you are interested in sponsoring this newsletter see here.