Drag the confidence slider to 99%, the screen returns a wider stop, and it feels safer. The question almost nobody asks: does the number keep its word?
It is a promise that can be checked
Most numbers in this industry cannot be checked — “strong trend”, “good support”, there is no way to score them right or wrong. A confidence level is different. A stop labelled 95% may only be taken out 5% of the time. True or false, and countable.
So we counted.
It overstates, and most at the setting that looks most careful
At 90% the gap is small. At 99% — the setting people reach for when they want to be really sure — it is breached half again as often as promised.
And the error runs one way only. Not noise: noise would land on both sides. It is always optimistic, for a reason no formula repairs — the rarest events are not in the data yet to be counted. You are estimating the tail from a sample with the tail cut off.
The irony worth keeping
The higher the confidence, the more cautious it looks, and the less it is standing on. Exactly when you most want certainty is when the number is thinnest.
What we do about it
Not hide it. The system knows in advance whether the figure it is about to print has enough behind it, and when it does not, it has to say so rather than printing something confident to two decimal places.
The mechanism is not published. The principle is, and it is the principle of the whole product: an honest measurement has to know what it does not know.
Why this matters more to you than to a fund
A fund survives a few miscalibrated stops — hundreds of positions, and the error averages out. You have five. A stop labelled “one in a hundred” that is really one in sixty-six will take out your largest position in the month you can least afford it.
That number is your rent. To them it is a cell in a table.