Here are eight that caught us, what happened, and how we check for each one now. This is method only; none of it needs our strategies to make sense.
Trading at the price you decided on
One comparison said spreading an order through the day cost far more than buying at the close. But the test decided on the closing price and filled at that same close, which no live order can do. Filled a day later, as a real order would be, the difference disappeared.
Now Fills come at least one session after the decision, and every report shows how far the result moves between the two.
Testing only the companies that survived
If the stock list holds only companies still trading today, every company that failed is missing from the past. We measured the inflation at roughly 2.4 to 4.2 percentage points a year, larger for longer lists. A later cross-check has not yet reconciled the exact size, so we treat it as an estimate.
Now Lists are built as they stood on each date, and an edge smaller than this bias counts as unproven.
Prices of zero at the start of history
To find where a stock’s history begins, a probe asked whether the data source returned any rows for a date. For one stock it returned months of rows with every price at zero, and the history appeared to start there. A zero next to a real price also looks like a crash.
Now A row counts as data only if its prices are valid, and start dates found the old way are suspect until recomputed.
Dropping what you cannot match
A pipeline dropped the index members it could not match to an identifier. All 16 dropped names turned out to be companies that had merged, been renamed or wound up: not one survivor. The quick repair then counted some companies twice, giving 59 members for a 50-name index, and the validator passed, because it checked each row on its own.
Now Before accepting a drop, we split what was dropped into survivors and non-survivors, and check counts against a size the table cannot supply itself.
A pass mark mistaken for power
A design called two standard errors its smallest detectable effect. That is only the pass mark: a true effect of exactly that size passes about half the time. Tests looked better powered than they were, and “not found” was read as “not there”.
Now Every design states the effect it can detect 80% of the time, apart from its pass mark. A result below that is “not powered”: neither a pass nor a fail.
A control that cannot fail
A negative control scrambles the data so that no real effect can survive. Ours returned the real run’s number to the last digit, and the check read “pass”. It happened twice in one day, through two different bugs, and nothing raised an error.
Now Every control’s score is compared with the real run’s, and identical digits stop the run.
A broken measurement that looks like progress
A code-quality gate counted errors and only let the count fall. A setting the tool silently ignored made the count depend on the shell, not the code. When the count dropped by 304, the gate recorded an improvement. None of those 304 was a real fix.
Now A falling count is suspect until explained, and a setting is tested where the tool reads it.
A language model that already knows the answer
A model trained on text that covers the test years may have seen the outcomes. We tried to measure that with other language models, but they share the same hindsight, and no check run with them can prove it absent.
Now We read such backtests one way only. A fail is a real negative; a pass is at best an upper bound. The clean test is forecasting results that have not happened yet.
The common thread
Each trap made a result look better, or a check look passed, without an error message. So we check good news hardest.
Have you been fooled by a trap that is not on this list? We would like to hear about it.