Known failures

What went wrong, how it was found, and what changed. Everything below is taken from the repository's own notes and commits; each item names its source.

1. Every backtest could see that day's closing price

Decisions run at 12:00 UTC (8 a.m. in New York) and asked the price source for bars "up to now". The source stamps each daily bar at midnight New York, which is earlier than 12:00 UTC, so the decision day's own bar came back, including its close. The written backtesting plan required prior-day prices only; nothing checked it.

• Found: 28 September 2026 by scripts/check_bar_timing.py, which asks the real price source what a decision would see. Both check days leaked. Comparing with real closes confirmed it for 938 of 938 stored decisions in three V1 runs.
• Fixed: bars whose 16:00 New York session had not closed before the decision are now dropped (588adef), trading days come from the market calendar, and fills use the decision day's open.
• Sources: scripts/check_bar_timing.py docstring, docs/engineering-log.md, PROJECT.md defect 8.

2. …and every backtest read that day's news before it happened

The news data carries dates, not times: 99.7% of the timestamps are exactly midnight UTC. The rule "published before the decision" therefore treated a headline dated the same day as already published, even when it was an after-the-close report. At 8 a.m. on Friday 2022-06-03 the sentiment analyst was reading a headline about Apple being the worst stock in the Dow that Friday.

• Found: 5 to 6 October 2026, while building the phase 09 test set, after the first "corrected" reruns had already been published as corrected. A re-check of the stored headlines gives 99.66% of the AAPL timestamps at midnight UTC.
• Fixed: a date-only headline is now available only once its date has ended in New York (6bebc48). The reruns' published numbers were relabelled as superseded (fd99811); only backtest 136 has both leaks fixed.
• Consequence: every originally published result had at least one leak. The in-sample runs have not been redone. Backtests 45 and 71 on this site are marked superseded for that reason.
• Why it survived: a written requirement never became a test, and this one survived the fix for the first.
• Sources: PROJECT.md defect 9 and the corrections section, docs/engineering-log.md.

3. Two fine-tuned adapters that did not beat a do-nothing guess

One database query explained both: across every backtest the teacher had produced 17 BUY, 3 SELL and 537 HOLD. Distilling a near-constant teacher yields a constant student. A rerun in October 2026 tracked in MLflow showed the PhraseBank adapter does learn its own dataset (91.3% on held-out PhraseBank against 59.3% for always-HOLD); the 84.0% that failed was the transfer to this project's headlines, and that test was not rerun.

Sources: PROJECT.md "Phase 04 — the fine-tuning work"; eval/mlflow/.

4. Other findings of the same kind

• An abstaining component made things worse. The sentiment node voted HOLD about 96% of the time; ablating it improved results by about 60 on one window, at roughly twice the measured run-to-run spread. One window, two replicates per configuration: directional, not conclusive.
• The pre-registered threshold was below its own test's resolving power. The return-prediction bar (IC at least 0.03) sat at about one standard error (0.029) of the 1,221-row split chosen to judge it. The conclusion survives through a larger split, which is reported rather than quietly substituted.
• Six database tests silently skipped for two phases. CI now fails if any test is skipped.

Sources: PROJECT.md "The ablation", "A flaw in the pre-registration itself", and the defects table.

What these have in common

None was visible to type checks, linting or re-reading. Each was found by checking a number that looked wrong against the data it came from, and each is now a test or a written practice. The full list is in the project record.