AAPL Backtest Results

Reading this honestly

• A look-ahead leak in the original runs, found in V2 (2026-09-28). Runs #3–#14 decided each day after seeing that day’s own close — confirmed for 938 of 938 stored decisions. As originally run, every one longer than a quarter underperformed buy-and-hold, by $39 to $225. The corrected reruns (#45, #71) also skip market holidays, fill at the open, and run on Claude rather than GPT-4o, so a difference from the originals can’t be attributed to the leak alone. They fixed the price leak only.
• A second look-ahead leak, in the news (found 2026-10-06). 99.7% of the headline data’s timestamps are dates at midnight UTC, so a decision at 8 a.m. New York treated headlines dated that day as already published — on 98–100% of decisions in every run, #45 and #71 included. At 8 a.m. on Friday 2022-06-03 the sentiment analyst read “Apple Was the Worst Stock in the Dow Friday”. So #3–#71 all carry at least one leak; #45 and #71 are a record of the first fix, not a result. #136 is the out-of-sample window with both fixed: V1’s rules over its decisions finish $72 ahead of buy-and-hold in a flat market — one run, about twice the replicate spread, not evidence of an edge. The in-sample runs are not yet redone.
• Runs #45 and #71 trade with V2’s VaR risk engine, which caps the position by a pre-registered 2% value-at-risk budget on top of V1’s rules. It held fewer shares in both: −$492 against V1’s rules in-sample, +$57 out-of-sample. V1’s rules were replayed exactly over the same decisions — the Portfolio Manager never sees positions — so those comparisons carry no model noise.
• A baseline error, found and corrected 2026-09-09. Backtest 4 ran 2022-06-01 to 2023-06-30; runs 7, 8, 9 and 14 ran 2022-06-03 to 2023-06-30. AAPL fell 148.73 to 145.39 across those two extra days, moving buy-and-hold by $66.80 on a 20-share basis. Earlier writeups compared the ablation runs against backtest 4's +$903.80 and concluded they "matched buy and hold"; against their own window's +$970.60 they were $41–74 below it. Every row in the comparison table above uses its own window's baseline.
• Runs 7 and 8 are ablations. The Sentiment Analyst is replaced by a fixed neutral stand-in while the Portfolio Manager keeps making real LLM calls, so the graph structure is held constant and only the node's content is removed. They outperformed the runs that kept the node — which is what prompted rewriting it.
• Run 14 changed two things at once — the sentiment node and the Portfolio Manager prompt that reads it. Its heavy trading may come from either. One further ablation separates them; it hasn't been run.
• Backtests #1 and #2 aren't shown. #1 ran before a transaction bug was fixed (position cap never engaged — 25 buys against a 20-share limit). #2 landed entirely inside a gap in the FNSPID headline data (Jan–Mar 2022), so the Sentiment Analyst never saw a single headline — it wasn't a real test of the two-agent system.
• "Buy & hold, 20 sh" is what simply buying the position cap on day one and holding to the last day of the window would have made — the honest baseline, not a strawman.
• Win rate is realized trades only. An open lot at window end isn't included until it closes, which can flatter or understate the picture — see the mark-to-market note above the P&L chart where it applies.
• Slippage: 5bps against the fill price, both directions — the decision-day close in original runs, the decision-day open in corrected ones. Commission: $0 (matches Alpaca). No benchmark beyond buy-and-hold is computed here.