A multi-agent paper-trading system, built across six phases and evaluated honestly. It runs, it reasons, it deploys — and it does not beat buying the stock and holding it. This is the report of that result.
Every backtest longer than a single quarter underperformed buy-and-hold, by $39 to $225 on a 20-share basis. The out-of-sample run — executed once, with no code, prompt or parameter changes made after seeing the in-sample result — lost $63 in a market that was essentially flat.
The project's founding rule was that an honestly evaluated failure is the correct thing to report. So it is reported, and the machinery built to detect it is the actual deliverable.
Corrections, 5 and 6 October 2026: every backtest in this report could see the future, twice over.
The first step of V2 checked when each simulated decision got its prices. Backtests decided at 12:00 UTC, but the price source stamps a day’s bar at midnight New York, so each decision was handed that day’s own close — a price that would not exist for another eight hours. Confirmed for all 938 stored decisions in runs #4, #6, #9 and #14. Fixed, along with two smaller problems found next to it: decisions made on market holidays, and fills at a price already gone by the open.
Two runs were redone with those fixes, and on Claude rather than GPT-4o. They are not results either: a second look-ahead leak was found on 6 October. The news data carries dates, not times, so every decision at 8 a.m. read headlines dated that same day — on 98–100% of decisions in every run, the reruns included. At 8 a.m. on Friday 2022-06-03 the sentiment analyst was reading “Apple Was the Worst Stock in the Dow Friday”. The headline of this report does not survive. Redone with both leaks fixed, the out-of-sample run finished $72 ahead of buy-and-hold in a flat market (+$73.88 against +$2.20) — one run on one stock, not evidence of an edge. The in-sample runs are not yet redone. Every figure below is from the original runs and is left as published.
Six phases · five built · one closed on a negative result
The premise was to close six engineering gaps by building one real system rather than six disconnected exercises. Each phase produces something the next one depends on, and the sequence is real: nothing here is a numbered list of parallel tasks.
Five of the six shipped. The sixth — the fine-tuned sentiment model — was investigated to a conclusion, and the conclusion was negative. That is a different outcome from being skipped, and is marked as such below.
Four nodes, deliberately split by whether judgment is actually required. A Technical Analyst computes RSI-14 and SMA-20 in plain Python — arithmetic is not worth an LLM. A Sentiment Analyst reads headlines through GPT-4o-mini. A Portfolio Manager (GPT-4o) synthesises both into a proposal. A Risk Manager, rule-based again, holds final authority and can veto or scale any trade.
That last split — propose versus gate — is what made most of this report possible. Because the Risk Manager is code and every input it consumed is stored, alternative risk rules could later be tested for free.
Seven runs · real Alpaca prices · real LLM calls · 5bps slippage
Headlines came from FNSPID, a 23 GB historical dataset, imported locally. Prices came from Alpaca's IEX feed. Every run made genuine LLM calls; nothing here is simulated reasoning.
| Run | Configuration | Closed | Result | Buy & hold | Difference |
|---|---|---|---|---|---|
| #3 | categorical sentiment · pilot quarter | 4 | +6.21 | −15.40 | +21.61 |
| #4 | categorical sentiment · in-sample | 32 | +864.62 | +903.80 | −39.18 |
| #6 | categorical sentiment · out-of-sample | 2 | −63.10 | +2.20 | −65.30 |
| #7 | sentiment ablated | 33 | +929.84 | +970.60 | −40.76 |
| #8 | sentiment ablated | 33 | +896.15 | +970.60 | −74.45 |
| #9 | categorical sentiment · replicate of #4 | 31 | +840.45 | +970.60 | −130.15 |
| #14 | continuous score node | 62 | +745.86 | +970.60 | −224.74 |
The single positive row is a three-month pilot with four closed trades. It beat the baseline mainly because AAPL fell over that window — buy-and-hold lost money and a largely idle system did not. That is not evidence of skill.
Corrected since (see above): every row here had the look-ahead leak. Corrected reruns: #6 → +66.35 against +2.20; the #14 configuration on #4’s window → +920.81 against +837.80.
Realized P&L alone made run #3 look like a clean 100% win rate and made run #14 look far worse than it was. Both had open lots carrying unrealized moves. Any run ending with open positions gets marked at the window's last price before being compared to anything.
A baseline error, found by checking a number that looked slightly wrong.
Run #4 covered 2022-06-01 → 2023-06-30. Runs #7, #8, #9 and #14 covered 2022-06-03 → 2023-06-30. AAPL fell 148.73 → 145.39 across those two extra days, moving buy-and-hold by $66.80.
Every earlier writeup had compared the ablation runs against run #4's baseline and concluded they "matched buy and hold". Against their own window they were $41–74 below it. The comparison tool had been printing the window for every run the whole time; nobody read it. It now refuses to be ignored.
The most instructive failure in the project
The goal was to replace the sentiment node's API call with a locally fine-tuned model. Four things were tried. All four are measured, and every measurement points the same way.
| Attempt | Headline number | Majority-class baseline | Verdict |
|---|---|---|---|
| Financial PhraseBank QLoRA | 84.0% | 96.8% | below baseline |
| Distilled from GPT-4o-mini | 90.9% | 90.9% | exactly baseline |
The second adapter predicted HOLD on all 44 held-out rows. It had learned the constant function. The first invented SELL calls on neutral headlines and reached 3.6% precision on BUY.
A database inventory explained both at once: across every backtest the teacher had produced 17 BUY, 3 SELL and 537 HOLD, on one symbol, over overlapping date ranges. Distilling a teacher that says HOLD 96.8% of the time yields a student that always says HOLD. The problem was never the hyperparameters.
Before spending roughly 6,000 labelling calls on more training data, the node was removed and the system re-run — same graph, same Portfolio Manager, only the sentiment content replaced by a neutral stand-in. Removing it improved results by about 60 on a 13-month window, consistently, at roughly twice the measured run-to-run noise.
The mechanism was visible in the action counts. The node's near-constant HOLD votes read to the Portfolio Manager as a vote against trading, suppressing around nine BUYs and six SELLs that would have been profitable. An agent that abstains most of the time is not automatically harmless.
The node was rewritten to emit a continuous −1.0 to +1.0 tone score instead of a categorical vote — fixing the root cause of both failed adapters, since a score has real variance even on boring news and is therefore a trainable regression target.
It succeeded at exactly that. Across 281 days: range −0.50 to +0.50, mean +0.057, standard deviation 0.235, zero exact zeros, 184 positive and 97 negative days. No collapse.
And it produced the worst trading result of any configuration tested: +745.86 against a bar of +896/+930. It traded roughly twice as much — 62 closed trades against 32–33 everywhere else, at a 32% win rate. "The component works as specified" and "the system got better" are independent claims.
A confound, named rather than buried.
That run changed two things at once: the sentiment node and the Portfolio Manager prompt that reads it. The prompt change was designed to stop abstentions suppressing trades — and it worked, possibly too well. This run cannot attribute the extra trading to either cause. One further ablation would separate them; it has not been run.
Phase 04 was then reframed rather than continued down the same road. The remaining labelling calls would have bought a better imitation of a component every measurement says to delete — so the question changed.
Every v1 failure had the same suspect: the label. Imitating a teacher that said HOLD 96.8% of the time makes the constant function the loss minimum, and two independent training runs found it. So v2 threw the teacher away and labelled from price data instead — each headline block tagged with the stock’s actual 5-day excess return, market move subtracted, z-scored per symbol. 41,701 examples across 104 symbols over 19 months, with labels whose standard deviation is ~1.0. Free labels, real variance, none of v1’s defects.
A TF-IDF and ridge baseline was run first, before any GPU time, because a low error on a centred label means nothing on its own. It found an information coefficient of +0.002. The bar for the fine-tuned model was then written down — IC ≥ 0.03 — before training started.
| Validation split | n | Spearman IC | R² | Sign acc. | Base rate |
|---|---|---|---|---|---|
| later dates, seen symbols | 9,336 | −0.0074 | −0.007 | 0.502 | 0.501 |
| unseen symbols, overlapping dates | 4,774 | +0.0104 | −0.005 | 0.499 | 0.509 |
| unseen symbols and dates | 1,221 | −0.0125 | −0.011 | 0.477 | 0.524 |
Below the bar, below the TF-IDF baseline, and below the accuracy of simply always guessing “up”.
| Validation split | n | Spearman IC | R² | Sign acc. | Base rate |
|---|---|---|---|---|---|
| later dates, seen symbols | 9,336 | −0.0003 | −0.004 | 0.495 | 0.501 |
| unseen symbols, overlapping dates | 4,774 | +0.0181 | −0.001 | 0.501 | 0.509 |
| unseen symbols and dates | 1,221 | +0.0278 | +0.000 | 0.508 | 0.524 |
The run first cut down for cost, done in full (5.9 GPU hours). Same answer: +0.028 sits just under the bar on a split that cannot tell it from zero (95% interval −0.028 to +0.084); the 9,336-row split gives −0.0003 and excludes 0.03. Prediction sd 0.040 — collapsed to the mean again.
The prediction spread is the number that matters. Predictions had a standard deviation of 0.051 against labels at ~1.0 — the model collapsed to predicting the mean. That superficially resembles v1’s always-HOLD collapse and is the opposite thing. In v1 the labels were constant and the model reproduced a degenerate target. Here the labels vary and the model still predicts the mean, because predicting the mean is the correct loss-minimising answer when the input carries no information about the target. A model making confident, varied predictions on this data would be the broken one.
A flaw in the pre-registration itself, found afterwards and recorded rather than dropped.
The bar was IC ≥ 0.03 and the 1,221-row clean split was named as deciding. A Spearman correlation on 1,221 rows has a standard error of about 0.029 — so the bar sat at roughly one standard error of the split chosen to judge it, with a 95% interval of [−0.069, +0.044] that contains 0.03. That split could never have separated a true 0.03 from zero, whichever way it landed. Resolving 0.03 at two standard errors needs about 4,400 rows.
The conclusion survives, but through the larger split: 9,336 rows, interval [−0.028, +0.013], which does exclude 0.03. Its weaker isolation permits only a leak that would help a model, and it still found nothing. The fix next time is holding out thirty symbols instead of fifteen.
Five attempts, two entirely different framings, one answer. Phase 04 is answered, not abandoned — the data was the suspect in v1, and v2 removed it as a suspect. Stated limits: 1.5B parameters at most, one seed per run (the cut-down 0.5B run has since been repeated at full size, with the same result), a single five-day horizon, one market regime. Absence of a signal this setup can detect is not proof that none exists.
Offline replay · zero API cost
The Risk Manager is rule-based, the Portfolio Manager's proposals are stored, and the price the agent saw each day is stored. So "what would a different risk rule have done?" is arithmetic over recorded data, not a new backtest. A replay tool re-gates every recorded proposal — and refuses to print anything unless its baseline reproduces the recorded run first. On the out-of-sample window it matches to $0.0003.
It was used to test the standing assumption that a drawdown rule would have saved the losing run. It would not have.
| Rule | Result | vs. baseline |
|---|---|---|
| baseline (position cap only) | −63.10 | +0.00 |
| no adding when down 2% | −63.10 | +0.00 |
| no adding when down 1% | −49.24 | +13.86 |
| stop-loss 3% | −49.03 | +14.06 |
| trailing stop 5% | −64.94 | −1.84 |
| min 3 trading days between buys | −49.24 | +13.86 |
It also corrected the project's own recorded diagnosis. Three consecutive December buys into a decline had been named as the cause; they are 43% of the loss. The largest single contributor, 54%, is one September lot caught by a two-day 6.4% gap down that no add-to-position rule can anticipate. And the December decline was only ~1.1% from the first buy to the third, which is why a 2% threshold changes literally nothing.
Note also that three "different" rules land on exactly −49.24. They block the same two trades by three mechanisms. A table of seven rules was really testing about three ideas — worth knowing before anyone reports "we evaluated seven risk rules".
Corrected since: this replay ran over the leaky run #6, so the loss it was trying to explain does not survive the correction — rerun, #6 finishes $66 ahead. The replay tool itself carried over into V2 unchanged: it is how V2’s risk engine was compared with V1’s rules on identical decisions.
Each one changed a conclusion or hid one
Decisions ran at 12:00 UTC and asked for bars “up to now”; the price source stamps each day’s bar at midnight New York, so the decision day’s own bar, close included, came back. The backtesting plan required prior-day prices from the start; nothing checked it. Prices are now filtered to bars whose session closed before the decision.
99.7% of the headline data’s timestamps are dates at midnight UTC. The rule “published before the decision” therefore admitted headlines dated the decision day itself, published at any hour of it. It was found only while building an evaluation set, after reruns that fixed the first leak had been published as corrected. A date-only headline now counts as available once its date has ended in New York.
The first insert wasn't wrapped in its own transaction, so every later conn.transaction() silently downgraded to a savepoint and no day's writes were visible to the next day. The symptom was the position cap never engaging. Found before any result was trusted.
On Windows, psycopg's pool cannot run on the default event loop. It raises PoolTimeout, which subclasses OperationalError, which the "no database available" guard caught. The suite reported green while running 33 of 39 tests. CI now fails if any test skips.
Two days' difference between runs moved the baseline by $66.80 and turned "matched buy and hold" into "$41–74 below it".
Correct for a process on the host, wrong inside a container where it is the container's own loopback. Migrations failed with "Connection refused" while docker ps showed Postgres healthy — both true at once.
most_recent = true meant a newly published image made Terraform propose destroying a running instance for no reason anyone asked for. A plan you cannot trust is a plan you stop reading.
The documented subject is repo:owner/name; the real one appends immutable account and repository IDs. AWS returns only "Not authorized" — uninformative by design, since naming the failed condition would let an attacker probe the policy. CloudTrail carries the actual claim.
Four of the six found in V1 were only findable by running the thing for real. That is the argument for the deployment costing a few cents rather than being left as validated code. The two found in V2 were visible in the code and data all along; each survived because a written requirement was never turned into a check — and the second survived the fix for the first.
Applied, verified, destroyed — same day
Terraform builds a purpose-made VPC, a t3.micro running the container image, an ECR registry, four encrypted parameters, scoped IAM roles, and an EventBridge schedule wired through SSM. Images reach the registry through GitHub OIDC — short-lived tokens, no access key stored anywhere.
Defaults are chosen to be safe rather than useful. The schedule ships disabled, because every firing spends real money. The managed database is off, because a container costs nothing and the trade-off belongs to whoever pays. There are no inbound rules at all; shell access is through SSM Session Manager, so there is no open port, no bastion and no key to lose.
It ran one real cycle on AWS — real prices, real LLM calls, a real database write — and returned run_id 1 · SUCCESS · AAPL BUY conf=0.65. Then it was destroyed. Keeping it running would cost about $13 a month to trade a strategy already measured as having no edge; proving the deployment works is worth a few cents, and running it indefinitely is not.
And what it deliberately does not
A working FastAPI service over Postgres with hand-written SQL and a hand-rolled migration runner. A four-node LangGraph agent where each node's model choice is justified by whether judgment is required. QLoRA fine-tuning end to end — classification, distillation from a teacher, and a regression head trained on a 41,000-example dataset built from scratch with leak-free splits. Containerisation, CI against a real database, and infrastructure-as-code applied to a live cloud account.
More usefully, it demonstrates an evaluation discipline: out-of-sample runs executed once and never re-tuned; accuracy numbers checked against the majority-class baseline before being believed; components ablated before being improved; noise floors measured rather than assumed; and a confound named in the writeup instead of smoothed over.
It does not demonstrate a profitable trading strategy, and no part of this report should be read as investment advice. The system never touched real capital. Its own conclusion is that it should not.