Post-mortem
Our memecoin sniper lost 59%. The bugs were in our own measurements.
It showed profit in simulation for weeks. Then we let it trade, and it didn't. Everything interesting turned out to be in the gap between those two facts.
We built the whole thing. Live feed of new token launches, a feature store, labels at four horizons, a grid search over entry and exit rules, a paper engine, then a live executor with real money in it.
Live, it returned −59% and we shut it off.
Our first instinct was to blame execution. Latency, slippage, bad fills, the usual suspects. That was wrong, and chasing it wasted about a week. The simulation had been lying to us, and it was lying in four different places at once.
The 0.90 that should have scared us
One classifier came back at 0.90 AUC. We were pleased.
Nobody gets 0.90 predicting which token launch pumps. If we had treated that number as a bug report instead of a milestone we'd have saved a month.
The top feature was inflow_peak: the largest net inflow inside a 90-minute window.
The label was whether the token pumped inside that same window. So the feature was looking at the
pump and reporting it back to us as a prediction. We had built a very confident rear-view mirror.
Rebuilt with only data available at launch time, the same model separated winners by about 2.5×. Real, but not enough to pay for fees, and nothing like 0.90.
Counting the same money twice
Best paper strategy: +0.20 SOL. It was inflated 1.75×.
The strategy locked part of a position and let the rest ride. We cached quoted value keyed on the token mint. Not on the size. So the locked chunk and the runner each pulled the same full-size quote, and both legs got valued as if they were the entire position.
What makes this one annoying is that the evidence was sitting in front of us the whole time. One token showed +75% in the sim and −9% live on the same trade. We assumed that was slippage and moved on.
The bug that hid a real signal
This is the one worth reading twice, because it runs the other way.
We assumed exits would cost 20–28% in slippage. Thin pools, obviously brutal, seemed prudent. When we finally stopped assuming and quoted our actual position size against a live router, the median price impact came back at 0%. We were too small to move anything.
Those inflated costs had been sitting inside the grid search the entire time, quietly killing strategies that were genuinely positive. We had spent weeks searching a space we'd already poisoned, and we never questioned it because pessimism feels responsible.
Overstated costs lie exactly as much as understated ones. It just feels safer while you're doing it.
403
An API started returning 403 for our account. Obviously we'd been flagged. We built a
whole theory about it.
Then someone thought to try a freshly generated address that had never touched the service. Also
403. The endpoint just wanted authentication. Nothing had happened to us at all.
Two minutes of control test, several days of nonsense.
What the numbers said once we could trust them
With the measurement fixed, the answer stopped being ambiguous. Four separate ways in.
| Question | Answer | |
|---|---|---|
| Is entry predictable? | AUC 0.45–0.48 | Four horizons, out of sample. At or below chance on all of them. |
| Is the average trade positive? | half fade −25% | The middle of the distribution is negative, not flat. |
| Does a smarter exit rescue it? | 7 geometries | None beat a plain stop. Cutting early just whipsaws you out of the dips that come before the real moves. |
| How fast do winners run? | peak at 49s | Median time from launch to top. |
That last one is what actually settles it. If the winners are done inside a minute, you're not in an analysis contest. You're in a latency contest, against people who own that problem professionally. No amount of feature engineering fixes being 400 milliseconds late.
The peaks were real. We just couldn't reach them.
Worth separating out, because it inverts what everyone assumes.
The spikes that looked fake in our data weren't fake. At micro size we measured a round trip in a $5,600 pool costing 2.6%, confirmed on chain. The move was real and it was reachable, as long as you stayed tiny.
Grow the position and it evaporates. Which is a strange kind of dead end: the edge exists, it just has almost no capacity, and the infrastructure to chase it costs more than it can ever return.
If you're building one now
We're not going to tell you not to. We'd have ignored that too.
What we'd say instead is that all four of our bugs survived for the same reason. We interrogated every disappointing result and none of the flattering ones. A number that made us happy went straight into the deck.
So before any of it touches money:
| Check | How |
|---|---|
| Leakage | Shuffle the labels and rerun. Still scoring well? Your features can see the future. |
| Accounting | Reconcile one simulated trade against one live trade, line by line. Just one. |
| Costs | Quote your real size against a live router. Guessing high is not the safe option. |
| Controls | Run every diagnostic against a case where you already know the answer. |
Where this ended up
Months of compute and an account down 59%. Since we paid for the lessons anyway, we turned the useful ones into checks that anyone can call for a fraction of a cent.
The one that comes straight out of this story is /v1/exit-cost. It quotes what leaving
a position would actually cost at your size, right now. That's the measurement whose absence cost us
59%, and it takes about a second.
There's also a pool safety check, which at the time of writing flags a token trading at $0.927 sitting inside three separate pools labelled "stablecoin" with yields up to 685%. And a chain check, for working out whether an ecosystem is quiet or actually abandoned.