We built a tool that makes overfitting easier. That’s why we count.

The problem, stated plainly

Overfitting in backtesting is driven above all by one thing: how many variants were tried before you kept the one you kept. Test enough rules against the same history and something will look good by luck alone, and the more you test the more certain that becomes.

Our product removes the friction from exactly that step. Describe an idea in plain English, and the harness builds it, tests it, varies it and tests it again. What used to be an afternoon of scripting is now a sentence, and the research loop generates and discards candidates continuously.

So a fair reading of what we have built is: a machine for raising the trial count. That is not a distortion of the product, it is a description of it, and the ordinary marketing move here is to talk about speed and let nobody make the connection.

Why say it out loud

Partly because it is true and somebody was going to work it out. Mostly because it is the reason the evaluation harness exists, and the argument does not land without it.

If you believe searching harder makes results less trustworthy, then a tool that searches very hard needs to be paired with something that knows how hard it searched. That pairing is not a feature we added on top. It is the shape of the thing: the reason we built a separate, deterministic evaluator at all is that we knew what our own generator was going to do to the trial count.

Leaving it out would also have been a specific kind of dishonest — the kind where every individual sentence is true and the picture assembled from them is wrong. We would rather be the ones to point at it.

What we actually do about it

Four things, and the first one is the only one that is unusual.

1. We count. Because the harness generates the candidates itself, the trial count is a by-product of the search rather than a number anyone has to remember and report honestly. This is the part almost nothing else in the category can do, and not because it is difficult — because it requires having kept the candidates you threw away.
2. We raise the bar with the count. That count feeds a deflated Sharpe ratio and a probability of backtest overfitting, so a strategy that survived a large search has to clear a higher threshold than one that survived a small search. Searching more makes the standard stricter, not the result better.
3. We hold data back by construction. The evaluation pipeline runs on data the proposing model never saw, and the separation is enforced by the system rather than by anyone’s discipline. Optional validation is validation that gets skipped exactly when the result already looks good.
4. We let the answer be no. The evaluator is deterministic code, it runs separately from the model that proposed the strategy, and nothing downstream retries until it passes. A pipeline that re-rolls until it gets a pass has no evaluator, only a delay.

The honest summary of all four: we did not solve the problem we created. We made it measurable, and then we made the measurement bind.

Backtested and hypothetical results are not indicative of future performance. Nothing here is investment advice.

What this costs us

It makes for a worse demo. The natural thing to show a prospective user is a strategy that passed, and a system built to be sceptical of its own output produces a lot of verdicts that say this does not hold up. That is the correct behaviour and it is a harder sell than a rising equity curve.

It also means we will sometimes be less impressive than a competitor on identical inputs. Given the same idea and the same data, a product that reports a raw Sharpe ratio will show a better number than one that reports a deflated Sharpe ratio, every single time. Ours is not a worse result. It is the same result with the search subtracted, and the search was always there.

We think that is the right trade, and we are aware that saying so before launch costs nothing. It will cost something later, which is the point at which it means anything.

What we still cannot do

A list, because a piece like this is worthless without one.

• We cannot tell you a strategy will work. Nothing can. Every number the harness produces describes how much a backtest is worth as evidence, which is a different question from what happens next.
• We cannot detect that an effect has decayed before it decays. If a real edge stops working because enough people found it, no amount of validation catches that in advance, and any product claiming otherwise is describing something it does not have.
• We cannot correct for mistakes, only for selection. A deflated Sharpe ratio adjusts for how many variants were tried. It does nothing about leaked data or a cost model that was wrong to begin with — those have to be caught separately, and they are the two most common reasons a backtest is nonsense.
• We cannot count the searching you did before you got here. If you arrive with an idea you have already tuned by hand for a month, our trial count starts at one, and it is wrong. That is a real limitation and we do not have a good answer to it.

The last one bothers us most, because it is the case where our own number is misleading in the flattering direction. We would rather have it written down here than discovered by a user.