A model should not grade its own work

What the products actually say

Read the marketing copy in this category and a pattern comes out quickly. One of the largest platforms describes its AI agent this way, on its own homepage: it "builds limitless strategies", "backtests everything against history", and "optimizes live on the chart". Another sells the ability to "create thousands of algorithmic trading strategies at the click of a button" while promising to "identify lying backtests". Read in July 2026.

Every one of those sentences describes the same architecture: one system generates candidates, evaluates them, decides which are good, and reports back. The generating and the judging happen inside the same loop, optimizing the same objective.

Described as a product feature, that sounds like efficiency. Described as an experiment, it is a scientist grading their own submission — and we would not accept it in any other field that produces numbers.

Why this is worse than it sounds

The failure mode is not that the model lies. It is that the model does exactly what it was asked, and what it was asked is subtly the wrong thing.

Point an optimizer at "make this backtest look better" and it will make the backtest look better. There are many ways to do that, and only one of them is finding a real effect. The others include fitting the noise more closely, drifting the parameters toward whatever the sample happened to reward, quietly favoring the period where the idea worked, and — the most common — simply trying more variants until one lands.

None of that requires anything going wrong. It is the instruction being followed competently. The optimizer has no way to distinguish "this rule captures something real" from "this rule matches this history", because on the data it was given those two things produce an identical signal. Distinguishing them requires evidence the optimizer was never allowed to see.

And a language model brings a second problem that a parameter sweep does not. It has read the literature. It knows which strategies are famous, which periods were kind to them, and what a persuasive research write-up sounds like. Ask it to evaluate its own proposal and you get a fluent, well-structured, superficially rigorous case for the thing it just made up. The output is more convincing than a spreadsheet and no more reliable.

What separation actually requires

It is a stronger condition than it first appears, and most of the ways to do it partially do not work.

• The evaluator is code, not a model. A second model asked to critique the first is still a model that can be argued with, and it shares the failure modes of the first. Determinism is the point: the same strategy and the same data must produce the same verdict every time, or the verdict is negotiable.
• The proposing model cannot see the evaluation data. Not "is instructed not to use it" — cannot see it. An instruction is a request; a boundary is a boundary.
• The proposing model cannot see the grading criteria in enough detail to target them. A rubric handed to the thing being graded is a specification for how to pass.
• The evaluator counts the proposals. This is the one most systems cannot retrofit, because it requires having recorded every candidate the search generated, and most throw them away.
• The verdict is allowed to be negative, and nothing downstream is allowed to retry until it is not. A pipeline that re-rolls until it gets a pass has no evaluator, only a delay.

That last one is where most well-intentioned implementations fail. Separation is easy to build and easy to defeat, and the way it gets defeated is not by removing the check — it is by running the check repeatedly until it passes, which is the same overfitting problem wearing a lab coat.

Backtested and hypothetical results are not indicative of future performance. Nothing here is investment advice.

The thing that makes this checkable

The argument above is easy to assert and would be worth very little if it stopped there, because every vendor in this category can claim rigor and most do.

What makes it more than a claim is that it is a structural property rather than a quality one. You can ask a specific question about any product in this space, and the answer is either yes or evasive: does the thing that decides whether a strategy is good share an objective with the thing that produced it?

That question does not require trusting anybody. It is architecture, and architecture is describable. It is a fairer test than "are your results good", which nobody can verify before launch and which every vendor answers the same way.

So the honest way to read this essay is as an invitation to ask it of us too. We are pre-launch. Our results cannot be checked yet by anyone, including by us. The architecture can be described now, and it is the thing worth holding us to later.