Strategies and backtests

AI backtesting: what ChatGPT and Claude get right, and where they invent results

ChatGPT and Claude can both run real Python on a price file you hand them. Left to answer on their own, either one can also describe a backtest, a return, a win rate and all, without a single line of that code ever touching a real price. Here is what their own docs say about the sandbox, a prompt that pushes them to actually run it, the traps that are specific to letting an AI do the work, how to check what comes back, and how Ethy’s backtester handles the same job.

[

November 16, 2026

]
•
[

15

 

min to read

]

The short answer

ChatGPT and Claude can run real Python code on data you give them, inside a sandboxed container with no live market feed. Left to answer directly, though, either one can describe a backtest, numbers, win rate and all, without running any code at all. Ask for the trade list, not just the summary, and check it.

What happens when you ask ChatGPT or Claude to backtest a strategy?

Both assistants can genuinely run code. OpenAI’s Code Interpreter tool, the API side of what the ChatGPT app calls data analysis, gives the model “a container object,” which its docs describe as “a fully sandboxed virtual machine that the model can run Python code in,” with a memory tier you choose (1g, 4g, 16g or 64g). Claude’s code execution tool does the same job with Bash and Python together, in what its docs call “a secure, sandboxed environment,” pre-loaded with pandas, numpy, scipy, scikit-learn, statsmodels and matplotlib. Point either one at a real price file and ask it to test a rule, and it can write the loop, run it, and hand back a number that came out of actual arithmetic on actual prices.

The part that catches people out is that neither assistant runs code every time you ask it to. Claude’s own docs spell out the rule for when its sandbox fires: it runs code for “non-trivial math,” “data analysis, file parsing, or visualization,” “algorithm execution or simulation,” and “explicit requests to ‘run,’ ‘compute,’ or ‘execute.’” In cases such as “simple arithmetic and well-known math facts” and “factual, conversational, or creative requests,” it “answers directly without running code.” The docs add the line that matters most here: “If you want Claude to run code for a borderline request, ask explicitly.” A one-line “backtest this RSI rule on BTC” can read as a request to describe what a backtest would show, not a request to run one, and the model has no obligation to tell you which path it took.

How to backtest a trading strategy covers the method itself, the steps, the mistakes and a real worked example on NVDA. This guide is about the extra layer that shows up only when an AI assistant is the one running (or not running) the numbers. Reading a chart with AI has the same split, between a model that reads real data and one that estimates from a picture.

What data can ChatGPT or Claude actually use for a backtest?

Only what you hand it, or what a separate tool fetches for it. OpenAI’s Code Interpreter container accepts a fixed list of file types: code files, data formats such as .csv, .json, .xml and .xlsx, documents, images and archives. There is no “get me BTC’s price history” button inside the sandbox; the price history is a file you upload or a table you paste. Claude’s separate file-creation feature, the one that turns a prompt into a spreadsheet or a chart, works the same way: its help article says it can “perform sophisticated data analysis and data science work” on files you give it, with “the maximum file size… 30MB per file for both uploads and downloads,” and on the Free, Pro, Max and Ultra plans its sandbox can also reach approved package sources such as PyPI and GitHub.

The code execution tool itself is more locked down again. Claude’s docs state plainly: “The container has no internet access, so Claude can’t download packages at runtime: only the pre-installed libraries are available.” That container also has limits worth knowing before you run anything long: 5 GiB of RAM, one CPU, and a 90-second wall-clock limit on each Python cell when the model calls tools from inside its code. OpenAI’s container has its own clock: “A container expires if it is not used for 20 minutes,” and an expired one can’t be revived, only replaced, which means files and variables from half an hour ago can simply be gone.

What each sandbox can and can’t reach
QuestionOpenAI Code InterpreterClaude’s code execution tool
Internet access inside the sandboxSet by the developer: off, or only the domains on an allowlist, per OpenAI’s API referenceExplicitly none: “no internet access”
File types it can read.csv, .json, .xml, .xlsx, plus code, documents, images, archivesCSV, Excel, JSON, XML, images, text files
How long it keeps your filesExpires after 20 minutes idle; an expired container can’t be reusedCheckpointed after about 5 minutes idle; the container and its files last up to 30 days
What runs the numbersReal Python, on the files in that containerReal Python and Bash, on the files in that container

So a backtest built this way is only as current as the file you uploaded. Neither sandbox is quietly checking Hyperliquid or any exchange for you. If you want an assistant to pull live prices on the side and then reason about them, that is a separate browsing or search tool bolted on next to the sandbox, with its own accuracy problems: our guide to trading prompts covers what web search gets wrong about prices and dates.

Why can an AI-written backtest show numbers nobody computed?

Because producing a confident, well-formatted answer is exactly what these models are built and graded to do, and a backtest’s normal output, a return, a win rate, a drawdown, a little table, is unusually good at looking like it came from real work even when it didn’t. A September 2025 paper by researchers at OpenAI and Georgia Tech traces this to how models are trained and scored: “language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty,” because, in the authors’ words, “language models are optimized to be good test-takers, and guessing when uncertain improves test performance.” A specific number beats “I’d need to actually run this” on almost every scoring rubric a model was shaped by, even when the specific number is fiction.

Put that next to the default behavior from the first section. If your prompt reads as a question to answer rather than a program to run, Claude can skip the sandbox entirely and still return something that looks exactly like a backtest result: an entry count, a percentage, a sentence about drawdown. Nothing in the numbers themselves tells you whether a real loop over real rows produced those figures or whether the model pattern-matched on what a backtest like that “usually” shows. The same risk compounds with the data limits above: a sandbox with no file and no internet access has nothing real to compute from, so if you never supplied prices, any numbers that come back were not pulled from a market either.

The tell is simple, and it is the same one our backtesting guide uses: a return with no trade list behind it is not a backtest. Ask for every trade, dated and priced, and the exact rule that generated it. Code that ran against real rows can print that list in a second. A description of a plausible-sounding result usually can’t produce one without quietly generating it on the spot, which is itself a sign to rerun the request and watch what the model actually does this time.

What prompt gets an AI to actually run a real backtest?

Name the rule exactly, name where the data should come from, and say the word “run.” Each line below closes a door the model would otherwise leave open.

A backtest prompt that asks for the real thing

Backtest this rule using your code execution tool. Actually run the code; don’t describe what a backtest would probably show. Rule: [state the market, the timeframe, the exact entry and exit, the stop, and the position size]. Data: use only the file I’ve attached. Don’t invent, estimate or fill in any price. If the file doesn’t cover the dates or the market I named, stop and tell me instead of substituting anything. Costs: charge a commission and a slippage amount on every trade; state the numbers you used. Rules for the result: fill each trade at the next bar’s open after the signal, not the bar that produced it. Report the return, the max drawdown, the number of trades and the win rate next to buying and holding over the same dates. Show me: the code you ran, the full list of trades with their dates and prices, and the exact start and end date of the data you used. Flag anything you had to guess or any point where data was missing.

The “show me” line is the one to never skip. A model that ran real code will produce the trade list without friction, because it is reading it straight out of the output it already generated. A model that didn’t will often backfill one, and a backfilled list tends to have round numbers, suspiciously clean dates or a trade count that does not match how often the rule should have fired on real data, which is exactly what the next section is for.

What traps are specific to letting an AI run the backtest itself?

The classic backtesting mistakes, overfitting, look-ahead bias and the rest, are covered in full in how to backtest a trading strategy. These three show up in a shape that is specific to handing the keyboard to an AI.

  • Look-ahead bias hidden inside code you didn’t write. A model can compute an indicator across the whole file first and then slice out a test window, which lets a later price quietly shape an earlier signal. The bug looks identical in style to code running correctly, and you won’t catch it by reading the chart; you have to read the code and check which rows each calculation actually touches.
  • Placeholder prices dressed as real ones. Neither sandbox comes with a market-data feed, as the data-limits section above shows. Ask for a backtest with no file attached, and a model that wants to be helpful can write a price series that looks statistically reasonable and run a perfectly real loop over it, numbers computed honestly, on prices nobody ever quoted. The giveaway is whether it asked you for a file before it started, or announced the exact source and dates of what it used.
  • Overfitting with no record of the tries. Ask for “a few variations” in one chat and the model will happily test several, but unless you make it log every variant it tried, you only ever see the one it chose to show you. Our guide to quantitative trading walks through why the number of tries matters as much as the result: the best of several silent attempts needs far stronger evidence than a single rule written down in advance, and a chat transcript that only shows the winner hides exactly that count.

A fourth trap is more mundane: OpenAI’s container expiring after 20 minutes idle, or a fresh request simply opening a new container, means a long back-and-forth about one strategy can lose the file, the variables or the exact version of the rule you were just discussing, and the model may paper over the gap rather than tell you it had to start over.

How do you check an AI-written backtest before you trust it?

Anthropic’s own guide to reducing hallucinations describes general techniques for catching a model that is guessing. Applied to a backtest, they turn into a short, concrete routine.

  1. Ask for the code and the full trade list.

    Not the summary stats alone. This is the backtest version of what Anthropic’s guide calls grounding a claim in a direct quote: real execution can show its work instantly, in detail, because the detail already exists.

  2. Run the identical prompt a second time.

    Anthropic calls this “best-of-N verification”: “Run Claude through the same prompt multiple times and compare the outputs. Inconsistencies across outputs could indicate hallucinations.” Code running on the same file should return the same return, down to the decimal, every single time. If it doesn’t, something isn’t being computed the same way twice.

  3. Restrict it to only the data you gave it.

    Anthropic’s guide calls this “external knowledge restriction”: telling the model to use only what you provided, not its general sense of how a rule like this “usually” performs, closes off the easiest place for an invented number to sneak back in.

  4. Spot-check two or three trades by hand.

    Pick a couple of dates from the trade list and look up what the market actually did that day. This is the same check our backtesting guide recommends for a backtest a chat assistant ran, and it catches both a fabricated list and a real but buggy one.

  5. Check that the trade count makes sense.

    A rule that should fire every few weeks but reports two trades over a year, or forty trades in a month, is telling you something is off before you even look at the return.

Audit an AI-written backtest

Here are the results of a backtest you ran for me earlier: [paste the summary, the code and the trade list]. Re-run the exact same code on the exact same file and compare the new result with the old one line by line. Then check: does every trade’s entry and exit price appear in the data file I gave you? Is any price, date or statistic in the summary missing from the code’s actual output? List anything that doesn’t match.

None of this replaces the out-of-sample test that makes any backtest, human or AI-written, worth trusting: running the unchanged rule on dates and markets it has never seen. That step, and the rest of the checklist for a backtest in general, is in our full backtesting guide.

How does Ethy’s backtester work, and how is it different?

Ethy skips the “did it actually run” question by design: you describe a rule in plain language, Ethy shows you the exact rules it read back, and the backtest it runs is the same spec that would deploy live or on paper, not a separate description of one. For the wider field of tools, see best backtesting software. There is no file to upload and no sandbox to lose; Ethy supplies the market data. Ethy’s AI backtesting tool walks through it screen by screen.

  • Markets. Every perpetual future on Hyperliquid’s main exchange and on trade.xyz, with its stocks, indices and commodities, from a real candle archive, plus spot tokens priced from CoinGecko. A market Ethy can’t trade can still be tested as research only.
  • Windows. Up to 365 days, depending on the timeframe you pick; the free plan runs the shortest window of each timeframe, 180 days on the daily.
Ethy’s backtest windows by timeframe
TimeframeHyperliquid windowSpot token window
5 minuteup to 15 daysup to 15 days
15 minuteup to 90 daysup to 90 days
1 hourup to 180 daysup to 90 days
4 hourup to 365 daysup to 90 days
Dailyup to 365 daysup to 365 days
  • A sized account, not an abstract percentage. By default a $5,000 simulated balance with $1,000 of margin per position; give Ethy only a balance and each position gets a fifth of it as margin, or give it only a per-position amount and the balance is five times that. Liquidation is modeled at the market’s real Hyperliquid cap.
  • Costs charged the same way for everyone. A trading cost and a small amount of slippage are charged on every simulated fill, at one consistent rate regardless of plan. User backtests don’t model funding on perpetual futures; if you’re testing a rule that holds a perp for days or weeks, read that cost in separately, the way our quantitative trading guide does for its own worked example.
  • The numbers a real backtest should show. Return against buy and hold, trade count, win rate, profit factor, average and largest win and loss, the longest losing streak; max drawdown, Sharpe, Sortino and Calmar; a monthly breakdown and the market’s regime at each entry; the equity curve and the drawdown curve. Every entry and exit is marked on the chart.
  • A built-in out-of-sample check. Ethy’s optimizer tests thousands of parameter combinations against real market history to surface the best-performing setups, ranks them on a training window, and checks each one on a held-out window it never tuned against, the split this guide’s checking section asks you to do by hand with an AI’s output.
  • Unlimited runs. Every plan can run as many backtests as you want; a failed run is refunded automatically.

Ethy presents a backtest’s result as an estimate on past data, not a promise, the same caution that applies to any backtest anywhere. It is software for research and analysis and doesn’t give investment advice.

Summary

  • ChatGPT’s data analysis (Code Interpreter in OpenAI’s API) and Claude’s code execution tool can both run real Python against data you give them, in a sandboxed container with no live market feed of its own.
  • Claude’s own docs say it answers simple requests directly, without running code, and that for a “borderline” request you should ask explicitly; a one-line backtest request can get a description instead of a computation.
  • A clean-looking return, win rate and drawdown prove nothing about how they were produced. Ask for the code and the full trade list every time.
  • The AI-specific traps are look-ahead bias hidden inside code you didn’t write, placeholder prices dressed up as real ones, and overfitting with no record of how many variations were tried.
  • To check a result: ask for the trade list, rerun the identical prompt and compare, restrict the model to only the data you gave it, and spot-check a couple of trades by hand.
  • Ethy’s backtester runs the exact spec you’d deploy, on real market history, with consistent costs and a built-in train-and-test split, and presents results as an estimate, not a promise.

Questions people ask

Can ChatGPT backtest a trading strategy?

Yes, if you give it a price file and explicitly ask it to run the code with its data analysis tool rather than describe a result. Without a file, there is nothing real to compute from. Always ask for the full trade list, not just the summary, and check a few trades against the file yourself.

Can Claude backtest a trading strategy?

Yes, through its code execution tool, which runs in a sandbox with no internet access and pandas, numpy and similar libraries pre-installed. Claude’s own docs say it skips running code for requests that look simple unless you ask explicitly, so say “run this” and ask for the code and the trade list back.

Why did my AI-written backtest give a different answer the second time I asked?

It is a sign that something wasn’t computed the same way twice. The same program on the same file returns the exact same number every time, so an answer that drifts between runs either came from different code each time or was generated fresh rather than read off a real calculation. Ask for the code both times and compare it, which is why running the same prompt twice is a useful check on its own.

Does ChatGPT or Claude have real market data for a backtest?

No, not inside the sandbox itself. Both tools work on the files you upload or the text you paste: Claude’s API docs state its code execution container has no internet access, and OpenAI’s container reaches only the domains a developer allows, if any. Any live price feed comes from a separate browsing or search feature, with its own accuracy limits.

How many trades should an AI-written backtest show before you trust it?

The same answer as any backtest: more than a couple, across rising, falling and flat stretches, the point our backtesting guide makes with its own two-trade NVDA example. A suspiciously round trade count, or one that doesn’t match how often the rule should fire on real data, is a reason to ask for the trade list and check it.

Can Ethy backtest a trading idea for me?

Yes. Describe the rule in plain English and Ethy backtests the exact spec it read back to you, on real Hyperliquid and spot-token history up to 365 days depending on the timeframe, with every trade marked on the chart and consistent costs charged on every fill.

Sources

  1. OpenAI API docs: Code Interpreter tool (container, file types, 20-minute expiry)developers.openai.com
  2. OpenAI API reference: Containers (network policy)developers.openai.com
  3. ChatGPT docs: Use ChatGPT (file uploads, ChatGPT Work)learn.chatgpt.com
  4. Claude docs: Code execution tool (sandbox, no internet access, when Claude runs code, resource limits)platform.claude.com
  5. Claude Help Center: Create and edit files with Claude (file size limits, data analysis)support.claude.com
  6. Claude docs: Reduce hallucinations (verification techniques)platform.claude.com
  7. Kalai, Nachum, Vempala and Zhang, Why Language Models Hallucinate (arXiv, September 2025)arxiv.org

Trading involves risk, including the loss of your money. Ethy AI is a research and trading tool and does not give investment advice or recommendations. Backtests are simulations on historical data and do not predict future results.

Ethy blog

Read more

The $ETHY ecosystem

Stop watching charts.
Start living.

Deploy your AI trading agent in one click.