ChatGPT and Claude can both run real Python on a price file you hand them. Left to answer on their own, either one can also describe a backtest, a return, a win rate and all, without a single line of that code ever touching a real price. Here is what their own docs say about the sandbox, a prompt that pushes them to actually run it, the traps that are specific to letting an AI do the work, how to check what comes back, and how Ethy’s backtester handles the same job.
November 16, 2026
15
min to read
The short answer
ChatGPT and Claude can run real Python code on data you give them, inside a sandboxed container with no live market feed. Left to answer directly, though, either one can describe a backtest, numbers, win rate and all, without running any code at all. Ask for the trade list, not just the summary, and check it.
Both assistants can genuinely run code. OpenAI’s Code Interpreter tool, the API side of what the ChatGPT app calls data analysis, gives the model “a container object,” which its docs describe as “a fully sandboxed virtual machine that the model can run Python code in,” with a memory tier you choose (1g, 4g, 16g or 64g). Claude’s code execution tool does the same job with Bash and Python together, in what its docs call “a secure, sandboxed environment,” pre-loaded with pandas, numpy, scipy, scikit-learn, statsmodels and matplotlib. Point either one at a real price file and ask it to test a rule, and it can write the loop, run it, and hand back a number that came out of actual arithmetic on actual prices.
The part that catches people out is that neither assistant runs code every time you ask it to. Claude’s own docs spell out the rule for when its sandbox fires: it runs code for “non-trivial math,” “data analysis, file parsing, or visualization,” “algorithm execution or simulation,” and “explicit requests to ‘run,’ ‘compute,’ or ‘execute.’” In cases such as “simple arithmetic and well-known math facts” and “factual, conversational, or creative requests,” it “answers directly without running code.” The docs add the line that matters most here: “If you want Claude to run code for a borderline request, ask explicitly.” A one-line “backtest this RSI rule on BTC” can read as a request to describe what a backtest would show, not a request to run one, and the model has no obligation to tell you which path it took.
How to backtest a trading strategy covers the method itself, the steps, the mistakes and a real worked example on NVDA. This guide is about the extra layer that shows up only when an AI assistant is the one running (or not running) the numbers. Reading a chart with AI has the same split, between a model that reads real data and one that estimates from a picture.
Only what you hand it, or what a separate tool fetches for it. OpenAI’s Code Interpreter container accepts a fixed list of file types: code files, data formats such as .csv, .json, .xml and .xlsx, documents, images and archives. There is no “get me BTC’s price history” button inside the sandbox; the price history is a file you upload or a table you paste. Claude’s separate file-creation feature, the one that turns a prompt into a spreadsheet or a chart, works the same way: its help article says it can “perform sophisticated data analysis and data science work” on files you give it, with “the maximum file size… 30MB per file for both uploads and downloads,” and on the Free, Pro, Max and Ultra plans its sandbox can also reach approved package sources such as PyPI and GitHub.
The code execution tool itself is more locked down again. Claude’s docs state plainly: “The container has no internet access, so Claude can’t download packages at runtime: only the pre-installed libraries are available.” That container also has limits worth knowing before you run anything long: 5 GiB of RAM, one CPU, and a 90-second wall-clock limit on each Python cell when the model calls tools from inside its code. OpenAI’s container has its own clock: “A container expires if it is not used for 20 minutes,” and an expired one can’t be revived, only replaced, which means files and variables from half an hour ago can simply be gone.
| Question | OpenAI Code Interpreter | Claude’s code execution tool |
|---|---|---|
| Internet access inside the sandbox | Set by the developer: off, or only the domains on an allowlist, per OpenAI’s API reference | Explicitly none: “no internet access” |
| File types it can read | .csv, .json, .xml, .xlsx, plus code, documents, images, archives | CSV, Excel, JSON, XML, images, text files |
| How long it keeps your files | Expires after 20 minutes idle; an expired container can’t be reused | Checkpointed after about 5 minutes idle; the container and its files last up to 30 days |
| What runs the numbers | Real Python, on the files in that container | Real Python and Bash, on the files in that container |
So a backtest built this way is only as current as the file you uploaded. Neither sandbox is quietly checking Hyperliquid or any exchange for you. If you want an assistant to pull live prices on the side and then reason about them, that is a separate browsing or search tool bolted on next to the sandbox, with its own accuracy problems: our guide to trading prompts covers what web search gets wrong about prices and dates.
Because producing a confident, well-formatted answer is exactly what these models are built and graded to do, and a backtest’s normal output, a return, a win rate, a drawdown, a little table, is unusually good at looking like it came from real work even when it didn’t. A September 2025 paper by researchers at OpenAI and Georgia Tech traces this to how models are trained and scored: “language models hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty,” because, in the authors’ words, “language models are optimized to be good test-takers, and guessing when uncertain improves test performance.” A specific number beats “I’d need to actually run this” on almost every scoring rubric a model was shaped by, even when the specific number is fiction.
Put that next to the default behavior from the first section. If your prompt reads as a question to answer rather than a program to run, Claude can skip the sandbox entirely and still return something that looks exactly like a backtest result: an entry count, a percentage, a sentence about drawdown. Nothing in the numbers themselves tells you whether a real loop over real rows produced those figures or whether the model pattern-matched on what a backtest like that “usually” shows. The same risk compounds with the data limits above: a sandbox with no file and no internet access has nothing real to compute from, so if you never supplied prices, any numbers that come back were not pulled from a market either.
The tell is simple, and it is the same one our backtesting guide uses: a return with no trade list behind it is not a backtest. Ask for every trade, dated and priced, and the exact rule that generated it. Code that ran against real rows can print that list in a second. A description of a plausible-sounding result usually can’t produce one without quietly generating it on the spot, which is itself a sign to rerun the request and watch what the model actually does this time.
Name the rule exactly, name where the data should come from, and say the word “run.” Each line below closes a door the model would otherwise leave open.
Backtest this rule using your code execution tool. Actually run the code; don’t describe what a backtest would probably show. Rule: [state the market, the timeframe, the exact entry and exit, the stop, and the position size]. Data: use only the file I’ve attached. Don’t invent, estimate or fill in any price. If the file doesn’t cover the dates or the market I named, stop and tell me instead of substituting anything. Costs: charge a commission and a slippage amount on every trade; state the numbers you used. Rules for the result: fill each trade at the next bar’s open after the signal, not the bar that produced it. Report the return, the max drawdown, the number of trades and the win rate next to buying and holding over the same dates. Show me: the code you ran, the full list of trades with their dates and prices, and the exact start and end date of the data you used. Flag anything you had to guess or any point where data was missing.
The “show me” line is the one to never skip. A model that ran real code will produce the trade list without friction, because it is reading it straight out of the output it already generated. A model that didn’t will often backfill one, and a backfilled list tends to have round numbers, suspiciously clean dates or a trade count that does not match how often the rule should have fired on real data, which is exactly what the next section is for.
The classic backtesting mistakes, overfitting, look-ahead bias and the rest, are covered in full in how to backtest a trading strategy. These three show up in a shape that is specific to handing the keyboard to an AI.
A fourth trap is more mundane: OpenAI’s container expiring after 20 minutes idle, or a fresh request simply opening a new container, means a long back-and-forth about one strategy can lose the file, the variables or the exact version of the rule you were just discussing, and the model may paper over the gap rather than tell you it had to start over.
Anthropic’s own guide to reducing hallucinations describes general techniques for catching a model that is guessing. Applied to a backtest, they turn into a short, concrete routine.
Not the summary stats alone. This is the backtest version of what Anthropic’s guide calls grounding a claim in a direct quote: real execution can show its work instantly, in detail, because the detail already exists.
Anthropic calls this “best-of-N verification”: “Run Claude through the same prompt multiple times and compare the outputs. Inconsistencies across outputs could indicate hallucinations.” Code running on the same file should return the same return, down to the decimal, every single time. If it doesn’t, something isn’t being computed the same way twice.
Anthropic’s guide calls this “external knowledge restriction”: telling the model to use only what you provided, not its general sense of how a rule like this “usually” performs, closes off the easiest place for an invented number to sneak back in.
Pick a couple of dates from the trade list and look up what the market actually did that day. This is the same check our backtesting guide recommends for a backtest a chat assistant ran, and it catches both a fabricated list and a real but buggy one.
A rule that should fire every few weeks but reports two trades over a year, or forty trades in a month, is telling you something is off before you even look at the return.
Here are the results of a backtest you ran for me earlier: [paste the summary, the code and the trade list]. Re-run the exact same code on the exact same file and compare the new result with the old one line by line. Then check: does every trade’s entry and exit price appear in the data file I gave you? Is any price, date or statistic in the summary missing from the code’s actual output? List anything that doesn’t match.
None of this replaces the out-of-sample test that makes any backtest, human or AI-written, worth trusting: running the unchanged rule on dates and markets it has never seen. That step, and the rest of the checklist for a backtest in general, is in our full backtesting guide.
Ethy skips the “did it actually run” question by design: you describe a rule in plain language, Ethy shows you the exact rules it read back, and the backtest it runs is the same spec that would deploy live or on paper, not a separate description of one. For the wider field of tools, see best backtesting software. There is no file to upload and no sandbox to lose; Ethy supplies the market data. Ethy’s AI backtesting tool walks through it screen by screen.
| Timeframe | Hyperliquid window | Spot token window |
|---|---|---|
| 5 minute | up to 15 days | up to 15 days |
| 15 minute | up to 90 days | up to 90 days |
| 1 hour | up to 180 days | up to 90 days |
| 4 hour | up to 365 days | up to 90 days |
| Daily | up to 365 days | up to 365 days |
Ethy presents a backtest’s result as an estimate on past data, not a promise, the same caution that applies to any backtest anywhere. It is software for research and analysis and doesn’t give investment advice.
Yes, if you give it a price file and explicitly ask it to run the code with its data analysis tool rather than describe a result. Without a file, there is nothing real to compute from. Always ask for the full trade list, not just the summary, and check a few trades against the file yourself.
Yes, through its code execution tool, which runs in a sandbox with no internet access and pandas, numpy and similar libraries pre-installed. Claude’s own docs say it skips running code for requests that look simple unless you ask explicitly, so say “run this” and ask for the code and the trade list back.
It is a sign that something wasn’t computed the same way twice. The same program on the same file returns the exact same number every time, so an answer that drifts between runs either came from different code each time or was generated fresh rather than read off a real calculation. Ask for the code both times and compare it, which is why running the same prompt twice is a useful check on its own.
No, not inside the sandbox itself. Both tools work on the files you upload or the text you paste: Claude’s API docs state its code execution container has no internet access, and OpenAI’s container reaches only the domains a developer allows, if any. Any live price feed comes from a separate browsing or search feature, with its own accuracy limits.
The same answer as any backtest: more than a couple, across rising, falling and flat stretches, the point our backtesting guide makes with its own two-trade NVDA example. A suspiciously round trade count, or one that doesn’t match how often the rule should fire on real data, is a reason to ask for the trade list and check it.
Yes. Describe the rule in plain English and Ethy backtests the exact spec it read back to you, on real Hyperliquid and spot-token history up to 365 days depending on the timeframe, with every trade marked on the chart and consistent costs charged on every fill.
Trading involves risk, including the loss of your money. Ethy AI is a research and trading tool and does not give investment advice or recommendations. Backtests are simulations on historical data and do not predict future results.
Deploy your AI trading agent in one click.
