How a language model picks the next word
A language model is a program that, given some text, predicts what text is likely to come next. It does this one small piece at a time, called a token, roughly a word or part of one. At each step it has a list of candidates with a probability attached to each. "Gross margin was" might be followed by "41" with a 30 percent chance, "40" with a 25 percent chance, "about" with a 15 percent chance, and so on.
Sampling is the act of choosing from that list. The model does not always take the top candidate. It draws from the list in proportion to the probabilities, which is what makes the same prompt produce different sentences on different runs. One early choice, "about" instead of "41", changes every prediction that follows.
Temperature is the setting that controls how adventurous the draw is. At a high temperature the model spreads its bets. At zero it is supposed to take the top candidate every time, which should make it repeatable. It mostly does. Not entirely, which is the surprising part.
The experiment that got 80 answers from one question
On 10 September 2025 the research lab Thinking Machines published a test in which the same prompt was sent to a model 1,000 times at temperature zero, the setting that is meant to remove randomness. They got 80 unique outputs from the 1,000 completions, and the first divergence between them came at token 103.
The cause was not the sampling. It was arithmetic. A model's answer is the result of billions of floating-point operations, and when a server processes your request in a batch with other people's requests, the batch size changes the order in which some of those operations happen. Floating-point addition is not perfectly associative: adding three numbers in a different order can produce a result that differs in the last decimal place. That tiny difference occasionally flips which candidate token is on top, and from that point on it is a different answer. Thinking Machines showed the fix is possible, but it costs speed, and the public chat products do not promise it.
So when an owner pastes a P&L into ChatGPT on Monday, asks for gross margin, and gets a different figure on Tuesday, nothing went wrong in the ordinary sense. The tool worked as designed. The design does not guarantee the same number twice.
What the research says about finance tasks
A 2025 study posted to arXiv (paper 2503.16974) tested exactly this on finance and accounting tasks, running each task 50 times across several models and collecting more than 3.4 million outputs. The models were nearly perfectly consistent on classification, the kind of job where the answer is "this expense belongs in travel". On complex generation and prediction tasks, the kind where the model has to analyze, compute and conclude, the authors found greater variability. Aggregating three to five runs improved reliability. And newer models were not uniformly more consistent than older ones, which cuts against the assumption that the next release will fix it.
Even the easy jobs are only mostly right. A benchmark published by the accounting software company Digits on 5 June 2026 ran 13 frontier models over 2,000 real transactions; the best general-purpose model categorized 80.8 percent correctly in one pass and 86.8 percent with extra tooling. Digits sells a competing product, so weigh the framing, but one transaction in seven left to check is the shape of it.
What goes wrong in practice
The failures owners hit are rarely a wildly wrong total. They are plausible totals built on quiet mistakes.
A finance writer at the CFO Impulse newsletter tested ChatGPT-4o on a 200-row dataset of the kind a small company might export from its accounting system. The regional breakdown was wrong, and stayed wrong after the model was told about the error and asked to redo it. An "Other" bucket in the industry breakdown turned out to be hiding three whole industries. Getting a usable table took five rounds of prompting, and the writer's conclusion was that it was impossible to verify correctness from the output alone, because the model presents every version with the same confidence.
Suppose the owner of a $2,940,000 distribution business pastes the year-to-date P&L into a chat and asks for gross margin. On the first run the model puts freight-in below the line and reports 41.2 percent. On the second run, in a fresh chat with the same file, it treats freight as a cost of goods and reports 38.7 percent. Both are defensible accounting choices. Neither run says which one it made. The owner now has two margins, a bank that asked for one, and no way to pick without doing the calculation herself, which is what she was trying to avoid. An AI that computes can be wrong in ways that look right, and one that is inconsistent removes the check most people rely on, which is to ask again.
Hallucination, and why longer documents make it worse
A hallucination is an answer the model states as fact that is not supported by the material it was given, or by anything. It is a different problem from inconsistency, though they compound each other.
Vectara publishes a hallucination leaderboard that measures how often models invent content when asked to summarize a document. In the 11 May 2026 edition the best model scored 1.8 percent, with a small OpenAI model at 3.1 percent and a small Google model at 3.3 percent. That sounds low until you count questions. The November 2025 edition of the same leaderboard found that the reasoning models many owners now use by default, including versions of Claude, GPT and Grok, all scored above 10 percent, and that rates rise with the length and complexity of the document. A 40-line P&L is short. A general ledger export is not.
For context on how far this has moved, the FinanceBench study of November 2023 found that GPT-4 Turbo, given the relevant documents, answered incorrectly or refused on 81 percent of a set of financial questions. That figure is nearly three years old and describes models nobody uses now, so treat it as a marker of where things started rather than where they are. The direction of travel is good. The destination is not "zero".
What getting the same answer twice requires
The common advice is to write a better prompt. We think that misses the point, because prompting shapes which path the model takes and cannot remove the randomness that lives inside it, and because it puts the burden on the owner to become an expert in a tool that was sold as removing the need for one.
What works is a division of labor. The number is computed by ordinary software, from the ledger, the same way every time: revenue minus cost of goods, divided by revenue, using the accounts the bookkeeper set up. The language model is then asked to explain the number, not to produce it. Explanation is a task where variation is harmless. Two slightly different sentences about why margin fell are both fine. Two different margins are not.
This is the split Navigator uses. Figures come from the QuickBooks Online ledger through a read-only connection and are computed the same way on every refresh, and the AI, which is included on every plan, explains what they mean. Every figure in an answer opens to show the company and the transactions behind it, so "ask again" is replaced by "look at where it came from". The test we suggest to anyone comparing tools, including ours, is to ask the same question of the same data twice and see what comes back.
There are limits to this too. A number computed correctly from a wrong ledger is still wrong, and no amount of consistency fixes a bookkeeper who is six weeks behind.
How to test any tool
Ask it a question with a single correct answer, such as last month's total revenue for one company. Then ask again in a fresh session. If the two answers differ, you have learned what you need to know about that tool for arithmetic. If they match, ask where the number came from, and go and look. A tool that can point to the report or the transaction has earned some trust. One that can only restate the figure has not.
The same test applies to the connectors that now link ChatGPT and Claude to QuickBooks directly, which we cover in what the QuickBooks connectors read, what they can change, and how to keep them read-only. And before pasting anything at all, it is worth knowing which account you are using and what it does with your data.
Questions owners ask
Why does ChatGPT give different answers every time?
Because a language model chooses each word by probability rather than by rule, and a small early difference sends the rest of the answer down a different path. Even with randomness set to zero, the way servers batch requests changes rounding inside the model. A September 2025 test got 80 distinct answers from 1,000 identical prompts.
Is ChatGPT reliable for financial analysis?
For explaining a number you already have, yes. For computing the number from raw data, less so. A 2025 study of 3.4 million outputs found near-perfect consistency on simple classification and greater variability on complex analysis. In practice the errors are subtle: a regional total that is slightly off, a category that quietly absorbs three others.
How do I make ChatGPT more consistent?
Give it the arithmetic already done, and ask it to explain rather than compute. If you must have it compute, run the same prompt three to five times and compare; the same 2025 study found that averaging runs improves reliability. Better prompts help at the margin, but they do not remove the randomness, which lives inside the model.
What is an AI hallucination?
A hallucination is an answer that is stated confidently and is not supported by the source the model was given, or by anything at all. Vectara's leaderboard measures it on document summaries; in May 2026 the best model was at 1.8 percent, and the November 2025 edition found reasoning models above 10 percent, rising with document length.
How can I verify an AI's financial answer?
Ask it where the number came from and open that source. If it cites a report or a transaction, check the figure there. If it cannot point to one, treat the answer as an estimate. Then ask the same question a second time in a fresh chat. Two matching answers with a source are worth acting on. Two different ones are not.
Related
If you are deciding whether to connect a chatbot to your books at all, start with the ChatGPT and Claude QuickBooks connectors and how to keep them read-only. The privacy half of the question is in whether it is safe to upload financial statements to ChatGPT or Claude. And for what an AI tool costs next to a bookkeeper, a controller or a fractional CFO, see AI CFO, fractional CFO or bookkeeper at 25 to 50 staff.
If you want to run the ask-twice test on your own books, the trial connects read-only in about fifteen minutes and needs no card: navigatorhq.ai.
Published . Last updated . Reviewed by a CFO on the Navigator team.