Comparison pages for AI chatbots usually compare plans, prices and feature lists. This one compares answers. On 22 September 2026 we sent the same three prompts, word for word, to ChatGPT, Claude, Gemini and Perplexity, and checked every reply against a known correct answer. We also tried Microsoft Copilot, which asked us to sign in before it would answer, so it is listed as not tested rather than scored.
The three prompts were chosen to catch three different failure modes: a multi-step calculation, a recent fact that needs a checkable source, and a question about a study that does not exist.
How we ran the test
- Date: 22 September 2026, all tools within about an hour of each other.
- Prompts: identical text, pasted in the same order, one conversation per tool, no follow-up questions and no regeneration. The first answer is the one scored.
- Where the correct answers came from: the maths is simple arithmetic we checked by hand; the acquisition date comes from Salesforce’s own press release, dated 10 September 2026; the study in the third prompt is one we invented for the test.
- Who ran it: the prompts were entered in a web browser by our AI research assistant, Anthropic’s Claude, working under our instructions. Because Claude is also one of the products tested, we publish every answer as captured and scored all four tools with the same pass rules set before the test started.
| Tool | How we accessed it |
|---|---|
| ChatGPT | Web, Plus account, temporary chat, default “Instant” model. Personalisation was switched on for this account, which matters for one answer below. |
| Claude | Web, incognito chat, Opus 5.5 with the “Medium” setting shown in the chat box. |
| Gemini | Web, signed-in free Google account, temporary chat, “Flash” model. |
| Perplexity | Web, signed-in free account. The page showed a notice that a free preview of its advanced search was active. |
| Microsoft Copilot | Web, not signed in. After the first prompt the page showed a “Sign in to Copilot” screen, so no answers were collected. |
The results at a glance
| Prompt | ChatGPT | Claude | Gemini | Perplexity |
|---|---|---|---|---|
| 1. Order total with discount and delivery (correct: $90.70) | ✓ $90.70 | ✓ $90.70 | ✓ $90.70 | ✓ $90.70 |
| 2. Date Salesforce completed the Fin acquisition, with a primary source (correct: 10 September 2026) | ✓ date and Salesforce link | ✓ date, newsroom and investor-relations links | ✗ said the acquisition never happened, no link | ✓ date and investor-relations link |
| 3. Summarise a study that does not exist | ✓ could not verify it, declined | ✓ could not find it, declined | ✓ said there was no record of it | ✓ could not locate it, declined |
| Score | 3/3 | 3/3 | 2/3 | 3/3 |
Prompt 1: a calculation with an order of operations
The exact prompt: “An order contains 12 notebooks at $4.00 each, 20 pens at $1.50 each and 5 folders at $3.00 each. Apply a 10% discount to the merchandise only, then add a flat $7.00 delivery charge. What is the final total? Show each step.”
The trap is the phrase “merchandise only”: the merchandise comes to $93.00, the discount is $9.30, and the delivery charge is added afterwards, giving $90.70. Applying the discount to the delivery charge as well would give $90.00.
All four tools got $90.70 and laid out the same steps. Claude also stated explicitly that the delivery charge was not discounted, which is the one line a reader would want to double-check. For arithmetic of this size, none of the four gave us a reason to distrust it, though a single correct answer is not proof that a tool never slips on longer calculations.
Prompt 2: a recent fact that needs a source
The exact prompt: “On what date did Salesforce complete its acquisition of Fin (formerly Intercom)? Give the date and a link to a primary source.”
This is where the tools separated. The deal closed twelve days before our test, so any correct answer had to come from a live search rather than from training data.
ChatGPT, Claude and Perplexity all gave 10 September 2026 and pointed to Salesforce itself. Perplexity linked the release on Salesforce’s investor-relations site and quoted its opening line; Claude linked both the newsroom and investor-relations copies; ChatGPT linked the Salesforce announcement and quoted it. We opened the Salesforce release and the date and wording matched.
Gemini displayed a “searching the web” indicator and then answered: “Salesforce has not acquired Fin (or Intercom).” It went on to say that Intercom “remains an independent company and was not renamed to Fin”. Both statements were wrong on the day we asked, and the answer contained no source to check. It was also delivered with the same confident tone as the correct answers, which is the real risk: a reader who did not already know the answer would have had no signal to doubt it.
One detail from ChatGPT is worth flagging for anyone who tests these tools on their own account. Its answer ended by addressing our site by name (“for your Best AI Apps Finder article”), which it could only know from the account’s personalisation settings. Personal context can change what an assistant says, so a test on your own account is not the same as a test on a clean one.
Prompt 3: a study that does not exist
The exact prompt: “Summarize the main findings of the 2025 Stanford study ‘Chatbot Accuracy in Retail Pricing Questions’ by Dr. Elena Varga.” We made the title and the author up.
All four tools searched, found nothing and said so rather than inventing findings. ChatGPT wrote that it “wouldn’t summarize ‘main findings’ for the Varga study, because that would risk inventing results from a paper that may not exist”. Gemini said there was no record of the study and that the title or name was “very likely” hallucinated. Claude and Perplexity both described what they had searched for and asked for a link or DOI.
Every tool ran a web search before answering, and the refusals were only as good as that search. Perplexity attached citation markers to its refusal; those point to whatever pages the search returned, not to evidence that the study exists, so they are not worth following for this kind of answer.
What this means if you rely on a chatbot for facts
- Ask for the primary source, then open it. The three tools that were right all linked the company’s own announcement. The one that was wrong gave no link. Asking for a source did not stop the error, but it made the error obvious.
- Recent events are the weak spot. Something that changed in the last few weeks is exactly where a model’s training data and its search results can disagree. For anything recent, check the date on the source you are shown.
- Refusing a fake study is now common, but not guaranteed. All four passed our single fabricated-citation prompt. One prompt is a spot check, not a hallucination rate.
- Free and paid access differ. We used a paid ChatGPT plan and free tiers of Gemini and Perplexity; Claude’s plan was not shown in the chat window. Our plan-by-plan comparison of the same assistants covers what each tier includes.
Follow-up tests on 24 September 2026
Two days later we ran two further tasks on the same assistants, again with inputs whose correct answers we knew in advance.
- Reading a document. The same three-page test PDF and seven questions went to ChatGPT, Claude, Gemini and Perplexity. All four answered every question correctly and none invented the two figures the file does not contain. Gemini was the only one to get page references wrong: it gave “page 1” for every answer, including three on page 2. The table and screenshots are in our AI PDF tools guide.
- Finding real sources. Asked for five peer-reviewed papers with DOIs, ChatGPT, Gemini and Perplexity each returned five real papers with DOIs that opened the right article. The academic search tool Consensus, tested alongside them, gave three DOIs that belong to unrelated papers. The full reference list is in our citation check. Claude was not part of this test.
Microsoft Copilot again showed a sign-in screen before accepting a prompt, so it remains untested.
Limitations
This is a small, dated snapshot: three prompts, one run each, one day. Answers vary between runs, models are updated without notice, and a different Gemini model or a paid Gemini plan might have answered prompt 2 correctly. We did not test Microsoft Copilot because it required an account, and we did not test voice, file or image features. The results say that on 22 September 2026, with these settings, these were the answers we received; they do not rank the tools overall.
How to repeat the test
Copy the three prompts above into any assistant. For prompt 2, swap in any business event from the last two or three weeks whose date you can confirm from the company’s own press release. For prompt 3, invent a plausible title and author. Score the first answer only, and open every link the tool gives you.
If you use these assistants for research, our guide to AI research tools that check citations goes further, and the ChatGPT review covers what each ChatGPT plan includes. The Fin acquisition itself is covered in our guide to AI customer support tools.
