Artificial intelligence (AI) promises to make accountants faster, but in tax and finance the stakes are accuracy. Experts warn that generic AI chatbots like ChatGPT or public copilot tools, often produce answers that are outdated, misleading or incorrect without careful prompting and oversight. In other words, speed is pointless if the results aren’t reliable.

The pitfalls of Generic AI

Using off-the-shelf AI can introduce serious risks. For example:

• Outdated or incorrect data: Many AI chatbots such as OpenAI’s GPT based ChatGPT are trained on broad internet sources and can cite stale or incomplete information. In a BBC report, 19% of AI answers introduced factual errors. ICAEW cautions that AI lacks the nuanced understanding required for complex tax planning and compliance, particularly in areas with rapidly changing regulations.
• Fabricated (hallucinated) answers: AI can confidently invent plausible but false facts. In a UK High Court case (Al-Haroun v Qatar National Bank [2025] EWHC 1383), a lawyer submitted 18 non-existent case citations generated by AI. In a recent tax case, HMRC v Gunnarsson [2025] UKUT 247 (TCC) an unrepresented taxpayer used a chatbot to justify his argument, citing three fake cases and was rapped by the tribunal judge for wasting HMRC’s time researching non-existent cases. These examples highlight a broader, global pattern of AI hallucinations in professional contexts.
• Lack of nuance:  AI can handle straightforward queries reasonably well, but it often struggles with context and complexity. In areas such as tax, audit, or accounting, where rules are highly technical and depend on precise circumstances, AI may fail to provide accurate or reliable analysis, leading to oversimplified or misleading conclusions. In practice, this means an AI’s answer might violate subtle rules or misapply a regulation.
• Unverified output: Generic AI models typically do not cite their sources, meaning their conclusions cannot be independently traced or validated. Without verifiable references, users must treat outputs as unsubstantiated claims rather than authoritative information, which poses risks in fields that demand accuracy, such as law, tax, auditing, and accounting.

In domains built on accuracy, trust, and compliance such as tax and finances, these problems matter. Regulatory bodies such as the HMRC, Financial Reporting Council (FRC) and other professional standard setters require that all work is verified.

Trust through verifiable sources

The real game changer in AI is using models built on verified, expert-curated data, rather than the fastest internet scrape. Croner-i’s AI solution, for example, leverages decades of expert-written tax, audit and accounting content, providing responses with links to statutes, case law, and detailed analysis, ensuring each answer is accurate, reliable, and verifiable.

This stands in contrast to generic chatbots, which act as ‘black boxes’ and cannot show the sources or reasoning behind their outputs.

Key takeaways: Professionals should treat AI like a research assistant, not an oracle. Always cross-check AI-generated answers against authoritative sources. Prefer specialised platforms or RAG (retrieval-augmented generation) powered systems that surface and cite real tax and accounting expertise. In the end, the value of AI comes from quality, verifiable insight, not just raw speed.

Find out more about Croner-i AI-Powered Search

About the author

Jude Fletcher is director of search and generative AI, Croner-i

Articles you may like

AI Essentials, part 1: the shift in accounting and tax research | 20 Aug 2025

AI chatbot hallucinates cases in failed £13k tax appeal | 19 Aug 2025

Barrister ridiculed for using AI to make up cases at High Court | 9 May 2025