Council Post: Hallucinations: Why You Might Be Using The Wrong Kind Of AI
Kunal Chopra is the CEO of Certivo, an AI-powered compliance system of record for global manufacturers.gettyIn June 2023, New York attorney Steven Schwartz used ChatGPT to research a brief in Mata v. Avianca, I...
Kunal Chopra is the CEO of Certivo, an AI-powered compliance system of record for global manufacturers.
getty
In June 2023, New York attorney Steven Schwartz used ChatGPT to research a brief in Mata v. Avianca, Inc. The AI delivered fabricated precedents that read as legitimate. Judge P. Kevin Castel sanctioned Schwartz and his firm, and the case became a cautionary tale across the legal profession.
Mata was not an isolated incident. There have been more than 1,700 cases globally of attorneys filing AI-generated briefs with fabricated citations in the 30 months since, and the American Bar Association issued its first formal ethics opinion on lawyers' use of generative AI in July 2024.
Executives are caught in the middle here. Boards often push for AI adoption, but risk officers urge caution. The confusion starts with treating "AI" as a single thing. The hallucination problem belongs to one specific kind of AI doing work it was never built for.
The Category Error At The Heart Of The AI Debate
When most people say "AI" today, they mean generative AI: large language models that produce text by predicting the probabilistic next tokens. Models like GPT-4 and Claude can draft contracts, summarize long documents and write working code. The same mechanism that makes them creative makes them unreliable when accuracy matters. They produce outputs that statistically resemble correct outputs, which is often good enough and occasionally catastrophic.
Generative AI is one branch of a larger family. Another tool, often called deterministic AI, works differently. It recognizes patterns and applies rules, matching inputs against verified knowledge (regulations, substance lists, reporting thresholds, etc.) and returning answers that trace back to a source.
Deterministic AI cannot hallucinate. With no mechanism to invent facts outside its knowledge base, it tells you when the answer isn't there.
Different AI For Different Jobs
For some applications, generative AI is exactly the right tool. Drafting a marketing email rewards speed and fluency, and an imperfect phrase is just an edit away.
In a compliance department, that same technology becomes a liability. A fabricated citation produces audit findings, fines and recalls. Compliance work requires applied accuracy, and deterministic AI is built for it.
Compliance isn't unique. The same logic applies anywhere outside parties hold you to defined rules: tax calculation, financial reporting, drug interaction checking, sanctions screening, claims adjudication, contract review and more. Generative AI is the wrong tool for any of them.
"AI adoption" is not one decision but rather dozens of decisions, each about which kind of AI fits which task. Choosing the wrong AI tool can come with a high cost.
The Hybrid Architecture Leaders Should Ask For
The strongest AI deployments I see today combine generative and deterministic systems, using each for the work it's built for.
A generative model can serve as the natural-language interface, translating a compliance officer's plain-English question into a structured query. The answer to the question itself (e.g., "What does regulation X require for substance Y in market Z?") can then come from a deterministic engine running against a verified knowledge base.
This architectural choice may separate the AI tools that survive in regulated industries from those that don't. Generative AI alone is too risky for high-stakes work; deterministic AI alone is too rigid for how humans actually communicate. Combined, they can produce AI that is both accessible and accountable.
Why Compliance Is The Proving Ground
At my company, our customers face this distinction daily. They sell into a regulatory landscape that grows more complex every year: They have to consider REACH, RoHS, Prop 65, PFAS restrictions, conflict minerals disclosures and mounting state-level chemical reporting rules. An incorrect determination can cost a product its market access, halt a shipment or trigger a recall.
When a manufacturer asks whether a product can ship into the European Union next quarter, the answer must be accurate and traceable to its source: It must cite the specific regulation, data points and rules that produced it. That kind of accountability cannot be retrofitted; it has to be built in from the start.
The Data Problem Most Compliance Software Can't Solve
Compliance data lives in supplier declarations, safety data sheets formatted dozens of different ways, technical specifications written for engineers, emails, supplier portals and product files spanning thousands of components across hundreds of suppliers in dozens of jurisdictions. The work is unstructured by nature.
Legacy compliance software was built on the opposite assumption. It requires clean, structured inputs and organizes work by regulatory framework, with separate modules for REACH, RoHS, conflict minerals and the rest, each treating compliance as a discrete problem. Teams have to clean supplier data before the software can use it, and a product assessed across five frameworks gets assessed five times. Compliance functions spend more time wrangling data than acting on it.
This is what the right AI architecture is for. A well-built deterministic system can read the unstructured documents teams already receive, pattern-match them against applicable regulations and connect compliance across every framework that touches a product.
A Better Question For The Next AI Pitch
Executives no longer need to debate whether to deploy AI in compliance work. Instead, they must determine which AI deserves their trust and where to deploy it. Regulatory change accelerates yearly, and new supply chains routinely add countries, tiers and substances to their infrastructure. The legacy compliance stack cannot keep pace.
When a vendor pitches an AI solution for compliance, supply chain, financial reporting or any domain where being approximately right means being entirely wrong, ask, "What happens when the answer isn't in your data?" If the system invents an answer anyway, you have your answer. That's not the AI you want anywhere near regulated decisions.
The leaders who get AI right will be the ones who can tell one architecture from another—and deploy each where it earns its place.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?