Artificial intelligence can help professionals research faster, summarize complex information, draft documents, and automate repetitive work. But in regulated industries, an AI-generated mistake can carry consequences that go far beyond an embarrassing error.
Two lawyers in Oregon recently learned that lesson in unusually costly fashion. As reported by the ABA Journal, a federal judge imposed roughly $110,000 in sanctions and attorney fees after court filings contained AI-generated legal authorities that did not exist.
The court found that three briefs contained citations to 15 nonexistent cases and fabricated quotations attributed to eight legitimate authorities. The court ultimately struck the briefing and dismissed the plaintiffs’ claims with prejudice. A subsequent order allocated more than $94,000 in attorney fees between two attorneys, in addition to monetary sanctions imposed against lead counsel.
The case is a particularly visible example of a much broader problem: AI doesn’t transfer professional accountability from the person using the tool to the tool itself.
For law firms, accounting firms, financial advisors, and other organizations entrusted with sensitive information or regulated work, that distinction should shape how AI is implemented.
The real AI risk is unchecked output
Generative AI is very good at producing convincing language. That’s precisely what makes errors dangerous.
A traditional software error may produce an obvious failure message or an incorrect calculation that’s relatively easy to spot. Generative AI can instead produce an answer that’s polished, detailed, and authoritative, but wrong.
In the Oregon case, some of the fabricated material survived across multiple filings even after opposing counsel had identified nonexistent cases. The problem therefore went beyond the initial generation of inaccurate information. It also involved verification, supervision, escalation, and accountability.
Regulated organizations should take note. Training employees to write better prompts helps, but it’s only a start. Organizations need processes that determine where AI may be used, what information may be entered into it, which outputs require verification, and who ultimately owns the result.
Why law firms face particularly high AI stakes
Law firms are a clear example because their work depends heavily on accuracy and confidentiality.
The American Bar Association’s Formal Opinion 512 makes clear that lawyers’ existing ethical duties continue to apply when they use generative AI. Those duties include competence, client confidentiality, supervision, candor toward courts, and appropriate review of AI-produced work.
An AI system might save hours drafting a memorandum or summarizing discovery. But the lawyer remains responsible for determining whether the work is accurate. The same applies to confidential information: entering privileged or client-related information into an inappropriate AI platform can create risks even when the resulting answer is correct.
A policy that just says “use AI responsibly” won’t help a lawyer decide whether a tool is safe for privileged information. The policy needs to name approved tools, set limits on what goes into them, and make clear who reviews the work before it’s filed.
Accounting firms face the same accuracy problem in a different form
For accounting firms, the fabricated-citation problem has an obvious parallel.
AI can generate a convincing explanation of a tax issue, summarize a financial document, help draft client correspondence, or analyze large volumes of information. But a plausible answer isn’t necessarily a supportable one.
An inaccurate assumption can make its way into a calculation, or a summary can leave out an important exception. AI-generated explanations also tend to sound more certain than the underlying guidance supports. And putting client financial data into unapproved tools raises the same confidentiality concerns law firms face.
Banning AI outright would give up real time savings. Sorting workflows by risk keeps those savings where the stakes are low. Using AI to rewrite an internal email is very different from relying on it to produce conclusions that will affect a tax filing, audit work, financial reporting, or advice delivered to a client.
The higher the stakes, the closer the human review needs to be.
Financial services firms face added regulatory scrutiny on AI governance
Financial services organizations face another layer of complexity because existing regulatory obligations still apply when AI becomes part of a workflow.
FINRA’s 2026 regulatory guidance explicitly notes that its rules remain applicable when member firms use generative AI. It identifies considerations involving supervision, communications, recordkeeping, cybersecurity, accuracy, governance, testing, monitoring, and human review.
Investment advisers have faced scrutiny as well. The SEC has previously brought enforcement actions over misleading claims concerning firms’ use of AI, reinforcing that introducing AI doesn’t reduce existing expectations for accurate representations and effective compliance controls.
For financial services firms, the question is whether they can explain how an AI tool is being used, what information it accesses, how its outputs are reviewed, and what controls govern the process.
AI governance must come before AI scale
Many organizations already have employees experimenting with ChatGPT, Microsoft Copilot, and other AI applications. Waiting until everyone has selected their own tools and developed their own practices before establishing governance makes the problem harder.
Xantrion’s guidance on why AI readiness is an executive issue in 2026 makes the same point. Leaders need clear ownership, an understanding of data and security readiness, measurable use cases, and defined human review.
For regulated organizations, a practical AI governance program should establish:
- Approved tools and use cases. Identify which AI platforms employees may use, and which activities require additional approval.
- Data-handling rules. Define what confidential, client, financial, personally identifiable, or proprietary information may, and may not, be provided to AI systems.
- Human-review requirements. Specify which outputs need independent verification before they can influence client advice, filings, financial decisions, or external communications.
- Clear ownership. Assign responsibility for approving tools, investigating problems, updating policies, and making escalation decisions.
- Vendor and security review. Understand retention, access controls, model training practices, permissions, auditability, and other relevant security considerations.
- Ongoing training and monitoring. Give employees practical examples of acceptable use, and revisit controls whenever tools, business processes, and regulations change.
A formal governance structure doesn’t have to become another layer of bureaucracy. Xantrion’s AI Steering Committee template, for example, is designed to document decision authority, approval criteria, risk thresholds, responsibilities, and escalation processes.
Safe AI adoption doesn’t mean avoiding AI
Even the Oregon State Bar sees value in AI. Its general counsel has noted that AI can provide meaningful efficiency when it’s integrated appropriately, but that human review remains essential.
An organization that responds to AI risk simply by banning the technology may drive experimentation underground, creating “shadow AI” that leadership can’t see or manage. A better approach is to give employees a controlled environment, approved workflows, practical guidance, and clear boundaries.
Xantrion has collected additional AI resources for business leaders covering strategy, implementation, governance, security, and industry-specific considerations for legal and financial services organizations. The recommended approach starts with readiness and governance and expands from lower-risk applications into more consequential use cases as appropriate controls mature.
For organizations that have already started experimenting and now need more consistency, Xantrion’s AI Enablement Sprint provides a structured way to establish AI usage guidelines, data-handling expectations, human-review requirements, role-specific workflows, and a centralized prompt library.
AI doesn’t eliminate accountability
The most important lesson for regulated firms is straightforward: AI can accelerate the work, but it can’t assume responsibility for the work.
Lawyers still need to verify their legal authorities, and accountants still need to support their conclusions. Financial professionals answer to the same supervisory and regulatory requirements they always have. Leadership remains responsible for protecting sensitive information and showing that appropriate controls exist.
The organizations that benefit most from AI are unlikely to be those that allow employees to use it everywhere as quickly as possible. They will be the organizations that know where AI belongs, where it doesn’t, and what safeguards have to be in place before its output becomes part of a consequential business decision.
