What happens when an AI system is uncertain

The most important AI output may be: "I do not have enough evidence to do this safely."

Language models are built to continue text, and fluent language can hide weak evidence. A system that always produces an answer may look helpful in a demo while quietly converting uncertainty into invented facts.

Uncertainty therefore needs an operational response, not a disclaimer below the chat box.

Confidence is not one number

A workflow can be uncertain because:

• the request is ambiguous;
• required data is missing;
• sources conflict or are stale;
• retrieval found weak evidence;
• the case differs from evaluation data;
• a tool failed or returned an unexpected state;
• the model's own interpretation is unstable;
• policy does not cover the case.

A model-generated confidence statement does not reliably measure all of these. Research on question answering has repeatedly found calibration problems, especially when tasks differ from familiar training conditions. Kadavath and colleagues and Jiang and colleagues

Detect uncertainty from several signals

Use observable signals where possible:

• retrieval score and presence of an authoritative source;
• agreement across independent checks;
• required fields and validation rules;
• distance from known examples;
• model disagreement across repeated or alternative evaluations;
• tool errors and stale timestamps;
• policy-defined high-risk topics;
• user correction or reviewer override patterns.

Each signal has limitations. A high retrieval score can point to the wrong document. Several models can repeat the same misconception. Combine signals and test them on real cases.

Give the system four possible responses

An uncertain workflow should be able to:

1. ask: request a missing order number or clarify the user's meaning;
2. retrieve or verify: consult an approved source or deterministic system;
3. abstain: state that the available evidence is insufficient;
4. escalate: send the evidence and unresolved question to a named person.

For low-consequence drafting, the system might proceed with uncertainty clearly marked. For a payment, account deletion or eligibility decision, uncertainty should block action.

Make abstention useful

"I can't help" is safe but often useless. A good abstention says what is missing, which source was checked and what the next step is.

I found two policies with different return windows, dated March and July. I have not processed the refund. The case has been sent to the policy owner with both sources.

This response preserves progress without pretending the conflict is resolved.

Set thresholds from consequences

Do not use one confidence threshold for every action. A content tag can tolerate more error than a customer-facing price. Define separate thresholds for answering, recommending and executing.

NIST's AI Risk Management Framework calls for systems to fail safely beyond their knowledge limits, for human oversight to be defined, and for performance to be monitored under conditions similar to deployment. NIST AI RMF Core

The Government of Canada's agentic AI guide similarly recommends stop conditions, bounded authority, recoverability and escalation paths, especially when agents can take actions. Government of Canada

Do not confuse explanation with accuracy

A long explanation can make a weak answer feel more trustworthy. A 2024 Nature Machine Intelligence study found that explanations could increase people's confidence in AI advice without a corresponding gain in decision accuracy in the studied setting. It is one controlled study, not a universal effect, but it is a useful warning against judging truth by eloquence. Nature Machine Intelligence

Show evidence that can be inspected. Separate quoted source facts from the model's inference. For calculations and critical identifiers, use deterministic validation.

Monitor the uncertainty system

Track:

• abstention and escalation rates;
• false confidence, where wrong answers passed the threshold;
• unnecessary abstention, where safe cases were rejected;
• reviewer agreement and override;
• reasons for missing evidence;
• outcomes by user group, topic and data source;
• changes after model, prompt or source updates.

The aim is not zero uncertainty. It is to stop hiding uncertainty inside confident prose and to route it according to consequence.

Source check

Calibration studies use particular models, datasets and experimental designs. They support the need to test uncertainty handling, not a universal error rate. Thresholds must be validated on the deployed workflow.

Sources

• Kadavath et al.: Language Models (Mostly) Know What They Know
• Jiang et al.: How Can We Know When Language Models Know?
• NIST AI RMF Core
• Government of Canada: Guide on the use of agentic artificial intelligence
• Nature Machine Intelligence: Explanations and reliance on AI advice
• OpenAI: Why language models hallucinate