AI SafetyAI SecurityDuty to WarnPrivacyThreat Detection

When ChatGPT Sees a Threat: What Duty to Warn Should AI Companies Have?

Yoni Fraimorice

22/9/2026

British Columbia is suing OpenAI over what it says was a missed chance to warn police before the February 2026 mass shooting in Tumbler Ridge.

The province says OpenAI's safety systems flagged violent ChatGPT activity months before the attack, but the company did not contact law enforcement. The lawsuit argues that one warning could have helped prevent the deaths of eight people.

These are legal allegations, not proven findings. British Columbia's attorney general has also said she has not seen the conversations and that OpenAI has refused the province's request to disclose them.

OpenAI has confirmed an important part of the timeline. In a February letter to Canadian ministers, the company said it detected and banned one account in June 2025 after automated and human review. It said the available activity did not meet its threshold for "credible and imminent planning." OpenAI later found a second account and said that, under its updated process, it would now refer the first account to law enforcement.

This creates a hard question for every AI provider: when a system detects a possible plan for real-world violence, what duty does the company have to warn someone?

Detection is not the same as understanding

A threat-detection pipeline can scan millions of conversations for signals such as weapons, targets, timing, maps, repeated violent questions, or attempts to bypass safety controls.

But each signal is weak by itself. A student may be writing fiction. A journalist may be researching an attack. A victim may be describing a threat made against them. A user may quote violent text while asking for help.

The useful unit is not one message. It is a pattern:

text

violent content
  + real target
  + access to means
  + operational detail
  + time pressure
  + repeated behavior
  = higher escalation priority

Even this is not a formula for guilt. It is a queue for trained human review.

OpenAI's letter shows why context matters. The company said its newer policy is more flexible because a user may never state the target, means, and timing in one clear sentence. Risk can appear across many conversations or accounts.

That means a serious pipeline needs four stages.

1. Detect behavior, not forbidden words

A keyword filter will create too many false alarms and miss coded language. Detection should combine conversation content with behavior: repeated planning, attempts to get around refusals, account recreation after a ban, and growing detail over time.

The model that chats with the user should not make the final reporting decision. It may be persuasive, inconsistent, or easy to manipulate. A separate safety system should score the activity and preserve the evidence used for review.

2. Give reviewers enough context

A reviewer cannot judge one isolated screenshot. They may need the relevant conversation history, prior policy violations, linked accounts, approximate location, and the reason each automated system raised an alert.

More context can improve safety, but it also increases privacy risk. Access should therefore be limited, logged, and time-bound. Reviewers should see only information needed for the decision.

3. Use a written escalation policy

"Use judgment" is not an operational policy. A provider needs clear levels:

yaml

level_1:
  action: block harmful assistance

level_2:
  action: human safety review

level_3:
  action: senior review and evidence preservation

level_4:
  action: emergency disclosure to the correct authority

The policy should define the evidence required at each level, who can approve escalation, how disagreement is handled, and how quickly a case must move.

The highest-risk decision should require two trained people, including someone who understands local law and emergency services. A reviewer should also be able to escalate uncertainty when the possible harm is severe.

4. Record the decision

The most important output may be a short decision record:

json

{
  "risk": "credible threat to others",
  "evidence": ["identified target", "weapon access", "recent planning"],
  "decision": "refer",
  "approvers": 2,
  "disclosed": ["account identifier", "relevant messages", "location signal"]
}

This record helps later audits answer the real questions: What did the company know? When did it know it? Which rule was applied? Who overruled whom? Was the disclosed data limited to what authorities needed?

Privacy law is not simply a reason to stay silent

Privacy protection and public safety are both real obligations.

Canada's PIPEDA permits an organization to disclose personal information without consent when an emergency threatens a person's life, health, or security. It also permits an organization to contact a government institution when it has reasonable grounds to believe information relates to a law that has been, is being, or is about to be broken.

Permission is not the same as a general legal duty to report. The law still leaves difficult questions about thresholds, jurisdiction, and responsibility. OpenAI may hold data in one country about a user in another, while the possible target is somewhere else.

A broad reporting rule would be dangerous. It could turn private AI services into mass-surveillance systems, flood police with weak alerts, and discourage people from seeking help. It could also harm users discussing intrusive thoughts without any plan to act.

A rule that is too narrow has the opposite failure: a company can identify a serious pattern, ban an account, and then treat the danger as somebody else's problem.

A narrow duty to warn is the better line

AI companies should not have to report every violent conversation. They should have a duty to act when all four conditions are met:

The threat concerns serious physical harm to another person.
The available evidence makes the threat credible, not merely disturbing.
The provider has enough identifying or location information for a useful warning.
A trained human review confirms the threshold, or the risk is so urgent that an emergency process applies.

Regulators should support this duty with a legal safe harbor for good-faith, minimal disclosures. In return, providers should publish aggregate numbers: cases reviewed, referrals made, false positives found, review times, and policy changes after incidents.

Independent auditors should test the whole pipeline, including account-ban evasion and disagreements between reviewers. They should not receive unrestricted access to ordinary user chats.

The real failure mode is an unresolved alert

The Tumbler Ridge case will test whether existing negligence and product-liability law can reach an AI company that allegedly saw warning signs but did not report them.

Whatever the court decides, the engineering lesson is already clear. A safety classifier is not a safety system. Detection only matters when it connects to context, trained people, a deadline, a documented decision, and a lawful path to the right authority.

AI providers should not become police. They also should not build systems that can recognize a credible threat, close the alert, and leave nobody responsible for what happens next.

Sources

Hero image: FEMA emergency operations briefing, photo by Mark Wolfe, public domain.