There is a difference between asking AI to help you brainstorm a blog post and asking AI to help determine whether a foreign government is transporting nuclear-weapons material.
That difference should be obvious.
Unfortunately, apparently it wasn’t obvious enough.
According to reporting published by CNN on September 18, 2026, the U.S. military came dangerously close to intercepting a Chinese ship in the Middle East after an intelligence report incorrectly claimed that the vessel was carrying components connected to a nuclear weapons program. The report had been produced with the help of an AI chatbot, which apparently misidentified what the ship was actually carrying.
And this wasn’t some harmless little AI mistake.
The military reportedly had aircraft in the air and armed personnel preparing to board the Chinese vessel before officials took a closer look at the underlying intelligence and discovered that the report was wrong. According to one source cited by CNN, the report was “entirely false.” Another source described the incident as having “almost started a war.”
Let that sink in for a second.
An AI system got something wrong.
Humans trusted the resulting information.
That information entered an intelligence process.
Military personnel began acting on it.
And the mistake was discovered only shortly before the planned operation.
That is an entirely different category of AI failure from ChatGPT giving you the wrong date for a historical event.
This is potentially life-or-death shit.
And this is exactly why I keep saying that AI needs human judgment
I use AI.
Quite openly.
I use it to help me develop ideas, organize thoughts, research things, brainstorm, revise material, and work through my opinion pieces. Sometimes I’ll throw a half-formed idea at AI and use the conversation to figure out what the hell I’m actually trying to say.
But here’s the important distinction:
AI helps me make things. It does not make my decisions for me.
I’m still the person responsible for what I publish.
I’m still supposed to question things.
I’m still supposed to verify information.
And if AI tells me something that sounds questionable, I don’t get to shrug and say, “Well, the robot said it.”
That’s not how responsibility works.
And it certainly shouldn’t be how intelligence or military decision-making works.
AI can be incredibly useful. It can process enormous amounts of information. It can identify patterns. It can summarize material. It can help humans work faster.
But one of the fundamental problems with generative AI is that it can produce something that sounds completely fucking convincing while being completely wrong.
That’s what makes this particular incident so disturbing.
The problem wasn’t that the AI produced obvious nonsense.
The problem was that it apparently produced an answer convincing enough to become part of a serious intelligence assessment.
CNN reported that the chatbot had combined open-source information with classified signals intelligence and ultimately reached the incorrect conclusion about the ship’s cargo.
That’s the nightmare scenario.
Not an obviously broken machine.
A machine that is wrong convincingly.
And then there’s the question I can’t stop thinking about
If this is something we know about, what don’t we know about?
That’s the part that bothers me.
We know about this incident because someone discovered the mistake and the story was reported.
But how many other AI-assisted mistakes are happening inside governments, militaries, corporations, hospitals, financial institutions, law enforcement agencies, intelligence agencies, and everywhere else that never become public?
I’m not claiming that there are thousands of secret disasters happening. We don’t know that.
And that’s precisely the point.
We don’t know.
The CNN reporting itself says the hallucination involved in this particular case was not necessarily an isolated phenomenon within the intelligence community. Sources described AI-generated errors as an ongoing concern as these systems become more integrated into government work.
That’s why transparency and oversight matter.
Because when a human makes a mistake, there is at least supposed to be a human chain of accountability.
When an AI system makes a mistake, we need to know:
Who used it?
What information did they give it?
What information did it generate?
Who reviewed the output?
Was the information independently verified?
Who ultimately made the decision?
And perhaps most importantly:
Was there actually a human making the decision, or was a human merely rubber-stamping what the machine said?
Those are not the same thing.
“Human in the loop” shouldn’t just mean having a human click the button
This is something that I think gets misunderstood.
People will say, “Don’t worry! There’s still a human involved.”
Okay.
But what does that actually mean?
If an AI system generates an intelligence assessment, a human glances at it for five seconds and approves it because the AI sounds confident, that’s technically human involvement.
But is that meaningful human judgment?
I would argue that the distinction matters enormously.
A human being involved needs to mean that the human has the authority, knowledge, time, and responsibility to question the machine.
That’s especially important when the potential consequence isn’t “this blog post has a factual error.”
We’re talking about decisions involving military operations.
Potentially civilians.
International relations.
Nuclear weapons.
Human lives.
War.
There are some decisions where “the algorithm gave us an 87 percent confidence score” simply isn’t good enough.
This isn’t an argument against AI
And I want to be very clear about that because I already know somebody will read this and assume that’s what I’m saying.
I’m not anti-AI.
I’m literally someone who uses AI in my own creative work.
I think AI can be an extraordinary tool for writers, artists, musicians, researchers, programmers, businesses, scientists, and ordinary people.
But a tool being useful doesn’t mean it should be trusted without limits.
We regulate other technologies precisely because their consequences can be enormous.
We don’t say, “Well, cars can be useful, therefore anyone should be allowed to drive one however they want.”
We don’t say, “Medical equipment sometimes makes mistakes, so doctors should just blindly trust it.”
We don’t say, “Aircraft computers are extremely sophisticated, therefore pilots aren’t necessary.”
We understand that technology works best when it operates within systems of human responsibility, safeguards, training, oversight, and accountability.
AI shouldn’t somehow get a free pass from those principles.
Especially when the technology is being used by governments
The stakes become even higher when we’re talking about government agencies.
A private individual asking an AI chatbot whether a movie came out in 1997 and getting the wrong answer is annoying.
A government agency receiving incorrect information about another country’s military or nuclear capabilities is potentially catastrophic.
And that’s why I think there needs to be serious discussion about regulation and standards for high-stakes AI use.
Not necessarily banning AI.
Not pretending AI has no legitimate uses.
Not stopping technological development.
But establishing boundaries around how much authority we are willing to give these systems.
There should be meaningful requirements for human oversight.
There should be verification procedures for high-stakes AI-generated claims.
There should be documentation of when AI materially contributed to an intelligence assessment or consequential decision.
There should be accountability when people deploy these systems irresponsibly.
And there should be clear situations where AI can assist humans but cannot independently determine the outcome.
Because there is a massive difference between:
“AI helped me analyze this information.”
and
“AI told me this was happening, so we acted.”
That difference could literally be the difference between peace and war.
The technology isn’t the only problem
Ultimately, I don’t think the lesson here should be “AI is dangerous because AI lies.”
The deeper lesson is that humans are dangerous when they forget that a tool can be wrong.
We’ve done this before with plenty of technologies.
We become enamored with efficiency.
We assume automation means accuracy.
We assume computers are objective.
We assume something generated by a machine must be more reliable than something generated by a person.
But AI doesn’t magically transform uncertain information into truth.
It can transform uncertain information into something that looks incredibly authoritative.
And that’s arguably even more dangerous.
Because confidence and correctness are not the same thing.
A machine can be spectacularly confident while being spectacularly wrong.
And if the people using it don’t understand that distinction, the machine doesn’t have to be malicious to cause enormous damage.
It just has to make one mistake.
And then have enough people trust it.
So yes, I will continue using AI
I’m not going to stop using AI because of this.
I’ll continue using it to help me write.
I’ll continue experimenting with AI-assisted music.
I’ll continue using it as a brainstorming and creative tool.
I’ll continue figuring out ways to incorporate it into my work.
But I’m also going to continue believing that I am responsible for what I do with it.
That’s the part I think we need to preserve as AI becomes increasingly embedded in society.
AI can assist.
AI can analyze.
AI can generate.
AI can suggest.
AI can help us move faster.
But there are some decisions where humans need to slow the fuck down.
Read the source material.
Question the output.
Ask what could be wrong.
Get another human involved.
Verify the claim independently.
And then, after all of that, a human being needs to make the decision.
Because if we’re going to build a world where AI is everywhere, the answer shouldn’t be giving AI unlimited authority.
The answer should be learning how to use an incredibly powerful tool without forgetting who is ultimately responsible for what happens next.
And when the alternative is potentially starting a war between nuclear-armed countries?
That’s not a philosophical question anymore.
That’s a fucking necessity.