How to Read AI News Without Getting Played: 7 Questions That Cut Through the Hype

AI news has a signal problem. Every week brings a “breakthrough,” a “first,” a model that “beats humans,” and a fresh prediction that everything changes in eighteen months. Some of it is real. Much of it is a press release wearing a lab coat.

You don’t need a PhD to tell the difference. You need seven questions.

1. Is this a product or a demo?

The single most useful filter. A demo is a controlled recording of a system doing something once, under conditions chosen by the people filming it. A product is something you can use today, that fails in public, at a price someone will state out loud.

The gap between the two is routinely one to three years — and some demos never become products at all. When a story is built on a demo video, mentally file it under “possible future,” not “news about the present.”

2. Who measured it, and against what?

“Beats doctors at diagnosis.” “Outperforms lawyers.” “PhD-level reasoning.” Every one of these claims hides a benchmark, and benchmarks have two chronic problems:

Contamination. Models train on internet-scale data. If the test questions (or things like them) were in the training data, a high score measures memory, not ability.
Narrowness. Beating humans at a multiple-choice medical exam is not the same as practicing medicine, which involves ambiguity, patients who omit things, and consequences.

A claim measured by the company’s own benchmark, announced in the company’s own blog post, is marketing until independently reproduced.

3. What does it cost to run?

Capability headlines almost never mention economics. A system that solves a hard problem using thousands of dollars of compute per attempt is a research result, not a coming wave of automation. When cost is missing from a story about AI replacing some job, the story is incomplete — the entire question of automation is capability divided by cost.

4. Who benefits from me believing this?

Not cynicism — just bookkeeping. AI labs raising money benefit from breakthrough narratives. Incumbents benefit from “it’s all hype” narratives. Doomers and accelerationists both benefit from “everything changes imminently.” Journalists benefit from drama in either direction.

None of this makes a claim false. It tells you how much independent verification to demand before updating your beliefs.

5. Is the impressive part the AI, or the humans behind the curtain?

Many “autonomous” systems ship with quiet human scaffolding: content reviewers, operators handling edge cases, engineers who curated the demo tasks. The tell is in qualifiers — “with human oversight,” “in supervised deployments,” “assisted.” Read for those words. They’re doing heavy lifting.

6. What happened to the last version of this story?

AI news has short-term amnesia. Before absorbing a claim, ask what happened to its predecessor. The pattern that repeats: the technology does arrive, but slower, narrower, and weirder than the headline version promised. Radiologists weren’t replaced — they got AI tools. That’s the shape most AI stories eventually take: not replacement, but reshaping.

7. Would this survive a boring restatement?

Strip the adjectives and restate the story in flat language. “Company releases model that scores 4% higher than its previous model on coding benchmarks” is real but modest. “AI achieves human-level coding” is the same fact, inflated. If a story evaporates when you remove the excitement, the excitement was the story.

The honest middle

Here’s the tension worth holding: the hype is real and so is the technology. The same ecosystem that produces inflated headlines is also producing genuinely startling systems — and dismissing everything as hype has aged as badly as believing everything.

The goal isn’t to become a cynic. It’s to become someone whose beliefs update on evidence instead of press cycles. Seven questions is all that takes.

New here? Start with our plain-English glossary of the 15 AI terms in every headline.