Hardly a week goes by without someone on LinkedIn or in the news presenting the conclusion of a supposedly important AI study. AI makes developers slower. AI creates more work, not less. 95 percent of AI projects fail. ChatGPT makes you dumb. Look a little closer, and you can only shake your head. The relevance of these studies usually ends the moment you check two details: when the data was collected, and which AI models - in which versions - were actually tested.
What leaves me speechless again and again is that it is not just laypeople passing these results around. Genuinely intelligent people in my industry and in the media pick them up and present them as relevant scientific findings, as if they said anything about how we work with AI today, let alone where this technology is heading.
I call this human slop in science. I don't mean sloppy work. I mean studies that may be methodologically sound and still miss reality, because they describe a state of the world that no longer exists. ChatGPT launched roughly four years ago. A study measuring the state of the art from one or two years back is looking back across a quarter to half of that entire history. For today's decisions, it is largely worthless. As a forecast, it is useless.
The Wright Flyer problem
Imagine someone in 2026 investing serious effort into finding the most fuel-efficient way to fly the Wright brothers' 1903 Flyer. You can do that. It might even be carefully measured. But nobody would draw conclusions from it for operating a modern long-haul jet. And if a newspaper ran the headline "Study shows flying is inefficient", we would laugh.
With AI, this is exactly what happens, and hardly anyone notices. To me, the leap between the models of 2024 and today's models feels similarly large. It just happened in two years instead of a hundred.
Seven studies: what they examined and what became of them
The following examples show the pattern. For each, three questions are worth asking: what was studied and how, when and with which models, and what did the media and self-proclaimed experts make of it?
1. Developer productivity (METR)
- What was studied: 16 experienced open-source developers worked on real tasks, sometimes with AI, sometimes without. With AI, they took 19 percent longer on average, even though they felt faster.
- When and with what: February to June 2025, mainly Cursor Pro with Claude 3.5 and 3.7 Sonnet; published in July 2025. The authors explicitly wrote that their results do not imply that future models will not speed up developers (arXiv).
- What it became: "Think AI coding tools are speeding up work? Think again - they're actually slowing developers down" (ITPro). Yet in February 2026, METR itself flagged the study as out of date. According to METR, a follow-up experiment no longer produced a reliable signal, partly because more and more developers did not want to work without AI; the institute also cites lower pay as a factor (METR).
2. AI intensifies work (UC Berkeley)
- What was studied: Two researchers observed a U.S. tech company with around 200 employees that offered commercial AI subscriptions without mandating their use. Employees worked faster, took on more tasks and extended their work into more hours of the day.
- When and with what: April to December 2025, no models named; interim findings published in Harvard Business Review in February 2026 (HBR).
- What it became: A German business magazine ran a headline that translates as "Harvard study: AI doesn't make us more productive - it drives us into burnout" (Business Punk). The study is not from Harvard, and according to heise, productivity actually rose significantly (heise). What was observed is an organization in its first year of voluntary AI adoption. That people experiment a lot in such a phase and sometimes would have been faster by hand looks to me like a learning curve, not a permanent state.
3. 95 percent without returns (MIT NANDA)
- What was studied: more than 300 publicly disclosed AI initiatives, interviews with 52 organizations and a survey of 153 senior leaders. No model was measured; what was measured were the returns of early adoption projects.
- When and with what: January to June 2025, published in July 2025. According to Fortune, the report did not blame model quality, but a learning gap in tools and organizations (Fortune).
- What it became: "95 percent of AI projects fail", a figure that, as Fortune reported, even spooked investors. A snapshot of early pilot projects turned into a verdict on the technology.
4. The Danish labor market (Humlum, Vestergaard)
- What was studied: surveys of 25,000 workers in eleven occupations, linked to labor market data. Result: no measurable effect of AI chatbots on earnings or hours, with average time savings of 3 percent (BFI Working Paper).
- When and with what: surveys from late 2023 and 2024, AI chatbots in general; published in April and May 2025.
- What it became: "Generative AI is not replacing jobs or hurting wages at all, economists claim" (The Register). What was measured was the chatbot era before agents. Concluding that AI will barely touch the labor market in the future means drawing a line through a single point.
5. Your Brain on ChatGPT (MIT Media Lab)
- What was studied: the brain activity of 54 people writing essays, measured by EEG, one group of them with AI support.
- When and with what: over four months, no dates given, using GPT-4o; published in June 2025 as a preprint without peer review (arXiv).
- What it became: headlines such as "ChatGPT users show alarming drop in brain activity: MIT study" (YourStory). The research team went on to publish a list of terms journalists should avoid and stressed that its conclusions were preliminary (MIT Media Lab).
6. Chatbots as health advisors (Oxford)
- What was studied: nearly 1,300 people in the UK assessed medical scenarios with AI assistance and did no better than a control group.
- When and with what: late 2024 according to Science News, with GPT-4o, Llama 3 and Command R+ (Science News); preprint in April 2025, published in Nature Medicine in February 2026.
- What it became: "Chatbots Make Terrible Doctors, New Study Finds" (404 Media). The researchers' core insight - that the problem lay mainly in the interaction between people and models - is a smart one. But a 2026 headline about chatbots that rests on 2024 models does not describe the chatbot anyone has on their phone today.
7. Do AGENTS.md files help?
- What was studied: whether instruction files given to AI coding agents for a software project improve their success rate. Result: no meaningful improvement, but costs rose by roughly a fifth (arXiv).
- When and with what: data collection period not stated; tested with Claude Sonnet 4.5 (released September 2025), GPT-5.2 (December 2025), GPT-5.1 mini and a small Qwen model. First published in February 2026, revised at the end of September 2026 with the same models.
- What it became: headlines such as "Paper finds AGENTS.md files hurt coding agent task success" (AI Weekly). For the question of how a company should instruct agents built on today's models, this study, in my view, tells us practically nothing.
Not an anecdote - it can be measured
By now, the phenomenon itself has become a research subject. A bibliometric audit from May 2026 examined more than 18,000 papers on language models published between 2022 and 2026. The models tested trailed the contemporaneous frontier by a clear margin at the median, and that gap is widening every year. About half of the audited abstracts state their conclusions about "AI" as a category rather than about the specific model tested (arXiv). A second audit from July 2026 found that across 40 studies, the newest model named was a median of 281 days old at the time of publication (arXiv).
Why we keep looking in the rearview mirror
The first reason is the pace of academia. The MIT team behind the ChatGPT study notes that peer review typically takes anywhere from four months to two years. This craft served science well for a long time, because the object of study changed more slowly than the research process. A language model is history in two years.
The second reason is self-inflicted. A fixed, older model is easier to control and reproduce. And "AI can't do X" gets more attention than "GPT-4o couldn't do X in late 2024 under these conditions." The limitations many authors carefully spell out rarely make it into the headline.
And there is an irony. Some of these studies are probably a by-product of the AI hype itself. Because everyone is talking about AI, there is funding, attention and publication pressure even for work that is foreseeably irrelevant to practice. This is not an accusation against individual researchers, who often act in the best of faith. It is an incentive system that rewards volume, not relevance.
The most uncomfortable reason concerns all of us. People are often said to struggle with imagining exponential change. What I observe in the AI debate goes further: many people seem unable to imagine any change at all. They look at the status quo, note that AI has not yet made something more productive, and conclude that it never will. It is like watching a three-year-old play chess and concluding they will never learn the game.
Meanwhile, the bastions are falling
Anyone who looks around with open eyes can see one bastion after another falling. On the Remote Labor Index, which tests whether AI can complete real, paid freelance projects to client standard, the best model a year ago managed only a few percent; today, Claude Fable 5.2 completes roughly four in ten (KI Woche). OpenAI released more than 700 mathematical manuscripts, 162 of them with machine-checked proofs (Takes on AI). Together with universities, Google had teams of AI agents prove a conjecture in information theory that had been open since 2013 (KI Woche). And AI agents uncovered hundreds of vulnerabilities in the Linux kernel in a single run (KI Woche).
Against this backdrop, some studies read as if they came from a parallel universe. A world in which Sonnet 4.5 is the measure of all things and developers happily work without AI. That world no longer exists.
The rift in science
Let me be clear: this is not about science as a whole. There is outstanding AI research - inside the AI labs, at universities and at independent institutes. The Remote Labor Index from the Center for AI Safety and Scale AI is continuously updated. METR publicly flagged its own, widely cited study as outdated and redesigned its experiment. And researchers recently used a current model to check hundreds of mathematical papers for errors (KI Woche).
That is exactly why a rift is opening. On one side are researchers who work with current systems, date their findings and keep measuring. On the other side are many who, probably with the best intentions, use yesterday's methods to produce studies about the technology of the day before yesterday. Some findings do hold across model generations, for example that people misjudge their own productivity with AI. The problem starts when a dated observation about a specific model is read as a statement about "AI". A study on GPT-4o is a historical source, not a look ahead.
Large parts of science will have to fundamentally change the way they look at AI if they want to stay relevant outside their own bubble. They need to move from snapshots to instruments. And anyone building their AI strategy on measurements of last year's models is driving with their eyes on the rearview mirror. On a straight road, that works for a while. The road we are on right now is anything but straight.
That's my take.