Substack recently integrated Pangram’s AI detector into its platform, allowing readers to scan posts, Notes, and comments for an estimated human-to-AI percentage breakdown. The feature is free, on-demand, and opt-in. Pangram markets a claimed false-positive rate of roughly 1 in 10,000 (and, for its latest model, as low as one in 24,000), backed by evaluations from the University of Chicago’s Becker Friedman Institute. On paper, the metrics look pristine.
The question is not whether Pangram can tell obvious AI text from obvious human text. It can. The real question is whether it can survive the actual workflow of human writing: drafting, typo fixes, line edits, and light AI assistance.
When actual writers tested the mechanism on real-world text, it failed that exact test.
One author fixed a single typo in a chapter that had previously scored 100% human. The new score instantly flipped to 100% AI. The author changed one word and deliberately broke a sentence. The score returned to 100% human. They restored the original word. The tool flagged it as 100% AI again. A single keystroke determined whether the tool classified human writing as synthetic.
Writer Katharine English ran an original short story she had been crafting for weeks. Pangram returned a 31% AI rating. The only concrete phrase the software highlighted as evidence was “moral clarity”. English performed standard human editing—tightening sentences, swapping the clunky phrase for something better, and cutting redundancies. Following these human improvements, the AI score nearly doubled.
Days later, this broken pattern played out in public. UC Berkeley mathematics professor Zvezdelina Stankova published an op-ed in the San Francisco Standard—the product of several hundred hours of intensive deliberation and 80 hours of her own direct work. She deployed an LLM for light copy-editing only: minor grammar fixes and sentence tightening. Pangram scored the piece as 33% AI-generated. A post highlighting this metric by Berkeley Law professor Chris Hoofnagle drew nearly three million views on X, immediately diverting the public conversation from her mathematical arguments to a bogus algorithmic verdict.
This is the product.
These detectors are trained and validated primarily on clean binaries: pre-LLM human text versus pure chatbot output. Real human writing is never that sterile. Humans edit iteratively. We fix typos. We tighten rhythms. We run basic grammar tools. We revise over days or weeks. The moment text leaves a controlled laboratory distribution, the statistical signals the model relies on become noisy or inverted.
A 2025 ACL paper by Shoumik Saha and Soheil Feizi, “Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing,” documented precisely this architectural flaw. State-of-the-art detectors frequently misclassify lightly AI-polished or hybrid human text as machine-generated, with misclassification rates ranging from 10% to 75% depending on the depth of polish applied. Minor human edits routinely flip scores. The tool ends up measuring deviation from a narrow training set, not actual authorship.
The Corporate Contrast
In mid-August 2026, New York Times consumer tech columnist Brian X. Chen tested the standalone Pangram product. Feeding it personal essays alongside chatbot text instructed to imitate his writing, he reported that the detector successfully separated raw extremes. He cited the 1-in-10,000 claim, described the experience as empowering, and compared it to putting on the reality-revealing sunglasses from They Live.
Notice the setup: polar extremes. Pristine personal essays versus raw chatbot output. A basic statistical classifier excels at separating the two. It is a low-bar, closed-world benchmark—entirely irrelevant to how the tool performs when real writers do what real writers actually do: edit, polish, fix typos, and iterate.
Chen paid $20 a month out of pocket for this illusion. Substack baked the exact same technology into its publishing platform for free.
The corporate anxiety driving this move is transparent. Substack’s business model depends entirely on a parasocial trust contract: readers pay subscriptions because they believe they are buying direct access to a human mind. If the platform becomes choked by generic AI ‘thought leadership,’ the trust evaporates. Substack needed to appear as the guardian of authentic human expression.
So they partnered with Pangram. But the tool they chose to protect human writing is fundamentally incapable of distinguishing human writing from human writing that has simply been edited.
Pangram’s lab numbers and Menlo Ventures’ $9 million funding round in July 2026 are real. But so is the chasm between controlled corporate benchmarks and messy, iterative human writing.
My own posts follow the same hybrid process the detector cannot parse—directed by a human, crafted across multiple models, then edited by hand.
Version history, process disclosure, and human editorial judgment remain far stronger evidence of authorship than any fluctuating percentage score. When a verification system misclassifies human refinement as a machine signature, it stops measuring authorship and starts measuring statistical deviation from a rigid training set.
The slop detector is the slop.
This post was directed and edited by a human, with research, drafting, and iterative refinement crafted by the multi-model AI council: GPT, Claude, Grok, and Gemini. Run it through Pangram and watch the score shift.
References
Katharine English, “Pangram Flagged My Own Writing as AI,” Nobody Famous (Substack), 23 July 2026.
Charlie Finch, “Fooling Pangram: How One Word Made Me Human,” (Substack), 27 July 2026.
Brian X. Chen, “I Tested a Popular A.I. Slop Detector. It Felt Empowering,” The New York Times, 13 August 2026. https://www.nytimes.com/2026/08/13/technology/personaltech/pangram-ai-detector-test.html
Coverage of Zvezdelina Stankova op-ed and Pangram score, The Guardian, 19 August 2026. https://www.theguardian.com/us-news/2026/aug/19/uc-berkeley-professor-ai
Shoumik Saha & Soheil Feizi, “Almost AI, Almost Human: The Challenge of Detecting AI-Polished Writing,” Findings of the Association for Computational Linguistics: ACL 2025. https://aclanthology.org/2025.findings-acl.1303/
Substack Help Center, “How can I detect AI on Substack?” (updated August 2026).
Pangram claims on false-positive rates and third-party evaluations. https://www.pangram.com/blog/all-about-false-positives-in-ai-detectors
Menlo Ventures investment note, “Investing in Pangram to stop AI slop on the internet,” 29 July 2026.