LLM Text Detection with Classical ML — TF-IDF + SVM That Still Works (2026)
lyc8503's AITextDetector uses scikit-learn TF-IDF + LinearSVC — ~85% sentence accuracy, 7-model voting, under 0.01% human false positives. Demo, limits, HN debate, and why not to punish students without review.
AI DetectionMachine LearningNLPAI Ethicsscikit-learnContent Integrity
July 2026 — A Hacker News thread (~162 points) resurfaced a counter-narrative to billion-parameter AI detectors: classical machine learning still catches a lot of LLM prose. Researcher lyc8503 trains TF-IDF + LinearSVC (scikit-learn) with seven binary classifiers and majority voting, reports ~85% sentence-level accuracy, and ships a browser demo at lyc8503.github.io/AITextDetector/. The stack is fully inspectable — unlike black-box SaaS — which makes it worth understanding even if you disagree with automated academic punishment.
TL;DR — classical ML detector at a glance
Question
Answer
Algorithm?
TF-IDF features + LinearSVC (scikit-learn)
Ensemble?
7 binary classifiers — one per training LLM; ≥2 votes = AI
Most 2024–2026 detectors market neural embeddings or perplexity from a reference LM. lyc8503's bet: bag-of-words statistics still separate LLM-regenerated text from human web fiction because:
Training objective artifacts — function-word ratios, sentence length variance, dialogue tag patterns
Twin regeneration — same plot, different function words → classifier learns distribution shift, not plagiarism
Interpretability — linear weights inspectable; no GPU farm to retrain a demo
snippet
Human fiction (pre-2022)
│
├─► TF-IDF vector ──► LinearSVC #1 (vs Gemini regen)
├─► TF-IDF vector ──► LinearSVC #2 (vs Qwen regen)
├─► ... (7 models)
│
New text ─► same TF-IDF ──► 7 scores ──► majority vote (≥2) ──► AI / human
This is not magic — it is supervised text classification on a narrow genre (web fiction). Performance will drop on legal briefs, STEM homework, or Claude Opus 4.8 fiction (author admits this arms-race gap).
Training setup — twins, not crawlers alone
Component
Detail
Human corpus
~10k web fiction chapters pre-2022 (before ChatGPT-era contamination)
Positive class
Same stories regenerated by 7 LLMs (Gemini, Qwen, GLM, Kimi K2.5, Doubao, DeepSeek, etc.)
Features
TF-IDF — high-dimensional sparse vectors
Classifier
LinearSVC per source model
Decision rule
Majority voting — ≥2 classifiers flag AI → label AI
Why pre-2022 human text? Reduces label noise from humans pasting ChatGPT into fanfic platforms — a common failure mode for detectors trained on "everything after 2023."
Kimi K2.5 in the training set connects to explainx.ai's Chinese open-model coverage — detectors must track non-English-West generators, not just GPT clones.
Accuracy claims — in-distribution vs unseen models
Test slice
Approx. sentence accuracy
Held-out same distribution
~85%
Unseen Claude Sonnet 4.6
~70%
Unseen GPT 5.2
~70%
Human fanfic @ 70% threshold
FP < 0.01% (10k samples)
70% on unseen frontier models is remarkable for linear bag-of-words — and simultaneously too low for automatic expulsion from school. A 30% miss rate on adversarial or novel-model prose is a feature, not a bug, if your policy requires human review.
Author honesty: Claude Opus 4.8 fiction is harder now — expect retrains or score drift monthly.
Paste a paragraph → get per-sentence scores and ensemble vote. Good for journalists, moderators, and researchers auditing slop — bad as a sole disciplinary record.
Lofter field test — 32% trending tags above 50% AI
The author scanned Lofter (Chinese creative blogging) trending tags — 32% of sampled posts scored above 50% AI by the ensemble.
Reading
Caveat
Platform slop influx is real
Tag trending ≠ random sample
Genre match helps detector
Fiction-like posts align with training
Cultural context
Western educators should not import Lofter stats blindly
Ties to AI copying creativity — when copying is free, detection becomes an arms race, not a morality substitute.
Bypass attempts — weaker than HN feared
Author-tested evasions:
Bypass
Result
Translate roundtrip (e.g. EN→ZH→EN)
Still often flagged
"Reduce AI flavor" prompt
Still often flagged
HN skeptics note stronger paraphrase models and humanization services will adapt. Classical detectors age faster when attackers optimize against public weights — the GitHub repo is a double-edged sword (auditability vs evasion).
HN debate — skeptics, Pangram, and harm
Thread themes:
Arms race — every public method becomes training noise for the next generator
Pangram cited — commercial SOTA neural detector with enterprise API (name only — no competitor link per explainx.ai policy)
Author humility — Opus 4.8 fiction evades; retrains required
explainx.ai position: Use scores as triage, like SEO/GEO audits — not verdicts. For images, prefer C2PA content credentials where signing exists; prose lacks universal credentials today.
Extend with your domain corpus (internal docs, support tickets) — do not assume fanfic weights transfer to engineering RFCs without retraining.
Classical vs neural detectors — when to pick which
Criterion
TF-IDF + SVM (lyc8503)
Neural / commercial SOTA
Auditability
High — sparse weights
Low — black box
Offline demo
107MB JSON in browser
Usually API-only
Genre sensitivity
High — fiction-tuned
Varies by vendor
Unseen model robustness
~70% (author)
Often higher — unverified here
Maintenance
You retrain
Vendor chases models
Cost
Open source
Subscription
For multimodal provenance, pair prose scores with C2PA on LinkedIn/X — different problem, complementary signal.
Summary
lyc8503's AITextDetector proves TF-IDF + LinearSVC with 7-model majority voting still hits ~85% sentence accuracy on fiction-like text and ~70% on unseen Claude Sonnet 4.6 / GPT 5.2 — with under 0.01% human false positives on pre-2022 fanfic at a 70% threshold. The 107MB browser demo is transparent and privacy-friendly; Lofter trending data shows real-world slop pressure. Do not auto-punish students or employees from scores alone — use human review, disclose limits, and expect Opus 4.8-class fiction to evade. See AI copying debate for why detection ≠ creativity policy.
Accuracy figures and Lofter statistics accurate as of July 17, 2026 per author blog and HN thread. Retrain on your genre before high-stakes use; model vendors ship weekly.
