Detecting LLM-Generated Web Fiction with "Classical" Machine Learning (AIGC Text Detection)

This article is currently an experimental machine translation and may contain errors. If anything is unclear, please refer to the original Chinese version. I am continuously working to improve the translation.TL;DR & DemoAs of early 2026, mainstream LLM-generated text exhibits strong statistical patterns that can be effectively distinguished from human-written content using traditional machine learning models. I suspect this is how many so-called “AI plagiarism checkers” actually work under the hood.Online Demo: https://lyc8503.github.io/AITextDetector/The model used in this demo is not trained on general-purpose data, nor has it undergone rigorous optimization or iteration. Its single-sentence detection accuracy is approximately 85% on the test set. Please read through this article before use to understand potential limitations.The core code (drafts) and trained model files are available on GitHub: lyc8503/AITextDetectorBackground (aka Useless Rambling)Back when I was still writing my thesis at school half a year ago, rumors were already spreading about checking papers for AIGC (AI-generated content). I tested several platforms—CNKI, Wanfang, and some third-party AIGC detection services—and found they could indeed distinguish between my hand-written text and LLM-generated text with decent accuracy.That sparked my curiosity about how AIGC detection actually works(and how to bypass it).But I was juggling too many things at the time—obsessed with radio, Minecraft, Touhou—and after a few failed attempts, I shelved the idea.Eventually, I faked my way through the thesis, and life moved on. But recently, while browsing Lofter, I stumbled upon entire tags flooded with low-quality, wildly out-of-character AI-generated fanfics.How can I tell they’re AI at a glance? Well, some folks(or gals) don’t even bother cleaning up Markdown formatting or AI-generated section headers before posting—and then they slap half the article behind a paywall 😓Most AI-generated texts, however, are harder to spot—they’re buried among diverse writing styles, varied prompts, and not immediately obvious. By the time you realize something feels off, it’s too late. Some texts are borderline impossible to prove as AI, leaving me paranoid. After swallowing a few too many AI-generated turds, I’d finally had enough.

Lofter browsing stops here—time to open VS Code!Yes, that’s how I ended up reviving my weekend project idea: building an AI-generated text detector…Research Attempt – No LuckThe internet is now almost entirely polluted with ads when searching for AIGC detection. Every result is just another essay-AI-rewriting service. Back then, I dug through the noise and found something called text perplexity.The idea is simple: use an existing LLM to estimate the probability of each word appearing in a given sentence. If nearly every word ranks high in the LLM’s predictions (Top-N), the sentence is likely AI-generated. Conversely, if many words are unexpected, it’s more likely human-written.Sounds promising, right? I spent some time trying this method, but results were disappointing—plenty of false positives and false negatives, and no reasonable threshold could be set. Plus, there are practical issues: high inference cost, poor cross-model generalization, difficulty deploying large models locally, and closed-weight models being hard to integrate. Overall, this approach isn’t elegant or reliable.Failed attempt—got tricked by a bunch of soft-ad "tutorials"A (Somewhat) Successful Attempt – scikit-learn SVMSince online resources were useless, back to old-school alchemy.Scikit-learn, activate! Following its Roadmap, we can directly pick Linear SVC and Naive Bayes as good starting points for our classification task.(Whisper: this also matched my gut feeling—LLMs have detectable word-choice patterns; even a Naive Bayes classifier should pick them up. I just didn’t expect the signal to be this strong.)Data GenerationOld-school alchemy traditional classifiers need labeled data—so we need human-written texts and confirmed LLM-generated texts for training.My approach: I pulled data I’d scraped in 2023 from a certain Ford-like and River-like platform, filtering for articles published between 2010–2022 (pre-ChatGPT). I only filtered out extremely low-engagement or very short pieces, then randomly sampled nearly 10,000 multi-thousand-character texts as human-written samples.Then, I used an LLM to generate chapter summaries of these texts, fed the summaries back into the LLM, and had it regenerate full articles.

This gave me a roughly equal number of LLM-generated samples, diverse in genre and closely matching the original human content.In theory, at least. But LLM APIs are expensive, and I wasn’t about to spend thousands on a weekend project. So I got creative—and skirted the rules leveraged multiple low-cost or free API channels:Gemini: Used CLIProxyAPI to convert Antigravity/Gemini CLI quota into API access—just pay ~$20 for an AI Pro accountQwen: qwen-code lets you reverse-engineer the Qwen Plus API—freeGLM-5: Lucky timing—OpenRouter was offering free GLM-5 public beta (Pony Alpha)Kimi, Deepseek, Doubao, GLM-4.7: Signed up during a promotional coding plan—$8.9 first month, API access unlockedDisclaimer: This is not a recommendation. These actions violate platform ToS and may get you banned. But the platforms are too busy with marketing hype to care, and I wasn’t about to pay full price.Many programming-focused LLM APIs strangely charge per call, but we can abuse optimize this by batching tasks into massive inputs, forcing the LLM to generate more content per call. And so…What do you mean I used over 300M Gemini tokens worth $2000 at full price?!Ultimately, I used gemini-3-flash to generate summaries, and seven different models (gemini-3-pro, qwen-coder-plus, glm-5, glm-4.7, kimi-k2.5, doubao-seed-code, deepseek-v3.2) to generate seven sets of LLM-generated samples.The generated filesTrainingWhile I was halfway through data generation, I couldn’t wait and started training.I asked Claude to write the classifier code, and it naively dumped the entire raw text into the model—achieving a suspicious 99.45% accuracy… Wait, really?Claude’s useless. I’ll do it myself. For training, I split all texts into sentences using Chinese punctuation, cleaned non-Chinese/English characters, then applied scikit-learn’s TF-IDF → LinearSVC.

After cleaning up some noise, sentence-level classification still hit ~85% accuracy!Even this buggy first version hit 88% accuracy (later optimized to 85%)Individual sentences carry limited info, but 85% accuracy per sentence means that for a longer article, we can be highly confident in judging whether it’s AI-generated. This performance far exceeded my expectations. Old-school ML still slaps—way better than those dumb online tools that just ask an LLM, “Hey, is this text AI-generated?”After finishing all data, I tried training an 8-class model (human + 7 AIs), but the LLMs seem too similar—probably distilled from each other—so classification was messy, with only ~50% accuracy.Multi-class results—apparently not separable. Maybe my model sucks, but whatever, not importantEventually, I trained seven separate binary classifiers and used majority voting: a sentence is flagged as AI if ≥2 models detect it.1234567891011121314151617181920212223242526272829303132333435loaded 8536 samplestrain chapter size: 6820, test chapter size: 1716[gemini] Train: 917,374 Test: 228,051[gemini] full TF-IDF + SVC ... 3,336,446 features -> acc=0.8809 f1=0.8082 [tn=143688 fp=10650 fn=16503

tp=57210][qwen] Train: 1,315,338 Test: 328,636[qwen] full TF-IDF + SVC ... 3,989,603 features -> acc=0.8911 f1=0.8974 [tn=136293 fp=18045 fn=17739 tp=156559][pony] Train: 1,128,044 Test: 278,663[pony] full TF-IDF + SVC ... 3,688,143 features -> acc=0.8493 f1=0.8286 [tn=135085 fp=19253 fn=22755 tp=101570] [kimi25] Train: 1,088,007 Test: 269,567[kimi25] full TF-IDF + SVC ... 3,976,027 features -> acc=0.8721 f1=0.8473 [tn=139390 fp=14948 fn=19534 tp=95695][glm47] Train: 1,124,430 Test: 279,109[glm47] full TF-IDF + SVC ... 3,980,772 features -> acc=0.8436 f1=0.8222 [tn=134461 fp=19877 fn=23786 tp=100985][doubao] Train: 1,063,395 Test: 264,121[doubao] full

TF-IDF + SVC ... 4,243,728 features -> acc=0.8940 f1=0.8700 [tn=142420 fp=11918 fn=16089 tp=93694][deepseekv32] Train: 1,176,294 Test: 289,042[deepseekv32] full TF-IDF + SVC ... 4,361,691 features -> acc=0.8529 f1=0.8403 [tn=134625 fp=19713 fn=22819 tp=111885]===================================== SUMMARY===================================== model s1 acc s1 f1 gemini 0.8809 0.8082 qwen 0.8911 0.8974 pony 0.8493 0.8286 kimi25 0.8721 0.8473 glm47 0.8436 0.8222 doubao 0.8940 0.8700 deepseekv32 0.8529 0.8403All models achieved over 85% accuracy and over 80% F1—pretty solid! I also noticed that AI-generated texts were often flagged by multiple models, so voting made perfect sense.I tried MultinomialNB and SGDClassifier, but accuracy dropped slightly. BERT gave a minor boost but required too much GPU time—discarded. Even tested AutoGluon, which somehow managed only 53% binary accuracy. Won’t dive into those.JS Implementation for Web DemoAt this point, I could’ve just published the repo and called it a day. But launching Python every time is way too inconvenient. I could’ve hosted a Python API, but that means server maintenance—violates my strict Serverless philosophy.

My original plan: export model to ONNX, run inference via ONNX Web Runtime in Wasm. But when I asked my silicon servant Claude to help, I didn’t specify clearly—and it went off-script, trimming and exporting the model as a JSON… then implemented TF-IDF + SVM entirely in JavaScript for browser inference.Hmm… actually not a bad idea. I tested it on a 1-million-character text—it took about 10 seconds on my machine, acceptable. For typical few-thousand-character inputs, it’s instant.Fine, since this is just a demo, and the JS approach is more transparent, I’ll keep this slightly silly implementation. (Blame Claude, not me.)As for accuracy: I tested different feature limits. Ultimately prioritized performance and kept 500k features. Stored as JSON, it’s a bloated 107MB (though gzipped server-side, it’s ~38MB). Smaller versions (50k–80k) only lost 3–4% accuracy, but final AI detection rates varied significantly—especially on human texts, with ±50% relative differences, leading to false positives. So I stuck with 500k.Final accuracy drop: ~1%, as shown below:1234567891011121314============================================================ SUMMARY top-500,000 C=1.0============================================================ model s1 acc s2 acc Δacc s1 f1 s2 f1 Δf1 gemini 0.8809 0.8721 -0.0088 0.8082 0.7986 -0.0096 qwen 0.8911 0.8819