Original test
Are AI detectors biased against non-native English writers? We tested ours
In 2023, Stanford researchers showed that popular AI detectors labelled most essays by non-native English speakers as machine-written. We ran GPTTrace on the same public essays. Our deep scan wrongly flagged 4 of 91, and our standard check none.
The study that started the debate
In April 2023, Weixin Liang, Mert Yuksekgonul, Yining Mao, Eric Wu and James Zou at Stanford published “GPT detectors are biased against non-native English writers”, later printed in the journal Patterns. They gave seven widely used AI detectors 91 TOEFL essays written by non-native English speakers. In the authors’ words, the detectors “misclassified over half of the TOEFL essays as ‘AI-generated’ (average false positive rate: 61.22%).” Essays by US eighth-graders, by contrast, were classified almost perfectly.
The explanation is uncomfortable. Many detectors lean heavily on how predictable a text is: language models choose likely words, so predictable writing looks machine-made. Writers working in a second language often use a smaller, safer vocabulary and simpler sentence patterns, exactly what such a detector treats as suspicious. The study became the most cited evidence that AI detection can punish people for writing in a language they learned, and it is a large part of why several universities have since stepped back from automated detection.
What happened when we ran GPTTrace on the same essays
The authors published their data, so on 13 October 2026 we ran GPTTrace’s two text checks on it in our evaluation harness, using the same WebAssembly code and model files the site serves. We used the same 91 TOEFL essays, two sets of essays by native English-speaking students, and the same TOEFL essays after the study’s authors had GPT-4 improve their word choices to sound more like a native speaker. Nothing was tuned on these essays, and neither our neural model’s training data (the RAID and MAGE research corpora) nor our calibration set contains them. “Flagged” means the result reached our “Likely AI” line, a probability of 70% or more.
| Essays | Written by | n | Standard check flagged | Deep scan flagged |
|---|---|---|---|---|
| TOEFL exam essays | Non-native English speakers | 91 | 0 (0%) | 4 (4.4%) |
| Hewlett Foundation essays | US 8th-grade students | 88 | 2 (2.3%) | 0 (0%) |
| College essays | Native English-speaking students | 70 | 0 (0%) | 0 (0%) |
| TOEFL essays, polished by GPT-4 | Human draft, AI rewrite | 91 | 12 (13.2%) | 46 (50.5%) |
For comparison, the seven detectors in the original study averaged 61.22% on the first row. GPTTrace’s deep scan, the stricter of our two checks, flagged about one essay in twenty-three.
How much to read into it
These are small samples, and honesty about that matters more than the headline. With 91 essays, 4 flagged is consistent with a true rate anywhere from roughly 2% to 11% (a 95% Wilson interval). Zero of 91 for the standard check means the true rate is probably below about 4%, not that it is zero. The essays are all one genre, short timed exam answers on everyday topics, so they say nothing about how GPTTrace handles a non-native writer’s dissertation chapter or business email.
The comparison with 2023 also has limits. The detectors in the study have been updated since, and we did not re-test them, so we are not claiming any of them still scores 61%. What the comparison does show is that the bias the study found is not inevitable: a detector can be built and calibrated to keep it small. We keep academic writing by non-native authors in our regular test set for that reason, and our threshold is set from the human side first. Our page on false positives explains the method.
The hard case: human writing polished by AI
The last row is the most interesting. Each of those essays was written by a person and then rewritten by GPT-4 to use more sophisticated vocabulary, which is what many students do with grammar tools and chatbots today. Our deep scan flagged half of them, our standard check about one in eight. Whether those flags are right depends on your rules. If AI rewriting counts as AI use, the deep scan missed half; if only AI-written ideas count, it flagged half of the writers for editing help. No detector can settle that question for you, and the rules of a class or a publication should say which side of the line polishing falls on.
What this means if you teach or assess writing
Four of these writers would still have received a “Likely AI” label from our deep scan, and any detector you use will have some rate like that, published or not. Treat a flag on a non-native speaker’s essay with extra care. Compare it with work they wrote in class, ask about their drafting process, and look at the specific evidence the result lists rather than the percentage. The essay checker page sets out a fair process, and everything runs in your browser, so student work never leaves your computer.
Data: the authors’ public repository, github.com/Weixin-Liang/ChatGPT-Detector-Bias (folders TOEFL_real_91, HewlettStudentEssay_real_88, CollegeEssay_real_70 and TOEFL_gpt4polished_91). We report only aggregate results. Full method: methodology.