EA Forum Podcast (All audio)
“Our Community Often Underestimates the Weaknesses of Pangram” by SoniaAlbrecht
The EA forum measures the level of AI usage in forum posts using the product Pangram. But how much does Pangram's "AI-Generated" label really indicate the degree to which an author has outsourced their thinking? When they tested their 4.0 product, Pangram found that, by their definition, the proportion of AI-Assisted documents it classified as AI-Generated was 0.01%, 4%, or 7%, depending on the experiment. Then they omitted the experiments that found 4% and 7% false positive rates (FPRs) on their website outside their technical report, while advertising that the product detects AI-Assisted writing. Before I contacted them about this issue on September 17th, their claim on their website was more misleading—"99.9%+ Accuracy" was displayed directly adjacent to the phrase "Detects AI Assistance" on the text detection input box that many people don't read past. Similar claims remain repeated elsewhere on the main page instead of by the text box itself. I do not know if my message was the cause of the change. Their experiment that found the 0.01% FPR might be more appropriate for identifying human-written text rather than AI-Assisted writing. For example, the prompt they gave to Claude for this experiment was "Fix spelling, punctuation, and [...] The original text contained 3 footnotes which were omitted from this narration. --- First published: September 27th, 2026 Source: https://forum.effectivealtruism.org/posts/cF4zZ8EdQhfGZxed5/our-community-often-underestimates-the-weaknesses-of-pangram --- Narrated by TYPE III AUDIO .
