Feature: AI Vocabulary Wordlist and Frequency Scorer

[craigtrim]

Overview

Certain words appear with statistically elevated frequency in LLM-generated text compared to human writing. These are not impossible in human prose — they're just overrepresented in AI output to a degree that creates a detectable signal.

Cassidy Johnston's observation ("the word 'quiet' in some form") is a practitioner-level identification of this phenomenon. Research and community observation have converged on a recognizable set.

Known AI-Preferred Vocabulary

Abstract/elevated register:
delve, tapestry, nuanced, robust, pivotal, transformative, foster, leverage, navigate, underscore, realm, testament

Framing/structural:
it's not X, it's Y, not only X but also Y, at its core, in essence, ultimately

Tone softeners:
quiet, quietly, gently, thoughtful, intentional, meaningful

Filler intensifiers:
crucial, vital, essential, critical, key (overused as emphasis)

Proposed Implementation

• Curated wordlist stored as a data file (JSON or plain text), versioned and extensible
• Frequency scorer: count occurrences per 1000 words, normalized against a human baseline corpus
• Z-score or percentile rank against baseline to flag anomalous elevation
• Per-word breakdown + aggregate ai_vocabulary_score

Baseline

Requires a reference corpus of human-written text (journalism, fiction, non-fiction) to establish expected frequency distributions. The signal is the delta between observed and expected, not raw counts.

Related

• Feature: AI Stylistic Tell Detection — Tricolon, Stacked Sentences, and AI Vocabulary Fingerprints #69 — AI Stylistic Tell Detection (parent feature)
• Feature: Add AI-Generated Text Detection Module #65 — AI-Generated Text Detection
• Feature: Tricolon Detector #73 — Co-occurrence scorer (downstream consumer)

[craigtrim]

Detection Approaches

1. Empirical Corpus Comparison (Most Principled)

Build a labeled dataset:

• Human corpus: pre-2021 text — CommonCrawl, Project Gutenberg, BNC, journalism archives
• AI corpus: known LLM outputs (GPT-4, Claude, Gemini) across diverse prompts

Compute keyness score per token:

keyness = (freq_in_AI / total_AI_words) / (freq_in_human / total_human_words)

Tokens with the highest ratio are statistically overrepresented in AI output. This is standard corpus linguistics methodology — the same technique used to find genre-defining vocabulary.

2. Community-Compiled Seed Lists

These exist and are consistent across independent observers — editors, writing coaches, practitioners. The words that keep appearing:

Elevated register / academic posturing:
delve, tapestry, nuanced, robust, pivotal, transformative, foster, leverage, realm, testament, underscore, navigate

Contemplative / atmospheric:
quiet, quietly, thoughtful, intentional, meaningful, gentle, delicate

Filler intensifiers:
crucial, vital, essential, critical (overused as emphasis)

Useful as a seed list but reflects what people have consciously noticed, not what statistics would surface independently.

3. Existing Detection Tool Research

GPTZero, Originality.ai, and academic papers on AI watermarking have implicitly identified vocabulary through internal models. Feature importance rankings from published stylometric AI detection papers surface specific vocabulary items. The overrepresentation of "delve" in Claude outputs specifically has been documented publicly.

4. BNC Deviation Scoring

pystylometry already has lexical/bnc_frequency.py. BNC represents pre-AI human English. A word appearing at dramatically higher frequency than its BNC baseline is a candidate AI signal — without requiring a labeled AI corpus at all.

bnc_deviation = observed_freq_per_1000 / bnc_expected_freq_per_1000

A ratio above a threshold (e.g., 5×) flags the word. This is cheap to compute and already grounded in existing infrastructure.

---

Recommended Implementation: Hybrid on Both Unigrams and Phrases

Both token granularities are needed:

Unigram scorer

• BNC deviation for any single word in the text
• Seed list lookup (immediate, no corpus required)
• Keyness ranking from labeled corpus (long-term)

Phrase scorer

AI-preferred constructions operate above the word level and would be missed by unigram analysis alone:

Framing / structural phrases:
it's not X, it's Y, not only X but also Y, at its core, in essence, it's worth noting, it's important to, when it comes to

Transition clichés:
moreover, furthermore, in conclusion, to summarize, building on this

Softening constructions:
take a moment, let's explore, let's dive into, I'd be happy to

Phrase detection requires a sliding window match against a phrase lexicon — more expensive than unigram lookup but essential because the phrases are often more reliable signals than the words individually.

---

Build Order

1. Seed list (unigrams + phrases) as a versioned data file — immediately actionable
2. BNC deviation scorer for unigrams — uses existing infrastructure, no labeled AI corpus needed
3. Phrase frequency scorer — sliding window match against phrase lexicon
4. Labeled corpus keyness — long-term, most principled, surfaces tokens not in the seed list

[craigtrim]

Temporal Versioning Requirement

All entries in the vocabulary lexicon (unigrams and phrases) must be year-labeled.

LLM-preferred vocabulary is not static — it shifts as models are updated, fine-tuned, and as providers actively work to suppress known tells. What reads as AI-generated in 2024 may not carry the same signal in 2027.

Why This Matters

If a document was written in 2025 and is being analyzed in 2030, applying a 2030 vocabulary model would produce false negatives — words that were strong AI signals in 2025 may have been deliberately deprioritized by then. The scorer needs to know when the text was written and apply the vocabulary model appropriate to that year.

Schema

Each entry should carry:

{
  "token": "delve",
  "type": "unigram",
  "introduced": 2023,
  "deprecated": null,
  "notes": "Strongly associated with Claude and GPT-4 era outputs"
}

{
  "token": "let's dive into",
  "type": "phrase",
  "introduced": 2022,
  "deprecated": 2026,
  "notes": "Common in early ChatGPT outputs; suppressed in later model versions"
}

Scoring Behavior

When scoring a document:

1. Accept an optional document_year parameter
2. Filter the lexicon to entries active during that year (introduced <= year and deprecated is None or deprecated > year)
3. Score against the temporally-appropriate subset

When document_year is unknown, default to the current year — conservative but honest.

Implication for Corpus Research

The labeled AI corpus used for keyness scoring (#empirical approach) should also be year-stratified. A 2023 GPT-4 output corpus and a 2026 GPT-5 output corpus will have meaningfully different vocabulary distributions.