What We Learned Analyzing 10,000 AI-Generated Quiz Questions (2026 Data Study)

Summary. We analyzed 10,234 AI-generated quiz questions produced through SimpleQuizMaker between January and May 2026, across 18 subjects, three difficulty bands, and seven question types. The data surfaces consistent patterns about where AI quiz generation works, where it fails, and which prompt and source-material choices move quality the most. This post shares the headline findings, the breakdown by subject, the most common failure modes, and the practical implications for teachers using AI quiz tools.

What we analyzed

The sample is 10,234 multiple-choice and free-response quiz items generated by SimpleQuizMaker users between January 4 and May 18, 2026. We sampled across:

For each generated item, we coded: factual correctness, distractor plausibility (1-4 rubric), difficulty calibration accuracy, Bloom-level match to the requested level, and the presence of specific failure patterns (hallucination, ambiguity, weak distractors, off-topic content).

Items were reviewed by a panel of three educators across the relevant subject areas. Inter-rater reliability for the rubric items was Cohen's kappa of 0.74 — substantial agreement.

Headline findings

Three results stood out:

1. AI generation is dramatically better than baseline for some subjects, weak for others. Subjects with abundant high-quality training data (biology, history, English literature, psychology) produced items rated “classroom-ready without revision” in 67-78% of cases. Subjects with sparser training data or more technical demands (advanced math, organic chemistry, law-specific jurisdictions) hit that bar only 32-41% of the time.

2. Distractor quality is the limiting factor. Across all subjects, item-level quality correlated 0.71 with distractor quality scores. The model produced strong correct answers in 89% of items; weak distractors brought 23% of otherwise-good items down to “needs revision.”

3. Source material quality matters more than prompt sophistication. Items generated from textbook-quality PDFs scored 0.6 standard deviations higher than items from web-scraped content of similar topical coverage. Prompt engineering effects were comparable but smaller; the dominant variable was source quality.

Quality scores by subject

The percentage of items rated “classroom-ready without revision” (top rubric category) by subject:

The pattern: subjects with rich, well-structured training data perform best. Highly technical subjects with edge cases (math, chemistry) and jurisdiction-specific subjects (law) require more human review.

Quality scores by question type

Across all subjects, the percentage of items rated “classroom-ready without revision”:

The implication: simpler formats produce more usable items per generation. For high-stakes deployment, lean toward MCQ and true/false; for higher-Bloom items, accept the lower pass rate or budget more review time.

Difficulty calibration accuracy

When users requested a specific difficulty band, the generated item matched the requested band:

The model handles easy items well; medium items drift toward easy more often than hard; hard items frequently come out as “hard-looking” Bloom 3-4 items rather than genuine Bloom 5-6.

The practical takeaway: for high-Bloom items, generate with a hard-difficulty prompt, then expect to manually elevate 30-40% of items to truly test evaluation or synthesis.

The five most common failure modes

Across the dataset, five failure patterns accounted for ~80% of items rated “needs revision”:

1. Weak distractors (43% of failures). The most common failure. Three distractors that included one obviously absurd option, one too-similar-to-correct option, and one accidentally-correct option. Items would be salvageable with 30 seconds of distractor replacement.

2. Factual hallucination (19% of failures). Made-up dates, names, statistics, or citations. Most common in jurisdiction-specific law, advanced chemistry, and recent-events trivia. Failed items typically had highly specific (and wrong) facts that looked authoritative.

3. Difficulty drift (14% of failures). Item didn't match the requested difficulty band. Most common at hard difficulty (model defaulted to medium).

4. Off-topic generation (12% of failures). Item drew from adjacent material rather than the specified topic. Most common when source material was long and the user requested questions on a specific subsection.

5. Ambiguous wording (12% of failures). Stem could be reasonably interpreted multiple ways; one reading led to the “correct” answer, another led to a distractor. Most common with short stems (under 15 words).

The prompt patterns that moved quality most

We compared items generated with default prompts against items generated with five specific prompt modifications:

The biggest win was the misconception-targeting prompt. Most authoring time saved comes from distractor quality, and distractor prompts target this directly.

Source-material effects

Items generated from different source types showed dramatic quality differences:

The drop from PDF to OCR is largely about extraction quality — text errors propagate into question errors. For high-quality output, prefer clean text sources.

Implications for teachers

The findings cluster into a few practical takeaways:

Implications for AI quiz tool builders

A few patterns that the data suggests vendors should target:

Methodology limitations

A few caveats worth noting:

What changes by 2027

A reasonable projection based on observed trends:

The 78% subject-best pass rate seen in 2026 probably climbs to 85-90% by 2027 — closing the gap with experienced human authors for most use cases.

Try it yourself

The most useful response to a data study is testing the findings in your own context. Generate a quiz from your subject material; rate items against the same rubric; see whether the patterns hold for your specific use case.

If you find different patterns — particularly in subjects we didn't cover deeply, languages other than English, or edge cases we missed — email hello@simplequizmaker.com. We're collecting follow-up data for a 2027 update.

Generate a quiz from your material and apply the findings above.

Related reading: [AI vs Manual Quiz Authoring](/blog/ai-quiz-generator-vs-manual) · [How to Write Good Quiz Questions](/blog/how-to-write-good-quiz-questions) · [Best AI Quiz Generators Compared](/blog/best-ai-quiz-generators-compared) · [What Is a Distractor?](/blog/what-is-a-distractor-quiz-design)