---
title: Annex A — AI-specific assessment items
methodology_version: v1.0.0
parent: core.md
status: stable
license: CC BY 4.0
created: 2026-05-26
---

# Annex A — AI-specific Assessment Items

> Mandatory when the recommender being evaluated is an AI (LLM response). Eleven items, each Pass/Fail with evidence. **Failure of any single item triggers automatic Axis ⑤ (Hallucination) Fail.** When the recommender is a human, this annex does not apply, and Axis ⑤ runs on its five base items only (see `core.md` §3).

> **Why this annex exists separately.** v1.0.0 folded six AI-specific items into Axis ⑤. The author's adversarial pre-review surfaced that this both (a) under-detected the breadth of AI-specific failure modes, and (b) made Axis ⑤ unevenly heavy when the recommender happened to be an AI. v2 splits the AI-specific items into this annex, expands them to eleven, and makes any single Annex A failure equivalent to Axis ⑤ Fail without contaminating the base axis.

---

## A.1 Model identification
Pass criteria:
- Model name, version, call date, `temperature`, `top_p`, `max_tokens` — all disclosed
- System prompt preserved (when present)
- Call path disclosed (API / web / mobile app)
- **Evidence**: full call metadata block, copyable

Failure mode this catches: a recommendation produced under GPT-3.5 in 2023 with no system prompt cannot be reproduced or judged the same way as one produced under Claude Opus 4 in 2026 with a careful system prompt.

## A.2 Prompt reproducibility
Pass criteria:
- Original prompt text preserved (Markdown / HTML encoding stripped)
- **Three reproductions** performed under the same parameters; all three responses preserved
- For each of the three responses, the recommendation target (ticker / asset) is marked as: match / mismatch / partial match
- **Evidence**: the three full responses, side by side

Failure mode this catches: prompt-engineered cherry-picking.

## A.3 Answer-distribution stability (Stability Score)
Derived from A.2. Pass criteria:
| Distribution | Meaning | Pass criterion |
|---|---|---|
| 3/3 match | Model produces this recommendation stably | ⭕ Pass |
| 2/3 match | Model agrees by majority | △ Partial — answer distribution must be disclosed in body |
| 1/3 or 0/3 | Model produces unstable recommendations | ❌ Fail — the recommendation itself is an artifact of LLM non-determinism |

**The Stability Score must be public-facing** in the published evaluation.

## A.4 Web search / RAG usage
Pass criteria:
- Web search tool usage disclosed (RAG vs pure training-data inference)
- If RAG was used, **all retrieved result URLs preserved**
- Tier distribution table for the RAG results (Tier 1 / 2 / 3 per `tier-rulebook.md`)
- **Evidence**: RAG retrieval log

Failure mode this catches: RAG sleight-of-hand (retrieve a Tier 1 source and a Tier 3 blog, then cite the blog).

## A.5 Training-data cutoff
Pass criteria:
- Model's training data cutoff estimated, with explicit handling of post-cutoff events
- Official model-card cutoff cited (URL)
- If the body references post-cutoff events **without** RAG having been used, this is automatic Fail (hallucination signal)
- **Evidence**: model-card URL, post-cutoff event handling note

## A.6 Non-existent-citation verification
Pass criteria:
- The AI has not recommended a non-existent **fund, ticker, report, paper, or news article**
- The evaluator personally verifies each cited source in the body
  - For ≤ 10 citations: **exhaustive verification** (no sampling)
  - For > 10 citations: **stratified random sample**, with sampling method documented (patch-pending)
- If even one citation refers to non-existent material → Verdict = Hallucinated, immediately at D+0
- **Evidence**: per-citation verification log, marked verified / unverifiable

Failure mode this catches: the canonical AI failure mode — confident citation of a paper/report/ticker that does not exist.

## A.7 Tense and year accuracy
Pass criteria:
- Explicit year/quarter labels ("Q3 2024" ✓, "the most recent quarter" ✗)
- Every cited event has its actual occurrence date cross-checked against a primary source
- **Evidence**: timestamps cross-referenced

## A.8 Correlation-vs-causation confabulation
Pass criteria:
- When the body asserts "X causes Y" / "X drives Y", it must offer a **causal mechanism**, not only a historical correlation
- A claim of causation purely on historical correlation triggers Axis ⑥ (Causal Chain) Fail
- **Evidence**: extracted causal mechanism (one paragraph) or noted absence

## A.9 Self-contradiction check
Pass criteria:
- Cross-check the three A.2 reproductions for **mutual contradiction** (e.g., "buy SOXX" vs "sell SOXX")
- Any contradiction discovered must be disclosed in the body
- **Evidence**: contradiction matrix across the three responses

## A.10 Recursive-AI evaluation ban
Pass criteria:
- If a **second AI** is used to evaluate the recommendation, the second AI must also satisfy Annex A in full
- Use of a second AI in scoring must be disclosed in frontmatter (`evaluator.ai_assist_in_scoring`)
- **Evidence**: second AI's metadata block, prompts, full responses

Failure mode this catches: the recursive hallucination trap.

## A.11 Inter-rater trigger above public-influence threshold
Pass criteria:
- Above a public-influence threshold (follower count ≥ 10,000 / citation count ≥ 100 / media mentions ≥ 5), an **external evaluator** must participate
- Both verdicts reported, with **agreement / disagreement noted**; disagreement explained in one paragraph
- **Evidence**: external evaluator's signed verdict block

> **Open issue (core.md §12):** the external evaluator pool is not yet constituted; until it is, A.11 cannot be applied to above-threshold recommenders.

---

## Annex A composite block (for the evaluation page)
```
[Annex A — AI-specific] (recommender is an AI)
 A.1 Model identification:           ✅/❌
 A.2 Prompt reproducibility (3x):     ✅/❌
 A.3 Stability Score:                 3/3 | 2/3 | 1/3 | 0/3
 A.4 RAG / web search:                used / not used   (Tier distribution: T1=N / T2=N / T3=N)
 A.5 Training cutoff:                 YYYY-MM-DD; post-cutoff events handled: ✅/❌
 A.6 Non-existent citations:          0 / N  (verification method: exhaustive / sampled)
 A.7 Tense and year accuracy:         ✅/❌
 A.8 Correlation-vs-causation:        ✅/❌  (causal mechanism present: ✅/❌)
 A.9 Self-contradiction:              ✅/❌  (contradictions disclosed: ✅/❌)
 A.10 Second AI in scoring:           none / used (Annex A applied to it: ✅/❌)
 A.11 Inter-rater (above threshold):  N/A / N evaluators, agreement X%
```
Any single Fail in A.1–A.11 → automatic Axis ⑤ Fail in the main 6-axis sheet.

## Known limitations of Annex A
- **Cost**: three reproductions × one large model × per evaluation (track in `evaluator.llm_cost_incurred`, patch-pending).
- **Reproducibility**: API model versions are silently revised; preserve the provider snapshot ID (`gpt-4o-2024-08-06`, not `gpt-4o`).
- **Recommender prompt secrecy**: many AI recommenders do not publish prompts; A.2 then cannot be performed — flag as a structural limitation or wait for disclosure.

**End of Annex A.**