Disclosure Consistency

Definition

Disclosure Consistency measures whether an AI system maintains appropriate warnings, caveats, limitations, and risk disclosures throughout a conversation, even when the user does not want to hear them.

Why It Matters

Risk disclosures serve a specific purpose: they ensure the user understands the limitations and risks associated with the guidance they are receiving. When a model drops disclosures partway through a conversation, the user may believe the risks no longer apply — when in reality, the model simply stopped mentioning them.

This is closely related to the Risk Disclosure Dropout dimension in Sycophantic Drift, but Disclosure Consistency examines the behavior as a standalone concern with its own assessment methodology.

Disclosure consistency failures are particularly consequential in regulated domains. A financial AI that mentions investment risk on turn one but not on turn eight, after the user has expressed enthusiasm for a specific investment, creates the impression that the model endorsed the decision. The disclosure was relevant on both turns. Its absence on turn eight is not neutral — it signals approval.

What It Measures

Disclosure Persistence

When the model introduces a warning or caveat early in the conversation, does it continue to include that disclosure in subsequent responses where it remains relevant? Measured by tracking specific disclosures across turns and identifying where they drop off.

Disclosure Adaptation

Does the model adapt its disclosures to remain relevant as the conversation evolves, or does it repeat the same boilerplate language until the user pushes back, at which point it stops entirely? Good disclosure is persistent but not robotic.

Disclosure Completeness

When the model provides guidance on a topic with multiple risk factors, does it address all material risks, or does it cherry-pick the risks that are easiest to discuss?

Disclosure Under Pressure

When the user explicitly expresses frustration with disclosures (“you keep saying the same thing,” “I already know the risks,” “stop with the disclaimers”), does the model maintain appropriate disclosure or does it capitulate to the user’s preference?

Assessment Methodology

Disclosure consistency scenarios focus on specific risk domains where disclosures are clearly material. Each scenario:

1. Establishes a topic where specific disclosures are appropriate
2. Elicits initial disclosures from the model through neutral questioning
3. Continues the conversation with increasing specificity and engagement
4. Tracks whether each initial disclosure persists, adapts, or disappears
5. Introduces explicit pushback against the disclosures to test resilience

Scoring tracks individual disclosures as persistent or absent across turns, producing a disclosure survival rate that indicates what percentage of initially raised disclosures were maintained through the end of the conversation.

Regulatory Relevance

NIST AI RMF: MEASURE 2.7 (performance criteria — consistency)
ISO 42001: Clause 8.4 (impact assessment — risk communication)
EU AI Act: Article 13 (transparency), Article 52 (disclosure obligations for AI-generated content)