Home Recent Research People Papers Contact

DARMA: Drift-Aware Retrieval Metrics and Alignment

Measure when changing knowledge changes AI behavior.

Knowledge-dependent AI systems can change behavior even when the model, software, and prompts have not changed. A funding solicitation is revised, a clinical guideline is replaced, an eligibility rule is updated, or a technical standard changes; the new material is simply loaded into the system and becomes the basis for future answers and recommendations. The difficult problem is not knowing that the document changed. It is knowing whether the change in meaning has altered the system somewhere that was not intended. Organizations often have no reliable way to determine which routine knowledge updates are consequential, which prior conclusions or workflows may now behave differently, or which parts of an AI system actually need to be revalidated. In high-consequence settings, that creates a serious blind spot: system behavior can shift silently, as the knowledge beneath it changes.

Case in point

A health system's pharmacy informatics team triages several thousand FDA label revisions, safety communications, and REMS updates a year across its formulary, deciding which ones require an urgent alert to prescribers and which are routine. They do it today by redline review.

That practice fails in both directions. A periodic FDA reformat touches every line of a label and produces a diff too large to read closely, so reviewers skim. A changed dosing ceiling or a newly added contraindication is often the smallest edit in the batch, a single altered number, one inserted “not”, indistinguishable by size from a hundred cosmetic ones beside it. A tool that just signals how much text changed is measuring the wrong thing, and the edit that mattered is the one most likely to be missed.

The same problem exists in MCP description poisoning (e.g. the rug-pull attack), government programmatics, legal-citation sanctions, and AI-enabled document management.

DARMA addresses this as a problem of semantic-change-impact rather than document comparison. Its drift metrics characterize how meaning moves across versions, providing the measurement layer for a risk-directed validation system for knowledge-dependent AI. DARMA's capability is not to flag every revision or preserve obsolete content. It is to identify when a knowledge change is likely to matter, where its effects may propagate, and where targeted revalidation is warranted. The problem appears wherever authoritative knowledge drives AI behavior: scientific program management and funding analysis, research opportunity matching, clinical guidance, regulatory and compliance systems, government benefits, defense and intelligence analysis, engineering standards, financial and insurance decision support, and enterprise policy assistants. See a whitepaper on scientific program management; continue reading more about DARMA.

Retrieval-augmented generation (RAG) solves one reliability problem by grounding a model in external documents. It leaves another largely unmeasured: those documents are revised. A passage can remain highly relevant to a query while no longer carrying the same meaning it carried in an earlier version. DARMA is a research platform for studying that change. It provides a repeatable experimental path from versioned documents to aligned passages, semantic geometry, controlled drift measurement, and downstream RAG testing. At its center is a four-metric drift suite developed to characterize distinct forms of semantic movement rather than collapsing revision into a single similarity score.

From relevance to semantic movement

DARMA treats semantic drift as a measurable source of downstream AI risk. Standard retrieval can identify material that is relevant to a query, but it does not tell us whether the meaning of that material changed enough to alter the behavior of systems that depend on it. DARMA makes that change experimentally observable, providing a platform for developing and validating metrics that distinguish consequential semantic movement from routine revision.

Identify changes that may require revalidation

Measure where authoritative knowledge has changed semantically, helping distinguish routine revision (e.g. style changes) from changes that warrant closer downstream testing (e.g. scope changes).

Characterize how meaning changed

Use DARMA's metric suite to measure translational movement, neighborhood rewiring, distributional change, and changes in semantic concentration rather than reducing drift to a single similarity score.

Separate semantic change from editing noise

Test signals against null, style-only, and substantive-scope conditions so meaningful change can be distinguished from measurement noise and superficial rewording.

Test downstream behavioral impact

Run matched questions against different knowledge versions to determine whether detected semantic change is accompanied by changes in retrieval-grounded answers and where further validation may be warranted.

From knowledge change to measurable system impact

DARMA treats semantic drift as both a measurement problem and an experimental systems problem. The platform follows the change from aligned source passages through semantic measurement and controlled calibration, then tests whether that change is reflected in downstream RAG behavior. The experimental approach was as follows:

01

Capture the knowledge change

Start with document revisions whose relationship is known well enough to construct controlled comparisons.

02

Align corresponding content

Split each version into passages, then match corresponding chunks so the comparison is between semantic counterparts rather than arbitrary text windows.

03

Measure semantic movement

Represent aligned passages in embedding space and apply DARMA's CMD drift metrics to characterize how meaning changed, including displacement and changes in local semantic structure.

04

Separate signal from stylistic revision

Calibrate the measurements against null and style-only controls so substantive semantic change can be distinguished from measurement noise and superficial rewriting.

05

Test the downstream consequence

Run matched questions against different knowledge versions and measure whether the detected semantic change produces a corresponding change in retrieval-grounded answers.

A multi-metric view of semantic change

DARMA does not depend on a newly invented drift metric. It brings established geometric and information-theoretic measures together as complementary instruments for examining how meaning changes across aligned document versions. Each measure exposes a different property of the embedding space, allowing the experiments to test which signals distinguish substantive semantic change from routine rewriting and which are most informative for downstream validation.

Semantic displacementCosine Drift

Uses cosine distance between aligned passage embeddings to measure how far a passage moves from its previous semantic position. It provides the most direct measure of version-to-version displacement.

Neighborhood rewiringkNN Change

Compares the nearest-neighbor sets surrounding a passage across versions. A passage may move only modestly while its relationships to surrounding concepts change substantially, revealing structural change that displacement alone may miss.

Neighborhood distributionJensen-Shannon Divergence

Compares the distributions of similarity within a passage's local semantic neighborhood. It measures whether the pattern of relationships around the passage has changed, rather than only whether the passage itself moved.

Semantic concentrationEntropy Difference

Measures whether the local semantic neighborhood becomes more concentrated or more diffuse between versions. This provides another view of structural change that is complementary to displacement and neighborhood divergence.

Individual drift scores show that semantic structure changed, but different combinations of change can represent different kinds of revision. DARMA combines translational displacement and neighborhood rewiring into a geometric interpretation framework that separates stable passages from distinct modes of semantic movement.

Quadrant framework: endpoint translation (Cosine Drift) on the horizontal axis against  neighborhood rewiring (Jensen-Shannon Divergence) on the vertical axis, dividing the space into four labelled modes of semantic shift.
Translation and neighborhood rewiring reveal different patterns of semantic change. DARMA preserves that structure so knowledge changes can be characterized before testing whether they affect downstream AI behavior.

Testing whether the signal is real

A semantic-change signal is useful only if it responds to changes in meaning rather than ordinary editing or measurement noise. DARMA tests this directly using controlled document conditions with known differences. The same measurement pipeline is applied to a near-stable baseline, a style-only revision, and a substantive scope change, creating a falsifiable test of whether the observed geometry is actually sensitive to meaning.

The near-stable condition establishes the empirical noise floor: how much apparent drift can occur when little or no meaningful change is present. The style condition changes wording and presentation while preserving meaning, providing a negative control for superficial revision. The scope condition introduces substantive changes in ideas and content. Together, these conditions test whether the measured signal remains quiet for routine rewriting but becomes stronger, and geometrically different, when the underlying meaning actually changes.

Content granularity shapes detection sensitivity

Within RAG, semantic characteristics cannot be measured independently of the retrieval architecture. DARMA tested 81 combinations of chunk size and overlap and found that sensitivity to substantive change varied across the configuration space. Style-only revisions remained comparatively stable, while the strength of the scope-change signal depended substantially on how the document was divided. This makes chunking part of the measurement system rather than a neutral preprocessing choice.

The experiments select the operational chunking configuration empirically rather than assuming a conventional setting will generalize. This has an important implication for validation: drift thresholds and expected signal behavior should be calibrated for the knowledge representation on which they will actually operate. A detector validated under one chunking architecture cannot automatically be assumed to behave the same way under another.

Grid of heatmaps showing quadrant proportions across 81 chunking configurations for scope and style conditions.
Configuration sensitivityThe style-control signature remains stable across the 81 chunking configurations, while the strength of the substantive-change signal varies. Chunking is therefore part of the measurement design, not a neutral preprocessing choice.

Calibrating semantic-change detection

Detecting semantic movement is only useful if the measurement can distinguish meaningful change from variations produced by the representation itself. DARMA first establishes an empirical noise floor using a near-stable control condition, then uses that boundary to examine whether scope changes separate from revisions that alter wording (e.g. style changes), while preserving meaning. The null condition establishes a baseline for expected variation. DARMA then compares style and scope revisions against that baseline to test whether substantive changes produce a distinguishable semantic signal.

Histogram of Cosine Drift under the null control condition, heavily concentrated near zero, mean 0.0038, with a 95th-percentile noise floor marked at 0.0258.
Empirical noise floorAt the selected 720/28 chunk/overlap configuration, null-control Cosine Drift is concentrated near zero, with a mean of 0.0038. The 95th percentile, 0.0258, defines the study-specific boundary above which observed displacement exceeds the expected background variation at a 5% false-positive rate.
Scatter of scope-change and style-change passages in Cosine Drift versus Jensen-Shannon Divergence space, with dashed detection thresholds.
Scope versus style Using empirically derived detection boundaries, style-only revisions remain predominantly in the low-change region while substantive scope revisions occur more often in regions showing translational displacement, neighbourhood restructuring, or both. The separation provides evidence that the measurement framework is responding to changes in meaning rather than simply to rewritten text.

Measuring detection performance

After establishing the baseline and selecting the content configuration, DARMA evaluates how well each of the four measures distinguishes substantive scope change from the null-control condition. All four measures perform above chance, but they do not contribute equally. Cosine Drift provides the strongest individual discrimination in this experiment, with an AUROC of 0.758, followed closely by kNN Change at 0.742. Jensen-Shannon Divergence reaches 0.694 and Entropy Difference 0.663. The result supports using multiple measurements to characterize semantic change while also showing that translational displacement carried the clearest individual detection signal under the selected configuration.

ROC curves for the four drift metrics: Cosine Drift AUROC 0.758, kNN Change 0.742, Jensen-Shannon Divergence 0.694, Entropy Difference 0.663, all above the random-classifier diagonal.
Detection performanceAll four drift metrics carry signal above chance at the selected configuration. Cosine Drift leads with AUROC 0.758, followed by kNN Change at 0.742.

Testing whether knowledge change reaches AI behavior

Detecting semantic change in the source material is only part of the problem. DARMA next tests whether those changes are reflected in the behavior of a RAG system grounded in different document versions. The same questions are asked against each version, allowing answers tied to revised sections to be compared with answers tied to unchanged material. Questions targeting revised sections produced lower paired-answer similarity and greater divergence than questions targeting unchanged sections. The effect was strongest in the x1-grounded analysis, where all five answer-level measures were statistically significant. In the x6-grounded analysis, the same directional pattern remained, although only ROUGE-L and BERTScore F1 retained statistical significance. These results provide evidence that changes in source knowledge can propagate into generated answers, while also showing that the strength of the effect depends on the evaluation condition.

Per-question answer-level scores across five metrics for unchanged versus changed document sections.
Downstream RAG effectQuestions targeting revised sections produce lower paired-answer similarity and higher divergence than questions targeting unchanged sections. The effect is strongest and most consistently significant in the x1-grounded analysis.

The answer-level measures also move together in the expected directions. Cosine Divergence is negatively associated with the similarity measures, with the strongest relationship observed with BERTScore F1 at r = −0.84. This does not measure a direct correlation between source-document drift and answer divergence; it shows that the different answer-level instruments are capturing a coherent pattern of response change.

Correlation matrix of answer-level metrics; Cosine Divergence correlates negatively with all similarity measures, most strongly with BERTScore F1 at r = -0.84.
Answer-metric agreementAt the answer level, Cosine Divergence varies inversely with the similarity measures; its strongest relationship is with BERTScore F1 (r = −0.84). This is evidence that the answer-level measures are capturing a coherent divergence signal.

A research platform for semantic change and downstream impact

DARMA extends beyond the results of a single controlled study. It provides reusable experimental infrastructure for studying how changes in authoritative knowledge can be measured, characterized, and traced into the behavior of AI systems that depend on that knowledge. The current experiments establish the measurement foundation; the broader platform supports continued work on identifying which changes matter, where their effects appear, and when targeted revalidation is warranted.

Version-aware semantic comparison

Align corresponding content across document versions and measure change at the passage level rather than reducing an entire revision to a single document-level score.

Multi-axis change characterization

Use complementary geometric and information-theoretic measures to distinguish displacement, neighbourhood change, distributional divergence, and changes in semantic concentration.

Controlled validation

Construct null, style-only, and substantive-change conditions to test whether observed signals respond to changes in meaning rather than surface rewriting or measurement noise.

Representation sensitivity

Test how content size, overlap, and segmentation affect semantic-change detection so measurement behavior can be calibrated to the representation actually used by the system.

Detection calibration

Establish empirical baselines and configuration-specific operating thresholds instead of assuming that one fixed cutoff will generalize across knowledge bases and retrieval architectures.

Downstream behavior testing

Run matched questions against different knowledge versions to determine whether semantic changes in source material are accompanied by changes in retrieval-grounded answers.

Impact-directed experimentation

Extend semantic-change measurements toward identifying which revised knowledge is most likely to affect system behavior and where additional validation should be concentrated.

Why this matters for knowledge-dependent AI

An AI system can remain technically unchanged while its behavior shifts because the knowledge it depends on has changed. DARMA makes that source of change experimentally tractable by measuring semantic movement across document versions, testing whether the observed signal exceeds expected variation, and examining whether those changes are reflected in generated answers.

The longer-term objective is risk-directed validation: using measurable properties of knowledge change to identify which updates are most likely to affect system behavior, where those effects may appear, and where revalidation should be concentrated. The current DARMA experiments establish the measurement and testing foundation needed to investigate that problem.