Measure when changing knowledge changes AI behavior.
Knowledge-dependent AI systems can change behavior even when the model, software, and prompts have not changed. A funding solicitation is revised, a clinical guideline is replaced, an eligibility rule is updated, or a technical standard changes; the new material is simply loaded into the system and becomes the basis for future answers and recommendations. The difficult problem is not knowing that the document changed. It is knowing whether the change in meaning has altered the system somewhere that was not intended. Organizations often have no reliable way to determine which routine knowledge updates are consequential, which prior conclusions or workflows may now behave differently, or which parts of an AI system actually need to be revalidated. In high-consequence settings, that creates a serious blind spot: system behavior can shift silently, as the knowledge beneath it changes.
A health system's pharmacy informatics team triages several thousand FDA label revisions, safety communications, and REMS updates a year across its formulary, deciding which ones require an urgent alert to prescribers and which are routine. They do it today by redline review.
That practice fails in both directions. A periodic FDA reformat touches every line of a label and produces a diff too large to read closely, so reviewers skim. A changed dosing ceiling or a newly added contraindication is often the smallest edit in the batch, a single altered number, one inserted “not”, indistinguishable by size from a hundred cosmetic ones beside it. A tool that just signals how much text changed is measuring the wrong thing, and the edit that mattered is the one most likely to be missed.
The same problem exists in MCP description poisoning (e.g. the rug-pull attack), government programmatics, legal-citation sanctions, and AI-enabled document management.
DARMA addresses this as a problem of semantic-change-impact rather than document comparison. Its drift metrics characterize how meaning moves across versions, providing the measurement layer for a risk-directed validation system for knowledge-dependent AI. DARMA's capability is not to flag every revision or preserve obsolete content. It is to identify when a knowledge change is likely to matter, where its effects may propagate, and where targeted revalidation is warranted. The problem appears wherever authoritative knowledge drives AI behavior: scientific program management and funding analysis, research opportunity matching, clinical guidance, regulatory and compliance systems, government benefits, defense and intelligence analysis, engineering standards, financial and insurance decision support, and enterprise policy assistants. See a whitepaper on scientific program management; continue reading more about DARMA.
Retrieval-augmented generation (RAG) solves one reliability problem by grounding a model in external documents. It leaves another largely unmeasured: those documents are revised. A passage can remain highly relevant to a query while no longer carrying the same meaning it carried in an earlier version. DARMA is a research platform for studying that change. It provides a repeatable experimental path from versioned documents to aligned passages, semantic geometry, controlled drift measurement, and downstream RAG testing. At its center is a four-metric drift suite developed to characterize distinct forms of semantic movement rather than collapsing revision into a single similarity score.
DARMA treats semantic drift as a measurable source of downstream AI risk. Standard retrieval can identify material that is relevant to a query, but it does not tell us whether the meaning of that material changed enough to alter the behavior of systems that depend on it. DARMA makes that change experimentally observable, providing a platform for developing and validating metrics that distinguish consequential semantic movement from routine revision.
Measure where authoritative knowledge has changed semantically, helping distinguish routine revision (e.g. style changes) from changes that warrant closer downstream testing (e.g. scope changes).
Use DARMA's metric suite to measure translational movement, neighborhood rewiring, distributional change, and changes in semantic concentration rather than reducing drift to a single similarity score.
Test signals against null, style-only, and substantive-scope conditions so meaningful change can be distinguished from measurement noise and superficial rewording.
Run matched questions against different knowledge versions to determine whether detected semantic change is accompanied by changes in retrieval-grounded answers and where further validation may be warranted.
DARMA treats semantic drift as both a measurement problem and an experimental systems problem. The platform follows the change from aligned source passages through semantic measurement and controlled calibration, then tests whether that change is reflected in downstream RAG behavior. The experimental approach was as follows:
Start with document revisions whose relationship is known well enough to construct controlled comparisons.
Split each version into passages, then match corresponding chunks so the comparison is between semantic counterparts rather than arbitrary text windows.
Represent aligned passages in embedding space and apply DARMA's CMD drift metrics to characterize how meaning changed, including displacement and changes in local semantic structure.
Calibrate the measurements against null and style-only controls so substantive semantic change can be distinguished from measurement noise and superficial rewriting.
Run matched questions against different knowledge versions and measure whether the detected semantic change produces a corresponding change in retrieval-grounded answers.
DARMA does not depend on a newly invented drift metric. It brings established geometric and information-theoretic measures together as complementary instruments for examining how meaning changes across aligned document versions. Each measure exposes a different property of the embedding space, allowing the experiments to test which signals distinguish substantive semantic change from routine rewriting and which are most informative for downstream validation.
Uses cosine distance between aligned passage embeddings to measure how far a passage moves from its previous semantic position. It provides the most direct measure of version-to-version displacement.
Compares the nearest-neighbor sets surrounding a passage across versions. A passage may move only modestly while its relationships to surrounding concepts change substantially, revealing structural change that displacement alone may miss.
Compares the distributions of similarity within a passage's local semantic neighborhood. It measures whether the pattern of relationships around the passage has changed, rather than only whether the passage itself moved.
Measures whether the local semantic neighborhood becomes more concentrated or more diffuse between versions. This provides another view of structural change that is complementary to displacement and neighborhood divergence.
Individual drift scores show that semantic structure changed, but different combinations of change can represent different kinds of revision. DARMA combines translational displacement and neighborhood rewiring into a geometric interpretation framework that separates stable passages from distinct modes of semantic movement.
A semantic-change signal is useful only if it responds to changes in meaning rather than ordinary editing or measurement noise. DARMA tests this directly using controlled document conditions with known differences. The same measurement pipeline is applied to a near-stable baseline, a style-only revision, and a substantive scope change, creating a falsifiable test of whether the observed geometry is actually sensitive to meaning.
The near-stable condition establishes the empirical noise floor: how much apparent drift can occur when little or no meaningful change is present. The style condition changes wording and presentation while preserving meaning, providing a negative control for superficial revision. The scope condition introduces substantive changes in ideas and content. Together, these conditions test whether the measured signal remains quiet for routine rewriting but becomes stronger, and geometrically different, when the underlying meaning actually changes.
Within RAG, semantic characteristics cannot be measured independently of the retrieval architecture. DARMA tested 81 combinations of chunk size and overlap and found that sensitivity to substantive change varied across the configuration space. Style-only revisions remained comparatively stable, while the strength of the scope-change signal depended substantially on how the document was divided. This makes chunking part of the measurement system rather than a neutral preprocessing choice.
The experiments select the operational chunking configuration empirically rather than assuming a conventional setting will generalize. This has an important implication for validation: drift thresholds and expected signal behavior should be calibrated for the knowledge representation on which they will actually operate. A detector validated under one chunking architecture cannot automatically be assumed to behave the same way under another.
Detecting semantic movement is only useful if the measurement can distinguish meaningful change from variations produced by the representation itself. DARMA first establishes an empirical noise floor using a near-stable control condition, then uses that boundary to examine whether scope changes separate from revisions that alter wording (e.g. style changes), while preserving meaning. The null condition establishes a baseline for expected variation. DARMA then compares style and scope revisions against that baseline to test whether substantive changes produce a distinguishable semantic signal.
After establishing the baseline and selecting the content configuration, DARMA evaluates how well each of the four measures distinguishes substantive scope change from the null-control condition. All four measures perform above chance, but they do not contribute equally. Cosine Drift provides the strongest individual discrimination in this experiment, with an AUROC of 0.758, followed closely by kNN Change at 0.742. Jensen-Shannon Divergence reaches 0.694 and Entropy Difference 0.663. The result supports using multiple measurements to characterize semantic change while also showing that translational displacement carried the clearest individual detection signal under the selected configuration.
Detecting semantic change in the source material is only part of the problem. DARMA next tests whether those changes are reflected in the behavior of a RAG system grounded in different document versions. The same questions are asked against each version, allowing answers tied to revised sections to be compared with answers tied to unchanged material. Questions targeting revised sections produced lower paired-answer similarity and greater divergence than questions targeting unchanged sections. The effect was strongest in the x1-grounded analysis, where all five answer-level measures were statistically significant. In the x6-grounded analysis, the same directional pattern remained, although only ROUGE-L and BERTScore F1 retained statistical significance. These results provide evidence that changes in source knowledge can propagate into generated answers, while also showing that the strength of the effect depends on the evaluation condition.
The answer-level measures also move together in the expected directions. Cosine Divergence is negatively associated with the similarity measures, with the strongest relationship observed with BERTScore F1 at r = −0.84. This does not measure a direct correlation between source-document drift and answer divergence; it shows that the different answer-level instruments are capturing a coherent pattern of response change.
DARMA extends beyond the results of a single controlled study. It provides reusable experimental infrastructure for studying how changes in authoritative knowledge can be measured, characterized, and traced into the behavior of AI systems that depend on that knowledge. The current experiments establish the measurement foundation; the broader platform supports continued work on identifying which changes matter, where their effects appear, and when targeted revalidation is warranted.
Align corresponding content across document versions and measure change at the passage level rather than reducing an entire revision to a single document-level score.
Use complementary geometric and information-theoretic measures to distinguish displacement, neighbourhood change, distributional divergence, and changes in semantic concentration.
Construct null, style-only, and substantive-change conditions to test whether observed signals respond to changes in meaning rather than surface rewriting or measurement noise.
Test how content size, overlap, and segmentation affect semantic-change detection so measurement behavior can be calibrated to the representation actually used by the system.
Establish empirical baselines and configuration-specific operating thresholds instead of assuming that one fixed cutoff will generalize across knowledge bases and retrieval architectures.
Run matched questions against different knowledge versions to determine whether semantic changes in source material are accompanied by changes in retrieval-grounded answers.
Extend semantic-change measurements toward identifying which revised knowledge is most likely to affect system behavior and where additional validation should be concentrated.
An AI system can remain technically unchanged while its behavior shifts because the knowledge it depends on has changed. DARMA makes that source of change experimentally tractable by measuring semantic movement across document versions, testing whether the observed signal exceeds expected variation, and examining whether those changes are reflected in generated answers.
The longer-term objective is risk-directed validation: using measurable properties of knowledge change to identify which updates are most likely to affect system behavior, where those effects may appear, and where revalidation should be concentrated. The current DARMA experiments establish the measurement and testing foundation needed to investigate that problem.