A research platform for measuring semantic change in versioned knowledge, developing and testing drift metrics, and tracing changed meaning into RAG behavior.
Retrieval-augmented generation solves one reliability problem by grounding a model in external documents. It leaves another largely unmeasured: those documents are revised. A passage can remain highly relevant to a query while no longer carrying the same meaning it carried in an earlier version.
DARMA is a research platform for studying that change. It provides a repeatable experimental path from versioned documents to aligned passages, semantic geometry, controlled drift measurement, and downstream RAG testing. At its center is a four-metric drift suite developed to characterize distinct forms of semantic movement rather than collapsing revision into a single similarity score.
What it is for
Standard retrieval is optimized to find material that is similar to a query. It does not characterize how the semantic content of that material changed across versions. DARMA turns version-to-version meaning change into an observable research variable and provides an experimental platform for developing, comparing, and validating ways to measure it.
Compare semantically corresponding passages across revisions rather than relying on timestamps, lexical diffs, or document-level similarity alone.
Use DARMA's developed metric suite to distinguish translational movement, neighbourhood rewiring, distributional change, and changes in semantic concentration instead of treating drift as one undifferentiated score.
Use null, style-only, and substantive-scope conditions so a positive signal has to survive tests against both measurement noise and superficial rewording.
Run matched questions against different document versions and quantify how revised knowledge changes the resulting answers.
How it works
DARMA treats semantic drift as both a measurement problem and an experimental systems problem. The platform is built around a reproducible chain from versioned text to aligned semantic geometry, metric evaluation, controlled calibration, and downstream response analysis.
Start with document revisions whose relationship is known well enough to construct controlled comparisons.
Split each version into passages, then match corresponding chunks so the comparison is between semantic counterparts rather than arbitrary text windows.
Represent each aligned passage in embedding space and compute complementary drift metrics describing displacement and local structural change.
Use the null condition to estimate the measurement noise floor and the style-only condition as a negative control for superficial rewriting.
Load different source versions into the retrieval pipeline, ask matched questions, and compare the resulting answers with lexical and semantic divergence measures.
Core metric innovation
The metric suite is a central scientific contribution of DARMA. Rather than treating semantic drift as one distance value, the platform operationalizes four complementary measurements over aligned passage embeddings, each exposing a different geometric property of revision.
Measures how far an aligned passage moves from its earlier direction in embedding space.
Measures whether the passage's nearest semantic neighbours change across versions.
Compares the passage's neighbourhood-attraction profile between versions.
Measures whether the local semantic relationship becomes more concentrated or more diffuse.
Scientific design
The first DARMA validation study uses three conditions: a null control for the noise floor, a style-only revision that changes wording without intended scope change, and a substantive scope revision. The style condition is especially important because it gives the platform a falsification test: if a candidate drift measure fires on rewriting alone, it is not yet measuring what it claims to measure.
What the study found
The strongest of the four document-level detectors for substantive scope change at the selected 720/28 configuration.
The design was swept across nine chunk sizes and nine overlaps rather than assuming one segmentation scheme was neutral.
Across most configurations, substantively modified chunks enriched the high-translation, low-rewiring mode described in the discussion.
Downstream validation
When questions target revised sections, paired answers generated from different document versions show lower similarity and greater divergence than answers for unchanged sections. In the x1-grounded analysis, all five answer-level measures were statistically significant; in the x6-grounded analysis, the directional pattern remained but only ROUGE-L and BERTScore F1 retained significance.
The answer metrics also agree with one another. Cosine Divergence is negatively correlated with BLEU, ROUGE-L, and BERTScore F1, with the strongest relationship against BERTScore F1 at r = −0.84. That correlation is among answer-level measurements; it is evidence of measurement coherence, not a direct source-drift-to-answer correlation.
Research platform
DARMA is more than the four scores produced in one study and more than a single operating threshold. It is reusable experimental infrastructure for turning document revision into aligned, testable semantic measurements, evaluating how those measurements behave, and tracing meaningful change into retrieval-grounded model behavior.
Use a common controlled framework to compare DARMA's four drift metrics, examine their complementary behavior, and evaluate additional candidate measures against the same semantic-change conditions.
Construct controlled comparisons across document revisions and measure change at passage level rather than relying on whole-document summaries.
Study semantic change as displacement, neighbourhood rewiring, distributional divergence, and concentration change, preserving structure that a single aggregate score can hide.
Treat chunk size and overlap as experimental variables, exposing when a drift result depends on segmentation architecture.
Estimate an empirical baseline under a null condition and derive configuration-specific operating points rather than importing an arbitrary global threshold.
Use a style-only negative control to test whether a detector is responding to semantic change rather than surface reformulation.
Ask the same questions against different source versions and measure how changed knowledge propagates into changed model responses.
Current scope. The study establishes the first validated DARMA configuration on a controlled versioned-document corpus and a specific RAG pipeline. Its thresholds are configuration-dependent, and the paper's broader threshold-calibration phase remains unfinished. DARMA is therefore best understood at this stage as a research platform for controlled semantic-drift measurement and experimentation, not as a universal production alarm with one fixed cutoff.
Why it matters
RAG reliability depends not only on retrieving the right source, but on knowing what happened to that source as it changed. DARMA provides a research platform for studying that problem experimentally: develop and test drift measurements, separate substantive change from rewriting, characterize the geometry of revision, test the sensitivity of the measurement system itself, and observe what reaches the generated answer.
DARMA turns changing knowledge into an experimental object: measure how meaning moved, test how well the metrics detect it, and observe what changes downstream.
DARMA is a reusable research platform within the Computational Meaning Dynamics Research Group's broader work on making semantic change observable, measurable, and experimentally tractable in AI systems.