Semantic sensemaking surveillance for agentic analyst pipelines
AI-mediated workflows increasingly transform evidence before a person or downstream system acts on it. Reports become summaries, events become labels, cases become priorities, and changing evidence is compressed into persistent objects such as alerts, scores, recommendations, and dashboard states. Once that happens, operational decisions may depend less on the underlying evidence than on the interpretation the workflow continues to use.
This creates a serious assurance gap. The evidence can change enough to alter what an event means while the workflow continues to execute normally. Decision support processes and automation can continue to execute correctly, preserve the same identifiers and labels, and produce coherent outputs even after the evidence has moved into a materially different interpretation. Nothing has to crash, hallucinate, or violate a process rule and the system will appear healthy while the interpretation driving attention, prioritization, confidence, escalation, or action no longer reflects what is happening.
AI systems increasingly turn changing evidence into persistent operational state: a case classification, risk score, recommendation, eligibility determination, threat assessment, care plan, or agent memory. Once created, that state is reused by later steps, other models, downstream software, and people who may never reconstruct the original evidence for themselves.
Dangerous failures can occur when the evidence changes enough that the maintained interpretation should change, but the state does not. The system may continue producing internally consistent outputs because nothing in its software, workflow, schema, or stored object is technically invalid. A fraud case can remain low risk after the transaction pattern changes. A clinical recommendation can remain appropriate to an earlier evidence state. A compliance determination can persist after the governing interpretation shifts. A threat or incident can continue to be treated under a frame that new evidence no longer supports.
As AI systems take on more persistent roles in decision-making, this gap becomes increasingly consequential. The system is still working, but there is no assurance that what it believes the situation to be still matches what is actually happening.
semOSIris addresses this as a problem of semantic continuity rather than workflow correctness. Its research framework examines whether the meaning represented by an AI system continues to change with the situation it is intended to represent. The goal is not to replace judgment or react to every new piece of information, but to identify when semantic change becomes significant enough that an existing interpretation should no longer be assumed to remain valid.
The research asks a different question from conventional AI assurance: as a situation changes, does the interpretation being used by the system change with it? semOSIris treats an evolving information stream as a sequence of semantic states and measures how those states move and reorganize over time. This makes it possible to distinguish ordinary variation from directional movement, instability, and more substantial changes in meaning without relying on the system's own labels or judgments to define the change. Doing that requires more than a single measure of difference; different forms of semantic movement reveal different properties of how meaning is changing.
A changing interpretation is not adequately described by a single similarity score. Meaning can move steadily away from where it began, shift abruptly between adjacent observations, reorganize around different concepts and relationships, or become more dispersed as competing interpretations emerge. semOSIris therefore measures several properties of semantic movement rather than treating every change as the same kind of change.
Translational drift measures how far the current semantic state has moved from an earlier reference state. It captures cumulative movement and helps identify cases where an interpretation has gradually become substantially different from where it began.
Local drift and semantic velocity measure change between successive states. They distinguish gradual evolution from periods in which meaning begins moving rapidly, even when the total distance from the starting point is still modest.
Neighborhood measures examine whether the concepts, terms, and semantic relationships surrounding an event are being reorganized. Jaccard-based and embedding-distribution measures can reveal a change in framing even when the underlying subject remains the same.
Entropy-based measures characterize whether the semantic state is becoming more concentrated or more dispersed. Changes in dispersion can help distinguish movement toward a new coherent interpretation from periods of unresolved semantic churn.
Reframing pressure combines movement away from the earlier state with evidence that the surrounding semantic structure is also changing. The resulting signal increases when meaning has moved substantially and reorganized around a different frame.
These measurements describe different properties of the same evolving semantic trajectory. Used together, they make it possible to distinguish a relatively stable interpretation from smooth directional change, semantic instability, and more substantial reframing.
The study tested using 1,032 real-world event-reporting records spanning 10 event families and 37 adjudicated reporting windows. Human coders established whether the operational frame had changed, while semantic-dynamics measures and multiple LLM coding conditions were evaluated independently against the same reference.
The best performing semantic signal was reframing pressure, followed closely by translational drift and embedding-distribution change. The best tested LLM condition still missed 8 of 17 human-adjudicated frame shifts. The result is notable because the semantic measurements were not derived from either the human labels or the LLM judgments: they provided an independent signal that meaning had materially changed.
Human coders outperformed every tested LLM condition on the frame-shift judgment. LLM agreement varied materially by model, prompt policy, and uncertainty handling. Even the best-performing tested LLM condition missed 8 of 17 human-adjudicated frame shifts. The result argues against treating an LLM judge as neutral ground truth: its agreement is itself a variable that must be measured and interpreted alongside independent semantic signals.
An interesting operational result came from triangulating reframing pressure with the LLM judgment rather than trusting either signal alone. When both indicated a shift, the evidence for reframing was more compelling. More importantly, high reframing pressure frequently identified human-adjudicated shifts that the LLM missed, creating a practical basis for deciding which cases should be reviewed and which can pass forward.
semOSIris is a working tool built around this research. It brings connects open source intelligence (OSI) with evolving events, their underlying information, model judgments, and semantic-dynamics measurements into the same operational view so changes in interpretation can be examined as they develop rather than reconstructed after the fact.
The tool organizes active events and their relationships in a common operational environment, allowing individual situations to be explored as new information accumulates and their interpretation develops over time.
The semantic view exposes the measurements behind an event's current state, including displacement, reorganization, dispersion, and reframing pressure, making it possible to see why an interpretation is being characterized as stable or changing.
The study suggests that semantic continuity is a distinct assurance problem. Independently measured semantic change recovered shifts in interpretation that LLM judges sometimes missed, while combining semantic measurements with model judgment provided a more reliable basis for deciding when an interpretation could continue to be used and when it should be reviewed. The implication is broader than the event-analysis setting used to illustrate the CMD approach: any AI system that maintains and reuses an interpretation of a changing situation can face the same failure.
AI systems increasingly have to interpret situations that continue to change after an initial classification, recommendation, or assessment has been made. The difficult problem is not simply detecting that new information arrived, but recognizing when enough has changed that the meaning of the situation itself should be reconsidered. That distinction matters wherever decisions depend on an interpretation that persists over time. semOSIris shows how that research can become a usable capability. It makes semantic movement visible, separates ordinary evolution from more substantial reframing, and combines independent semantic signals with model judgments to focus attention on the cases most likely to matter.
This reflects a broader goal of the Computational Meaning Dynamics Research Group: not only to study how meaning changes, but to turn those findings into instruments and tools that can be used in realistic settings. semOSIris is one implementation of that approach. The larger research opportunity is to extend these capabilities into other domains where AI systems must understand changing situations, preserve useful state over time, and recognize when what they previously believed no longer fits what is happening.
If you want to see a talk on the paper this research is based on click here. The presentation was created using a fully automated local small model pipeline and includes an AI voice clone of Stephen Russell narrating.