Incremental Updates and Triggers — Continuous Cost Lens Recomposition

Status & scope

• Status: Draft — ready to implement.
• Module: lens/runtime/incremental/ (new).
• Definition of done: streaming replay test green — a recorded LayerChangeEvent sequence replays to byte-identical RecompositionEvent/ThresholdCrossingEvent hashes (threat domain).
• Sister specs: parallax SPEC-11 (Correlation Persistence), parallax SPEC-27 (Continuous Fusion) — external repo.
• Milestone: Phase 4 (continuous operation).
• Date: 2026-04-27.

1. Problem

Cost Lens (Prism) today is request-response. An operator asks for a route, a threat surface, a coverage window, an action window — the framework composes the answer from current layer data and returns it. Evidence is captured per request.

This works for analytical workflows ("plan a route through the AO this morning") and demo scenarios. It fails for operational scenarios where:

1. Layer data changes continuously. A gauge reading updates; precipitation forecast revises; a TLE epoch advances; a no-fly polygon is added. The threat surface, route, coverage window, or action window the operator was looking at thirty seconds ago is now stale.
2. Re-composing the whole surface on every change is expensive. A flood threat surface over the Houston AO is a 1km grid of ~50,000 cells. A precipitation update at one gauge changes ~200 cells in a 5km radius. Recomposing all 50,000 cells on every gauge update is wasteful at best, untenable at sub-second latency.
3. No event-driven trigger. Source adapters (SPEC-13) currently fetch on demand. There is no path from "the source data just changed" to "recompose the affected portion of the lens output and emit a signal if a threshold crossed."
4. No streaming evidence chain. SPEC-04 (Evidence Model) captures evidence per composition. There is no event chain that says "at T, layer L changed; at T+50ms, cells C₁..C_n recomposed; at T+60ms, threshold X was crossed at cell C_k; signal emitted."
5. No replay determinism for the streaming case. FED-SPEC-02 (replay determinism) covers batch composition. Streaming replay — replaying a sequence of layer changes and verifying the same recomposition events fire byte-for-byte — has no spec yet.

The result: Prism is an analytical tool when the customer needs an operational system. Streaming Mesh (Xanadu) provides the transport, but transport without incremental compute is plumbing. This spec defines the compute.

2. Decision

Define an incremental recomposition pipeline that:

• Listens for layer-change events (source-data update, scheduled refresh, composition cascade, manual)
• Computes the change footprint for each event (the spatial / temporal / topological region affected)
• Recomposes only the affected portion of the lens output (sparse update over the footprint)
• Maintains an append-only event chain in the evidence model (extends SPEC-04) so the streaming case is replayable
• Emits signals when threshold crossings occur during recomposition
• Cascades to dependent lenses via lens:// references (extends SPEC-12)

This is not a rewrite of lens.runtime.compose. Batch composition stays for analytical and demo use. Incremental recomposition is a new pipeline that reuses the same primitives (cost models, weights, thresholds) but operates on layer-change deltas instead of full layer reads.

3. Architecture

Source-data update           Scheduled refresh           Composition cascade
(adapter detects change)     (layer TTL expires)         (referenced lens recomposed)
        │                            │                            │
        └────────────────────────────┼────────────────────────────┘
                                     │
                                     ▼
                       ┌─────────────────────────────┐
                       │  LayerChangeEvent           │
                       │   - layer_id                │
                       │   - change_kind             │
                       │   - footprint               │
                       │   - prev_data_hash          │
                       │   - new_data_hash           │
                       │   - source_event_ref        │
                       └──────────────┬──────────────┘
                                      ▼
       ┌──────────────────────────────────────────────────────────┐
       │  IncrementalRecompositionPipeline                        │
       │                                                          │
       │  1. RESOLVE FOOTPRINT                                    │
       │     Compute affected output region per domain            │
       │     (traversal: edge → routes; threat: cell footprint;   │
       │      observation: sat × time × grid; temporal:           │
       │      constraint → windows)                               │
       │                                                          │
       │  2. RECOMPOSE                                            │
       │     Apply weights × layer values over footprint only     │
       │     Cached per-layer rasters; only composition recomputes│
       │                                                          │
       │  3. THRESHOLD CHECK                                      │
       │     Per cell / route / window in the footprint, did any  │
       │     threshold transition? (above → below or vice versa)  │
       │                                                          │
       │  4. EMIT EVENTS                                          │
       │     RecompositionEvent (always)                          │
       │     ThresholdCrossingEvent (if any threshold crossed)    │
       │     CascadeEvent (if dependent lenses must recompose)    │
       │                                                          │
       │  5. SIGNAL EMISSION                                      │
       │     For threshold crossings, emit typed signal           │
       │     (signal_class = "computed", producing_engine =       │
       │      "prism", producing_lens_id = lens_id)               │
       └──────────────────────────────────────────────────────────┘
                                      │
                                      ▼
                       Promotion boundary into DES
                       (signal → InsightRoot if filter passes)

4. Per-domain incremental update patterns

Each Cost Lens domain has a different change-propagation shape. The pipeline dispatches to a per-domain footprint resolver.

4.1 Traversal (routing)

• Layer change: edge weight updates (weather, traffic, threat overlay, terrain wetness from threat-domain composition)
• Footprint: affected edges → routes that traverse them
• Strategy: maintain a route → edge reverse index. On edge-weight change, invalidate routes that touched that edge. Rerun A* only on invalidated routes (or on the affected sub-graph if the operator's query was an isochrone).
• Cost: O(invalidated_routes × A) — typically far less than O(all_routes × A).

4.2 Threat (risk surface)

• Layer change: one input layer updates (e.g., precipitation, temperature, threat polygon)
• Footprint: spatial extent of the layer change. A precipitation update at one gauge affects cells within an interpolation radius. A new threat polygon affects cells inside the polygon plus a buffer.
• Strategy: per-layer rasters are cached at the grid resolution. On layer change, write the new layer raster (full or sparse delta). Recompose the surface only over the changed footprint by re-applying weights × layer values.
• Cost: O(affected_cells × layer_count) — typically far less than O(total_cells × layer_count).

4.3 Observation (sensor coverage)

• Layer change: TLE update (satellite ephemeris advances), new satellite added, sky-mask change at a ground station
• Footprint: affected satellite × affected time range × visibility-affected grid
• Strategy: SGP4 propagation is fast (~µs per epoch). Re-propagate only the affected satellite over the affected time range. Recompute coverage on the visibility-affected grid cells.
• Cost: O(1 satellite × affected_time × affected_grid) — typically far less than O(all_sats × full_time × full_grid).

4.4 Temporal (action windows)

• Layer change: one constraint updates (weather forecast revision, fuel availability change, mission-window shift)
• Footprint: time interval where the constraint changed × downstream windows that intersected that interval
• Strategy: maintain windows as interval sets. On constraint change, recompute the intersection only over the affected time interval.
• Cost: O(affected_intervals) — typically far less than O(all_constraints × full_time).

5. Triggers

Five trigger sources, dispatched through one entry point.

Trigger	Source	Latency target	Notes
Source-data change event	Adapter detects (poll diff, CDC, ES write trigger, file watcher)	Sub-second once detected	Adapter responsibility extended in SPEC-13
Streaming-mesh message	Xanadu peer-to-peer event	Sub-second	Profile 3 deployment only
Scheduled refresh	Layer TTL expires (per SPEC-05)	Per TTL — minutes to hours	Time-decay layers (temporal forecasts, weather)
External webhook	Customer system pushes update	Sub-second once received	Customer-side integration; bounded by adapter validation
Composition cascade	A referenced lens recomposes (lens:// reference)	Sub-second after dependency recomposes	Topological dispatch via SPEC-12 composer

All triggers produce a LayerChangeEvent with a uniform schema (§7). The pipeline is trigger-source-agnostic past that point.

6. Evidence chain — extension to the evidence model

SPEC-04 captures evidence per composition. This spec extends it for the streaming case. Three new event types appended to the lens evidence chain:

6.1 LayerChangeEvent

event_type: layer_change
lens_id: houston_flood_threat_v1
layer_id: precipitation_gauge_data
timestamp: 2026-04-27T14:23:11.412Z
change_kind: data_update    # data_update | layer_added | layer_removed | weight_change
footprint:
  type: spatial
  bounding_box: [...]
  affected_cells: 217
prev_data_hash: sha256:...
new_data_hash: sha256:...
source_event_ref: adapter:gauge_KHOU_2026-04-27T14:23:11
trigger_source: source_change

6.2 RecompositionEvent

event_type: recomposition
lens_id: houston_flood_threat_v1
triggering_layer_change_id: lce_...
timestamp: 2026-04-27T14:23:11.487Z
footprint_summary:
  affected_cells: 217
  affected_routes: 0
  affected_windows: 0
prev_output_hash: sha256:...    # over the affected footprint only
new_output_hash: sha256:...
composition_version: 1.0.0      # weights / cost models version

6.3 ThresholdCrossingEvent

event_type: threshold_crossing
lens_id: houston_flood_threat_v1
triggering_recomposition_id: rec_...
timestamp: 2026-04-27T14:23:11.491Z
location: { cell_id: HC_2174 }   # or route_id / window_id
threshold_id: high_risk
prev_value: 0.71
new_value: 0.83
direction: crossed_above
signal_emitted: sig_...

6.4 Replay determinism — streaming case

Replay walks the event chain in order. Given the same source-data update sequence, the same LayerChangeEvent → RecompositionEvent → ThresholdCrossingEvent sequence must fire with byte-identical hashes. This is the streaming extension of FED-SPEC-02.

Constraints required for byte-identical replay:

• Event ordering — events must be totally ordered (timestamps + sequence number; ties broken deterministically).
• Footprint resolution determinism — given the same layer change, the same footprint must be computed (no sets, no hash-randomization-dependent ordering).
• Composition determinism — already required by FED-SPEC-02 for the batch case; same property must hold per-cell.
• Threshold check determinism — float comparison ordering pinned (no parallel reductions that change order between runs).

7. Schema

# lens/models/incremental.py (new)

@dataclass(frozen=True)
class LayerChangeEvent:
    event_id: str
    lens_id: str
    layer_id: str
    timestamp: datetime
    change_kind: ChangeKind  # DATA_UPDATE | LAYER_ADDED | LAYER_REMOVED | WEIGHT_CHANGE
    footprint: Footprint     # spatial / temporal / topological — domain-specific
    prev_data_hash: str
    new_data_hash: str
    source_event_ref: str
    trigger_source: TriggerSource

@dataclass(frozen=True)
class RecompositionEvent:
    event_id: str
    lens_id: str
    triggering_layer_change_id: str
    timestamp: datetime
    footprint_summary: dict
    prev_output_hash: str
    new_output_hash: str
    composition_version: str

@dataclass(frozen=True)
class ThresholdCrossingEvent:
    event_id: str
    lens_id: str
    triggering_recomposition_id: str
    timestamp: datetime
    location: dict           # cell_id | route_id | window_id
    threshold_id: str
    prev_value: float
    new_value: float
    direction: CrossingDirection  # CROSSED_ABOVE | CROSSED_BELOW
    signal_emitted: str | None

8. Public API

# lens/runtime/incremental/__init__.py

class IncrementalPipeline:
    def __init__(
        self,
        lens_spec: LensSpec,
        layer_cache: LayerCache,
        evidence_writer: EvidenceWriter,
        signal_emitter: SignalEmitter,
    ): ...

    def on_layer_change(self, event: LayerChangeEvent) -> list[Event]:
        """Process a layer change. Returns the cascade of events emitted."""
        ...

class FootprintResolver(Protocol):
    """Per-domain footprint computation. One implementation per Cost Lens domain."""
    def resolve(self, change: LayerChangeEvent, lens_spec: LensSpec) -> Footprint: ...

9. File layout

lens/
  runtime/
    incremental/
      __init__.py
      pipeline.py                ← IncrementalPipeline orchestration
      footprint/
        __init__.py
        traversal.py             ← TraversalFootprintResolver
        threat.py                ← ThreatFootprintResolver
        observation.py           ← ObservationFootprintResolver
        temporal.py              ← TemporalFootprintResolver
      triggers/
        __init__.py
        source_change.py         ← Adapter-driven trigger
        scheduled.py             ← TTL-driven trigger
        cascade.py               ← Composition-cascade trigger
        webhook.py               ← External webhook trigger
        streaming_mesh.py        ← Xanadu peer event trigger
      replay.py                  ← Streaming replay verification
  models/
    incremental.py               ← LayerChangeEvent, RecompositionEvent, ThresholdCrossingEvent

10. Acceptance criteria

• [ ] LayerChangeEvent, RecompositionEvent, ThresholdCrossingEvent data classes implemented (frozen)
• [ ] IncrementalPipeline.on_layer_change() dispatches to the per-domain resolver and emits the event chain
• [ ] Per-domain footprint resolvers implemented for all four Cost Lens domains (traversal, threat, observation, temporal) with at least one canonical scenario each
• [ ] Trigger sources implemented: source-change adapter callback, scheduled (TTL), composition cascade. Streaming-mesh and webhook may follow.
• [ ] Threshold-crossing detection emits typed signals through the existing signal emitter (extend SPEC-04 if needed)
• [ ] Streaming replay test: given a recorded sequence of LayerChangeEvents, replay produces byte-identical RecompositionEvent and ThresholdCrossingEvent hashes
• [ ] Performance test (threat domain reference scenario — Houston flood): single gauge update produces sub-second recomposition over the affected footprint (<200ms target on TRM-1 reference machine)
• [ ] Performance test (traversal domain reference scenario): single edge-weight change produces sub-second route invalidation + reroute over the affected sub-graph
• [ ] Composition-cascade verified: when a referenced lens recomposes, the dependent lens recomposes via the same pipeline (no special path)
• [ ] FED-SPEC-02 streaming-replay benchmark passes for the threat domain reference scenario

11. Non-goals

• Not a transport layer. Streaming Mesh (Xanadu) is the transport for Profile 3 deployments; this spec defines the compute that runs on top of it.
• Not a stream-processing framework. No Flink, no Beam, no Kafka Streams. Pure Python; events flow through the pipeline in-process; coordination across nodes is Xanadu's job.
• Not Entity Lens. Parallax has SPEC-11 (incremental fusion math) and SPEC-27 (continuous fusion pipeline). This spec is the Cost Lens equivalent.
• Not a UI / operator surface. Beacon's incremental rendering is downstream and out of scope; this spec defines the events Beacon will consume.

12. Open questions

• Per-cell evidence vs. footprint-summary evidence. SPEC-04 captures evidence per composition. For streaming, do we capture per-cell evidence on every threshold crossing, or footprint-summary evidence with per-cell drill-down on demand? Lean: footprint-summary by default; per-cell on threshold crossings.
• Cascade depth limits. SPEC-12 caps composition depth at 3. Same limit applies for cascade triggers? Lean: yes — same invariant.
• Backpressure under high-frequency layer changes. When a layer changes faster than the pipeline can recompose, what's the policy? Drop, batch, or queue with explicit lag metrics? Lean: batch within a configurable time window (default 100ms) with a debounce; emit one LayerChangeEvent per batch summarizing the deltas.
• Multi-layer simultaneous changes. When two layers change in the same window, is recomposition done once over the union footprint, or twice sequentially? Lean: once over the union — but the event chain captures both LayerChangeEvents with a single RecompositionEvent referencing both.

13. Cross-references

Spec	Relationship
parallax SPEC-11	Sister spec for Entity Lens incremental fusion math (UnionFind absorbing new records). This spec is the Cost Lens equivalent.
parallax SPEC-27	Sister spec for Entity Lens continuous fusion pipeline (event-driven trigger + signal emission). This spec mirrors its shape for Cost Lens.
SPEC-04 (axonis-lens)	Evidence model — extended here with three new event types.
SPEC-05 (axonis-lens)	Operational lifecycle — Init→Execute→Sync. The streaming case lives inside Execute.
SPEC-12 (axonis-lens)	Composition — extended here with composition-cascade triggers.
SPEC-13 (axonis-lens)	Source adapters — extended here with change-detection responsibilities.
FED-SPEC-02 (themis)	Replay determinism — streaming case is the V1 gap closed by §6.4 of this spec.
Streaming Mesh / Xanadu	Transport for Profile 3 deployments; the conduit on which LayerChangeEvents flow between Edge Nodes.
Themis Phase 4	Instruments Scenario A (5-node Profile 3) for benchmarking the streaming-replay determinism property defined here.

14. Implementation notes

This spec is DRAFT. Recommended implementation order:

1. Models first — LayerChangeEvent, RecompositionEvent, ThresholdCrossingEvent as frozen dataclasses. No pipeline yet.
2. Threat-domain footprint resolver — first concrete domain. Houston flood is the reference scenario.
3. Source-change trigger — adapter callback hook. Polls / CDC / file-watcher are the V0 detection mechanisms.
4. Pipeline orchestration — wire 1+2+3 together; emit event chain to existing evidence writer.
5. Streaming replay test — verify byte-identical replay on the threat domain reference. This closes the FED-SPEC-02 streaming gap for one domain.
6. Remaining domains — traversal, observation, temporal footprint resolvers (in this order; traversal has the most customer pull).
7. Cascade trigger — composition-cascade via SPEC-12.
8. Streaming-mesh trigger — Xanadu peer event ingestion. Requires Profile 3 deployment.
9. Webhook trigger — customer-side integration shape.

Themis Phase 4 instruments the resulting pipeline for the FED-SPEC-02 streaming-replay benchmark.

---

Owner: CPO. Aligned with parallax SPEC-11 + SPEC-27. Next review when the threat-domain reference implementation lands.

---

Depends on: component.prism.composition, component.prism.evidence-model, component.prism.operational-lifecycle, component.prism.source-adapters

Realizes: product.lens