Skip to content

Federated Quorum

Status & scope

  • Stage: DRAFT — ready to implement
  • Module: parallax/ops/fusion/federation/quorum.py (new) + additions to parallax/ops/fusion/models/lens_spec.py
  • Depends on: scoring-engine, correlation-persistence, three-phase-protocol, existing LineageAction + CorrelationStatus enums
  • Part of: Full Fusion Cycle initiative · enables the dissent-fidelity benchmark (themis FED-SPEC-03.2, external)
  • Milestone: M6 (Federation Protocol)
  • Definition of done: pytest tests/test_quorum.py green — all acceptance boxes checked

Problem

The three-phase protocol (SPEC-16) and run_multi_fusion (SPEC-16 §13) produce a match verdict per node pair and then recover transitivity with UnionFind. There is no notion of multiple federated nodes agreeing on a single correlation. When N nodes each hold an opinion on whether record X and record Y are the same entity, today's pipeline silently merges those opinions: UnionFind unions any pair above threshold, and the per-node verdicts that disagreed are lost.

Four failure modes this creates:

  1. No declared agreement rule. A lens cannot say "this correlation is only CONFIRMED if a majority of participating nodes agree." Agreement is implicit in whatever the clustering step happens to do, which is not auditable and not configurable per lens.
  2. Absence is conflated with rejection. When a node is offline or times out during a fusion run, its silence is treated identically to a no_match verdict. The audit trail cannot distinguish "node B said these are different people" from "node B never answered." FusionRun.missing_federates records that a node was absent at the run level, but nothing records per correlation that the node abstained.
  3. No quorum outcome on the record. A CorrelationRecord carries a single confidence and status. There is no field that says "4 of 5 nodes confirmed this; the quorum policy was majority; the policy was satisfied." A reviewer cannot reconstruct how a multi-party decision was reached.
  4. Dissent has nothing to attach to. SPEC-30 (Dissent Recording) needs a well-defined "the quorum decided X" event so it can record the verdicts that disagreed with X. Without an explicit quorum outcome, "dissent" is undefined.

themis FED-SPEC-03 §4 makes this the gating requirement for the dissent fidelity benchmark (FED-SPEC-03.2): "when contributors disagree, both perspectives survive in the audit chain with attribution." That benchmark is build status in FED-SPEC-12 precisely because SPEC-29 quorum + SPEC-30 dissent do not exist yet.

Decision

Add an explicit, declarative quorum layer between per-node verdicts and the correlation outcome.

  1. Quorum policy is declared in the lens. A new QuorumConfig on LensSpec (or, when running ad-hoc, passed to the evaluator directly) declares the policy: unanimous, majority, weighted, or n_of_m. No default that silently confirms — a lens with no quorum config falls back to the existing bilateral pipeline behaviour, unchanged.
  2. Quorum evaluation is a pure function over per-node verdicts. evaluate_quorum() takes a tuple of NodeVerdict and a QuorumConfig, and returns a QuorumOutcome. No I/O, no infrastructure, no clock dependency beyond an injectable timestamp. Deterministic: the same verdicts and policy always produce the same outcome.
  3. Absence is a first-class verdict: ABSTAIN. A node that is absent, timed out, or declined to participate is recorded as NodeVerdict(vote=ABSTAIN), never as NO_MATCH. The quorum policy decides how abstentions count (most policies treat them as non-votes against the denominator; see §"Partial participation").
  4. The quorum outcome is recorded on the correlation as an append-only event. A new LineageAction.QUORUM_EVALUATED event carries the policy, the tally, the per-node verdicts, and the resulting decision. The CorrelationRecord.status transitions per the outcome (CONFIRMED / PROPOSED / REJECTED), but the event is the truth — the status is a materialisation.
  5. No auto-routing, no state machine. evaluate_quorum() computes an outcome and record_quorum_outcome() appends an event. Neither advances a workflow, notifies a human, or triggers downstream actions. AI assists; a human still attests (Invariant 6). A satisfied quorum produces a PROPOSED-or-CONFIRMED correlation that a human may still attest, invalidate, or correct (SPEC-24).

Architecture

Quorum sits after Phase 3 consensus scoring (SPEC-16 §3) and before correlation persistence (SPEC-11). It is the bridge from "N nodes each scored this pair" to "the federation has an attributable, policy-governed opinion on this pair."

SPEC-16 Phase 3 (per-pair, per-node scores)
            │
            ▼
   collect_verdicts()       ── one NodeVerdict per participating node,
            │                  ABSTAIN for absent/timed-out nodes
            ▼
   evaluate_quorum(verdicts, policy)   ── PURE FUNCTION
            │
            ▼
      QuorumOutcome  ── {decision, policy, tally, dissenting_node_ids, ...}
            │
            ├──► record_quorum_outcome()  ── append QUORUM_EVALUATED event,
            │                                 transition CorrelationRecord.status
            │
            └──► SPEC-30 record_dissent()  ── for every verdict that disagrees
                                              with the quorum decision

Standalone mode: collect_verdicts() is fed from the per-node, per-pair scores already produced by run_multi_fusion / the three-phase pipeline (both run in-process, no infrastructure). Platform mode: the same NodeVerdict tuple is assembled by titan's coordinator from the FUSION_RUN_COMPLETE messages each node returns. parallax never sees the wire — it sees the verdict tuple.

Partial participation

A federated run never guarantees every node answers. SPEC-29 treats non-response as information, not failure.

  • A node that times out, is offline, or declines (no consent, policy boundary) yields NodeVerdict(vote=ABSTAIN, reason=...) for every pair it would have voted on.
  • count_abstentions_as controls the arithmetic:
  • "non_vote" (default): abstentions are excluded from participants (the denominator). A 3-node majority among 5 declared nodes where 2 abstained needs 2 of the 3 voters.
  • "against": abstentions count as NO_MATCH for the agreement test (strict mode for lenses where silence should block confirmation).
  • If participants < min_participants, the decision is INDETERMINATE and the correlation stays PROPOSED — never auto-confirmed on thin participation.
  • Consistency with the run-level audit: a node in FusionRun.missing_federates produces ABSTAIN verdicts in every correlation it touched, and the run is status == "partial" by SPEC-16 §7 construction.

Public API

Configuration model (additions to lens_spec.py)

# parallax/ops/fusion/models/lens_spec.py  (additions)

class QuorumPolicy(str, Enum):
    UNANIMOUS = "unanimous"   # every participating (non-abstaining) node must agree
    MAJORITY = "majority"     # > 50% of participating nodes agree
    WEIGHTED = "weighted"     # sum of weights of agreeing nodes >= weight_threshold
    N_OF_M = "n_of_m"         # at least `min_agreeing` nodes agree


@dataclass(frozen=True)
class QuorumConfig:
    """Declares how N federated nodes reach agreement on a correlation.

    Declared in the lens YAML under `identity_fusion.quorum`, or passed
    directly to evaluate_quorum() for ad-hoc runs. Absent => no quorum
    layer; the pipeline behaves exactly as it does today (SPEC-16 §13).
    """
    policy: QuorumPolicy
    # N_OF_M:
    min_agreeing: int = 0
    # WEIGHTED:
    node_weights: tuple[tuple[str, float], ...] = ()   # (node_id, weight) pairs
    weight_threshold: float = 0.0
    # Shared:
    min_participants: int = 2          # below this, quorum is INDETERMINATE
    count_abstentions_as: str = "non_vote"   # "non_vote" | "against"
    # Per-node agreement is decided against this threshold.
    confirmation_threshold: float = 0.70

QuorumConfig is attached to IdentityFusionConfig as an optional field (quorum: QuorumConfig | None = None), so existing lenses parse unchanged.

Verdict + outcome models (new — federation/quorum.py)

# parallax/ops/fusion/federation/quorum.py  (new module)

class NodeVote(str, Enum):
    MATCH = "match"          # node scored the pair >= confirmation_threshold
    NO_MATCH = "no_match"    # node scored the pair < confirmation_threshold
    ABSTAIN = "abstain"      # node absent / timed out / declined — NOT a no_match


@dataclass(frozen=True)
class NodeVerdict:
    """One federated node's opinion on a single candidate pair.

    Pure data. Assembled from SPEC-16 Phase 3 per-node scores (standalone)
    or from titan FUSION_RUN_COMPLETE messages (platform). `score` is the
    node's confidence in [0,1]; for ABSTAIN it is 0.0 and `reason` explains
    the absence ("timeout", "offline", "declined", "no_consent").
    """
    node_id: str
    vote: NodeVote
    score: float = 0.0
    per_field_scores: tuple[tuple[str, float], ...] = ()
    reason: str = ""              # populated for ABSTAIN
    lens_version: str = ""
    timestamp: str = ""


class QuorumDecision(str, Enum):
    CONFIRMED = "confirmed"           # quorum satisfied for MATCH
    REJECTED = "rejected"             # quorum satisfied for NO_MATCH
    NOT_REACHED = "not_reached"       # policy not satisfied either way -> stays PROPOSED
    INDETERMINATE = "indeterminate"   # fewer than min_participants voted


@dataclass(frozen=True)
class QuorumTally:
    match_votes: int
    no_match_votes: int
    abstentions: int
    participants: int                 # match + no_match (excludes abstentions
                                      # when count_abstentions_as == "non_vote")
    match_weight: float = 0.0         # WEIGHTED policy only
    no_match_weight: float = 0.0


@dataclass(frozen=True)
class QuorumOutcome:
    """Result of evaluating a quorum policy over a set of NodeVerdicts.

    Frozen and self-describing: carries enough to reconstruct WHY the
    decision was reached without re-running the evaluator. This is the
    object recorded on the correlation and read by SPEC-30.
    """
    decision: QuorumDecision
    policy: QuorumPolicy
    tally: QuorumTally
    verdicts: tuple[NodeVerdict, ...]            # all node verdicts, in node_id order
    agreeing_node_ids: tuple[str, ...]           # nodes that voted with the decision
    dissenting_node_ids: tuple[str, ...]         # voted AGAINST the decision (drives SPEC-30)
    abstaining_node_ids: tuple[str, ...]
    threshold_detail: str = ""                   # human-readable rule that fired

Evaluation — the pure function

def evaluate_quorum(
    verdicts: tuple[NodeVerdict, ...],
    config: QuorumConfig,
) -> QuorumOutcome:
    """Evaluate a quorum policy over per-node verdicts. PURE.

    - Deterministic: identical (verdicts, config) -> identical QuorumOutcome.
    - No I/O, no clock, no infrastructure.
    - ABSTAIN verdicts are partitioned out per config.count_abstentions_as:
        "non_vote"  -> excluded from participants/denominator
        "against"   -> counted as NO_MATCH for the agreement test
    - If participants < config.min_participants -> decision = INDETERMINATE.
    - Policy rules:
        UNANIMOUS : every participant voted MATCH -> CONFIRMED;
                    every participant voted NO_MATCH -> REJECTED; else NOT_REACHED.
        MAJORITY  : match_votes > participants/2 -> CONFIRMED;
                    no_match_votes > participants/2 -> REJECTED; else NOT_REACHED.
        N_OF_M    : match_votes >= min_agreeing -> CONFIRMED;
                    no_match_votes >= min_agreeing -> REJECTED; else NOT_REACHED.
        WEIGHTED  : match_weight >= weight_threshold -> CONFIRMED;
                    no_match_weight >= weight_threshold -> REJECTED; else NOT_REACHED.
    - dissenting_node_ids = participants whose vote != the decision's vote.
      (For NOT_REACHED / INDETERMINATE there is no decision, so there is no
       dissent yet — dissent is defined only relative to a reached decision.)

    Raises QuorumConfigError for incoherent config (e.g. N_OF_M with
    min_agreeing == 0, WEIGHTED with empty node_weights).
    """


def collect_verdicts(
    pair: tuple[str, str],
    per_node_scores: dict[str, ScoredPair | None],
    expected_nodes: tuple[str, ...],
    confirmation_threshold: float,
    absent_reason: dict[str, str] | None = None,
    timestamp: str = "",
) -> tuple[NodeVerdict, ...]:
    """Assemble NodeVerdicts for one candidate pair.

    For each node in expected_nodes:
      - score present and >= confirmation_threshold -> MATCH
      - score present and  < confirmation_threshold -> NO_MATCH
      - score is None / node missing                -> ABSTAIN with reason
        (from absent_reason, default "no_response")

    `per_node_scores` is keyed by node_id and produced from SPEC-16 Phase 3
    (standalone) or coordinator-assembled FUSION_RUN_COMPLETE results
    (platform). Pure — no I/O.
    """

Recording the outcome on the correlation

# parallax/ops/fusion/federation/quorum.py

def record_quorum_outcome(
    cr: CorrelationRecord,
    outcome: QuorumOutcome,
    fusion_run_id: str,
    timestamp: str = "",
) -> CorrelationRecord:
    """Append a QUORUM_EVALUATED LineageEntry and transition status.

    Append-only (Invariant 2): never rewrites prior lineage. The event's
    `details` carries the serialised tally, policy, and per-node verdicts so
    the outcome is reconstructable from the ledger alone.

    Status transition (materialisation of the event, never the source of truth):
        CONFIRMED      -> CorrelationStatus.CONFIRMED, accuracy CONFIRMED(+3)
        REJECTED       -> CorrelationStatus.REJECTED,  accuracy KNOWN_WRONG(-3)
        NOT_REACHED    -> CorrelationStatus.PROPOSED  (unchanged; awaits more
                          nodes or human attestation)
        INDETERMINATE  -> CorrelationStatus.PROPOSED  (insufficient participation)

    Does NOT auto-route, notify, or advance any workflow. A human may still
    attest/invalidate/correct afterwards (SPEC-24); quorum is machine
    agreement, attestation is human sign-off (Invariant 6).
    """

New LineageAction value

# extends LineageAction enum (correlation.py)
class LineageAction(str, Enum):
    # ... existing ...
    QUORUM_EVALUATED = "quorum_evaluated"   # NEW — SPEC-29

New exception

# parallax/ops/fusion/lens/errors.py  (addition)
class QuorumConfigError(LensValidationError):
    """A QuorumConfig is internally incoherent or cannot be satisfied:
    N_OF_M with min_agreeing <= 0, WEIGHTED with empty node_weights or
    weight_threshold <= 0, min_participants < 1, or count_abstentions_as
    not in {"non_vote", "against"}. Fail fast at parse/evaluate time."""

Lens YAML surface

identity_fusion:
  initial_threshold: 0.50
  confirmation_threshold: 0.70
  match_function: [...]
  quorum:                          # NEW — optional. Omit => no quorum layer.
    policy: majority               # unanimous | majority | weighted | n_of_m
    min_participants: 2
    count_abstentions_as: non_vote # non_vote | against
    # n_of_m only:
    # min_agreeing: 3
    # weighted only:
    # node_weights: { sc_central: 2.0, firm_a: 1.0, firm_b: 1.0 }
    # weight_threshold: 3.0

parse_lens() (SPEC-01) parses this block into QuorumConfig and validates it via evaluate_quorum's coherence checks (raising QuorumConfigError). A lens without a quorum: block parses with IdentityFusionConfig.quorum = None.

Interaction with the Three-Phase Protocol (SPEC-16)

  • Quorum consumes the output of Phase 3 (consensus scoring). It does not change any phase. Phase 1/2/3 privacy guarantees are untouched: quorum operates on confidence scores only, never raw PII or feature vectors.
  • For a bilateral run (2 nodes), MAJORITY and UNANIMOUS degenerate to "both agree." This is the existing SPEC-16 behaviour, now made explicit.
  • For N-party run_multi_fusion (SPEC-16 §13), per-pair scores already exist for every node pair. collect_verdicts() reframes a pair's per-node scores into votes; evaluate_quorum() applies the policy. UnionFind clustering is unchanged and still runs — quorum gates which pairwise edges are CONFIRMED before the edge feeds clustering, it does not replace clustering.
  • A node absent at the run level (FusionRun.missing_federates) yields ABSTAIN verdicts for every pair it would have voted on. The run-level FusionRun.status == "partial" and the per-correlation ABSTAIN records are consistent by construction.

Invariants

  1. Quorum is declarative. The policy lives in the lens (or an explicit QuorumConfig argument), never hard-coded in the evaluator.
  2. Absence is ABSTAIN, never NO_MATCH. A missing or timed-out node never counts as a vote against a match. collect_verdicts() enforces this.
  3. Evaluation is pure and deterministic. evaluate_quorum() has no I/O and no clock; identical inputs yield identical outputs. This is what makes the benchmark reproducible (themis FED-SPEC-03 statistical methodology).
  4. The event is the truth. record_quorum_outcome() appends a QUORUM_EVALUATED event carrying the full tally and verdicts. The CorrelationRecord.status is a materialisation of that event.
  5. No auto-routing. Quorum computes and records. It never advances a workflow, notifies, or triggers downstream action. Humans still attest (Invariant 6).
  6. Append-only. A re-run that re-evaluates quorum appends a new QUORUM_EVALUATED event; it never edits the prior one (Invariant 2).

Testing

Tests in tests/test_quorum.py.

Policy-evaluation tests (pure function)

  • test_unanimous_all_match_confirms
  • test_unanimous_one_no_match_not_reached
  • test_majority_three_of_five_confirms
  • test_majority_tie_not_reached
  • test_n_of_m_meets_min_agreeing_confirms
  • test_weighted_meets_threshold_confirms
  • test_weighted_below_threshold_not_reached
  • test_below_min_participants_indeterminate
  • test_evaluate_quorum_is_deterministic — same input, 100 runs, identical output

Abstention handling

  • test_abstain_excluded_when_non_vote — abstentions don't change the denominator
  • test_abstain_counts_against_when_configured
  • test_absent_node_is_abstain_not_no_match — collect_verdicts with missing node
  • test_timeout_reason_preserved_on_abstain

Recording

  • test_record_quorum_outcome_appends_event — lineage grows by exactly one
  • test_quorum_event_carries_full_tally — details reconstruct the outcome
  • test_confirmed_outcome_sets_status_confirmed
  • test_not_reached_leaves_status_proposed
  • test_record_is_append_only — prior lineage byte-identical after recording

Config validation

  • test_n_of_m_zero_min_agreeing_raises
  • test_weighted_empty_weights_raises
  • test_lens_without_quorum_block_parses_quorum_none

Integration

  • End-to-end 5-node run_multi_fusion with a majority lens: verify each confirmed correlation carries a QUORUM_EVALUATED event whose tally matches the per-node scores; verify P018 (the false-positive trap) is NOT_REACHED because fewer than a majority of nodes scored it above threshold.
  • Partial-failure: drop one of five nodes; verify that node's ABSTAIN appears in every affected correlation's quorum event and that FusionRun.status == "partial".

Acceptance

  • [ ] QuorumPolicy enum + QuorumConfig dataclass added to lens_spec.py
  • [ ] IdentityFusionConfig.quorum: QuorumConfig | None field added (back-compatible)
  • [ ] parse_lens() parses + validates the quorum: block; raises QuorumConfigError
  • [ ] federation/quorum.py with NodeVote, NodeVerdict, QuorumDecision, QuorumTally, QuorumOutcome
  • [ ] evaluate_quorum() pure function with all four policies
  • [ ] collect_verdicts() maps per-node scores to verdicts, absence -> ABSTAIN
  • [ ] record_quorum_outcome() appends QUORUM_EVALUATED, transitions status
  • [ ] LineageAction.QUORUM_EVALUATED added to enum
  • [ ] QuorumConfigError added to lens/errors.py
  • [ ] All tests pass
  • [ ] One documented example: examples/quorum_majority_demo.py (5-node majority run)

What themis FED-SPEC-03 measures against this spec

The dissent-fidelity benchmark (FED-SPEC-03.2, ticket THEMIS-2-2) treats SPEC-29 as the quorum half of its gate. The harness will assert:

  • Q1 — Quorum outcome is reconstructable. For every correlation produced under a quorum lens, the QUORUM_EVALUATED event's tally and verdict list reproduce the QuorumOutcome when re-evaluated by evaluate_quorum(). (Determinism invariant.)
  • Q2 — Abstention is not rejection. In a fixture where one node is offline, the harness asserts that node's verdict is ABSTAIN (with a reason), never NO_MATCH, in every correlation it touched.
  • Q3 — Dissent set is well-defined. QuorumOutcome.dissenting_node_ids is exactly the set of participants whose vote disagreed with the reached decision — the input SPEC-30 consumes. The harness checks this set is non-empty on the deliberate disagreement fixtures (FED-SPEC-07 §5.2 Multi-INT, 2 of 8 ground-truth entities).

Layering: this spec vs the consensus session (SPEC-36)

This spec is the pure compute layer: evaluate_quorum() is a deterministic function over a fixed set of verdicts. It has no clock, no deadlines, no state.

The service layer that accumulates contributions over time — deadlines, required_contributors, minimum_authority_sum, WITHDRAWN/IN_CONFLICT session states — is SPEC-36 (Consensus Session), implemented at the titan/REST layer per CL-06. The session calls this spec's evaluation when its quorum-satisfaction rules say the verdict set is complete. The split keeps the state machine out of the parallax library (CLAUDE.md Task Pattern boundary) while making the evaluation independently testable and replayable.

The combiner (SPEC-28) produces a joint_confidence and conflict_indicator from weighted per-contributor scores; a weighted quorum policy MAY consume the combiner's contributor weights (composite_rating), and conflict_indicator > conflict_threshold is a second dissent-emission path recorded by SPEC-30.

Cross-references

  • SPEC-30 (Dissent Recording) — consumes QuorumOutcome.dissenting_node_ids; every dissenting verdict becomes a DissentRecord and a DISSENT_RECORDED event.
  • SPEC-16 (Three-Phase Protocol) — quorum consumes Phase 3 output; privacy guarantees untouched.
  • SPEC-24 (Attestation Rationale) — a satisfied quorum still produces a record a human may attest/invalidate/correct; quorum is machine agreement, attestation is human sign-off (Invariant 6).
  • SPEC-11 (Correlation Persistence) — record_quorum_outcome() writes via the CorrelationStore.append_lineage() interface; works against any backend.
  • SPEC-28 (Multi-Contributor Combination) — combiner math; weights and conflict_indicator feed quorum policies and dissent emission.
  • SPEC-36 (Consensus Session) — the service-layer session (formerly circulated as SPEC-29 in the CPO-direction series) that drives when evaluation happens.
  • themis FED-SPEC-03 §4 (themis/docs/FED-SPEC-03-FEDERATION-TRUST.md) — the dissent-fidelity benchmark this spec gates.

Open Questions

  1. Weighted policy with abstaining weighted node. When a node carrying weight in node_weights abstains, does its weight drop from the denominator entirely, or does weight_threshold stay fixed (making confirmation harder)? Current draft leaves weight_threshold fixed — i.e. an abstaining heavyweight makes the bar harder to clear. Needs a fixture-backed decision with the themis owner.
  2. Tie-break for even-N MAJORITY. A 2-of-4 / 2-of-4 split is NOT_REACHED today. Some customers may want a configurable tie-break (e.g. defer to a designated authority node). Deferred — NOT_REACHED is the safe default until a customer asks.
  3. Re-quorum cadence. When a previously-absent node comes back online in a later run, do we re-evaluate quorum automatically (appending a second QUORUM_EVALUATED) or only on explicit re-run? This intersects SPEC-25 (decay/reconfirm scheduling) and SPEC-27 (continuous fusion); resolve jointly there rather than building a trigger here (no auto-routing — Task Pattern).
  4. Per-correlation vs per-pair quorum identity. For N>2 a correlation may be built from several pairwise edges. This draft evaluates quorum per candidate pair and lets UnionFind cluster confirmed edges. Whether a cluster-level quorum (all members agree the cluster is one entity) is additionally required is left to a follow-up once the per-pair version is measured by themis.

Depends on: component.parallax.correlation-persistence, component.parallax.scoring-engine, component.parallax.three-phase-protocol

Realizes: product.fusion

Required by: component.parallax.consensus-session, component.parallax.dissent, component.parallax.wire-message-families