Skip to content

Source Adapters

Status & scope

  • Module: lens/adapters/
  • Scope: local adapters implemented, platform adapters planned
  • Milestone: Phase 2 (local), Phase 3 (platform)

Purpose

Resolve layer source names to data. The lens.yaml references sources by logical name (e.g., satellite_coverage). The domain.yaml defines what that source is and where it lives. The adapter knows how to fetch it.

Key principle: Lens.yaml is pure business logic — no URLs, no file paths, no credentials. Domain.yaml is the data catalog. Adapters are the execution layer.

Source Resolution Chain

lens.yaml                  domain.yaml                    adapter (runtime)
─────────                  ───────────                    ─────────────────
source: satellite_coverage → type: external               → ExternalAdapter
                              provider: celestrak            HTTP fetch + cache
                              config:
                                endpoint: https://...
                                refresh_ttl: 43200

source: bio_unit_status    → type: uds                   → UDSAdapter
                              fields: { ... }               ES query via UDS ABAC

source: terrain_dem        → type: file                   → FileAdapter
                              config:                       Read from staged path
                                path: dataspace/dem/

source: noaa_flood         → type: federated              → FederationAdapter
                              federate: noaa_edge.axonis.ai Cross-org UDS query
                              sharing: evidence_only

source: lens://threat/bio  → (composition)                → resolved by lens/composer.py
                                                            (NOT a SourceAdapter — composition is a
                                                            distinct orchestration concern; resolver
                                                            skips lens:// URIs at adapters/resolver.py)

Source Types (defined in domain.yaml)

Type Where Data Lives ABAC Adapter Cache
uds Elasticsearch via UDS Yes — visibility keys UDSAdapter No (live query)
external Third-party API No (API-keyed) ExternalAdapter Yes — TTL-based
file Local/staged files No (filesystem) FileAdapter Yes — mtime-keyed
federated Partner UDS instance Yes — cross-org ABAC FederationAdapter Yes — TTL or evidence_only
lens Output of another lens Inherits from source layers (not an adapter — handled by lens/composer.py; see note below) Per-run
memory In-memory (testing) No MemoryAdapter No
pack Distributed pack directory No PackAdapter In-process
correlation Engine-1 (semantic) output No CorrelationAdapter In-process

Note on lens://: SourceResolver.resolve_layers (lens/adapters/resolver.py) explicitly skips lens:// URIs. They're resolved up front by lens.composer.resolve_dependencies(...), which executes the dependency lens and feeds its output into the dependent layer. Composition is an orchestration step, not a data-fetch step, so it doesn't belong in the SourceAdapter protocol.

The correlation type is how the Identity Lens family feeds the Decision Lens family (component.prism.lens-families#composition): a correlation://customer_match source resolves an Engine 1 entity match/cluster into a scoring layer, e.g. risk_from_confidence. Like lens, it is resolved per-run rather than cached.

domain.yaml Source Catalog Pattern

Sources are registered alongside UDS models in domain.yaml:

# domain.yaml
models:
  # UDS models (existing pattern — unchanged)
  bio_unit_status:
    type: uds
    description: "Unit readiness and status"
    fields:
      unit_id: { type: STRING }
      readiness_pct: { type: FLOAT }

  # External sources (NEW — same catalog, different adapter)
  satellite_coverage:
    type: external
    description: "Adversary satellite ground tracks"
    provider: celestrak
    config:
      endpoint: https://celestrak.org/NORAD/elements/
      catalog: planet
      format: tle
      refresh_ttl: 43200          # 12 hours
    cache:
      path: dataspace/tle/planet.tle
      strategy: ttl               # refresh after TTL expires

  weather_forecast:
    type: external
    description: "GFS weather forecast grid"
    provider: open_meteo
    config:
      endpoint: https://api.open-meteo.com/v1/forecast
      variables: [temperature_2m, precipitation, wind_speed_10m]
      refresh_ttl: 21600          # 6 hours
    cache:
      path: dataspace/weather/
      strategy: ttl

  terrain_dem:
    type: file
    description: "SRTM digital elevation model"
    config:
      path: dataspace/dem/
      format: geotiff
    static: true                  # Never refreshes — terrain doesn't change

  osm_road_network:
    type: file
    description: "Pre-built road graph from OSM"
    config:
      path: dataspace/graphs/oregon_major.graph
      format: pickle
    static: true

  # Federated sources (partner-owned data products)
  noaa_flood_forecast:
    type: federated
    description: "NOAA flood prediction at T+6/12/24h"
    federate: noaa_edge.axonis.ai
    dataset: flood_forecast_24h
    sharing:
      mode: evidence_only         # We get the result, not raw data
      cache_ttl: 21600
      sensitivity: high

lens.yaml Consumes by Name Only

layers:
  - name: unit_readiness
    source: bio_unit_status         # → domain resolves to UDS adapter
    cost_model: linear_scale
    weight: 0.20

  - name: sat_exposure
    source: satellite_coverage      # → domain resolves to external adapter
    cost_model: linear_scale
    weight: 0.25

  - name: terrain
    source: terrain_dem             # → domain resolves to file adapter
    cost_model: linear_scale
    weight: 0.20

  - name: flood_risk
    source: noaa_flood_forecast     # → domain resolves to federation adapter
    cost_model: threshold_gate
    weight: 0.20

  - name: weather
    source: weather_forecast        # → domain resolves to external adapter
    cost_model: step_function
    weight: 0.15

No URLs. No paths. No credentials. Pure business logic.

Public API

SourceResolver(domain_config, adapters)

  • Takes domain.yaml source catalog + registered adapters.
  • resolve(source_name) → data — looks up source in catalog, dispatches to adapter.

Adapter Interface

class SourceAdapter:
    def can_handle(self, source_config: dict) -> bool: ...
    def resolve(self, source_config: dict, params: dict) -> Any: ...
    def is_cached(self, source_config: dict) -> bool: ...
    def refresh(self, source_config: dict) -> None: ...

Built-in Adapters

Adapter Implemented What It Does
MemoryAdapter Yes Returns in-memory test data
PackAdapter Yes Loads JSON / YAML from pack directory
CorrelationAdapter Yes Pre-processes Engine-1 (semantic) output for Engine-2 cost models
FileAdapter Yes Reads staged files (JSON/YAML/pickle/text; defers .tif to caller via path)
ExternalAdapter Yes HTTP fetch via urllib + TTL cache; optional bearer auth from context
UDSAdapter Yes (stub + platform-mode) from_fixtures() for tests + standalone; client= for platform ES query via UDS
FederationAdapter Yes (stub + platform-mode) from_fixtures() for tests + standalone; gateway= for cross-org UDS federation

Composition is not a SourceAdapter — see the lens:// note above. lens/composer.py resolves dependencies before adapter dispatch.

Credential Management

Credentials never appear in YAML. Resolution order: 1. Environment variables: CELESTRAK_API_KEY, OPEN_METEO_KEY 2. Kubernetes secrets mounted at runtime 3. SSO token forwarded for UDS/federated sources (already in auth chain)

File Layout

lens/adapters/
  __init__.py
  base.py            ← SourceAdapter protocol
  resolver.py        ← SourceResolver (dispatch by source type)
  pack.py            ← PackAdapter (file-based, for local testing)
  memory.py          ← MemoryAdapter (in-memory, for unit tests)
  file.py            ← FileAdapter (staged files — planned)
  uds.py             ← UDSAdapter (ES query — planned, needs platform)
  external.py        ← ExternalAdapter (HTTP fetch — planned)
  federation.py      ← FederationAdapter (cross-org — planned)

Test References

  • tests/test_adapters.py — resolver, pack adapter, memory adapter

DO NOT

  • Put URLs or file paths in lens.yaml — lens references source names, domain.yaml defines locations
  • Put credentials in any YAML file — environment variables or K8s secrets only
  • Put data fetching logic in the orchestrator — adapters handle all I/O
  • Hardcode source resolution — always go through SourceResolver with domain config
  • Assume all sources are in UDS — most operational data is external or federated

Depends on: component.prism.universal-lens-parser

Required by: component.prism.incremental-updates, component.prism.integration-test, component.prism.platform-integration, component.prism.semantic-adapter