Skip to content

axonis-core Shared Library

Status: Implemented — axonis-core repo active; auth, elastic, redis, gateway, userspace, uds, schema, logger all in use across services. Athena slim extraction tracks remaining flatten work. Package: axonis-core (PyPI: axonis-core) Depends on: None (foundation package) Milestone: P1 (prerequisite for all other platform specs)

Purpose

axonis-core is the lightweight shared library imported by every Axonis service and client. It provides authentication, Elasticsearch access, Redis access, memory, the UDS CRUD base class, all userspace domain object classes, schema constants, protocol contracts, logging, and utility functions.

It must have no ML, imaging, geospatial, or heavy compute dependencies. Target: <20 direct dependencies, all under 5MB total.

Package Structure

axonis/
  __init__.py
  schema.py                      # Schema index aliases, INDICES mapping, Token — platform constants only
  logger.py                      # Logging configuration
  exceptions.py                  # Shared exception hierarchy
  decorators.py                  # @timeit, @deprecated, @ignore_exception, @federate, @required
  ansi_colors.py                 # Terminal color codes (used by display.py and logger.py)

  # Utility modules (split from former misc.py)
  env.py                         # getenv_bool, strtobool, is_edge_node, send_to_federation
  settings.py                    # AxonisSettings — pydantic-settings base class; extend in every service (see platform.service-configuration)
  ports.py                       # ServicePorts — canonical port assignments for all services
  singletons.py                  # CombinedSingletonMeta, ThreadSafeSingletonMeta, lazy_property, init_request_singleton_scope
  text.py                        # clean_elastic_id, clean_id, clean_term, abbreviate, finditem, find_keys, flatten_dict
  timeutils.py                   # datetime_from_string, now_in_milliseconds
  files.py                       # load_yaml_into_dict, unzip
  display.py                     # display_bytes, display_seconds, display_duration, show_size
  k8s.py                         # get_cpu_size_in_cores, get_pod_size_in_bytes, get_kube_resources

  auth/
    __init__.py
    oauth.py                     # OauthAuthentication singleton (Keycloak)
    oauth_token.py               # OAuth token helpers
    token_payload.py             # TokenPayload dataclass
    decorators.py                # @requires_auth, @requires_role
    middleware.py                # OAuthMiddleware (ASGI; allows /health + /service-info unauthenticated)

  elastic/
    __init__.py
    client.py                    # Elasticsearch connection + CRUD
    async_client.py              # Async Elasticsearch client
    query.py                     # Query builder (ElasticQuery)
    validation.py                # Schema validation (validate_schema, validate_queries)

  redis/
    __init__.py
    client.py                    # Redis connection wrapper

  gateway/
    __init__.py
    client.py                    # Federation HTTP client
    federate.py                  # Federate wrapper
    oracle.py                    # OracleRegistration (lifespan register/deregister)
    service_info.py              # ServiceInfo, ServiceDiscovery

  memory/
    __init__.py
    service.py                   # MemoryService — wraps Memory(UDS) + Redis with graceful degradation
    conversation.py              # ConversationStore — Redis-backed conversation turns
    embedder.py                  # embed() helper
    extractor.py                 # MemoryExtractor — LLM-driven fact extraction from chat turns

  audit/
    __init__.py
    middleware.py                # AuditMiddleware

  http/
    __init__.py
    client.py                    # RestClient (OAuth2 — was rest_client.py)

  mcp/
    __init__.py
    client.py                    # MCPClient async wrapper (was mcp_client.py; requires mcp extra)
    models.py                    # Shared Pydantic response models (MCPError, OperationResult, etc.)
    utils.py                     # tool_result, error_result, clean_tool_schemas helpers

  llm/
    __init__.py
    spec.py                      # Spec, Response, ToolCall, StreamChunk — config + normalized response/stream types
    client.py                    # Client — fully-featured provider-agnostic client: complete() + stream(), tool-aware, all 5 providers

  k8s/
    __init__.py
    client.py                    # KubernetesClient — thin wrapper around the kubernetes Python SDK
    ingress.py                   # IngressManager — create/update/delete networking.k8s.io/v1 Ingress resources
                                 # Used by Oracle for dynamic service routing (see component.oracle.gateway, platform.ingress-routing)
                                 # Base class moved from Forge; Forge retains its Dask/Seldon deployment subclasses

  uds.py                         # UDS base class
  fusion.py                      # Message, MessageType (federation wire types)

  operations/
    __init__.py
    dispatcher.py                # Op dispatch + attach_token helper (two-cluster execution)

  userspace/
    __init__.py                  # Re-exports all domain classes
    data.py                      # DatasetMetrics, Embedding, Query
    intelligence.py              # Insight, InsightRoot, InsightEvent, Signal, Block, Memory,
                                 # Domain, Experience, Accountability, DecisionTemplate,
                                 # EditionSnapshot, Profile, QueryIntent, Report, Task,
                                 # TaskTemplate, WorkflowTask, VizIntent
    alerting.py                  # AlertEvent, Notification, Subscriber, AlertSubscriber, AlertThreshold
    common.py                    # UserPreferences

  # ML ops dispatch — NOT in axonis-core. Lives at rest/uds/:
  #
  #   rest/uds/ops/      — feature_engineer dispatch + OPS registry +
  #                        22 op modules (bin, builtin, command,
  #                        correlation, encode, engineer, feature, fft,
  #                        freetext, fuse_wherenull, image, label,
  #                        operations, pca_transform,
  #                        pca_transform_visualization, scale, sequence,
  #                        timeseries, tsne, umap, user). Imported as
  #                        `from uds.ops import ...`.
  #
  #   rest/uds/libs/     — codecs, topological_graph, imagery,
  #                        augmentation, utils, mergers, plot_object,
  #                        _imgaug_compat. Imported as `from uds.libs
  #                        import ...`.
  #
  #   rest/uds/userspace/ — Dataset, TrainedModel, Model, Predict,
  #                        Serving, DataPipeline, etc. Imported as
  #                        `from uds.userspace import ...`.
  #
  #   rest/uds/core/     — analyzer, dataspace, dmoset, storage, parser,
  #                        federate_query, dmo, airflow, dm_utils, utils,
  #                        templates/. Imported as `from uds.core import ...`.
  #
  #   rest/uds/adapters/ — adapter, artifact, attribute, normalize, type,
  #                        utils, image, ingest, constant. Imported as
  #                        `from uds.adapters import ...`.
  #
  # axonis-core does NOT re-export these. Consumers (titan, FATE-Flow)
  # import from `uds.*` directly.

TransformSchema (PCA name, SEED constant) lives in axonis.schema alongside Schema and Token (axonis/schema.py:212). Other ML-specific schema classes (TrainingStatus, NLP, Pytorch, LibrarySchema) stayed with the ops/libs migration; check current axonis.schema for their exact home.

Ops dependency groups live in rest/pyproject.toml (uv [dependency-groups], not pip extras under axonis-core): [ops] (numpy/pandas<2/dask/dask-ml/sklearn/scipy/imageio/setuptools), [ops-feature] (umap-learn), [ops-image] (imgaug, opencv-python-headless, matplotlib, scikit-image), [ops-timeseries] (statsmodels), [ops-nlp] (nltk, spacy, huggingface-hub, sentence-transformers, en_core_web_md), [ops-geo] (geodex>=1.0.0). No [ops-fusion] group exists — parallax fusion ops are dormant until the numpy>=2.2.6 / numpy<1.29 standoff between parallax and the ops base group is resolved (see rest/pyproject.toml comment block, lines 117-121).

axonis-core itself defines only three extras: memory, otel, dev.

Lives elsewhere: - userspace/fusion.py (Lens, LensBinding, EntityMatch, EntityCluster, FusionRun, Correlation) — parallax - Heavy userspace (Dataset, Model, TrainedModel, Predict, Serving, DataPipeline, DataPipelineConnection, Asset, Project, Notebook, Logs, Workflow, Plot, Plotter, Embedding, Explain, Federate, Query, DatasetMetrics, DatasetObject, ConfusionMatrix, ModelArtifact, ModelSource, ModelStorage) — rest/uds/userspace/ - ML core modules (analyzer, dataspace, dmoset, storage, parser, federate_query, dmo, airflow, dm_utils, utils, templates/, federate_registration) — rest/uds/core/ - UDS adapters (adapter, artifact, attribute, normalize, type, utils, image, ingest, constant) — rest/uds/adapters/

Dependencies

[project]
requires-python = ">=3.10"
dependencies = [
    "Authlib>=0.15.5",
    "requests>=2.28.1",
    "urllib3>=1.26.0",
    "elasticsearch>=8.0.0,<9.0.0",
    "redis>=4.0.0",
    "dill>=0.3.0",
    "PyYAML>=6.0",
    "PyJWT>=2.8.0",
    "jsonschema>=4.0.0",
    "jsonpath-ng>=1.5.0",
    "httpx>=0.27.0",
    "pytz>=2022.1",
]

[project.optional-dependencies]
mcp = ["mcp>=1.26.0", "anyio>=4.0"]

Invariants

  1. No heavy dependencies. axonis-core must never depend on numpy, pandas, torch, GDAL, dask, scikit-learn, or any package >5MB. These belong in rest/uds/.
  2. Every userspace class extends UDS. All CRUD operations go through UDS.create(), UDS.read(), UDS.update(), UDS.delete(). No direct Elasticsearch calls from userspace classes. Exception: infrastructure-orchestrator classes in rest/uds/userspace/ that coordinate external systems (Airflow, Seldon/K8s) may extend Elastic directly so their super() CRUD calls reach ES without the UDS federation fan-out layer. These classes must remain in rest/uds/ and must not be added to axonis-core.
  3. Schema constants are the single source of truth. Every index alias, object type, and INDICES mapping lives in schema.py. Services must not define their own constants.
  4. Authentication is mandatory. Every service importing axonis-core must call OauthAuthentication for token validation. The @requires_auth decorator enforces this on REST endpoints.
  5. Memory is a UDS object. The Memory(UDS) class persists facts in the Elasticsearch memory index (mapping defined by the deployment's templates dir; see memory_mapping.json in rest/uds/templates/). Semantic recall composes ElasticQuery with kNN filters over the embedding dense_vector field. Recall through MemoryService is strictly per-service — the service field on every record gates reads. Cross-service knowledge transfer is Apollo's job (component.oracle.apollo), not a property of the shared index.
  6. MCPClient requires the mcp extra. Import fails gracefully with a clear message if not installed.
  7. MemoryService is the only memory abstraction. Services must not construct Memory(UDS) directly for conversational memory. Use MemoryService for all memory operations. Direct Memory(UDS) construction is only acceptable for non-conversational bulk reads (e.g., admin tools, analytics).
  8. Memory scoping is enforced by MemoryService. Callers pass conversation_id; MemoryService enforces the scoping rules. Services must not implement their own scoping logic.
  9. ConversationStore is ephemeral. Redis conversation turns are not permanent memory. Permanent facts must be written to Memory(UDS) via MemoryService.store(). Do not treat Redis conversation turns as durable state.

Auth Pattern

from axonis.auth.oauth import OauthAuthentication

# In Flask/Starlette middleware:
auth = OauthAuthentication()
token_payload = auth.validate(request.headers.get("Authorization"))
# token_payload.username, token_payload.roles, token_payload.federate_domain

All services must validate tokens on every request. The gateway (oracle) validates once; backend services validate the forwarded token.

UDS Pattern

from axonis.uds import UDS
from axonis.schema import Schema

class Lens(UDS):
    def __init__(self, alias=Schema.LENS):
        super().__init__(alias)
        self.namespace = Schema.USERSPACE

All userspace classes follow this pattern, except infrastructure-orchestrator classes (see invariant 2 exception).

Memory Pattern

Memory(UDS) at axonis.userspace.intelligence. UDS shell over the memory index. Index mapping (embedding as dense_vector) ships with the deployment's templates dir (see rest/uds/templates/memory_mapping.json).

Any service can store and recall memories, but recall is strictly per-service: every MemoryService.recall(...) is scoped to (user_id, service) — a service only ever reads memories it wrote. The service field on each record gates reads, not just writes; cross-service knowledge transfer is the responsibility of Apollo (component.oracle.apollo), not MemoryService. Apollo observes activity across every service, synthesises distilled artifacts (PromptShims, FailurePatterns, etc.), and pushes them back to each service through the guidance attach channel.

All services use MemoryService rather than constructing Memory(UDS) directly:

from axonis.memory.service import MemoryService

memory = MemoryService(conversation_id=..., user_id=..., profile_id=...)
await memory.store(content, memory_type="fact", tags=[...])
results = await memory.recall(query, limit=10)

MemoryService availability tiers

MemoryService degrades gracefully — no errors are raised when backends are unavailable:

Condition Write behaviour Read behaviour
Redis + ES available Write Redis (cache, TTL-bound) + ES (persist) Read Redis first
Redis only Write + read Redis Skip ES writes silently
ES only Skip Redis; write + read ES directly —
Neither available No-op writes Empty reads; services operate without memory

Memory scoping convention

Enforced by MemoryService, not by callers. Every recall is filtered by (user_id, service) first; additional per-memory-type rules layer on top:

  • All recalls — scoped to service == self.service. A service only ever reads memories it wrote. No cross-service recall through MemoryService for any memory_type. Cross-service knowledge transfer is Apollo's responsibility — see component.oracle.apollo for the guidance-attach contract.
  • memory_type="context" — additionally scoped by source_conversation_id. Only the originating conversation reads its own context memories.
  • memory_type="preference", "fact", "instruction" — user-wide within the same service. If the same user expresses the same preference to two different services, each service records its own copy; Apollo's synthesis is responsible for noticing the pattern and proposing a cross-service artifact through its guidance channel.

Redis key format

axonis:conv:{conversation_id}:{key} — no service prefix. Service-neutral so multiple services sharing the same Redis instance do not collide or interfere.

Low-level access (non-conversational only)

For non-conversational bulk reads (admin tools, analytics), direct Memory(UDS) access is acceptable:

  • Store: memory.create({"content": ..., "embedding": embed(content), "memory_type": ..., "tags": [...]}). Embedding is computed by the caller (oracle exposes the endpoint) — axonis-core stays ML-free.
  • Filter recall: memory.read({"query": {"bool": {"filter": [...]}}}).
  • Semantic recall: ElasticQuery(memory, {}).add_filter({"knn": {"field": "embedding", "query_vector": ..., "k": 10, "filter": [...]}}).

Reference: rest/uds/userspace/query.py:25.

ConversationStore Pattern

Redis-backed store for active conversation turns. Falls back to stateless (empty history) if Redis is unavailable.

  • Keys: axonis:conv:{conversation_id}:turns — list of {role, content, timestamp} dicts
  • TTL: configurable, default 3600s
  • Max turns: configurable, default 100 (FIFO eviction)
  • No service prefix — service-neutral
  • All services that implement chat use this
from axonis.memory.conversation import ConversationStore

store = ConversationStore(conversation_id=...)
await store.append(role="user", content="...")
history = await store.get_history()
await store.clear()

LLM Pattern

axonis-core owns the platform's single LLM client; each service still owns its own tool-loop and guardrail logic. Core's responsibility is to make that client fully-featured enough that a consumer-owned loop is trivial to build on it — the provider abstraction, tool-awareness, and streaming live in core; the loop does not.

Client (llm/client.py) — one provider-agnostic client for every backend. Client(Spec).complete(messages, tools=None) returns a typed Response; Client(Spec).stream(...) yields StreamChunks for incremental delivery. Providers: anthropic (Messages SDK) and openai / groq / ollama / trinity (OpenAI-compatible /chat/completions). The only hard SDK dependency is anthropic; the OpenAI-compatible providers use httpx. Client dispatches internally by Spec.provider — there is no separate provider router.

Response (llm/spec.py) — the single normalized result, returned by complete() and carried on the terminal of a stream:

  • text, model, input_tokens, output_tokens, raw
  • tool_calls: list — normalized {id, name, input}; empty when the model returned plain text
  • stop_reason — e.g. end_turn, tool_use

tool_calls is always fully-formed: while streaming, argument fragments are accumulated internally and the parsed tool calls are attached only to the terminal Response.

StreamChunk (llm/spec.py) — one stream increment: content_delta (a text fragment to forward immediately) or final (the assembled Response, set only on the terminal chunk). Draining a stream and reading .final yields a Response equivalent to complete()'s — same text, tool_calls, stop_reason, model, and token counts — so streaming is a strict superset of complete(). (raw may be None on a streamed Response for OpenAI-compatible providers, which expose no single response object; streamed token counts depend on the provider emitting a usage chunk.)

The tool-loop stays with the consumer. Core does not ship the multi-turn loop. Because Client is tool-aware (tools in → tool_calls out) and streaming-capable, a service wraps its own short loop around it: call → if tool_calls, run them under the service's guardrails and append the results → re-prompt → repeat until the model returns text. This keeps the loop, guardrails, tool dispatch, and observation where they belong — in the service:

  • Oracle runs its own loop with HTTP-registry dispatch + role guardrails + Apollo guidance/observation, and streams via Client.stream() (component.oracle.gateway).
  • Standalone services run a local-dispatch loop, or — for one-shot completions with no tools — call Client.complete directly (axonis.mcp.chat.register_chat_tools).
  • Apollo is the deliberate exception: it runs its own purpose-built client (oracle/apollo/llm.py), not core Client. Its Curator / admin-chat needs — strict-JSON synthesis, tool_choice control, an in-process local model, a deterministic stub harness — are intentionally outside core's lightweight surface (see component.oracle.apollo §Apollo owns its LLM client).

Spec.from_env(prefix="{SERVICE}_LLM") carries per-service selection (provider / model / keys); is_configured() is False when no model or key is set.

Import boundaries

Services import from axonis.* for foundation (auth, elastic, redis, gateway, memory, MCP, LLM, UDS base) and from uds.* for the heavy ML surface (uds.userspace, uds.ops, uds.libs, uds.adapters, uds.core). No re-export shims.

Operations Dispatcher

axonis.operations.dispatcher is the shared entry point for feature-engineering op calls. It lives in axonis-core rather than any one service because the same payload (command + parameters) is dispatched from two Dask topologies: a frontend-hosted LocalCluster for sample previews and the Titan-hosted DistributedCluster for training (component.fedai-rest.dataspace, component.titan.runtime). Routing follows operation_dispatch in fedai-rest/charts/fedai-rest/values.yaml — peer-owned commands go via REST/MCP, the rest fall through to uds.ops.feature_engineer.Engineer.

DistributedCluster workers have no Authenticator singleton. Callers must therefore pass the token through attach_token(body, token) (writes body["_auth_token"]) before submission; the helper is idempotent and a no-op on LocalCluster, so submission code calls it unconditionally.

Test Expectations

  • All userspace classes must have CRUD roundtrip tests (mock Elasticsearch)
  • Auth must have token validation tests (mock Keycloak)
  • Memory must have store/recall tests (mock Redis)
  • MCPClient must have connection + tool call tests (mock MCP server)
  • Zero imports of numpy, pandas, torch, or any ML library in the test suite
  • MemoryService must have availability degradation tests (mock Redis unavailable; mock ES unavailable; mock both unavailable)
  • MemoryService must have scoping tests (context memories not visible across conversation_id boundaries; preference/fact/instruction memories visible user-wide)
  • ConversationStore must have fallback tests (returns empty history when Redis unavailable)

Required by: component.axonis-core.library, component.beacon.ticketing, component.beacon.workbench, component.conduit.service, component.cortex.intelligence, component.geodex.operations, component.oracle.apollo, component.oracle.gateway, component.parallax.service-interface, component.postern.proxy, component.prism.service-interface, component.sentinel.alerting, component.titan.runtime, component.xanadu.messaging, platform.apollo, platform.axonis-client, platform.devops-cicd, platform.ingress-routing, platform.observability, platform.service-configuration, platform.service-contract