Self-Aware Infrastructure

1 · Abstract

The problem

Modern observability tooling is good at collecting signals. Metrics, logs, traces, and events all arrive reliably. What stays manual is the interpretation: separating noise from signal, correlating related failures, deciding whether something is an incident, remembering what happened last time, and finding the root cause.

This stack addresses that gap with a layered architecture:

• Reflection. The triage agent reduces raw workload state to a single authoritative Ready level per application — still the only signal that opens an incident on its own. The same agent streams four precursor signal families over the same channel: RED (request rate, error rate, duration → RequestErrorAnomaly) and USE (cgroup/PSI saturation and utilization → ResourcePressureAnomaly), both derived from an embedded eBPF collector with application code unchanged; observed service topology, also eBPF-derived, compared against the declared graph (→ GraphDrift); and windowed per-service log error/warn counts (→ LogErrorAnomaly). All four are capped at signal class: they absorb into incidents and backfill onto them, never open one directly. Kubernetes events, provider webhook events, and human reports feed the same pipeline.
• Awareness. Triage classifies every event (deterministic rules first, embeddings second, an LLM only as final arbiter), maintains a live service graph as the level source of truth, and manages the incident lifecycle against it.
• Cognition. X-Ray provides the durable memory substrate. Every observation and every conclusion is a MemoryCell custom resource, indexed in an in-process HNSW semantic graph and consolidated by reflection and knowledge loops into stable facts and operational patterns.
• Intelligence. The intelligence layer sits on top: root-cause analysis over a five-layer fractal incident graph, executable Kubernetes skills, and conversational AI over the whole system. Today, a human closes the loop: the system observes, classifies, correlates, predicts, and reports, and a 24/7 team follows every triage incident and decides what to do about it. Acting on a trusted root cause directly, through the same MCP and A2A tool surfaces, is a roadmap phase (§11), not current behavior.

The whole stack runs as Kubernetes controllers. CRDs serve as the event schema and the memory format, controllers manage the agent lifecycle, etcd is the durable store, and informers feed the reflection layer.

2 · Design philosophy

The Alive Agent pattern

The design starts from an observation recorded in ADR-000: Kubernetes already provides the hard parts of an agent runtime. Distributed state, desired-versus-actual convergence, self-healing, and schema extension are solved problems there. The stack uses them in three ways:

• CRD as event schema and memory format. Anything that happened or is known becomes a custom resource: an Application's health, an incident (MemoryCell kind=active_concern), a learned fact (kind=stable_fact). Operators can inspect what the agent knows with standard cluster tooling.
• Controller as agent runtime. The agent is not request-scoped. It is a controller-managed goroutine holding world state, an event hub, a session service, and tool registries in process, restarted only when its config actually changes. Per ADR-001, this is the first design to drive an LLM agent directly from the Kubernetes reconcile loop rather than from an external queue.
• Reconciliation as consistency maintenance. The same watch-and-converge pattern that keeps Deployments healthy keeps the agent's beliefs consistent: a level reconciler converges incident state to graph truth, a cell syncer converges the semantic index to etcd, and strict one-way write topologies prevent the loops from feeding each other.

A second principle applies at every layer: decisions are made by the cheapest adequate mechanism. Static rules run first, vector similarity second, and a generative model only when neither is enough. LLM calls are the most expensive and least predictable component, and when one fails the pipeline fails closed.

2.1 Reasoning tiers: reflex, System 1, System 2

The cheapest-adequate-mechanism principle generalizes into a three-tier gradation of reasoning depth. The System 1 / System 2 names come from Daniel Kahneman's dual-process model (Thinking, Fast and Slow, 2011): System 1 is fast, automatic, and cheap, recognizing patterns without effort; System 2 is slow, deliberate, and expensive, and it reasons. Kahneman's model has only the two systems; the zero-token reflex tier beneath them is this architecture's extension. The tiers are orthogonal to the pipeline layers: each layer contains work at more than one tier.

Three properties make the gradation robust rather than decorative:

• Each tier degrades to the one below. The guardrail fails closed. When the System-1 daily budget exhausts, reflection, knowledge, routing, and RCA suspend while the reflex tier keeps classifying, reconciling, and sweeping, and the world context reports the degraded mode explicitly. If the System-2 session disappears, triage behaves exactly as it did before one existed.
• System 2 is a consumer, not a component. The substrate is model-agnostic context infrastructure. The System-2 tier connects through the same OAuth MCP endpoint as any external client, reads the same world context, incident, prediction, and memory tools, and writes through the same memory write path (watch items, corrections, superseded patterns). It upgrades on its own cadence, is billed on its own budget, and holds no privileged access.
• The schedule is policy, not code. The TriagePolicy cell governs the System-1 consolidation cadence. The system2-delegate preset sets the reflection and knowledge intervals to zero for deployments where an external System-2 session drives memory synthesis on demand through build_memory and build_knowledge; the passes stay functional, only their timers stop.

The division of labor follows the tiers' cognitive profiles. Reflex owns everything with a deterministic answer, including dimensions a model would waste tokens rediscovering (the chronic-signal sweep exists because event count and age need arithmetic, not reasoning). System 1 owns fast, frequent, bounded interpretation. System 2 owns what needs breadth: noticing that a memory cell, a reopen counter, and a prediction disagree about the same service, or that a "self-recovers" pattern learned from per-cycle observations is normalized deviance at the weekly timescale. The first production System-2 sessions (2026-07) produced exactly this class of output: provenance-stamped world context, a budget self-report, reopen-count surfacing, the chronic-signal sweep itself, and correction cells that the next System-1 consolidation consumes.

2.2 CoALA: the memory and action structure underneath

Kahneman's tiers describe how expensively a decision gets made. Cognitive Architectures for Language Agents (CoALA; Sumers, Yao, Narasimhan & Griffiths, 2023) describes what the decision runs on: an agent as a set of memory modules (working, episodic, semantic, procedural), an action space split into internal actions that update those modules (retrieval, reasoning, learning) and external actions that touch the world (grounding), and a decision loop that proposes, evaluates, and selects the next action. The two frameworks are orthogonal by construction — one about cost, the other about structure — and this stack maps onto CoALA's structure directly, because the memory substrate was built to that shape before this document existed:

• Working memory is LiveState and the per-tenant status cells rendered into WorldContext() — the live, in-process view every pipeline stage and every model call reads.
• Episodic memory is incidents and events persisted as MemoryCells (active_concern, event), plus the fractal incident graph's historical layer (§8) — past occurrences a root-cause pass can retrieve.
• Semantic memory is the stable_fact and operational_pattern cells the knowledge pass writes, the service graph itself as structured world knowledge, and the corpus maps that index code, docs, and runbooks by meaning.
• Procedural memory is the closest fit, and worth stating plainly: the compiled classifier caps are the system's innate, unlearned procedure, and learned reflex_rule cells are the system rewriting that procedure from its own incident history — CoALA's procedural-memory update, not an analogy for it. The TriagePolicy cell, which governs how every tier behaves, belongs to this module too.
• Internal actions are retrieval (HNSW semantic search, the corpus maps' hybrid keyword/semantic fusion, RAG over similar past incidents), reasoning (RCA, the routing LLM), and learning (the reflection and knowledge passes writing their conclusions back into memory).
• External actions — grounding — are, today, notifications, incident records, and status surfaces: the loop still closes through a human who reads them and decides. The MCP and A2A tool surfaces are the grounding channel a future decision would act through; autonomous remediation over that channel is a roadmap item (§11), not current behavior.
• The decision loop is the classifier pipeline's staged, fail-fast evaluation (§6.1), followed by the routing LLM's proposal and selection wherever reflex could not settle the case.

Three places the mapping is an adoption rather than a copy, and they are the places this stack differs from the paper on purpose. The decision procedure is compiled code, not a prompt-driven planner: reflex proposes and selects on its own for most events, and the model is consulted as the arbiter of the remainder rather than as the thing that runs the loop. Working memory is durable — WorldContext() is refreshed into a singleton note cell, so the agent's live view of the world survives a restart instead of being the ephemeral scratchpad CoALA describes. And the cycle is not agent-invoked: it is a controller reconcile that runs on every event and every sweep tick, per tenant, whether or not anyone is asking a question.

2.3 Self-tuning classification

The procedural module has one loop that closes on its own. A periodic, leader-elected pass picks the tenants whose flap-suppression counters or reopen counts suggest their current thresholds are mis-set, and spends one System-1 call per candidate on an evidence snapshot: incident lifecycle timelines, readiness at the moment of each recurrence, how often suppression fired and how often the not-ready bypass let a recurrence through, the neighbouring recovery and reopen windows that shape the same pattern without being knobs, and excerpts from the tenant's own memory about that service. The model returns a structured proposal. Deterministic code disposes of it.

What keeps the loop bounded is the disposal, not the prompt. The tunable surface is one field pair on one section of one tenant overlay — the flap-suppression minimum-reopens count and the not-ready bypass window — inside bounds tighter than the policy validator's own, and a proposal outside them is rejected rather than silently clamped. Proposals land in a separate policy cell the loop owns, merged after the operator's document, so the operator's value always wins and reverting the entire trial is a single cell deletion. At most one applied change per tenant per day. Every proposal, applied or not, is ledgered as a dated note cell carrying the old and new values, the model's rationale, and the snapshot it was derived from, so the loop is auditable after the fact rather than only observable while it runs. The whole thing sits behind a kill switch that is off by default and disables both halves at once: no proposals, and the loop's cell ignored at merge time. Its first mode is shadow — proposals are ledgered and never applied.

One prompt rule is worth stating because it is easy to violate by accident: the system prompt may explain the mechanism a knob acts on, and may never contain a rule for which direction to move it. A prescriptive when X, raise Y is a hardcoded heuristic wearing a model's costume, and the stack already has a deterministic sweep that mines reopen counts one tier below. This loop exists for the case that sweep cannot express — reasoning about a pattern from evidence — so the prompt hands over accurate mechanism and gets out of the way.

4 · Reflection

Ground truth and event sources

4.1 Applications: virtual apps from workload labels

An application in Triage is a virtual app. The cluster push agent synthesizes it in memory straight from workload labels, grouping Deployments, StatefulSets, and DaemonSets by part-of → instance → name, and aggregates their health into a single Ready condition. A service reports live readiness the moment it carries standard labels, with no manifest ever deployed. Namespace onboarding is two kubectl annotate commands (§4.2); nothing else is installed on the application side. There is no app.k8s.io Application custom resource anywhere in this path: the core reads virtual apps only.

The agent's synthesis carries anti-flap health semantics:

• It is a read-only aggregator. The agent never modifies the workloads it observes. The Ready condition transition, with its lastTransitionTime, is the only downstream signal: one edge, one timestamp.
• Anti-flap health semantics. A workload counts as Ready when it is serving, before it reaches full desired count: a Deployment with Available=True qualifies even mid-scale-up, and readiness deliberately does not gate on observedGeneration, because generation skew during HPA activity was itself a flap source. A 30-second stabilization period damps NotReady-to-Ready transitions. Crash-looping Argo Rollouts are detected via the Available condition rather than the Progressing phase, which reports progress even while pods crash-loop.
• Two object sets. The Ready gate aggregates workload kinds only; the componentsReady "N/M" string counts all matched objects (Services, PVCs, and so on) for human context. Consumers must treat "2/3" as context, never as the health gate.
• Push mode. For clusters with no inbound API access, the agent streams outbound over an HMAC-authenticated WebSocket: virtual-app status, Kubernetes Warning events, and the USE RED signal families (§5). This is the producer side of Triage's cluster-agent receiver. The hello frame declares which collectors are armed, so the receiver can distinguish a collector that is off from one that is quiet.
• Unprivileged collectors, aggregates only. Every collector the agent ships runs without privileged containers, and only aggregates cross the wire: counts, rates, pressure percentages. Request payloads and log lines never leave the cluster.
• Counter honesty. Kernel and runtime counters are lifetime values; on first sighting the agent records a baseline and reports deltas only, so a restarted agent can never present a pod's history as a fresh burst.

As a result, the layer above never interprets pod-level detail. Each virtual app in each cluster has one authoritative, flap-damped answer to whether it is healthy, with a durable timestamp for its last transition.

4.2 The triageagent.dev/* annotation contract

The annotation contract is how owners steer observation without deploying a manifest. The triage agent's reconciler does not read these keys. They are read by the cluster push agent: at synthesis time for grouping and scope, carried with the pushed status to the core where they annotate the resulting graph node, and one key reaches down into the eBPF probe's discovery config (§5). This is the standard Kubernetes opt-in idiom, the same mechanism as istio-injection or Flux's reconcile annotations, and the security boundary is namespace-edit RBAC itself.

Any of the workload-level keys set on the Namespace object act as defaults inherited by every application in that namespace, applied live on annotation change; a workload's own annotation always wins over the namespace default. Between triageagent.dev/tenant for scope, namespace defaults for metadata, and the synthesis floor above, a team onboards a whole namespace with two kubectl annotate commands and no deployment at all.

4.3 Event sources

Triage ingests four classes of signal, all normalized into one InfraEvent shape with a source path (k8s/<cluster>/Kind/ns/name, app/<cluster>/ns/name, mcp/<service>, msteams/…):

Remote clusters connect through the cluster-agent WebSocket. Each cluster presents a per-cluster Bearer token, and the Secret's keys double as the allowlist: the token proves cluster identity, and cluster identity selects the tenant stamped onto every event. Tenancy is decided once, at the receiver, and carried through classification, incidents, and memory.

There is no dedicated security engine. Security findings are one case of a broader class — any security-finding source (a CNAPP/CSPM tool, a vulnerability scanner, a bug-bounty program) rides the same pipeline as every other provider event, and adding one is configuration, not code: the events server accepts POST /events/<provider>/ under Bearer auth for an arbitrary provider segment, so pointing a new source at it is a routing entry, not a new code path. Wiz is the live, verified example: a webhook at /events/wiz/ delivers a finding — an exposed host, a CVE, a compromised dependency — as an event, and a bug-bounty submission would arrive the same way, say at /events/bugbounty/. It passes through the same classifier, resolves to a service and a blast radius from the service graph, opens an incident, notifies the owning channel, and correlates with whatever else is open for that service — one more event source, not a separate product surface. Today, a human on the incident team acts on that finding; the agent closing that loop itself, through the same MCP and A2A tool surfaces, is a roadmap phase (§11).

The same posture rules out a receiver embedded in the core for either OTLP or Pub/Sub: Pub/Sub-sourced signals already ride that one webhook today — gcpevents, a small Cloud Run function outside the core, subscribes to Cloud Billing cost anomalies and Personalized Service Health events and posts them to /events/gcp-cost/<service> and /events/gcp-status/<service> — and the telemetry an OTLP receiver was meant to carry already reaches the core as the RED and USE families over the cluster-agent WebSocket (§5). A small ingress surface plus an outside translator per protocol, not a receiver per protocol inside the core, is the deliberate shape.

That same pipeline has a second, cheaper surface: detection from inside the cluster, through infrastructure the agent already runs. The embedded eBPF collector is on every node for RED and USE, and the Kubernetes event watch already forwards every Warning event in a watched namespace — a security detector on either side is a new signal type on an existing pipe, not new infrastructure. Kyverno is the concrete case: its audit-mode policy violations are ordinary Kubernetes Warning events, so they already reach the same k8s/ path a pod failure does wherever the namespace is in --k8s-event-namespaces, with no Kyverno-specific code — its separate PolicyReport resources are not parsed here. Cilium/Hubble already feeds RED metrics on the same collector (§5); turning its flow data into anomaly detection, and eBPF-derived runtime-security signals more broadly, are integration options, not shipped capability today.

6 · Awareness

Triage

6.1 The classifier pipeline

Every event passes through a fail-fast pipeline ordered from cheapest to most expensive stage:

Two caps in the pipeline encode trust decisions. Pod-failure types (crash, backoff, OOM) are capped at Signal: the Application layer's AppDegraded is the only authoritative incident trigger, so pod churn can never open incidents directly. Human sources are also capped at Signal: a Teams message or MCP report cannot fast-path an incident open; it has to convince the routing LLM with evidence. The generative model appears once in the main path, as the final arbiter for what the deterministic and semantic stages could not settle, behind a semantic routing cache that is validated against live incident state before reuse.

Log-based metrics (ADR-039) add a precursor signal without shipping log lines. The agent's own log collector (the log role of the embedded eBPF engine, §5) fingerprints each error at the edge, collapsing variable tokens into a template so a thousand near-identical errors become one series, gates those series on an error-rate threshold, and streams windowed per-service error/warn/total counts over the cluster-agent WS. The receiver emits one LogErrorAnomaly event per service on the same app/<cluster>/<ns>/<service> source shape the service graph already resolves. Like pod-failure types, it is capped at Signal: it absorbs into an open same-service incident and attaches as predictive evidence when one later opens, but never opens an incident on its own.

6.2 The service graph as level truth

Each tenant owns an isolated service graph: an in-memory HNSW index over the service catalog, continuously enriched by live informers with per-cluster application status and per-instance RED/USE snapshots. Blast radius (transitive downstream impact) is computed from graph topology and stamped on every event. Tenant isolation defaults to DENY: an event from tenant A never resolves tenant B's services, never absorbs into B's incidents, and RAG over incident history never crosses tenants. Graphs are delivered as prebuilt artifacts inside mental_map MemoryCells, so the graph itself travels through the same memory substrate as everything else.

6.3 Incident lifecycle

Incident state is edge-driven for latency: an AppRecovered edge resolves an incident within seconds. But edges can be missed. A restart loses in-memory hysteresis timers, a snapshot seed is deliberately silent for healthy apps, and a message can be dropped. Before the level reconciler existed, a missed edge stranded an incident indefinitely and produced a visible split brain: the graph shows the app 3/3 Ready while the incident stays open.

ADR-032 makes incident state converge to the graph. The service graph is the level source of truth, and a level reconciler sweeps every minute (and immediately at startup), auto-resolving any open app/-sourced incident whose instance has been continuously Ready beyond the hysteresis window. It uses the virtual app's Ready-condition transition time, which survives restarts because the cluster push agent maintains it. Reconciliation flows strictly one way, from graph to incidents. Incident state is never injected back into the graph: the graph is the classifiers' ground truth fed by live informers, and contaminating it with derived state would corrupt every classification downstream.

Identity and recurrence follow the same discipline. One identity matcher applies everywhere (exact source first, then same service within the same tenant, never across app instances), and a recurrence within six hours reopens the resolved incident, preserving history and reopen count, rather than minting a chain of near-duplicate incidents.

Application removal is treated as lifecycle, not failure. In ephemeral environments (preview namespaces), apps are deleted routinely; an AppRemoved event can never open an incident. If an incident is open for the removed app, it is auto-resolved and a low-severity watch entry is recorded instead, so the disappearance stays visible without masquerading as an outage.

7 · Cognition

X-Ray

7.1 One substrate for observations and conclusions

X-Ray uses a single memory format: the MemoryCell CRD (xray.ai/v1). Triage's inputs and outputs are the same kind of object:

• Source artifacts. Every processed event (kind=event) and every incident (kind=active_concern) persist as cells.
• Outcome artifacts. Reflection and knowledge passes emit stable_fact, operational_pattern, note, and system_identity cells, including the agent's own prompts, its world-status note, and each tenant's service graph (mental_map).

Because ingest and insight share a substrate, the whole cognitive state is uniform: durable in etcd, inspectable with kubectl get memcell, segmentable by subject (the multi-tenant key), scoped public/private, and rendered by X-Ray's memory-graph UI without any triage-specific integration.

The runtime representation is an in-process HNSW graph that serves as both the similarity engine and the memory topology (X-Ray ADR-000 describes it as a living semantic graph). CRs are the durable truth; HNSW is the fast read surface, reseeded from CRs on restart. Embeddings come from a llama.cpp sidecar serving nomic-embed-text truncated to 256 dimensions: local, cheap, and provider-independent.

Cells carry typed ThoughtLink edges (caused_by, triggers, correlates_with, precedes, explains, contradicts). Triage's knowledge pass emits link operations, and its get_memory_context traverses the resulting associative graph from an anchor cell. In the X-Ray product runtime, link traversal at retrieval time is schema-ready but still on the roadmap.

7.2 The reflection and knowledge loops

A single goroutine drives two consolidation passes:

Reflection is deliberately timer-only. Incident changes used to trigger it, and a flapping incident could re-trigger an expensive consolidation pass on every degrade/recover cycle. That wiring is gone: resolving an incident updates the live status summary deterministically, without a model call, while reflection consolidates on its own schedule. Both passes run per tenant, so consolidation never mixes tenants' memories. Under the system2-delegate policy preset both timers are zero and an external System-2 session (section 2.1) invokes the passes on demand instead.

Reflection works on current state: live incidents and fresh causal context. Knowledge extracts what recurs across resolved incidents (stable facts, operational patterns) into a knowledge digest. This is where proactive detection grows from: a pattern like "explore-persona flaps after catalog deploys" becomes an operational-pattern cell that informs future classification and conversation.

7.3 Write topology

Two reconcilers connect memory and etcd, and they are strictly one-way. The mutation watch persists live-state changes to cells (memory to etcd, the only writer in that direction). The MemoryCell reconciler seeds the index and mirrors external edits (etcd to memory). The governing invariant: every edge between them must be a no-op when state is already equal. A mirror that stamps timestamps, or a note refresh that rewrites identical content, creates a self-sustaining write loop; ADR-033 documents one that shipped and how it was closed. This invariant is what makes memory-as-CRDs workable in practice.

The same discipline extends across controllers: configuration cells are never embedded or indexed, and cells owned by another instance are never mirrored, so two controllers sharing a namespace cannot feed each other's write loops.

7.4 Operational state as memory

The substrate also carries the agent's own operational state: runtime statistics, authenticated user sessions (stored hashed, with encrypted identity), and per-user voice usage persist in a system-configuration cell. A pod restart neither logs users out nor resets quotas, and the state remains inspectable with the same tooling as everything else.

7.5 Composition

Triage and X-Ray do not share databases and do not reach across namespaces. They compose through two narrow channels: the MemoryCell CRD (Triage writes cells; the triage-xray release reads, indexes, and visualizes them) and MCP (X-Ray's voice sessions register Triage's MCP server, so voice sessions call live Triage tools alongside memory tools). Both X-Ray releases run the same binary and chart; separation comes from namespaces and the subject key.

8 · Intelligence

The intelligence layer

The intelligence layer is the conversational surface of the stack, and it draws on everything below it.

• Root-cause analysis runs over the fractal incident graph: a five-layer context assembly per incident. (1) blast-radius summary from the service graph, (2) live application status, (3) tagged causal events, (4) 30-day event history for the service, (5) similar resolved incidents retrieved from the vector store. Each layer comes from a different subsystem; the fractal graph is where reflection, awareness, and memory converge into one reasoning context.
• Skills: a delegated sub-agent with a Kubernetes tool set (get, describe, logs, events, CRUD across clusters, routed by the cluster segment of the incident source), reporting progress through a task-poll pattern so long-running investigations do not block the conversation.
• Interfaces: a mobile PWA, web push on new incidents, MS Teams threads, MCP tools for other agents, and optional A2A for agent-to-agent delegation.
• Voice quota: a per-user daily voice token budget is tracked and enforced (ADR-038); quotas survive restarts.

The layering discipline holds here too. The intelligence layer keeps no private state: its memory is X-Ray cells, its incident context is the fractal graph, its ground truth is the service graph.

9 · Walkthrough

The life of an event

A concrete end-to-end trace: a bad deploy of explore-persona in the dev namespace of a remote cluster.

1. Reflection

   New pods crash-loop. The Rollout reports Available=False; the cluster agent flips the app's Ready condition to False (an edge, with lastTransitionTime recorded) and updates componentsReady to 2/3. No Kubernetes event is emitted; the condition change is the signal.

2. Transport

   The cluster-agent streams the virtual app's status over its WebSocket. Its per-cluster token authenticates the cluster, which resolves the tenant stamped on the resulting AppDegraded event (app/tenant-a-aks/dev/explore-persona).

3. Awareness

   Stage 0 resolves the service in the tenant's graph and stamps blast radius. AppDegraded is the authoritative trigger. The classifier fast-path finds no active incident for the source, finds one resolved 40 minutes ago, and reopens it (reopen count incremented, history preserved) instead of minting a duplicate. The pod-level crash and backoff events arriving in parallel are capped at Signal and absorbed as contributing evidence, not new incidents.

4. Cognition

   The incident and its events persist as MemoryCells. The reflection pass builds the fractal graph, matches the pattern against last week's flap, and records it; a later knowledge pass distills an operational_pattern cell.

5. Intelligence

   Web push fires. The operator opens a conversation; RCA over the fractal graph correlates the deploy with the crash evidence and the similar past incident, and a skill sub-agent pulls the failing container's logs from the right cluster.

6. Convergence

   The deploy is rolled back. Pods serve; after the 30-second stabilization the Ready condition flips True; the recovery edge resolves the incident after the 5-minute hysteresis. If that edge had been lost (restart, drop, missed timer), the level reconciler would have resolved the incident within a minute of the hysteresis window expiring, from the CR's durable transition time. Both paths converge to the same resolved state.

10 · Tenancy & onboarding

Tenancy, security, onboarding

Multi-cluster and multi-tenant. One controller serves N tenants with isolated graphs, classifiers, and incident spaces; one Helm chart also deploys as fully separate per-project releases when isolation must be physical. Tenancy is stamped once at the receiver and defaults to DENY across every boundary.

Security posture. Human input is guardrailed (INFRA / NOISE / INJECTION) and fails closed; prompt-injection defense is a dedicated design; human sources cannot fast-path incidents; per-cluster tokens bind cluster identity and hot-reload on rotation; API auth composes static tokens and SSO sessions without breaking browser or M2M clients, with per-tenant access grants enforced on every surface; voice usage is quota-capped per authenticated user per day.

Self-observability. The agent monitors itself the way it monitors infrastructure: OTel traces to Phoenix, per-call LLM token and latency stats, and a pytest eval suite (batch evals scoring live traces, plus end-to-end RCA scenarios judged by an LLM), so prompt and pipeline changes are evaluated against a baseline.

Implementation status. ThoughtLink retrieval-time traversal ships in Triage's memory context but not yet in the X-Ray product runtime. X-Ray's own reflection loop and the three-tier write-approval pipeline (guardrails, model-as-judge, human-in-the-loop) are designed but not fully implemented. The cluster agent's push protocol is a hand-synced mirror of the receiver's and can drift.

10.1 Onboarding a tenant

Bringing a new tenant onto the stack takes three steps. The work is front-loaded into the service map; the agent itself needs no code change.

1. 1 · Prepare the service map

   List the tenant's services and how they depend on each other, including the alternate names each service goes by in alerts and deployments. Topology is the load-bearing part: blast radius, event correlation, and noise gating are all computed from these dependency edges.

2. 2 · Enrich with domain data

   Add what only the owning team knows: business domain, criticality tier, ownership, what breaks when the service does, and how its failures actually read in alerts. This is the text the classifier matches incoming events against, so enrichment quality directly sets classification quality. The enriched map is compiled once into the tenant's service-graph artifact.

   services:
     - name: checkout-api            # 1 · service map
       aliases: [helmrelease/checkout]
       calls: [payments, cart]
       called_by: [storefront]
       domain: Commerce              # 2 · domain data
       tier: 1
       team: platform
       description: Order checkout API. Outage blocks all purchases.
       symptoms: 502 bad gateway, checkout timeout, payment step failing

3. 3 · Install the cluster agent

   Deploy the cluster agent into each of the tenant's clusters in push mode. It aggregates workload health into one Ready level per application and streams status and warning events outbound over an authenticated WebSocket, so the tenant's cluster needs no inbound access. On the triage side, register the tenant with its service graph and clusters and issue one credential per cluster. From the first event, tenancy is stamped at the receiver and isolation is default-DENY: the tenant gets its own service graph, classifier, incident space, and memory visibility.

11 · Roadmap

Planned work

• Drift → graph update: auto-PREPARE tooling that turns a drift report into a graph update without a hand-written catalog edit, and drift surfaced on the incident views themselves. Detection already runs end to end — collector, hub, observed edges, CLI badges and catalog export (ADR-047, ADR-050).
• Replica scale-out: shared budgets: exact peer-sum gating, so the LLM daily token budget and the guardrail reserve are enforced across replicas instead of per replica — and moving the chronic sweep, the correlation pass and the embedding self-write sweep off per-replica execution. Leader-gated singletons and the merged replica_counters read surface already ship.
• Autonomous remediation: once a root cause is trusted for an incident class, the agent acts on it directly — through the same MCP and A2A tool surfaces it already exposes for skills and conversation — instead of leaving every finding, security findings included, for the incident team to act on by hand.
• The AI teammate: the feedback loop: a System-2 delegate can already hold the standing role for a tenant's status cells, with heartbeat-gated failover back to reflection when it goes dark. Still unbuilt is the reward channel itself: feedback cells synthesized from the system's own contradictions (reopens after auto-resolve, reconciler catches, cascade-gate denials, human corrections), zero-token consumers for it (classifier exemplars, verified link priors), consult_request emission on novelty, blast-radius, and reopen-storm triggers, and a graduated autonomy ledger where action classes earn autonomy from proposal-approval history.

12 · References

Design records

Triage: ADR-000 (Alive Agent pattern), 001 (architecture), 002 (classifier pipeline), 003 (service graph HNSW), 004 (recovery hysteresis), 007 (prompt-injection defense), 009/010 (fractal incident graph / fractal RAG), 012 (skill system), 014 (knowledge graph), 017 (multi-cluster sources), 019 (MCP server), 028 (MemoryCell backend adoption), 029 (cluster-agent WS), 030/031 (multi-tenant graphs / prebuilt graph artifact), 032 (incident state reconciliation), 033 (memory-cell write topology), 034 (context headroom / token compression), 035 (debounced stats persistence), 036 (voice session management), 037 (GitHub session persistence), 038 (per-user voice token quotas), 039 (log-based metrics source), 040 (prediction-aware reflection), 041 (policy cell), 042 (replica scale-out), 043 (cell TTL and forgetting), 044 (mindset cell), 045/046 (RED / USE metrics sources), 047/050 (graph drift / observed edges), 048 (service weight), 051 (token capability bits), 052 (private memory scope), 053 (redacted memory stream), 054 (SAI vocabulary), 055 (case interface).

X-Ray: ADR-000 (architecture; HNSW as a living graph), 020 (associative memory: kind taxonomy, consolidation, approval pipeline), 021 (reflection/knowledge loop), 037 (scope as privacy axis), 043/045 (code-graph cognition), 044 (session recap continuity), 051 (MCP tools as general agentic memory; subject as tenant key).

Triage agent: ADR-0001 (stabilization-period fix), 0003 (anti-flap readiness predicates; no-events contract), 0004 (dynamic component watches), 0005 (push mode).

Literature: Daniel Kahneman, Thinking, Fast and Slow (Farrar, Straus and Giroux, 2011), the dual-process model behind the System 1 / System 2 reasoning tiers (section 2.1). Tim Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas L. Griffiths, "Cognitive Architectures for Language Agents" (arXiv:2309.02427, 2023), the memory-module/action-space/decision-loop framework behind section 2.2.