Triage Core

The core is one Go binary run as a Kubernetes controller. It reconciles MemoryCell objects, serves the HTTP API with the MCP server on port 8080, runs a second MCP listener on 8083, an events server on 8086 for alert webhooks and the agent WebSocket, an MS Teams connector on 8085, health probes on 8082 and, with several replicas, a gossip mesh on 8088. Nothing is compiled in for the UI; a ui cell names an OCI artifact that the core pulls and serves.

All persistent state is a MemoryCell of the xray.ai/v1 kind in the core's namespace: events, issues, knowledge, patterns, but also the configuration (policy, runtime document, prompts, notification routing, the tenant registry and each tenant's service graph). The chart ships that one CRD. The state that is only useful while the process runs, such as the event hub, the service ledger, the situation frame and the rate limiters, lives in memory, belongs to one replica and starts empty after a restart.

Two kinds of decision run side by side. The event path is deterministic code driven by policy, the service graph and recorded patterns, with one decision model for typed questions; it runs on every event. The reasoning loops are model passes that run a few times a day per tenant and write what the event path reads next. No model opens an issue.

Runtime

Edges of the system

• The core opens connections outward only to model providers, the OAuth and OIDC providers, Firebase, notification webhooks, OCI registries and the remote MCP or A2A agents the runtime document lists. Agents connect to the core; the core never connects into a cluster.
• Optional pieces fail soft. With no decision model the engine keeps its own severity; with no reasoning model the loops do not run; with no homeserver the messenger routes answer 503.

A tenant is the unit of isolation: its own service graph, events, issues, memory, goals, notification channels, budgets and agents. People see a tenant as a workspace. Every event, cell, frame and API row carries a tenant, and every read and write is checked against what the caller may do on that tenant.

Tenants come from one of two places. With tenants.registry on, the registry cell is the source and the API creates and edits them. With it off, a tenant exists when a service-graph cell for it is applied. In both cases there is no default tenant and no static tenant flag; write ingress fails closed until a tenant exists.

Registry record

Members and roles

A member holds one of three workspace roles, stored as a Role per role and a RoleBinding per member in the core namespace, labelled with the tenant id and named triage-tenant-<tenant>-…. The subject is <provider>:<id>, at most 200 characters without a space or slash.

Trial workspaces

A person who is not an instance admin can create a workspace when they hold a GitHub session whose organisations include the requested account. They become its owner. The limits come from flags and never from the request: --tenant-trial-days (14), --tenant-trial-max-per-org (1) and --tenant-trial-licences (3).

Isolation in memory

A cell is visible to every tenant when it carries the shared tag and to one tenant when it carries tenant/<name>. An untagged cell of a tenant-scoped kind resolves to no tenant on a multi-tenant instance and is not served. The rule is applied on the model-facing paths, on the REST surface and on MCP, and a caller who is not an admin cannot write a tenant/, shared or author/ tag. Private cells carry scope: private and the hash of one user; only that user sees them, and MCP treats them as absent.

Every event enters through one router that stamps the tenant, applies the admission limits and collapses repeats. The source string decides how much the pipeline trusts the event: sources that begin with k8s/, app/, flux/ or events/ are machine sources; everything else, including mcp/, msteams/ and a2a/, is a human source.

A webhook names its tenant with X-Tenant, the tenant query parameter or a body field. The value auto resolves the tenant from the tenants' service graphs and fails closed when no graph or more than one graph matches. --events-ingest-authz is enforce by default: a scoped token without the i bit on the resolved tenant gets 403. In log mode the request proceeds and the decision is counted, which is the way to migrate tokens.

Admission

Screening

A regex pre-screen for prompt injection runs on every event. A human-source match is dropped. A machine-source match increments a counter and is logged, and the event is never dropped on the regex. Human events then go to the guardrail model, on by default (models.guardrailEnabled), which labels each INFRA, NOISE or INJECTION; an error or an empty answer drops the event (fail closed), and a semantic cache (similarity 0.88, window 24 h) answers repeats. With the decision model on, its human-guardrail label is applied as well, and NOISE and INJECTION end the event. Machine events never reach a guardrail model.

Event text that reaches a prompt is wrapped as untrusted data between per-call random fences with an instruction not to follow it.

An issue (a concern in the code) is the unit a person acts on: a service, a severity, a summary, the events behind it, a cause and the patterns that bear on it. Its status is new, investigating or resolved. Figure 2 shows the path from an event to an issue.

Allocation and the ledger

The service graph decides which service and instance an event belongs to. It never decides whether the event matters: noise means unallocated, an event no graph node takes, and it opens nothing. An allocated event is never dropped for its tier, its dependents or its blast radius. Every error-bearing event is credited to its service instance in the service ledger by signal family: level, request, log, resource, pod and provider. A row keeps 60 one-minute and 288 five-minute slots, rows are capped at 2,000 per tenant, and the least recently credited row goes first. The ledger answers the question "why did Triage say nothing" and feeds the situation frame, the tenant ledger endpoint and the MCP tools inspect_service and get_tenant_situation.

Openers

An issue opens on evidence that something happened to a service, never on blast radius. These are the only openers; each is code, and a model takes no part in the decision.

Pod-family events (BackOff, Unhealthy, Failed, FailedScheduling and the rest of the pod-failure role) are one instance fact and never open an issue by themselves. A crash loop opens through O1, a second family through O2, and a probe that keeps failing on a Ready workload through O3.

Severity

The band comes from blast radius, the share of the catalog that depends on the service. At or above the tenant's INCIDENT threshold an issue is critical; between the NOISE and INCIDENT thresholds it is warning; below the NOISE threshold, for a leaf, or for a noise-tier node it is info. Evidence raises an info issue to warning and never to critical. A provider-critical event keeps critical. Each channel's minimum severity decides who is paged.

With the decision model on, one score question on the ordered scale info, warning, critical is asked per new issue, from evidence only (the change, what was already said in the last 24 hours, the service's place in the graph, its seven-day issues, its situation line and its ledger families in the last 10 minutes). The issue takes the engine's severity moved at most one step toward the answer. A failing or absent model leaves the engine's severity.

Holds

Author pass

After every machine open, one model pass rewrites the engine's templated summary into words an on-call engineer reads in five seconds and, with link_concerns, draws caused-by and duplicate-of edges to the tenant's other active issues. It may not create an issue, change its severity or source, or call any other tool. An issue is authored once; the pass runs again only when its severity has changed since the words were written. The write is its own mutation kind and does not move the issue's LastSeen, since words are not evidence.

Resolution

A resolve is evidence. The resolved record, its verdict and its root cause are what the judges and the pattern track read next. Patterns link to issues in both directions: matches is written by code at open and reopen from the tenant's active patterns on the service, and explains is written when a person or the reflection pass names the pattern as the cause.

Memory is one substrate. Every cell is an xray.ai/v1 MemoryCell with a kind, a subject (the producing instance), tags, an origin, links to other cells and an optional embedding. The core watches the cells in its namespace whose subject matches --memory-subject. Embeddings come from text-embedding-005 on Vertex AI, or from a gateway or sidecar (nomic-embed-text, 256 dimensions); an HNSW index serves similarity search.

Kinds

Provenance

Every cell has an origin: observed (code wrote it from facts), derived (a model wrote it), human (an identified user wrote it through MCP, chat or voice) or verified (derived, then passed a check in section 6). A cell without an origin is derived. Prompts receive the origin on every cell with one rule: human outranks derived, and a derived cell is a prior claim, never evidence. Code enforces the rule where a model chooses what to rely on: an id accepted as a pattern or a basis must be observed or human.

Retention

Windows are tenant-overridable in policy. Knowledge writes go through an upsert that merges a near-duplicate in place, never against a shared or a foreign-author cell. Event cells collapse per (tenant, source, type) fingerprint.

How cells are applied

Configuration cells carry their document as base64 bytes under spec.data. The reconciler decodes the document with strict field checking and then a validator. A rejected document keeps the previous good state, so a typo never replaces a working configuration. A cell write is a Get then Create or Update with optimistic concurrency on the resource version, with labels set wholesale and the embedding preserved. A service graph is either inline (gzip of a gob, decoded under a 64 MiB bound with a panic-to-error guard) or a digest-pinned OCI reference; a tag-only reference is refused and a failed pull keeps the loaded graph.

The loops are the slow half of the core. A deterministic situation frame describes each tenant; reflection and knowledge passes read the frame and memory and write what the event path reads next; RSI tunes the knobs and the patterns from the outcomes people confirm. Figure 3 shows the loop.

Situation frame

The frame is built without a model from the graph, the RED and USE snapshots, the signals ring, the event cells, the classifier's matches and the ledger. It has one entry per notable service, each with its facts, how long they have held and a one-word reason, capped at situation.maxServices (12). A service is notable when it is not Ready; when memory has stood above the floor (85 percent of its limit) for the hold (30 minutes); when OOM kills are increasing; when log, request or resource anomalies appear in more than streamWindows (5) windows of the last hour; when the error rate is above its floor (5 percent); when an issue is older than the stale age (12 hours) with no level evidence; when more than one issue is open on a service; or when a catalog node's workload names never matched. It also lists sharedMatches, the graph nodes that the anomaly text of two or more other services matched. The frame is served at GET /v2/tenants/{tenant}/situation and by get_tenant_situation, and it wakes reflection when it changes, at most every wakeEvery (10 minutes) and wakeCooldown (30 minutes).

The core describes its own dependencies the way it describes a tenant's. A ledger of self-signals records, for the decision model, the reasoning model and the embedder, the last error with its status, the failures in the last hour and the last success. It appears in every frame, in the RCA brief when the issue's service is the core, and in get_triage_agent_status.

Reflection and knowledge

Each loop has a cooldown, one hour by default, that separates two passes on one tenant. Model output is parsed as strict JSON into operations; kinds outside the allowed set are dropped, a mind-map operation must name a service in the pass's hot set, every tenant/ and shared tag a model writes is replaced by the sub-pass's tenant, and malformed output yields a summary and no operations.

Verified reasoning

A derived claim is checked before it is trusted. With loops.verify on, a fact's cause is tested against live evidence, with at most 3 checks per cause and 500 a day and a minimum support of 0.6. A fact whose cause cannot be tested is demoted to a hypothesis, which expires; a fact that passes becomes verified. Edges in the observed graph carry a role, request, telemetry or control, taken from the destination port, the protocol and a policy list of telemetry-sink names. A telemetry edge stays in the graph for provenance but is not a dependency: blast radius, the dependency layer, the classifier's "Depends on" line and the RCA call path never see it.

Predictions

A precursor matcher raises candidate services from the sequences that preceded past issues (predictions.window 14 days, precisionWeighting on). Each candidate is ranked by the decision model (how likely an issue is within the hour, from its evidence), and the answer becomes the prediction's confidence. An elevated prediction needs a confidence of 0.5; a notification needs 0.8 and re-arms after the confidence falls below 0.5.

Patterns and RSI

A pattern is an operational_pattern cell: a state of one service that people recorded as expected, such as a nightly scale-down, with an optional window. It holds machine opens (section 4), is linked to the issues it explains, and is the base of knowledge: the knowledge pass writes the fact behind it. RSI (recursive self-improvement) reads the outcomes the core can measure and proposes changes. It is off by default.

The gates are a consistency streak (applyAfter, 2 passes), the applyTenants list, one applied change per tenant per day, an evaluator that reverts a change that did not improve its objective by minImprovement (0.2) over the evaluation window (6 hours), and the operator's own settings, which win field by field over the RSI overlay. The loop runs every 30 minutes, and a burst of alerts (wakeAlertsPerHour, 6) wakes it with a 30-minute cooldown. The knobs are registered with their bounds: the flap gate, the alert repeat interval and the like.

The pattern track has three parts. New: from one issue a person resolved with a verdict or a root cause, or a flap the machine closed after at least minReopens reopens with nobody's verdict, it drafts a pattern. Revise: when an issue opened on a service after its pattern was recorded and a person resolved it as expected or noise, the reasoning model reads the pattern and the misses and answers with the text or window that would have held them, or with nothing. Retire: a derived pattern that explains nothing is pruned. People see all of it: list_patterns and the memory explorer show the author tag rsi, the status block counts decided and pending proposals, and refute_memory_item, expire_memory_item and approve_pattern remain for a person who disagrees.

The core consumes models and hosts none. Every role is configured in the runtime document, and a role's credential is a Secret reference resolved into memory; there is no API-key flag. Vertex AI uses Workload Identity when googleCloudProject is set on the role's block.

The decision model

The decision model answers typed questions about a JSON state: a probability, a score on an ordered scale or a choice among options. It is the only judge on the event path. A call has a 15-second timeout and a 1 MiB response cap. It is consulted at these points, each with its own switch in decisions (all on by default, tenant-overridable, and all inert when no model is wired):

The URL must be https, or http to a loopback host or a .svc name, checked when the document is validated and on every send. A token file is read on every call and must resolve under tokenFileRoots (default one directory); the service-account directory is refused. A 429 with Retry-After up to 10 seconds is waited out once; a longer one makes the client refuse calls until the time is up, and no pause exceeds 60 seconds.

Budgets

Every figure is a sum over today's rows of one usage ring, per tenant and per call kind, so the gate and the status read the same numbers. Failed calls count, since the ring records what the provider billed. One resolver reports spent, cap and the limit that blocked: instance, kind or tenant. The day is UTC.

An agent opens a WebSocket to /events/ws and proves who it is in the first frame. The HTTP request is upgraded without a credential and authenticated in-band; the first frame has a 90-second deadline and an 8 MiB read limit, and a same-origin check admits requests that carry no Origin. Authentication is the join certificate and nothing else. A hello without a certificate is refused.

Certificate join

• Joining is enabled when the agent CA is configured (clusterAgents.join.caSecret). The CA is Ed25519; a previous CA can be set for rotation and cross-signs the current one.
• The core sends a challenge nonce first. The agent's join, renew or hello frame signs it, and the agent verifies the core's CA signature over the same nonce.
• A CSR carries the cluster name and the tenant list. It is issued only when an approved agent with the same fingerprint and tenant list exists. Certificates last 24 hours (--agent-cert-validity), a certificate issued before the latest approval is refused, and revoking an agent closes its live sessions.
• A request that is not yet approved waits in a pending list (100 entries, 5 per source, 5 per cluster, 24-hour TTL). An admin approves, denies or revokes through /v2/agents and can set an agent's own daily event cap.
• Join attempts are throttled per presented identity (6 a minute) and per source (60 a minute). The source is the peer address, or the Nth X-Forwarded-For entry from the right when trustedProxyHops is N.

One-time keys

A workspace manager mints a key (tak_<id>_<secret>, 256 bits of secret, only its SHA-256 stored) that lasts seven days and is spent by the first successful join. Kinds: cluster (several tenants), host (one tenant, a Linux host) and otlp (one tenant, no Kubernetes watch). The join frame carries the key, a CSR and an Ed25519 proof over the nonce; the CSR's cluster name and tenant list must equal the key's, the certificate is issued at once, and the core assigns the cluster name (the label plus an eight-character suffix). The API returns the install command: a Helm line with core.host, core.caFingerprint, core.clusterName, core.tenants and core.agentKey, or for a host the install.sh line.

Bindings

A key-joined agent holds one binding per tenant, each with an optional namespace list (* or empty means all). The receiver filters snapshot, delta, event, log, RED, USE and topology frames to the binding's namespaces and drops frames for tenants outside the bindings. Heartbeat and sensor-state frames are not tenant-filtered, but each OTLP source entry in a sensor-state frame carries a tenant stamp that is checked against the bindings. Bindings change through POST /v2/agents/{cluster}/bindings: replace the namespaces of one, add a tenant the caller manages, or file a request that the other tenant's managers answer (7-day TTL, 1,000 requests globally). An agent with no binding left is revoked, and deleting a tenant ends its bindings and unused keys at once.

Frames

Every frame carries a tenant. The hello's tenant, an id or a label unique among the certificate's tenants, and every frame's stamp are validated against the certificate's tenant set. A second connection for the same cluster is refused while the first is live.

The HTTP API, the WebSockets and the MCP server accept two credentials: an SSO session, and a Kubernetes ServiceAccount token verified by TokenReview with audience triage-core. Static bearer tokens are not accepted there; the keys of the api-tokens Secret authenticate the events server only. With no SSO provider and no ServiceAccount, every gated request gets 401. sso.authDisabled (--auth-disabled) is the only way to run open, for a lab.

Credentials

Grants

Authorization is a map from tenant to a set of six capability bits. Bits never imply one another, and the wildcard tenant * is the instance.

Grants come from Kubernetes RBAC: Roles and RoleBindings in the core namespace, with rules on the resource tenants of the group triageagent.dev and the tenant names (or *) in resourceNames. Verbs map to bits: get and list to r, create and update to w, execute to x, ingest to i, voice to v, observe to o, and * to all. Subjects are User github:<login>, Group group:<name> and a ServiceAccount in the core namespace. The chart ships the six named Roles on every tenant. Grants are resolved per request for a session, so a change applies at once. Older grant sources (admin-user lists and team-grant maps in the api-tokens Secret) still exist; configuring any of them arms enforcement, and an unarmed instance gives every session the admin grant while RBAC-resolved callers stay scoped.

Enforcement

• REST and WebSockets resolve grants once per request (captured at upgrade for sockets). A ?tenant= route answers 403 for a name the caller cannot read, whether or not it exists; an unknown name is listed back only to a caller who could read some tenant of that name.
• MCP filters every list, get and search by what the caller may read, hides write-scoped tools from a read-only caller and rejects them if called.
• Voice and chat need v on the bound tenant. The pprof handlers need the admin grant. The tenant name list needs o.
• Workspace management (keys, agents, members, binding requests, notification channels) needs write and execute on the tenant; account, licences, trial dates and restore need admin.

MCP OAuth server

When --public-url is set and an SSO provider exists, the core is its own OAuth 2.1 authorization server for MCP clients, co-located on port 8080: protected-resource and authorization-server metadata, dynamic client registration restricted to loopback and custom-scheme redirects, PKCE S256 required, a resource indicator that must match, single-use authorization codes that last two minutes, and an EdDSA token whose signing key derives from the configured client secrets so replicas agree. Tokens carry a scope, triageagent:mcp.read or triageagent:mcp.write, and either a grants string or an RBAC subject whose grants are read from the cache on every request. A shorter lifetime for write tokens can be set. Authorization codes live in one pod, so several replicas need session affinity for this flow.

Every service is a ClusterIP by default. A Gateway API route can expose one hostname that carries the MCP path, the events paths and the API; an nginx Ingress and a GCP backend policy are optional. The messenger homeserver has no route at all.

HTTP API

The API is versioned under /v2 and described by an OpenAPI 3.1 document the core serves live at /v2/openapi.json and /v2/openapi.yaml, with Swagger UI at /docs. The groups:

The status WebSocket also sends {"changed":[…]} frames that name what changed, from the set apps, graph, analytics, events, memory, tenants, goals and ledger. A frame carries names only, at most one per topic per gap of 1 to 30 seconds, and a client refetches what it shows.

MCP

The MCP server defines 55 tools; six of them exist only when the messenger is wired. Tools split by who may call them:

List-shaped results are compressed to 500 characters a row (--headroom-limit, 0 disables). analyze_issue uses the client's own model through MCP sampling and costs the core no tokens.

Voice and chat

Voice is a Gemini Live WebSocket; chat is a text WebSocket on the same tool set. Both answer 503 while disabled or without a credential, both need the v bit on the bound tenant before the upgrade, and both check the origin. A write-class tool call asks the client to confirm, except save_session_note and ask_spot. Quotas are in section 7.

UI bundle

A ui cell holds a digest-pinned OCI reference (uiRef) and optionally the name of a pull Secret. The core pulls the artifact, takes every regular file of every layer (64 MiB and 5,000 files at most, names normalised) and serves it on port 8080 for every path that is not an API or infrastructure path. A failed pull keeps the bundle being served. The home document carries an enforcing Content-Security-Policy naming its inline script hash, and a kill-switch service worker is served at /ui/sw.js.

Each workspace has a chat built on a Matrix homeserver that only the core can reach. The core is the only Matrix client: people use /v2/messenger, and the homeserver has no public route. The routes answer 503 while no homeserver is configured.

Rooms

Every room route checks the read grant on the room's workspace per call. The write grant gates deleting other people's messages and pinning. A message the core posts for a person into a workspace room carries their level on the workspace in the event key app.triage.grant (read, write or admin); a forward carries the forwarder's level. Removing a member kicks their Matrix user from the workspace room and from their private chat, and clears their push registrations if they are in no other workspace room. Text is 1 to 4,000 characters and an upload up to 2 MiB.

An @tria mention in a room is answered as the mentioner, with their read and voice grants. The 40 earlier messages become turns, confirm-class tools and ask_spot are not offered, and the answer carries the asker's level. It gets 16 tool calls, 8,192 output tokens a call, a 4-minute window and a 30-second posting deadline, and counts against the per-user chat quota and the global chat budget. With messenger.multimodal on in the runtime document, the images and files of a room are shown to the model when it answers.

Spots

A spot is a per-workspace agent that lives outside the core, acts through MCP tools and speaks in the workspace's rooms under its spotName. It holds no Matrix credential. The core gives it a trigger stream, a budget and a status line.

One component, the interrupt shield, decides everything a channel hears. The engine only offers a candidate: a new issue, a resolved issue, a prediction crossing or an observation. The shield sets a new issue's severity with the decision model (section 4), queues the change for its tenant, and lets reflection write the message in its next pass. A new issue wakes a scoped reflection pass at most once per tenant every 10 minutes; changes that arrive meanwhile ride in the same pass, and recoveries, risk crossings and observations never wake one.

• A new issue that is critical once its severity is set is sent at once, verbatim.
• A repeat gate suppresses the same (kind, tenant, service) inside alerts.repeatInterval (1 hour, bounds 0 to 24 hours). Critical is never suppressed. RSI tunes this knob.
• With the decision model's gate switch on, a non-critical issue that outlives the blip window reaches reflection only above 0.5.

Channels

A URL must be https, name a host (no IP literal, no localhost), carry no user information and resolve to no blocked address range where DNS answers; the check runs at load and, for workspace channels, at write. Delivery uses one shared client with a 10-second timeout and no redirects, one drain worker and a 256-deep queue, and only the leader announces. URLs are never logged or shown; status surfaces list keys only. The test endpoint sends one fixed message, one a minute per channel, and returns the provider's error text up to 512 bytes. --issue-url-template substitutes {id} and {tenant} into the link a message carries.

Push

Web Push uses VAPID keys read from a Secret at start (a per-pod pair is generated and gossiped when none is set). A subscription's endpoint is validated at subscribe, at gossip apply and at restore with the same rules as a webhook URL plus a dial-time check. Android registrations are Firebase tokens stored as fcm:<token>, sent to Firebase with a service account or Workload Identity. The server overwrites any grants a client supplies with the caller's own, and delivery is filtered by them.

Observations

When memory notices something before anything is down, such as a service that has held 95 percent of its memory limit for six hours, the reflection write emits an observation. It goes through the same router and the same channel floors, leader-gated, with a minimum interval between them.

The Deployment can run several replicas. With controller.leaderElection on, one replica holds a Lease and does the work that must happen once; with several replicas (or gossip.enabled) the chart renders the gossip mesh, and every replica serves the API, MCP and the intake.

The core has few flags and no configuration file. Behaviour is set in cells that are applied while the core runs, and the Helm chart renders them from values.

Policy

Mechanism stays in code and policy is parameters. There is no rules engine and no expression language. The schema cannot express turning off the safety invariants: the guardrail fails closed, cross-tenant access is denied by default, issue state follows the graph one way, cell writes follow the write topology, and every model call has an output cap. A preset typo cannot become a security incident. Sections:

apiVersion: triage.policy/v1
name: example
version: 1
classification:
  flapSuppression:
    minReopens: 5
  quiet:
    serviceTokens: [synthetic, e2e, canary, smoke]
budgets:
  llmDailyTokens: 1000000
  llmTenantDailyDefaultTokens: 100000
rsi:
  mode: shadow
tenants:
  tenant-a:
    classification:
      quiet:
        namespaces: [sandbox]

Five presets ship with the repository: balanced (the compiled defaults as a document), cost-saver, high-recall, quiet and system2-delegate. Offline validators for the policy document, the runtime document, notification routing and a service catalog run the same checks as the reconciler. A document that fails validation, or fails to decode with strict field names, is rejected and the previous good one stays in force.

Chart

The chart triage-core groups its values as: the controller (replicas, leader election, probe and metrics addresses), the HTTP API and MCP listeners, the events server, the MS Teams connector, auth and access (grant sources), sso, runtimeConfig, prompts, embedding and chatModel sidecars, messenger, voice, tenants (the registry, trials, members), clusterAgents.join (the agent CA), gossip, the exposure objects (httpRoute, externalIngress, health and backend policies) and pod settings. With nothing set the chart generates an api-tokens Secret with one admin key that works on the events server only.