X-Ray

X-Ray is one binary that holds two kinds of memory. The memory server holds authored memory: items that agents and people write, kept as MemoryCell objects. xray graph holds derived memory: a graph of code, documentation and Kubernetes manifests, rebuilt from sources that already exist. The server reads the same snapshots, so one endpoint answers search_memory and search_graph (section 09). Either half runs without the other, on a laptop or in a cluster.

Data crosses three boundaries: from clients into the HTTP and MCP servers, from the server to the Kubernetes API through a controller-runtime cache, and from the server to model providers for embeddings and voice. With a cluster, the MemoryCell objects are the store. Without one, the server runs on process memory alone and warns at start.

Listeners

Base components

Out of scope

• Model-driven background work. The server makes no model call except to embed text and to hold a voice session. Consolidation, summarising and generalising belong to the agent, which writes the result back with upsert_memory.
• Anything beyond the API and the web UI it serves: sign-in providers other than GitHub and Microsoft Entra, and tenant switching (section 08).

A memory item is a MemoryCell in the group xray.ai, version v1, short name mcell. Its spec carries the item.

apiVersion: xray.ai/v1
kind: MemoryCell
metadata:
  name: checkout-retries-3f2a9c1e
  namespace: xray
spec:
  id: checkout-retries
  kind: operational_pattern
  scope: public
  subject: checkout
  title: Checkout retries on 503
  content: |
    The payment client retries a 503 three times with jitter.
  confidence: high
  status: active
  tags: [payments]
  createdAt: "2026-10-24T09:00:00Z"
  updatedAt: "2026-10-24T09:00:00Z"

Kinds

The first ten rows are the kinds upsert_memory lists. The last three are written by the server and the chart.

Scope

Privacy is a scope, separate from kind. A private item is isolated to its subject: reading, fetching or deleting it needs the caller to present that subject, and its live events go only to the owner. A public item is shared knowledge, visible on the graph and in search. One predicate, private when scope is private or the kind is system_configuration, guards every read path. Callers that set a scope are trusted for it: the subject is a tenant key, not an authenticated identity. Voice writes are always private, with the subject set to a hash of the signed-in user.

Provenance

Every cell carries an origin. A write is human when its credential names a person: an MCP token issued through sign-in, or a voice session of a signed-in user. It is derived when it does not: the shared MCP secret names nobody. The caller cannot choose; the server sets it. observed (written by code from facts) and verified (derived, then checked) are for cells that code writes or a cluster admin applies. A cell stored without an origin reads as derived.

A derived write cannot replace a human cell; the upsert returns an error. A person can replace a derived one. upsert_memory takes evidence_refs, the ids a cell was drawn from, and refute_memory follows them: it retires a cell that is wrong (status stale), records the reason and evidence under data.refuted, and retires every active derived cell that cites it by evidence ref or link, one hop. Human, observed and verified cells are never retired by the cascade.

Writing rules

A write through MCP or a voice session needs a kind from the table above (the server-written kinds are not writable), and a stable_fact needs confidence high: a fact that is stable is one the writer is sure of. A kind filter on search or list must name a searchable kind, so a mistyped one is an error and not an empty result. A cell can be replaced (supersede_memory writes the new cell, lists the old id in supersedes and retires the old one as superseded), expired when it is out of date (expire_memory, status stale) or refuted when it is wrong.

Duplicates

An upsert that restates a stable_fact, operational_pattern, watch_item or hypothesis lands on the cell it repeats (status merged, tags unioned) instead of creating a second one. The cells must share kind, scope and subject, and a derived restatement is never merged into a human cell. Similarity is the overlap of title and content tokens: 0.3 Jaccard, or 0.9 of the smaller text contained in the larger. Texts under eight tokens need 0.8. A short token counts when it has a digit, so "Q3 plan" and "Q4 plan" stay two cells. --memory-write-dedup=false turns this off. consolidate_memory finds the same clusters among cells already stored and, with apply, retires the restatements as superseded, listing them on the keeper.

Kubernetes holds authored memory. The in-memory indexes are a cache of it and are rebuilt at start. Derived memory ships as gob snapshots.

Gob snapshots

A map is a pre-built graph saved as <name>.graph.gob.gz with its HNSW index. Loading a map embeds nothing in the pod. A snapshot reaches the server by one of two routes:

• A file. A snapshot in the directory named by --mental-maps-dir. The server walks it and loads every snapshot it finds, and watches it for new files. The xray-graph chart mounts snapshot images there as image volumes; the memory chart mounts none.
• Carried in a cell. A cell holds the gob in data.graph. The cell syncer decodes it and registers the map, and skips the work when the bytes are unchanged.

Snapshots must match the server's embedding fingerprint: the model, --embedding-dims and --embedding-max-input-chars.

Search ranks every candidate lexically. When an embedding endpoint is configured it also ranks by meaning, and fuses the two lists by rank. Without an endpoint, search is lexical only.

Lexical rank

A candidate is admitted when it matches enough of the query: every token of a one- or two-token query, all but one of three or more. A token matches as a substring or within one edit of a word (none for tokens of one or two characters), so a misheard word still finds the item. An item's id is part of what it matches, and a query that is exactly an id returns that item first.

The score is a bonus for the whole query in the title, a graded match per token (title above content, exact above fuzzy), a small trigram overlap, and a penalty for each token missed. Tokens are weighted by how distinctive they are among the candidates, as a smoothed inverse document frequency: a word nearly every item contains counts for little, matched or missed, and a word few contain dominates both. For a query of three or more tokens an item that fails the coverage rule is still admitted when the tokens it did match carry 35% of the query's weight, so "what is wrong with the bigquery export" finds the item that names BigQuery.

Semantic rank

The query is embedded and compared to the HNSW index by cosine similarity. The default model is gemini-embedding-001 at 128 dimensions, from Vertex AI (--gcp-project) or any OpenAI-compatible endpoint (--embedding-endpoint). Input is cut to 8000 characters unless --embedding-max-input-chars says otherwise. --semantic-score-gap drops results more than a given cosine gap below the top hit.

A cell is embedded as its kind, scope, subject, title and content joined by spaces, cut to 1800 bytes on a character boundary (the batch limit of a local embedder is 2048 tokens, and a longer input is rejected outright). The store knows its embedding dimension and refuses a vector or a query of another length. If a search's whole top window is rejected on lookup, because every hit is a cell deleted or retired on another replica and still indexed here, the search ranks every cell exactly instead, so the cells that qualify are not starved. The embedding and index phases of each semantic search are reported separately as xray.memory.search.phase.duration.

Fusion

The two lists are combined by reciprocal rank, 1/(60+rank) summed over the lists an item appears in. Ranks are used because cosine similarity and lexical scores are on different scales. An item in one list contributes one term. By default every query fuses both lists. --search-lexical-shortcircuit restores the older behaviour of skipping the embedding call when a keyword hit already covers the query.

Shaping

The fused pool, over-fetched to four times the limit (at most 60), is shaped before it is cut. Near-duplicate facts of the four kinds above collapse to the best-ranked one, which reports how many it absorbed in collapsed. The pool is then re-ranked: each item scores its rank times a boost of 1.7 for an active watch item that has not expired, times a mild recency factor that falls from 1 to 0.85 with a seven-day half-life. It reorders close results; it does not override relevance. --search-recall-shaping=false returns the fused order. When a cross-encoder is installed it orders the pool in place of the re-rank.

Kind-filtered semantic search scans the items of that kind exactly instead of asking the graph for nearest neighbours and dropping the wrong kinds, which returned nothing for a kind that is a small share of the store. Query embeddings are cached in a bounded LRU, so a repeated search does not call the embedder again.

Searches scope to a subject and also return public items with no subject. The store also has opt-in hooks for one-hop link expansion, source collapse and a cross-encoder rerank; they are Go calls for programs that embed the package, with no flag or chart value.

Result limits

Each ceiling holds whatever limit the caller asks for. --headroom-content-limit bounds the characters of each result's content, to control the cost of voice context.

MCP tools

Read tools need the scope xray:memory.read; upsert_memory, delete_memory, refute_memory, consolidate_memory, expire_memory and supersede_memory need xray:memory.write. The shared bearer secret carries both.

When a map is loaded, the same endpoint also serves the graph tools, all read scope (section 09).

{
  "jsonrpc": "2.0",
  "id": 1,
  "method": "tools/call",
  "params": {
    "name": "upsert_memory",
    "arguments": {
      "kind": "operational_pattern",
      "title": "Checkout retries on 503",
      "content": "The payment client retries a 503 three times with jitter.",
      "subject": "checkout",
      "confidence": "high",
      "tags": ["payments"]
    }
  }
}

Go packages

X-Ray is also a Go library, importable as github.com/triageagent-dev/x-ray/v2. Three packages cover the two ways an agent uses it, and they share one interface per kind of memory, so an agent written against the interface runs on either.

A refusal, such as an unknown kind or a missing item, is a ToolError in all three, and IsNotFound tests for the missing item. The embedded store applies the same rules the server does at its boundary: a kind from the vocabulary, high confidence on a stable fact, a subject on a private item. The credential decides how a write is classed. Over MCP an OAuth token that names a person records human and the shared secret derived; in process, the program chooses with WithOrigin.

To make X-Ray the memory of an agent, it is used in three places: a search before the agent answers, memory tools while it answers, and a write afterwards that cites the items it drew on, so that refuting one retires the rest. A separate adapter module implements the Google ADK memory service on either an embedded store or a client, so X-Ray plugs into the ADK runner and its load_memory and preload_memory tools. Each user's conversations are stored as private notes under the pair of app and user, and shared memory is added to every search.

REST

The memory routes are read-only. Writes arrive by MCP, by voice, or as objects applied with kubectl.

Live events

GET /v1/memory/events carries two event types: a hit (the item ids, query and kinds of a recall that just ran) and a change (an upsert or delete). A recall publishes a hit when it is made by search_memory or get_memory_context over MCP, or by the voice search and list tools. Private items are never named: a hit lists the shared items a search returned, and a private change goes only to its owner. A client connected to one replica sees the events of that replica (section 11).

How a recall shows up

The web UI turns a hit into light, the way optogenetics does: a neuron made to carry a light-gated channel fires when blue light reaches it. The kinds in the event name the region, so a blue fibre lights it even when the recalled cell is not one of the nodes drawn. The cells in the region wash blue in the order the light reaches them, nearest the fibre first. The recalled cells fire hardest, with a double flare, three spike rings and an afterglow of a few seconds. A pulse of light then runs from each recalled cell along its links to the memories it connects to. It is laid over the category-coloured flash the graph already draws, and a repeat hit fires again. The region treatment is one fast attack and a smooth decay, about two pulses a second at most, because flicker over a large area is a photosensitivity risk; reduced motion gets one slow wash. The embedded web UI listens to this stream; a Triage deployment carries the same events over a WebSocket.

Every route under /v1 needs a session except the sign-in routes, /v1/version, /v1/categories, /healthz, /manifest.json and /sw.js. --dev turns authentication off and is for local use only.

--public-url is the externally reachable base URL. It scopes CORS, the OAuth callbacks and the origin check on the voice WebSocket. --trusted-proxy-hops says how many proxies in front of the server append the client address, which decides how X-Forwarded-For is read for rate limiting. --max-request-bytes caps request bodies.

A browser upgrades GET /v1/voice to a WebSocket and the server opens a Gemini Live session on its behalf. Two goroutines per session relay audio each way, and tool calls from the model run in parallel and return as tool responses. The server rejects a WebSocket whose Origin is not --public-url.

Each session reads the system_configuration cell at its start, so edits apply to the next session without a restart.

The voice tools are search_memory, list_memory, get_memory, fetch_memory, recent_memory, upsert_memory, save_memory, delete_memory and end_session. Voice writes are private to the user. When a map is loaded the graph tools are available to the model too.

Limits

• --user-daily-audio-token-limit: audio tokens, in and out, a user may spend per UTC day. 0 is unlimited. The remaining amount is pushed to the browser after each turn.
• --max-concurrent-voice: simultaneous sessions across all users.
• --voice-max-output-tokens: output cap per turn, 8192 by default, so a runaway turn fails fast.

Voice needs Gemini credentials: --gcp-project for Vertex AI, or --gemini-api-key for the Google AI API. Without either, the voice route is off.

The web UI is part of the binary. It is a client of the API above, built as a static export and embedded, and the server serves it at / on the same origin as the API. API paths, /healthz, the PWA files, /mcp and the OAuth endpoints go to the server; everything else is the UI. It runs the same locally and in a cluster: make run opens it on port 8080 with sign-in off, and the chart serves it through the same Service. --ui=false serves the API alone, and a binary built with UI=0 has no UI at all.

One origin is required. The voice WebSocket URL is built in the browser from its own host, the session cookie has no Domain, and the server checks the WebSocket's Origin against --public-url. Embedding gives it, so --public-url is simply the address users reach, and it is also the OAuth callback base.

Each document is served with a script-src that names the hashes of its own inline scripts, so no script runs without one; 'unsafe-inline' is allowed for styles only. Hashed bundles are cached for a year, and HTML is revalidated so an upgrade is seen at once.

The UI speaks this server's own API, /v1: voice at /v1/voice, live events as server-sent events, per-user notes, no tenants. The same sources build for a Triage deployment, which serves the same surface under /v2 with a WebSocket event stream and tenants. The code graph has a smaller page of its own at /ui on xray graph serve.

Categories

The UI groups items into categories. With none configured it shows the native taxonomy: Concerns (active_concern), Patterns (operational_pattern), Facts (stable_fact), Identity (system_identity), Events (event) and Notes (note). A deployment overrides this by setting ui.categories in the configuration cell. The server returns them from /v1/categories before sign-in.

xray graph builds derived memory: a graph of source code, documentation and Kubernetes manifests, embedded for search and saved as a snapshot. Nothing in it is authored, so it can be thrown away and rebuilt. It answers the structural questions that text search cannot: who calls this symbol, what a change reaches, where a decision was written down. It needs no cluster and no database, and it sends nothing off the host except text to the embedding endpoint you point it at.

Two builds

Only the Tree-sitter parser needs a C toolchain, so the build tag marks the boundary. The static build is the one deployed to a cluster. Asked to parse, it exits with a message pointing at the CGO build. The container image with the parser is published as a tag of the same repository, <version>-parser.

What it indexes

• Code. Go with resolved call, import and inherit edges; TypeScript, TSX, JavaScript, Python, Java, Rust, Ruby, C, C++, PHP and C# through a generic tags extractor that stores symbol bodies and call edges.
• Documentation. Markdown: a node per file and a node per second-level heading.
• Manifests. YAML, Helm templates included: a node per document, with reference edges to images, Secrets and ConfigMaps. It reads files and never contacts a cluster.
• Infrastructure. Compose files, Dockerfiles, application config and SQL DDL, which feed the service catalogue that services derives.

Node names are qualified by repository, repo//path::Symbol, so many repositories share one endpoint without collisions.

Parse

The walk skips ignored directories and does not descend into symlinked ones unless asked. Call resolution rests on declared evidence only: a bare call in the caller's own package, a pkg.Func() in the package the import names, a typed receiver in the package its type came from, a chained receiver followed through declared field and result types. There is no repository-wide name matching, and an ambiguous call yields no edge.

Embed

A node is not embedded as its source. Documentation nodes embed their prose; a file node becomes a one-line description; a code node leads with its signature and path, then folds in the first 600 characters of its body. The signature keeps exact-name queries strong and the body slice lets a paraphrase reach the function. Vectors are cached by that text, not by node identity, because every rebuild constructs fresh nodes; misses go out in one batch. The defaults are 256 dimensions with Matryoshka truncation and an input cap of 8192 characters, so an oversized node is truncated and not rejected.

Snapshot

A snapshot is a gzip stream holding two gob values, a header and a body, so a server can register many maps by decoding only the headers.

A mismatch in dimensions between the snapshot and the server is a hard rejection, because those vectors are not comparable with a query embedding. A difference in the model name only warns, since an embedding server answers with whatever model it has loaded. When the stored index is absent the loader rebuilds it; when present it is deserialised, which skips the insert loop that dominates a cold load. A snapshot built with -no-embed carries no vectors and is searched lexically only.

Encrypted snapshots

Strings in a gob are plain bytes, so an unencrypted snapshot is a readable dump of every node, and maps travel as OCI images. A snapshot can be written through age instead. The file keeps its name and layout and starts with the age header in place of the gzip magic; readers sniff for either, so one directory can hold both.

The key reads and the recipient writes. A builder needs only the public recipient (-snapshot-recipient), so a CI job never holds a secret. A server holds the identity (-snapshot-key, or --mental-maps-snapshot-key on the memory server). A map read encrypted is re-encrypted to the identity's own public key on every flush, and the flush refuses rather than write plaintext. The identity encrypts nothing else, so a plaintext map and the notes map stay plaintext unless a recipient is also given. An encrypted map found with no matching key is skipped with a warning that names the flag, and the other maps still serve. This protects the file at rest. The running server holds every map in memory and answers in the clear, so access to the server stays the boundary for the data itself.

Route

There is no merged graph. Each map keeps its own graph and index, with its own HNSW and embedding cache, so ranking never crosses projects. An exact map label is a fast path that needs no embedding. Otherwise the map hint and the query are embedded together and cosine-ranked against the summary vector of every resident map; an explicit hint reduces the fan-out to one, and --mental-maps-route-fanout sets it otherwise. An idle map is evicted after --mental-maps-ttl-min minutes; its small routing vector stays, so it is still reachable and reloads from disk on the next hit. The summary comes from an operator-written description, a README, or the symbol names, in that order, and the header records which.

Retrieve

Each search makes one query embedding, the only embedder call on the serve path, and then runs two channels.

• Semantic. HNSW, fetched at three times the limit and at least 20, so fusion has depth. A kind or repository filter applies after the index returns, so the fetch widens to compensate.
• Lexical. Per-word trigram matching behind a coverage gate: a query of one or two tokens must match all of them, a longer one two thirds, rounded up. Trigrams point at the words that contain them and words point at the nodes that use them, so a token is matched once per distinct word and only matching words are expanded to nodes. In code, trigrams discriminate poorly across declarations (every declaration contains func) and well across the vocabulary, which is why the index is built over words.

The channels are combined by reciprocal rank fusion with k = 60. Rank is used and not score, because cosine similarity sits in a narrow band while the keyword score is an unbounded integer. Replies are lean: 5 pointers by default and 20 at most (up to 100 when a kind, attribute or ordering selects a set), and get_graph_node returns a full body with its relations in both directions.

Update

The serving process holds the live graph, and update is a thin HTTP client onto it, so it works in the static build too. Changed files are re-parsed, and because the cache is keyed by node text only nodes that changed reach the embedder. A static build indexes Markdown and YAML in a batch and reports code files as skipped. A path that resolves to no served repository, through ../ or a symlinked directory the build did not follow, comes back as unowned and is never read.

Graph tools

These are served by xray graph serve and serve-snapshots, and by the memory server when it loads maps (--mental-maps-dir), on the same endpoint as the memory tools. A cell that carries a graph in data.graph registers as a map without a restart. Every read tool is annotated read-only, non-destructive and idempotent.

Agent notes

Three more tools keep short notes of what was asked, answered and decided, so a later session can read them back. They need a server started with -writable. Notes go to one map named by -memory-map (default session), created on first use and never mixed into a code or catalogue map.

Expired notes are swept on read and hourly. These notes are separate from MemoryCell memory (section 02): they are not typed, scoped or shared through the cluster, and they live in a snapshot file.

Access

Both graph servers listen on loopback unless told otherwise (127.0.0.1:8084 for serve, 127.0.0.1:8085 for serve-snapshots). Every request except /healthz passes three checks.

The memory server's own access rules (section 06) apply when it serves maps.

In a cluster

Two ways to run maps. The memory server takes them as cells that carry a graph, or as files in --mental-maps-dir. The separate xray-graph chart runs serve-snapshots as its own deployment, with maps from image volumes (Kubernetes 1.33 or later) or a volume seeded from a maps image, bearer-token authentication, an age identity, an HPA and an ingress. A maps image is a scratch image holding the snapshots under /maps; the build target never pushes it.

Memory decays by deadline. The only background work in the server is the deletion of expired items, and it makes no model call.

• Default deadlines. A newly created event gets 30 days, a recent_change or note 90 days, a hypothesis one day and an unknown seven, unless the caller sets staleAfter. Other kinds never expire by default. The defaults are fixed in code. Restating a hypothesis or unknown (the write merges into the existing cell) renews its deadline.
• Retention sweep. Every --retention-sweep-interval (one hour; 0 disables) the server hard-deletes items past staleAfter. Durable and configuration kinds are skipped.
• Episodic memory. Older than --episodic-ttl (7 days) it is dropped from recent_memory at read time, and a reaper deletes the cell.

A delete of a cell that another replica already removed counts as done, so the sweep is safe on several replicas.

The chart runs one replica, and one is the supported setting. Cells, sessions and tokens are shared through the cluster. The remaining state is held by the pod.

There is no leader election, because the server has no work that must run once. Search-hit events have publish hooks (SetHitPublishHook) and an apply call for remote events, but no transport is wired to them, so those pulses stay on the pod that served the search.

Flags set the baseline. The configuration cell and the chart's values.yaml override or supply them as noted. xray -help lists every flag.

Chart

The chart installs the MemoryCell CRD from its crds directory on first install. Helm does not upgrade CRDs, so apply a changed CRD by hand.