ego

ego is one process. ego serve listens on :8001 for the REST API, the web UI, the live feed and the MCP endpoint, on :4318 for OTLP/HTTP, on :4317 for OTLP/gRPC, and on :9090 for health. The UI is built into the binary. The image carries nothing else, so there is no runtime to install and no SDK to add to an agent.

Every receiver feeds the same session store through the same conversion path. Sessions live in memory up to a bound and are archived to one SQLite file, written without CGO. The archive holds sessions, the session index, runs and their result rows, eval sets, role bindings, sender tokens, the token usage ledger, online scores, alerts, findings, corrections and labels. Older sessions are read back from the archive when someone asks for them.

The commands are run (score trace files from the command line), serve, mcp (the tool registry over standard input and output, as a client of a running serve), auth mint-token and version.

Persistence and replicas

Persistence is optional. Without a session database, ego keeps the newest sessions in memory only and the run history, online evaluation and every view that needs the archive answer 503. One SQLite file allows one replica, so ego does not scale out; a restart is a short gap in intake until the pod is back. Large migrations at start run in batches and the chart has a long startup probe for them.

Traces arrive as OTLP over HTTP (POST /v1/traces, gzip accepted) or gRPC, or as files in Jaeger JSON, OTLP JSON or JSONL, which the loader tells apart on its own. Three attribute conventions are understood: ADK (gcp.vertex.agent.*), the OpenTelemetry GenAI semantic conventions and OpenInference. The converter turns a trace into invocations: the user input, the tool calls, and the final answer. The answer is the last call of the agent that opened the invocation, so a verifier or guard that speaks later is not scored as the answer.

Sessions

Spans group into sessions by ego.session_name, then gen_ai.conversation.id, then a synthetic name, always under the sender and the tenant label. A session completes after 30 s of quiet (3 s after a root span) and is queued for online scoring then. A late span reopens it, and a session already scored is not scored twice. A session holds at most 10000 spans.

Always-on agents: threads and segments

For an agent that wakes every few minutes for weeks, a session id names the agent and not a unit of work. A sender that declares the resource attribute ego.thread gets three levels. The thread is the declared value. A segment is what ego stores, lists, scores and cites as a session: a run of whole traces whose first one keeps the thread's id and whose n-th is <thread>~n. A turn is one trace, and a segment never cuts one. A trace goes to the segment its first span chose, so late spans still land in it. Segmentation is opt-in; every session without the attribute behaves as described above.

A closed segment takes no new trace and is scored once, with the end of the previous segment as fenced context for the Gemini judges. A single runaway trace is capped at 50000 spans. GET /api/threads and the list_threads tool give one row per declared thread.

Claude Code

Claude Code's native traces (claude_code.* spans) are translated to GenAI attributes at ingest: sessions come from session.id, the agent from service.name, and tokens count cache reads and writes in the input. A tool call that failed or was rejected is a failed step. The cache parts are kept in the usage ledger so a cost estimate is not inflated by an agent that mostly reads its prompt from the cache.

A token identifies a sender. A sender sends many tenants, labelled on its spans, and the tenant key is <sender>/<label>. Two senders can both send a tenant called acme and stay apart. The label is read from the span or resource attribute named by --usage-tenant-attr; a span with no label belongs to default. The shared --otlp-token names no sender, so its tenants are bare labels.

Names

A tenant key identifies a tenant but is hard to read. A sender can say the display name in the span or resource attribute ego.tenant.name. The first name seen is kept while the tenant has none, only a named sender can set one, a name is at most 80 characters with no control or format characters, and an admin's name always wins. A bound token takes its name when the admin mints it.

Senders and tokens

An admin creates senders and mints their tokens with POST /api/admin/senders/{id}/tokens. The body names the token's label and, for a bound token, the tenant and its display name. Each sender is limited to 5000 open sessions, 1000 tenants and 1000 names. A failed token check counts against its source, and a source with more than 30 failures in a minute is answered 429 (gRPC ResourceExhausted).

Removing tenants

POST /api/admin/purge-tenants removes tenants that no longer exist. The body lists tenant keys, never patterns, and has a dry-run flag. It deletes the sessions whose tenants are all among the keys, their usage and online-score rows, and the names. A session that also belongs to another tenant stays, and so do the labels people gave. The caller is written to the log.

Every result says whether an algorithm or a model produced it. Algorithms run locally with no key. The other families call a judge: Gemini through the GenAI client, the Vertex AI Managed Evaluation service through Application Default Credentials, or Jev, the decision model.

The default judge model is gemini-3-flash-preview, and each evaluator can name another. Quoted trace text in a judge prompt is fenced, block markers inside it are defanged, and every call opens with a notice that the quoted text is data. A judge call is capped at 8192 output tokens.

Interactive evaluation

POST /api/evaluate and its streaming twin score traces against an optional golden eval set. The request names stored sessions by id or uploads files, and a golden set can be built from a session. The streaming form reports progress per trace and per metric, can be stopped from the page, and a request can limit itself to chosen turns of the sessions. A run is recorded when it starts and closes as succeeded, cancelled or failed with its reason, so the run list shows it while it works. A run still marked running after a restart is closed as failed.

A one-off run scores traces when someone supplies them. Online evaluation adds a continuous loop over the traces ego already holds: sample, score, detect, notify. It takes no side in what produced the traces. The call kind is gen_ai.agent.name, else the root span name, and nothing in the code names a producer.

Sampling

A profile matches the kind of a session, and only a matching session is scored. The sampler reads each session's features: kind, tenant, service, service.version, model, error spans, tokens, latency and a fingerprint of the kind, the normalised first input and the tool-call names. Four rules decide what happens next.

• Interest. A session with error spans, with tokens or latency above the kind's rolling mean plus three deviations on a log scale, or running a version or model not seen for the kind in 24 hours, goes in the boost lane. A change starts a burst: the next afterChangeSessions sessions of the kind are boosted.
• Trivial. A session with no tool calls, a first message of at most 40 characters and a last reply of at most 100 is skipped before the rate buckets and counted as trivial. A session with error spans or unreadable text is never skipped this way.
• Dedup. A fingerprint scored inside dedupWindow without an interest reason is skipped.
• Rate and budget. Two token buckets per tenant and kind, one per lane, then a daily cap on judge tokens and on Jev calls. The scheduler reserves an estimate before scoring and books the actual cost after. Vertex metrics and metrics that need a golden set are refused, because the budget cannot meter the first and no golden answer exists for the second.

Every decision is counted by reason at GET /api/online/status.

Configuration

Profiles live in a JSON file, --online-config. Evaluators have the shape of the interactive request, with numSamples defaulting to 1, since one judge sample per invocation is the cost lever. maxInvocations scores only the last N invocations of a long session, and a profile may also match a service or a tenant, the most specific winning. Live traffic has no golden answers, so profiles use reference-free metrics.

{
  "budget":   { "judgeTokensPerDay": 2000000, "jevCallsPerDay": 1000 },
  "sampling": { "perKindPerHour": 6, "boostPerKindPerHour": 6,
                "afterChangeSessions": 20, "dedupWindow": "6h" },
  "profiles": [
    { "kind": "rca", "maxInvocations": 3,
      "evaluators": [ { "name": "jev_grounded" },
                      { "name": "hallucinations_v1", "numSamples": 1 } ] }
  ],
  "detector": { "window": "6h", "baseline": "168h", "minSamples": 12, "minDrop": 0.1 },
  "sinks":    [ { "type": "discord", "urlEnv": "EGO_DISCORD_WEBHOOK" } ]
}

Detection and alerts

Every 10 minutes, for each tenant, kind and metric, the detector compares the scores in the window with those in the baseline before it. A regression needs at least minSamples on both sides, a drop of the mean of at least minDrop, and a drop above 2.5 standard errors (Welch). One bad score never alerts. When one service.version or model holds at least half of the current samples and under a tenth of the baseline, it is named as the suspected change.

An alert is open, updated when the drop grows by another minDrop, and resolved after two checks in a row with the drop under half of minDrop, so it does not flap. Discord gets open, update and resolve with the baseline and current means, the suspect and the three worst sessions. The webhook carries ego.quality.v1 events, and the event sink posts EvalQualityDegraded and EvalQualityRecovered to a receiver that takes that shape. A daily digest goes to Discord at 08:00 UTC.

Online evaluation ends at a notification and a person reads it. The rest of the loop lets the agent that produced the traces read the verdicts, change itself, and have ego check that the change worked. ego never changes an agent. It diagnoses from the traces it holds, records what the agent says it changed, and reports whether the scores recovered.

Findings

A findings pass diagnoses low-scoring series with one judge call. It runs for each open alert, refreshed when the drop grows, and in a sweep every findings.every over series with at least findings.minSessions sessions in findings.window and a mean under findings.minScore. A finding holds the failure mode, a summary, a suggested correction and the evidence sessions, selected and fenced as data. The call is metered against the daily judge budget. Without a judge model, or when its answer is unusable, the finding is written from the evidence alone. States are open, corrected, resolved and stale.

Corrections

A correction records what the agent or a person changed, with the service.version that carries it. It marks its finding corrected and starts a sampler burst, as a new version does. On every detector tick it is verified against the scores since it was registered, at its version when given. Verified resolves the finding. Failed reopens it with the attempt in its history. A tenant and kind hold at most 20 pending corrections, and what is capped at 2000 characters.

Labels and the score of record

A label is a person's verdict on one metric of one session: agree or disagree, with an optional score and note, stored with the judge's score at that time and the caller's identity. Every aggregate read of scores joins each score with its newest label. A person's score replaces the judge's, an agreement keeps the judge's score and marks it confirmed, and a disagreement without a score leaves the session out. The judge's row is never rewritten. A sender's ingest token is refused by the label endpoint, so an agent cannot write the score of record on itself.

The agent's side is one skill, served by the server as the MCP prompt rsi_loop: check the findings, read their evidence, make one change per finding, register it, and come back for the verdict.

Sessions and runs

The Sessions view pages from the server, newest first, with a cursor and a total. Its filters (time range, tenant, agent, complete, LLM calls only, search) run against an index table in the archive, and the caller's role is checked on every row. A search reads memory and the newest 2000 archived rows that mention the text. A session shows its spans with attributes and errors, and its online scores with thumbs for labels. The Runs view lists every evaluation and groups them by eval set or agent, with pass rate and metric trends.

Activity of an agent

An agent that works in turns, one root span per prompt or message, has an Activity view. GET /api/activity lists the agents with turns, one row per agent and tenant, from sessions that hold claude_code.interaction spans or declared a thread. GET /api/activity/turns returns one row per turn across the agent's newest 80 sessions, oldest first, at most 3000 rows: start, duration, model requests and tool calls, the time of the model, of the tools and of the wait for approval, tokens with the cached part, which tool calls failed and what followed what. The window runs from 1 minute to 30 days.

Token usage

The usage ledger is built from gen_ai.usage.* on LLM spans and grouped by call kind, tenant, model and service. Each row also carries the cache read and cache write tokens, parts of the input. The UI prices Claude models with the cache read at a tenth and the cache write at 1.25 times the input rate, and marks a cost as an approximation: billing is computed from the traces and is not exact. A failed request that reported no tokens has no ledger row, so it counts as an error span and not as a failed call. GET /api/usage/limits shows the caps in force and what is used, read only.

Alerts flag

The red flag in the sidebar lists the latest alert of each tab, five at most: a session with a failed LLM call, a daily budget at 90% or more, a run that failed or had under half its decided results pass, and the latest low online score or open alert. Tenant-limited roles get their tenants' only. The quality, usage and runs views all follow the tenant chosen in the switcher.

One tool registry serves two transports. ego mcp on standard input and output is a REST client of a running server. serve mounts the same tools at POST /mcp (streamable HTTP, stateless; GET answers 405, and a body over 1 MiB is refused with 413) and runs them in process against the stores, so a role's agent and tenant limits apply to every tool. No tool is defined twice.

/mcp sits behind the same login and roles as the API. A sender's ingest token also authenticates there, and only there, as a member limited to its own tenants. It is the token the agent already sends its traces with, so one token sends traces and reads verdicts. The shared OTLP token does not, because it names no sender.

The evidence a tool returns is read from the stored online results: the per-sentence verdicts of hallucinations_v1 (unsupported, contradictory and disputed first), per-rubric verdicts, trajectory comparisons and the rationale of the jev_* decisions. A tool that needs online evaluation answers an error when it is not configured. The prompt rsi_loop is registered on both transports.

Authentication is active only with a session secret. Sign-in is GitHub OAuth restricted to an organisation and gives a signed session cookie. API tokens come from ego auth mint-token, and a raw GitHub token is accepted when its owner is a member of the organisation. Cookies and API tokens are different kinds, each with an id and an issue time. A token carries the role it was minted with. An admin can revoke one token or everything issued to a subject, and logging out revokes the cookie.

A binding names a user or a GitHub team and may be limited. An agent scope limits sessions, traces, runs and the live feed to named agents and fails closed: views that cannot be attributed to an agent, such as usage, quality and eval sets, are closed to it. A session with several agents is visible to a role scoped to any one of them. A tenant limit (keys, <sender>/* or *) narrows the same views to those tenants, and a tenant-limited role reads runs only for the online runs of its tenants. Unknown team membership is refused and never takes the default role. --default-role none gives nobody access without a grant. The whole feature is inert when no session secret is set.

A bound ingest token, on /mcp, reads only its own tenant. Nobody who holds an ingest token can write labels.

Every flag of ego serve can be set from the environment as EGO_<NAME>. The settings that matter most:

The judge uses a Gemini key or, with GOOGLE_GENAI_USE_VERTEXAI=true, Application Default Credentials. Vertex managed evaluation needs GOOGLE_CLOUD_PROJECT and, optionally, GOOGLE_CLOUD_LOCATION.

ego runs from the binary on a laptop, or in Kubernetes from a Helm chart. The image is a static binary on a minimal base with no shell. The chart renders the online evaluation config into a ConfigMap, runs a single replica with a persistent volume for the database, sets a startup probe that tolerates long migrations, and exposes the API through a gateway route. A new root-level API path must be added to the route's list of root paths or the gateway answers 404.

Each tagged release publishes through CI: binaries for Linux and macOS on amd64 and arm64 with checksums.txt and its signature bundle, a signed multi-arch container image, and the Helm chart as an OCI artifact. ego version prints the installed version, and the install script verifies its download against the checksums. CI on every push and pull request builds the UI, then runs go build, go vet, gofmt, the tests, the vulnerability scan and a container build.

Two simplifications are known. The ROUGE-1 tokenizer does not stem and does not split unspaced scripts, so response match runs lower on inflected English and means nothing for those scripts. A GitHub organisation member who is removed from the organisation keeps a cookie for up to 12 hours.