Triage Agent

Data crosses three boundaries: from kernel and host files into the sensor roles, from the node into sensorstream over gRPC, and from the cluster to the core over a secure WebSocket (WSS) the agent opens. Nothing outside the cluster can open a connection into it.

The sensor roles send one OTLP snapshot per role every --sensor-tick, default 15 s. sensorstream keeps the latest per node and serves it to the collectors through metricset.Source. Collectors are manager.Runnables that run only on the leader, and only the leader holds the connection to the core. The controller can run several replicas: every replica takes sensor streams and gossips what its own sensors send to the others over mutual TLS (headless Service <fullname>-peers, port 9471, pinned to the same sensor certificate), keeping the newest snapshot per node. The leader therefore sees every node whichever replica a sensor connected to, and a newly elected leader already holds the state.

The agent is one container image, run in two shapes. The DaemonSet hosts the sensor roles; the controller Deployment hosts everything that talks to the Kubernetes API or the core. A namespace is observed only when it is annotated with a tenant or listed in --target-namespaces, and every frame leaves stamped with that tenant.

Collectors are off by default. Virtual apps and Kubernetes Warning events flow as soon as push mode is configured; RED, USE, log and topology collectors each need their own flag and the sensor stream.

Out of scope for the agent: comparing the observed topology with the declared graph, opening issues, and any model call. The core does those.

High availability

The controller runs as several replicas behind one leader. Leader election is on whenever replicaCount is above 1, through a Lease named triage-agent in the agent's namespace. Every replica serves the sensor stream and the OTLP receiver; only the leader runs the collectors and holds the connection to the core.

A single replica runs without leader election and without a PodDisruptionBudget; a restart is a gap in intake and frames until the pod is back.

Linux host mode

With --mode=host the agent runs as one systemd service on a VM or bare-metal host, with no Kubernetes API. The host reports the same frames as a cluster, so the core and the protocol are unchanged: a host is a cluster of one with one synthetic namespace. There is no controller Deployment, no DaemonSet, no sensor stream and no listener; the sensor roles run in the agent's own process and the collectors read them directly.

Install is a binary or a deb or rpm package with triage-agent.service, which reads its flags from /etc/triage-agent/env and runs as root with the same six capabilities as the DaemonSet. The hello carries the capability host. The acceptance steps have not yet been run on a real host.

Each item the agent sees, whether a workload, an event or a metric series, is resolved to a tenant through its namespace before it is queued. An annotated namespace always goes to its annotation's tenant. Tenant mode adds a primary tenant for namespaces listed in --target-namespaces; cluster mode works from annotations alone.

Annotations

Objects

Objects the chart installs in the agent namespace.

Watched API types

The controller follows these Kubernetes API types through informers. All access is get, list and watch.

Custom types are added with workloadResources (flag --workload-resources), one <group>/<version>/<resource> per entry, and a matching read rule in rbac.extraWorkloadRules. A non-empty list replaces the defaults, so keep the ones still needed. A type the API server does not serve is skipped at startup with one log line.

workloadResources:
  - apps/v1/deployments
  - apps/v1/statefulsets
  - apps/v1/daemonsets
  - argoproj.io/v1alpha1/rollouts
  - helm.toolkit.fluxcd.io/v2/helmreleases   # Flux HelmRelease
rbac:
  extraWorkloadRules:
    - apiGroups: ["helm.toolkit.fluxcd.io"]
      resources: ["helmreleases"]
      verbs: ["get", "list", "watch"]

Signals

Five sources feed the agent. RED and topology come from OBI (OpenTelemetry eBPF Instrumentation, vendored at third_party/obi), which chooses its own hooks; the agent loads no eBPF program of its own.

OTLP receiver

A cluster that already runs OpenTelemetry, or cannot run the DaemonSet, pushes OTLP metrics to the controller. The receiver keeps each resource's newest cumulative point per series and hands it to sensorstream, so the same collectors, baseline gates and replica gossip apply. Metric names route it: the HTTP, gRPC and database histograms and obi.network.flow.bytes go to RED and topology, use.* to USE, log.errors, log.warns and log.lines to LOG. Where a sensor and an OTLP sender report the same service or pod, the sensor's numbers are used and the OTLP ones dropped, so nothing is counted twice.

A Collector that has only traces, kubelet access and logs covers all three frameworks without the DaemonSet. The receiver translates two inputs it would otherwise ignore:

Points need service.name and k8s.namespace.name and cumulative temporality. Older conventions are converted: http.server.duration in milliseconds, http.status_code, a numeric rpc.grpc.status_code. The workload comes from k8s.owner.name or the k8s.deployment.name family that the k8sattributes processor adds. Delta temporality and exponential histograms are refused and reported in the response. Limits: 5000 resources, 20000 series each; a resource silent for 10 minutes is dropped.

The receiver serves the controller's certificate, which senders pin from -sensor-ca. A sender authenticates with a client certificate that chains to otlp.clientCASecret, which is the OTel exporter's own tls.cert_file and key_file, or with a projected ServiceAccount token, audience triage-agent-otlp, of an account in otlp.allowedServiceAccounts, checked by TokenReview.

exporters:
  otlp/triage:
    endpoint: <fullname>-sensors.<namespace>.svc:4317
    tls:
      ca_file: /etc/otel/triage/ca.crt       # from -sensor-ca
      cert_file: /etc/otel/certs/client.crt  # chains to otlp.clientCASecret
      key_file: /etc/otel/certs/client.key

Virtual apps come from workloads only, synthesized by workload name or by label when a namespace opts in. The workload kinds and their readiness rules are under Watched API types above.

Sandboxed runtimes. eBPF works on the host kernel, so sandboxes limit it. The agent cannot run under gVisor, Kata or user-namespaced pods, which have no host kernel or host PID namespace to probe. Workloads that run under them are mostly invisible to the traffic role: their traffic passes through the sandbox's own kernel, so RED, SIG and L7 topology are missing and at most L4 flows at the sandbox's interface are seen.

Gated frames carry only services that crossed their baseline in the window. Summary frames carry every service, ungated, so the core can see quiet services too.

Wire formats

Inside the cluster the agent speaks native OpenTelemetry. OBI emits OTel metrics under the semantic conventions, and the sensor roles forward them unchanged as OTLP ResourceMetrics with cumulative temporality. No custom metric model sits between OBI and the collectors.

The sensor stream wraps those metrics in a small protobuf envelope. Each tick a sensor sends one Snapshot per role and one SensorState; the controller acknowledges each message by sequence number.

service SensorService { rpc Stream(stream StreamRequest) returns (stream StreamResponse); }

message StreamRequest { uint64 seq; Timestamp time; oneof { Hello | Snapshot | SensorState } }
message Snapshot {
  string role = 1;
  repeated opentelemetry.proto.metrics.v1.ResourceMetrics resource_metrics = 2;
}
message StreamResponse { Ack { uint64 seq } }

The collectors turn the cumulative histograms into per-window request count, error count and p99 per service, then gate them. What leaves the cluster is one JSON frame per WebSocket text message: a shared envelope with exactly one payload field named after the kind. The red_metrics example below uses illustrative values.

{ "v": 1, "kind": "red_metrics", "cluster": "<cluster>", "tenant": "<tenant>",
  "redMetrics": {
    "windowStart": 1759834800, "windowSec": 60,
    "services": [{
      "namespace": "shop", "service": "checkout",
      "requestCount": 1840, "errorCount": 41,
      "sample": "GET /api/orders -> 502 x41",
      "p99Ms": 812, "baselineP99Ms": 140, "latencyBreach": true,
      "breach": "op-latency", "operation": "POST /api/orders",
      "observed": 812, "baseline": 140, "evidenceSec": 180
    }]
  } }

The payload fields are hello, snapshot, appDelta, k8sEvent, logMetrics, redMetrics, redSummary, useMetrics, useSummary, topologySnapshot and sensorState. Field names are a pinned contract with the core and must stay byte-identical on both sides; fields added later are optional.

Wire caps

A log_metrics frame carries one sample line per service so the core can see what the errors say. The fingerprint that groups errors is masked; the sample that leaves the cluster is the line with secrets and email addresses replaced by ***, then cut to length.

Both links are encrypted and authenticated in both directions, and node roles open no port. Between the controller and the core, each side proves itself with a certificate inside the WebSocket: the core first, by signing a challenge nonce with its agent CA, then the agent, with its admin-approved Ed25519 certificate and a signature over the same nonce. This is mutual certificate authentication over TLS 1.3, carried in the WebSocket rather than as TLS client certificates.

Between a sensor and the controller, one-way TLS plus a token gives mutual authentication with nothing extra to run. The controller holds a self-signed ECDSA P-256 certificate that each sensor pins from the -sensor-ca ConfigMap, so the sensor knows it reaches the real controller. The sensor presents no client certificate. It sends its projected ServiceAccount token, audience triage-agent-sensors, and the controller checks it with a TokenReview, a call to the Kubernetes API server that verifies the token's signature, expiry, audience and bound pod and returns the ServiceAccount and node it was issued for. The controller needs only create on tokenreviews for this. The token must belong to the -ebpf ServiceAccount, and the node named in the token must match the node in the sensor's hello. The kubelet issues the token per pod, valid for one hour and rotated automatically, so there is no client CA, no per-node certificate and no shared client key. A compromised node cannot send data as another node.

• Listeners. Only the controller listens: 9470 for the sensor stream when --sensor-listen is set, and 4317 for OTLP when --otlp-listen is set. Metrics on 8080 are off by default (--metrics-addr=0). No pprof in the repository.
• Keepalive. 30 s with a 10 s timeout on both ends of the gRPC stream.
• Certificate life. The sensor certificate is valid for 10 years and regenerated 30 days before expiry. The join certificate renews with less than a third of its life left.

RBAC is read-only on workloads. The agent has no cluster-wide Secret or ConfigMap access and updates three named objects in its own namespace, each by resourceNames.

Process privilege

Both workloads run the same image, whose default user is nonroot (UID 65532, Chainguard static base). The controller keeps that user and needs no host access. The DaemonSet overrides it with runAsUser: 0 in its pod spec, in both modes below. It runs all sensor roles in one container, agent, so the traffic, resource and logs-based metrics roles share one security context. The chart offers two privilege modes for it, selected by ebpf.privileged.

Root scope. UID 0 is what makes the six capabilities effective: Kubernetes has no field for ambient capabilities, and file capabilities on the binary are blocked by no_new_privs. Root here is not full privilege; it is UID 0 with every other capability dropped, privileged: false, no_new_privs, seccomp and a read-only root filesystem. hostPID already requires the privileged Pod Security level, with or without root. Comparable eBPF agents such as Beyla, Cilium, Falco and Tetragon also run as root, most with wider capabilities or full privilege.

Extra capabilities

Both widen the container beyond the audited default; the chart advises trying without them first.

Host access

With --topology-enabled, the RED collector builds a topology_snapshot from OBI's client-side metrics and, when enabled, L4 flows. Counts accumulate across RED ticks and flush every 600 s as one full snapshot, so a lost frame heals on the next flush. The snapshot keeps the top 500 edges by calls and the first 200 nodes, and is skipped when every scrape in the tick failed.

The agent has no drift comparator. The core compares the snapshot with the declared service graph. A snapshot request does not resend topology; the next flush carries it.

Each connector is its own Go module, deployed apart from the agent, that turns a provider's signals into events for the core events API.

promevents runs non-root with a read-only root filesystem and all capabilities dropped; its Role in the core namespace grants verb ingest on named tenants only.

Requirements

Flags and values

third_party/obi is an owned fork of OBI v0.12.2 with 11 local commits, including eBPF C changes, and is not tracked upstream; Apache-2.0 attribution stays in NOTICE. Its Go offsets table, used for binaries without debug info, covers Go up to 1.26.4 and grpc-go up to 1.81.1. Refresh it with OBI's tracker when new Go or grpc-go releases land, and keep scanning its go.mod for CVEs.