Observability Setup

VoidBox exposes structured run events and guest telemetry by default. OTLP trace and metric export is available when built with the opentelemetry feature and when you run the observable execution paths.

What’s Included

OTLP traces — Per-box spans, tool call events, pipeline-level trace context.
Metrics — Token counts, cost, and duration per stage.
Structured logs — [vm:NAME] prefixed and trace-correlated for easy filtering.
Guest telemetry — procfs metrics (CPU, memory) exported from the guest to the host via vsock.

Enable OTLP

Build with the opentelemetry feature flag and set the OTLP endpoint:

cargo build --features opentelemetry

Then set the endpoint environment variable when running an observable example:

VOIDBOX_OTLP_ENDPOINT=http://localhost:4317 \
cargo run --features opentelemetry --example claude_in_voidbox_example

Configuration

Environment Variable	Description
`VOIDBOX_OTLP_ENDPOINT`	OTLP gRPC endpoint (e.g. `http://localhost:4317`)
`OTEL_EXPORTER_OTLP_ENDPOINT`	Standard OpenTelemetry fallback for the endpoint
`VOIDBOX_SERVICE_NAME`	Service name for exported telemetry (default: `void-box`)
`OTEL_EXPORTER_OTLP_HEADERS`	Reliable way to pass OTLP auth headers today (`key=value,key2=value2`)
`VOIDBOX_OTLP_HEADERS`	Parsed internally, but not yet wired into the exporter path; do not rely on it yet for collector auth

Trace Structure

When observable execution is enabled, traces follow a hierarchy from the pipeline level down to individual tool calls within each VM stage:

Pipeline span
  └─ Stage 1 span (box_name="data_analyst")
       ├─ tool_call event: Read("input.json")
       ├─ tool_call event: Bash("curl ...")
       └─ attributes: tokens_in, tokens_out, cost_usd, model
  └─ Stage 2 span (box_name="quant_analyst")
       └─ ...

Each stage span carries attributes for token counts, cost, model used, and duration. Tool call events are recorded as span events within the stage span.

Guest Telemetry

The guest-agent inside each micro-VM periodically reads /proc/stat and /proc/meminfo, then sends TelemetryBatch messages over vsock to the host. On the host side, the TelemetryAggregator ingests these batches and exports them as OTLP metrics.

Guest telemetry gives you per-VM resource utilization without any agent-side instrumentation. CPU and memory metrics flow automatically as long as the guest-agent is running.

Playground

The repository includes a ready-to-run observability stack in the playground/ directory with pre-configured:

Grafana — Dashboards for pipeline traces and metrics.
Tempo — Distributed trace backend for OTLP ingestion.
Prometheus — Metrics collection and storage.

See the playground/ directory in the repository for setup instructions.