Skip to main content

Documentation Index

Fetch the complete documentation index at: https://oxy.tech/docs/llms.txt

Use this file to discover all available pages before exploring further.

Oxy can record every agent run, workflow execution, and tool call as a trace — a hierarchical view of what the AI did, how long each step took, which SQL was generated, how many tokens were used, and whether anything errored. Traces show up in the Observability section of the Developer Portal so you can debug slow runs, audit what agents produced, and spot patterns across many executions. Observability has two independent switches:
  • OXY_OBSERVABILITY_BACKEND — turns span capture on. Set it to duckdb, postgres, or clickhouse and Oxy starts writing traces to that store. No capture happens if it’s unset.
  • --enterprise — mounts the observability UI and API routes (the Traces / Metrics / Execution Analytics pages and their /api/* endpoints). Without it, traces are still captured (if a backend is set) but are only reachable via direct DB access.
For the normal “I want to see traces in the Developer Portal” path you want both.

​
Enabling observability

Pick a storage backend by setting OXY_OBSERVABILITY_BACKEND, then start the server with --enterprise to expose the UI. 
export OXY_OBSERVABILITY_BACKEND=duckdb
oxy serve --enterprise
Or with the Docker-managed PostgreSQL stack: 
export OXY_OBSERVABILITY_BACKEND=duckdb
oxy start --enterprise
When --local is set, DuckDB is automatically used as the default backend — you don’t need to set OXY_OBSERVABILITY_BACKEND for local development. 
oxy serve --local --enterprise   # DuckDB is the implicit default in --local mode
Open http://localhost:3000/ide/observability/traces to see traces. Run an agent or workflow and the corresponding trace appears within a second or two.

​
Choosing a backend

There is no default outside --local. Pick the backend that matches your deployment:
BackendWhen to useStorage
duckdbSingle-instance deployments, local devEmbedded file at $OXY_STATE_DIR/observability.duckdb
postgresMulti-instance / Kubernetes deploymentsReuses OXY_DATABASE_URL — same database as the app
clickhouseHigh-volume installs (>1M spans/day)Dedicated ClickHouse instance
DuckDB is fastest for small-to-medium volumes and needs zero extra infrastructure. The trade-off is it only supports a single writer, so it can’t be shared across multiple Oxy pods.Postgres is the right pick for horizontally-scaled deployments. It reuses the connection pool Oxy already uses for its own state, so there’s no extra database to provision.ClickHouse is overkill for most installs. Only reach for it if DuckDB’s single-writer limit is an actual bottleneck for you. oxy start starts a ClickHouse container automatically when you pick this backend.
If you start the server with --enterprise but forget to set OXY_OBSERVABILITY_BACKEND, the Traces page renders a “not configured” banner and nothing is recorded. The server also prints a warning on startup.

​
What’s recorded

Oxy emits spans for the major execution boundaries:
SpanDescription
agent.run_agentTop-level agent run. Parent span for everything the agent does.
workflow.run_workflowTop-level workflow execution.
analytics.runAgentic analytics pipeline run.
llm.callIndividual LLM request (includes prompt/completion token counts).
tool.callTool invocation — SQL execution, semantic query compilation, Looker lookup, etc.
Each span carries attributes like the agent name, model, execution status, and error messages if any. Nested spans are stitched into a waterfall view in the Trace Detail page so you can see where time was spent.

​
Environment variables

VariableDefaultDescription
OXY_OBSERVABILITY_BACKEND(unset)duckdb, postgres, or clickhouse. Defaults to duckdb when --local is set. Unset disables observability entirely.
OXY_DATABASE_URL(required for postgres)PostgreSQL connection string. Reused by the postgres backend.
OXY_CLICKHOUSE_URL(required for clickhouse)ClickHouse HTTP endpoint, e.g. http://localhost:8123.
OXY_CLICKHOUSE_USERdefaultClickHouse username.
OXY_CLICKHOUSE_PASSWORD(empty)ClickHouse password.
OXY_CLICKHOUSE_DATABASEobservabilityClickHouse database name.
OXY_OBSERVABILITY_LOG_LEVELdebugFilter for span capture. Independent of OXY_LOG_LEVEL — console verbosity does not affect what is recorded.
OXY_SERVICE_NAMEoxyService name attached to every span.

​
Retention

Trace data is automatically pruned after 90 days. Retention is derived from the longest time-window the Traces UI exposes, so the UI and the retention policy always agree. For DuckDB and Postgres, a background task deletes rows older than the cutoff every 6 hours. For ClickHouse, the same policy is applied at the engine level via ALTER TABLE ... MODIFY TTL on schema init, so expiration is handled by ClickHouse’s own background merges.

​
Switching backends

Change OXY_OBSERVABILITY_BACKEND and restart the server. Trace data is not migrated across backends — each one has its own storage, so switching from DuckDB to Postgres starts you fresh. If you need to keep historical traces, export them before switching.

​
No traces showing up?

  1. Confirm the server was started with --enterprise.
  2. Confirm OXY_OBSERVABILITY_BACKEND is set (or that you’re using --local, which defaults to duckdb). Check the startup output for a line like Observability: duckdb (...).
  3. If the Traces page shows a “not configured” banner, the backend env var wasn’t set when the server started.
  4. OXY_LOG_LEVEL does not affect what is recorded. Traces are captured even at warn.
  5. For Postgres: make sure OXY_DATABASE_URL is set and the observability_spans table exists (it’s created by a migration on first boot).
  6. For ClickHouse: check docker logs oxy-clickhouse (if using oxy start) or your external instance’s logs.
For deeper architecture details see the internal observability docs.