Traces in Phoenix or Langfuse

This page is for operators who run their own OttoTester instance.

Each agent run can send a trace to a tracing tool: a record of everything the run’s agents did. It shows the run’s planning, writing and fixing stages, each agent inside them, and each model call with the model’s name, what was sent, what came back and how many tokens it used. The scores the run was graded with are attached to it. Passwords and keys the run signed in with are taken out before anything is sent.

OttoTester sends traces to one of two tools: Arize Phoenix or Langfuse. Only one at a time.

Pick the tool

Open Global Admin → Observability. Under Send traces to, choose Off, Phoenix or Langfuse, and press Save. It applies at once, with no restart.

Below it is one card for each tool, with the settings in the tables below. Fill in the tool’s card and press Save. Each field says where its value comes from: saved here, from the settings file, or not set.

A value saved on the page wins over the settings file. Clear a field, or press Use the settings file, to go back to the file.
Keys and headers are stored encrypted. Once saved, the card only says they are set, with Replace and Remove.
Test connection checks that the tool answers and takes its keys. Send test trace sends one trace named “OttoTester test trace”. Both use the card as it stands, and neither saves anything.
Save refuses a tool that is missing a setting it needs, and names the setting.
When you switch, every step that finishes after the save goes to the new tool. A run already going can have its trace split across both.
Every change is written to the Audit Log, with keys masked.

You can also set the tool in your settings file, with TRACING_TOOL set to phoenix, langfuse or off, then restart the server. A choice saved on the page wins over it.

Only the chosen tool’s settings are read. The other tool’s are ignored, even when they are filled in.
Left empty, TRACING_TOOL means Phoenix when PHOENIX_ENDPOINT is set, and off otherwise. An install that sent traces to Phoenix before this setting existed keeps doing so.
If the chosen tool is missing a setting it needs, nothing is sent, and the server says which setting is missing when it starts. Global Admin → Observability says so too.

Phoenix

Setting	Needed	What it is
`PHOENIX_ENDPOINT`	Yes	Where the server sends traces, such as `http://phoenix:6006` inside Docker.
`PHOENIX_API_KEY`	No	Only when your Phoenix asks for one.
`PHOENIX_PROJECT`	No	The Phoenix project traces go under. With none, `default`.
`PHOENIX_PUBLIC_URL`	No	The address your browser opens for “Open in Phoenix”. With none, runs show no Phoenix link.
`PHOENIX_HEADERS`	No	Extra headers sent to Phoenix, for a proxy or a hosted Phoenix, written like `{"X-Team": "qa"}`. Leave out `Authorization`; the API key sets it.

Langfuse

Setting	Needed	What it is
`LANGFUSE_BASE_URL`	Yes	Where the server sends traces, such as `https://cloud.langfuse.com`.
`LANGFUSE_PUBLIC_KEY`	Yes	Your Langfuse project’s public key, from its settings page (it starts with `pk-lf-`).
`LANGFUSE_SECRET_KEY`	Yes	The secret key made with it (it starts with `sk-lf-`).
`LANGFUSE_PUBLIC_URL`	No	The address your browser opens for “Open in Langfuse”, when it differs from `LANGFUSE_BASE_URL`.

OttoTester sends Langfuse the standard OpenTelemetry format, at LANGFUSE_BASE_URL/api/public/otel/v1/traces, and posts each score through Langfuse’s scores API.

When the server starts

The server starts with the settings file’s values, then switches to the ones saved on the page once its database is up. With a tool picked and its settings filled in, it checks that the tool answers and takes its keys, and writes one line to its log: connected, or what went wrong. It does the same after each save. If the tool later refuses the keys, the log says so, at most once a minute. Runs carry on either way; only their traces are lost.

Find a run’s trace

Open Global Admin → Observability. The top line says which tool traces go to. At the bottom, Recent runs lists the 50 newest runs, with Open in Phoenix or Open in Langfuse beside each. The link opens that run’s trace in a new tab.

Each run remembers which tool got its trace. If you switch tools, older runs keep linking to the tool that has their trace, for as long as that tool’s settings stay filled in. A run that saved no trace, because tracing was off or it ran before runs saved one, says “No trace saved”.

Running them beside OttoTester for development

The development stack in docker-compose.dev.yml runs Phoenix by default, at http://localhost:6006. Langfuse is there too, behind a profile, with its user, project and keys made on first start:

Set TRACING_TOOL=langfuse in .env.
Run COMPOSE_PROFILES=langfuse pnpm restart.
Open http://localhost:3300 and sign in as dev@ottotester.local with the password ottotester-local-only.

Those are throwaway local values. Never use them for a Langfuse anyone else can reach.

Reference → Environment variables