# Traces

> Distributed tracing for understanding where requests spend their time.

The traces add-on collects distributed traces from your app and shows them in Grafana. Use it when "this request was slow" needs to become "this request was slow because of this downstream call to this service."

## Attach it

```bash
watasu addons:create traces --app my-app
```

If this is your first observability add-on, Grafana is provisioned automatically.

## What your app needs to do

Tracing requires your app to emit spans. Use a standard tracing library — OpenTelemetry is the recommended path:

* instrument the inbound request boundary (your HTTP framework or RPC server)
* propagate context across outgoing calls (HTTP clients, RPC calls, queue producers)
* let the auto-instrumentation libraries cover database and cache calls

You don't need to configure an export destination. When the traces add-on attaches, your app receives:

| Variable                             | Value                                        |
| ------------------------------------ | -------------------------------------------- |
| `OTEL_EXPORTER_OTLP_ENDPOINT`        | The app's private trace ingest endpoint      |
| `OTEL_EXPORTER_OTLP_TRACES_ENDPOINT` | The same endpoint with the `/v1/traces` path |
| `OTEL_EXPORTER_OTLP_PROTOCOL`        | `http/protobuf`                              |

Standard OpenTelemetry SDKs read these automatically — instrument your app and the spans flow. Watasu collects, stores, and presents the traces in Grafana.

## What you can do in Grafana

* search by service name, operation, duration, or tags
* inspect individual traces with the full call tree
* compare latency distributions across operations
* jump from a slow trace into the matching log lines (correlation works out of the box)

## When traces help most

Traces pay off when:

* your app talks to several services or has multiple process types
* one inbound request fans out into many downstream calls
* p99 latency matters and logs alone can't explain it
* you're debugging a system, not a single binary

For a single-process app with two endpoints, traces are overkill. For a five-service mesh with queues in between, they're indispensable.

## Tips

* instrument the main user-facing request path first, then expand
* use service and operation names a human can read at 3am
* make sure spans reflect actual work boundaries (one giant 200ms span tells you nothing)
* use traces alongside logs and metrics — none of the three alone is enough

[View this page on the web](https://docs.watasu.io/observability/traces/)