> Source: https://wexa.ai/docs/governance/observability

# Observability

An **observability trace** is the end-to-end record of one request as it crosses the platform's services. Use it to see which step took the time and where a failure started. Anyone who calls Wexa gets a trace identifier back. Reading the trace needs access to the collector your deployment exports to, or to the console's Observability module.

1. Take the trace identifier from the failing response's `X-Trace-Id` header, or from the error your SDK raised.
2. In the console, open **Governance**, then **Observability** (`/governance/observability`). The Traces tab lists traces and searches by operation or service.
3. To open one directly by identifier, go to `/governance/observability/trace/{traceId}`. The identifier is the value of `X-Trace-Id`, which is the OpenTelemetry trace id the gateway minted.
4. Or paste the identifier into your own collector's search.

Both SDKs attach the identifier to the error they raise: `trace_id` in Python and `traceId` in TypeScript.

## Limits

| Behavior | What to expect |
| --- | --- |
| Trace reach | The caller, the gateway, and whatever does the work: the context graph, the knowledge base, an execution. |
| Untraced hops | Appear as the gateway's own client span with nothing nested under it. You see that the call happened and how long it took, not what happened inside. Resolving a role while checking a grant is the common example. |
| Redacted arguments | Values whose key contains token, secret, password, key, auth, credential, cookie or session are masked. Result previews are truncated. |

**How tracing works and what you get by default**

## How it works

1. **The gateway opens a root span** for every request, before authentication, so refused requests are traced too.
2. **It names the trace in the response**, in the `X-Trace-Id` header, whatever the outcome.
3. **Each governed tool call nests under the root**: one span for the call and one for each lifecycle stage.
4. **The gateway exports** traces, metrics and logs in OpenTelemetry formats, over OTLP, to a collector you provide.
5. **You read the trace** in your collector or in the console.

The gateway only exports: it keeps no trace store and serves no trace over the Wexa API.

The console's Observability module reads traces from a telemetry backend through its own observability API, and its Settings page shows that backend's retention windows and sampling. Those reads are not governed calls, so they never appear inside a trace.

**Read the trace identifier of a failed call**

**Python SDK**

```python
from wexa import Fabric, WexaError

fabric = Fabric()
try:
    result = fabric.query_context(query="MATCH (n) RETURN n LIMIT 1")
except WexaError as e:
    print(e.trace_id, e.lifecycle_id)
```

**TypeScript SDK**

```ts
import { Fabric, WexaError } from '@wexa-fabric/sdk'

const fabric = new Fabric()
try {
  const result = await fabric.queryContext({ query: 'MATCH (n) RETURN n LIMIT 1' })
} catch (e) {
  if (e instanceof WexaError) console.log(e.traceId, e.lifecycleId)
}
```

**REST API**

```bash
curl -si -X POST https://fabric.wexa.ai/v1/query-context \
  -H "Authorization: Bearer $FABRIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"MATCH (n) RETURN n LIMIT 1"}' | grep -i '^x-trace-id\|^server-timing'
```

### What you get without configuring anything

These hold whether or not telemetry export is switched on.

**Trace context propagates.** The gateway installs the W3C trace-context propagator unconditionally. It honours an inbound `traceparent`, and its outbound calls to the services behind it carry the context onward. A deployment where only some services export still produces connected traces.

**Every response names its trace.** The gateway sets `X-Trace-Id` and a `Server-Timing` entry before the handler runs. The identifier is present on successes, on refusals and on requests that never authenticated. Quote it when reporting a problem.

**The shape of a trace**

## Read the shape of a trace

One tool call produces a nest of spans, in layers.

| Span | What it is | What it carries |
| --- | --- | --- |
| `gateway.request` | The root: one server span per HTTP request, opened before authentication | Method and path, client address as resolved through proxy headers, origin and user agent, the kind of credential presented, whether the request came over the Wexa MCP server or the REST API, and the response status |
| `lifecycle.<tool>` | The tool call, a child of the request span | `lifecycle_id`, tool name, surface, project mode, organization, project, actor and role, originating client and connection, a redacted rendering of the arguments, and, when the call was stopped, the reason and the stage that stopped it |
| `lifecycle.S1:authenticate` to `lifecycle.S10:record` | One child per [lifecycle](/docs/governance/audit-and-lifecycle) stage | Stage name, status, a sequence number, a redacted input and output, and stage detail |

The stop reason and stage sit on the parent span, so you do not have to hunt through children to learn why a call failed.

Stage detail includes:

- At `S3`, the quota counts and ceilings the call saw.
- At `S5`, the rule, verdict, reasoning and decision identifier.
- At `S6`, the approval identifier and reason.

Two details of the shape are deliberate.

- **`S8:execute` is opened live**, while the work happens, rather than replayed at the end. That is what makes the downstream service call nest inside `S8` instead of beside it, so time spent waiting on the graph is visibly inside the execute stage.
- **The sequence number orders the stages.** Governance stages run in well under a millisecond, so their start timestamps collapse onto the same instant. A viewer that sorts siblings by timestamp alone shows them jumbled.

**Debug a slow or failing run, and when there is no trace**

## Debug a slow run

Open the trace by its identifier and read down the waterfall.

1. **Compare the root span's duration with `lifecycle.<tool>`.** A large gap means the time went somewhere other than the tool call, usually authentication or a role lookup, which appears as a client span with nothing beneath it.
2. **Look at which stage is wide.** The governance stages other than `S8:execute` are sub-millisecond by nature. If one is wide, that is the finding, not a rounding artefact.
3. **If `S8:execute` is the wide one, descend into it.** The downstream call nests there, such as the graph read behind [`query-context`](/docs/context/query-context), or the agent run behind [`run-agent`](/docs/agents/run-agent). A wide child means a slow service. A narrow child means time spent in the gateway around the call.
4. **Check the quota counts on `S3`.** A project near its ceiling shows there, and it explains a run that is slow because its callers are backing off.
5. **For a multi-call run, sort by the `lifecycle_id`, not by the tool.** An agent making twenty calls produces twenty traces. The one you want is the one whose identifier the caller reported.

## Debug a failing run

The failure reason is on the parent span, so start there rather than at the deepest red child.

- **`S5:policy` failed:** a rule refused the call. The rule name and its reasoning are on that stage's span, and the same decision is retrievable at `GET /v1/policy-decisions`.
- **`S3:rate-quota` failed:** a [ceiling](/docs/administration/usage-quota-and-billing) was hit. The counts on the stage say which bucket.
- **`S6:approval` failed on a call with a resume token:** the token was rejected. It was spent, expired, or bound to a different tool, project or user.
- **`S6` pending:** not a failure. The call parked at an [approval](/docs/governance/policy-and-approvals) and the trace ends there legitimately. The rest of the story is in the trace of the resumed call.
- **`S8:execute` failed:** the tool or the service behind it failed. The child span carries the error, and its status tells you whether the gateway gave up waiting or the service answered badly.

## When there is no trace

If the `X-Trace-Id` you were given finds nothing in your collector, the likely causes are, in order:

1. Export is off for that deployment.
2. The collector endpoint is not reachable from the gateway. This degrades silently by design, so telemetry can never take the platform down.
3. The trace was not sampled. Head sampling is configurable and samples everything by default, so a deployment that has lowered it is missing traces in proportion.

The console shows the sampling ratio and the retention windows for traces, logs and metrics at `/governance/observability/settings`.

**Logs, metrics and what a trace never holds**

## Limits and behavior

| Behavior | What to expect |
| --- | --- |
| Logs | One structured log per lifecycle stage, with `lifecycle_id`, stage, status and detail, tool, organization, project and actor. Each request also gets an ingress log and a completion log. |
| Log correlation | Each record is stamped with the trace identifier, so a log line and its trace join without correlating by time. |
| Metrics | A counter of gateway requests and a histogram of request duration, both by route and response status. They come from the same middleware as the root span, so refused requests are counted. |
| Data-flow figures | Structural only: records in, records out, and the top-level field names of the result. Never the values. |
| Graph reads and writes | Recorded by identity only: the label, the name of the identity key and that key's reference value. |

Related: [lifecycle and audit](/docs/governance/audit-and-lifecycle) describes the stages the waterfall renders and the durable account of the same call, [policy and approvals](/docs/governance/policy-and-approvals) explains what a refusal at `S5` means, and [usage, quota and billing](/docs/administration/usage-quota-and-billing) covers the counts recorded at `S3` and what a `429` in a trace is telling you.