Observability

How it works

1. The gateway opens a root span for every request, before authentication, so refused requests are traced too.
2. It names the trace in the response, in the X-Trace-Id header, whatever the outcome.
3. Each governed tool call nests under the root: one span for the call and one for each lifecycle stage.
4. The gateway exports traces, metrics and logs in OpenTelemetry formats, over OTLP, to a collector you provide.
5. You read the trace in your collector or in the console.

The gateway only exports: it keeps no trace store and serves no trace over the Wexa API.

The console's Observability module reads traces from a telemetry backend through its own observability API, and its Settings page shows that backend's retention windows and sampling. Those reads are not governed calls, so they never appear inside a trace.

curl -si -X POST https://fabric.wexa.ai/v1/query-context \
  -H "Authorization: Bearer $FABRIC_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"query":"MATCH (n) RETURN n LIMIT 1"}' | grep -i '^x-trace-id\|^server-timing'

import { Fabric, WexaError } from '@wexa-fabric/sdk'

const fabric = new Fabric()
try {
  const result = await fabric.queryContext({ query: 'MATCH (n) RETURN n LIMIT 1' })
} catch (e) {
  if (e instanceof WexaError) console.log(e.traceId, e.lifecycleId)
}

from wexa import Fabric, WexaError

fabric = Fabric()
try:
    result = fabric.query_context(query="MATCH (n) RETURN n LIMIT 1")
except WexaError as e:
    print(e.trace_id, e.lifecycle_id)

What you get without configuring anything

These hold whether or not telemetry export is switched on.

Trace context propagates. The gateway installs the W3C trace-context propagator unconditionally. It honours an inbound traceparent, and its outbound calls to the services behind it carry the context onward. A deployment where only some services export still produces connected traces.

Every response names its trace. The gateway sets X-Trace-Id and a Server-Timing entry before the handler runs. The identifier is present on successes, on refusals and on requests that never authenticated. Quote it when reporting a problem.

Read the shape of a trace

One tool call produces a nest of spans, in layers.

The stop reason and stage sit on the parent span, so you do not have to hunt through children to learn why a call failed.

Stage detail includes:

• At S3, the quota counts and ceilings the call saw.
• At S5, the rule, verdict, reasoning and decision identifier.
• At S6, the approval identifier and reason.

Two details of the shape are deliberate.

• S8:execute is opened live, while the work happens, rather than replayed at the end. That is what makes the downstream service call nest inside S8 instead of beside it, so time spent waiting on the graph is visibly inside the execute stage.
• The sequence number orders the stages. Governance stages run in well under a millisecond, so their start timestamps collapse onto the same instant. A viewer that sorts siblings by timestamp alone shows them jumbled.

Debug a slow run

Open the trace by its identifier and read down the waterfall.

1. Compare the root span's duration with lifecycle.<tool>. A large gap means the time went somewhere other than the tool call, usually authentication or a role lookup, which appears as a client span with nothing beneath it.
2. Look at which stage is wide. The governance stages other than S8:execute are sub-millisecond by nature. If one is wide, that is the finding, not a rounding artefact.
3. If S8:execute is the wide one, descend into it. The downstream call nests there, such as the graph read behind query-context, or the agent run behind run-agent. A wide child means a slow service. A narrow child means time spent in the gateway around the call.
4. Check the quota counts on S3. A project near its ceiling shows there, and it explains a run that is slow because its callers are backing off.
5. For a multi-call run, sort by the lifecycle_id, not by the tool. An agent making twenty calls produces twenty traces. The one you want is the one whose identifier the caller reported.

Debug a failing run

The failure reason is on the parent span, so start there rather than at the deepest red child.

• S5:policy failed: a rule refused the call. The rule name and its reasoning are on that stage's span, and the same decision is retrievable at GET /v1/policy-decisions.
• S3:rate-quota failed: a ceiling was hit. The counts on the stage say which bucket.
• S6:approval failed on a call with a resume token: the token was rejected. It was spent, expired, or bound to a different tool, project or user.
• S6 pending: not a failure. The call parked at an approval and the trace ends there legitimately. The rest of the story is in the trace of the resumed call.
• S8:execute failed: the tool or the service behind it failed. The child span carries the error, and its status tells you whether the gateway gave up waiting or the service answered badly.

When there is no trace

If the X-Trace-Id you were given finds nothing in your collector, the likely causes are, in order:

1. Export is off for that deployment.
2. The collector endpoint is not reachable from the gateway. This degrades silently by design, so telemetry can never take the platform down.
3. The trace was not sampled. Head sampling is configurable and samples everything by default, so a deployment that has lowered it is missing traces in proportion.

The console shows the sampling ratio and the retention windows for traces, logs and metrics at /governance/observability/settings.

Limits and behavior

Related: lifecycle and audit describes the stages the waterfall renders and the durable account of the same call, policy and approvals explains what a refusal at S5 means, and usage, quota and billing covers the counts recorded at S3 and what a 429 in a trace is telling you.