> Source: https://wexa.ai/docs/context/ingestion

# Ingestion

Ingestion brings a connected source's data into the context graph and the data catalog. Syncing a connector needs `fabric:connector.sync`; a catalog synchronization needs `fabric:catalog.write`. To start it in the console:

1. Open **Data Management**, then **Ingestion**.
2. Select **Start** on a connector's row, or **Start All** at the top. Start triggers an immediate full sync and schedules a continuous polling worker that runs until stopped.
3. Watch the row. A green **Ingesting** badge means a workflow is running. An amber **Status unavailable** badge means the status check is not getting through, and the connector may still be ingesting.
4. Select **Stop** to end the connector's active workflows, or **Stop All** for every connector.

## Limits

| Behavior | Detail |
|---|---|
| Acknowledgement | A call answers when work is accepted, not when it finishes. |
| Repeat passes | Merge onto the same nodes. No duplicates. |
| Where data lands | Under the calling scope's project, taken from your token. |
| Overlapping catalog passes | A second trigger is refused with `409` while one runs. |
| No synchronization configured | `503`. It needs an administrator. |
| Grants | `fabric:catalog.write` to trigger a catalog pass, `fabric:connector.sync` to sync a connector. |

**More details**

**Ingestion** is bringing a connected source's data into Wexa so that it appears in the [context graph](/docs/context/context-graph) and the [data catalog](/docs/context/create-ontology), which is the step that makes a source present after [provisioning a connector](/docs/connectors/overview) made it reachable.

You need a connector already provisioned in the project. Syncing a connector through the API needs the `fabric:connector.sync` grant, and triggering a catalog synchronization needs `fabric:catalog.write`. A project that has provisioned a connector and not ingested it has a connector and an empty graph.

### Start ingestion for a connector

Status refreshes every 15 seconds and node counts every 5 minutes. How often the worker polls depends on your organization's plan. A connector can set its own interval under **Sync frequency** on its card in **Connect**, **Connectors**.

Through the API, [`sync-connector`](/docs/connectors/sync-connector) does what **Sync this connector** does on a connector's card. It answers at once with a `sync_id`.

All three tabs make that same call.

**Sync a connector**

**REST API**

```bash
curl -sS https://fabric.wexa.ai/v1/sync-connector \
  -H "authorization: Bearer $FABRIC_API_KEY" \
  -H "content-type: application/json" \
  -d '{"connector_id":"6ab3608bf19beb4f0ed39e77"}'
```

**TypeScript SDK**

```ts
import { Fabric } from '@wexa-fabric/sdk'

const fabric = new Fabric()
const sync = await fabric.syncConnector({
  connector_id: '6ab3608bf19beb4f0ed39e77',
})
```

**Python SDK**

```python
from wexa import Fabric

fabric = Fabric()
sync = fabric.sync_connector(
    connector_id="6ab3608bf19beb4f0ed39e77",
)
```

To block until the refresh ends, add `wait` on either SDK, which polls `get-connector-sync` for you. `wait` waits only for the refresh of the connector's actions and ontology. It never waits for ingestion to finish, because a native connector cannot report when its data is all in.

### More limits

| Behavior | Detail |
|---|---|
| Governance | A pass runs under a scope, needs the relevant grant, draws on the project's quota and writes to the audit record. |
| Classifications | A classification arriving from a source feeds the restricted set the policy engine enforces. Labelling a column sensitive at the source tightens what agents may do with it in Wexa. |

See [policy and approvals](/docs/governance/policy-and-approvals) for how a decision is made.

### How ingestion works

1. **You start a pass.** In the console you start a connector. Through the API you call `sync-connector`, or trigger a catalog synchronization.
2. **Wexa accepts it.** The call answers as soon as the work is accepted, and the work continues after the response has been sent.
3. **The pass crosses to the source.** How long it takes depends on that system, not on Wexa.
4. **Two kinds of record land under your project.** Metadata becomes catalog assets, and content becomes nodes and relationships in the context graph.
5. **You ask whether it finished.** Read the status of the connector's ingestion or the synchronization statistics. The sections below show each one.

### What one pass produces

**Metadata becomes catalog assets.** The structure of the source lands as catalog assets and the relationships between them. That means its databases, tables, columns, dashboards and models, with the lineage between them and any classifications and glossary terms attached.

**Content becomes context.** The entities a source is actually about, such as messages, tickets, records and people, land as nodes and relationships in the context graph, typed according to the project's vocabulary.

Both land under the calling scope's project. Neither is a copy you keep in step by hand. Ingestion is repeatable, and repeated passes merge onto the same nodes instead of accumulating duplicates.

### Why it is asynchronous

Ingestion crosses a network to a system Wexa does not control. A source with a hundred tables and one with a hundred thousand differ by orders of magnitude. Holding an HTTP request open for the second would leave you with a timeout as your only signal.

So an ingestion call answers when the work is *accepted*. That shifts a burden onto you, because you now ask whether it finished.

### Follow the ingestion of a connector

In the console, open **Data Management**, then **Ingestion** (`/data-management/ingestion`). Each connector's row shows its state and node counts.

Through the API, [`get-connector-ingestion`](/docs/connectors/get-connector-ingestion) shows what the console's connector page shows: the pipeline's stages, its state and the last run's node and edge counts. A failed run carries its error.

**Read a connector's ingestion**

**REST API**

```bash
curl -sS https://fabric.wexa.ai/v1/get-connector-ingestion \
  -H "authorization: Bearer $FABRIC_API_KEY" \
  -H "content-type: application/json" \
  -d '{"connector_id":"6b1a3d5e7f9c0b2a4d6e8f01"}'
```

**TypeScript SDK**

```ts
import { Fabric } from '@wexa-fabric/sdk'

const fabric = new Fabric()
const ingestion = await fabric.getConnectorIngestion({
  connector_id: '6b1a3d5e7f9c0b2a4d6e8f01',
})
```

**Python SDK**

```python
from wexa import Fabric

fabric = Fabric()
ingestion = fabric.get_connector_ingestion(connector_id="6b1a3d5e7f9c0b2a4d6e8f01")
```

To follow the refresh itself, pass the `sync_id` to [`get-connector-sync`](/docs/connectors/get-connector-sync).

### Trigger and check a catalog synchronization

A catalog synchronization is triggered on demand and reports back separately. These routes have no SDK method, so you call them over REST.

**Trigger a catalog synchronization**

**REST API**

```bash
curl -sS -X POST https://fabric.wexa.ai/v1/catalog/sync \
  -H "authorization: Bearer $FABRIC_API_KEY"

curl -sS https://fabric.wexa.ai/v1/catalog/sync/stats \
  -H "authorization: Bearer $FABRIC_API_KEY"
```

The trigger returns immediately and leaves the pass running behind it. Reading the statistics afterwards tells you what happened:

- how many tables and columns were seen;
- how many assets were created and how many updated;
- how many lineage edges were recorded at each grain;
- how many edges were rejected.

Three answers to the trigger mean different things:

| Answer | Meaning |
|---|---|
| Accepted (`202`) | The pass is running. Read the statistics for the result. |
| Already running (`409`) | A pass is in flight, and a second is refused rather than started alongside it. Watch the statistics for the pass already going. |
| Not configured (`503`) | No synchronization exists for this organization, so there is nothing to run. It needs an administrator, not patience. |

A deployment can also be notified when a source changes, rather than waiting for the next scheduled pass. The cadence of scheduled passes is a deployment setting. See the [catalog reference](/docs/connectors/catalog).

### Follow a code-sync ingestion job

The [code sync](/docs/context/code-sync) path has an explicit job model, the clearest example of the shape. A bulk ingestion is submitted and the response carries a job identifier. A separate call reads that job's status by identifier, and you poll it until the job reports that it is finished.

Everything about the request is pinned to your scope on the way through. The organization and project the data lands in are taken from your token and overwritten on the payload. A client cannot ingest into a project it was not granted, whatever it sends.

### Read live instead of ingesting

Ingestion is not the only way to read a source. [`connector-read`](/docs/context/connector-read) runs a read-only query against the source live and copies nothing into Wexa. For a large warehouse or database that is usually the better answer. You get current rows without a second copy, and without paying to ingest data you will read once.

The division that works in practice is **ingest the structure, read the rows live**. The catalog gives you the shape of the source, which is small, slow-changing and worth having locally so agents can reason about it without a round trip. The rows are large and fast-changing, so fetch them when they are wanted. See [connectors](/docs/connectors/overview).