Ingest data

This guide uses the insert API in RawTree Cloud or RawTree BYOC. The service handles credentials, table creation, and preprocessing. RawTree parses and stores the resulting rows.

RawTree accepts unstructured data without a schema step. The service can create a table on first insert when your credential permits it.

Inline data

rtree insert --table events \
  --data '{"action":"signup","user_id":1}'

Use an array to insert multiple rows:

rtree insert --table events \
  --data '[{"action":"signup","user_id":1},{"action":"purchase","user_id":1,"amount":42}]'

Files

rtree insert --table events --file ./events.jsonl

JSONL is useful for large event streams because each line is one event.

Public URLs

rtree insert --table events --url https://example.com/events.jsonl

URL ingest waits for the import to finish. The API returns JSON with inserted (or null if the row count is unavailable). The X-ClickHouse-Query-Id response header contains the supplied ID, or the ID generated by RawTree when omitted. Errors return a normal HTTP error response.

Built-in transforms

Use transforms when the source format needs to be flattened before insert. A transform is a built-in preprocessor for JSON body inserts: the service reads the source JSON, emits one or more flat rows, then inserts those rows into the table.

rtree insert --table traces \
  --data '{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"api"}}]},"scopeSpans":[{"spans":[{"name":"GET /health","spanId":"abc"}]}]}' \
  --transform otlp-traces

The API accepts the same transform through the transform query parameter on a JSON body insert:

curl -X POST "https://api.rawtree.com/v1/tables/traces?transform=otlp-traces" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"resource":{"attributes":[{"key":"service.name","value":{"stringValue":"api"}}]},"scopeSpans":[{"spans":[{"name":"GET /health","spanId":"abc"}]}]}'

OpenTelemetry SDKs and collectors can also send native OTLP to the service without a transform query parameter. See the OpenTelemetry guide for endpoints, SDK environment variables, Collector config, and a smoke test.

Transforms are not supported with URL inserts. If you use ?url=, transform the data before hosting it.

See Transforms for supported input shapes and emitted rows.

API insert

curl -X POST "https://api.rawtree.com/v1/tables/events" \
  -H "Authorization: Bearer $API_KEY" \
  -H "Content-Type: application/json" \
  -d '[{"action":"signup","user_id":1}]'

Verify the insert

rtree table describe events
rtree query "SELECT count() FROM events"

Upload failures and retries

Raw JSON inserts without a transform stream to the database. The service prepares a missing table before it forwards the upload. It does not keep a complete copy for server-side replay. Gzip uploads follow the same rules. The body-size limit applies to decoded data.

Keep the full payload until the request succeeds. For temporary delivery failures, including a table that is not visible on the destination replica, the API returns 503 with Retry-After. Retry the full upload after that delay. Use backoff if failures continue. Correct invalid JSON, invalid gzip, oversized payloads, and permission errors before you retry.

A failed or interrupted upload can leave a table or some inserted rows. The database can also accept data before a timeout or lost response. A retry can therefore insert duplicate rows. A failed response does not guarantee rollback.