Ingestion model
Understand table creation, evolving data, insert results, and retries.
Ingestion model
Ingestion turns source records into rows in a RawTree table. You select a cluster, database, and destination table. RawTree accepts JSON without an upfront column definition, while permissions still control where you can write.
From a payload to a table
RawTree parses and stores rows. RawTree Cloud and RawTree BYOC supply the insert API, source integrations, and table-creation orchestration described here. SQL ingestion and the insert API have separate request contracts.
A normal JSON insert sends one object or a collection of objects. Each object becomes a row. The service creates a missing table when the caller has permission to use automatic table creation. A key restricted to database roles can insert into existing tables, but it does not receive automatic table-creation rights.
A later insert can introduce fields that earlier rows do not contain. Existing
rows do not need a rewrite to acquire those fields. A known field absent from
an earlier row returns NULL when queried. Values at the same path can have
different types. See Raw tables and
Dynamic fields.
A table's schema describes the fields RawTree can expose. It is not a promise that every row contains each field or uses the same type.
Ingestion paths
| Path | What happens |
|---|---|
| JSON body insert | The service forwards ordinary JSON to RawTree for parsing and insertion. |
| URL import | The database reads a public source URL. The request waits for the import to finish. |
| Transform | A built-in preprocessor extracts or reshapes records before insertion. |
| App endpoint | A supported protocol, such as OpenTelemetry, maps incoming data to table rows. |
| Connector | A managed process reads an external source and delivers records to a configured table. |
These paths share tables, but their response and retry contracts differ. Transforms do not apply to URL imports. Native OpenTelemetry exports can report partial success, including the number of rejected records. See Apps, connectors, and transforms.
What an insert result means
For ordinary JSON inserts, the database reports summary counters such as
written_rows. The compatibility field inserted contains the numeric row
count, or null when that count is unavailable. A reported count can describe
a deduplicated request without adding new stored rows.
The response describes that insert request. It does not report completion of every replica or downstream workflow. Queries, connector delivery, and workflow runs have their own outcomes.
An unsuccessful upload can still create a table or write some rows. A timeout or lost response can also occur after the database accepts data. Failure does not imply rollback.
Retries and duplicate rows
Ordinary JSON uploads stream to the database. The API does not retain a complete payload for automatic replay. The sender owns the payload and retry decision. Temporary failures can include a retry delay. Invalid data and permission errors need correction before another attempt.
A retry can write the same records again. Ordinary JSON body inserts disable
insert deduplication by default. The API exposes deduplicate_insert and
insert_deduplication_token for deliberate retry handling. A stable token must
identify the same intended insert, not unrelated batches.
Insert deduplication is an engine mechanism with configuration and retention limits. It does not establish permanent row-level uniqueness or make a series of requests atomic. Applications that require unique business events need an explicit event identity and duplicate-handling policy.
Batches and visibility
Batch size affects request overhead, memory use, and the amount of data that needs a retry. The request-body limit bounds each upload. For compressed uploads, the limit applies after decompression.
Replication and background work can continue independently of a client request. A table that is not yet available on the destination replica can produce a temporary insert failure. See Storage and compute.