GuidesIngest

Connect NATS JetStream

Create and manage a NATS JetStream connector for a RawTree table.

Connect NATS JetStream

Use a NATS connector to read JSON messages from JetStream and write them to a RawTree table. Each destination reads one stream through an existing consumer. Each row contains the message's JSON fields plus nats.subject and nats.timestamp. The connector runs in RawTree Cloud or RawTree BYOC, wherever the connector service is enabled.

Before you begin

  • Sign in as an organization administrator.
  • Create a destination database and table for API requests. The dashboard can create them during setup.
  • Save the cluster ID from GET /v1/clusters, the database name, and the table name.
  • Enable JetStream on your NATS account.
  • Prepare a reachable NATS server URL and its credentials.

By default, RawTree accepts only publicly routable broker addresses. Private endpoints work only in regions that allow them. Authentication does not replace network access. Use TLS with a trusted server certificate. Custom CA certificates, client certificates, and certificate verification overrides are not supported. Keep passwords, tokens, and NKey seeds out of shared files and shell history.

Prepare a consumer

Create the stream and consumer on your NATS server before you create the connector. RawTree does not create them.

For example, configure stream EVENTS to capture subject events.created. Create consumer rawtree-events on that stream with these settings:

SettingValue
Durable namerawtree-events
Consumer modePull, with no delivery subject
Acknowledgement policyExplicit
Filter subjectevents.created
Delivery policyAll, to read retained messages and future messages

Use a dedicated consumer for RawTree. Other applications that pull from the same consumer compete for its messages. The consumer's filter and delivery policy determine which messages RawTree reads. See the NATS consumer documentation for consumer settings.

Give the connector credentials permission to read stream and consumer information, pull messages, and acknowledge them. Set stream retention and consumer delivery limits to cover your expected outages.

Create a connector in the dashboard

  1. Open Connectors in your organization and cluster.
  2. Start a new connector and select NATS JetStream.
  3. Enter a Connection name.
  4. Enter one or more comma-separated URLs in Servers, such as tls://nats.example.com:4222.
  5. Enable TLS encryption for the encrypted connection.
  6. Enable Authentication and select Username and password, Token, or NKey.
  7. Enter the credentials for the selected method.
  8. In the destination step, enter EVENTS as Stream and rawtree-events as Consumer.
  9. Select the destination database and table.
  10. Optional: under Delivery tuning, change Batch size (default 1000) and Batch timeout (seconds) (default 1).
  11. Review the configuration and create the connector.

Add a separate destination for each additional stream and consumer pair. Each destination must use a distinct database and table pair. RawTree checks the broker, stream, and consumer before it saves the connector. NATS has no message preview or stream discovery step.

Create a connector through the API

Send POST /v1/connectors to https://api.rawtree.com with the cluster ID in the X-RawTree-Cluster-Id header. Use a user session or user OAuth token with full_access. Connector management does not accept API keys or workload tokens.

Set the connector type to nats. This example uses an existing analytics.events table:

{
  "name": "nats-events",
  "type": "nats",
  "destinations": [
    {
      "nats_stream": "EVENTS",
      "nats_consumer": "rawtree-events",
      "database": "analytics",
      "table": "events"
    }
  ],
  "settings": {
    "servers": "tls://nats.example.com:4222",
    "tls": { "enabled": true },
    "auth": {
      "strategy": "user_password",
      "user": "rawtree-reader",
      "password": "replace-with-a-secret"
    },
    "batch": {
      "max_events": 1000,
      "timeout_secs": 1
    }
  }
}

The API returns HTTP 201 with the connector, destination IDs, status, revision, and runtime state. It encrypts stored credentials and omits their secret values from responses. Save the connector id for later requests. See the NATS API reference for settings and a response example.

Verify ingestion

Publish a JSON object to the stream's subject through your existing NATS publisher:

{"event_id":"nats-check-1","event_type":"connector_test"}

Open the connector details and check its status and delivered-event metrics. Query the destination table to verify the message:

SELECT event_id, event_type
FROM analytics.events
WHERE event_id = 'nats-check-1'
LIMIT 10;

The result should contain the published event. Duplicates are possible because delivery is at least once.

For API checks, read GET /v1/connectors/{connector_id} with the same X-RawTree-Cluster-Id header. A runtime.state of configured confirms configuration publication, not successful message delivery. Verify rows or delivered-event metrics separately. Use GET /v1/metrics for telemetry, as described in Monitor usage.

Pause, resume, or delete

Send PATCH /v1/connectors/{connector_id} with the same X-RawTree-Cluster-Id header. To pause the connector, use:

{ "status": "paused" }

To resume it, use:

{ "status": "active" }

Omitted settings and credentials remain unchanged. When you update destinations, include existing destination IDs to preserve their identities. The supplied destination list replaces the existing list.

Send DELETE /v1/connectors/{connector_id} to remove the connector. The response contains deleted: true. RawTree leaves the NATS stream, consumer, and destination table in place.

Delivery and retries

NATS uses the same disk buffer as Kafka and Postgres connectors. RawTree acknowledges JetStream messages after their decoded events enter that buffer. The ingestion service then owns delivery and retries retryable write failures. An acknowledgement does not mean the row is already queryable.

A restart can recover buffered events from retained storage. Loss of that storage can lose acknowledged events that remain undelivered. When the buffer blocks, progress acknowledgements extend the consumer's deadline. Malformed messages remain unacknowledged and can delay later messages. RawTree does not automatically move them to a dead-letter stream.

Troubleshoot setup

Broker validation failures return HTTP 422 with one of these error codes:

ErrorAction
nats_unreachableCheck server URLs, DNS, port, and firewall rules.
nats_host_blockedUse a publicly routable broker address, unless the region allows private endpoints.
nats_auth_failedCheck the selected authentication method and credentials.
nats_tls_failedCheck the server certificate, hostname, and certificate chain.
nats_jetstream_unavailableEnable JetStream and check account permissions.
nats_stream_not_foundCheck the stream name and stream information permissions.
nats_consumer_not_foundCheck the consumer name and consumer information permissions.
nats_consumer_invalidUse a durable pull consumer with explicit acknowledgements.

If no rows arrive, check the consumer filter, retained messages, JSON payloads, and pull permissions. Check the connector metrics for write failures and buffer growth.