Skip to main content

Send data to Zerobus Ingest from Apache NiFi

info

For the endpoints, the token request, and the REST insert format, see Write a client.

Maintained by: the Apache NiFi project, in the Apache NiFi repository.

Apache NiFi moves records through a flow of processors. In this flow, an InvokeHTTP processor posts each FlowFile to the Zerobus Ingest REST endpoint. A bounded retry and a held queue handle failed requests. A StandardOauth2AccessTokenProvider controller service gets the OAuth token from a service principal and refreshes it, so the flow needs no token code. You don't need to run a message bus between NiFi and the lakehouse.

When to use this path​

  • Your data already flows through NiFi, and you want to land it in a Delta table without another hop.
  • Your producer sends small batches or individual records. For high-volume streaming, use a Zerobus Ingest SDK over gRPC instead. The REST endpoint is stateless and governed by a request-rate quota (see Zerobus Ingest quotas).

How it works​

The StandardOauth2AccessTokenProvider requests a token from <workspace-url>/oidc/v1/token with the client credentials grant. The token request carries the authorization_details that Zerobus Ingest needs to authorize a table write. Each FlowFile becomes one POST to /zerobus/v1/tables/<catalog>.<schema>.<table>/insert, where the body is a JSON array of objects. A successful request returns HTTP 200 with an empty JSON object. Zerobus Ingest can deliver a record more than one time when the client retries records that aren't acknowledged, so a retry can duplicate a row.

Requirements​

  • A workspace in a region that supports Zerobus Ingest. See Availability.
  • A service principal with an OAuth secret and the Workspace access entitlement. See Configure authentication.
  • Target tables that meet the Zerobus Ingest table requirements.
  • Apache NiFi 2.7.0 and above, which supports the FORM. dynamic properties that Step 4 uses. The examples use apache/nifi:2.12.0 from the apache/nifi image.
  • Docker, to run the NiFi image in Step 2.

Step 1: Create the target table and grant access​

Replace <catalog>.<schema>.<table> with your table name. Then run the statements in Databricks SQL.

SQL
CREATE TABLE <catalog>.<schema>.<table> (
roll_id STRING,
player STRING,
sides INT,
value INT,
rolled_at TIMESTAMP
) USING DELTA;

Grant the service principal access. Replace <service-principal-uuid> with its application ID:

SQL
GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-uuid>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service-principal-uuid>`;
GRANT MODIFY, SELECT ON TABLE <catalog>.<schema>.<table> TO `<service-principal-uuid>`;

Step 2: Install and start NiFi​

Start NiFi from the Apache image, with a single-user login. The password must be at least 12 characters. Read it into an environment variable so that it doesn't appear in your shell history or in the docker run command line:

Bash
read -rsp 'NiFi password: ' SINGLE_USER_CREDENTIALS_PASSWORD; echo
export SINGLE_USER_CREDENTIALS_PASSWORD

docker run --name nifi -p 127.0.0.1:8443:8443 -d \
-e SINGLE_USER_CREDENTIALS_USERNAME=<username> \
-e SINGLE_USER_CREDENTIALS_PASSWORD \
apache/nifi:2.12.0

Open https://localhost:8443/nifi and sign in with the username and password you set. The port is bound to loopback, so only this host can reach the UI. For production, keep NiFi's conf directory and its repositories on persistent storage, and back them up together. The conf directory holds the flow definition and nifi.properties, which sets the sensitive properties key that encrypts sensitive values such as client_secret. See Upgrade recommendations in the NiFi Administration Guide.

Step 3: Create the parameter context​

In the NiFi UI, create a process group named zerobus-ingest. Then create a parameter context, and assign it to the process group. Add these parameters:

Parameter

Sensitive

Value

client_id

No

The service principal's application ID (<service-principal-uuid>).

client_secret

Yes

The service principal's OAuth secret.

zerobus_token_url

No

<workspace-url>/oidc/v1/token

zerobus_workspace_id

No

Your workspace ID (<workspace-id>).

zerobus_server_endpoint

No

The Zerobus Ingest endpoint for your workspace.

target_table

No

<catalog>.<schema>.<table>

Parameter

Sensitive

Value

client_id

No

The service principal's application ID (<service-principal-uuid>).

client_secret

Yes

The service principal's OAuth secret.

zerobus_token_url

No

<workspace-url>/oidc/v1/token

zerobus_workspace_id

No

Your workspace ID (<workspace-id>).

zerobus_server_endpoint

No

The Zerobus Ingest endpoint for your workspace.

target_table

No

<catalog>.<schema>.<table>

To find your endpoint, see Get your workspace URL and Zerobus Ingest endpoint. The endpoint has this format:

https://<workspace-id>.zerobus.<region>.gcp.databricks.com

Don't mark client_id as sensitive. NiFi doesn't let a non-sensitive property, such as Client ID, reference a sensitive parameter. Because client_id isn't sensitive, an exported parameter context includes it, so treat exports as sensitive.

Step 4: Add the OAuth 2.0 access token provider​

In the process group, open Controller Services and add a StandardOauth2AccessTokenProvider named zerobus-oauth2-token. Set these properties:

Property

Value

Authorization Server URL

#{zerobus_token_url}

Grant Type

Client Credentials

Client ID

#{client_id}

Client Secret

#{client_secret}

Scope

all-apis

Resource

api://databricks/workspaces/#{zerobus_workspace_id}/zerobusDirectWriteApi

Property

Value

Authorization Server URL

#{zerobus_token_url}

Grant Type

Client Credentials

Client ID

#{client_id}

Client Secret

#{client_secret}

Scope

all-apis

Resource

api://databricks/workspaces/#{zerobus_workspace_id}/zerobusDirectWriteApi

Add a dynamic property named FORM.authorization_details with the following value. The value scopes the token to writes on your table. Replace <catalog> and <schema> with your catalog and schema, and <table> with your table name:

JSON
[
{
"type": "unity_catalog_privileges",
"privileges": ["USE CATALOG"],
"object_type": "CATALOG",
"object_full_path": "<catalog>"
},
{
"type": "unity_catalog_privileges",
"privileges": ["USE SCHEMA"],
"object_type": "SCHEMA",
"object_full_path": "<catalog>.<schema>"
},
{
"type": "unity_catalog_privileges",
"privileges": ["SELECT", "MODIFY"],
"object_type": "TABLE",
"object_full_path": "<catalog>.<schema>.<table>",
"operations": ["zerobuswrite"]
}
]

Enable the controller service. Its state should be Enabled.

Step 5: Add the processors and send a record​

Add these processors to the process group, and connect them in this order:

  1. GenerateFlowFile (or the processor that produces your records). Set Data Format to Text, Unique FlowFiles to false, Custom Text to a JSON array of one object, and Run Schedule to 10 sec. NiFi's default of 0 sec runs the processor as often as possible. Stop the processor when your test is done. The following example sends a dice roll:

    JSON
    [{"roll_id":"${UUID()}","player":"ada","sides":6,"value":${random():mod(6):plus(1)},"rolled_at":${now():toNumber():multiply(1000)}}]

    The rolled_at value is epoch microseconds, which Zerobus Ingest reads as a TIMESTAMP. A string timestamp is rejected.

  2. InvokeHTTP. Set these properties:

    Property

    Value

    HTTP Method

    POST

    HTTP URL

    #{zerobus_server_endpoint}/zerobus/v1/tables/#{target_table}/insert

    Request Content-Type

    application/json

    Request OAuth2 Access Token Provider

    zerobus-oauth2-token (the controller service from Step 4)

    Property

    Value

    HTTP Method

    POST

    HTTP URL

    #{zerobus_server_endpoint}/zerobus/v1/tables/#{target_table}/insert

    Request Content-Type

    application/json

    Request OAuth2 Access Token Provider

    zerobus-oauth2-token (the controller service from Step 4)

    Route the relationships as follows:

    • Original: auto-terminate. The request FlowFile isn't needed after a successful response.

    • Response (HTTP 200 to 299): route to a logging processor, or auto-terminate.

    • On each logging processor, auto-terminate its success relationship. A processor is valid and can run only when each of its relationships is connected or auto-terminated.

    • Retry (HTTP 500 to 599), Failure (socket errors), and No Retry (HTTP 400 to 499): on the processor's Relationships tab, select Automatically Retry for these three relationships. Set Number of Retry Attempts to 3 and Retry Back Off Policy to Penalize. The retries wait 30, 60, and 120 seconds.

      Zerobus Ingest documents HTTP 429 as retryable, and most other 4xx responses as not retryable without changing the request (see Zerobus Ingest errors). This example retries every 4xx response, so a 4xx that isn't 429 repeats until the retry limit, then goes to the held queue.

    • After the last retry, route Retry, Failure, and No Retry to the held queue in the next paragraph. Route No Retry to a logging processor too, so you can see the error.

    Add a LogAttribute processor named hold-zerobus-failures, and disable it rather than only stopping it. Starting a process group doesn't start a disabled processor, so the held records stay in their queue. Connect Retry, Failure, and No Retry to it. On each of those connections, set Back Pressure Object Threshold to 1000 and Back Pressure Data Size Threshold to 1 GB. Records that exhaust their retries wait in those connections' queues, with their JSON unchanged. When a queue is full, NiFi stops the upstream processors instead of dropping records, so ingestion pauses until you drain or reroute the queue. Size the thresholds for your failure rate, and watch the queue. To review the held records, right-click a connection and select List Queue.

  3. Start the processors, except hold-zerobus-failures, which stays disabled. A successful insert routes the FlowFile to Response, with HTTP 200 in the invokehttp.status.code attribute.

Query the table to confirm the flow wrote rows. The newest rows show player as ada, sides as 6, and value from 1 to 6:

SQL
SELECT roll_id, player, sides, value, rolled_at FROM <catalog>.<schema>.<table> ORDER BY rolled_at DESC LIMIT 10;

Troubleshooting​

Symptom

Cause

Fix

The token request returns HTTP 400 with invalid_authorization_details

The FORM.authorization_details property is missing or misspelled. NiFi silently drops a property whose name doesn't match.

Check that the dynamic property is named exactly FORM.authorization_details.

The insert returns HTTP 401 with an empty body

InvokeHTTP isn't sending the token, because Request OAuth2 Access Token Provider is empty.

Set Request OAuth2 Access Token Provider to the controller service from Step 4.

The insert returns HTTP 400 with INVALID_PARAMETER_VALUE and "invalid digit found in string"

rolled_at is sent as a string. TIMESTAMP values are epoch microseconds.

Send rolled_at as a number of epoch microseconds. Records already in the held queue still contain the string value, so correct or remove them before you resend them.

Records accumulate in the held queue

Each record still failed after three retries.

Check the error in the log and fix its cause. Records stay in the queue until you remove or reroute them.

The controller service is Invalid with "cannot reference Parameter ... Sensitivity of the parameter does not match"

A non-sensitive property, such as Client ID, references a sensitive parameter, such as a client_id marked sensitive.

Re-create client_id as a non-sensitive parameter. Keep Sensitive set on client_secret, which the sensitive Client Secret property references.

NiFi's UI fails with "Invalid SNI"

The UI was opened by IP address. NiFi answers on the hostname only.

Open the UI by hostname, such as https://localhost:8443/nifi.

Symptom

Cause

Fix

The token request returns HTTP 400 with invalid_authorization_details

The FORM.authorization_details property is missing or misspelled. NiFi silently drops a property whose name doesn't match.

Check that the dynamic property is named exactly FORM.authorization_details.

The insert returns HTTP 401 with an empty body

InvokeHTTP isn't sending the token, because Request OAuth2 Access Token Provider is empty.

Set Request OAuth2 Access Token Provider to the controller service from Step 4.

The insert returns HTTP 400 with INVALID_PARAMETER_VALUE and "invalid digit found in string"

rolled_at is sent as a string. TIMESTAMP values are epoch microseconds.

Send rolled_at as a number of epoch microseconds. Records already in the held queue still contain the string value, so correct or remove them before you resend them.

Records accumulate in the held queue

Each record still failed after three retries.

Check the error in the log and fix its cause. Records stay in the queue until you remove or reroute them.

The controller service is Invalid with "cannot reference Parameter ... Sensitivity of the parameter does not match"

A non-sensitive property, such as Client ID, references a sensitive parameter, such as a client_id marked sensitive.

Re-create client_id as a non-sensitive parameter. Keep Sensitive set on client_secret, which the sensitive Client Secret property references.

NiFi's UI fails with "Invalid SNI"

The UI was opened by IP address. NiFi answers on the hostname only.

Open the UI by hostname, such as https://localhost:8443/nifi.

Additional resources​