Send data to Zerobus Ingest from Apache NiFi
For the endpoints, the token request, and the REST insert format, see Write a client.
Maintained by: the Apache NiFi project, in the Apache NiFi repository.
Apache NiFi moves records through a flow of processors. In this flow, an InvokeHTTP processor posts each FlowFile to the Zerobus Ingest REST endpoint. A bounded retry and a held queue handle failed requests. A StandardOauth2AccessTokenProvider controller service gets the OAuth token from a service principal and refreshes it, so the flow needs no token code. You don't need to run a message bus between NiFi and the lakehouse.
When to use this path
- Your data already flows through NiFi, and you want to land it in a Delta table without another hop.
- Your producer sends small batches or individual records. For high-volume streaming, use a Zerobus Ingest SDK over gRPC instead. The REST endpoint is stateless and governed by a request-rate quota (see Zerobus Ingest quotas).
How it works
The StandardOauth2AccessTokenProvider requests a token from <workspace-url>/oidc/v1/token with the client credentials grant. The token request carries the authorization_details that Zerobus Ingest needs to authorize a table write. Each FlowFile becomes one POST to /zerobus/v1/tables/<catalog>.<schema>.<table>/insert, where the body is a JSON array of objects. A successful request returns HTTP 200 with an empty JSON object. Zerobus Ingest can deliver a record more than one time when the client retries records that aren't acknowledged, so a retry can duplicate a row.
Requirements
- A workspace in a region that supports Zerobus Ingest. See Availability.
- A service principal with an OAuth secret and the Workspace access entitlement. See Configure authentication.
- Target tables that meet the Zerobus Ingest table requirements.
- Apache NiFi 2.7.0 and above, which supports the
FORM.dynamic properties that Step 4 uses. The examples useapache/nifi:2.12.0from theapache/nifiimage. - Docker, to run the NiFi image in Step 2.
Step 1: Create the target table and grant access
Replace <catalog>.<schema>.<table> with your table name. Then run the statements in Databricks SQL.
CREATE TABLE <catalog>.<schema>.<table> (
roll_id STRING,
player STRING,
sides INT,
value INT,
rolled_at TIMESTAMP
) USING DELTA;
Grant the service principal access. Replace <service-principal-uuid> with its application ID:
GRANT USE CATALOG ON CATALOG <catalog> TO `<service-principal-uuid>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<service-principal-uuid>`;
GRANT MODIFY, SELECT ON TABLE <catalog>.<schema>.<table> TO `<service-principal-uuid>`;
Step 2: Install and start NiFi
Start NiFi from the Apache image, with a single-user login. The password must be at least 12 characters. Read it into an environment variable so that it doesn't appear in your shell history or in the docker run command line:
read -rsp 'NiFi password: ' SINGLE_USER_CREDENTIALS_PASSWORD; echo
export SINGLE_USER_CREDENTIALS_PASSWORD
docker run --name nifi -p 127.0.0.1:8443:8443 -d \
-e SINGLE_USER_CREDENTIALS_USERNAME=<username> \
-e SINGLE_USER_CREDENTIALS_PASSWORD \
apache/nifi:2.12.0
Open https://localhost:8443/nifi and sign in with the username and password you set. The port is bound to loopback, so only this host can reach the UI. For production, keep NiFi's conf directory and its repositories on persistent storage, and back them up together. The conf directory holds the flow definition and nifi.properties, which sets the sensitive properties key that encrypts sensitive values such as client_secret. See Upgrade recommendations in the NiFi Administration Guide.
Step 3: Create the parameter context
In the NiFi UI, create a process group named zerobus-ingest. Then create a parameter context, and assign it to the process group. Add these parameters:
Parameter | Sensitive | Value |
|---|---|---|
| No | The service principal's application ID ( |
| Yes | The service principal's OAuth secret. |
| No |
|
| No | Your workspace ID ( |
| No | The Zerobus Ingest endpoint for your workspace. |
| No |
|
To find your endpoint, see Get your workspace URL and Zerobus Ingest endpoint. The endpoint has this format:
https://<workspace-id>.zerobus.<region>.cloud.databricks.com
Don't mark client_id as sensitive. NiFi doesn't let a non-sensitive property, such as Client ID, reference a sensitive parameter. Because client_id isn't sensitive, an exported parameter context includes it, so treat exports as sensitive.
Step 4: Add the OAuth 2.0 access token provider
In the process group, open Controller Services and add a StandardOauth2AccessTokenProvider named zerobus-oauth2-token. Set these properties:
Property | Value |
|---|---|
Authorization Server URL |
|
Grant Type |
|
Client ID |
|
Client Secret |
|
Scope |
|
Resource |
|
Add a dynamic property named FORM.authorization_details with the following value. The value scopes the token to writes on your table. Replace <catalog> and <schema> with your catalog and schema, and <table> with your table name:
[
{
"type": "unity_catalog_privileges",
"privileges": ["USE CATALOG"],
"object_type": "CATALOG",
"object_full_path": "<catalog>"
},
{
"type": "unity_catalog_privileges",
"privileges": ["USE SCHEMA"],
"object_type": "SCHEMA",
"object_full_path": "<catalog>.<schema>"
},
{
"type": "unity_catalog_privileges",
"privileges": ["SELECT", "MODIFY"],
"object_type": "TABLE",
"object_full_path": "<catalog>.<schema>.<table>",
"operations": ["zerobuswrite"]
}
]
Enable the controller service. Its state should be Enabled.
Step 5: Add the processors and send a record
Add these processors to the process group, and connect them in this order:
-
GenerateFlowFile (or the processor that produces your records). Set Data Format to
Text, Unique FlowFiles tofalse, Custom Text to a JSON array of one object, and Run Schedule to10 sec. NiFi's default of0 secruns the processor as often as possible. Stop the processor when your test is done. The following example sends a dice roll:JSON[{"roll_id":"${UUID()}","player":"ada","sides":6,"value":${random():mod(6):plus(1)},"rolled_at":${now():toNumber():multiply(1000)}}]The
rolled_atvalue is epoch microseconds, which Zerobus Ingest reads as aTIMESTAMP. A string timestamp is rejected. -
InvokeHTTP. Set these properties:
Property
Value
HTTP Method
POSTHTTP URL
#{zerobus_server_endpoint}/zerobus/v1/tables/#{target_table}/insertRequest Content-Type
application/jsonRequest OAuth2 Access Token Provider
zerobus-oauth2-token(the controller service from Step 4)Route the relationships as follows:
-
Original: auto-terminate. The request FlowFile isn't needed after a successful response.
-
Response (HTTP 200 to 299): route to a logging processor, or auto-terminate.
-
On each logging processor, auto-terminate its success relationship. A processor is valid and can run only when each of its relationships is connected or auto-terminated.
-
Retry (HTTP 500 to 599), Failure (socket errors), and No Retry (HTTP 400 to 499): on the processor's Relationships tab, select Automatically Retry for these three relationships. Set Number of Retry Attempts to
3and Retry Back Off Policy to Penalize. The retries wait 30, 60, and 120 seconds.Zerobus Ingest documents HTTP 429 as retryable, and most other 4xx responses as not retryable without changing the request (see Zerobus Ingest errors). This example retries every 4xx response, so a 4xx that isn't 429 repeats until the retry limit, then goes to the held queue.
-
After the last retry, route Retry, Failure, and No Retry to the held queue in the next paragraph. Route No Retry to a logging processor too, so you can see the error.
Add a LogAttribute processor named
hold-zerobus-failures, and disable it rather than only stopping it. Starting a process group doesn't start a disabled processor, so the held records stay in their queue. Connect Retry, Failure, and No Retry to it. On each of those connections, set Back Pressure Object Threshold to1000and Back Pressure Data Size Threshold to1 GB. Records that exhaust their retries wait in those connections' queues, with their JSON unchanged. When a queue is full, NiFi stops the upstream processors instead of dropping records, so ingestion pauses until you drain or reroute the queue. Size the thresholds for your failure rate, and watch the queue. To review the held records, right-click a connection and select List Queue. -
-
Start the processors, except
hold-zerobus-failures, which stays disabled. A successful insert routes the FlowFile to Response, with HTTP 200 in theinvokehttp.status.codeattribute.
Query the table to confirm the flow wrote rows. The newest rows show player as ada, sides as 6, and value from 1 to 6:
SELECT roll_id, player, sides, value, rolled_at FROM <catalog>.<schema>.<table> ORDER BY rolled_at DESC LIMIT 10;
Troubleshooting
Symptom | Cause | Fix |
|---|---|---|
The token request returns HTTP 400 with | The | Check that the dynamic property is named exactly |
The insert returns HTTP 401 with an empty body | InvokeHTTP isn't sending the token, because Request OAuth2 Access Token Provider is empty. | Set Request OAuth2 Access Token Provider to the controller service from Step 4. |
The insert returns HTTP 400 with |
| Send |
Records accumulate in the held queue | Each record still failed after three retries. | Check the error in the log and fix its cause. Records stay in the queue until you remove or reroute them. |
The controller service is Invalid with "cannot reference Parameter ... Sensitivity of the parameter does not match" | A non-sensitive property, such as Client ID, references a sensitive parameter, such as a | Re-create |
NiFi's UI fails with "Invalid SNI" | The UI was opened by IP address. NiFi answers on the hostname only. | Open the UI by hostname, such as |
Additional resources
- Write a client: the REST and SDK examples, and the token request in full.
- Zerobus Ingest errors: the error codes Zerobus Ingest returns.
- Zerobus Ingest quotas: the request-rate quota for REST ingestion.
- Apache NiFi documentation: processors, parameter contexts, controller services, and the OAuth 2.0 token provider.