Databricks Voices content is authored and published by Databricks employees that want to share their own individual learnings and Databricks knowledge. This content supplements the official Databricks product documentation found in other sections of this site.

Stream syslog into Lakewatch with syslog-ng and Zerobus Ingest

Alysson Souza · Product Specialist

Syslog is still the common denominator for security telemetry. Firewalls, switches, load balancers, Linux hosts, and a long tail of appliances can all send it, and many of them can send nothing else. Getting that data into a security information and event management (SIEM) system usually means running a message bus, landing files in object storage, or maintaining a custom pipeline.

This page shows a simpler pattern. syslog-ng receives syslog from your endpoints and writes it straight into a Unity Catalog table through the Zerobus Ingest REST API. A Lakewatch datasource then reads that table and parses it into a governed table with observables, ready for detections. There's no Kafka, no file landing zone, and no extra pipeline to operate.

How it works

syslog-ng sends syslog from your endpoints to Zerobus Ingest, which writes it to a raw table that a Lakewatch datasource parses for detections

syslog-ng parses each message and sends it to Zerobus Ingest as JSON, with one field per table column.
Zerobus Ingest writes the records to a raw Delta table in Unity Catalog.
A Lakewatch datasource reads that table with a parser, which converts the field types, sets the event time, and extracts observables such as the hostname.
The parser writes the result to a parsed table that detection rules query.

The raw table uses the same field names as the built-in Fluentd Syslog (RFC 3164) parser, so the parser works without changes. This is also why the pattern uses the Zerobus Ingest REST API rather than its OpenTelemetry endpoint. OpenTelemetry tables have a fixed schema with the syslog fields nested under attributes, while the REST API writes to a table with the columns you define.

Requirements

A Databricks workspace with Lakewatch enabled. Lakewatch is in Beta, and workspace admins enable it from the Previews page.
A Unity Catalog catalog and schema that Lakewatch uses as a datasource folder.
A SQL warehouse. Lakewatch uses it to validate the datasource when it's created or updated.
A service principal with an OAuth secret. syslog-ng authenticates as this principal.
syslog-ng 4.11 and above with the http and cloud-auth modules. cloud-auth handles the OAuth token request and renewal.
Outbound HTTPS (443) from syslog-ng to your workspace and to its Zerobus Ingest endpoint. For the endpoint format on each cloud, see Get your workspace URL and Zerobus Ingest endpoint.

Step 1: Create the raw table and grant access

Zerobus Ingest doesn't create or change tables, and it rejects any field that isn't a column. Create the table first, and keep syslog-ng's output matching it exactly. The columns are the fields the built-in RFC 3164 parser reads, plus received_at for clustering.

To create the raw table, run the following:

SQL
CREATE TABLE <catalog>.<schema>.syslog_raw (
  host        STRING,    -- sending device hostname (parser: required)
  ident       STRING,    -- program / app name (parser: required)
  pid         BIGINT,    -- process ID, when numeric
  pri         SMALLINT,  -- syslog priority (facility * 8 + severity)
  tag         STRING,    -- e.g. syslog.auth.warning (parser: required)
  message     STRING,    -- free-text message (parser: required)
  time        STRING,    -- 'yyyy-M-d HH:mm:ss xx', converted by the parser
  received_at TIMESTAMP  -- when syslog-ng received the event
) CLUSTER BY (received_at);

The service principal needs the following grants. They match the privileges syslog-ng requests in its token in Step 2. To grant them, run the following:

SQL
GRANT USE CATALOG ON CATALOG <catalog> TO `<sp-application-id>`;
GRANT USE SCHEMA ON SCHEMA <catalog>.<schema> TO `<sp-application-id>`;
GRANT SELECT, MODIFY ON TABLE <catalog>.<schema>.syslog_raw TO `<sp-application-id>`;

Step 2: Configure syslog-ng

The configuration has three parts: a source for your endpoints, a rewrite for the two fields that need logic, and an http() destination that posts batches to Zerobus Ingest. Set your values one time in the @define block at the top. Set ZEROBUS_SP_SECRET in syslog-ng's environment so the secret stays out of the file.

The following is the complete syslog-ng configuration:

Text
@version: 4.11
@include "scl.conf"

# Hostname only, for example dbc-a1b2c3d4-e5f6.cloud.databricks.com
@define workspace_host  "<workspace-hostname>"
@define workspace_id    "<workspace-id>"
# Zerobus endpoint for your cloud, for example
# <workspace-id>.zerobus.<region>.cloud.databricks.com on AWS
@define zerobus_host    "<zerobus-endpoint>"
@define client_id       "<sp-application-id>"
@define catalog         "<catalog>"
@define schema          "<schema>"
@define table           "syslog_raw"

options {
  # Use the device's own hostname, not the collector's view of it
  keep-hostname(yes);
  # A failed type cast drops that field, not the whole message
  on-error("drop-property");
};

source s_endpoints {
  network(transport("udp") port(514));
  network(transport("tcp") port(514));
  # For TLS, add:
  # syslog(transport("tls") port(6514)
  #        tls(key-file("...") cert-file("...")));
};

rewrite r_zerobus {
  # Event time with the SENDER's UTC offset, in the format the parser expects
  set("${S_YEAR}-${S_MONTH}-${S_DAY} ${S_HOUR}:${S_MIN}:${S_SEC}"
      value("zb.time"));
  set("${zb.time} $(substr ${S_TZOFFSET} 0 3)$(substr ${S_TZOFFSET} 4 2)"
      value("zb.time"));
  # Only pass numeric PIDs: one bad value rejects the whole batch
  set("${PID}" value("zb.pid") condition(match("^[0-9]+$" value("PID"))));
};

destination d_zerobus {
  http(
    url("https://`zerobus_host`/zerobus/v1/tables/`catalog`.`schema`.`table`/insert")
    method("POST")
    headers("Content-Type: application/json")

    cloud-auth(oauth2(
      client_id("`client_id`")
      client_secret("`ZEROBUS_SP_SECRET`")    # read from the environment
      token_url("https://`workspace_host`/oidc/v1/token")
      scope("all-apis")
      # Required: a plain all-apis token gets 401 "Invalid token audience"
      resource("api://databricks/workspaces/`workspace_id`/zerobusDirectWriteApi")
      authorization_details('[
        {"type": "unity_catalog_privileges", "object_type": "CATALOG",
         "object_full_path": "`catalog`",
         "privileges": ["USE CATALOG"]},
        {"type": "unity_catalog_privileges", "object_type": "SCHEMA",
         "object_full_path": "`catalog`.`schema`",
         "privileges": ["USE SCHEMA"]},
        {"type": "unity_catalog_privileges", "object_type": "TABLE",
         "object_full_path": "`catalog`.`schema`.`table`",
         "privileges": ["SELECT", "MODIFY"]}
      ]')
      auth_method(basic)
    ))

    # tag is required by the parser. received_at is a TIMESTAMP column,
    # sent as integer epoch microseconds.
    body("$(format-json
      host=$HOST ident=$PROGRAM pid=${zb.pid} pri=int($PRI)
      tag=syslog.${FACILITY}.${LEVEL} message=$MESSAGE time=${zb.time}
      received_at=int64($(* ${R_UNIXTIME} 1000000)))")
    # Send each batch as a JSON array
    body-prefix("[") delimiter(",") body-suffix("]")

    # Batches of up to 4 MiB, well under the 10 MiB request limit
    batch-lines(500) batch-bytes(4194304) batch-timeout(5000)
    # A 1 GiB on-disk buffer keeps events through outages
    disk-buffer(reliable(yes) capacity-bytes(1073741824))
  );
};

log { source(s_endpoints); rewrite(r_zerobus); destination(d_zerobus); };

Each record syslog-ng sends looks like the following:

JSON
{
  "host": "fw01",
  "ident": "sshd",
  "pid": "4243",
  "pri": 36,
  "tag": "syslog.auth.warning",
  "message": "Failed password for invalid user admin from 203.0.113.9 port 40022 ssh2",
  "time": "2026-11-09 15:59:01 +1100",
  "received_at": 1762664341000000
}

Step 3: Verify ingestion

To check the configuration and restart syslog-ng to load it, run the following:
sh
```
syslog-ng --syntax-only
sudo systemctl restart syslog-ng
```
Send a test message to the collector from any endpoint. On Linux, the standard logger utility can do this. Include --rfc3164, because logger sends RFC 5424 by default and network() expects RFC 3164. Run the following:
sh
```
logger --rfc3164 -n <collector-host> -P 514 -t zbtest "Zerobus pipeline test"
```

After a minute or so, confirm the message is in the raw table by running the following query:

SQL
SELECT received_at, host, ident, tag, message
FROM <catalog>.<schema>.syslog_raw
WHERE message LIKE '%Zerobus pipeline test%';

If the message doesn't arrive, run syslog-ng-ctl stats | grep d_zerobus on the collector. A rising queued or dropped count means Zerobus Ingest is rejecting the requests. syslog-ng's own log (for example, journalctl -u syslog-ng) shows the HTTP status and error message.

Step 4: Create the Lakewatch datasource

The built-in Fluentd Syslog (RFC 3164) parser (version 1) reads the table from Step 1 as is. Its definition is the following:

YAML
description: 'RFC 3164 syslog events collected and structured by Fluentd.'
fields:
  - name: pri
    type: SMALLINT
    comment: Priority is calculated by (Facility * 8 + Severity). The lower this value,
      the higher importance of the log message.
  - name: host
    type: STRING
    required: true
    observables:
      - hostname
    comment: Hostname identifies the machine that originally sent the syslog message.
  - name: ident
    type: STRING
    required: true
    comment: Appname identifies the device or application that originated the syslog
      message.
  - name: pid
    type: BIGINT
    comment: ProcID is often the process ID, but can be any value used to enable log
      analyzers to detect discontinuities in syslog reporting.
  - name: message
    type: STRING
    required: true
    comment: Message contains free-form text that provides information about the event.
  - name: tag
    type: STRING
    required: true
    comment: Tag of the syslog message
time:
  coalesce:
    - path: time
      time_formats:
        - yyyy-M-d HH:mm:ss xx

A row is parsed only when all required fields are present. tag is a Fluentd routing tag that syslog-ng doesn't produce, which is why the Step 2 configuration builds it. For the full parser syntax, see Parser reference.

Create the datasource in the UI

For details on this ingestion method, see Existing table.

In Lakewatch, open Datasources and create a new datasource.
For the ingestion method, choose Existing table and select <catalog>.<schema>.syslog_raw. Choose whether to include rows that already exist.
Add the Fluentd Syslog (RFC 3164) parser.
Set the schedule (see schedule versus cost), then create the datasource.

Create the datasource with the CLI

To look up the built-in parser's ID, run the following:

sh
databricks api get /api/2.0/lakewatch/builtin-parsers \
  | jq -r '.. | objects
      | select(.parser_name? == "fluentd_syslog3164") | .parser_id'

Save the following request body as syslog-datasource.json:

JSON
{
  "catalog_name": "<catalog>",
  "schema_name": "<schema>",
  "datasource_name": "syslog",
  "warehouse_id": "<warehouse-id>",
  "schedule": { "at_least_every": "1h" },
  "existing_table": {
    "catalog": "<catalog>",
    "schema": "<schema>",
    "name": "syslog_raw",
    "include_existing_rows": true
  },
  "parsers": [
    {
      "id": "<parser-id>",
      "version": "1",
      "target_table": {
        "catalog_name": "<catalog>",
        "schema_name": "<schema>",
        "table_name": "fluentd_syslog3164_syslog"
      }
    }
  ]
}

To create the datasource, run the following:

sh
databricks lakewatch create-datasource --json @syslog-datasource.json

The first run starts within a few minutes and creates the parsed table. To run it again without waiting for the schedule, run the datasource's ingestion job. Use the name (datasources/<id>) that create-datasource returns, and run the following:

sh
JOB_ID=$(databricks lakewatch get-datasource datasources/<id> -o json \
  | jq -r .status.ingestion_job.job_id)
databricks jobs run-now $JOB_ID

Step 5: Validate parsing

After the first run, check that events are parsed with the right time and observables by running the following query:

SQL
SELECT time, host, ident, tag, lw_observables.hostnames, message
FROM <catalog>.<schema>.fluentd_syslog3164_syslog
ORDER BY time DESC
LIMIT 20;

time should match when each event happened on the device, and hostnames should contain the sending host.

Only rows that match the parser reach the parsed table. To find raw rows with no parsed match, run the following query:

SQL
-- Raw rows that never reached the parsed table
SELECT r.*
FROM <catalog>.<schema>.syslog_raw r
LEFT ANTI JOIN <catalog>.<schema>.fluentd_syslog3164_syslog p
  ON r.message = p.message AND r.host = p.host;

Rows that arrived after the last run appear here until the next run. An older row usually has an empty required field, typically ident or tag.

Use a custom parser for richer fields

With the built-in parser, message stays free text. Every endpoint is searchable by host and program, and detections can match on message content with SQL (LIKE, SEARCH).

For first-class fields such as src_ip, user, or action, extract them in syslog-ng before sending:

app-parser() for supported sources, including Cisco, Check Point, Fortigate, Juniper, Palo Alto, iptables, and sudo
kv-parser() for key=value messages
csv-parser() and regexp-parser() for everything else

Then, for each source type:

Create a raw table with the extracted fields as columns.
Add a destination like the one in Step 2 for that table, with the token scoped to it and the fields in the format-json body named exactly like the columns.
Create a datasource on the new table with a custom parser that types the fields and adds observables, for example ip on src_ip and username on user. You can clone the built-in parser or write your own.
Validate it with the Step 5 queries.

Operational tips

Schedule versus cost: A frequent schedule keeps the parsed table fresh for detections but runs compute more often. Zerobus Ingest writes to the raw table continuously, independent of the datasource schedule.
Filter noise at the collector: Load balancer health checks and similar probes can make up a large share of syslog volume. Drop them with a syslog-ng filter before they're ingested.
Index message for search at scale: Investigations often search the free-text message column. On large tables, a full-text search index lets those searches skip most files. Databricks reports speed-ups of 100x or more on very large tables. Full-text search indexes are in Beta, and a workspace admin enables them from the Previews page. Create the index on the parsed table, with the catalog and schema in the index name, then use search() or isearch() in place of LIKE:
SQL
```
CREATE SEARCH INDEX <catalog>.<schema>.syslog_message_idx
  ON <catalog>.<schema>.fluentd_syslog3164_syslog (message);

SELECT time, host, message
FROM <catalog>.<schema>.fluentd_syslog3164_syslog
WHERE isearch(message, 'failed password');
```
Predictive optimization keeps the index up to date as new rows arrive. A small table has too few files to skip, so the benefit shows as the table grows.
Stay within the REST quota: Zerobus Ingest REST allows 10,000 requests per second by default. syslog-ng batches up to 500 records per request (batch-lines in the Step 2 configuration), so this covers high event volumes. For very high, sustained volumes, use the gRPC SDKs instead of REST. See Zerobus Ingest quotas.
Expect occasional duplicates: If a request times out and syslog-ng retries it, the same rows can be written twice. Deduplicate in queries when exact counts matter.
Run syslog-ng close to your endpoints: Endpoints only need to reach the collector. Only the collector needs outbound 443 to Zerobus Ingest.

Summary

When syslog-ng does the structuring and writes straight into Unity Catalog through Zerobus Ingest, you can bring syslog from any endpoint into Lakewatch with two moving parts: a collector and a datasource. Start with the built-in RFC 3164 parser to make every source searchable, then add per-source parsing in syslog-ng and custom parsers where your detections need them.

How it works​

Requirements​

Step 1: Create the raw table and grant access​

Step 2: Configure syslog-ng​

Step 3: Verify ingestion​

Step 4: Create the Lakewatch datasource​

Create the datasource in the UI​

Create the datasource with the CLI​

Step 5: Validate parsing​

Use a custom parser for richer fields​

Operational tips​

Summary​

Additional resources​