Skip to main content
Last updated on

Create a custom parser

Preview

Lakewatch is in Private Preview. To request access, reach out to your Databricks account team.

Lakewatch provides built-in parsers for common security log sources. Write a custom parser when you ingest a source that no built-in parser covers, or when you want to tailor the columns, validators, or observables a built-in parser produces.

A parser is a versioned specification that describes the output columns of a log source. You build it in the parser editor in either UI mode (a guided, point-and-click builder) or YAML mode, and it's created in your current Lakewatch environment. For the full syntax (field attributes, types, timestamps, validators, and observables), see the Parser reference.

Open the parser editor​

From the Lakewatch sidebar, select Parsers, then select Create parser. The parser list also shows every built-in and custom parser in your environment, with its type, attached datasources, and latest version.

To start from a built-in parser instead of a blank one, find it in the list, open its ⋮ actions menu, and select Clone. Cloning copies the definition into a new, editable custom parser (version 1) and leaves the built-in parser unchanged.

Define the parser​

On the Add new parser page, give the parser a Parser name and an optional Description. The name is a display name and can contain spaces.

Build the output columns under Fields & Observables, in either UI or YAML mode. You can switch between them at any time.

Choose sample data​

Select Choose sample data and pick the raw records this parser is for, so fields come from real data. You can sample from a Datasource, External location, Volume, or Table.

With sample data chosen, you can optionally select Build parser with Genie: Genie reads the sample and drafts the event time and fields for you to review and adjust.

Set the event time and fields​

  • Event time (required): the timestamp of each event. Every parser needs one before it can be saved. In YAML, this is the top-level time block (not a field), because it drives clustering and query performance. See Event time.
  • Fields: the columns of the parsed table and where each one is read from. For each field, set name, type, and (when the source key differs from the column name) path; mark fields that must be present with required: true; add validate to constrain a STRING field, and observables to extract values (such as ip or email) into the shared lw_observables column. Express nested objects as VARIANT with inner fields, and lists as ARRAY with an element.

Define your fields carefully. During classification, an event matches only when its required fields are present, its typed fields convert to the declared types, and all validators pass. An event that matches no attached parser is routed to a dead-letter sentinel rather than failing the pipeline, so an over-strict parser silently drops events.

Preview and save​

In the Data preview panel, select Preview to run the parser against your sample data and confirm the columns, types, and extracted observables are what you expect. When it looks right, select Save. Saving stores the definition as version 1, records you as its owner, and creates the parser in your current environment. A parser document is at most 256 KB.

Attach the parser to a datasource​

Attach the custom parser to a datasource the same way you attach a built-in parser. You can attach more than one parser to a datasource; incoming events are matched in declaration order and land in the first parser they match. See Attach parsers.

Test and iterate​

After the datasource runs, confirm the parser behaves as expected:

  • Query the parsed table to verify the columns, types, and extracted observables match what you defined.
  • Check for events in the dead-letter sentinel. Events land there when they don't match any attached parser: usually because a required field is missing, a value doesn't convert to its declared type, or a validator fails.

To refine the parser, edit it and save. Editing appends a new immutable version; earlier versions are never changed and version numbers are never reused. A new version keeps the parser's name and changes only the definition. A datasource pins a specific version, so update the datasource to pick up your new version: see Change a parser's version. Existing parsed data isn't reprocessed; the new version applies to events ingested after you save the change.

Additional resources​

On this page