Skip to main content

CREATE FLOW (DLT)

Use the CREATE FLOW statement to create flows or backfills for your DLT tables.

Syntax​

CREATE FLOW flow_name [COMMENT comment] AS
{
  INSERT INTO [ONCE] target_table BY NAME query
}

Parameters​

  • flow_name

    The name of the flow to create.

  • COMMENT

    An optional description for the flow.

  • ONCE

    Optionally define the flow as a one time flow, such as a backfill. Using ONCE changes the flow in two ways:

    • The source query or apply_changes_spec is not a streaming table.
    • The flow is run one time by default. If the pipeline is updated with a complete refresh, then the ONCE flow will run again to recreate the data.

  • target_table

    The table to update. This must be a Streaming table.

  • INSERT INTO

    Defines a table query that is inserted into to the target table. If the ONCE option is not supplied, the query must be a streaming query. Use the STREAM keyword to use streaming semantics to read from the source. If the read encounters a change or deletion to an existing record, an error is thrown. It is safest to read from static or append-only sources. To ingest data that has change commits, you can use Python and the SkipChangeCommits option to handle errors.

Examples​

SQL
-- Create a streaming table, and add two flows that apply changes to it:
CREATE OR REFRESH STREAMING TABLE target_table;

-- first flow into target_table:
APPLY CHANGES INTO target_table
FROM stream(cdc_data.users)
KEYS (userId)
APPLY AS DELETE WHEN
 operation = "DELETE"
SEQUENCE BY sequenceNum
COLUMNS * EXCEPT (operation, sequenceNum)
STORED AS SCD TYPE 2;

-- second flow into target_table:
CREATE FLOW my_append_flow AS
INSERT INTO ONCE target_table
SELECT * FROM my_backfill_table
Last updated on