Skip to main content
Last updated on

Create a NetSuite ingestion pipeline

Preview

The NetSuite connector is in Public Preview.

Learn how to create a NetSuite ingestion pipeline using Databricks APIs or the Databricks CLI. You can't create NetSuite pipelines using the data ingestion UI wizard.

Requirements​

Before you begin, you must meet the following requirements:

  • Serverless Lakeflow Spark Declarative Pipelines must be enabled for your workspace. See Configure a serverless pipeline.
  • You must have a Unity Catalog connection to NetSuite. See Configure NetSuite for ingestion into Databricks and NetSuite.
  • You must have one of the following privileges on the Unity Catalog connection: USE CONNECTION or ALL PRIVILEGES.
  • You must have the USE CATALOG privilege on the target catalog.
  • You must have one of the following privilege sets on the target schema:
    • USE SCHEMA, CREATE TABLE, CREATE VOLUME on an existing schema
    • CREATE SCHEMA on the target catalog

Download the SuiteAnalytics Connect JDBC driver​

The NetSuite connector requires that you upload your SuiteAnalytics Connect JDBC driver JAR file to a Unity Catalog volume.

  1. Log in to your NetSuite account.
  2. In the Settings portlet, click Set up Analytics Connect.
  3. Download the JAR file to your local machine.

Create a Unity Catalog volume​

Create a volume to store the JAR file.

See Create a volume.

Users who create NetSuite ingestion pipelines must have access to this volume.

Upload the JAR file to the volume​

  1. In the Catalog pane, navigate to the volume you created.
  2. Click Upload to volume.
  3. Select the JAR file you downloaded and click Upload.

Create the pipeline​

You can create a NetSuite ingestion pipeline using a notebook or the Databricks CLI. The data ingestion UI wizard isn't supported for NetSuite pipelines.

Databricks notebook​

  1. Create personal access tokens for workspace users.

  2. Import the following notebook into your workspace:

    Create a NetSuite ingestion pipeline notebook

    Open notebook in new tab

  3. In cell #1, replace the value for api-token.

  4. Leave the values in cell #2. Don't modify this cell.

  5. Modify the following values in cell #3:

    • pipeline_name: A name for your Netsuite ingestion pipeline.
    • connection_name: The name of the Unity Catalog connection that stores authentication details for NetSuite.
    • destination_catalog: The catalog you want the connector to write to.
    • destination_schema: The schema you want the connector to write to.
    • source_tables: The tables to ingest.

  6. (Optional) Add additional tables to the objects array to ingest multiple tables in a single pipeline. The maximum number of tables per pipeline is 200.

  7. Click Run all.

Databricks CLI​

You can create a NetSuite pipeline using the Databricks CLI with a JSON configuration file.

  1. Create a JSON file named netsuite-pipeline.json with the following content:

    JSON
    {
      "name": "<pipeline-name>",
      "catalog": "<destination-catalog>",
      "target": "<destination-schema>",
      "serverless": true,
      "ingestion_definition": {
        "connection_name": "<netsuite-connection-name>",
        "table_configuration": {
          "netsuite_jar_path": "/Volumes/<catalog>/<schema>/<volume>/NQjc.jar"
        },
        "objects": [
          {
            "table": {
              "source_schema": "netsuite2",
              "source_table": "<table-name>",
              "destination_catalog": "<destination-catalog>",
              "destination_schema": "<destination-schema>",
              "table_configuration": {
                "scd_type": "SCD_TYPE_1"
              }
            }
          }
        ]
      }
    }

  2. Replace the placeholder values:

    • <pipeline-name>: A unique name for the ingestion pipeline.
    • <destination-catalog>: The catalog where ingested data will be written.
    • <destination-schema>: The schema where ingested data will be written.
    • <netsuite-connection-name>: The name of the Unity Catalog connection you created.
    • <catalog>/<schema>/<volume>: The path to the volume where you uploaded the NetSuite JAR file.
    • <table-name>: The name of the NetSuite table to ingest.

  3. Create the pipeline using the Databricks CLI:

    Bash
    databricks pipelines create --json @netsuite-pipeline.json

Additional resources​