Skip to main content
Last updated on

Ingest data from Jira

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

Learn how to create a managed Jira ingestion pipeline using Databricks Lakeflow Connect.

note

The Jira connector automatically retries with exponential backoff when rate limits are encountered. If rate limit errors persist, see Rate limit errors.

Requirements​

  • To create an ingestion pipeline, you must first meet the following requirements:

    • Your workspace must be enabled for Unity Catalog.

    • Serverless compute must be enabled for your workspace. See Serverless compute requirements.

    • To create a new connection, you must have CREATE CONNECTION privileges on the metastore. See Manage privileges in Unity Catalog.

      If the connector supports UI-based pipeline authoring, an admin can create the connection and the pipeline at the same time by completing the steps on this page. However, if the users who create pipelines use API-based pipeline authoring or are non-admin users, an admin must first create the connection in Catalog Explorer. See Connect to managed ingestion sources.

    • To use an existing connection, you must have USE CONNECTION privileges or ALL PRIVILEGES on the connection object.

    • You must have USE CATALOG privileges on the target catalog.

    • You must have USE SCHEMA and CREATE TABLE privileges on an existing schema or CREATE SCHEMA privileges on the target catalog.

  • To ingest from Jira, you must first complete the steps in Create a Jira connection.

Create an ingestion pipeline​

Each source table is ingested into a streaming table or a snapshot table, depending on the source. For a list of supported source tables, see Jira connector reference.

  1. In the sidebar of the Databricks workspace, click Data Ingestion.
  2. On the Add data page, under Databricks connectors, click Jira.
  3. On the Connection page of the ingestion wizard, select the connection that stores your Jira access credentials. If you have the CREATE CONNECTION privilege on the metastore, you can click Plus icon. Create connection to create a new connection with the authentication details in Create a Jira connection.
  4. Click Next.
  5. On the Ingestion setup page, enter a unique name for the pipeline.
  6. Select a catalog and a schema to write event logs to. If you have USE CATALOG and CREATE SCHEMA privileges on the catalog, you can click Plus icon. Create schema in the drop-down menu to create a new schema.
  7. Click Create pipeline and continue.
  8. On the Source page, select the tables to ingest. You can optionally filter the data by Jira spaces or projects. Use exact project keys, not project names or IDs.
  9. Click Save and continue.
  10. On the Destination page, select a catalog and a schema to load data into. If you have USE CATALOG and CREATE SCHEMA privileges on the catalog, you can click Plus icon. Create schema in the drop-down menu to create a new schema.
  11. Click Save and continue.
  12. (Optional) On the Schedules and notifications page, click Plus icon. Create schedule. Set the frequency to refresh the destination tables.
  13. (Optional) Click Plus icon. Add notification to set email notifications for pipeline operation success or failure, then click Save and run pipeline.

Examples​

Use these examples to configure your pipeline.

Ingest a single source table​

(Recommended) The following pipeline definition file ingests a single source table.

YAML
variables:
  dest_catalog:
    default: main
  dest_schema:
    default: ingest_destination_schema

# The main pipeline for jira_dab
resources:
  pipelines:
    pipeline_jira:
      name: jira_pipeline
      catalog: ${var.dest_catalog}
      schema: ${var.dest_schema}
      ingestion_definition:
        connection_name: <jira-connection>
        objects:
          # An array of objects to ingest from Jira. This example ingests the issues table.
          - table:
              source_schema: default
              source_table: issues
              destination_catalog: ${var.dest_catalog}
              destination_schema: ${var.dest_schema}

Ingest multiple source tables​

(Recommended) The following pipeline definition file ingests multiple source tables.

YAML
variables:
  dest_catalog:
    default: main
  dest_schema:
    default: ingest_destination_schema

# The main pipeline for jira_dab
resources:
  pipelines:
    pipeline_jira:
      name: jira_pipeline
      catalog: ${var.dest_catalog}
      schema: ${var.dest_schema}
      ingestion_definition:
        connection_name: <jira-connection>
        objects:
          # An array of objects to ingest from Jira. This example ingests the issues and projects tables.
          - table:
              source_schema: default
              source_table: issues
              destination_catalog: ${var.dest_catalog}
              destination_schema: ${var.dest_schema}
          - table:
              source_schema: default
              source_table: projects
              destination_catalog: ${var.dest_catalog}
              destination_schema: ${var.dest_schema}

Ingest all source tables​

The following pipeline definition file ingests all available Jira source tables in one pipeline. Make sure that your OAuth application includes all scopes required by the full table set and the authenticating user has the necessary Jira permissions. Pipelines fail if any required scope or permission is missing.

YAML
variables:
  dest_catalog:
    default: main
  dest_schema:
    default: ingest_destination_schema

# The main pipeline for jira_dab
resources:
  pipelines:
    pipeline_jira:
      name: jira_pipeline
      catalog: ${var.dest_catalog}
      schema: ${var.dest_schema}
      ingestion_definition:
        connection_name: <jira-connection>
        objects:
          # An array of objects to ingest from Jira. This example ingests all tables in the default schema.
          - schema:
              source_schema: default
              destination_catalog: ${var.dest_catalog}
              destination_schema: ${var.dest_schema}

Declarative Automation Bundles job definition file​

The following is an example job definition file to use with Declarative Automation Bundles. The job runs every day, exactly one day from the last run.

YAML
resources:
  jobs:
    jira_dab_job:
      name: jira_dab_job

      trigger:
        periodic:
          interval: 1
          unit: DAYS

      email_notifications:
        on_failure:
          - <email-address>

      tasks:
        - task_key: refresh_pipeline
          pipeline_task:
            pipeline_id: ${resources.pipelines.pipeline_jira.id}

Filter by project​

You can filter ingested data to specific Jira projects by adding connector_options.jira_options to any table object in your pipeline definition. Specify exact project keys in include_jira_spaces. Project keys are case-sensitive and differ from project names and IDs.

To find your project key, look at the URL from in the project. The key appears after /projects/. Alternatively, go to Project Settings > Details to see the current and previous keys, or look at the top of a project issue card.

For more information, see Edit a space's details in the Jira documentation.

YAML
- table:
    source_schema: default
    source_table: issues
    destination_catalog: ${var.dest_catalog}
    destination_schema: ${var.dest_schema}
    connector_options:
      jira_options:
        include_jira_spaces:
          - KEY1
          - KEY2

Track deletes for the issues table​

important

By default, the Jira connector does not track deletes for the issues table. Deleted Jira issues remain in the destination table. To propagate deletes, set use_audit_logs to true in jira_options. This requires a paid Jira plan and a connection user with Jira admin permissions.

To detect and propagate deletes for the issues table, the connector reads the Jira audit log. Reading the audit log requires a paid Jira plan and admin permissions, so the connector does not read it by default. When use_audit_logs is false (the default), the issues table retains all previously ingested rows and reflects only inserts and updates. When you set use_audit_logs to true, the connector removes a row from the issues table when the corresponding issue is deleted in Jira.

To track deletes, set use_audit_logs to true under connector_options.jira_options on the issues table object, as shown in the following examples. This option affects only the issues table.

You can set use_audit_logs to true on an existing pipeline without a full refresh. It takes effect on the next pipeline update, when the connector reconciles the deletes that are still available in the Jira audit log. Jira retains audit log records for a limited period that depends on your Jira plan, so issues deleted before that retention window remain in the destination table. To remove those older deleted rows, run a full refresh of the issues table.

For the requirements to enable delete tracking, see Delete tracking and How do I enable audit logs in Jira?.

YAML
- table:
    source_schema: default
    source_table: issues
    destination_catalog: ${var.dest_catalog}
    destination_schema: ${var.dest_schema}
    connector_options:
      jira_options:
        use_audit_logs: true

Common patterns​

For advanced pipeline configurations, see Common patterns for managed ingestion pipelines.

Next steps​

Start, schedule, and set alerts on your pipeline. See Common pipeline maintenance tasks.

Additional resources​

On this page