Skip to main content
Last updated on

Veeva Vault connector concepts

Beta

This feature is in Beta. Workspace admins can control access to this feature from the Previews page. See Manage Databricks previews.

This page describes key concepts for the Veeva Vault connector in Lakeflow Connect.

How the connector works​

The Veeva Vault connector uses Veeva's Direct Data API to ingest your Vault data into Databricks. Veeva generates two types of archives:

  • Full archives: Generated once daily. Contain a complete snapshot of all data.
  • Incremental archives: Generated every 15 minutes. Contain only records that changed since the last archive.

On the first pipeline run, the connector downloads and processes the most recent full archive. Subsequent runs use incremental archives to ingest only the changed records, improving performance and reducing costs.

The connector downloads archives to a Unity Catalog volume (configured in your connection) before processing. This staging volume caches archives between pipeline triggers.

Archive retention and full refresh behavior​

Veeva retains incremental archives for 10 days and full archives for 2 days. If a pipeline falls more than 10 days behind, the incremental archive chain is broken and a full refresh is required.

When a full refresh is triggered, the process spans two pipeline updates:

  1. The first update clears the staged archive state from the Unity Catalog volume.
  2. The second update downloads and processes a new full archive.

Ingested objects​

By default, the connector ingests all objects from your Vault instance, including:

  • Standard objects (__v)
  • Custom objects (__c)
  • System tables
  • Audit trail tables

You can also configure the pipeline to ingest specific objects instead of the full Vault.

Pricing​

Managed connectors have a compute-based pricing model. The Veeva Vault connector runs on serverless infrastructure and incurs serverless Lakeflow Spark Declarative Pipelines DBU charges. For rate details, see the Lakeflow Spark Declarative Pipelines pricing page. Additional charges may apply, such as serverless egress fees.

On this page