Databricks

Install

See the Install guide for the full setup, including Windows PowerShell.

curl -fsSL https://install.skippr.io/install.sh | shClick to copy

Uploads data to Databricks as Parquet files via the Files API, with optional COPY INTO.

Configuration

warehouse:
  kind: databricks
  workspace_url: "https://my-workspace.cloud.databricks.com"
  token: ${DATABRICKS_TOKEN}
  warehouse_id: "abc123"
  catalog: main
  schema: default

Field	Default	Description
workspace_url	(required)	Databricks workspace URL
token	(required)	Personal access token
warehouse_id		SQL warehouse ID (enables COPY INTO)
catalog	main	Unity Catalog name
schema	default	Schema name

CLI

skippr connect warehouse databricks \
  --workspace-url https://dbc-xxxxxxxx.cloud.databricks.com \
  --token "dapi..." \
  --warehouse-id abc123 \
  --catalog main \
  --schema default

Flag	Description
--workspace-url	Databricks workspace URL
--token	Personal access token
--warehouse-id	SQL warehouse ID
--catalog	Unity Catalog name (default: main)
--schema	Target schema for bronze/raw data (default: default)

Authentication

Configure workspace_url, token, and optional warehouse_id directly when you connect the warehouse or in skippr.yaml.

For security best practices, we strongly advise against storing the token in skippr.yaml. Use environment variable interpolation instead: replace the token value with your own ${ENV_VAR} reference.

The relevant part of skippr.yaml looks like this:

warehouse:
  kind: databricks
  token: ${DATABRICKS_TOKEN}

Set the env var before running skippr:

macOS / Linux

export DATABRICKS_TOKEN="dapi..."

Windows PowerShell

$env:DATABRICKS_TOKEN = "dapi..."

Windows Command Prompt

set DATABRICKS_TOKEN=dapi...

Permissions or Network Requirements

The token must be allowed to write to the selected catalog and schema. If you use warehouse_id, the SQL warehouse must also be reachable and authorized for COPY INTO or query execution.

Troubleshooting

Symptom	Fix
authentication failed	Verify the workspace URL, token, and any SQL warehouse ID values.
writes or COPY INTO fail	Check catalog and schema permissions, warehouse availability, and whether the token can use that SQL warehouse.

CDC Support

Databricks supports CDC with exactly-once final-state reconciliation using Unity Catalog MERGE. Skippr automatically creates _skippr_order_token columns and tombstone tables.

See CDC Destinations -- Databricks for details.

Next steps

• Pair this destination with a source from Source Connectors.
• For CDC semantics, see CDC Destinations, CDC Guarantees, and CDC Operations.

Install

See the Install guide for the full setup, including Windows PowerShell.

curl -fsSL https://install.skippr.io/install.sh | shClick to copy