DuckLake destination

The DuckLake destination is in private alpha and available only to approved organizations. Request access before following this guide.

Replicate Postgres tables to DuckLake for current-state lakehouse queries. Prepare resources, configure the destination, then query replicated data.

Source table requirements#

Insert-only tables don't require a primary key or replica identity. Updates and deletes require a published Postgres row identity. See supported replica identities.

Prepare DuckLake resources #

Prepare a Postgres catalog, object storage, and a compatible query engine:

You can query replicated tables, but treat them and their underlying catalog and object-storage state as read-only. Writes outside Pipelines can conflict with replication and background maintenance.

Configure DuckLake as a destination #

Choose a mode below for its resource requirements and configuration steps.

Follow Set up Pipelines, select DuckLake, then choose Use Supabase or Custom parameters for the catalog and storage.

Use Supabase#

Use this mode to back the DuckLake with Supabase projects. Pipelines provisions its own catalog and object-storage credentials when you create the destination.

Before you begin:

• Choose active, healthy, non-branch projects from the same organization for the catalog and storage. You can use the same project for both.
• Make sure your organization role can administer SQL in the catalog project and Storage in the storage project.
• Create a private standard Storage bucket, or create one from the destination form.
• Choose a metadata schema unique to this DuckLake. Use only letters, numbers, and underscores.
• Keep catalog and storage in the same region when possible, near the managed pipeline region.

To configure the destination:

1. Select Use Supabase.
2. Choose the Catalog project, Pool size, and Metadata schema. Pool size allows 1 to 6 concurrent DuckDB connections; the default is 4.
3. Choose the Storage project and private Bucket.
4. Click Create and start pipeline and complete the validation and cost confirmations.

Credential-provisioning warnings are expected before creation: catalog and Storage credentials are provisioned when you save the destination. Review the selected resources before proceeding.

Custom parameters#

Use this mode with a Postgres catalog and S3-compatible object storage that you control.

Prepare the following resources:

1. A Postgres database reachable from managed Pipelines. Create a dedicated user that can create and modify the DuckLake metadata schema and its tables.
2. An S3-compatible bucket and a dedicated prefix for this DuckLake.
3. Object-storage credentials that can list, read, write, and delete objects under that prefix. Delete access is required for managed file cleanup.

Use a new catalog metadata schema and data prefix for each destination. Reusing an existing schema or prefix can mix the metadata or files of different DuckLakes.

Configure these fields in the destination form:

• Catalog URL: A postgres:// or postgresql:// connection URL, including credentials and an sslmode appropriate for your provider
• Data path: An s3://<bucket>/<prefix> URL
• Pool size: From 1 to 6; the default is 4
• S3 access key ID and S3 secret access key: A credential pair for the data path
• S3 region: The storage provider's region
• S3 endpoint: The provider endpoint without http:// or https://
• S3 URL style: path for Supabase Storage and many S3-compatible providers, or vhost for virtual-host-style addressing
• Use SSL: Keep enabled for production endpoints
• Metadata schema: A unique Postgres schema for DuckLake metadata, using only letters, numbers, and underscores

Click Create and start pipeline and complete the validation and cost confirmations.

The catalog URL and storage credentials are stored as secrets and aren't returned after creation. When editing the destination, leave a secret field empty to keep its stored value, or enter a new value to replace it.

How replication works#

Pipelines creates current-state tables, copies rows according to the initial sync selection, then applies published changes and supported schema changes.

Source schema and table names are preserved, while ASCII uppercase letters in column names are converted to lowercase. DuckDB compares schema, table, and column identifiers without ASCII case distinctions, so don't publish names that differ only by ASCII case. Use distinct lowercase column names. Source column names matching the generated supabase_etl_ducklake_dropped_<ordinal>_<hash> shape are reserved for schema-change recovery.

A source TRUNCATE truncates the DuckLake table. A table restart drops and recreates it. Removing a table from the publication leaves its destination data in place.

Query replicated data #

Connect DuckDB with its ducklake extension, or another compatible engine, to the same catalog and storage path. Query through the catalog; reading raw Parquet files can miss inlined changes, delete files, and the current snapshot.

For Use Supabase mode, create separate read credentials for the selected catalog and Storage projects. The writer credentials generated for Pipelines aren't exposed. For Custom parameters, use separate read-only credentials when your catalog and storage provider support them.

After attaching the catalog under an alias such as my_ducklake, source schemas and tables are available as qualified DuckLake tables:

1
select *
2
from my_ducklake.public.orders;

See the DuckDB connection guide for the current ducklake extension and ATTACH syntax. If your query client can't read a Supabase-backed catalog during private alpha, contact support.

Replica identity#

Replica-identity changes affect only new WAL. If retained updates or deletes still fail after a pipeline restart, restart replication for the affected table.

Type mapping#

Pipelines creates DuckLake columns with these mappings:

DuckDB decimals support precision from 1 to 38 and a scale between 0 and the precision. Postgres numeric values declared outside that range, unconstrained numeric, and numeric types with unsupported modifiers are stored as varchar to preserve their serialized value.

Numeric arrays use varchar[], even when their declared precision and scale would fit a scalar DuckLake decimal.

Schema change support#

Pipelines supports:

• Adding, renaming, or dropping columns
• Dropping NOT NULL from an existing column
• Adding, changing, or removing supported column defaults
• Adding or removing published columns on tracked tables

New columns are created as nullable so existing destination rows remain valid. Adding NOT NULL keeps an existing destination column nullable. Supported defaults are stored in DuckLake metadata; other defaults are skipped. Postgres still supplies the source values through replication.

Previously excluded columns are added without a default, leaving historical rows NULL. Removing a published column drops its destination values; adding it again does not restore them.

For type changes, unsupported changes, and interrupted schema changes, see the shared schema-change behavior and recovery.

Troubleshooting#

Use pipeline monitoring to inspect errors. For unresolved failures, contact support with the pipeline ID and error details.

Additional resources#

• DuckLake documentation
• DuckLake architecture
• Monitor pipeline status