Configuration

ftm-lakehouse can be configured via environment variables or YAML configuration files.

Environment Variables

Core Settings

Variable	Description	Default
`LAKEHOUSE_URI`	Base path to lakehouse storage	`./data`
`LAKEHOUSE_JOURNAL_URI`	SQLAlchemy URI for statement journal	`sqlite:///:memory:`
`LAKEHOUSE_GRACE_PERIOD_DAYS`	Default tombstone grace period used by `maintenance optimize` (rows with `deleted_at` older than this are physically dropped in the merge step)	`30`
`LAKEHOUSE_MAX_BUFFER_ROWS`	Hard cap on rows held in an in-memory `EntityBuffer` before a flush is required. Bulk-import paths that hit the cap raise `BufferFullError` and the caller flushes + retries.	`1_000_000`
`LAKEHOUSE_JOURNAL_POOL_SIZE`	Postgres journal connections kept warm between writers. ADBC ships no pool of its own and a cold connection costs ~60ms, which the journal would pay per writer; a checkout is ~0.3ms including the liveness ping that keeps a connection the server has since dropped (`idle_session_timeout`, a pgbouncer reap, a failover, a restart) from reaching a writer. This bounds only what is kept idle – writers beyond it open their own connection rather than queueing, so peak connections follow write concurrency either way. It applies per cached dataset journal, so a worker writing many datasets holds up to this many idle connections for each of them: that is the figure to size against postgres `max_connections`. Set to `0` to pool nothing.	`5`
`LAKEHOUSE_LOCK_MAX_RETRIES`	Retry bound for every wait on the dataset write fence: acquiring the exclusive maintenance lock (`.LOCK`), an append backing off while `.LOCK` is held, and maintenance draining in-flight append markers (`.LOCK-APPENDS/`). Retry `n` sleeps `n` + jitter seconds, so the total wait is roughly `N²/2` seconds; the default gives up after ~4.5 minutes with a `RuntimeError` instead of waiting forever. Stale locks or markers from crashed writers need a manual `ftm-lakehouse maintenance unlock`.	`22`
`LAKEHOUSE_DUCKDB_MEMORY_LIMIT`	Per-DuckDB-connection RAM ceiling. Queries exceeding it spill to disk rather than growing toward all available RAM. Format follows DuckDB's `SET memory_limit` (e.g. `8GB`, `512MB`).	`8GB`
`LAKEHOUSE_DUCKDB_TEMP_DIRECTORY`	Spill-to-disk path for queries that overflow `LAKEHOUSE_DUCKDB_MEMORY_LIMIT`	Defaults under the OS temp directory. Point at a fast, capacity-controlled volume for heavy workloads. Set to a real volume if a container uses `/tmp` as `tmpfs` to avoid RAM spilling. Set empty to use default.
`LAKEHOUSE_DUCKDB_EXTENSION_DIRECTORY`	Directory DuckDB loads its extensions from (and auto-installs into when one is missing). Unset = `$HOME/.duckdb/extensions`, which fails in containers running without a writable `HOME` (`Failed to create directory "/.duckdb"`). The shipped Docker image pre-installs the `delta` extension at build time and sets this to `/opt/duckdb/extensions`, so runtime needs neither a writable `HOME` nor network access.	(unset)
`LAKEHOUSE_ON_ZFS`	Enable ZFS dataset creation for local storage	`false`
`LAKEHOUSE_ZFS_POOL`	ZFS dataset path for the lakehouse root (e.g. `zpools/tank/lakehouse`). Transport / agent settings (`ZFS_SOCKET`, `ZFS_OWNER`, ...) belong to the zfs-agent package -- see ZFS Integration	(required when `ON_ZFS` is enabled)
`LAKEHOUSE_API_KEY` / `LAKEHOUSE_API_SECRET`	Client-side auth headers attached to outgoing lakehouse-API requests (authenticate through the reverse proxy in front of the API server)	(unset)
`LAKEHOUSE_PUBLIC_URL_PREFIX`	Public URL prefix for blob URLs (supports a `${dataset}` placeholder)	(unset)
`LOG_LEVEL`	Logging level (DEBUG, INFO, WARNING, ERROR)	`INFO`
`DEBUG`	Enable debug mode	`false`

There is deliberately no environment setting for the shard count: it is per-dataset configuration (shards in config.yml, default 0 = single shard). Placement is enforced on write – ParquetStore.append derives each row's shard from its entity_id against the count the store resolved – so a process with a different environment can't mis-shard an existing dataset. Huge datasets should configure 8 or more at creation; changing it later means rewriting every partition with maintenance shard – see Sharding.

Basic Usage

# Local filesystem
export LAKEHOUSE_URI=./my_lakehouse

# S3 storage
export LAKEHOUSE_URI=s3://my-bucket/lakehouse
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret

# With persistent journal (for production)
export LAKEHOUSE_JOURNAL_URI=postgresql://user:pass@localhost/journal

Dataset Configuration

Each dataset can have its own config.yml file that follows the ftmq.model.Dataset specification:

name: my_dataset  # also known as "foreign_id"
title: An Awesome Dataset
shards: 0  # entity-id hash shards; configure 8+ for huge datasets at creation
compression: zst  # compress exported artifacts (gz / zst; unset = uncompressed)
description: >
  A detailed description of this dataset,
  its sources, and contents.
updated_at: 2024-09-25
category: leak  # or: sanctions, pep, etc.
publisher:
  name: Data and Research Center – DARC
  url: https://dataresearchcenter.org

Write it with ftm-lakehouse -d my_dataset configure -c config.yml (or update_dataset() from Python). Both merge – keys absent from the yaml keep their current value – and keep a versioned snapshot of each write.

Storage Backends

Local Filesystem

export LAKEHOUSE_URI=/path/to/lakehouse

Amazon S3

export LAKEHOUSE_URI=s3://bucket-name/prefix
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_REGION=us-east-1

S3-Compatible (MinIO, etc.)

export LAKEHOUSE_URI=s3://bucket-name/prefix
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_ENDPOINT_URL=https://minlake.example.com

Google Cloud Storage

Requires extra install: pip install gcsfs

export LAKEHOUSE_URI=gs://bucket-name/prefix
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json

Azure Blob Storage

Requires extra install: pip install adlfs

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_ACCOUNT_NAME=your_account
export AZURE_STORAGE_ACCOUNT_KEY=your_key

Or using connection string:

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"

Or using SAS token:

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_ACCOUNT_NAME=your_account
export AZURE_STORAGE_SAS_TOKEN="?sv=2021-06-08&ss=b&srt=sco&sp=rwdlacyx..."

Or using Azure AD / Service Principal:

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_ACCOUNT_NAME=your_account
export AZURE_STORAGE_TENANT_ID=your_tenant_id
export AZURE_STORAGE_CLIENT_ID=your_client_id
export AZURE_STORAGE_CLIENT_SECRET=your_client_secret

Journal Database

The statement journal buffers writes before flushing to Delta Lake storage. For production use, configure a persistent database:

SQLite (File-based)

export LAKEHOUSE_JOURNAL_URI=sqlite:///path/to/journal.db

PostgreSQL

export LAKEHOUSE_JOURNAL_URI=postgresql://user:password@host:5432/database

In-Memory (for debugging / testing)

export LAKEHOUSE_JOURNAL_URI=sqlite:///:memory:

Warning

The in-memory journal is lost when the process exits. Use a persistent database for production workloads.

Python Configuration

You can also configure programmatically:

from ftm_lakehouse import get_entities, get_lakehouse

# Get lakehouse with custom URI
lake = get_lakehouse(uri="s3://my-bucket/lakehouse")

# Repositories per dataset (uri derived from the catalog)
entities = get_entities("my_dataset", lake.dataset_uri("my_dataset"))

Multi-Dataset Configuration

A lakehouse can contain multiple datasets, each with different configurations:

lakehouse/
  dataset_a/
    config.yml         # Dataset A config
    archive/
    ...
  dataset_b/
    config.yml         # Dataset B config (could point to remote storage)
    ...

A dataset can reference remote storage while appearing in a local catalog:

# lakehouse/remote_dataset/config.yml
name: remote_dataset
title: Remote Dataset
# This dataset's data lives in S3
storage:
  uri: s3://remote-bucket/dataset

Catalog

The catalog is the storage root itself – any directory under the lakehouse uri that contains a config.yml is a dataset. get_lakehouse().list_datasets() enumerates them; there is no catalog-level configuration file.