Skip to content

Configuration

ftm-lakehouse is configured via environment variables and a per-dataset config.yml.

Environment Variables

Core Settings

Variable Description Default
LAKEHOUSE_URI Base path or URI of the lakehouse storage data
LAKEHOUSE_STATEMENTS_URI Root of every dataset's statements table, as {uri}/{dataset}/statements – e.g. on SSDs while the archive stays on HDDs. Config, tags, locks and exports stay in the dataset. It applies to every dataset, whatever uri it is opened with – see Separate statements storage. (unset: {LAKEHOUSE_URI}/{dataset}/statements)
LAKEHOUSE_JOURNAL_URI SQLAlchemy URI of the statement journal – see Journal Database sqlite:///:memory:
LAKEHOUSE_GRACE_PERIOD_DAYS Days a tombstone (deleted_at) is kept before the merge of maintenance optimize drops it and the rows it shadows 30
LAKEHOUSE_MAX_BUFFER_ROWS Row cap of an in-memory EntityBuffer. Bulk imports that hit it raise BufferFullError; the caller flushes and retries. 1_000_000
LAKEHOUSE_JOURNAL_DRAIN_ROWS Rows per Arrow table a journal flush hands to the parquet store – one parquet file per partition the table spans. Higher means fewer, bigger files; lower it if a flush runs out of memory, as each table is held whole while it is written. 1_000_000
LAKEHOUSE_WRITE_BUFFER Memory a bulk import (entities import / statements import) holds across partitions before it writes the largest ones, as a byte size. Peak memory is about twice it; on a store with many partitions it also sets how big the import's files get. 1GiB
LAKEHOUSE_JOURNAL_POOL_SIZE Postgres journal connections kept idle between writers, one pool per journal uri and process, shared by all datasets. Writers beyond it open their own connection rather than queue. Multiply by the process count to size against postgres max_connections; 0 pools nothing. 5
LAKEHOUSE_LOCK_MAX_RETRIES Retry bound for every wait on a dataset lock – acquiring .LOCK or .LOCK-MERGE, and appends backing off while .LOCK is held. The total wait is about N²/2 seconds (a minute at the default), then RuntimeError. A lock left by a crashed process needs ftm-lakehouse maintenance unlock. 10
LAKEHOUSE_DUCKDB_MEMORY_LIMIT DuckDB's memory budget; queries beyond it spill to disk. A byte size (8GB, 512MiB) – a share such as 80% fails at startup, since the merge and the export sweep split it between their workers – see Sizing parallel workers. 8GB
LAKEHOUSE_WORKERS Processes the merge of maintenance optimize and the export sweep spread their partitions over; 1 (the minimum) runs in-process. CPU threads and the memory limit are split between them – see Sizing parallel workers. An export uses at most one worker per (shard, bucket) pair (shards × at most five buckets); TMPDIR holds the parts of the pairs in flight and a merge's temporary runs. 1
LAKEHOUSE_DUCKDB_TEMP_DIRECTORY Where DuckDB spills queries that outgrow the memory limit, each DuckDB instance into its own subdirectory, removed on close (a killed process leaves it behind). Point it at a fast volume with room – not a tmpfs /tmp, which spills into RAM (a tmpfs / ramfs spill directory logs a warning). Empty uses DuckDB's own default, .tmp in the working directory. {OS temp dir}/duckdb
LAKEHOUSE_DUCKDB_EXTENSION_DIRECTORY Where DuckDB loads (and auto-installs) extensions. Unset means $HOME/.duckdb/extensions, which fails without a writable HOME. The Docker image pre-installs delta into /opt/duckdb/extensions and sets this. (unset)
LAKEHOUSE_ON_ZFS Create tuned ZFS datasets for new datasets – see ZFS Integration false
LAKEHOUSE_ZFS_POOL ZFS dataset path new datasets are created under (e.g. zpools/tank/lakehouse). Transport settings (ZFS_SOCKET, ZFS_OWNER, …) belong to the zfs-agent package. (required with LAKEHOUSE_ON_ZFS)
LAKEHOUSE_ZFS_STATEMENTS_POOL ZFS dataset path the statements datasets are created under when LAKEHOUSE_STATEMENTS_URI puts them on another volume – see Separate statements pool. On ZFS, set both or neither. (unset)
LAKEHOUSE_API_KEY / LAKEHOUSE_API_SECRET Sent as X-Api-Key / X-Api-Secret on a client's lakehouse-API requests, for the reverse proxy in front of the server to check (unset)
LAKEHOUSE_PUBLIC_URL_PREFIX Public URL prefix for blob URLs, ${dataset} substituted. A dataset's own public_url_prefix takes precedence. (unset)
LOG_LEVEL Logging level (DEBUG, INFO, WARNING, ERROR) INFO
DEBUG Enable debug mode false

There is no environment setting for the shard count: shards is per-dataset configuration in config.yml (default 0, a single shard), set at creation – 8 or more for huge datasets. Changing it later rewrites every partition (maintenance shard) – see Sharding.

Sizing parallel workers

Merge. A worker's memory follows the merge's constants, not its partition: measured end to end, a worker peaks at about 2.5× the sum of its run (1GB), dedupe chunk (512MB) and write chunk (512MB) – about 5GB – plus a 16MB batch of every sorted run, 32 at most. Budget 7GB a worker: 16 workers in 112GB, 24 in 168GB. DuckDB's limit is the worker's share of LAKEHOUSE_DUCKDB_MEMORY_LIMIT. A partition with more than 1GB (uncompressed) of unmerged files writes temporary runs to TMPDIR, about their size on disk per worker; a huge one first merges its runs in rounds of 32, so the final merge never reads more than that. One entity larger than a chunk makes DuckDB spill for that chunk.

Export. A worker's DuckDB gets LAKEHOUSE_DUCKDB_MEMORY_LIMIT / LAKEHOUSE_WORKERS. A merged pair streams in file order and needs little; an unmerged partition is sorted through the dedupe with the full-text values of the document and page buckets in memory, so a share too small for the heaviest one fails it with DuckDB's Out of Memory Error – run optimize first (make does).

RAM has to fit workers × (share + 1–2GB), plus the ARC on ZFS. When the sum does not fit, the kernel kills a worker and the run stops with BrokenProcessPool; a merge keeps the partitions it committed, so a rerun picks up the rest.

The largest partitions on disk (after a vacuum, since unreferenced files count too):

du -sh "${LAKEHOUSE_STATEMENTS_URI:-$LAKEHOUSE_URI}/<dataset>/statements"/shard=*/bucket=*/origin=* | sort -h | tail

Workers are spawned processes that re-import the main module. The CLI is safe; a script calling merge or export with more than one worker needs an if __name__ == "__main__": guard, and cannot be piped into python on stdin.

Basic Usage

# Local filesystem
export LAKEHOUSE_URI=./my_lakehouse

# S3 storage
export LAKEHOUSE_URI=s3://my-bucket/lakehouse
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret

# With persistent journal (for production)
export LAKEHOUSE_JOURNAL_URI=postgresql://user:pass@localhost/journal

Dataset Configuration

Each dataset has its own config.yml, following the ftmq.model.Dataset specification:

name: my_dataset  # also known as "foreign_id"
title: An Awesome Dataset
shards: 0  # entity-id hash shards; configure 8+ for huge datasets at creation
compression: zst  # compress exported artifacts (gz / zst; unset = uncompressed)
description: >
  A detailed description of this dataset,
  its sources, and contents.
updated_at: 2024-09-25
category: leak  # or: sanctions, pep, etc.
publisher:
  name: Data and Research Center – DARC
  url: https://dataresearchcenter.org

Write it with ftm-lakehouse -d my_dataset configure -c config.yml (or update_dataset() from Python). Both merge – keys absent from the yaml keep their current value – and keep a versioned snapshot of each write.

Storage Backends

Local Filesystem

export LAKEHOUSE_URI=/path/to/lakehouse

Separate statements storage

The statements table is the hot path – flushes, merges, exports and queries all read or rewrite it – while the archive is mostly cold blobs. LAKEHOUSE_STATEMENTS_URI puts every dataset's table on its own volume:

export LAKEHOUSE_URI=/mnt/hdd/lakehouse             # config, archive, exports, tags, locks
export LAKEHOUSE_STATEMENTS_URI=/mnt/ssd/lakehouse  # {dataset}/statements

The setting moves no data. On an existing lake every store reads empty until its table is moved, and an export in between replaces the artifacts with empty ones. With writers and maintenance stopped, copy each table first – the Delta log's paths are table-relative, so a copy is a valid table:

rsync -a "$LAKEHOUSE_URI/<dataset>/statements/" "$LAKEHOUSE_STATEMENTS_URI/<dataset>/statements/"

Then set the variable, compare ftm-lakehouse -d <dataset> statements sql "SELECT count(*) FROM statement" with the count before the move, and remove the old directory.

Amazon S3

export LAKEHOUSE_URI=s3://bucket-name/prefix
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_REGION=us-east-1

S3-Compatible (MinIO, etc.)

export LAKEHOUSE_URI=s3://bucket-name/prefix
export AWS_ACCESS_KEY_ID=your_key
export AWS_SECRET_ACCESS_KEY=your_secret
export AWS_ENDPOINT_URL=https://minlake.example.com

Google Cloud Storage

Requires the gcs extra: pip install "ftm-lakehouse[gcs]"

export LAKEHOUSE_URI=gs://bucket-name/prefix
export GOOGLE_APPLICATION_CREDENTIALS=/path/to/credentials.json

Azure Blob Storage

Requires the azure extra: pip install "ftm-lakehouse[azure]"

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_ACCOUNT_NAME=your_account
export AZURE_STORAGE_ACCOUNT_KEY=your_key

Or using connection string:

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_CONNECTION_STRING="DefaultEndpointsProtocol=https;AccountName=...;AccountKey=...;EndpointSuffix=core.windows.net"

Or using SAS token:

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_ACCOUNT_NAME=your_account
export AZURE_STORAGE_SAS_TOKEN="?sv=2021-06-08&ss=b&srt=sco&sp=rwdlacyx..."

Or using Azure AD / Service Principal:

export LAKEHOUSE_URI=az://container-name/prefix
export AZURE_STORAGE_ACCOUNT_NAME=your_account
export AZURE_STORAGE_TENANT_ID=your_tenant_id
export AZURE_STORAGE_CLIENT_ID=your_client_id
export AZURE_STORAGE_CLIENT_SECRET=your_client_secret

Journal Database

The statement journal buffers writes until they are flushed into the parquet store. Use a persistent database in production.

SQLite (File-based)

export LAKEHOUSE_JOURNAL_URI=sqlite:///path/to/journal.db

PostgreSQL

Requires the postgres extra (pip install "ftm-lakehouse[postgres]"); without it, creating the journal fails.

export LAKEHOUSE_JOURNAL_URI=postgresql://user:password@host:5432/database

In-Memory (for debugging / testing)

export LAKEHOUSE_JOURNAL_URI=sqlite:///:memory:

Warning

The in-memory journal is lost when the process exits. Use a persistent database for production workloads.

Python Configuration

from ftm_lakehouse import get_entities, get_lakehouse

# Get lakehouse with custom URI
lake = get_lakehouse(uri="s3://my-bucket/lakehouse")

# Repositories per dataset (uri derived from the catalog)
entities = get_entities("my_dataset", lake.dataset_uri("my_dataset"))

Multi-Dataset Configuration

A lakehouse holds multiple datasets, each with its own configuration:

lakehouse/
  dataset_a/
    config.yml         # Dataset A config
    archive/
    ...
  dataset_b/
    config.yml         # Dataset B config (could point to remote storage)
    ...

A dataset can reference remote storage while appearing in a local catalog:

# lakehouse/remote_dataset/config.yml
name: remote_dataset
title: Remote Dataset
# This dataset's data lives in S3
storage:
  uri: s3://remote-bucket/dataset

Catalog

The catalog is the storage root itself – any directory under the lakehouse uri that contains a config.yml is a dataset. get_lakehouse().list_datasets() enumerates them; there is no catalog-level configuration file.