CLI Reference

ftm-lakehouse is a Typer CLI organised into sub-command groups.

ftm-lakehouse [OPTIONS] <group> <command> [ARGS]

Group	Purpose
`archive`	Content-addressed file storage
`entities`	Read and write FtM entities
`statements`	Read and write raw FtM statements
`maintenance`	Storage maintenance (flush, optimize, shard, migrate, unlock)
`zfs`	ZFS dataset management

Top-level commands: ls (dataset names), datasets (metadata), configure (write dataset configuration), make (build or update a dataset), export (write every export artifact), crawl (ingest documents into the archive).

Environment variables configure storage and behaviour – see the configuration reference.

Examples

export LAKEHOUSE_URI=./data

# Initialise the dataset – no data yet, so flush only
ftm-lakehouse -d my_dataset make --no-exports

# Record its configuration (title, summary, shards, compression, ...)
ftm-lakehouse -d my_dataset configure -c config.yml

# Crawl some files
ftm-lakehouse -d my_dataset crawl /path/to/documents

# Bulk-load a pre-built entities.ftm.json (skips the journal)
cat entities.ftm.json | ftm-lakehouse -d my_dataset entities import

# Several times faster for trusted input – same statements, no FtM validation
cat entities.ftm.json | ftm-lakehouse -d my_dataset entities import --unsafe

# Flush the journal, optimize the store and build all exports – the default
ftm-lakehouse -d my_dataset make

# The export on its own – every artifact from one pass over the entities
ftm-lakehouse -d my_dataset export

# Drain the journal on its own – one dataset, or the whole catalog
ftm-lakehouse -d my_dataset maintenance flush
ftm-lakehouse maintenance flush --all

# Compact the store, on a schedule in production: merge each dirty
# (shard, bucket, origin) partition into one file, drop tombstones older than
# LAKEHOUSE_GRACE_PERIOD_DAYS, then vacuum the replaced files
ftm-lakehouse -d my_dataset maintenance optimize

# Change the shard count of an existing dataset – rewrites every partition.
# Run with writers stopped, then `maintenance optimize`
ftm-lakehouse -d my_dataset maintenance shard --shards 8

# Bring a store written by an older version up to date; `--all` sweeps the
# catalog (the docker entrypoint runs it). Run with writers stopped
ftm-lakehouse -d my_dataset maintenance migrate
ftm-lakehouse maintenance migrate --all

`configure`

ftm-lakehouse -d <dataset> configure -c <config.yml> writes dataset configuration and nothing else. The yaml follows the dataset configuration schema. Only the keys in the file are written, so a partial file leaves the rest (notably shards) untouched; name and uri come from -d / the catalog. Each write keeps a versioned snapshot.

Set shards before the dataset is written to. Changing it on a store that holds rows leaves the existing rows under the old count while reads prune by the new one, so entity_id lookups miss them. Use maintenance shard --shards <n> instead – see Re-sharding.

`make`

make runs the whole pipeline; every stage is on by default:

Flag	Default	Effect
`-c <config.yml>`	–	Same merge-write as `configure`, before anything else
`--flush` / `--no-flush`	on	Flush outstanding journal statements into the parquet store
`--exports` / `--no-exports`	on	Write every export artifact – statements, entities, documents, parents, their diffs, the statistics and the index. `--no-exports` flushes only
`--optimize` / `--no-optimize`	on	Run optimize (merge + vacuum) before exporting; only with `--exports`. Exports are correct either way – an optimized store exports faster and takes less disk
`--force-optimize`	off	Re-merge every partition from scratch, dirty or not
`--force-exports`	off	Export even when the tags say the exports are fresh

`maintenance flush`

ftm-lakehouse -d <dataset> maintenance flush drains the journal into the parquet store and prints how many statements landed – the first stage of make, on its own.

--all drains every dataset in the catalog, printing a count per dataset and the total; it can't be combined with -d. An empty journal is a cheap no-op, so ftm-lakehouse maintenance flush --all works as a cron entry. The first dataset that fails aborts the sweep.

Commands

Generated from the CLI at docs build time:

`ftm-lakehouse`

Usage:

$ ftm-lakehouse [OPTIONS] COMMAND [ARGS]...

Options:

--version / --no-version: Show version [default: no-version]
--settings / --no-settings: Show current settings [default: no-settings]
--uri <str>: Lakehouse uri (path)
-d <str>: Dataset name (also known as foreign_id)
--dataset-uri <str>: Dataset lakehouse uri
--install-completion: Install completion for the current shell.
--show-completion: Show completion for the current shell, to copy it or customize the installation.
--help: Show this message and exit.

Commands:

ls: Show dataset names in the current catalog.
datasets: Show metadata for all datasets in the...
configure: Update the dataset configuration from a...
crawl: Crawl documents from local or remote...
make: Make or update a dataset.
export: Export the dataset: statements.csv,...
archive: Access the file archive
entities: Read and write FtM entities
maintenance: Dataset maintenance operations
statements: Read and write raw FtM statements
zfs: ZFS dataset management for the lakehouse

`ftm-lakehouse ls`

Show dataset names in the current catalog.

Usage:

$ ftm-lakehouse ls [OPTIONS]

Options:

-o <str>: [default: -]
--help: Show this message and exit.

`ftm-lakehouse datasets`

Show metadata for all datasets in the current catalog.

Usage:

$ ftm-lakehouse datasets [OPTIONS]

Options:

-o <str>: [default: -]
--help: Show this message and exit.

`ftm-lakehouse configure`

Update the dataset configuration from a yaml file.

Merges into the existing config.yml and keeps a versioned snapshot. Layout-affecting settings (shards) only take effect on a dataset that has not been written to yet.

Usage:

$ ftm-lakehouse configure [OPTIONS]

Options:

-c <str>: Configuration yml to store as config.yml [required]
--help: Show this message and exit.

`ftm-lakehouse crawl`

Crawl documents from local or remote sources into the archive.

Usage:

$ ftm-lakehouse crawl [OPTIONS] {uri}

Arguments:

uri: [required]

Options:

-o <str>: Write results to this destination [default: -]
--exclude <str>: Exclude paths glob pattern
--include <str>: Include paths glob pattern
--make-entities / --no-make-entities: Create entities from crawled files [default: make-entities]
--existing <overwrite|skip-checksum>: How to handle existing files [default: overwrite]
--help: Show this message and exit.

`ftm-lakehouse make`

Make or update a dataset.

By default this flushes the journal, optimizes the parquet store (merge + vacuum) and regenerates every export. Use --no-exports to flush only, or --no-optimize to export without the maintenance pass – exports read reconciled rows, so they are correct on an un-merged store too.

Usage:

$ ftm-lakehouse make [OPTIONS]

Options:

-c <str>: Configuration yml to store as config.yml
--flush / --no-flush: Flush outstanding journal statements to store [default: flush]
--exports / --no-exports: Write the exports – statements, entities, documents, parents, their diffs, the statistics and the index. --no-exports flushes only. [default: exports]
--optimize / --no-optimize: Optimize (merge) the statement store before exporting (with --exports). Reads reconcile un-merged rows, so exports are correct either way; an optimized store exports faster and reclaims disk. [default: optimize]
--force-optimize / --no-force-optimize: Re-merge every partition from scratch, dirty or not. [default: no-force-optimize]
--force-exports / --no-force-exports: Re-compute full exports pipeline even if up-to-date. [default: no-force-exports]
--help: Show this message and exit.

`ftm-lakehouse export`

Export the dataset: statements.csv, entities.ftm.json, documents.csv (per origin scope), parents.csv and statistics.json from a single pass over the entities, with their diffs, then index.json registering what they wrote.

Usage:

$ ftm-lakehouse export [OPTIONS]

Options:

--force / --no-force: Run regardless of freshness state. [default: no-force]
--help: Show this message and exit.

`ftm-lakehouse archive`

Access the file archive

Usage:

$ ftm-lakehouse archive [OPTIONS] COMMAND [ARGS]...

Options:

--help: Show this message and exit.

Commands:

get: Retrieve a file by content hash and write...
ls: List the checksums of all files in the...
download: Download all archive files to a local...

`ftm-lakehouse archive get`

Retrieve a file by content hash and write it to an output URI.

Usage:

$ ftm-lakehouse archive get [OPTIONS] {content_hash}

Arguments:

content_hash: [required]

Options:

-o <str>: [default: -]
--help: Show this message and exit.

`ftm-lakehouse archive ls`

List the checksums of all files in the dataset archive.

Usage:

$ ftm-lakehouse archive ls [OPTIONS]

Options:

-o <str>: [default: -]
--help: Show this message and exit.

`ftm-lakehouse archive download`

Download all archive files to a local directory.

Usage:

$ ftm-lakehouse archive download [OPTIONS]

Options:

-o <str>: [required]
--help: Show this message and exit.

`ftm-lakehouse entities`

Read and write FtM entities

Usage:

$ ftm-lakehouse entities [OPTIONS] COMMAND [ARGS]...

Options:

--help: Show this message and exit.

Commands:

iterate: Iterate entities from the parquet store as...
stream: Stream FtM entities from the pre-exported...
import: Bulk-import FtM entities straight into the...

`ftm-lakehouse entities iterate`

Iterate entities from the parquet store as FtM JSON lines.

Live read – the parquet table as it is post-flush, duplicates and tombstones reconciled. For the frozen pre-exported view use stream.

Filter with either -q (Aleph filter params) or --rql (nested RQL), as in ftmq q.

Usage:

$ ftm-lakehouse entities iterate [OPTIONS]

Options:

-o <str>: [default: -]
-q, --query <str>: Filter query string, e.g. 'filter:schema=Person&filter:group.countries=de'
--rql <str>: RQL query string (nested & | ~), e.g. 'and(eq(schema,Person),or(eq(group.countries,de),eq(group.countries,at)))'
--help: Show this message and exit.

`ftm-lakehouse entities stream`

Stream FtM entities from the pre-exported entities.ftm.json.

Usage:

$ ftm-lakehouse entities stream [OPTIONS]

Options:

-o <str>: [default: -]
--help: Show this message and exit.

`ftm-lakehouse entities import`

Bulk-import FtM entities straight into the parquet store, bypassing the Journal.

Can as well take unsorted fragments as input for migration from followthemoney-store into ftm-lakehouse keeping fragments and origin provenance. (Use ftmq fragments iterate-fragments -d ... for export.)

Usage:

$ ftm-lakehouse entities import [OPTIONS]

Options:

-i <str>: [default: -]
--origin <str>: Default data origin if the input carries none [default: bulk]
--override-origin / --no-override-origin: Force the given origin over input-carried origins [default: no-override-origin]
--role <str>: Default role (who asserts) if the input carries none
--bulk-size <int>: Number of statements buffered before flush to parquet. [default: 1000000]
--last-seen <%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S>: Default last_seen timestamp if the input has none
--unsafe: Fast path: map input straight to parquet rows, skipping FtM object construction and validation. Trusted input only.
--help: Show this message and exit.

`ftm-lakehouse maintenance`

Dataset maintenance operations

Usage:

$ ftm-lakehouse maintenance [OPTIONS] COMMAND [ARGS]...

Options:

--help: Show this message and exit.

Commands:

flush: Drain outstanding journal statements into...
optimize: Optimize the statement store: Merge dirty...
shard: Re-shard the statement store: rewrite it...
migrate: Apply the storage-layout migrations a...
unlock: Forcibly release the dataset write fence.

`ftm-lakehouse maintenance flush`

Drain outstanding journal statements into the parquet store.

With --all every dataset in the catalog is swept in turn. It addresses the whole catalog, so it is mutually exclusive with -d.

Duplicates and tombstones land as new rows – run maintenance optimize afterwards to collapse them. In api mode the flush is delegated to the server.

Usage:

$ ftm-lakehouse maintenance flush [OPTIONS]

Options:

--all: Sweep the whole catalog (not combinable with -d)
--help: Show this message and exit.

`ftm-lakehouse maintenance optimize`

Optimize the statement store: Merge dirty partitions into sorted files with statements deduped, tombstones applied and expired ones deleted (older than LAKEHOUSE_GRACE_PERIOD_DAYS). Reclaims storage afterwards via vacuuming. --force re-merges every partition from scratch.

Usage:

$ ftm-lakehouse maintenance optimize [OPTIONS]

Options:

--retention-hours <int>: Vacuum: retain obsolete files newer than this many hours. [default: 0]
--force / --no-force: Run regardless of freshness state. [default: no-force]
--help: Show this message and exit.

`ftm-lakehouse maintenance shard`

Re-shard the statement store: rewrite it onto a new shard count and record that count in config.yml.

The shard count is otherwise fixed at dataset creation, since every reader and writer resolves it from the config. Growing it is the fix for a dataset whose partitions have become too big to query well; see docs/architecture.md for how to size it.

A full rewrite of the store: the journal is drained first, every (bucket, origin) group is streamed into its new shard partitions, and the config is written last. Nothing is deduped or sorted on the way, so follow up with maintenance optimize.

Run with writers stopped – the write fence holds off parquet appends, but statements journalled under the old count and flushed afterwards land in the wrong partition.

Usage:

$ ftm-lakehouse maintenance shard [OPTIONS]

Options:

--shards <int>: Target number of entity-id hash shards (0/1 = single). [required]
--force / --no-force: Run regardless of freshness state. [default: no-force]
--help: Show this message and exit.

`ftm-lakehouse maintenance migrate`

Apply the storage-layout migrations a dataset hasn't seen yet.

Migrations bring a store written by an older version up to the layout the current code reads. Each is stamped when it completes, so this is a no-op on an up-to-date dataset and a half-finished run resumes where it stopped.

With --all every dataset in the catalog is swept in turn – how the docker entrypoint runs it. Run with writers stopped: a migration takes the exclusive write fence.

Usage:

$ ftm-lakehouse maintenance migrate [OPTIONS]

Options:

--all: Sweep the whole catalog (not combinable with -d)
--force / --no-force: Run regardless of freshness state. [default: no-force]
--help: Show this message and exit.

`ftm-lakehouse maintenance unlock`

Forcibly release the dataset write fence.

Use when a previous writer (merge, vacuum, re-shard, an export sweep) died with a lock held and later runs fail to acquire it. The locks are files at <dataset>/.LOCK and <dataset>/.LOCK-MERGE.

Confirm no process is actively writing before running – breaking a held lock can corrupt an in-flight write. No-op if no lock is held.

Local-only: the lock is a storage-side file – run this where the storage is directly accessible.

Usage:

$ ftm-lakehouse maintenance unlock [OPTIONS]

Options:

--help: Show this message and exit.

`ftm-lakehouse statements`

Read and write raw FtM statements

Usage:

$ ftm-lakehouse statements [OPTIONS] COMMAND [ARGS]...

Options:

--help: Show this message and exit.

Commands:

iterate: Iterate statements from the parquet store...
stream: Stream the pre-exported statements.csv...
import: Bulk-import raw statements (CSV) straight...
sql: Run a raw SQL query against the parquet...

`ftm-lakehouse statements iterate`

Iterate statements from the parquet store as CSV rows.

Live read – reflects current state of the parquet table. For the frozen pre-exported view use stream.

Usage:

$ ftm-lakehouse statements iterate [OPTIONS]

Options:

-o <str>: [default: -]
--help: Show this message and exit.

`ftm-lakehouse statements stream`

Stream the pre-exported statements.csv to the output.

Usage:

$ ftm-lakehouse statements stream [OPTIONS]

Options:

-o <str>: [default: -]
--help: Show this message and exit.

`ftm-lakehouse statements import`

Bulk-import raw statements (CSV) straight into the parquet store.

Mirrors entities import at the statement grain. Rows are parsed with the lakehouse read_csv_statements – which preserves the fragment supersession key (followthemoney's reader has no notion of it) – then buffered in EntityBuffer and handed to EntityRepository.write_stream. Bypasses the journal. With --unsafe, rows skip Statement construction entirely and map straight to parquet rows.

Usage:

$ ftm-lakehouse statements import [OPTIONS]

Options:

-i <str>: [default: -]
--origin <str>: Default data origin if the input carries none [default: bulk]
--override-origin / --no-override-origin: Force the given origin over input-carried origins [default: no-override-origin]
--role <str>: Default role (who asserts) if the input carries none
--bulk-size <int>: Number of statements buffered before flush to parquet. [default: 1000000]
--last-seen <%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S>: Default last_seen timestamp if the input has none
--unsafe: Fast path: map input straight to parquet rows, skipping FtM object construction and validation. Trusted input only.
--help: Show this message and exit.

`ftm-lakehouse statements sql`

Run a raw SQL query against the parquet store, rendered as a table – or, with --csv, streamed as csv to the output. Without a query argument the query is read from the input.

Two DuckDB views: statement – every physical row, duplicates and tombstones included – and statement_merged – the live rows a read sees, a plain scan on a merged store and the dedupe query otherwise. Filter statement where you can: on an unmerged store a predicate on statement_merged runs above the dedupe windows. Add a LIMIT when scanning large partitions.

Local-only: raw SQL is deliberately not exposed over the API – run this where the storage is directly accessible.

Usage:

$ ftm-lakehouse statements sql [OPTIONS] [query]

Arguments:

query: SQL query – read from -i when omitted

Options:

-i <str>: [default: -]
-o <str>: [default: -]
--csv: Write the result as csv to the output
--help: Show this message and exit.

`ftm-lakehouse zfs`

ZFS dataset management for the lakehouse

Usage:

$ ftm-lakehouse zfs [OPTIONS] COMMAND [ARGS]...

Options:

--help: Show this message and exit.

Commands:

init: Create ZFS datasets for a lakehouse dataset.

`ftm-lakehouse zfs init`

Create ZFS datasets for a lakehouse dataset.

Creates the parent, archive, and statements ZFS datasets with tuned properties under the given pool – the statements one under LAKEHOUSE_ZFS_STATEMENTS_POOL when that is set.

Usage:

$ ftm-lakehouse zfs init [OPTIONS] {dataset}

Arguments:

dataset: Dataset name to initialize [required]

Options:

-p, --pool <str>: ZFS pool path (or set LAKEHOUSE_ZFS_POOL)
--help: Show this message and exit.