Skip to content

CLI Reference

ftm-lakehouse provides a Typer-based command-line interface organised into sub-command groups.

ftm-lakehouse [OPTIONS] <group> <command> [ARGS]
Group Purpose
archive Content-addressed file storage
entities Read and write FtM entities
statements Read and write raw FtM statements
maintenance Storage maintenance (flush, optimize, unlock)
zfs ZFS dataset management

Top-level (no group), as frequently-used shortcuts: ls (dataset names), datasets (metadata), configure (write dataset configuration), make (build/update a dataset), export (produce the exports, or a single kind), crawl (ingest documents into the archive).

Environment variables configure storage locations and behavior – see the configuration reference.

Examples

export LAKEHOUSE_URI=./data

# Initialise the dataset – no data yet, so skip the exports pipeline
ftm-lakehouse -d my_dataset make --no-exports

# Record its configuration (title, summary, shards, compression, ...)
ftm-lakehouse -d my_dataset configure -c config.yml

# Crawl some files
ftm-lakehouse -d my_dataset crawl /path/to/documents

# Bulk-load a pre-built entities.ftm.json (skips the journal)
cat entities.ftm.json | ftm-lakehouse -d my_dataset entities import

# ... several times faster for trusted input (same statement ids and
# namespace stripping as the safe path, no FtM object construction):
cat entities.ftm.json | ftm-lakehouse -d my_dataset entities import --unsafe

# Flush the journal, optimize the store and build all exports – the default
ftm-lakehouse -d my_dataset make

# The export sweep on its own – every streamed artifact from one pass
ftm-lakehouse -d my_dataset export

# ... or a single kind
ftm-lakehouse -d my_dataset export statistics

# Drain the journal on its own – one dataset, or the whole catalog
ftm-lakehouse -d my_dataset maintenance flush
ftm-lakehouse maintenance flush --all

# Maintenance – async, run on a schedule in production. Merges duplicates per
# (shard, bucket, origin) partition, drops tombstones older than
# LAKEHOUSE_GRACE_PERIOD_DAYS, bin-packs small files, removes obsolete ones –
# always in one pass, held under the dataset write fence.
ftm-lakehouse -d my_dataset maintenance optimize

# Change the shard count of an existing dataset: rewrites every partition,
# then records the new count in config.yml. Run with writers stopped, and
# follow up with `maintenance optimize`.
ftm-lakehouse -d my_dataset maintenance shard --shards 8

# Bring a store written by an older version up to the current layout. No-op on
# an up-to-date dataset; `--all` sweeps the catalog (what the docker entrypoint
# runs). Run with writers stopped.
ftm-lakehouse -d my_dataset maintenance migrate
ftm-lakehouse maintenance migrate --all

configure

ftm-lakehouse -d <dataset> configure -c <config.yml> writes dataset configuration and nothing else – no flush, no exports. The yaml follows the dataset configuration schema; only the keys it actually contains are written, so a partial file leaves everything else (notably shards) untouched. name and uri are taken from -d / the catalog and ignored if present in the file. Each write keeps a versioned snapshot.

Layout-affecting settings (shards) belong in the config before a dataset is written to. Setting a different value on a store that already holds rows splits it: rows written from then on are placed under the new count, the rows already there keep their old partitions, and reads prune by the new count – so an entity_id-filtered query silently misses whichever half didn't move. maintenance shard --shards <n> is the operation that moves them – see Re-sharding.

make

make is the whole pipeline in one command; every stage is on by default and can be switched off:

Flag Default Effect
-c <config.yml> – Same merge-write as configure, before anything else
--flush / --no-flush on Flush outstanding journal statements into the parquet store
--exports / --no-exports on Build statements/entities/documents/statistics exports and diffs. With --no-exports only index.json is refreshed
--optimize / --no-optimize on Run the optimize pass before exporting (only applies with --exports)
--force-optimize off Optimize even when the store is already up-to-date
--force-exports off Re-compute the exports pipeline even when the tags say it is fresh

maintenance flush

ftm-lakehouse -d <dataset> maintenance flush drains outstanding journal statements into the parquet store and prints how many landed. It is the same drain make runs as its first stage, on its own – no optimize, no exports, so duplicates and tombstones stay as new rows until the next optimize.

--all sweeps every dataset in the catalog instead, printing a count per dataset plus the total. It addresses the whole catalog, so combining it with -d is an error rather than a silent override. Datasets with an empty journal are a cheap no-op – the drain probes for rows before it rotates anything – which makes ftm-lakehouse maintenance flush --all a reasonable cron entry for a lakehouse whose writers leave data in the journal. It fails fast: the first dataset that errors aborts the sweep.

Commands

The following reference is generated from the CLI itself at docs build time:

ftm-lakehouse

Usage:

$ ftm-lakehouse [OPTIONS] COMMAND [ARGS]...

Options:

  • --version / --no-version: Show version [default: no-version]
  • --settings / --no-settings: Show current settings [default: no-settings]
  • --uri <str>: Lakehouse uri (path)
  • -d <str>: Dataset name (also known as foreign_id)
  • --dataset-uri <str>: Dataset lakehouse uri
  • --install-completion: Install completion for the current shell.
  • --show-completion: Show completion for the current shell, to copy it or customize the installation.
  • --help: Show this message and exit.

Commands:

  • ls: Show dataset names in the current catalog.
  • datasets: Show metadata for all datasets in the...
  • configure: Update the dataset configuration from a...
  • crawl: Crawl documents from local or remote...
  • make: Make or update a dataset.
  • export: Export the dataset: all (the default –...
  • archive: Access the file archive
  • entities: Read and write FtM entities
  • maintenance: Dataset maintenance operations
  • statements: Read and write raw FtM statements
  • zfs: ZFS dataset management for the lakehouse

ftm-lakehouse ls

Show dataset names in the current catalog.

Usage:

$ ftm-lakehouse ls [OPTIONS]

Options:

  • -o <str>: [default: -]
  • --help: Show this message and exit.

ftm-lakehouse datasets

Show metadata for all datasets in the current catalog.

Usage:

$ ftm-lakehouse datasets [OPTIONS]

Options:

  • -o <str>: [default: -]
  • --help: Show this message and exit.

ftm-lakehouse configure

Update the dataset configuration from a yaml file.

Merges into the existing config.yml and keeps a versioned snapshot. Layout-affecting settings (shards) only take effect on a dataset that has not been written to yet.

Usage:

$ ftm-lakehouse configure [OPTIONS]

Options:

  • -c <str>: Configuration yml to store as config.yml [required]
  • --help: Show this message and exit.

ftm-lakehouse crawl

Crawl documents from local or remote sources into the archive.

Usage:

$ ftm-lakehouse crawl [OPTIONS] {uri}

Arguments:

  • uri: [required]

Options:

  • -o <str>: Write results to this destination [default: -]
  • --exclude <str>: Exclude paths glob pattern
  • --include <str>: Include paths glob pattern
  • --make-entities / --no-make-entities: Create entities from crawled files [default: make-entities]
  • --existing <overwrite|skip-path|skip-checksum>: How to handle existing files [default: overwrite]
  • --help: Show this message and exit.

ftm-lakehouse make

Make or update a dataset.

By default this flushes the journal, optimizes the parquet store and regenerates all exports. Use --no-exports to only flush and refresh index.json, or --no-optimize to export without the maintenance pass.

Usage:

$ ftm-lakehouse make [OPTIONS]

Options:

  • -c <str>: Configuration yml to store as config.yml
  • --flush / --no-flush: Flush outstanding journal statements to store [default: flush]
  • --exports / --no-exports: Include export statements/entities and diffs, compute stats [default: exports]
  • --optimize / --no-optimize: Optimize parquet store beforehand when using --exports [default: optimize]
  • --force-optimize / --no-force-optimize: Re-optimize even if up-to-date. [default: no-force-optimize]
  • --force-exports / --no-force-exports: Re-compute full exports pipeline even if up-to-date. [default: no-force-exports]
  • --help: Show this message and exit.

ftm-lakehouse export

Export the dataset: all (the default – statements.csv, entities.ftm.json and documents.csv from a single pass, with their diffs), or one of statements (statements.csv), entities (entities.ftm.json), documents (documents.csv), statistics (statistics.json), index (index.json).

Usage:

$ ftm-lakehouse export [OPTIONS] [kind]:<all|statements|entities|documents|statistics|index>

Arguments:

  • kind:<all|statements|entities|documents|statistics|index>: Which export to produce. [default: all]

Options:

  • --force / --no-force: Run regardless of freshness state. [default: no-force]
  • --help: Show this message and exit.

ftm-lakehouse archive

Access the file archive

Usage:

$ ftm-lakehouse archive [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • get: Retrieve a file by content hash and write...
  • head: Retrieve all metadata objects for a...
  • ls: List all files in the dataset archive.
  • download: Download all archive files to a local...

ftm-lakehouse archive get

Retrieve a file by content hash and write it to an output URI.

Usage:

$ ftm-lakehouse archive get [OPTIONS] {content_hash}

Arguments:

  • content_hash: [required]

Options:

  • -o <str>: [default: -]
  • --help: Show this message and exit.

ftm-lakehouse archive head

Retrieve all metadata objects for a content hash and write them out.

Usage:

$ ftm-lakehouse archive head [OPTIONS] {content_hash}

Arguments:

  • content_hash: [required]

Options:

  • -o <str>: [default: -]
  • --help: Show this message and exit.

ftm-lakehouse archive ls

List all files in the dataset archive.

Usage:

$ ftm-lakehouse archive ls [OPTIONS]

Options:

  • -o <str>: [default: -]
  • --keys / --no-keys: Show only keys [default: no-keys]
  • --checksums / --no-checksums: Show only checksums [default: no-checksums]
  • --help: Show this message and exit.

ftm-lakehouse archive download

Download all archive files to a local directory.

Usage:

$ ftm-lakehouse archive download [OPTIONS]

Options:

  • -o <str>: [required]
  • --help: Show this message and exit.

ftm-lakehouse entities

Read and write FtM entities

Usage:

$ ftm-lakehouse entities [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • iterate: Iterate entities from the parquet store as...
  • stream: Stream FtM entities from the pre-exported...
  • import: Bulk-import FtM entities straight into the...

ftm-lakehouse entities iterate

Iterate entities from the parquet store as FtM JSON lines.

Live read – reflects current state of the parquet table post-flush, but correctness is only guaranteed after maintenance optimize. For the frozen pre-exported view use stream.

Filter with either -q (Aleph filter params) or --rql (nested RQL), as in ftmq q.

Usage:

$ ftm-lakehouse entities iterate [OPTIONS]

Options:

  • -o <str>: [default: -]
  • -q, --query <str>: Filter query string, e.g. 'filter:schema=Person&filter:group.countries=de'
  • --rql <str>: RQL query string (nested & | ~), e.g. 'and(eq(schema,Person),or(eq(group.countries,de),eq(group.countries,at)))'
  • --help: Show this message and exit.

ftm-lakehouse entities stream

Stream FtM entities from the pre-exported entities.ftm.json.

Usage:

$ ftm-lakehouse entities stream [OPTIONS]

Options:

  • -o <str>: [default: -]
  • --help: Show this message and exit.

ftm-lakehouse entities import

Bulk-import FtM entities straight into the parquet store, bypassing the Journal.

Can as well take unsorted fragments as input for migration from followthemoney-store into ftm-lakehouse keeping fragments and origin provenance. (Use ftmq fragments iterate-fragments -d ... for export.)

Usage:

$ ftm-lakehouse entities import [OPTIONS]

Options:

  • -i <str>: [default: -]
  • --origin <str>: Default data origin if the input carries none [default: bulk]
  • --override-origin / --no-override-origin: Force the given origin over input-carried origins [default: no-override-origin]
  • --role <str>: Default role (who asserts) if the input carries none
  • --bulk-size <int>: Number of statements buffered before flush to parquet. [default: 1000000]
  • --last-seen <%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S>: Default last_seen timestamp if the input has none
  • --unsafe: Fast path: map input straight to parquet rows, skipping FtM object construction and validation. Trusted input only.
  • --help: Show this message and exit.

ftm-lakehouse maintenance

Dataset maintenance operations

Usage:

$ ftm-lakehouse maintenance [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • flush: Drain outstanding journal statements into...
  • optimize: Optimize the statement store: collapse...
  • shard: Re-shard the statement store: rewrite it...
  • migrate: Apply the storage-layout migrations a...
  • unlock: Forcibly release the dataset write fence.

ftm-lakehouse maintenance flush

Drain outstanding journal statements into the parquet store.

With --all every dataset in the catalog is swept in turn. It addresses the whole catalog, so it is mutually exclusive with -d.

Duplicates and tombstones land as new rows – run maintenance optimize afterwards to collapse them. In api mode the flush is delegated to the server.

Usage:

$ ftm-lakehouse maintenance flush [OPTIONS]

Options:

  • --all: Sweep the whole catalog (not combinable with -d)
  • --help: Show this message and exit.

ftm-lakehouse maintenance optimize

Optimize the statement store: collapse duplicates and reap expired tombstones, bin-pack small parquet files, delete obsolete files.

Tombstones older than LAKEHOUSE_GRACE_PERIOD_DAYS are dropped. Each step is held under the dataset write fence.

Usage:

$ ftm-lakehouse maintenance optimize [OPTIONS]

Options:

  • --retention-hours <int>: Vacuum: retain obsolete files newer than this many hours. [default: 0]
  • --force / --no-force: Run regardless of freshness state. [default: no-force]
  • --help: Show this message and exit.

ftm-lakehouse maintenance shard

Re-shard the statement store: rewrite it onto a new shard count and record that count in config.yml.

The shard count is otherwise fixed at dataset creation, since every reader and writer resolves it from the config. Growing it is the fix for a dataset whose partitions have become too big to query well; see docs/architecture.md for how to size it.

A full rewrite of the store: the journal is drained first, every (bucket, origin) group is streamed into its new shard partitions, and the config is written last. Nothing is deduped or sorted on the way, so follow up with maintenance optimize.

Run with writers stopped – the write fence holds off parquet appends, but statements journalled under the old count and flushed afterwards land in the wrong partition.

Usage:

$ ftm-lakehouse maintenance shard [OPTIONS]

Options:

  • --shards <int>: Target number of entity-id hash shards (0/1 = single). [required]
  • --force / --no-force: Run regardless of freshness state. [default: no-force]
  • --help: Show this message and exit.

ftm-lakehouse maintenance migrate

Apply the storage-layout migrations a dataset hasn't seen yet.

Migrations bring a store written by an older version up to the layout the current code reads. Each is stamped when it completes, so this is a no-op on an up-to-date dataset and a half-finished run resumes where it stopped.

With --all every dataset in the catalog is swept in turn – how the docker entrypoint runs it. Run with writers stopped: a migration takes the exclusive write fence.

Usage:

$ ftm-lakehouse maintenance migrate [OPTIONS]

Options:

  • --all: Sweep the whole catalog (not combinable with -d)
  • --force / --no-force: Run regardless of freshness state. [default: no-force]
  • --help: Show this message and exit.

ftm-lakehouse maintenance unlock

Forcibly release the dataset write fence.

Use when a previous writer (flush / merge / compact / vacuum / append) died with the lock held and subsequent writes hang trying to acquire it. The lock is just a file at <dataset>/.LOCK.

Confirm no process is actively writing before running – breaking a held lock can corrupt an in-flight write. No-op if no lock is held.

Local-only: the lock is a storage-side file – run this where the storage is directly accessible.

Usage:

$ ftm-lakehouse maintenance unlock [OPTIONS]

Options:

  • --help: Show this message and exit.

ftm-lakehouse statements

Read and write raw FtM statements

Usage:

$ ftm-lakehouse statements [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • iterate: Iterate statements from the parquet store...
  • stream: Stream the pre-exported statements.csv...
  • import: Bulk-import raw statements (CSV) straight...
  • sql: Run a raw SQL query against the parquet...

ftm-lakehouse statements iterate

Iterate statements from the parquet store as CSV rows.

Live read – reflects current state of the parquet table. For the frozen pre-exported view use stream.

Usage:

$ ftm-lakehouse statements iterate [OPTIONS]

Options:

  • -o <str>: [default: -]
  • --help: Show this message and exit.

ftm-lakehouse statements stream

Stream the pre-exported statements.csv to the output.

Usage:

$ ftm-lakehouse statements stream [OPTIONS]

Options:

  • -o <str>: [default: -]
  • --help: Show this message and exit.

ftm-lakehouse statements import

Bulk-import raw statements (CSV) straight into the parquet store.

Mirrors entities import at the statement grain. Rows are parsed with the lakehouse read_csv_statements – which preserves the fragment supersession key (followthemoney's reader has no notion of it) – then buffered in EntityBuffer and handed to EntityRepository.write_batches as one packed table. Bypasses the journal. With --unsafe, rows skip Statement construction entirely and map straight to parquet rows.

Usage:

$ ftm-lakehouse statements import [OPTIONS]

Options:

  • -i <str>: [default: -]
  • --origin <str>: Default data origin if the input carries none [default: bulk]
  • --override-origin / --no-override-origin: Force the given origin over input-carried origins [default: no-override-origin]
  • --role <str>: Default role (who asserts) if the input carries none
  • --bulk-size <int>: Number of statements buffered before flush to parquet. [default: 1000000]
  • --last-seen <%Y-%m-%d|%Y-%m-%dT%H:%M:%S|%Y-%m-%d %H:%M:%S>: Default last_seen timestamp if the input has none
  • --unsafe: Fast path: map input straight to parquet rows, skipping FtM object construction and validation. Trusted input only.
  • --help: Show this message and exit.

ftm-lakehouse statements sql

Run a raw SQL query against the parquet store, rendered as a table.

Queries the registered DuckDB views – statement (deduped-live) and statement_raw (physical rows). Results print as a rich table; add a LIMIT when scanning large partitions.

Local-only: raw SQL is deliberately not exposed over the API – run this where the storage is directly accessible.

Usage:

$ ftm-lakehouse statements sql [OPTIONS] {query}

Arguments:

  • query: [required]

Options:

  • --help: Show this message and exit.

ftm-lakehouse zfs

ZFS dataset management for the lakehouse

Usage:

$ ftm-lakehouse zfs [OPTIONS] COMMAND [ARGS]...

Options:

  • --help: Show this message and exit.

Commands:

  • init: Create ZFS datasets for a lakehouse dataset.

ftm-lakehouse zfs init

Create ZFS datasets for a lakehouse dataset.

Creates the parent, archive, and statements ZFS datasets with tuned properties under the given pool.

Usage:

$ ftm-lakehouse zfs init [OPTIONS] {dataset}

Arguments:

  • dataset: Dataset name to initialize [required]

Options:

  • -p, --pool <str>: ZFS pool path (or set LAKEHOUSE_ZFS_POOL)
  • --help: Show this message and exit.