Skip to content

Docs ftm-lakehouse on pypi PyPI Downloads PyPI - Python Version Python test and package pre-commit Coverage Status AGPLv3+ License Pydantic v2

ftm-lakehouse

ftm-lakehouse provides a data standard and archive storage for leaked data, private and public document collections and structured FollowTheMoney data.

It is a multi-tenant storage and retrieval layer for entity data, documents and their metadata. Tenants produce and consume it – for example investigraph, memorious, and platforms such as OpenAleph, ICIJ Datashare or Liquid Investigations.

What is a lakehouse?

Open formats

The file layout follows documented conventions and the statements are parquet, so third-party tools can populate and consume a lakehouse directly. Data, change history and versions all live in the storage backend; reading needs no running service. Writers use a SQL write-ahead journal (sqlite, or postgres in production).

Core Components

Entities

The entities interface is the primary way to work with FollowTheMoney data:

  • Write entities through a buffered journal
  • Query them from a Delta Lake statement store
  • Export them as JSON, CSV and statistics

Entities are stored as statements: one property value of one entity from one source (entity_id, schema, prop, value, dataset, plus provenance such as origin). That keeps the provenance of every value, and lets sources merge and update incrementally. A lakehouse dataset does no entity resolution, so canonical_id is not stored – it always equals entity_id.

from ftmq.query import C, Query

from ftm_lakehouse import ensure_dataset, get_entities

ensure_dataset("my_dataset")
entities = get_entities("my_dataset")

# Write entities through the journal (buffered, then flushed to parquet)
with entities.writer(origin="import") as writer:
    for entity in source:
        writer.add_entity(entity)
entities.flush()

# Read back
entity = entities.get("entity-id-123")

# Query the parquet store
for entity in entities.query(Query(C(origin="crawl"))):
    process(entity)

The statement store is partitioned by (shard, bucket, origin) and written append-only. Reads reconcile duplicates and tombstones; optimize compacts them – see Maintenance.

Archive

The archive interface manages source files, content-addressed by SHA256 checksum, so identical files are stored once. Files become FollowTheMoney entities as well.

from ftm_lakehouse import get_archive

archive = get_archive("my_dataset")

# Archive a file
checksum = archive.store("/path/to/document.pdf")

# Retrieve file content
with archive.open(checksum) as fh:
    content = fh.read()

Installation

Requires Python 3.12 or later.

pip install ftm-lakehouse

Optional extras:

pip install "ftm-lakehouse[postgres]"  # postgres journal (ADBC + psycopg)
pip install "ftm-lakehouse[api]"       # the lakehouse API server
pip install "ftm-lakehouse[s3]"        # S3-compatible object storage (s3fs)
pip install "ftm-lakehouse[gcs]"       # Google Cloud Storage (gcsfs)
pip install "ftm-lakehouse[azure]"     # Azure Blob Storage (adlfs)
pip install "ftm-lakehouse[http]"      # HTTP(S)-backed api stores (aiohttp)

Extras combine, e.g. pip install "ftm-lakehouse[s3,postgres]".

Quickstart

>> Get started here

Background

The design grew out of the FollowTheMoney data lake RFC discussion and prior art in mmmeta, Aleph's servicelayer archive, OpenSanctions dataset metadata and nomenklatura statements.

For contributing, development setup and testing see the repository README.

leakrfc (predecessor), (c) 2024 investigativedata.io

ftm-lakehouse, (c) 2024 investigativedata.io

ftm-lakehouse, (c) 2025-2026 Data and Research Center - DARC

ftm-lakehouse is licensed under the AGPLv3 or later license.