Skip to content

Conventions

ftm-lakehouse is convention-driven: the path layout, artifact names and freshness tags are stable contracts – third-party tools can populate or consume a lakehouse by following them, without going through this library.

Storage Layout

On-disk (or cloud storage) layout of a lakehouse storage root:

lakehouse/
└── {dataset}/
    ├── config.yml                # Dataset configuration (shards, compression, ...)
    ├── index.json                # Published dataset index with statistics
    ├── .LOCK                     # Exclusive maintenance fence
    ├── .LOCK-APPENDS/            # In-flight append markers (shared fence)
    ├── archive/                  # Content-addressed file storage
    │   └── {ch[0:2]}/{ch[2:4]}/{ch[4:6]}/{checksum}/
    │       ├── blob              # Raw file content (stored once)
    │       ├── {file_id}.json    # File metadata (one per source path)
    │       └── {origin}.txt      # Extracted text (one per engine)
    ├── statements/               # Delta Lake parquet store
    │   ├── _delta_log/
    │   └── shard={hex}/bucket={bucket}/origin={origin}/*.parquet
    ├── entities.ftm.json[.gz|.zst]   # Aggregated entities export
    ├── exports/
    │   ├── statements.csv[.gz|.zst]  # Sorted statements export
    │   ├── statistics.json           # Entity counts, facets
    │   └── documents.csv             # Document metadata
    ├── diffs/                    # Timestamped delta diff exports
    ├── versions/                 # Versioned snapshots (config, index, ...)
    │   └── YYYY/MM/{timestamp}/
    ├── tags/{tenant}/            # Freshness tags (workflow state)
    └── jobs/
        └── runs/{job_type}/{timestamp}.json

Freshness Tags

Operations use tags to track freshness and skip unnecessary work – is_latest(key, dependencies) returns True when key is newer than all its dependencies:

Tag Set by Meaning
journal/last_updated Statement writes Journal has uncommitted data
journal/last_flushed Flush operation Journal was flushed
statements/last_updated Flush / append Rows landed in the parquet store – not canonical yet
statements/last_optimized merge, on completion Canonical content changed. The clock every export, statistic and diff goes stale against
archive/last_updated File archive New file was archived
exports/statements.csv, entities.ftm.json, exports/documents.csv, exports/statistics.json, index.json Export operations Export target keys double as their freshness tags
operations/crawl/last_run Crawl operation Last crawl execution

Compression suffixes

When a dataset configures compression (gz / zst in config.yml), the streaming export artifacts carry the codec suffix – entities.ftm.json.zst, exports/statements.csv.zst – and index.json advertises the resulting names and urls. index.json and statistics.json themselves are always plain JSON.

Path conventions

All path construction goes through ftm_lakehouse.core.conventions.path – rendered here so the constants stay in sync with the code:

Path conventions for the FollowTheMoney data lakehouse.

The fundamental idea is to have a convention-based file system layout with well-known paths for metadata and information interchange between processing stages.

All paths are dataset-relative unless otherwise noted.

Dataset Layout

::

lakehouse/
    index.json                          # catalog index
    config.yml                          # catalog configuration
    versions/                           # versioned snapshots
        YYYY/MM/YYYY-MM-DDTHH:MM:SS/
            index.json
            config.yml

    [dataset]/
        index.json                      # dataset index
        config.yml                      # dataset configuration

        versions/                       # versioned snapshots
            YYYY/MM/...

        .LOCK                           # dataset-wide maintenance lock
        .LOCK-APPENDS/                  # in-flight append markers
        locks/{tenant}/                 # operation-specific locks
        tags/{tenant}/                  # workflow state / cache

        archive/                        # content-addressed file storage
            ab/cd/ef/{checksum}/        # SHA256 split into segments
                blob                    # file blob (stored once)
                {file_id}.json          # metadata (one per source path)
                {origin}.txt            # extracted text (one per engine)

        mappings/
            {content_hash}/
                mapping.yml             # current CSV mapping configuration
                versions/               # versioned snapshots
                    YYYY/MM/...

        statements/                     # statement store (shard-partitioned)
            shard={shard}/
                bucket={bucket}/
                    origin={origin}/
                        *.parquet

        entities.ftm.json[.zst|gzip]    # aggregated entities export

        exports/
            statistics.json             # entity counts, facets
            statements.csv              # sorted statements
            documents.csv               # document metadata
            graph.cypher                # neo4j export (optional)

        diffs/
            entities.ftm.json/
                20240116T103000000000Z.delta.json  # entities delta
            exports/
                documents.csv/
                    20240116T103000000000Z.diff.csv  # documents delta

        jobs/
            runs/
                {job_type}/
                    {timestamp}.json    # job run results

ARCHIVE = 'archive' module-attribute

Base path for archive

ARCHIVE_BLOB = 'blob' module-attribute

blob filename within checksum directory

CONFIG = 'config.yml' module-attribute

user editable config filename

DIFFS = 'diffs' module-attribute

Base path for diff exports

DIFFS_DOCUMENTS = f'{DIFFS}/{EXPORTS_DOCUMENTS}' module-attribute

Base path for document.csv diffs

DIFFS_ENTITIES = f'{DIFFS}/{ENTITIES_JSON}' module-attribute

Base path for entities.ftm.json diffs

ENTITIES_JSON = 'entities.ftm.json' module-attribute

aggregated entities file name

EXPORTS = 'exports' module-attribute

Base path for exports

EXPORTS_CYPHER = f'{EXPORTS}/graph.cypher' module-attribute

neo4j data export file path

EXPORTS_DOCUMENTS = f'{EXPORTS}/documents.csv' module-attribute

documents metadata to stream

EXPORTS_STATEMENTS = f'{EXPORTS}/statements.csv' module-attribute

complete sorted statements file path

EXPORTS_STATISTICS = f'{EXPORTS}/{STATISTICS}' module-attribute

entity counts, pre-computed facts file path

INDEX = 'index.json' module-attribute

generated index filename

JOBS = 'jobs' module-attribute

Job data prefix

JOB_RUNS = f'{JOBS}/runs' module-attribute

Job runs result storage prefix

LOCK = '.LOCK' module-attribute

dataset-wide maintenance lock key name

LOCKS = 'locks' module-attribute

Base path for storing locks

LOCK_APPENDS = '.LOCK-APPENDS' module-attribute

Prefix for per-writer append marker keys (shared side of the write fence)

MAPPING = 'mapping.yml' module-attribute

mapping file name

MAPPINGS = 'mappings' module-attribute

Base path for storing mappings

STATEMENTS = 'statements' module-attribute

Base path for storing statement data (partitioned by shard, bucket, origin)

STATISTICS = 'statistics.json' module-attribute

computed statistics filename

TAGS = 'tags' module-attribute

Base path for dataset tags cache

TENANT = 'lakehouse' module-attribute

Default tenant name

TS_FORMAT = '%Y%m%dT%H%M%S%fZ' module-attribute

Global format for timestamps in files

VERSIONS = 'versions' module-attribute

Base path for versions

archive_blob(checksum)

Get the blob path for a file in the archive.

Layout: archive/5a/6a/cf/5a6acf229ba576d9a40b09292595658bbb74ef56/blob

Parameters:

Name Type Description Default
checksum str

SHA256 checksum of file

required
Source code in ftm_lakehouse/core/conventions/path.py
def archive_blob(checksum: str) -> str:
    """
    Get the blob path for a file in the archive.

    Layout: archive/5a/6a/cf/5a6acf229ba576d9a40b09292595658bbb74ef56/blob

    Args:
        checksum: SHA256 checksum of file
    """
    return f"{archive_prefix(checksum)}/{ARCHIVE_BLOB}"

archive_meta(checksum, file_id)

Get a file metadata path for a specific file instance.

Multiple files with the same checksum but different source paths each get their own metadata file, keyed by their File.id.

Layout: archive/5a/6a/cf/.../file-abc123.json

Parameters:

Name Type Description Default
checksum str

SHA256 checksum of file

required
file_id str

The File.id (hash of source path + checksum)

required

Raises:

Type Description
ValueError

If checksum or file_id is malformed.

Source code in ftm_lakehouse/core/conventions/path.py
def archive_meta(checksum: str, file_id: str) -> str:
    """
    Get a file metadata path for a specific file instance.

    Multiple files with the same checksum but different source paths
    each get their own metadata file, keyed by their File.id.

    Layout: archive/5a/6a/cf/.../file-abc123.json

    Args:
        checksum: SHA256 checksum of file
        file_id: The File.id (hash of source path + checksum)

    Raises:
        ValueError: If ``checksum`` or ``file_id`` is malformed.
    """
    safe_name(file_id, "file_id")
    return f"{archive_prefix(checksum)}/{file_id}.json"

archive_prefix(checksum)

Get the directory path for a file in the archive.

Layout: archive/5a/6a/cf/5a6acf229ba576d9a40b09292595658bbb74ef56/

Parameters:

Name Type Description Default
checksum str

SHA256 checksum of file

required
Source code in ftm_lakehouse/core/conventions/path.py
def archive_prefix(checksum: str) -> str:
    """
    Get the directory path for a file in the archive.

    Layout: archive/5a/6a/cf/5a6acf229ba576d9a40b09292595658bbb74ef56/

    Args:
        checksum: SHA256 checksum of file
    """
    return f"{ARCHIVE}/{make_checksum_key(checksum)}"

archive_txt(checksum, origin)

Get a file text content path for a specific extraction origin.

Multiple text extractions can exist per file, keyed by origin (e.g., different OCR engines or extraction methods).

Layout: archive/5a/6a/cf/.../{origin}.txt

Parameters:

Name Type Description Default
checksum str

SHA256 checksum of file

required
origin str

The extraction origin/engine name

required

Raises:

Type Description
ValueError

If checksum or origin is malformed.

Source code in ftm_lakehouse/core/conventions/path.py
def archive_txt(checksum: str, origin: str) -> str:
    """
    Get a file text content path for a specific extraction origin.

    Multiple text extractions can exist per file, keyed by origin
    (e.g., different OCR engines or extraction methods).

    Layout: archive/5a/6a/cf/.../{origin}.txt

    Args:
        checksum: SHA256 checksum of file
        origin: The extraction origin/engine name

    Raises:
        ValueError: If ``checksum`` or ``origin`` is malformed.
    """
    validate_origin(origin)
    return f"{archive_prefix(checksum)}/{origin}.txt"

documents_diff(ts=None)

Get path for a documents diff export file.

Layout: diffs/exports/documents.csv/{ts}.diff.csv

Parameters:

Name Type Description Default
ts datetime | None

Compact timestamp (YYYYMMDDTHHMMSSZ), defaults to current time

None

Returns:

Type Description
str

Path to diff file

Source code in ftm_lakehouse/core/conventions/path.py
def documents_diff(ts: datetime | None = None) -> str:
    """
    Get path for a documents diff export file.

    Layout: diffs/exports/documents.csv/{ts}.diff.csv

    Args:
        ts: Compact timestamp (YYYYMMDDTHHMMSSZ), defaults to current time

    Returns:
        Path to diff file
    """
    if ts is None:
        ts = utc_now()
    ts_iso = ts.strftime(TS_FORMAT)
    return f"{DIFFS_DOCUMENTS}/{ts_iso}.diff.csv"

entities_diff(ts=None, suffix=None)

Get path for an entities diff export file.

Layout: diffs/entities.ftm.json/{ts}.delta.json

The delta file contains line-based JSON with operation envelopes

{"op": "ADD", "entity": {"id": "...", "schema": "...", "properties": {...}}} {"op": "MOD", "entity": {"id": "...", "schema": "...", "properties": {...}}} {"op": "DEL", "entity": {"id": "..."}}

Parameters:

Name Type Description Default
ts datetime | None

Compact timestamp (YYYYMMDDTHHMMSSZ), defaults to current time

None

Returns:

Type Description
str

Path to delta file

Source code in ftm_lakehouse/core/conventions/path.py
def entities_diff(ts: datetime | None = None, suffix: str | None = None) -> str:
    """
    Get path for an entities diff export file.

    Layout: diffs/entities.ftm.json/{ts}.delta.json

    The delta file contains line-based JSON with operation envelopes:
        {"op": "ADD", "entity": {"id": "...", "schema": "...", "properties": {...}}}
        {"op": "MOD", "entity": {"id": "...", "schema": "...", "properties": {...}}}
        {"op": "DEL", "entity": {"id": "..."}}

    Args:
        ts: Compact timestamp (YYYYMMDDTHHMMSSZ), defaults to current time

    Returns:
        Path to delta file
    """
    if ts is None:
        ts = utc_now()
    ts_iso = ts.strftime(TS_FORMAT)
    path = f"{DIFFS_ENTITIES}/{ts_iso}.delta.json"
    if suffix:
        path = f"{path}.{suffix}"
    return path

entity_shard(entity_id, shards)

Hex shard key for an entity id under a uniform shard count.

Uses the first 8 hex chars of the entity_id hash, taken mod shards, then zero-padded to shard_hex_width(shards).

Source code in ftm_lakehouse/core/conventions/path.py
def entity_shard(entity_id: str, shards: int) -> str:
    """Hex shard key for an entity id under a uniform shard count.

    Uses the first 8 hex chars of the entity_id hash, taken mod ``shards``,
    then zero-padded to ``shard_hex_width(shards)``.
    """
    if shards <= 1:
        return "0"
    bucket = int(hash_data(entity_id)[:8], 16) % shards
    return f"{bucket:0{shard_hex_width(shards)}x}"

lock(*parts, tenant=TENANT)

Generate a path to store a lock

Source code in ftm_lakehouse/core/conventions/path.py
def lock(*parts: str, tenant: str | None = TENANT) -> str:
    """Generate a path to store a lock"""
    return join_relpaths(LOCKS, tenant or TENANT, *parts)

mapping(content_hash)

Get the mapping.yml path for the given file SHA256.

Layout: mappings/{content_hash}/mapping.yml

Source code in ftm_lakehouse/core/conventions/path.py
def mapping(content_hash: str) -> str:
    """
    Get the mapping.yml path for the given file SHA256.

    Layout: mappings/{content_hash}/mapping.yml
    """
    return f"{MAPPINGS}/{content_hash}/{MAPPING}"

shard_hex_width(shards)

Hex width required to represent shards-1 (zero-padded).

Examples: 1→1, 8→1, 16→1, 32→2, 256→2, 4096→3.

Source code in ftm_lakehouse/core/conventions/path.py
def shard_hex_width(shards: int) -> int:
    """Hex width required to represent `shards-1` (zero-padded).

    Examples: 1→1, 8→1, 16→1, 32→2, 256→2, 4096→3.
    """
    if shards <= 1:
        return 1
    return max(1, ((shards - 1).bit_length() + 3) // 4)

statement_origin(origin)

Get path prefix for given origin, following parquet partition pattern

Parameters:

Name Type Description Default
origin str

The origin, or phase, or stage

required

Raises:

Type Description
ValueError

If origin is malformed.

Source code in ftm_lakehouse/core/conventions/path.py
def statement_origin(origin: str) -> str:
    """
    Get path prefix for given origin, following parquet partition pattern

    Args:
        origin: The origin, or phase, or stage

    Raises:
        ValueError: If ``origin`` is malformed.
    """
    validate_origin(origin)
    return f"{STATEMENTS}/origin={origin}"

tag(*parts, tenant=TENANT)

Generate a path to store a tag

Source code in ftm_lakehouse/core/conventions/path.py
def tag(*parts: str, tenant: str | None = TENANT) -> str:
    """Generate a path to store a tag"""
    return join_relpaths(TAGS, tenant or TENANT, *parts)

version(name, ts=None)

Get a versioned snapshot path for a file, e.g. for index.json or config.yml

Layout: versions/YYYY/MM/{TS_FORMAT}/

Parameters:

Name Type Description Default
name str

The file name to version (e.g. "config.yml", "index.json")

required
ts str | None

ISO timestamp, omit to use current time

None

Returns:

Type Description
str

Path like "versions/2025/01/20250115T103000/config.yml"

Source code in ftm_lakehouse/core/conventions/path.py
def version(name: str, ts: str | None = None) -> str:
    """
    Get a versioned snapshot path for a file, e.g. for index.json or config.yml

    Layout: versions/YYYY/MM/{TS_FORMAT}/<name>

    Args:
        name: The file name to version (e.g. "config.yml", "index.json")
        ts: ISO timestamp, omit to use current time

    Returns:
        Path like "versions/2025/01/20250115T103000/config.yml"
    """
    if ts is None:
        ts = utc_now().strftime(TS_FORMAT)

    year = ts[:4]
    month = ts[4:6]
    return f"{VERSIONS}/{year}/{month}/{ts}/{name}"

Tag conventions

Global tags used to identify actions. Used for cache keys of workflow runs etc.

Export operations don't have constants here – their freshness tag is the path.* export target itself (e.g. exports/statements.csv), touched by :meth:DatasetJobOperation._run_local after a successful run.

ARCHIVE_ORIGIN = 'archive' module-attribute

Default origin identifier for archived files (if not crawled)

ARCHIVE_UPDATED = 'archive/last_updated' module-attribute

Archive last updated (file added or removed)

CRAWL_ORIGIN = 'crawl' module-attribute

Default origin identifier for crawled files.

JOURNAL_FLUSHED = 'journal/last_flushed' module-attribute

Journal store last flushed into statement store

JOURNAL_UPDATED = 'journal/last_updated' module-attribute

Statement journal was updated

OP_CRAWL = 'operations/crawl/last_run' module-attribute

Last crawl (import files) execution

OP_DOWNLOAD_ARCHIVE = 'operations/download_archive/last_run' module-attribute

Last download archive execution

OP_MAKE = 'operations/make/last_run' module-attribute

Last make (full workflow) execution

STATEMENTS_OPTIMIZED = 'statements/last_optimized' module-attribute

Statement store was optimized (merge + compact + vacuum)

STATEMENTS_UPDATED = 'statements/last_updated' module-attribute

Statement store was updated

statements_partition_optimized(shard, bucket, origin)

Per-partition freshness tag: a (shard, bucket, origin) was merged.

Partition-scoped analog of :data:STATEMENTS_OPTIMIZED, stamped by :meth:ParquetStore.merge after it rewrites the partition. See :func:statements_partition_updated for the freshness comparison.

Parameters:

Name Type Description Default
shard str

Hex-padded shard value.

required
bucket str

FtM schema bucket (thing / interval / ...).

required
origin str

Source tag – validated so it stays a single path segment.

required
Source code in ftm_lakehouse/core/conventions/tag.py
def statements_partition_optimized(shard: str, bucket: str, origin: str) -> str:
    """Per-partition freshness tag: a ``(shard, bucket, origin)`` was merged.

    Partition-scoped analog of :data:`STATEMENTS_OPTIMIZED`, stamped by
    :meth:`ParquetStore.merge` after it rewrites the partition. See
    :func:`statements_partition_updated` for the freshness comparison.

    Args:
        shard: Hex-padded shard value.
        bucket: FtM schema bucket (``thing`` / ``interval`` / ...).
        origin: Source tag – validated so it stays a single path segment.
    """
    validate_origin(origin)
    return f"statements/{shard}/{bucket}/{origin}/last_optimized"

statements_partition_updated(shard, bucket, origin)

Per-partition freshness tag: a (shard, bucket, origin) was written.

Partition-scoped analog of :data:STATEMENTS_UPDATED, stamped by :meth:ParquetStore.append. :meth:ParquetStore.merge compares it against :func:statements_partition_optimized via :meth:TagStore.is_latest to skip partitions that haven't changed since their last merge.

Parameters:

Name Type Description Default
shard str

Hex-padded shard value.

required
bucket str

FtM schema bucket (thing / interval / ...).

required
origin str

Source tag – validated so it stays a single path segment.

required
Source code in ftm_lakehouse/core/conventions/tag.py
def statements_partition_updated(shard: str, bucket: str, origin: str) -> str:
    """Per-partition freshness tag: a ``(shard, bucket, origin)`` was written.

    Partition-scoped analog of :data:`STATEMENTS_UPDATED`, stamped by
    :meth:`ParquetStore.append`. :meth:`ParquetStore.merge` compares it
    against :func:`statements_partition_optimized` via
    :meth:`TagStore.is_latest` to skip partitions that haven't changed
    since their last merge.

    Args:
        shard: Hex-padded shard value.
        bucket: FtM schema bucket (``thing`` / ``interval`` / ...).
        origin: Source tag – validated so it stays a single path segment.
    """
    validate_origin(origin)
    return f"statements/{shard}/{bucket}/{origin}/last_updated"