Conventions
ftm-lakehouse is convention-driven: the path layout, artifact names and freshness tags are stable contracts – third-party tools can populate or consume a lakehouse by following them, without going through this library.
Storage Layout
On-disk (or cloud storage) layout of a lakehouse storage root:
lakehouse/
└── {dataset}/
├── config.yml # Dataset configuration (shards, compression, ...)
├── index.json # Published dataset index with statistics
├── .LOCK # Exclusive maintenance fence
├── .LOCK-APPENDS/ # In-flight append markers (shared fence)
│
├── archive/ # Content-addressed file storage
│ └── {ch[0:2]}/{ch[2:4]}/{ch[4:6]}/{checksum}/
│ ├── blob # Raw file content (stored once)
│ ├── {file_id}.json # File metadata (one per source path)
│ └── {origin}.txt # Extracted text (one per engine)
│
├── statements/ # Delta Lake parquet store
│ ├── _delta_log/
│ └── shard={hex}/bucket={bucket}/origin={origin}/*.parquet
│
├── entities.ftm.json[.gz|.zst] # Aggregated entities export
│
├── exports/
│ ├── statements.csv[.gz|.zst] # Sorted statements export
│ ├── statistics.json # Entity counts, facets
│ └── documents.csv # Document metadata
│
├── diffs/ # Timestamped delta diff exports
├── versions/ # Versioned snapshots (config, index, ...)
│ └── YYYY/MM/{timestamp}/
├── tags/{tenant}/ # Freshness tags (workflow state)
└── jobs/
└── runs/{job_type}/{timestamp}.json
Freshness Tags
Operations use tags to track freshness and skip unnecessary work – is_latest(key, dependencies) returns True when key is newer than all its dependencies:
| Tag | Set by | Meaning |
|---|---|---|
journal/last_updated |
Statement writes | Journal has uncommitted data |
journal/last_flushed |
Flush operation | Journal was flushed |
statements/last_updated |
Flush / append | Rows landed in the parquet store – not canonical yet |
statements/last_optimized |
merge, on completion |
Canonical content changed. The clock every export, statistic and diff goes stale against |
archive/last_updated |
File archive | New file was archived |
exports/statements.csv, entities.ftm.json, exports/documents.csv, exports/statistics.json, index.json |
Export operations | Export target keys double as their freshness tags |
operations/crawl/last_run |
Crawl operation | Last crawl execution |
Compression suffixes
When a dataset configures compression (gz / zst in config.yml), the streaming export artifacts carry the codec suffix – entities.ftm.json.zst, exports/statements.csv.zst – and index.json advertises the resulting names and urls. index.json and statistics.json themselves are always plain JSON.
Path conventions
All path construction goes through ftm_lakehouse.core.conventions.path – rendered here so the constants stay in sync with the code:
Path conventions for the FollowTheMoney data lakehouse.
The fundamental idea is to have a convention-based file system layout with well-known paths for metadata and information interchange between processing stages.
All paths are dataset-relative unless otherwise noted.
Dataset Layout
::
lakehouse/
index.json # catalog index
config.yml # catalog configuration
versions/ # versioned snapshots
YYYY/MM/YYYY-MM-DDTHH:MM:SS/
index.json
config.yml
[dataset]/
index.json # dataset index
config.yml # dataset configuration
versions/ # versioned snapshots
YYYY/MM/...
.LOCK # dataset-wide maintenance lock
.LOCK-APPENDS/ # in-flight append markers
locks/{tenant}/ # operation-specific locks
tags/{tenant}/ # workflow state / cache
archive/ # content-addressed file storage
ab/cd/ef/{checksum}/ # SHA256 split into segments
blob # file blob (stored once)
{file_id}.json # metadata (one per source path)
{origin}.txt # extracted text (one per engine)
mappings/
{content_hash}/
mapping.yml # current CSV mapping configuration
versions/ # versioned snapshots
YYYY/MM/...
statements/ # statement store (shard-partitioned)
shard={shard}/
bucket={bucket}/
origin={origin}/
*.parquet
entities.ftm.json[.zst|gzip] # aggregated entities export
exports/
statistics.json # entity counts, facets
statements.csv # sorted statements
documents.csv # document metadata
graph.cypher # neo4j export (optional)
diffs/
entities.ftm.json/
20240116T103000000000Z.delta.json # entities delta
exports/
documents.csv/
20240116T103000000000Z.diff.csv # documents delta
jobs/
runs/
{job_type}/
{timestamp}.json # job run results
ARCHIVE = 'archive'
module-attribute
Base path for archive
ARCHIVE_BLOB = 'blob'
module-attribute
blob filename within checksum directory
CONFIG = 'config.yml'
module-attribute
user editable config filename
DIFFS = 'diffs'
module-attribute
Base path for diff exports
DIFFS_DOCUMENTS = f'{DIFFS}/{EXPORTS_DOCUMENTS}'
module-attribute
Base path for document.csv diffs
DIFFS_ENTITIES = f'{DIFFS}/{ENTITIES_JSON}'
module-attribute
Base path for entities.ftm.json diffs
ENTITIES_JSON = 'entities.ftm.json'
module-attribute
aggregated entities file name
EXPORTS = 'exports'
module-attribute
Base path for exports
EXPORTS_CYPHER = f'{EXPORTS}/graph.cypher'
module-attribute
neo4j data export file path
EXPORTS_DOCUMENTS = f'{EXPORTS}/documents.csv'
module-attribute
documents metadata to stream
EXPORTS_STATEMENTS = f'{EXPORTS}/statements.csv'
module-attribute
complete sorted statements file path
EXPORTS_STATISTICS = f'{EXPORTS}/{STATISTICS}'
module-attribute
entity counts, pre-computed facts file path
INDEX = 'index.json'
module-attribute
generated index filename
JOBS = 'jobs'
module-attribute
Job data prefix
JOB_RUNS = f'{JOBS}/runs'
module-attribute
Job runs result storage prefix
LOCK = '.LOCK'
module-attribute
dataset-wide maintenance lock key name
LOCKS = 'locks'
module-attribute
Base path for storing locks
LOCK_APPENDS = '.LOCK-APPENDS'
module-attribute
Prefix for per-writer append marker keys (shared side of the write fence)
MAPPING = 'mapping.yml'
module-attribute
mapping file name
MAPPINGS = 'mappings'
module-attribute
Base path for storing mappings
STATEMENTS = 'statements'
module-attribute
Base path for storing statement data (partitioned by shard, bucket, origin)
STATISTICS = 'statistics.json'
module-attribute
computed statistics filename
TAGS = 'tags'
module-attribute
Base path for dataset tags cache
TENANT = 'lakehouse'
module-attribute
Default tenant name
TS_FORMAT = '%Y%m%dT%H%M%S%fZ'
module-attribute
Global format for timestamps in files
VERSIONS = 'versions'
module-attribute
Base path for versions
archive_blob(checksum)
Get the blob path for a file in the archive.
Layout: archive/5a/6a/cf/5a6acf229ba576d9a40b09292595658bbb74ef56/blob
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
checksum
|
str
|
SHA256 checksum of file |
required |
Source code in ftm_lakehouse/core/conventions/path.py
archive_meta(checksum, file_id)
Get a file metadata path for a specific file instance.
Multiple files with the same checksum but different source paths each get their own metadata file, keyed by their File.id.
Layout: archive/5a/6a/cf/.../file-abc123.json
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
checksum
|
str
|
SHA256 checksum of file |
required |
file_id
|
str
|
The File.id (hash of source path + checksum) |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in ftm_lakehouse/core/conventions/path.py
archive_prefix(checksum)
Get the directory path for a file in the archive.
Layout: archive/5a/6a/cf/5a6acf229ba576d9a40b09292595658bbb74ef56/
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
checksum
|
str
|
SHA256 checksum of file |
required |
Source code in ftm_lakehouse/core/conventions/path.py
archive_txt(checksum, origin)
Get a file text content path for a specific extraction origin.
Multiple text extractions can exist per file, keyed by origin (e.g., different OCR engines or extraction methods).
Layout: archive/5a/6a/cf/.../{origin}.txt
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
checksum
|
str
|
SHA256 checksum of file |
required |
origin
|
str
|
The extraction origin/engine name |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in ftm_lakehouse/core/conventions/path.py
documents_diff(ts=None)
Get path for a documents diff export file.
Layout: diffs/exports/documents.csv/{ts}.diff.csv
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ts
|
datetime | None
|
Compact timestamp (YYYYMMDDTHHMMSSZ), defaults to current time |
None
|
Returns:
| Type | Description |
|---|---|
str
|
Path to diff file |
Source code in ftm_lakehouse/core/conventions/path.py
entities_diff(ts=None, suffix=None)
Get path for an entities diff export file.
Layout: diffs/entities.ftm.json/{ts}.delta.json
The delta file contains line-based JSON with operation envelopes
{"op": "ADD", "entity": {"id": "...", "schema": "...", "properties": {...}}} {"op": "MOD", "entity": {"id": "...", "schema": "...", "properties": {...}}} {"op": "DEL", "entity": {"id": "..."}}
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
ts
|
datetime | None
|
Compact timestamp (YYYYMMDDTHHMMSSZ), defaults to current time |
None
|
Returns:
| Type | Description |
|---|---|
str
|
Path to delta file |
Source code in ftm_lakehouse/core/conventions/path.py
entity_shard(entity_id, shards)
Hex shard key for an entity id under a uniform shard count.
Uses the first 8 hex chars of the entity_id hash, taken mod shards,
then zero-padded to shard_hex_width(shards).
Source code in ftm_lakehouse/core/conventions/path.py
lock(*parts, tenant=TENANT)
mapping(content_hash)
Get the mapping.yml path for the given file SHA256.
Layout: mappings/{content_hash}/mapping.yml
shard_hex_width(shards)
Hex width required to represent shards-1 (zero-padded).
Examples: 1→1, 8→1, 16→1, 32→2, 256→2, 4096→3.
Source code in ftm_lakehouse/core/conventions/path.py
statement_origin(origin)
Get path prefix for given origin, following parquet partition pattern
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
origin
|
str
|
The origin, or phase, or stage |
required |
Raises:
| Type | Description |
|---|---|
ValueError
|
If |
Source code in ftm_lakehouse/core/conventions/path.py
tag(*parts, tenant=TENANT)
version(name, ts=None)
Get a versioned snapshot path for a file, e.g. for index.json or config.yml
Layout: versions/YYYY/MM/{TS_FORMAT}/
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
name
|
str
|
The file name to version (e.g. "config.yml", "index.json") |
required |
ts
|
str | None
|
ISO timestamp, omit to use current time |
None
|
Returns:
| Type | Description |
|---|---|
str
|
Path like "versions/2025/01/20250115T103000/config.yml" |
Source code in ftm_lakehouse/core/conventions/path.py
Tag conventions
Global tags used to identify actions. Used for cache keys of workflow runs etc.
Export operations don't have constants here – their freshness tag is the
path.* export target itself (e.g. exports/statements.csv), touched
by :meth:DatasetJobOperation._run_local after a successful run.
ARCHIVE_ORIGIN = 'archive'
module-attribute
Default origin identifier for archived files (if not crawled)
ARCHIVE_UPDATED = 'archive/last_updated'
module-attribute
Archive last updated (file added or removed)
CRAWL_ORIGIN = 'crawl'
module-attribute
Default origin identifier for crawled files.
JOURNAL_FLUSHED = 'journal/last_flushed'
module-attribute
Journal store last flushed into statement store
JOURNAL_UPDATED = 'journal/last_updated'
module-attribute
Statement journal was updated
OP_CRAWL = 'operations/crawl/last_run'
module-attribute
Last crawl (import files) execution
OP_DOWNLOAD_ARCHIVE = 'operations/download_archive/last_run'
module-attribute
Last download archive execution
OP_MAKE = 'operations/make/last_run'
module-attribute
Last make (full workflow) execution
STATEMENTS_OPTIMIZED = 'statements/last_optimized'
module-attribute
Statement store was optimized (merge + compact + vacuum)
STATEMENTS_UPDATED = 'statements/last_updated'
module-attribute
Statement store was updated
statements_partition_optimized(shard, bucket, origin)
Per-partition freshness tag: a (shard, bucket, origin) was merged.
Partition-scoped analog of :data:STATEMENTS_OPTIMIZED, stamped by
:meth:ParquetStore.merge after it rewrites the partition. See
:func:statements_partition_updated for the freshness comparison.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
str
|
Hex-padded shard value. |
required |
bucket
|
str
|
FtM schema bucket ( |
required |
origin
|
str
|
Source tag – validated so it stays a single path segment. |
required |
Source code in ftm_lakehouse/core/conventions/tag.py
statements_partition_updated(shard, bucket, origin)
Per-partition freshness tag: a (shard, bucket, origin) was written.
Partition-scoped analog of :data:STATEMENTS_UPDATED, stamped by
:meth:ParquetStore.append. :meth:ParquetStore.merge compares it
against :func:statements_partition_optimized via
:meth:TagStore.is_latest to skip partitions that haven't changed
since their last merge.
Parameters:
| Name | Type | Description | Default |
|---|---|---|---|
shard
|
str
|
Hex-padded shard value. |
required |
bucket
|
str
|
FtM schema bucket ( |
required |
origin
|
str
|
Source tag – validated so it stays a single path segment. |
required |