ZFS Integration

On a ZFS pool, ftm-lakehouse can create a tuned ZFS dataset per storage type when a dataset is first touched. The transport – a local zfs subprocess, or a host-side socket agent for containers – is the external zfs-agent package; ftm-lakehouse contributes the tuning and the call.

Local Mode

If the lakehouse runs directly on a ZFS-backed filesystem, enable ZFS dataset creation:

export LAKEHOUSE_URI=/zpools/tank/lakehouse
export LAKEHOUSE_ON_ZFS=1
export LAKEHOUSE_ZFS_POOL=zpools/tank/lakehouse

LAKEHOUSE_ZFS_POOL is the ZFS dataset path (no leading slash) the per-dataset children are created under; it must match the pool layout.

Each new dataset gets these ZFS datasets (zfs create via zfs-agent):

Both children also set sync=standard and logbias=throughput.

Separate statements pool

With the statements tables on their own volume (LAKEHOUSE_STATEMENTS_URI), name the ZFS dataset path that volume is mounted from, as LAKEHOUSE_ZFS_POOL names the one of LAKEHOUSE_URI:

export LAKEHOUSE_URI=/zpools/hdd/lakehouse
export LAKEHOUSE_STATEMENTS_URI=/zpools/ssd/lakehouse
export LAKEHOUSE_ON_ZFS=1
export LAKEHOUSE_ZFS_POOL=zpools/hdd/lakehouse
export LAKEHOUSE_ZFS_STATEMENTS_POOL=zpools/ssd/lakehouse

The parent and archive are created under LAKEHOUSE_ZFS_POOL; the statements child under a parent of its own ({dataset}/, same properties) in LAKEHOUSE_ZFS_STATEMENTS_POOL. On ZFS the two statements settings go together – one without the other fails at startup. A socket agent creates under one --pool only, so this works in local mode, or with an agent whose pool holds both paths.

Mountpoint Ownership

ZFS creates mountpoints owned by root:root. Set the zfs-agent package's ZFS_OWNER to chown new mountpoints; unset, no chown happens:

export ZFS_OWNER=1000:1000

• Local mode: set ZFS_OWNER wherever ftm-lakehouse or ftm-lakehouse zfs init runs.
• Socket mode: ownership is the agent's – pass --owner to zfs-agent or set ZFS_OWNER where it runs. The client sends none.

Socket Agent Mode

Containers typically lack the ZFS tools. Instead, the host runs the zfs-agent daemon, which listens on a Unix socket and runs zfs create for the container.

flowchart LR
    subgraph container["Docker Container"]
        app["ftm-lakehouse<br/>ensure_zfs_dataset()"]
    end

    subgraph host["Host"]
        agent["zfs-agent"]
        zfs["zfs create ..."]
        agent --> zfs
    end

    app -- "JSON over /run/zfs.sock" --> agent

On the host (requires pip install zfs-agent):

zfs-agent --socket /run/zfs.sock --pool zpools/tank/lakehouse --owner 1000:1000 --allowed-uid 1000

The agent checks the peer UID (SO_PEERCRED), restricts the socket to 0600, allowlists properties and restricts creates to its pool – see the zfs-agent documentation for the protocol and its ZFS_* environment.

Mount the socket into the container and set the environment:

services:
  api:
    image: ftm-lakehouse
    user: "1000:1000"
    environment:
      LAKEHOUSE_URI: /zpools/tank/lakehouse
      LAKEHOUSE_ON_ZFS: "1"
      LAKEHOUSE_ZFS_POOL: zpools/tank/lakehouse
      ZFS_SOCKET: /run/zfs.sock
    volumes:
      - /run/zfs.sock:/run/zfs.sock
      - /zpools/tank/lakehouse:/zpools/tank/lakehouse

The container's user: UID must match the agent's --allowed-uid – peer credentials cross the bind-mounted socket unchanged. With ZFS_SOCKET set, creates go over the socket instead of a local zfs subprocess.

Manual Initialization

To create a dataset's ZFS datasets without starting the application:

ftm-lakehouse zfs init my_dataset --pool zpools/tank/lakehouse

This creates the parent, archive and statements ZFS datasets. --pool defaults to LAKEHOUSE_ZFS_POOL; the statements dataset goes to LAKEHOUSE_ZFS_STATEMENTS_POOL when that is set.

Environment Variables

Lakehouse-side: LAKEHOUSE_ON_ZFS, LAKEHOUSE_ZFS_POOL and LAKEHOUSE_ZFS_STATEMENTS_POOL – see the configuration reference. Transport-side (ZFS_SOCKET, ZFS_OWNER, ZFS_POOL, ZFS_ALLOWED_UID, ZFS_EXTRA_PROPS, ZFS_LOG_LEVEL): the zfs-agent package.