Usage

Ingest file can be used stand-alone or in the context of OpenAleph

Depending on setup, the commands have to be run with or without the docker container. When using docker, make sure to mount the volumes needed and attach to a running postgres container if needed.

docker

docker run -it ghcr.io/openaleph/ingest-file ingestors ...

locally

ingestors ...

Directory or File Ingestion

Store files in archive and dispatch tasks. This accepts as well a single file as path argument.

ingestors ingest -d my_dataset ./path/to/files`

Resuming an ingest

With the lakehouse storage backend (OPENALEPH_LAKEHOUSE=1), --resume skips the files a previous run already crawled. They are looked up in the dataset's documents crawl export (exports/documents.crawl.csv) by their path below the crawl root, so run it against the same (relative) root as before.

ingestors ingest -d my_dataset --resume ./path/to/files

Start worker(s)

procrastinate worker -q ingest --concurrency 8

One-shot ingestion

Instead of having long-running workers, run the worker in sync mode. It will stop after all tasks are processed:

procrastinate worker -q ingest --one-shot --concurrency 8

debug mode

This will run the worker in-memory, so no additional worker command needed:

DEBUG=1 ingestors ingest -d my_dataset ./path/to/files

CLI Reference

CLI reference