Architecture and design principles

OpenAleph Procrastinate

openaleph-procrastinate acts as a shared library for services that can handle tasks within the OpenAleph infrastructure. It holds a Job model (via pydantic), procrastinate.App logic and helpers for reading/writing archive files and followthemoney data via the repository.

This base library doesn't know about the actual queues and tasks. The worker services would have openaleph-procrastinate as a dependency and import the helpers and Job model from there.

For task inspection, openaleph-procrastinate provides the manage module: dataset, batch, queue and task status aggregation plus bulk job operations, queried directly against the procrastinate tables and exposed via the command line.

Services

Services subscribing to openaleph_procrastinate are completely independent programs that could run on separate (or same) infrastructure or as containers. They only need to be able to connect to the postgresql database holding the procrastinate task queue data.

Due to the possibility to defer Jobs to unknown tasks (outside of the codebase of the current running service), openaleph-procrastinate, a service and the OpenAleph programs doesn't need to share much codebase, and by using mostly out-of-the-box logic from procrastinate the implementation stays simple and maintainable.