Skip to content

openaleph_procrastinate.settings

DeferSettings

Bases: BaseSettings

Adjust the worker queues and tasks for different stages.

This is useful e.g. for launching a priority queuing setup for a specific dataset:

Example 
# ingest service
export OPENALEPH_INGEST_QUEUE=ingest-prio-dataset
export OPENALEPH_ANALYZE_QUEUE=analyze-prio-dataset
ingestors ingest -d prio_dataset ./documents
procrastinate worker -q ingest-prio-dataset --one-shot  # stop worker after complete

# analyze service
procrastinate worker -q analyze-prio-dataset --one-shot  # stop worker after complete
Source code in openaleph_procrastinate/settings.py 
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
class DeferSettings(BaseSettings):
    """
    Adjust the worker queues and tasks for different stages.

    This is useful e.g. for launching a priority queuing setup for a specific dataset:

    Example:
        ```bash
        # ingest service
        export OPENALEPH_INGEST_QUEUE=ingest-prio-dataset
        export OPENALEPH_ANALYZE_QUEUE=analyze-prio-dataset
        ingestors ingest -d prio_dataset ./documents
        procrastinate worker -q ingest-prio-dataset --one-shot  # stop worker after complete

        # analyze service
        procrastinate worker -q analyze-prio-dataset --one-shot  # stop worker after complete
        ```
    """

    model_config = SettingsConfigDict(
        env_prefix="openaleph_",
        env_nested_delimiter="_",
        env_file=".env",
        nested_model_default_partial_update=True,
        extra="ignore",  # other envs in .env file
    )

    ingest: ServiceSettings = ServiceSettings(
        queue="ingest", task="ingestors.tasks.ingest"
    )
    """ingest-file"""

    analyze: ServiceSettings = ServiceSettings(
        queue="analyze", task="ftm_analyze.tasks.analyze"
    )
    """ftm-analyze"""

    transcribe: ServiceSettings = ServiceSettings(
        queue="transcribe", task="ftm_transcribe.tasks.transcribe"
    )
    """ftm-transcribe"""

    geocode: ServiceSettings = ServiceSettings(
        queue="geocode", task="ftm_geocode.tasks.geocode"
    )
    """ftm-geocode"""

    assets: ServiceSettings = ServiceSettings(
        queue="assets", task="ftm_assets.tasks.resolve"
    )
    """ftm-assets"""

    # OpenAleph

    index: ServiceSettings = ServiceSettings(
        queue="openaleph", task="aleph.procrastinate.tasks.index"
    )
    """openaleph indexer"""

analyze = ServiceSettings(queue='analyze', task='ftm_analyze.tasks.analyze') class-attribute instance-attribute

ftm-analyze

assets = ServiceSettings(queue='assets', task='ftm_assets.tasks.resolve') class-attribute instance-attribute

ftm-assets

geocode = ServiceSettings(queue='geocode', task='ftm_geocode.tasks.geocode') class-attribute instance-attribute

ftm-geocode

index = ServiceSettings(queue='openaleph', task='aleph.procrastinate.tasks.index') class-attribute instance-attribute

openaleph indexer

ingest = ServiceSettings(queue='ingest', task='ingestors.tasks.ingest') class-attribute instance-attribute

ingest-file

transcribe = ServiceSettings(queue='transcribe', task='ftm_transcribe.tasks.transcribe') class-attribute instance-attribute

ftm-transcribe

OpenAlephSettings

Bases: BaseSettings

openaleph_procrastinate settings management using pydantic-settings

Note

All settings can be set via environment variables, prepending OPENALEPH_ (except for those with another alias) via runtime or in a .env file.

Source code in openaleph_procrastinate/settings.py 
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
class OpenAlephSettings(BaseSettings):
    """
    `openaleph_procrastinate` settings management using
    [pydantic-settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/)

    Note:
        All settings can be set via environment variables, prepending
        `OPENALEPH_` (except for those with another alias) via runtime or in a
        `.env` file.
    """

    model_config = SettingsConfigDict(
        env_prefix="openaleph_",
        env_nested_delimiter="_",
        env_file=".env",
        nested_model_default_partial_update=True,
        extra="ignore",  # other envs in .env file
    )

    instance: str = Field(default="openaleph")
    """Instance identifier"""

    debug: bool = Field(default=env.DEBUG, alias="debug")
    """Debug mode"""

    db_uri: str = Field(default=env.DATABASE_URI)
    """OpenAleph database uri"""

    procrastinate_db_uri: str = Field(default=env.DATABASE_URI)
    """Procrastinate database uri, falls back to OpenAleph database uri"""

    ftm_store_uri: str = Field(default=env.FTM_STORE_URI)
    """FollowTheMoney store uri"""

db_uri = Field(default=env.DATABASE_URI) class-attribute instance-attribute

OpenAleph database uri

debug = Field(default=env.DEBUG, alias='debug') class-attribute instance-attribute

Debug mode

ftm_store_uri = Field(default=env.FTM_STORE_URI) class-attribute instance-attribute

FollowTheMoney store uri

instance = Field(default='openaleph') class-attribute instance-attribute

Instance identifier

procrastinate_db_uri = Field(default=env.DATABASE_URI) class-attribute instance-attribute

Procrastinate database uri, falls back to OpenAleph database uri

ServiceSettings

Bases: BaseSettings

Settings for a specific service, like ingest-file or ftm-analyze

Source code in openaleph_procrastinate/settings.py 
 7
 8
 9
10
11
12
13
14
15
16
17
class ServiceSettings(BaseSettings):
    """
    Settings for a specific service, like `ingest-file` or `ftm-analyze`
    """

    queue: str
    """queue name"""
    task: str
    """task module path"""
    defer: bool = True
    """enable deferring"""

defer = True class-attribute instance-attribute

enable deferring

queue instance-attribute

queue name

task instance-attribute

task module path