Skip to content

Settings

Settings

Bases: OpenAlephSettings

ingest-file settings management using pydantic-settings

Note

All settings can be set via environment variables, prepending INGESTORS_ (except for those with another alias) via runtime or in a .env file.

Source code in ingestors/settings.py 
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
class Settings(OpenAlephSettings):
    """
    `ingest-file` settings management using
    [pydantic-settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/)

    Note:
        All settings can be set via environment variables, prepending
        `INGESTORS_` (except for those with another alias) via runtime or in a
        `.env` file.
    """

    model_config = SettingsConfigDict(
        env_prefix="ingestors_",
        env_file=".env",
        extra="ignore",  # other envs in .env file
    )

    convert_timeout: int = 300
    """Headless libreoffice document convert timeout in seconds"""

    tika_fallback: bool = False
    """Use Apache Tika as a text extraction fallback"""

    calamine: bool = False
    """Use the Rust calamine implementation (python-calamine) for spreadsheets"""

    lakehouse_flush_size: int = 10_000
    """Size cap until when the journal should be flushed to parquet"""

    @property
    def tags_database_uri(self) -> str:
        """Resolve the tags database URI from fragments_uri, forcing psycopg3
        driver for PostgreSQL."""
        uri = self.fragments_uri
        if uri and uri.startswith("postgresql://"):
            uri = uri.replace("postgresql://", "postgresql+psycopg://", 1)
        return uri

calamine = False class-attribute instance-attribute

Use the Rust calamine implementation (python-calamine) for spreadsheets

convert_timeout = 300 class-attribute instance-attribute

Headless libreoffice document convert timeout in seconds

lakehouse_flush_size = 10000 class-attribute instance-attribute

Size cap until when the journal should be flushed to parquet

tags_database_uri property

Resolve the tags database URI from fragments_uri, forcing psycopg3 driver for PostgreSQL.

tika_fallback = False class-attribute instance-attribute

Use Apache Tika as a text extraction fallback