Skip to content

Settings

FlairNerModels

Bases: BaseSettings

Easily overwrite specific language model for specific languages via:

FTM_ANALYZE_FLAIR_MODELS_DEU=de_core_news_lg

Source code in ftm_analyze/settings.py 
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
class FlairNerModels(_BaseSettings):
    """
    Easily overwrite specific language model for specific languages via:

    `FTM_ANALYZE_FLAIR_MODELS_DEU=de_core_news_lg`
    """

    eng: str = "ner"
    deu: str = "de-ner"
    fra: str = "fr-ner"
    spa: str = "es-ner-large"  # FIXME
    # rus: str = "ru_core_news_md"
    # por: str = "pt_core_news_sm"
    # ron: str = "ro_core_news_sm"
    # mkd: str = "mk_core_news_sm"
    # ell: str = "el_core_news_sm"
    # pol: str = "pl_core_news_sm"
    # ita: str = "it_core_news_sm"
    # lit: str = "lt_core_news_sm"
    nld: str = "nl-ner"

Settings

Bases: BaseSettings

ftm-analyze settings management using pydantic-settings

Note

All settings can be set via environment variables or .env file, prepending FTM_ANALYZE_ (except for those with another alias)

Source code in ftm_analyze/settings.py 
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
class Settings(BaseSettings):
    """
    `ftm-analyze` settings management using
    [pydantic-settings](https://docs.pydantic.dev/latest/concepts/pydantic_settings/)

    Note:
        All settings can be set via environment variables or `.env` file,
        prepending `FTM_ANALYZE_` (except for those with another alias)
    """

    model_config = SettingsConfigDict(
        env_prefix="ftm_analyze_",
        env_nested_delimiter="_",
        env_file=".env",
        extra="ignore",
    )

    ner_type_model_path: Path = Path("./models/model_type_prediction.ftz")
    """Local path to ftm type predict model"""

    ner_type_model_confidence: float = 0.85
    """Minimum confidence for ftm type predict model"""

    ner_engine: Literal["spacy", "flair", "bert"] = "spacy"
    """NER engine to use (may need install extra dependencies)"""

    lid_model_path: Path = Path("./models/lid.176.ftz")
    """Local path to lid model"""

    spacy_models: SpacyNerModels = SpacyNerModels()
    """Spacy models"""

    flair_models: FlairNerModels = FlairNerModels()
    """Flair models"""

    bert_model: str = "dslim/bert-base-NER"
    """Model when using BERT transformers"""

    ner_default_lang: str = "eng"
    """Default ner language, 3-letter code"""

    resolve_mentions: bool = False
    """Resolve known mentions via `juditha`"""

    refine_mentions: bool = False
    """Refine schema classification for mentions via `juditha` fasttext model"""

    annotate: bool = False
    """Insert annotations into `indexText` for resolved mentions"""

    validate_names: bool = False
    """Validate NER results against known name tokens via `juditha`"""

annotate = False class-attribute instance-attribute

Insert annotations into indexText for resolved mentions

bert_model = 'dslim/bert-base-NER' class-attribute instance-attribute

Model when using BERT transformers

flair_models = FlairNerModels() class-attribute instance-attribute

Flair models

lid_model_path = Path('./models/lid.176.ftz') class-attribute instance-attribute

Local path to lid model

ner_default_lang = 'eng' class-attribute instance-attribute

Default ner language, 3-letter code

ner_engine = 'spacy' class-attribute instance-attribute

NER engine to use (may need install extra dependencies)

ner_type_model_confidence = 0.85 class-attribute instance-attribute

Minimum confidence for ftm type predict model

ner_type_model_path = Path('./models/model_type_prediction.ftz') class-attribute instance-attribute

Local path to ftm type predict model

refine_mentions = False class-attribute instance-attribute

Refine schema classification for mentions via juditha fasttext model

resolve_mentions = False class-attribute instance-attribute

Resolve known mentions via juditha

spacy_models = SpacyNerModels() class-attribute instance-attribute

Spacy models

validate_names = False class-attribute instance-attribute

Validate NER results against known name tokens via juditha

SpacyNerModels

Bases: BaseSettings

Easily overwrite specific language model for specific languages via:

FTM_ANALYZE_SPACY_MODELS_DEU=de_core_news_lg

Source code in ftm_analyze/settings.py 
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
class SpacyNerModels(_BaseSettings):
    """
    Easily overwrite specific language model for specific languages via:

    `FTM_ANALYZE_SPACY_MODELS_DEU=de_core_news_lg`
    """

    eng: str = "en_core_web_sm"
    deu: str = "de_core_news_sm"
    fra: str = "fr_core_news_sm"
    spa: str = "es_core_news_sm"
    rus: str = "ru_core_news_sm"
    por: str = "pt_core_news_sm"
    ron: str = "ro_core_news_sm"
    mkd: str = "mk_core_news_sm"
    ell: str = "el_core_news_sm"
    pol: str = "pl_core_news_sm"
    ita: str = "it_core_news_sm"
    lit: str = "lt_core_news_sm"
    nld: str = "nl_core_news_sm"
    nob: str = "nb_core_news_sm"
    nor: str = "nb_core_news_sm"
    dan: str = "da_core_news_sm"