SentencePieceTokenizer.ConfigΒΆ

Component: SentencePieceTokenizer

class SentencePieceTokenizer.Config[source]

Bases: ConfigBase

All Attributes (including base classes)

sp_model_path: str = ''
max_input_text_length: Optional[int] = None

Default JSON

{
    "sp_model_path": "",
    "max_input_text_length": null
}