TokenTensorizer.Config¶

Component: TokenTensorizer

class TokenTensorizer.Config[source]: Bases: Tensorizer.Config

All Attributes (including base classes)

is_input: bool = True

column: str = 'text'

The name of the text column to parse from the data source.

tokenizer: Tokenizer.Config = Tokenizer.Config()

The tokenizer to use to split input text into tokens.

add_bos_token: bool = False

add_eos_token: bool = False

use_eos_token_for_bos: bool = False

max_seq_len: Optional[int] = None

vocab: VocabConfig = VocabConfig()

vocab_file_delimiter: str = ' '

Subclasses

SquadTensorizer.Config
SquadTensorizerForKD.Config
CharacterTokenTensorizer.Config

Default JSON

{
    "is_input": true,
    "column": "text",
    "tokenizer": {
        "Tokenizer": {
            "split_regex": "\\s+",
            "lowercase": true,
            "use_byte_offsets": false
        }
    },
    "add_bos_token": false,
    "add_eos_token": false,
    "use_eos_token_for_bos": false,
    "max_seq_len": null,
    "vocab": {
        "build_from_data": true,
        "size_from_data": 0,
        "min_counts": 0,
        "vocab_files": []
    },
    "vocab_file_delimiter": " "
}