ByteTensorizer.Config¶

Component: ByteTensorizer

class ByteTensorizer.Config[source]: Bases: Tensorizer.Config

All Attributes (including base classes)

is_input: bool = True

column: str = 'text'

The name of the text column to parse from the data source.

lower: bool = True

max_seq_len: Optional[int] = None

add_bos_token: Optional[bool] = False

add_eos_token: Optional[bool] = False

use_eos_token_for_bos: Optional[bool] = False

Default JSON

{
    "is_input": true,
    "column": "text",
    "lower": true,
    "max_seq_len": null,
    "add_bos_token": false,
    "add_eos_token": false,
    "use_eos_token_for_bos": false
}