GPT2BPETokenizer.ConfigΒΆ

Component: GPT2BPETokenizer

class GPT2BPETokenizer.Config[source]

Bases: ConfigBase

All Attributes (including base classes)

bpe_encoder_path: str = 'manifold://pytext_training/tree/static/vocabs/bpe/gpt2/encoder.json'
bpe_vocab_path: str = 'manifold://pytext_training/tree/static/vocabs/bpe/gpt2/vocab.bpe'
lowercase: bool = False

Default JSON

{
    "bpe_encoder_path": "manifold://pytext_training/tree/static/vocabs/bpe/gpt2/encoder.json",
    "bpe_vocab_path": "manifold://pytext_training/tree/static/vocabs/bpe/gpt2/vocab.bpe",
    "lowercase": false
}