WordPieceTokenizer.Config¶

Component: WordPieceTokenizer

class WordPieceTokenizer.Config[source]: Bases: ConfigBase

All Attributes (including base classes)

basic_tokenizer: BERTInitialTokenizer.Config = BERTInitialTokenizer.Config()

wordpiece_vocab_path: str = '/mnt/vol/nlp_technologies/bert/uncased_L-12_H-768_A-12/vocab.txt'

Default JSON

{
    "basic_tokenizer": {
        "split_regex": "\\s+",
        "lowercase": true
    },
    "wordpiece_vocab_path": "/mnt/vol/nlp_technologies/bert/uncased_L-12_H-768_A-12/vocab.txt"
}