pytext.data.tokenizers package

Submodules

pytext.data.tokenizers.tokenizer module

class pytext.data.tokenizers.tokenizer.BERTInitialTokenizer(basic_tokenizer)

Bases: pytext.data.tokenizers.tokenizer.Tokenizer

Basic initial tokenization for BERT. This is run prior to word piece, does white space tokenization in addition to lower-casing and accent removal if specified.

classmethod from_config(config: pytext.data.tokenizers.tokenizer.BERTInitialTokenizer.Config)

tokenize(text)

Tokenizes a piece of text.

class pytext.data.tokenizers.tokenizer.CppProcessorMixin

Bases: object

Cpp processors like SentencePiece don’t pickle well; reload them.

class pytext.data.tokenizers.tokenizer.DoNothingTokenizer

Bases: pytext.data.tokenizers.tokenizer.Tokenizer

Tokenizer that takes a list of strings and converts to a list of Tokens. Useful in cases where tokenizer is run before-hand

classmethod from_config(config: pytext.data.tokenizers.tokenizer.DoNothingTokenizer.Config)

tokenize(tokens: Union[List[str], str]) → List[pytext.data.tokenizers.tokenizer.Token]

torchscriptify()

class pytext.data.tokenizers.tokenizer.GPT2BPETokenizer(bpe: fairseq.data.encoders.gpt2_bpe_utils.Encoder, lowercase: bool = False)

Bases: pytext.data.tokenizers.tokenizer.Tokenizer

Tokenizer for gpt-2 and RoBERTa.

decode(sentence: str)

classmethod from_config(config: pytext.data.tokenizers.tokenizer.GPT2BPETokenizer.Config)

tokenize(input_str: str) → List[pytext.data.tokenizers.tokenizer.Token]

class pytext.data.tokenizers.tokenizer.PickleableGPT2BPEEncoder(encoder, bpe_merges, errors='replace')

Bases: fairseq.data.encoders.gpt2_bpe_utils.Encoder

Fairseq’s encoder stores the regex module as a local reference on its encoders, which means they can’t be saved via pickle.dumps or torch.save. This modified their save/load logic doesn’t store the module, and restores the reference after re-inflating.

class pytext.data.tokenizers.tokenizer.SentencePieceTokenizer(sp_model_path: str = '', max_input_text_length: Optional[int] = None)

Bases: pytext.data.tokenizers.tokenizer.Tokenizer, pytext.data.tokenizers.tokenizer.CppProcessorMixin

Sentence piece tokenizer.

classmethod from_config(config: pytext.data.tokenizers.tokenizer.SentencePieceTokenizer.Config)

tokenize(input_str: str) → List[pytext.data.tokenizers.tokenizer.Token]

torchscriptify()

class pytext.data.tokenizers.tokenizer.Token(value, start, end)

Bases: tuple

end

Alias for field number 2

start

Alias for field number 1

value

Alias for field number 0

class pytext.data.tokenizers.tokenizer.Tokenizer(split_regex='\s+', lowercase=True, use_byte_offsets=False)

Bases: pytext.config.component.Component

A simple regex-splitting tokenizer.

decode(sentence: str)

classmethod from_config(config: pytext.data.tokenizers.tokenizer.Tokenizer.Config)

tokenize(input: str) → List[pytext.data.tokenizers.tokenizer.Token]

torchscriptify()

class pytext.data.tokenizers.tokenizer.WordPieceTokenizer(wordpiece_vocab, basic_tokenizer, wordpiece_tokenizer)

Bases: pytext.data.tokenizers.tokenizer.Tokenizer

Word piece tokenizer for BERT models.

classmethod from_config(config: pytext.data.tokenizers.tokenizer.WordPieceTokenizer.Config)

static load_vocab(vocab_file)

Loads a vocabulary file into a dictionary.

tokenize(input_str: str) → List[pytext.data.tokenizers.tokenizer.Token]

Module contents

class pytext.data.tokenizers.GPT2BPETokenizer(bpe: fairseq.data.encoders.gpt2_bpe_utils.Encoder, lowercase: bool = False)

Bases: pytext.data.tokenizers.tokenizer.Tokenizer

Tokenizer for gpt-2 and RoBERTa.

decode(sentence: str)

classmethod from_config(config: pytext.data.tokenizers.tokenizer.GPT2BPETokenizer.Config)

tokenize(input_str: str) → List[pytext.data.tokenizers.tokenizer.Token]

class pytext.data.tokenizers.Token(value, start, end)

Bases: tuple

end

Alias for field number 2

start

Alias for field number 1

value

Alias for field number 0

class pytext.data.tokenizers.Tokenizer(split_regex='\s+', lowercase=True, use_byte_offsets=False)

Bases: pytext.config.component.Component

A simple regex-splitting tokenizer.

decode(sentence: str)

classmethod from_config(config: pytext.data.tokenizers.tokenizer.Tokenizer.Config)

tokenize(input: str) → List[pytext.data.tokenizers.tokenizer.Token]

torchscriptify()

class pytext.data.tokenizers.DoNothingTokenizer

Bases: pytext.data.tokenizers.tokenizer.Tokenizer

Tokenizer that takes a list of strings and converts to a list of Tokens. Useful in cases where tokenizer is run before-hand

classmethod from_config(config: pytext.data.tokenizers.tokenizer.DoNothingTokenizer.Config)

tokenize(tokens: Union[List[str], str]) → List[pytext.data.tokenizers.tokenizer.Token]

torchscriptify()

class pytext.data.tokenizers.WordPieceTokenizer(wordpiece_vocab, basic_tokenizer, wordpiece_tokenizer)

Bases: pytext.data.tokenizers.tokenizer.Tokenizer

Word piece tokenizer for BERT models.

classmethod from_config(config: pytext.data.tokenizers.tokenizer.WordPieceTokenizer.Config)

static load_vocab(vocab_file)

Loads a vocabulary file into a dictionary.

tokenize(input_str: str) → List[pytext.data.tokenizers.tokenizer.Token]

class pytext.data.tokenizers.CppProcessorMixin

Bases: object

Cpp processors like SentencePiece don’t pickle well; reload them.

class pytext.data.tokenizers.SentencePieceTokenizer(sp_model_path: str = '', max_input_text_length: Optional[int] = None)

Bases: pytext.data.tokenizers.tokenizer.Tokenizer, pytext.data.tokenizers.tokenizer.CppProcessorMixin

Sentence piece tokenizer.

classmethod from_config(config: pytext.data.tokenizers.tokenizer.SentencePieceTokenizer.Config)

tokenize(input_str: str) → List[pytext.data.tokenizers.tokenizer.Token]

torchscriptify()