pytext package¶
Subpackages¶
- pytext.common package
- pytext.config package
- Submodules
- pytext.config.component module
- pytext.config.config_adapter module
- pytext.config.contextual_intent_slot module
- pytext.config.doc_classification module
- pytext.config.field_config module
- pytext.config.module_config module
- pytext.config.pair_classification module
- pytext.config.pytext_config module
- pytext.config.query_document_pairwise_ranking module
- pytext.config.serialize module
- pytext.config.utils module
- Module contents
- pytext.data package
- Subpackages
- pytext.data.data_structures package
- pytext.data.featurizer package
- pytext.data.sources package
- pytext.data.test package
- Submodules
- pytext.data.test.batch_sampler_test module
- pytext.data.test.data_test module
- pytext.data.test.dynamic_pooling_batcher_test module
- pytext.data.test.mask_tensorizers_test module
- pytext.data.test.pandas_data_source_test module
- pytext.data.test.round_robin_batchiterator_test module
- pytext.data.test.simple_featurizer_test module
- pytext.data.test.tensorizers_test module
- pytext.data.test.tokenizers_test module
- pytext.data.test.tsv_data_source_test module
- pytext.data.test.utils_test module
- Module contents
- pytext.data.tokenizers package
- Submodules
- pytext.data.batch_sampler module
- pytext.data.bert_tensorizer module
- pytext.data.data module
- pytext.data.data_handler module
- pytext.data.dense_retrieval_tensorizer module
- pytext.data.disjoint_multitask_data module
- pytext.data.disjoint_multitask_data_handler module
- pytext.data.dynamic_pooling_batcher module
- pytext.data.masked_tensorizer module
- pytext.data.masked_util module
- pytext.data.packed_lm_data module
- pytext.data.roberta_tensorizer module
- pytext.data.squad_for_bert_tensorizer module
- pytext.data.squad_tensorizer module
- pytext.data.tensorizers module
- pytext.data.token_tensorizer module
- pytext.data.utils module
- pytext.data.xlm_constants module
- pytext.data.xlm_dictionary module
- pytext.data.xlm_tensorizer module
- Module contents
- Subpackages
- pytext.exporters package
- pytext.fields package
- pytext.loss package
- pytext.metric_reporters package
- Submodules
- pytext.metric_reporters.calibration_metric_reporter module
- pytext.metric_reporters.channel module
- pytext.metric_reporters.classification_metric_reporter module
- pytext.metric_reporters.compositional_metric_reporter module
- pytext.metric_reporters.compositional_utils module
- pytext.metric_reporters.dense_retrieval_metric_reporter module
- pytext.metric_reporters.disjoint_multitask_metric_reporter module
- pytext.metric_reporters.intent_slot_detection_metric_reporter module
- pytext.metric_reporters.language_model_metric_reporter module
- pytext.metric_reporters.mask_compositional module
- pytext.metric_reporters.metric_reporter module
- pytext.metric_reporters.pairwise_ranking_metric_reporter module
- pytext.metric_reporters.regression_metric_reporter module
- pytext.metric_reporters.seq2seq_compositional module
- pytext.metric_reporters.seq2seq_metric_reporter module
- pytext.metric_reporters.seq2seq_utils module
- pytext.metric_reporters.squad_metric_reporter module
- pytext.metric_reporters.word_tagging_metric_reporter module
- Module contents
- pytext.metrics package
- Submodules
- pytext.metrics.calibration_metrics module
- pytext.metrics.dense_retrieval_metrics module
- pytext.metrics.intent_slot_metrics module
- pytext.metrics.language_model_metrics module
- pytext.metrics.mask_metrics module
- pytext.metrics.seq2seq_metrics module
- pytext.metrics.squad_metrics module
- Module contents
- pytext.models package
- Subpackages
- pytext.models.decoders package
- Submodules
- pytext.models.decoders.decoder_base module
- pytext.models.decoders.intent_slot_model_decoder module
- pytext.models.decoders.mlp_decoder module
- pytext.models.decoders.mlp_decoder_query_response module
- pytext.models.decoders.mlp_decoder_two_tower module
- pytext.models.decoders.multilabel_decoder module
- Module contents
- pytext.models.embeddings package
- Submodules
- pytext.models.embeddings.char_embedding module
- pytext.models.embeddings.contextual_token_embedding module
- pytext.models.embeddings.dict_embedding module
- pytext.models.embeddings.embedding_base module
- pytext.models.embeddings.embedding_list module
- pytext.models.embeddings.mlp_embedding module
- pytext.models.embeddings.scriptable_embedding_list module
- pytext.models.embeddings.word_embedding module
- pytext.models.embeddings.word_seq_embedding module
- Module contents
- pytext.models.ensembles package
- pytext.models.language_models package
- pytext.models.output_layers package
- Submodules
- pytext.models.output_layers.distance_output_layer module
- pytext.models.output_layers.doc_classification_output_layer module
- pytext.models.output_layers.doc_regression_output_layer module
- pytext.models.output_layers.intent_slot_output_layer module
- pytext.models.output_layers.lm_output_layer module
- pytext.models.output_layers.multi_label_classification_layer module
- pytext.models.output_layers.output_layer_base module
- pytext.models.output_layers.pairwise_ranking_output_layer module
- pytext.models.output_layers.squad_output_layer module
- pytext.models.output_layers.utils module
- pytext.models.output_layers.word_tagging_output_layer module
- Module contents
- pytext.models.qna package
- pytext.models.representations package
- Subpackages
- pytext.models.representations.transformer package
- Submodules
- pytext.models.representations.transformer.multihead_attention module
- pytext.models.representations.transformer.multihead_linear_attention module
- pytext.models.representations.transformer.positional_embedding module
- pytext.models.representations.transformer.representation module
- pytext.models.representations.transformer.residual_mlp module
- pytext.models.representations.transformer.sentence_encoder module
- pytext.models.representations.transformer.transformer module
- Module contents
- pytext.models.representations.transformer package
- Submodules
- pytext.models.representations.attention module
- pytext.models.representations.augmented_lstm module
- pytext.models.representations.bilstm module
- pytext.models.representations.bilstm_doc_attention module
- pytext.models.representations.bilstm_doc_slot_attention module
- pytext.models.representations.bilstm_slot_attn module
- pytext.models.representations.biseqcnn module
- pytext.models.representations.contextual_intent_slot_rep module
- pytext.models.representations.deepcnn module
- pytext.models.representations.docnn module
- pytext.models.representations.huggingface_bert_sentence_encoder module
- pytext.models.representations.huggingface_electra_sentence_encoder module
- pytext.models.representations.jointcnn_rep module
- pytext.models.representations.ordered_neuron_lstm module
- pytext.models.representations.pair_rep module
- pytext.models.representations.pass_through module
- pytext.models.representations.pooling module
- pytext.models.representations.pure_doc_attention module
- pytext.models.representations.representation_base module
- pytext.models.representations.seq_rep module
- pytext.models.representations.slot_attention module
- pytext.models.representations.sparse_transformer_sentence_encoder module
- pytext.models.representations.stacked_bidirectional_rnn module
- pytext.models.representations.traced_transformer_encoder module
- pytext.models.representations.transformer_sentence_encoder module
- pytext.models.representations.transformer_sentence_encoder_base module
- Module contents
- Subpackages
- pytext.models.semantic_parsers package
- pytext.models.seq_models package
- Submodules
- pytext.models.seq_models.attention module
- pytext.models.seq_models.base module
- pytext.models.seq_models.contextual_intent_slot module
- pytext.models.seq_models.conv_decoder module
- pytext.models.seq_models.conv_encoder module
- pytext.models.seq_models.conv_model module
- pytext.models.seq_models.light_conv module
- pytext.models.seq_models.mask_generator module
- pytext.models.seq_models.nar_length module
- pytext.models.seq_models.nar_modules module
- pytext.models.seq_models.nar_output_layer module
- pytext.models.seq_models.positional module
- pytext.models.seq_models.projection_layers module
- pytext.models.seq_models.rnn_decoder module
- pytext.models.seq_models.rnn_encoder module
- pytext.models.seq_models.rnn_encoder_decoder module
- pytext.models.seq_models.seq2seq_model module
- pytext.models.seq_models.seq2seq_output_layer module
- pytext.models.seq_models.seqnn module
- pytext.models.seq_models.utils module
- Module contents
- pytext.models.decoders package
- Submodules
- pytext.models.bert_classification_models module
- pytext.models.bert_regression_model module
- pytext.models.crf module
- pytext.models.disjoint_multitask_model module
- pytext.models.distributed_model module
- pytext.models.doc_model module
- pytext.models.joint_model module
- pytext.models.masked_lm module
- pytext.models.masking_utils module
- pytext.models.model module
- pytext.models.module module
- pytext.models.pair_classification_model module
- pytext.models.query_document_pairwise_ranking_model module
- pytext.models.r3f_models module
- pytext.models.roberta module
- pytext.models.two_tower_classification_model module
- pytext.models.utils module
- pytext.models.word_model module
- Module contents
- Subpackages
- pytext.optimizer package
- Subpackages
- Submodules
- pytext.optimizer.activations module
- pytext.optimizer.adabelief module
- pytext.optimizer.fairseq_fp16_utils module
- pytext.optimizer.fp16_optimizer module
- pytext.optimizer.lamb module
- pytext.optimizer.madgrad module
- pytext.optimizer.optimizers module
- pytext.optimizer.privacy_engine module
- pytext.optimizer.radam module
- pytext.optimizer.scheduler module
- pytext.optimizer.swa module
- Module contents
- pytext.resources package
- pytext.task package
- pytext.torchscript package
- Subpackages
- pytext.torchscript.seq2seq package
- Submodules
- pytext.torchscript.seq2seq.beam_decode module
- pytext.torchscript.seq2seq.beam_search module
- pytext.torchscript.seq2seq.decoder module
- pytext.torchscript.seq2seq.encoder module
- pytext.torchscript.seq2seq.export_model module
- pytext.torchscript.seq2seq.scripted_seq2seq_generator module
- pytext.torchscript.seq2seq.seq2seq_rnn_decoder_utils module
- Module contents
- pytext.torchscript.tensorizer package
- pytext.torchscript.tokenizer package
- pytext.torchscript.seq2seq package
- Submodules
- pytext.torchscript.batchutils module
- pytext.torchscript.module module
- pytext.torchscript.utils module
- pytext.torchscript.vocab module
- Module contents
- Subpackages
- pytext.trainers package
- pytext.utils package
- Submodules
- pytext.utils.ascii_table module
- pytext.utils.config_utils module
- pytext.utils.cuda module
- pytext.utils.data module
- pytext.utils.distributed module
- pytext.utils.documentation module
- pytext.utils.embeddings module
- pytext.utils.file_io module
- pytext.utils.label module
- pytext.utils.lazy module
- pytext.utils.loss module
- pytext.utils.meter module
- pytext.utils.mobile_onnx module
- pytext.utils.model module
- pytext.utils.onnx module
- pytext.utils.path module
- pytext.utils.precision module
- pytext.utils.tensor module
- pytext.utils.test module
- pytext.utils.timing module
- pytext.utils.torch module
- pytext.utils.usage module
- Module contents
Submodules¶
pytext.builtin_task module¶
pytext.main module¶
-
pytext.main.
run_single
(rank: int, config_json: str, world_size: int, dist_init_method: Optional[str], metadata: Union[Dict[str, pytext.data.data_handler.CommonMetadata], pytext.data.data_handler.CommonMetadata, None], metric_channels: Optional[List[pytext.metric_reporters.channel.Channel]])[source]¶
pytext.workflow module¶
-
class
pytext.workflow.
LogitsWriter
(results: multiprocessing.context.BaseContext.Queue, output_path: str, use_gzip: bool, ndigits_precision: int)[source]¶ Bases:
object
Writes model logits to a file.
- The class is designed for use in an asynchronous process spawned by torch.multiprocessing.spawn, e.g.
- logits_writer = LogitsWriter(…) logits_writer_ctx = torch.multiprocessing.spawn(logits_writer.run, join=False) logits_writer_ctx.join()
-
pytext.workflow.
export_saved_model_to_caffe2
(saved_model_path: str, export_caffe2_path: str, output_onnx_path: str = None) → None[source]¶
-
pytext.workflow.
export_saved_model_to_torchscript
(saved_model_path: str, path: str, export_config: pytext.config.pytext_config.ExportConfig) → None[source]¶
-
pytext.workflow.
get_logits
(snapshot_path: str, use_cuda_if_available: bool, output_path: Optional[str] = None, test_path: Optional[str] = None, field_names: Optional[List[str]] = None, dump_raw_input: bool = False, batch_size: int = 16, ndigits_precision: int = 0, output_columns: Optional[List[int]] = None, use_gzip: bool = False, device_id: int = 0, fp16: bool = False)[source]¶
-
pytext.workflow.
prepare_task
(config: pytext.config.pytext_config.PyTextConfig, dist_init_url: str = None, device_id: int = 0, rank: int = 0, world_size: int = 1, metric_channels: Optional[List[pytext.metric_reporters.channel.Channel]] = None, metadata: pytext.data.data_handler.CommonMetadata = None) → Tuple[pytext.task.task.Task_Deprecated, pytext.trainers.training_state.TrainingState][source]¶
-
pytext.workflow.
prepare_task_metadata
(config: pytext.config.pytext_config.PyTextConfig) → pytext.data.data_handler.CommonMetadata[source]¶ Loading the whole dataset into cpu memory on every single processes could cause OOMs for data parallel distributed training. To avoid such practice, we move the operations that required loading the whole dataset out of spawn, and pass the context to every single process.
-
pytext.workflow.
reload_model_for_multi_export
(config: pytext.config.pytext_config.PyTextConfig)[source]¶
-
pytext.workflow.
save_and_export
(config: pytext.config.pytext_config.PyTextConfig, task: pytext.task.task.Task_Deprecated, metric_channels: Optional[List[pytext.metric_reporters.channel.Channel]] = None) → None[source]¶
-
pytext.workflow.
save_pytext_snapshot
(config: pytext.config.pytext_config.PyTextConfig) → None[source]¶
-
pytext.workflow.
test_model
(test_config: pytext.config.pytext_config.TestConfig, metric_channels: Optional[List[pytext.metric_reporters.channel.Channel]], test_out_path: str) → Any[source]¶
-
pytext.workflow.
test_model_from_snapshot_path
(snapshot_path: str, use_cuda_if_available: bool, test_path: Optional[str] = None, metric_channels: Optional[List[pytext.metric_reporters.channel.Channel]] = None, test_out_path: str = '', field_names: Optional[List[str]] = None)[source]¶
-
pytext.workflow.
train_model
(config: pytext.config.pytext_config.PyTextConfig, dist_init_url: str = None, device_id: int = 0, rank: int = 0, world_size: int = 1, metric_channels: Optional[List[pytext.metric_reporters.channel.Channel]] = None, metadata: pytext.data.data_handler.CommonMetadata = None) → Tuple[source]¶
Module contents¶
-
pytext.
batch_predict_caffe2_model
(pytext_model_file: str, caffe2_model_file: str, db_type: str = 'minidb', data_source: Optional[pytext.data.sources.data_source.DataSource] = None, use_cuda=False, task: Optional[pytext.task.new_task.NewTask] = None, train_config: Optional[pytext.config.pytext_config.PyTextConfig] = None, cache_size: int = 0)[source]¶ Gets predictions from caffe2 model from a batch of examples.
Parameters: - pytext_model_file – Path to pytext model file (required if task and training config is not specified)
- caffe2_model_file – Path to caffe2 model file
- db_type – DB type to use for caffe2
- data_source – Data source for test examples
- use_cuda – Whether to turn on cuda processing
- task – The pytext task object
- train_config – The pytext training config
- cache_size – The LRU cache size to use for prediction. 0 = no cache, -1 = boundless cache, [1, inf) = size of cache
-
pytext.
create_predictor
(config: pytext.config.pytext_config.PyTextConfig, model_file: Optional[str] = None, db_type: str = 'minidb', task: Optional[pytext.task.new_task.NewTask] = None, cache_size: int = 0) → Callable[[Mapping[str, str]], Mapping[str, numpy.array]][source]¶ Create a simple prediction API from a training config and an exported caffe2 model file. This model file should be created by calling export on a trained model snapshot.