pytext.optimizer.sparsifiers package

Submodules

pytext.optimizer.sparsifiers.blockwise_sparsifier module

class pytext.optimizer.sparsifiers.blockwise_sparsifier.BlockwiseMagnitudeSparsifier(sparsity, starting_epoch, frequency, block_size, columnwise_blocking, accumulate_mask, layerwise_pruning)

Bases: pytext.optimizer.sparsifiers.sparsifier.L0_projection_sparsifier

running blockwise magnitude-based sparsification

Parameters:

• block_size – define the size of each block
• columnwise_blocking – define columnwise block if true
• starting_epoch – sparsification_condition returns true only after starting_epoch
• frequency – sparsification_condition only if number of steps devides frequency
• accumulate_mask – if true, the mask after each .sparisfy() will be reused
• sparsity – percentage of zeros among the UNPRUNED parameters.
• on how the sparsifier work (Examples) –
• matrix (2D) –
• [ – 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
• ] –
• 3 X 1 block (define) –
• [ – ***** *** 0 1 2 3 4 ****** *** 5 6 7 8 9 ****** *** 10 11 12 13 14 ****** *** 15 16 17 18 19 ****** *** 20 21 22 23 24 ****** ***
• ] –
• l1 norm of each block and sort them. Retain blocks with largest (compute) –
• values until sparsity threshold is met (absolute) –

classmethod from_config(config: pytext.optimizer.sparsifiers.blockwise_sparsifier.BlockwiseMagnitudeSparsifier.Config)

get_current_sparsity(model)

get_masks(model: torch.nn.modules.module.Module, pre_masks: List[torch.Tensor] = None) → List[torch.Tensor]

Note: this function returns the masks only but do not sparsify or modify the weights

prune x% of weights among the weights with “1” in pre_masks

Parameters:

• model – Model
• pre_masks – list of FloatTensors where “1” means retained the weight and “0” means pruned the weight

Returns:

List[torch.Tensor], intersection of new masks and pre_masks, so that “1” only if the weight is selected after new masking and pre_mask

Return type:

masks

get_sparsifiable_params(model, requires_name=False)

pytext.optimizer.sparsifiers.sparsifier module

class pytext.optimizer.sparsifiers.sparsifier.CRF_L1_SoftThresholding(lambda_l1: float, starting_epoch: int, frequency: int)

Bases: pytext.optimizer.sparsifiers.sparsifier.CRF_SparsifierBase

implement l1 regularization:

min Loss(x, y, CRFparams) + lambda_l1 * ||CRFparams||_1

and solve the optimiation problem via (stochastic) proximal gradient-based method i.e., soft-thresholding

param_updated = sign(CRFparams) * max ( abs(CRFparams) - lambda_l1, 0)

classmethod from_config(config: pytext.optimizer.sparsifiers.sparsifier.CRF_L1_SoftThresholding.Config)

sparsify(state)

class pytext.optimizer.sparsifiers.sparsifier.CRF_MagnitudeThresholding(sparsity, starting_epoch, frequency, grouping)

Bases: pytext.optimizer.sparsifiers.sparsifier.CRF_SparsifierBase

magnitude-based (equivalent to projection onto l0 constraint set) sparsification on CRF transition matrix. Preserveing the top-k elements either rowwise or columnwise until sparsity constraint is met.

classmethod from_config(config: pytext.optimizer.sparsifiers.sparsifier.CRF_MagnitudeThresholding.Config)

sparsify(state)

class pytext.optimizer.sparsifiers.sparsifier.CRF_SparsifierBase(config=None, *args, **kwargs)

Bases: pytext.optimizer.sparsifiers.sparsifier.Sparsifier

get_sparsifiable_params(model: torch.nn.modules.module.Module)

get_transition_sparsity(transition)

sparsification_condition(state)

class pytext.optimizer.sparsifiers.sparsifier.L0_projection_sparsifier(sparsity, starting_epoch, frequency, layerwise_pruning=True, accumulate_mask=False)

Bases: pytext.optimizer.sparsifiers.sparsifier.Sparsifier

L0 projection-based (unstructured) sparsification

Parameters:

• weights (torch.Tensor) – input weight matrix
• sparsity (float32) – the desired sparsity [0-1]

apply_masks(model: pytext.models.model.Model, masks: List[torch.Tensor])

apply given masks to zero-out learnable weights in model

classmethod from_config(config: pytext.optimizer.sparsifiers.sparsifier.L0_projection_sparsifier.Config)

get_masks(model: pytext.models.model.Model, pre_masks: List[torch.Tensor] = None) → List[torch.Tensor]

Note: this function returns the masks only but do not sparsify or modify the weights

prune x% of weights among the weights with “1” in pre_masks

Parameters:

• model – Model
• pre_masks – list of FloatTensors where “1” means retained the weight and “0” means pruned the weight

Returns:

List[torch.Tensor], intersection of new masks and pre_masks, so that “1” only if the weight is selected after new masking and pre_mask

Return type:

masks

get_sparsifiable_params(model: pytext.models.model.Model)

sparsification_condition(state)

sparsify(state)

obtain a mask and apply the mask to sparsify

class pytext.optimizer.sparsifiers.sparsifier.SensitivityAnalysisSparsifier(pre_train_model_path, analyzed_sparsity, max_analysis_batches, max_skipped_weight, pre_analysis_path, sparsity, iterative_pruning, pruning_iterations, start_sparsity_ratio)

Bases: pytext.optimizer.sparsifiers.sparsifier.Sparsifier

apply_masks(model: pytext.models.model.Model, masks: List[torch.Tensor])

apply given masks to zero-out learnable weights in model

find_params_to_prune(metric_dict, max_skip_weight_num)

classmethod from_config(config: pytext.optimizer.sparsifiers.sparsifier.SensitivityAnalysisSparsifier.Config)

get_current_sparsity(model: pytext.models.model.Model) → float

get_mask_for_param(param, sparsity)

generate the prune mask for one weight tensor.

get_masks(model: pytext.models.model.Model) → List[torch.Tensor]

Note: this function returns the masks for each weight tensor if that tensor is required to be pruned

prune x% of weights items among the weights with “1” in mask (self._mask) indicate the remained weights, with “0” indicate pruned weights

Parameters:model – Model Returns:List[torch.Tensor], the prune mask for the weight of all layers Return type:masks

get_required_sparsifiable_params(model: pytext.models.model.Model)

get_sparsifiable_params(model)

increase_sparsity(state)

initialize(trainer, state, eval_data, metric_reporter, train_config)

layer_wise_analysis(param_name, param_dict, trainer, state, eval_data, metric_reporter)

load_analysis_from_path()

op_pre_epoch(trainer, state)

note: invoke this function at the begin of each pruning iteration. Each pruning iteration contains several epochs. In this function, we will: 1. update the sparsity, 2. reload the best model from the previous iteration, 3. generate the prune mask, and 4. apply the mask to prune the weight of the model with increased sparsity.

save_model_state_for_all_rank()

sensitivity_analysis(trainer, state, eval_data, metric_reporter, train_config)

Analysis the sensitivity of each weight tensor to the metric. Prune the weight tensor one by one and evaluate the metric if the correspond weight tensor is pruned. :param trainer: batch iterator of training data :type trainer: trainer :param state: the state of the current training :type state: TrainingState :param eval_data: batch iterator of evaluation data :type eval_data: BatchIterator :param metric_reporter: compute metric based on training :type metric_reporter: MetricReporter :param output and report results to console, file.. etc: :param train_config: training config :type train_config: PyTextConfig

Returns:a string of each layer sensitivity to metric. Return type:analysis_result

sparsification_condition(state)

sparsify(state)

apply the mask to sparsify the weight tensor

class pytext.optimizer.sparsifiers.sparsifier.Sparsifier(config=None, *args, **kwargs)

Bases: pytext.config.component.Component

get_current_sparsity(model: pytext.models.model.Model) → float

get_sparsifiable_params(*args, **kwargs)

initialize(*args, **kwargs)

op_pre_epoch(*args, **kwargs)

save_model_state_for_all_rank()

sparsification_condition(*args, **kwargs)

sparsify(*args, **kwargs)

class pytext.optimizer.sparsifiers.sparsifier.State

Bases: enum.Enum

An enumeration.

ANALYSIS = 'Analysis'

OTHERS = 'Others'

Module contents