pytext.optimizer.sparsifiers package

Submodules

pytext.optimizer.sparsifiers.blockwise_sparsifier module

class pytext.optimizer.sparsifiers.blockwise_sparsifier.BlockwiseMagnitudeSparsifier(sparsity, starting_epoch, frequency, block_size, columnwise_blocking, accumulate_mask, layerwise_pruning)

Bases: pytext.optimizer.sparsifiers.sparsifier.L0_projection_sparsifier

running blockwise magnitude-based sparsification

Parameters:

• block_size – define the size of each block
• columnwise_blocking – define columnwise block if true
• starting_epoch – sparsification_condition returns true only after starting_epoch
• frequency – sparsification_condition only if number of steps devides frequency
• accumulate_mask – if true, the mask after each .sparisfy() will be reused
• sparsity – percentage of zeros among the UNPRUNED parameters.
• on how the sparsifier work (Examples) –
• matrix (2D) –
• [ – 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
• ] –
• 3 X 1 block (define) –
• [ – ***** *** 0 1 2 3 4 ****** *** 5 6 7 8 9 ****** *** 10 11 12 13 14 ****** *** 15 16 17 18 19 ****** *** 20 21 22 23 24 ****** ***
• ] –
• l1 norm of each block and sort them. Retain blocks with largest (compute) –
• values until sparsity threshold is met (absolute) –

classmethod from_config(config: pytext.optimizer.sparsifiers.blockwise_sparsifier.BlockwiseMagnitudeSparsifier.Config)

get_current_sparsity(model)

get_masks(model: torch.nn.modules.module.Module, pre_masks: List[torch.Tensor] = None) → List[torch.Tensor]

Note: this function returns the masks only but do not sparsify or modify the weights

prune x% of weights among the weights with “1” in pre_masks

Parameters:

• model – Model
• pre_masks – list of FloatTensors where “1” means retained the weight and “0” means pruned the weight

Returns:

List[torch.Tensor], intersection of new masks and pre_masks, so that “1” only if the weight is selected after new masking and pre_mask

Return type:

masks

get_sparsifiable_params(model, requires_name=False)

pytext.optimizer.sparsifiers.sparsifier module

class pytext.optimizer.sparsifiers.sparsifier.CRF_L1_SoftThresholding(lambda_l1: float, starting_epoch: int, frequency: int)

Bases: pytext.optimizer.sparsifiers.sparsifier.CRF_SparsifierBase

implement l1 regularization:

min Loss(x, y, CRFparams) + lambda_l1 * ||CRFparams||_1

and solve the optimiation problem via (stochastic) proximal gradient-based method i.e., soft-thresholding

param_updated = sign(CRFparams) * max ( abs(CRFparams) - lambda_l1, 0)

classmethod from_config(config: pytext.optimizer.sparsifiers.sparsifier.CRF_L1_SoftThresholding.Config)

sparsify(state)

class pytext.optimizer.sparsifiers.sparsifier.CRF_MagnitudeThresholding(sparsity, starting_epoch, frequency, grouping)

Bases: pytext.optimizer.sparsifiers.sparsifier.CRF_SparsifierBase

magnitude-based (equivalent to projection onto l0 constraint set) sparsification on CRF transition matrix. Preserveing the top-k elements either rowwise or columnwise until sparsity constraint is met.

classmethod from_config(config: pytext.optimizer.sparsifiers.sparsifier.CRF_MagnitudeThresholding.Config)

sparsify(state)

class pytext.optimizer.sparsifiers.sparsifier.CRF_SparsifierBase(config=None, *args, **kwargs)

Bases: pytext.optimizer.sparsifiers.sparsifier.Sparsifier

get_sparsifiable_params(model: torch.nn.modules.module.Module)

get_transition_sparsity(transition)

sparsification_condition(state)

class pytext.optimizer.sparsifiers.sparsifier.L0_projection_sparsifier(sparsity, starting_epoch, frequency, layerwise_pruning=True, accumulate_mask=False)

Bases: pytext.optimizer.sparsifiers.sparsifier.Sparsifier

L0 projection-based (unstructured) sparsification

Parameters:

• weights (torch.Tensor) – input weight matrix
• sparsity (float32) – the desired sparsity [0-1]

apply_masks(model: pytext.models.model.Model, masks: List[torch.Tensor])

apply given masks to zero-out learnable weights in model

classmethod from_config(config: pytext.optimizer.sparsifiers.sparsifier.L0_projection_sparsifier.Config)

get_masks(model: pytext.models.model.Model, pre_masks: List[torch.Tensor] = None) → List[torch.Tensor]

Note: this function returns the masks only but do not sparsify or modify the weights

prune x% of weights among the weights with “1” in pre_masks

Parameters:

• model – Model
• pre_masks – list of FloatTensors where “1” means retained the weight and “0” means pruned the weight

Returns:

List[torch.Tensor], intersection of new masks and pre_masks, so that “1” only if the weight is selected after new masking and pre_mask

Return type:

masks

get_sparsifiable_params(model: pytext.models.model.Model)

sparsification_condition(state)

sparsify(state)

obtain a mask and apply the mask to sparsify

class pytext.optimizer.sparsifiers.sparsifier.Sparsifier(config=None, *args, **kwargs)

Bases: pytext.config.component.Component

get_current_sparsity(model: pytext.models.model.Model) → float

get_sparsifiable_params(*args, **kwargs)

sparsification_condition(*args, **kwargs)

sparsify(*args, **kwargs)

Module contents