LmFineTuning.Config

Component: LmFineTuning

class LmFineTuning.Config[source]

Bases: Scheduler.Config

All Attributes (including base classes)

cut_frac: float = 0.1
The fraction of iterations we increase the learning rate. Default 0.1
ratio: int = 32
How much smaller the lowest LR is from the maximum LR eta_max.
non_pretrained_param_groups: int = 2
Number of param_groups, starting from the end, that were not pretrained. The default value is 2, since the base Model class supplies to the optimizer typically one param_group from the embedding and one param_group from its other components.
lm_lr_multiplier: float = 1.0
Factor to multiply lr for all pretrained layers by.
lm_use_per_layer_lr: bool = False
Whether to make each pretrained layer’s lr one-half as large as the next (higher) layer.
lm_gradual_unfreezing: bool = True
Whether to unfreeze layers one by one (per epoch).
last_epoch: int = -1
Though the name is last_epoch, it means last batch update. last_batch_update: = current_epoch_number * num_batches_per_epoch + batch_id after each batch update, it will increment 1

Default JSON

{
    "cut_frac": 0.1,
    "ratio": 32,
    "non_pretrained_param_groups": 2,
    "lm_lr_multiplier": 1.0,
    "lm_use_per_layer_lr": false,
    "lm_gradual_unfreezing": true,
    "last_epoch": -1
}