WarmupScheduler.Config

Component: WarmupScheduler

class WarmupScheduler.Config

Bases: BatchScheduler.Config

All Attributes (including base classes)

warmup_steps: int = 10000

number of training steps over which to increase learning rate

inverse_sqrt_decay: bool = False

whether to perform inverse sqrt decay after the warmup phase

Default JSON

{
    "warmup_steps": 10000,
    "inverse_sqrt_decay": false
}