MultiStepLR

MultiStepLR is a scheduler that decays the learning rate with a certain multiplicative factor set by the user when a certain milestone is reached. The milestone is the number of epochs which is set by the user.
Major Parameters
Milestones
They are a pair of values that sets the number of epochs after which the learning rate is scaled.
Gamma
It is the multiplicative factor by which the learning rate is scaled.
Mathematical Demonstration
Let us demonstrate the functioning of the MultiStepLR with a simple calculation.
If Milestones are set to be 30 and 80 base learning rate being 0.05, and gamma 0.1, then
for, 0<=epoch<300<=epoch<300<=epoch<30
 , lr=0.05lr=0.05lr=0.05
 
for, 30<=epoch<8030<=epoch<8030<=epoch<80
 , lr=0.05∗0.1=0.005lr=0.05*0.1=0.005lr=0.05∗0.1=0.005
 
for , epoch>=80epoch>=80epoch>=80
 , lr=0.05∗0.12=0.0005lr=0.05*0.1^2=0.0005lr=0.05∗0.12=0.0005
 
Code Implementation
1
import torch
2
model = [Parameter(torch.randn(2, 2, requires_grad=True))]
3
optimizer = torch.optim.AdamW(model.parameters(), lr=learning_rate, weight_decay=0.01, amsgrad=False)
4
scheduler=torch.optim.lr_scheduler.MultiStepLR(optimizer, milestones=[30,80], gamma=0.1, last_epoch=-1, verbose=False)
5
for epoch in range(20):
6
    for input, target in dataset:
7
        optimizer.zero_grad()
8
        output = model(input)
9
        loss = loss_fn(output, target)
10
        loss.backward()
11
        optimizer.step()
12
    scheduler.step()
Copied!

Schedulers - Previous

CyclicLR

Next - Schedulers

StepLR

Last modified 9mo ago

Copy link

Contents

Major Parameters

Milestones

Gamma

Mathematical Demonstration

Code Implementation