RMSprop

RMSprop is  another optimization technique where there is a different learning rate for each parameter. The learning rate is varied by calculating the exponential moving average of the gradient squared and using it to further update the parameter.
Mathematically, the exponential moving average of the gradient squared is given as follows,
Sdw=α⋅Sdw+(1−α)⋅∂w2S_{dw}=\alpha \cdot S_{dw} +(1-\alpha) \cdot \partial w^2
Sdw​=α⋅Sdw​+(1−α)⋅∂w2
Here www
 is one of the parameters and β\betaβ
is the smoothing constant.
Value of α\alphaα
 is usually set to 0.99.
Then using SdwS_{dw}Sdw​
 to update the parameter,
w=w−η⋅∂wSdww=w-\eta \cdot \frac{\partial w}{\sqrt{S_{dw}}}w=w−η⋅Sdw​​∂w​
Here η\etaη
 is the Base Learning Rate.

Notice some implications of such an update. If the change of w with respect to objective function was very high, then the update would decrease since we are dividing with a high value. Similarly, if the change of w with respect to the objective function was low then the update would be higher.
Let us clarify with a help of contour lines,
​
Comparison of stochastic gradient descent and RMSprop
We can see that higher gradient in the vertical direction and the lower in the horizontal direction is slowing down the overall search process of the optimal solution. The use of RMSprop solves this problem by finding a better search “trajectory”.
Major Parameters
​Base Learning Rate​
​Momentum (SGD)​
​Epsilon Coefficient​
​Weight Decay​
Alpha
Centered
Alpha
Alpha is the same value at the α\alphaα
 in the aforementioned formula. Lower the value of alpha, lower the number of previous weights  taken into account for the calculation of exponential moving average of squared gradients.
Centered
If "Centered" is activated, then the RMS prop is calculated by normalizing the gradients with the variance. If not, then the uncentered second moment, as in the aforementioned formula, is used.
Centering might help in the training process but will be slightly more computationally expensive.
Code Implementation
1
# importing the library
2
import torch
3
import torch.nn as nn
4
​
5
x = torch.randn(10, 3)
6
y = torch.randn(10, 2)
7
​
8
# Build a fully connected layer.
9
linear = nn.Linear(3, 2)
10
​
11
# Build MSE loss function and optimizer.
12
criterion = nn.MSELoss()
13
​
14
# Optimization method using RMSprop
15
optimizer = torch.optim.RMSProp(linear.parameters(), lr=0.01, alpha=0.99,
16
 eps=1e-08, weight_decay=0, momentum=0, centered=False)
17
​
18
# Forward pass.
19
pred = linear(x)
20
​
21
# Compute loss.
22
loss = criterion(pred, y)
23
print('loss:', loss.item())
24
​
25
optimizer.step()
Copied!
​

Solvers / Optimizers 🧮 - Previous

Rprop

Next - Solvers / Optimizers 🧮

Adadelta

Last modified 9mo ago

Copy link

Contents

Major Parameters

Alpha

Centered

Code Implementation

​