tensornn.optimizers

This file contains optimizers that help tune your neural network. Optimizers enable us to improve our neural network efficiently.

Classes

`Adam`	Adaptive Moment Estimation, or famously known as Adam, is the most widely used optimizer in the machine learning community.
`Optimizer`	Base optimizer class.
`RMSProp`	Root Mean Square Propagation optimizer.
`SGD`	Stochastic gradient descent optimizer.

class tensornn.optimizers.Adam(learning_rate: float = 0.001, momentum: float = 0.9, decay: float = 0.999, epsilon: float = 1e-08)

Bases: Optimizer

Adaptive Moment Estimation, or famously known as Adam, is the most widely used optimizer in the machine learning community. By combining the ideas of momentum and RMSProp, people have found that Adam works surprisingly well for a lot of situations and is able to converge faster than other optimizers.

register(model: TensorNNObject) → None

Register the optimizer with the model.

Parameters:: model – the model to register the optimizer with

step() → None: Perform a step of optimization.

class tensornn.optimizers.Optimizer(learning_rate: float = 0.01)

Bases: ABC, TensorNNObject

Base optimizer class. All optimizers should inherit from this.

learning_rate: float

model: TensorNNObject

register(model: TensorNNObject) → None

Register the optimizer with the model.

Parameters:: model – the model to register the optimizer with

reset() → None: Reset the optimizer.

set_lr(lr: float) → None

Set the learning rate of the optimizer.

Parameters:: lr – the learning rate to set

abstractmethod step() → None: Perform a step of optimization.

class tensornn.optimizers.RMSProp(learning_rate: float = 0.001, decay: float = 0.9, epsilon: float = 1e-08)

Bases: Optimizer

Root Mean Square Propagation optimizer.

register(model: TensorNNObject) → None

Register the optimizer with the model.

Parameters:: model – the model to register the optimizer with

step() → None: Perform a step of optimization.

class tensornn.optimizers.SGD(learning_rate: float = 0.001, momentum: float = 0.9)

Bases: Optimizer

Stochastic gradient descent optimizer. But, this is actually mini-batch stochastic gradient descent. You can make it standard SGD by setting batch_size to 1 in training.

This optimizer is the most basic optimizer and is the most widely used optimizer. It works by taking the gradient of the loss function with respect to the weights and biases of the model and adjusting them in the opposite direction of the gradient to minimize the loss function.

This implementation of SGD also supports momentum. Momentum is a technique which helps the model ‘walk’ down the loss function as a ball would roll down a hill. Successive steps in the same direction are accelerated, and the model is able to escape local minima and saddle points more easily. To turn off momentum, set the momentum parameter to 0.