tenzr::optim::SGD¶

Stochastic gradient descent with momentum, weight decay, and Nesterov options.

#include <sgd.h>

Inherits from tenzr::optim::GradientTransformation

Public Functions¶

Additional inherited members¶

Public Functions inherited from tenzr::optim::GradientTransformation

Public Functions Documentation¶

function SGD¶

explicit SGD(
    double lr =1.0e-3,
    double momentum =0.0,
    double dampening =0.0,
    double weight_decay =0.0,
    bool nesterov =false,
    ScheduleFn schedule =nullptr
)

Parameters:

• lr Base learning rate.
• momentum Momentum factor.
• dampening Momentum dampening.
• weight_decay L2 penalty added to the gradient.
• nesterov Whether to use Nesterov momentum.
• schedule Optional learning-rate schedule by step.

Create an SGD transformation.

function Init¶

virtual OptState Init(
    const std::unordered_map< std::string, Tensor > & params
) override

Parameters:

• params Parameters to optimize, keyed by name.
• params Parameters to be optimized.

Return:

• Zero-initialized SGD state.
• Fresh optimizer state.

Reimplements: tenzr::optim::GradientTransformation::Init

Allocate momentum buffers when momentum is enabled.

function Update¶

virtual std::unordered_map< std::string, Tensor > Update(
    const std::unordered_map< std::string, Tensor > & grads,
    OptState & state,
    const std::unordered_map< std::string, Tensor > & params
) override

Parameters:

• grads Gradients keyed by parameter name.
• state Optimizer state, updated in place.
• params Current parameters keyed by name.
• grads Named gradients.
• state Optimizer state, updated in place.
• params Current parameter values.

Return:

• Parameter updates.
• Named updates to add to each parameter.

Reimplements: tenzr::optim::GradientTransformation::Update

Compute the SGD (optionally momentum/Nesterov) parameter updates.

function Lr¶

inline double Lr() const

Return: Learning rate.

function Momentum¶

inline double Momentum() const

Return: Momentum factor.

function WeightDecay¶

inline double WeightDecay() const

Return: L2 weight-decay coefficient.

function Nesterov¶

inline bool Nesterov() const

Return: true when Nesterov momentum is enabled.