tenzr::optim::Adam¶

Adam optimizer with optional AMSGrad and decoupled weight decay.

#include <adam.h>

Inherits from tenzr::optim::GradientTransformation

Inherited by tenzr::optim::AdamW

Public Functions¶

Protected Attributes¶

Additional inherited members¶

Public Functions inherited from tenzr::optim::GradientTransformation

Public Functions Documentation¶

function Adam¶

explicit Adam(
    double lr =1.0e-3,
    double beta1 =0.9,
    double beta2 =0.999,
    double eps =1.0e-8,
    double weight_decay =0.0,
    bool amsgrad =false,
    bool decoupled_weight_decay =false,
    ScheduleFn schedule =nullptr
)

Parameters:

• lr Base learning rate.
• beta1 First moment decay.
• beta2 Second moment decay.
• eps Numerical epsilon.
• weight_decay Weight-decay coefficient.
• amsgrad Whether to track the maximum second moment.
• decoupled_weight_decay Whether to decouple weight decay (AdamW).
• schedule Optional learning-rate schedule by step.

Create an Adam/AdamW transformation.

function Init¶

virtual OptState Init(
    const std::unordered_map< std::string, Tensor > & params
) override

Parameters:

• params Parameters to optimize, keyed by name.
• params Parameters to be optimized.

Return:

• Zero-initialized Adam state.
• Fresh optimizer state.

Reimplements: tenzr::optim::GradientTransformation::Init

Allocate the first/second moment buffers, plus the AMSGrad maximum when enabled.

function Update¶

virtual std::unordered_map< std::string, Tensor > Update(
    const std::unordered_map< std::string, Tensor > & grads,
    OptState & state,
    const std::unordered_map< std::string, Tensor > & params
) override

Parameters:

• grads Gradients keyed by parameter name.
• state Optimizer state, updated in place.
• params Current parameters keyed by name.
• grads Named gradients.
• state Optimizer state, updated in place.
• params Current parameter values.

Return:

• Parameter updates.
• Named updates to add to each parameter.

Reimplements: tenzr::optim::GradientTransformation::Update

Compute the bias-corrected Adam/AdamW parameter updates.

function Lr¶

inline double Lr() const

Return: Learning rate.

function Beta1¶

inline double Beta1() const

Return: First-moment decay rate.

function Beta2¶

inline double Beta2() const

Return: Second-moment decay rate.

function Eps¶

inline double Eps() const

Return: Numerical-stability epsilon.

function WeightDecay¶

inline double WeightDecay() const

Return: Weight-decay coefficient.

function Amsgrad¶

inline bool Amsgrad() const

Return: true when AMSGrad is enabled.

Protected Attributes Documentation¶

variable lr_¶

double lr_;

variable beta1_¶

double beta1_;

variable beta2_¶

double beta2_;

variable eps_¶

double eps_;

variable weight_decay_¶

double weight_decay_;

variable amsgrad_¶

bool amsgrad_;

variable decoupled_weight_decay_¶

bool decoupled_weight_decay_;

variable schedule_¶

ScheduleFn schedule_;