tenzr::optim::Lion¶

Lion optimizer using sign-based updates with first- and second-moment decays.

#include <lion.h>

Inherits from tenzr::optim::GradientTransformation

Public Functions¶

Additional inherited members¶

Public Functions inherited from tenzr::optim::GradientTransformation

Public Functions Documentation¶

function Lion¶

explicit Lion(
    double lr =1.0e-4,
    double beta1 =0.9,
    double beta2 =0.99,
    double weight_decay =0.0,
    ScheduleFn schedule =nullptr
)

Parameters:

• lr Base learning rate.
• beta1 Interpolation factor for the update direction.
• beta2 Momentum decay for the running estimate.
• weight_decay Decoupled weight-decay coefficient.
• schedule Optional learning-rate schedule by step.

Create a Lion transformation.

function Init¶

virtual OptState Init(
    const std::unordered_map< std::string, Tensor > & params
) override

Parameters:

• params Parameters to optimize, keyed by name.
• params Parameters to be optimized.

Return:

• Zero-initialized Lion state.
• Fresh optimizer state.

Reimplements: tenzr::optim::GradientTransformation::Init

Allocate the momentum buffer for each parameter.

function Update¶

virtual std::unordered_map< std::string, Tensor > Update(
    const std::unordered_map< std::string, Tensor > & grads,
    OptState & state,
    const std::unordered_map< std::string, Tensor > & params
) override

Parameters:

• grads Gradients keyed by parameter name.
• state Optimizer state, updated in place.
• params Current parameters keyed by name.
• grads Named gradients.
• state Optimizer state, updated in place.
• params Current parameter values.

Return:

• Parameter updates.
• Named updates to add to each parameter.

Reimplements: tenzr::optim::GradientTransformation::Update

Compute the sign-based Lion parameter updates.

function Lr¶

inline double Lr() const

Return: Learning rate.

function Beta1¶

inline double Beta1() const

Return: First-moment decay rate.

function Beta2¶

inline double Beta2() const

Return: Second-moment decay rate.

function WeightDecay¶

inline double WeightDecay() const

Return: Weight-decay coefficient.