tenzr::optim¶

Classes¶

	Name
class	tenzr::optim::Adam Adam optimizer with optional AMSGrad and decoupled weight decay.
class	tenzr::optim::AdamW Adam variant that applies decoupled weight decay.
class	tenzr::optim::ExponentialMovingAverage Maintains an exponential moving average of a parameter set.
class	tenzr::optim::GradientTransformation Abstract gradient transformation defining Init and Update for an optimizer.
class	tenzr::optim::Lion Lion optimizer using sign-based updates with first- and second-moment decays.
class	tenzr::optim::Optimizer Binds a parameter set to a gradient transformation and applies update steps.
struct	tenzr::optim::OptState Optimizer state holding the step counter and per-parameter tensors.
class	tenzr::optim::SGD Stochastic gradient descent with momentum, weight decay, and Nesterov options.

Types¶

	Name
using std::function< double(uint64_t step)>	ScheduleFn Learning-rate schedule mapping a global step to a learning rate.

Functions¶

	Name
std::unordered_map< std::string, Tensor >	ClipByValue(const std::unordered_map< std::string, Tensor > & grads, double min_val, double max_val)
std::unordered_map< std::string, Tensor >	ClipByGlobalNorm(const std::unordered_map< std::string, Tensor > & grads, double max_norm, double * calculated_norm =nullptr)
ScheduleFn	ConstantSchedule(double lr)
ScheduleFn	LinearWarmup(double peak_lr, uint64_t warmup_steps)
ScheduleFn	CosineDecay(double init_lr, uint64_t decay_steps, double alpha =0.0)
ScheduleFn	WarmupCosineDecay(double peak_lr, uint64_t warmup_steps, uint64_t total_steps, double min_lr =0.0)

Types Documentation¶

using ScheduleFn¶

using tenzr::optim::ScheduleFn = typedef std::function<double(uint64_t step)>;

Learning-rate schedule mapping a global step to a learning rate.

Functions Documentation¶

function ClipByValue¶

std::unordered_map< std::string, Tensor > ClipByValue(
    const std::unordered_map< std::string, Tensor > & grads,
    double min_val,
    double max_val
)

Parameters:

grads Gradients keyed by parameter name.
min_val Lower clamp bound.
max_val Upper clamp bound.
grads Named gradients to clip.
min_val Lower clamp bound.
max_val Upper clamp bound, which must be >= min_val.

Return:

Clipped gradients.
Newly allocated clipped gradients keyed by name.

Clamp every gradient element to [min_val, max_val]. Clamp every gradient element into [min_val, max_val].

function ClipByGlobalNorm¶

std::unordered_map< std::string, Tensor > ClipByGlobalNorm(
    const std::unordered_map< std::string, Tensor > & grads,
    double max_norm,
    double * calculated_norm =nullptr
)

Parameters:

grads Gradients keyed by parameter name.
max_norm Maximum allowed global norm.
calculated_norm Optional output receiving the pre-clip global norm.
grads Named gradients to clip; returned unchanged when already within the norm.
max_norm Maximum allowed global norm, which must be positive.
calculated_norm Optional out-parameter receiving the pre-clip global norm.

Return:

Clipped gradients.
Clipped gradients, or the originals when no clipping was needed.

Rescale gradients so their global L2 norm does not exceed max_norm. Rescale gradients so their global L2 norm does not exceed max_norm.

function ConstantSchedule¶

inline ScheduleFn ConstantSchedule(
    double lr
)

Parameters:

lr Learning rate returned at every step.

Return: Schedule function.

Constant learning-rate schedule.

function LinearWarmup¶

inline ScheduleFn LinearWarmup(
    double peak_lr,
    uint64_t warmup_steps
)

Parameters:

peak_lr Target learning rate.
warmup_steps Number of warmup steps; 0 applies peak_lr immediately.

Return: Schedule function.

Linearly ramp the learning rate from 0 to peak_lr.

function CosineDecay¶

inline ScheduleFn CosineDecay(
    double init_lr,
    uint64_t decay_steps,
    double alpha =0.0
)

Parameters:

init_lr Initial learning rate.
decay_steps Number of steps over which to decay.
alpha Final learning-rate fraction.

Return: Schedule function.

Cosine-anneal the learning rate from init_lr to init_lr * alpha.

function WarmupCosineDecay¶

inline ScheduleFn WarmupCosineDecay(
    double peak_lr,
    uint64_t warmup_steps,
    uint64_t total_steps,
    double min_lr =0.0
)

Parameters:

peak_lr Peak learning rate reached after warmup.
warmup_steps Number of warmup steps.
total_steps Total steps including warmup.
min_lr Floor learning rate after decay.

Return: Schedule function.

Linear warmup followed by cosine decay.