Move every weight a fixed distance toward zero each step: Monty's
attention/decay.py::LinearWeightDecay.
A weight of 0.7 with a rate of 0.1 becomes 0.6, and -0.7 becomes
-0.6. A weight already within rate of zero is not nudged past it — it
becomes exactly 0.0, so that a steady decay cannot make a weight oscillate
around zero or flip its sign, which would turn "look here" into "avoid here"
by forgetting.
Summary
Functions
A grid in which every weight has moved one step toward zero.
The default rate, so a config can say linear and mean what Monty means.
A decay of rate per step, which must not be negative.
Types
@type t() :: %Monty.Attention.Decay.LinearWeightDecay{rate: float()}
Functions
A grid in which every weight has moved one step toward zero.
@spec default_rate() :: float()
The default rate, so a config can say linear and mean what Monty means.
A decay of rate per step, which must not be negative.
A negative rate would move weights away from zero, which is the opposite of forgetting, so it is refused rather than accepted and misread.