research
SGCA
Sin-Gated Control Attention: separating attention retrieval from learned residual control.
sgcaattentiongating
Published work connected by this topic.
Sin-Gated Control Attention: separating attention retrieval from learned residual control.
Parameter-matched small language-model experiments with a sinusoidal attention gate.
A careful reading of parameter-matched Tiny Shakespeare results and their limits.