Phase I uses 8,240 tokens
The current context decision, separate from earlier long-context curricula.
Sin-Gated Control Attention: separating attention retrieval from learned residual control.
Attention looks back over earlier tokens and mixes their values into one number per feature, called A here. How much each token counts is its weight, and the weights always add up to 1.
Worked example: scores [2, 1, 0.1] become weights by softmax, w = e^s / Σe^s. Switching focus multiplies every score before softmax.
A = 0.66×1 + 0.24×-0.5 + 0.10×0.2 = 0.56One SGCA block, followed for a single feature with toy numbers: X = 2, A = 0.8, α = 0.5, WH+b = 1.2. Real values are vectors; this is the same arithmetic on one slot.
It plays by itself. Pick a step to stop it.
H = RMSNorm(X)SGCA passes the attention output A through sin(A), then scales it by a learned amplitude α. For small A, sin(A) ≈ A, so it behaves like a plain scaled copy. As A grows, the sine folds back.
Worked numbers, with α = 1: A = 0.5 gives sin(0.5) ≈ 0.48, A = 1.5 gives 1.00, and A = 3.0 gives 0.14.
Many squashing functions saturate: past a certain size, a bigger input barely changes the output. A sine does not saturate. It keeps responding, and it changes sign.
Whether that extra sensitivity helps a language model is a question for experiments, not for this picture.
To say the gate caused a difference, everything else must match. Each factor you leave unlocked is one more thing that could explain a gain.
The control branch adds parameters. To keep the count equal, the baseline gets the same number of extra parameters somewhere else, such as a wider feed-forward layer.
3 things differ: the gate + 2 unlocked. A gain could come from any of them.SGCA is an experimental decoder-only language-model mechanism. It retains grouped-query attention as retrieval, then uses its output inside a sinusoidal control term before the residual update. This is a research hypothesis, not a claim of general superiority.
The matched baseline adds retrieved information directly to the residual stream. SGCA computes an update from a learned projection of normalized input, modulated by attention output. Retrieval and the rule governing its influence become conceptually distinct.
: input residual stream.
: RMS-normalized input.
: output of GQA.
: learned linear control projection.
: elementwise multiplication.
: a learnable tensor, broadcast-compatible with .
The definition does not fix α to a scalar or matrix. Exact parameterization must be recorded in an implementation.
The block then proceeds through RMSNorm, SwiGLU, and the feed-forward residual.
The two paths share a GQA backbone. The control projection changes parameterization and computation; those differences must be accounted for.
A separate periodic control path may change representations, gradients, or optimization. It could also introduce instability or initialization sensitivity. Both possibilities require experiments.
Hold the tokenizer, data, order, batching, optimizer, schedule, token budget, context, precision, hardware class, and evaluation fixed where possible. Report parameter and compute differences and repeat across seeds.
Comparison | Question | Status |
|---|---|---|
SGCA / matched GQA | Effect of the complete control path | Planned validation |
sin / identity | Contribution of the sinusoid | Planned |
sin / tanh / sigmoid | Periodicity versus other gates | Planned |
learned / fixed α | Effect of learned modulation | Planned |
with / without projection | Contribution of WH + b | Planned |
parameter / geometry / compute matching | Sensitivity to matching method | Planned |
No outcomes are fabricated for these planned comparisons.
No current SGCA benchmark is published here. Earlier SinGatedLM experiments are related motivation and use a predecessor mechanism.
How should α be initialized? How sensitive is the path to attention-output scale? Does the gate help beyond a small dataset? What is its throughput cost at matched compute? Do gains survive multiple seeds and broader evaluation?
Complete public SGCA run artifacts are still needed. The Alethic overview identifies the available predecessor code and checkpoints.
Research description by Bekhruz Suleyman, updated 1 October 2026. No DOI or peer-reviewed publication is claimed. Cite the page URL and access date when referring to the formulation.
Language-model architecture research, from controlled small-model experiments to a planned ~1B Phase I.
Parameter-matched small language-model experiments with a sinusoidal attention gate.
Experimental pretraining and SinGatedAttention exploration on constrained compute.