b.
Color theme
Let’s talk
Explore
Appearance
Color theme
research / experimental

SGCA

Sin-Gated Control Attention: separating attention retrieval from learned residual control.

Alethic news / 10 Nov 2026The gate, not the narrower FFN, explains the gapGQA-narrow (the baseline with SGCA’s FFN width and no gate, 277.7M parameters) finished at 0.8110 BPB, only 0.0042 behind full GQA. About 90% of SGCA’s 0.0404 gap belongs to the gate.
Read the results

Step 1: retrieve a signal

Attention looks back over earlier tokens and mixes their values into one number per feature, called A here. How much each token counts is its weight, and the weights always add up to 1.

Worked example: scores [2, 1, 0.1] become weights by softmax, w = e^s / Σe^s. Switching focus multiplies every score before softmax.

A = 0.66×1 + 0.24×-0.5 + 0.10×0.2 = 0.56
Broad: every token contributes, so A is a blend.

Step 2: build the update

One SGCA block, followed for a single feature with toy numbers: X = 2, A = 0.8, α = 0.5, WH+b = 1.2. Real values are vectors; this is the same arithmetic on one slot.

It plays by itself. Pick a step to stop it.

H = RMSNorm(X)
The block input X is rescaled to H so its size is steady.

Step 3: the sine term

SGCA passes the attention output A through sin(A), then scales it by a learned amplitude α. For small A, sin(A) ≈ A, so it behaves like a plain scaled copy. As A grows, the sine folds back.

Worked numbers, with α = 1: A = 0.5 gives sin(0.5) ≈ 0.48, A = 1.5 gives 1.00, and A = 3.0 gives 0.14.

Move over the graph to probe a token position. This is an illustration of the mechanism, not a measured result.

Step 4: why try a sine?

Many squashing functions saturate: past a certain size, a bigger input barely changes the output. A sine does not saturate. It keeps responding, and it changes sign.

Whether that extra sensitivity helps a language model is a question for experiments, not for this picture.

From A = 2 to A = 3: tanh moves 0.96 → 0.995 (change 0.03). α·sin(A) moves 0.91 → 0.14 (change 0.77). Hover the graph to probe any A.

Step 5: a fair comparison

To say the gate caused a difference, everything else must match. Each factor you leave unlocked is one more thing that could explain a gain.

The control branch adds parameters. To keep the count equal, the baseline gets the same number of extra parameters somewhere else, such as a wider feed-forward layer.

3 things differ: the gate + 2 unlocked. A gain could come from any of them.

Abstract

SGCA is an experimental decoder-only language-model mechanism. It retains grouped-query attention as retrieval, then uses its output inside a sinusoidal control term before the residual update. This is a research hypothesis, not a claim of general superiority.

Motivation

The matched baseline adds retrieved information directly to the residual stream. SGCA computes an update from a learned projection of normalized input, modulated by attention output. Retrieval and the rule governing its influence become conceptually distinct.

Definition and mathematical formulation

H=RMSNorm(X)A=GQA(H)C=(WH+b)⊙[α⊙sin⁡(A)]X′=X+C\begin{gathered}H=\mathrm{RMSNorm}(X)\\A=\mathrm{GQA}(H)\\C=(WH+b)\odot[\alpha\odot\sin(A)]\\X'=X+C\end{gathered}
  • XX: input residual stream.

  • HH: RMS-normalized input.

  • AA: output of GQA.

  • WH+bWH+b: learned linear control projection.

  • ⊙\odot: elementwise multiplication.

  • α\alpha: a learnable tensor, broadcast-compatible with AA.

The definition does not fix α to a scalar or matrix. Exact parameterization must be recorded in an implementation.

Block architecture

The block then proceeds through RMSNorm, SwiGLU, and the feed-forward residual.

Baseline

Xbase′=X+AX′′=X′+SwiGLU(RMSNorm(X′))\begin{gathered}X'_{\mathrm{base}}=X+A\\X''=X'+\mathrm{SwiGLU}(\mathrm{RMSNorm}(X'))\end{gathered}

The two paths share a GQA backbone. The control projection changes parameterization and computation; those differences must be accounted for.

Hypotheses

A separate periodic control path may change representations, gradients, or optimization. It could also introduce instability or initialization sensitivity. Both possibilities require experiments.

Experimental controls

Hold the tokenizer, data, order, batching, optimizer, schedule, token budget, context, precision, hardware class, and evaluation fixed where possible. Report parameter and compute differences and repeat across seeds.

Ablation plan

Comparison

Question

Status

SGCA / matched GQA

Effect of the complete control path

Planned validation

sin / identity

Contribution of the sinusoid

Planned

sin / tanh / sigmoid

Periodicity versus other gates

Planned

learned / fixed α

Effect of learned modulation

Planned

with / without projection

Contribution of WH + b

Planned

parameter / geometry / compute matching

Sensitivity to matching method

Planned

No outcomes are fabricated for these planned comparisons.

Results

No current SGCA benchmark is published here. Earlier SinGatedLM experiments are related motivation and use a predecessor mechanism.

Limitations and open questions

How should α be initialized? How sensitive is the path to attention-output scale? Does the gate help beyond a small dataset? What is its throughput cost at matched compute? Do gains survive multiple seeds and broader evaluation?

Reproduction

Complete public SGCA run artifacts are still needed. The Alethic overview identifies the available predecessor code and checkpoints.

Citation

Research description by Bekhruz Suleyman, updated 1 October 2026. No DOI or peer-reviewed publication is claimed. Cite the page URL and access date when referring to the formulation.

From the workbench

Related threads

Keep exploring.

Back to research