b.
Color theme
Let’s talk
Explore
Appearance
Color theme
Alethic model news

SGCA led for 145M tokens. Then GQA passed it.

Alethic-300M is a 302,908,416-parameter decoder-only language model, trained from scratch by Bekhruz Suleyman, an independent researcher and engineer in Uzbekistan. It is a matched pair of language models trained on the same 1B tokens. One uses Sin-Gated Control Attention (SGCA). The other is a plain grouped-query attention (GQA) baseline. A third run shows where the gap comes from.

GQA0.8068Final validation bits per byte. Baseline, 302.9M parameters
GQA-narrow0.8110Final validation bits per byte. No gate, 277.7M parameters, +0.0042 against GQA
SGCA0.8472Final validation bits per byte. Sine gate, 302.9M parameters, +0.0404 against GQA

Validation loss relative to the GQA baseline

Bits per byte minus GQA's value at the same step. Below the GQA line is better than GQA.

  • SGCA
  • GQA-narrow (no gate, SGCA’s FFN width)
−0.15−0.10−0.05GQA+0.050M200M400M600M800M1BTraining tokensCurves cross near 145M tokensSGCA ahead by 0.140 at 66MSGCA +0.0404GQA-narrow +0.0042
Setup

What was compared

SGCA keeps grouped-query attention as the retrieval step and changes what is added to the residual stream. Attention’s output passes through a sine and is multiplied by a learned linear read of the same input.

C = (W·H + b) · (α · sin A),   A = o_proj(GQA(H)),   X′ = X + C

The gate adds 1,050,624 parameters per layer. To keep both models at exactly 302,908,416 parameters, SGCA’s feed-forward width drops from 2,560 to 2,218. So SGCA differs from the baseline in two ways at once: the gate, and a 13.4% narrower FFN.

Shape24 layers, d_model 1,024, 16 query and 4 key/value heads, context 4,096
Data1,000,013,824 tokens, one pass: Python 36.4%, other code 16.3%, FineWeb-Edu 28.6%, math 8.6%, Uzbek 10.1%
Training15,259 steps of 65,536 tokens, AdamW, peak LR 4e-4, cosine to 4e-5, 763 warmup steps, bf16, seed 1234
MetricBits per byte on 1,017,192 held-out tokens from six sources (lower is better)
HardwareOne H100 80 GB per run
Result

The curves

SGCA is well ahead at 66M tokens (1.463 against 1.602 bits per byte). The two validation curves cross near 145M tokens, and GQA stays ahead from there to the end. GQA-narrow sits almost on top of GQA the whole way.

Validation bits per byte

Measured every 1,000 steps and at the final step. Hover, or focus the chart and use the arrow keys, to read values.

  • GQA
  • SGCA
  • GQA-narrow
0.81.01.21.41.60M200M400M600M800M1BTraining tokens
Show the numbers as a table
StepTokensGQAGQA-narrowSGCASGCA − GQA
1,00066M1.60241.59791.4627−0.1397
2,000131M1.16191.17071.1564−0.0054
3,000197M1.05471.05981.0741+0.0195
4,000262M1.00281.00381.0285+0.0256
5,000328M0.95710.96040.9892+0.0321
6,000393M0.92730.93020.9612+0.0339
7,000459M0.90220.90510.9375+0.0353
8,000524M0.88220.88410.9184+0.0362
9,000590M0.86390.86830.9020+0.0381
10,000655M0.84880.85310.8879+0.0391
11,000721M0.83650.84020.8761+0.0395
12,000786M0.82580.82980.8659+0.0402
13,000852M0.81760.82140.8571+0.0395
14,000918M0.81170.81570.8515+0.0399
15,000983M0.80780.81180.8480+0.0402
15,2591.0B0.80680.81100.8472+0.0404
Ablation

Where the gap comes from

GQA-narrow is the baseline with SGCA’s narrower FFN and no gate. It has 277,693,440 parameters, 25.2M fewer than either matched model. Its distance from GQA is the cost of the FFN cut. The rest of SGCA’s gap belongs to the gate.

SGCA’s 0.0404 bits-per-byte gap to GQA at 1B tokens, split in two

  • Narrower FFN: 0.0042
  • Sine gate: 0.0362

The split is stable: from step 4,000 to the end the FFN share stays between 4% and 11%. SGCA has 25.2M more parameters than GQA-narrow and still finishes 4.5% worse.

Gap to GQA by source, at 1B tokens

Bits per byte above GQA. The FFN share is between 7% and 15% on every source.

  • SGCA
  • GQA-narrow
GQA+0.01+0.02+0.03+0.04+0.05Other code+0.0489+0.0042Uzbek (Latin)+0.0482+0.0051Math+0.0453+0.0040Uzbek (Cyrillic)+0.0383+0.0032Python+0.0378+0.0028FineWeb-Edu+0.0373+0.0056
Show the numbers as a table
SourceGQAGQA-narrowSGCAFFN share
Other code0.64650.65060.69539%
Uzbek (Latin)1.15471.15971.202911%
Math1.04941.05331.09469%
Uzbek (Cyrillic)0.76170.76490.79998%
Python0.56960.57250.60747%
FineWeb-Edu0.97660.98211.013815%
Mechanism

The gate as a surface

What one channel adds to the residual stream

  • GQA adds attention’s output A directly. The surface is a plane, and g does not exist.
  • SGCA adds g · sin A, where g = W·H + b. Near A = 0 the sine is almost the identity (sin 0.06 = 0.05996), so early in training SGCA behaves like a product of two linear reads.
  • Further out the sine bends. Its slope is cos A, which reaches zero at A = ±π/2, and A and π − A give the same output.

SGCA: the update can never be larger than |g|. A runs from −π to π and g from −1.5 to 1.5. Drag to rotate.

This shows the function each model applies, for one channel with α = 1. It illustrates the definition and is not a measurement of the trained models. Whether the trained SGCA model works in the bent region is an open question.

Cost

Speed

SGCA and GQA have the same parameter count and the same counted FLOPs per token (3.025 billion), yet SGCA trains 6.8% slower than GQA and 9.3% slower than GQA-narrow. The extra time is in the elementwise gate work, which the FLOP formula does not count.

ModelTokens per secondFLOP utilization
GQA148,94845.6%
GQA-narrow153,02344.5%
SGCA138,83442.5%
Limits

What this does not show

  • One seed per model.Differences below about 0.003 bits per byte are within the jitter visible in the curves. The 0.040 gap is more than ten times that.
  • One scale.1B tokens is 3.3 tokens per parameter, far below the roughly 20 that is compute-optimal. A 6B-token run could order the models differently. It has not been run.
  • One learning rate.It was chosen from a 100M-token sweep where SGCA won at every rate tried. SGCA may want a different schedule.
  • One form of the gate.Here the gated term replaces attention’s direct path and α starts at 1. A variant that keeps the direct path and starts as exactly GQA is untested.
Timeline

All updates

  1. Ablation

    The gate, not the narrower FFN, explains the gap

    GQA-narrow (the baseline with SGCA’s FFN width and no gate, 277.7M parameters) finished at 0.8110 BPB, only 0.0042 behind full GQA. About 90% of SGCA’s 0.0404 gap belongs to the gate.

  2. Result

    1B-token matched pair: GQA 0.8068, SGCA 0.8472

    Both 302.9M-parameter models trained on the same 1,000,013,824 tokens. SGCA led until about 145M tokens, then GQA passed it and stayed ahead on all six validation sources.

  3. Training

    Learning-rate sweep at 100M tokens

    Three rates (2e-4, 4e-4, 8e-4) for each model. SGCA was ahead at every rate at this early budget (1.2697 against 1.3942 BPB at 4e-4). 4e-4 was chosen for the 1B runs.

  4. Tokenizer

    Tokenizer frozen: 50,000-piece Unigram with exact round trips

    alethic-tokenizer-v1 decodes every test document back to the exact input, keeps Python indentation at 4, 8 and 12 spaces, and uses 2.17 tokens per word on Uzbek Latin and 2.55 on Uzbek Cyrillic.

  5. Infrastructure

    Alethic-300M rewrite started

    New repository with both model variants at exactly 302,908,416 parameters, a crash-safe data pipeline, resumable training and a test suite that runs on a laptop CPU.

Background

Read the research

The definition of SGCA, the earlier small-model experiments and the released models are in the research archive. Code, configs and the full research log are on GitHub. Weights, checkpoints, the tokenizer and tokenized data are public on Hugging Face.