Alethic
Language-model architecture research, from controlled small-model experiments to a planned ~1B Phase I.
Alethic-300M is a 302,908,416-parameter decoder-only language model, trained from scratch by Bekhruz Suleyman, an independent researcher and engineer in Uzbekistan. It is a matched pair of language models trained on the same 1B tokens. One uses Sin-Gated Control Attention (SGCA). The other is a plain grouped-query attention (GQA) baseline. A third run shows where the gap comes from.
Bits per byte minus GQA's value at the same step. Below the GQA line is better than GQA.
SGCA keeps grouped-query attention as the retrieval step and changes what is added to the residual stream. Attention’s output passes through a sine and is multiplied by a learned linear read of the same input.
The gate adds 1,050,624 parameters per layer. To keep both models at exactly 302,908,416 parameters, SGCA’s feed-forward width drops from 2,560 to 2,218. So SGCA differs from the baseline in two ways at once: the gate, and a 13.4% narrower FFN.
| Shape | 24 layers, d_model 1,024, 16 query and 4 key/value heads, context 4,096 |
| Data | 1,000,013,824 tokens, one pass: Python 36.4%, other code 16.3%, FineWeb-Edu 28.6%, math 8.6%, Uzbek 10.1% |
| Training | 15,259 steps of 65,536 tokens, AdamW, peak LR 4e-4, cosine to 4e-5, 763 warmup steps, bf16, seed 1234 |
| Metric | Bits per byte on 1,017,192 held-out tokens from six sources (lower is better) |
| Hardware | One H100 80 GB per run |
SGCA is well ahead at 66M tokens (1.463 against 1.602 bits per byte). The two validation curves cross near 145M tokens, and GQA stays ahead from there to the end. GQA-narrow sits almost on top of GQA the whole way.
Measured every 1,000 steps and at the final step. Hover, or focus the chart and use the arrow keys, to read values.
| Step | Tokens | GQA | GQA-narrow | SGCA | SGCA − GQA |
|---|---|---|---|---|---|
| 1,000 | 66M | 1.6024 | 1.5979 | 1.4627 | −0.1397 |
| 2,000 | 131M | 1.1619 | 1.1707 | 1.1564 | −0.0054 |
| 3,000 | 197M | 1.0547 | 1.0598 | 1.0741 | +0.0195 |
| 4,000 | 262M | 1.0028 | 1.0038 | 1.0285 | +0.0256 |
| 5,000 | 328M | 0.9571 | 0.9604 | 0.9892 | +0.0321 |
| 6,000 | 393M | 0.9273 | 0.9302 | 0.9612 | +0.0339 |
| 7,000 | 459M | 0.9022 | 0.9051 | 0.9375 | +0.0353 |
| 8,000 | 524M | 0.8822 | 0.8841 | 0.9184 | +0.0362 |
| 9,000 | 590M | 0.8639 | 0.8683 | 0.9020 | +0.0381 |
| 10,000 | 655M | 0.8488 | 0.8531 | 0.8879 | +0.0391 |
| 11,000 | 721M | 0.8365 | 0.8402 | 0.8761 | +0.0395 |
| 12,000 | 786M | 0.8258 | 0.8298 | 0.8659 | +0.0402 |
| 13,000 | 852M | 0.8176 | 0.8214 | 0.8571 | +0.0395 |
| 14,000 | 918M | 0.8117 | 0.8157 | 0.8515 | +0.0399 |
| 15,000 | 983M | 0.8078 | 0.8118 | 0.8480 | +0.0402 |
| 15,259 | 1.0B | 0.8068 | 0.8110 | 0.8472 | +0.0404 |
GQA-narrow is the baseline with SGCA’s narrower FFN and no gate. It has 277,693,440 parameters, 25.2M fewer than either matched model. Its distance from GQA is the cost of the FFN cut. The rest of SGCA’s gap belongs to the gate.
The split is stable: from step 4,000 to the end the FFN share stays between 4% and 11%. SGCA has 25.2M more parameters than GQA-narrow and still finishes 4.5% worse.
Bits per byte above GQA. The FFN share is between 7% and 15% on every source.
| Source | GQA | GQA-narrow | SGCA | FFN share |
|---|---|---|---|---|
| Other code | 0.6465 | 0.6506 | 0.6953 | 9% |
| Uzbek (Latin) | 1.1547 | 1.1597 | 1.2029 | 11% |
| Math | 1.0494 | 1.0533 | 1.0946 | 9% |
| Uzbek (Cyrillic) | 0.7617 | 0.7649 | 0.7999 | 8% |
| Python | 0.5696 | 0.5725 | 0.6074 | 7% |
| FineWeb-Edu | 0.9766 | 0.9821 | 1.0138 | 15% |
SGCA: the update can never be larger than |g|. A runs from −π to π and g from −1.5 to 1.5. Drag to rotate.
This shows the function each model applies, for one channel with α = 1. It illustrates the definition and is not a measurement of the trained models. Whether the trained SGCA model works in the bent region is an open question.
SGCA and GQA have the same parameter count and the same counted FLOPs per token (3.025 billion), yet SGCA trains 6.8% slower than GQA and 9.3% slower than GQA-narrow. The extra time is in the elementwise gate work, which the FLOP formula does not count.
| Model | Tokens per second | FLOP utilization |
|---|---|---|
| GQA | 148,948 | 45.6% |
| GQA-narrow | 153,023 | 44.5% |
| SGCA | 138,834 | 42.5% |
GQA-narrow (the baseline with SGCA’s FFN width and no gate, 277.7M parameters) finished at 0.8110 BPB, only 0.0042 behind full GQA. About 90% of SGCA’s 0.0404 gap belongs to the gate.
Both 302.9M-parameter models trained on the same 1,000,013,824 tokens. SGCA led until about 145M tokens, then GQA passed it and stayed ahead on all six validation sources.
Three rates (2e-4, 4e-4, 8e-4) for each model. SGCA was ahead at every rate at this early budget (1.2697 against 1.3942 BPB at 4e-4). 4e-4 was chosen for the 1B runs.
alethic-tokenizer-v1 decodes every test document back to the exact input, keeps Python indentation at 4, 8 and 12 spaces, and uses 2.17 tokens per word on Uzbek Latin and 2.55 on Uzbek Cyrillic.
New repository with both model variants at exactly 302,908,416 parameters, a crash-safe data pipeline, resumable training and a test suite that runs on a laptop CPU.
The definition of SGCA, the earlier small-model experiments and the released models are in the research archive. Code, configs and the full research log are on GitHub. Weights, checkpoints, the tokenizer and tokenized data are public on Hugging Face.
Language-model architecture research, from controlled small-model experiments to a planned ~1B Phase I.
Sin-Gated Control Attention: separating attention retrieval from learned residual control.
Experimental pretraining and SinGatedAttention exploration on constrained compute.