Phase I uses 8,240 tokens
The current context decision, separate from earlier long-context curricula.
Language-model architecture research, from controlled small-model experiments to a planned ~1B Phase I.
Lower layer: a retrieved signal A over tokens and channels. Upper layer: the same points after the sine term, sin(A). Scroll and the size of A grows, so the sine starts near-linear and then folds, changing sign and sensitivity as its input moves. The bright band marks one token position sweeping across; move the pointer to probe nearby points.
This depicts the mechanism only. It is not data, a benchmark, or a measured result.
Attention looks back over earlier tokens and mixes their values into one number per feature, called A here. How much each token counts is its weight, and the weights always add up to 1.
Worked example: scores [2, 1, 0.1] become weights by softmax, w = e^s / Σe^s. Switching focus multiplies every score before softmax.
A = 0.66×1 + 0.24×-0.5 + 0.10×0.2 = 0.56One SGCA block, followed for a single feature with toy numbers: X = 2, A = 0.8, α = 0.5, WH+b = 1.2. Real values are vectors; this is the same arithmetic on one slot.
It plays by itself. Pick a step to stop it.
H = RMSNorm(X)SGCA passes the attention output A through sin(A), then scales it by a learned amplitude α. For small A, sin(A) ≈ A, so it behaves like a plain scaled copy. As A grows, the sine folds back.
Worked numbers, with α = 1: A = 0.5 gives sin(0.5) ≈ 0.48, A = 1.5 gives 1.00, and A = 3.0 gives 0.14.
Many squashing functions saturate: past a certain size, a bigger input barely changes the output. A sine does not saturate. It keeps responding, and it changes sign.
Whether that extra sensitivity helps a language model is a question for experiments, not for this picture.
To say the gate caused a difference, everything else must match. Each factor you leave unlocked is one more thing that could explain a gain.
The control branch adds parameters. To keep the count equal, the baseline gets the same number of extra parameters somewhere else, such as a wider feed-forward layer.
3 things differ: the gate + 2 unlocked. A gain could come from any of them.Can attention retrieval and residual control be separated so that a learned sinusoidal control path determines how retrieved information influences the residual stream?
Alethic is my language-model research line. The current direction is Sin-Gated Control Attention (SGCA). The question concerns optimization, representation, and efficiency under a carefully matched decoder-only baseline.
Current state
Phase I is active research and preparation. Approximately 1B parameters and 20B training tokens are targets, not a completed model or a processed token count. The initial context target is 8,240 tokens.
The decoder block uses RMSNorm, GQA, SGCA, and SwiGLU. RoPE remains in the attention path.
is the output of GQA. The learned tensor must be broadcast-compatible with ; no scalar or fixed matrix shape is assumed. The block then applies RMSNorm, SwiGLU, and another residual update.
Both paths share GQA. The baseline adds the attention output directly; SGCA computes a separate control update.
Conceptual block comparison
Copy
# Matched GQA baseline
H=rms_norm(X)
A=gqa(H)
X_prime=X+A
output=X_prime+swiglu(rms_norm(X_prime))
# SGCA research block
H=rms_norm(X)
A=gqa(H)
C=linear(H)*(alpha*sin(A))
X_prime=X+C
output=X_prime+swiglu(rms_norm(X_prime))This is conceptual pseudocode, not a training implementation. The actual tensor shape and initialization of α belong in each run configuration.
Match the tokenizer, corpus, data order, optimizer, learning-rate schedule, token budget, context, precision, batch construction, initialization, hardware class, and evaluation where possible. Parameter matching and compute matching answer different questions and must be reported explicitly. Multiple seeds and non-sinusoidal gates are part of the validation plan.
SinGatedLMPreliminary ~64K and ~1M controlled experiments.
116M pretrainingHistorical predecessor released under the Ruzz name.
Alethic-151MExperimental pretraining and SinGatedAttention exploration.
SGCACurrent separate sinusoidal residual-control formulation.
Phase IPlanned ~1B scale, 8,240-token context, controlled baseline.
This is a research progression, not one benchmark series. The models did not all share the same architecture or setup.
The earlier parameter-matched results are documented on the SinGatedLM page. A later Alethic-151M record reports a best validation loss of approximately 5.0279. These separate experiments are not an apples-to-apples comparison with current SGCA. No Phase-I SGCA result is reported here.
Configuration | Phase-I target |
|---|---|
Parameters | Approximately 1B, planned |
Context | 8,240 tokens |
Tokenizer | Approximately 48K vocabulary, planned |
Training tokens | Approximately 20B for a full run, planned |
Attention / position | GQA / RoPE in attention path |
Control | SGCA with learned broadcast-compatible α |
Normalization / FFN | RMSNorm / SwiGLU |
Earlier 1.5B and 12K/54K plans are historical. Longer-context scaling is postponed until the smaller-scale mechanism has been evaluated.
Work has used free Colab T4 sessions and Kaggle environments, including 2×T4. Tokenizer preparation, export, source-ID deduplication, and binary packing are substantial parts of the infrastructure. The challenge is making runs resumable and comparable under constrained compute.
Small-model evidence does not establish SGCA superiority. Earlier experiments have limited seeds, dataset breadth, and model scale. Training stability, initialization sensitivity, and throughput remain questions to measure.
The earlier code and model releases are linked below. A public Phase-I repository, complete run configuration, seeds, tokenizer release, and final logs have not been supplied for this page.
1 October 2026: Overview documented with the current Phase-I target and separate historical model lineage.
Sin-Gated Control Attention: separating attention retrieval from learned residual control.
Parameter-matched small language-model experiments with a sinusoidal attention gate.
Experimental pretraining and SinGatedAttention exploration on constrained compute.