b.
Color theme
Let’s talk
Explore
Appearance
Color theme
research / active

Alethic

Language-model architecture research, from controlled small-model experiments to a planned ~1B Phase I.

Alethic news / 10 Nov 2026The gate, not the narrower FFN, explains the gapGQA-narrow (the baseline with SGCA’s FFN width and no gate, 277.7M parameters) finished at 0.8110 BPB, only 0.0042 behind full GQA. About 90% of SGCA’s 0.0404 gap belongs to the gate.
Read the results
Illustration · the sine term

Lower layer: a retrieved signal A over tokens and channels. Upper layer: the same points after the sine term, sin(A). Scroll and the size of A grows, so the sine starts near-linear and then folds, changing sign and sensitivity as its input moves. The bright band marks one token position sweeping across; move the pointer to probe nearby points.

This depicts the mechanism only. It is not data, a benchmark, or a measured result.

Step 1: retrieve a signal

Attention looks back over earlier tokens and mixes their values into one number per feature, called A here. How much each token counts is its weight, and the weights always add up to 1.

Worked example: scores [2, 1, 0.1] become weights by softmax, w = e^s / Σe^s. Switching focus multiplies every score before softmax.

A = 0.66×1 + 0.24×-0.5 + 0.10×0.2 = 0.56
Broad: every token contributes, so A is a blend.

Step 2: build the update

One SGCA block, followed for a single feature with toy numbers: X = 2, A = 0.8, α = 0.5, WH+b = 1.2. Real values are vectors; this is the same arithmetic on one slot.

It plays by itself. Pick a step to stop it.

H = RMSNorm(X)
The block input X is rescaled to H so its size is steady.

Step 3: the sine term

SGCA passes the attention output A through sin(A), then scales it by a learned amplitude α. For small A, sin(A) ≈ A, so it behaves like a plain scaled copy. As A grows, the sine folds back.

Worked numbers, with α = 1: A = 0.5 gives sin(0.5) ≈ 0.48, A = 1.5 gives 1.00, and A = 3.0 gives 0.14.

Move over the graph to probe a token position. This is an illustration of the mechanism, not a measured result.

Step 4: why try a sine?

Many squashing functions saturate: past a certain size, a bigger input barely changes the output. A sine does not saturate. It keeps responding, and it changes sign.

Whether that extra sensitivity helps a language model is a question for experiments, not for this picture.

From A = 2 to A = 3: tanh moves 0.96 → 0.995 (change 0.03). α·sin(A) moves 0.91 → 0.14 (change 0.77). Hover the graph to probe any A.

Step 5: a fair comparison

To say the gate caused a difference, everything else must match. Each factor you leave unlocked is one more thing that could explain a gain.

The control branch adds parameters. To keep the count equal, the baseline gets the same number of extra parameters somewhere else, such as a wider feed-forward layer.

3 things differ: the gate + 2 unlocked. A gain could come from any of them.

Research question

Can attention retrieval and residual control be separated so that a learned sinusoidal control path determines how retrieved information influences the residual stream?

Alethic is my language-model research line. The current direction is Sin-Gated Control Attention (SGCA). The question concerns optimization, representation, and efficiency under a carefully matched decoder-only baseline.

Current state

Phase I is active research and preparation. Approximately 1B parameters and 20B training tokens are targets, not a completed model or a processed token count. The initial context target is 8,240 tokens.

Architecture

The decoder block uses RMSNorm, GQA, SGCA, and SwiGLU. RoPE remains in the attention path.

H=RMSNorm(X),A=GQA(H)C=(WH+b)⊙[α⊙sin⁡(A)],X′=X+C\begin{gathered}H=\mathrm{RMSNorm}(X),\qquad A=\mathrm{GQA}(H)\\C=(WH+b)\odot[\alpha\odot\sin(A)],\qquad X'=X+C\end{gathered}

AA is the output of GQA. The learned tensor α\alpha must be broadcast-compatible with AA; no scalar or fixed matrix shape is assumed. The block then applies RMSNorm, SwiGLU, and another residual update.

Matched baseline

Both paths share GQA. The baseline adds the attention output directly; SGCA computes a separate control update.

Conceptual block comparison

Copy

text
# Matched GQA baseline
H=rms_norm(X)
A=gqa(H)
X_prime=X+A
output=X_prime+swiglu(rms_norm(X_prime))

# SGCA research block
H=rms_norm(X)
A=gqa(H)
C=linear(H)*(alpha*sin(A))
X_prime=X+C
output=X_prime+swiglu(rms_norm(X_prime))

This is conceptual pseudocode, not a training implementation. The actual tensor shape and initialization of α belong in each run configuration.

Experiment methodology

Match the tokenizer, corpus, data order, optimizer, learning-rate schedule, token budget, context, precision, batch construction, initialization, hardware class, and evaluation where possible. Parameter matching and compute matching answer different questions and must be reported explicitly. Multiple seeds and non-sinusoidal gates are part of the validation plan.

Model lineage

  1. SinGatedLMPreliminary ~64K and ~1M controlled experiments.

  2. 116M pretrainingHistorical predecessor released under the Ruzz name.

  3. Alethic-151MExperimental pretraining and SinGatedAttention exploration.

  4. SGCACurrent separate sinusoidal residual-control formulation.

  5. Phase IPlanned ~1B scale, 8,240-token context, controlled baseline.

This is a research progression, not one benchmark series. The models did not all share the same architecture or setup.

Recorded evidence

The earlier parameter-matched results are documented on the SinGatedLM page. A later Alethic-151M record reports a best validation loss of approximately 5.0279. These separate experiments are not an apples-to-apples comparison with current SGCA. No Phase-I SGCA result is reported here.

Current experiment

Configuration

Phase-I target

Parameters

Approximately 1B, planned

Context

8,240 tokens

Tokenizer

Approximately 48K vocabulary, planned

Training tokens

Approximately 20B for a full run, planned

Attention / position

GQA / RoPE in attention path

Control

SGCA with learned broadcast-compatible α

Normalization / FFN

RMSNorm / SwiGLU

Earlier 1.5B and 12K/54K plans are historical. Longer-context scaling is postponed until the smaller-scale mechanism has been evaluated.

Compute and pipeline

Work has used free Colab T4 sessions and Kaggle environments, including 2×T4. Tokenizer preparation, export, source-ID deduplication, and binary packing are substantial parts of the infrastructure. The challenge is making runs resumable and comparable under constrained compute.

Limitations

Small-model evidence does not establish SGCA superiority. Earlier experiments have limited seeds, dataset breadth, and model scale. Training stability, initialization sensitivity, and throughput remain questions to measure.

Reproducibility

The earlier code and model releases are linked below. A public Phase-I repository, complete run configuration, seeds, tokenizer release, and final logs have not been supplied for this page.

Updates

1 October 2026: Overview documented with the current Phase-I target and separate historical model lineage.

From the workbench

Related threads

Keep exploring.

Back to research