What the small SinGated experiments actually show
A careful reading of parameter-matched Tiny Shakespeare results and their limits.
Parameter-matched small language-model experiments with a sinusoidal attention gate.
SinGatedLM is an earlier architecture experiment using attention-derived output as a sinusoidal multiplicative gate. It motivated further investigation in Alethic.
Here is derived from attention. This predecessor expression is not the current SGCA control formulation.
Two reported parameter-matched configurations used Tiny Shakespeare. Their settings are kept separate. Context and seed identifiers were not supplied for these reports.
Setup / model | Parameters | Dataset | Steps | Context | Seed | Val. loss |
|---|---|---|---|---|---|---|
~64K / baseline | ~64K | Tiny Shakespeare | Unreported | Unreported | Unreported | 2.6931 |
~64K / SinGated | ~64K | Tiny Shakespeare | Unreported | Unreported | Unreported | 2.5603 |
~1M / baseline | 1,027,047 | Tiny Shakespeare | 8,000 | Unreported | Unreported | 1.7592 |
~1M / SinGated | 1,027,048 | Tiny Shakespeare | 8,000 | Unreported | Unreported | 1.5902 |
Final reported metrics; different setups remain separate. Missing settings are marked unreported. No statistical significance is implied.
The ~1M comparison has 1,027,048 SinGated parameters versus 1,027,047 baseline parameters: one parameter apart. This controls one confounder; it does not prove that compute, optimization, and every initialization detail matched.
For the ~1M run, validation loss was 1.5902 versus 1.7592 after 8,000 steps, an observed difference of 0.1690. Reported perplexities were approximately 4.90 and 5.81.
The separate ~64K report recorded 2.5603 versus 2.6931, a difference of 0.1328. Other historical run values must not be silently merged into these configurations.
A complete per-step series has not been supplied. No interpolated or reconstructed training curve is shown.
Limitations
Tiny Shakespeare, small and shallow models, limited early seeds, and limited architecture breadth. These results motivated further investigation; they do not establish general architectural superiority.
Priorities include multiple seeds, broader datasets, deeper networks, sin/tanh/sigmoid/identity ablations, matched parameters, matched compute, and downstream evaluation.
Inspect the repository. Use its actual configurations and instructions. Resolve missing seed and context information before claiming an independently reproduced run.
Language-model architecture research, from controlled small-model experiments to a planned ~1B Phase I.
Sin-Gated Control Attention: separating attention retrieval from learned residual control.
Experimental pretraining and SinGatedAttention exploration on constrained compute.