b.
Color theme
Let’s talk
Explore
Appearance
Color theme
research / completed

SinGatedLM

Parameter-matched small language-model experiments with a sinusoidal attention gate.

Overview

SinGatedLM is an earlier architecture experiment using attention-derived output as a sinusoidal multiplicative gate. It motivated further investigation in Alethic.

Mechanism

f(x,y)=(Wx+b)⊙(αsin⁡(y))\begin{gathered}f(x,y)=(Wx+b)\odot(\alpha\sin(y))\end{gathered}

Here yy is derived from attention. This predecessor expression is not the current SGCA control formulation.

Experimental setups

Two reported parameter-matched configurations used Tiny Shakespeare. Their settings are kept separate. Context and seed identifiers were not supplied for these reports.

Setup / model

Parameters

Dataset

Steps

Context

Seed

Val. loss

~64K / baseline

~64K

Tiny Shakespeare

Unreported

Unreported

Unreported

2.6931

~64K / SinGated

~64K

Tiny Shakespeare

Unreported

Unreported

Unreported

2.5603

~1M / baseline

1,027,047

Tiny Shakespeare

8,000

Unreported

Unreported

1.7592

~1M / SinGated

1,027,048

Tiny Shakespeare

8,000

Unreported

Unreported

1.5902

Final reported metrics; different setups remain separate. Missing settings are marked unreported. No statistical significance is implied.

Parameter matching

The ~1M comparison has 1,027,048 SinGated parameters versus 1,027,047 baseline parameters: one parameter apart. This controls one confounder; it does not prove that compute, optimization, and every initialization detail matched.

Reading the results

For the ~1M run, validation loss was 1.5902 versus 1.7592 after 8,000 steps, an observed difference of 0.1690. Reported perplexities were approximately 4.90 and 5.81.

The separate ~64K report recorded 2.5603 versus 2.6931, a difference of 0.1328. Other historical run values must not be silently merged into these configurations.

Training curves

A complete per-step series has not been supplied. No interpolated or reconstructed training curve is shown.

Limitations

Tiny Shakespeare, small and shallow models, limited early seeds, and limited architecture breadth. These results motivated further investigation; they do not establish general architectural superiority.

What came next

Priorities include multiple seeds, broader datasets, deeper networks, sin/tanh/sigmoid/identity ablations, matched parameters, matched compute, and downstream evaluation.

Repository and reproduction

Inspect the repository. Use its actual configurations and instructions. Resolve missing seed and context information before claiming an independently reproduced run.

Related threads

Keep exploring.

Back to research