Phase I uses 8,240 tokens
The current context decision, separate from earlier long-context curricula.
A careful reading of parameter-matched Tiny Shakespeare results and their limits.
A benchmark becomes misleading when its configuration disappears. The SinGatedLM record contains multiple experiments; they should not collapse into one best number.
One ~1M run reports 1,027,048 SinGated parameters against 1,027,047 baseline parameters. After 8,000 steps on Tiny Shakespeare, validation losses were 1.5902 and 1.7592, with reported perplexities approximately 4.90 and 5.81.
A separate ~64K report gives losses of 2.5603 and 2.6931. Its training-step count, context, and seed are not supplied in the summary. Missing settings should remain visible rather than borrowed from another run.
The observation is lower reported validation loss for the gated model in these configurations. Parameter matching controls an obvious difference in the larger run.
The interpretation is narrower than general superiority. Tiny Shakespeare is small; early models were small and shallow, seeds and architecture breadth were limited, and current SGCA differs from the predecessor gate.
Useful conclusion
The results motivate further controlled investigation. They do not establish general architectural superiority.
Repeat across seeds, broaden data and depth, and compare sin with identity, tanh, and sigmoid. Report parameters and compute. Hold tokenizer, data order, optimizer, token budget, context, and evaluation fixed where possible.
The goal is to understand which change caused which effect, whether it repeats, and whether its cost is acceptable—not merely to collect favorable final losses.
Reproduction needs a source revision, split, tokenizer, seed, precision, context, batch construction, optimizer, schedule, and evaluation settings. The final number should lead back to that record.
The SinGatedLM page keeps the supplied configurations separate. It omits training curves because no complete per-step series was supplied. A plausible invented curve would weaken the evidence.
The SGCA research question, residual update, and requirements for a fair comparison.
Architecture targets, constrained compute, and infrastructure for an inspectable run.
What source-ID deduplication, supervised tokens, and packed windows say about preparation.