Planning a 1B language-model experiment independently
Architecture targets, constrained compute, and infrastructure for an inspectable run.
Experimental pretraining and SinGatedAttention exploration on constrained compute.
Alethic-151M belongs to the earlier SinGatedAttention lineage, before current SGCA Phase I. It is a base/pretraining experiment, not a finished instruction-tuned assistant.
Setting | Recorded value |
|---|---|
Parameters / dimension | Approximately 151M / 768 |
Layers / heads | 12 / 12 |
Tokenizer | SentencePiece Unigram, approximately 20K vocabulary |
Context / batch | 256 tokens / 6 |
Corpus | FineWeb-Edu-based |
Framework / precision | PyTorch / bfloat16 |
Compute | Colab Tesla T4 |
Attention | PyTorch SDPA, supported GQA path |
One preparation run records 424,051,527 total corpus tokens, with 419,811,011 train and 4,240,516 validation tokens. Corpus size is not the count consumed by the optimizer.
The referenced run reached approximately 24,392 steps at about 1,536 tokens per step. Later notes record a best validation loss of approximately 5.0279. This is not a matched current-SGCA result.
Short context, constrained compute, and incomplete evaluation limit conclusions. Exact seeds and evaluation settings are needed for fair comparison. Generated text should be assessed as base-model continuation rather than chat instruction following.
Open Alethic-151M on Hugging Face.
The historical 116M predecessor has approximately 115,944,960 parameters, a decoder-only architecture, a SentencePiece Unigram tokenizer, and FineWeb-Edu-derived data. Its Ruzz repository name is preserved as historical naming.
Language-model architecture research, from controlled small-model experiments to a planned ~1B Phase I.
Sin-Gated Control Attention: separating attention retrieval from learned residual control.
Parameter-matched small language-model experiments with a sinusoidal attention gate.