b.
Color theme
Let’s talk
Explore
Appearance
Color theme
research / experimental

Alethic-151M

Experimental pretraining and SinGatedAttention exploration on constrained compute.

alethictrainingmodels

Model identity

Alethic-151M belongs to the earlier SinGatedAttention lineage, before current SGCA Phase I. It is a base/pretraining experiment, not a finished instruction-tuned assistant.

Recorded configuration

Setting

Recorded value

Parameters / dimension

Approximately 151M / 768

Layers / heads

12 / 12

Tokenizer

SentencePiece Unigram, approximately 20K vocabulary

Context / batch

256 tokens / 6

Corpus

FineWeb-Edu-based

Framework / precision

PyTorch / bfloat16

Compute

Colab Tesla T4

Attention

PyTorch SDPA, supported GQA path

Data and training record

One preparation run records 424,051,527 total corpus tokens, with 419,811,011 train and 4,240,516 validation tokens. Corpus size is not the count consumed by the optimizer.

The referenced run reached approximately 24,392 steps at about 1,536 tokens per step. Later notes record a best validation loss of approximately 5.0279. This is not a matched current-SGCA result.

Limitations

Short context, constrained compute, and incomplete evaluation limit conclusions. Exact seeds and evaluation settings are needed for fair comparison. Generated text should be assessed as base-model continuation rather than chat instruction following.

Release and predecessor

Open Alethic-151M on Hugging Face.

The historical 116M predecessor has approximately 115,944,960 parameters, a decoder-only architecture, a SentencePiece Unigram tokenizer, and FineWeb-Edu-derived data. Its Ruzz repository name is preserved as historical naming.

Related threads

Keep exploring.

Back to research