The Stack v3: a packing snapshot
A supplied Python-data preparation record, separate from training progress.
A small window into the current direction of the work.
Alethic Phase I ~1B target, 8,240-token context, and a controlled matched baseline. SGCA Testing the separation between attention retrieval and residual control. Dataset and training infrastructure Export, source-ID deduplication, packed binaries, and resumable preparation. Efficient language-model architecture, experiment design, and the systems underneath training. A complete run record: actual configurations, tokenizer and data documentation, checkpoints, logs, and measured comparisons when available.
Follow the research