The Stack v3: a packing snapshot
A supplied Python-data preparation record, separate from training progress.
What source-ID deduplication, supervised tokens, and packed windows say about preparation.
A preparation log is not a training benchmark. One supplied Python portion of The Stack v3 records 125,215 JSONL records inspected, 20,638 / 32,767 train windows packed, 170,057,120 supervised tokens, 512 validation windows, and 62,375 identical source-ID duplicates skipped.
These describe a preparation stage, not final corpus size, optimizer-consumed tokens, or model quality.
A supervised token contributes a prediction target under the packer's accounting. Window size, separators, masks, padding, and prediction shift affect how the count relates to stored token arrays.
Multiplying a counter by a byte width gives file size only when the dtype and format are known. File bytes are measured from files; prediction targets are counted by the preparation or training procedure.
Skipping an identical source ID avoids processing the same identified upstream record again. It does not prove content or near-duplicate deduplication, nor absence of split contamination.
Identity keys, persistent state, and stable split assignment matter. A restarted export should not silently change what gets counted or where a source lands.
Resumption under session limits benefits from upstream position, completed shards, counters, and reproducible packing rules. Partially written output should be distinguishable from a finished artifact.
The preparation record belongs beside training. A final validation loss cannot explain a data bug that was never logged.
The baseline and SGCA run should share tokenizer and data construction where possible. Order, context, and token budget need the same care as the equation.
The lab note preserves the supplied snapshot as a preparation observation, not a live or final report.
The SGCA research question, residual update, and requirements for a fair comparison.
A careful reading of parameter-matched Tiny Shakespeare results and their limits.
Architecture targets, constrained compute, and infrastructure for an inspectable run.