Data Portrait: CrystalCoder training data
A Data Portrait of the pre-training data of IFM/Crystal (CrystalCoder, LLM360), decoded from the exact training sequences released in IFM/CrystalCoderDatasets (token ids, decoded with the model's tokenizer; fill-in-the-middle documents are put back in their original order):
- phase1: first half of SlimPajama
- phase2: second half of SlimPajama + StarCoder data (FIM undone)
- phase3: StarCoder Python/HTML/CSS/JavaScript (FIM undone)
Not covered: the ~10B tokens of SlimPajama re-used in phase 3 are not a separate phase-3 filter (which rows were drawn is not recorded, and all of that text is already in the phase-1/2 filters), and the stage-3a instruction-tuning data, which was never released.
Text is normalised as in data-portraits and split into 50-character n-grams with stride 50. Each stage is split into Bloom filters that fit in RAM (4 in total). A query ORs membership across filters: combined false-positive rate <= 0.01 over all of them (each filter 0.002509). Since every filter belongs to one stage, a match can be traced to a stage. Documents were packed into 2048-token rows, so n-grams that span a row boundary are missing.
Each partition is a separate revision of this repo (tag <stage>-p<i>of<k>, branch partition-<n>):
| stage | tag | files | size |
|---|---|---|---|
| phase1 | phase1-p0of1 |
1730 | 43 GiB |
| phase2 | phase2-p0of2 |
2401 | 45 GiB |
| phase2 | phase2-p1of2 |
2401 | 45 GiB |
| phase3 | phase3-p0of1 |
581 | 7 GiB |
partitions.json lists the exact files (pinned dataset revision) in each partition. Query with
crystal_portrait.query:
from crystal_portrait.query import SequentialPortrait
portrait = SequentialPortrait("aalkaswan/crystalcoder-portrait", ["phase1-p0of1", "phase2-p0of2", "phase2-p1of2", "phase3-p0of1"])
report = portrait.contains_from_text([text], sort_chains_by_length=True)[0]