The Synthetic Data Playbook
Generating Trillions of the Finest Tokens
LLMC @ NII
Why This Matters
The Data Black Box
Frontier labs (OpenAI, Google, Anthropic) don't disclose how they build their training data.
Neither do the Chinese labs (DeepSeek or Qwen).
Training data is the most important ingredient in building an LLM, yet the recipes are kept secret.
Digital Sovereignty
If you can't build the data, you can't build the model.
If you can't build the model, you depend on those who can.
This work puts the knowledge out in the open for everyone: governments, universities, startups, and individuals.
The Problem
You Start With the Entire Internet...
...and throw away 98.6% of it
DCLM: 240T tokens from Common Crawl → 1.4% survives as DCLM-Baseline
The Idea
Rewrite Instead of Filter
★★★ BeSt DeAls!!!
Photosynthesis is the process by wich plants convert sunlit into energy.
It occurs in the chloroplasts
Click here for more → → →
© 2019 AllScienceInfo.biz
Carbon dioxide and water are transformed into glucose and oxygen...
[AD] [AD] [POPUP]
Q: What is photosynthesis?
A: Photosynthesis is the process by which plants convert sunlight into chemical energy.
It occurs in organelles called chloroplasts.
Q: What are the inputs and outputs?
A: Plants take in carbon dioxide (CO₂) and water (H₂O), and using light energy,
produce glucose (C₆H₁₂O₆) and oxygen (O₂).
Same knowledge, better packaging.
You keep 100% of your data instead of discarding 90%.
Our Research
What's the Best Recipe?
Three knobs to tune: source data, prompt strategy, and generator model.
Methodology
Our Integration Test Suite
For each experiment, we:
- Train a 1.2B parameter model from scratch
- Feed it 20B tokens of synthetic and original data
- Test on 12 benchmarks (reading, math, reasoning, knowledge...)
- Compare against curated web datasets as baselines
This is expensive so we tried proxies:
- DCLM/Edu scores (used for filtering pretraining data)
- Smaller training runs
None correlated well enough.
No shortcuts: you must train and evaluate to know if your data is good.
Spoiler
FinePhrase Wins
Our best synthetic recipe outperforms all tested baselines, including curated web data.
Let's unpack how.
Finding #1
Prompt Design Is the #1 Lever
Structured prompts beat everything:
- Math reformatting
- Table extraction
- FAQ generation
- Tutorial rewriting
These beat curated web data and all prior synthetic baselines.
The prompt matters more than the model or the source data.
Finding #2