Reasoning model from scratch
Inference-time scaling, GRPO and distillation, implemented bit by bit.
PythonPyTorchJupyter
Built a reasoning model from scratch in PyTorch.
Things I implemented:
- Text generation with a pretrained LLM: loading weights, sampling
- Inference-time scaling: spending more compute per answer at generation time
- Self-refinement: the model critiques and improves its own outputs
- Reinforcement learning with a bare RLVR/GRPO training loop
- GRPO improvements added one script at a time: tracking, clip ratios, KL regularization, format rewards
- Distillation of the reasoning behaviour into a smaller model
- Evaluation scripts for scoring generated answers