GPT-2 from the ground up

Built GPT-2 from the ground up in PyTorch: tokenizer, attention, the full architecture, pretraining and two fine-tunings.

Things I implemented:

• BPE tokenizer, special tokens, sliding-window dataset with configurable stride
• Token and positional embeddings
• Self-attention from scratch: Q/K/V projections, scaled dot-product, causal masking, dropout
• Multi-head attention with the batched weight split
• Layer norm, GELU and the feed-forward block
• The full GPT model: embeddings, N transformer blocks, final norm, output head
• Text generation: greedy, temperature scaling, top-k sampling
• Loading OpenAI's GPT-2 weights: TensorFlow checkpoint parsing, fused QKV split
• Classification fine-tuning: a spam classifier head, all but the last block frozen
• Instruction fine-tuning: Alpaca prompts, custom collate with padding, target shift, -100 masking