Implement a Transformer Component From a Published Paper Specification
12 weeks · 0 milestones
Implement a core transformer component — attention mechanism, positional encoding, or byte-pair encoding tokenizer — directly from the specification in a published paper (Attention Is All You Need, or equivalent). The implementation must faithfully replicate the paper's equations in code with comments linking each line of code to the specific equation or paragraph in the paper it implements. Write-up must explain every design decision in terms of the specific constraint or property in the paper that motivated it. Proof: the implementation and write-up reviewed by an ML researcher or senior ML engineer who asks 'what would change in your attention output if you doubled the number of heads but kept the total dimension constant?' — you must answer by reasoning through your specific implementation.
Milestone map
Milestone map
3 milestones
Study the transformer architecture in depth: self-attention, multi-head attention, positional encoding, layer normalisation, and feed-forward sublayers. Implement a minimal transformer (encoder only or decoder only) in PyTorch without using HuggingFace Transformers or similar high-level libraries — the architecture must be implemented from attention matrices up. Train it on a small sequence modelling task.
Proof required
Submit: a public GitHub repository (or Colab) containing your transformer implementation with at least self-attention, multi-head attention, positional encoding, layer norm, and feed-forward sublayers coded from scratch; a training loss plot showing the model learns on a small task; and a written explanation (300 words) of why multi-head attention produces richer representations than single-head attention. An ML researcher or senior ML engineer must confirm the implementation is architecturally correct.
What gets checked
- Implementation builds the transformer from attention matrices — no HuggingFace Transformers or pre-built attention modules used
- Training loss plot shows learning on a small sequence task — not a flat loss curve
- An ML researcher or senior ML engineer has confirmed the architectural implementation is correct