Implement a Transformer Component From a Published Paper Specification

12 weeks · 0 milestones

Implement a core transformer component — attention mechanism, positional encoding, or byte-pair encoding tokenizer — directly from the specification in a published paper (Attention Is All You Need, or equivalent). The implementation must faithfully replicate the paper's equations in code with comments linking each line of code to the specific equation or paragraph in the paper it implements. Write-up must explain every design decision in terms of the specific constraint or property in the paper that motivated it. Proof: the implementation and write-up reviewed by an ML researcher or senior ML engineer who asks 'what would change in your attention output if you doubled the number of heads but kept the total dimension constant?' — you must answer by reasoning through your specific implementation.

Sign in to start this outcome and track your progress publicly.