Milestone map
Milestone map
2 milestones
Do the phase's lessons in public
1–1 weeks (about 10 hrs/week, est)
Work through the 16 lessons of Phase 7: Transformers Deep Dive from AI Engineering from Scratch (free) and keep the transformer lessons' notebooks (self-attention, multi-head attention, positional encoding, the full transformer, KV cache) in one public repo as you go. About 9 hours (est).
Proof required
Link your public repo with the transformer lessons' notebooks (self-attention, multi-head attention, positional encoding, the full transformer, KV cache), and show your self-check (quizzes passed, notebooks run, tests passing where they exist).
What gets checked
- Each notebook runs.
- Attention shapes and masking have tests that pass.
- The full transformer trains and its loss goes down (curve shown).
Common mistakes
- Copying the repo's solutions: verifiers read your history and ask about your code.
- Skipping the self-check, then asking for review too early.
Powstik Guide
A public repo with the transformer lessons' notebooks (self-attention, multi-head attention, positional encoding, the full transformer, KV cache).
Steps
- Make a public repo for this phase.
- Do one lesson at a time and commit after each.
- Self-check: Run each lesson's code; it must work on your machine as the lesson shows. Run each lesson's notebook top to bottom; its outputs must match the lesson.
- Write a short README: what is here and how to run it.
- Ask three people who know the topic to check it.
Template
README Phase 7: Transformers Deep Dive (AI Engineering from Scratch) What is here: How to run it: Self-check result: What was hardest:
What gets sent back
- One big upload with no history.
- Self-check not shown.
- Code that does not run from the README.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Each notebook runs.
- Attention shapes and masking have tests that pass.
- The full transformer trains and its loss goes down (curve shown).
- Ask about one choice in the code; the answer should be clear.
You'll sign in first, then come straight back here.
Train a small transformer you wrote
2–4 weeks (6–10 hrs/week)
Use what you learned. Make a small transformer written from scratch and trained on a text you choose, with the loss curve, sample outputs and tests for attention shapes and causal masking. It must run, be evaluated and be documented.
Proof required
Link a small transformer written from scratch and trained on a text you choose, with the loss curve, sample outputs and tests for attention shapes and causal masking, with a README that says how to run it and what the evaluation found.
What gets checked
- The tests pass.
- The loss curve and samples are shown.
- It is compared with a simple baseline (for example a bigram model).
Common mistakes
- A demo with no evaluation.
- Numbers without the data or code that produced them.
Powstik Guide
A small transformer written from scratch and trained on a text you choose, with the loss curve, sample outputs and tests for attention shapes and causal masking.
Steps
- Write one line: what it does and for whom.
- Build the smallest version that works.
- Write the evaluation first, then improve against it.
- Write the README with the results.
- Ask three people to check it.
Template
README What it does: How to run it: How it was evaluated: Results: Limits:
What gets sent back
- Does not run from the README.
- No evaluation.
- Claims more than the evaluation shows.
Resources
Foundationstart here
What a verifier looks for
- The tests pass.
- The loss curve and samples are shown.
- It is compared with a simple baseline (for example a bigram model).
- Run it, or watch it run: it must work, be evaluated and be documented.
- Check that the work is the person's own.
You'll sign in first, then come straight back here.