Milestone map
Milestone map
2 milestones
Do the phase's lessons in public
1–1 weeks (about 10 hrs/week, est)
Work through the 24 lessons of Phase 10: LLMs from Scratch from AI Engineering from Scratch (free) and keep the LLMs-from-scratch lessons' code and notebooks (tokenizer, data pipeline, pre-training a mini GPT, instruction tuning, DPO, evaluation, quantisation) in one public repo as you go. About 14 hours (est).
Proof required
Link your public repo with the LLMs-from-scratch lessons' code and notebooks (tokenizer, data pipeline, pre-training a mini GPT, instruction tuning, DPO, evaluation, quantisation), and show your self-check (quizzes passed, notebooks run, tests passing where they exist).
What gets checked
- Each lesson's code runs and its quiz is passed where there is one.
- The mini GPT trains and its loss goes down.
- Evaluation numbers are reported before and after tuning.
Common mistakes
- Copying the repo's solutions: verifiers read your history and ask about your code.
- Skipping the self-check, then asking for review too early.
Powstik Guide
A public repo with the LLMs-from-scratch lessons' code and notebooks (tokenizer, data pipeline, pre-training a mini GPT, instruction tuning, DPO, evaluation, quantisation).
Steps
- Make a public repo for this phase.
- Do one lesson at a time and commit after each.
- Self-check: Run each lesson's code; it must work on your machine as the lesson shows. Run each lesson's notebook top to bottom; its outputs must match the lesson. Each lesson has a quiz (quiz.json, also on the course site): pass it.
- Write a short README: what is here and how to run it.
- Ask three people who know the topic to check it.
Template
README Phase 10: LLMs from Scratch (AI Engineering from Scratch) What is here: How to run it: Self-check result: What was hardest:
What gets sent back
- One big upload with no history.
- Self-check not shown.
- Code that does not run from the README.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Each lesson's code runs and its quiz is passed where there is one.
- The mini GPT trains and its loss goes down.
- Evaluation numbers are reported before and after tuning.
- Ask about one choice in the code; the answer should be clear.
You'll sign in first, then come straight back here.
Train a tiny GPT end to end
2–4 weeks (6–10 hrs/week)
Use what you learned. Make a tiny GPT trained end to end: your tokenizer, pre-training on a corpus you choose, a small instruction-tuning step, and an evaluation before and after tuning. It must run, be evaluated and be documented.
Proof required
Link a tiny GPT trained end to end: your tokenizer, pre-training on a corpus you choose, a small instruction-tuning step, and an evaluation before and after tuning, with a README that says how to run it and what the evaluation found.
What gets checked
- It trains from the README.
- Evaluation before and after tuning is shown.
- Sample outputs are included, with their limits stated.
Common mistakes
- A demo with no evaluation.
- Numbers without the data or code that produced them.
Powstik Guide
A tiny GPT trained end to end: your tokenizer, pre-training on a corpus you choose, a small instruction-tuning step, and an evaluation before and after tuning.
Steps
- Write one line: what it does and for whom.
- Build the smallest version that works.
- Write the evaluation first, then improve against it.
- Write the README with the results.
- Ask three people to check it.
Template
README What it does: How to run it: How it was evaluated: Results: Limits:
What gets sent back
- Does not run from the README.
- No evaluation.
- Claims more than the evaluation shows.
Resources
Foundationstart here
What a verifier looks for
- It trains from the README.
- Evaluation before and after tuning is shown.
- Sample outputs are included, with their limits stated.
- Run it, or watch it run: it must work, be evaluated and be documented.
- Check that the work is the person's own.
You'll sign in first, then come straight back here.