powstik
Prove
LiveProof-of-Work · Oct–Nov: every verified proof counts. 28 days leftAll outcomes
Skills

Implement PPO or GRPO on a toy task

6 weeks · 2 milestones

Implement a policy-optimisation method yourself and test it properly. Research step 4 of the Varsity Curriculum: AI Research. Free resources; your code in a public repo; one person who knows the topic confirms it.

Part of: AI Research · Proves for: Engineers, PG

Milestone map

Milestone map

2 milestones

Implement PPO or GRPO yourself

2–4 weeks (about 8 hrs/week, est)

Build your own implementation of PPO (on a control task such as CartPole) or GRPO (on a small language model with a checkable reward such as arithmetic), trained over at least 5 seeds. Keep it in one public repo and commit as you go.

Proof required

Link the public repo with your implementation, the reward or environment, learning curves over at least 5 seeds, and a comparison with a reference implementation or a simple baseline.

What gets checked

  • The algorithm is your own code, not a library call.
  • Learning curves cover at least 5 seeds with the spread shown.
  • It beats a stated baseline, or the report explains why not.

Common mistakes

  • One big upload with no commit history.
  • Results that the repo cannot reproduce.

Powstik Guide

A public repo with your own implementation of PPO (on a control task such as CartPole) or GRPO (on a small language model with a checkable reward such as arithmetic), trained over at least 5 seeds.

Steps

  1. Make a public repo.
  2. Write the tests or the evaluation first.
  3. Build the smallest version that passes them.
  4. Write a README: what it is and how to reproduce the results.
  5. Ask someone who knows the topic to check it.

Template

README
Implement PPO or GRPO on a toy task
What is here:
How to reproduce:
Results:
Limits:

What gets sent back

  • No tests or evaluation.
  • Code that does not work from the README.
  • Copied code with no history.

Resources

Foundationstart here

What a verifier looks for

  • The algorithm is your own code, not a library call.
  • Learning curves cover at least 5 seeds with the spread shown.
  • It beats a stated baseline, or the report explains why not.
  • Open the repo: does the code match the results shown?

You'll sign in first, then come straight back here.

Test one design choice

1–2 weeks (about 6 hrs/week, est)

Write up one design choice tested (for example the clipping range, the group size or the KL penalty) with the result and what it means, in a short report. Put the report in the repo next to the code that produced it.

Proof required

Link the report in your public repo, with the plots and the code or commands that made them.

What gets checked

  • One choice is varied and everything else is held fixed.
  • Results cover several seeds.
  • The report says what it would test next.

Common mistakes

  • Claims the plots do not support.
  • A single seed reported as a result.

Powstik Guide

A short report in the repo with plots and the commands that made them.

Steps

  1. State the question in one line.
  2. Say what you held fixed and what you changed.
  3. Show the plot with the spread over seeds.
  4. Write what it shows, and what it does not.

Template

Report
Question:
Setup (fixed / changed):
Results (plot, seeds):
What it shows:
What it does not show:
Next:

What gets sent back

  • No plot.
  • No statement of what was held fixed.

Resources

Depthgo deeper

What a verifier looks for

  • One choice is varied and everything else is held fixed.
  • Results cover several seeds.
  • The report says what it would test next.
  • Is the conclusion honest about what was not tested?

You'll sign in first, then come straight back here.

Part of