Build an AI Agent from Scratch

Proof required

Submit a GitHub repository (public or shared) containing: the full agent source code, a README explaining the task, the tool(s) used, and how to run it, and a 3-minute screen recording demonstrating the agent completing 3 different inputs end-to-end. At least one input must be one you have not tested before recording.

What gets checked

• Agent runs end-to-end without manual intervention on all 3 demo inputs — stubs, hardcoded responses, or pre-cached outputs do not count
• Tool invocation is real: the recording shows an actual API call or file I/O occurring, not a mocked response
• README explains what the agent does, what tool it calls, and how to reproduce the demo — a missing README or a README that only describes installation is insufficient

Common mistakes

• Building a chatbot wrapper without any real tool use — the defining feature of an agent is that it takes actions in the world, not that it generates text
• Using a demo task that always succeeds on a single templated input rather than generalisable inputs — the agent must handle variation
• Skipping error handling so the agent crashes on any unexpected input — robust agents fail gracefully and report the failure

Resources

What a verifier looks for

• Ask the submitter to explain why they chose that tool over alternatives — a good answer names specific trade-offs, a weak answer says 'it was in the tutorial'
• Pose a novel input live and watch the agent run — does it handle it correctly or does it fail in a way the submitter cannot explain?
• Ask what happens when the tool returns an error — can they describe the error-handling path from memory, or do they need to look at the code?
• Check the README: can someone else reproduce the demo following it without asking the submitter for clarification?

You'll sign in first, then come straight back here.

Proof required

Submit the updated repository with: the multi-step agent code, an eval script, a results file showing the agent's pass rate on 10 test inputs with the pass/fail criterion stated, and a 200-word write-up explaining what broke when you added memory and how you fixed it.

What gets checked

• Eval harness runs automatically and produces a pass/fail result for each of 10 inputs — manual inspection of outputs does not count as an eval
• Multi-step execution is genuine: the recorded trace must show at least 3 tool calls where input to step N+1 depends on the output of step N
• Write-up names a specific failure that occurred when adding memory (e.g. context window overflow, stale state, prompt injection from memory) — 'everything worked fine' is not an acceptable answer

Common mistakes

• Building an eval that always passes because the pass criterion is trivially easy — the eval should surface real failures on at least 1–2 of the 10 test inputs
• Treating 'memory' as just including all previous messages in the prompt without addressing what happens when the context gets long
• Skipping the eval entirely and submitting screenshots of manual tests — without automation, reproducibility and regression testing are impossible

Resources

What a verifier looks for

• Ask the submitter to walk through a specific eval failure — can they trace which step produced the bad output and why?
• Ask what the pass rate was before and after adding memory — improvement should be demonstrable and explained
• Ask what the hardest test case was and why — a vague answer suggests the eval was not taken seriously
• Check that the eval script is runnable: attempt to run it on a clean environment; if it fails immediately, the submission is incomplete

You'll sign in first, then come straight back here.

Proof required

Submit a session record containing: the reviewer's name and a verifiable credential (LinkedIn profile URL or company page) showing 3+ years building production AI systems; a summary of the novel prompt posed live and the agent's actual response; and a 300-word post-session reflection naming two specific improvements the reviewer identified and how you plan to or did address them.

What gets checked

• Reviewer credential is verifiable: LinkedIn URL or equivalent showing 3+ years in production AI/ML engineering — a university researcher with no production experience is insufficient
• Novel prompt is documented with the actual agent response, not a description of what the agent should have done — the response can be wrong, but it must be real
• Two specific improvements are named: vague feedback such as 'improve the prompts' does not count — improvement must be specific enough to be actionable

Common mistakes

• Choosing a reviewer who is a colleague already familiar with the project — the adversarial value comes from someone seeing it fresh and asking questions you have not rehearsed
• Running a scripted demo with pre-tested prompts rather than letting the reviewer pose novel ones — the reviewer must drive the novel prompt, not the submitter
• Treating the post-session reflection as a formality with generic improvements rather than specific actionable changes to code or architecture

Resources

What a verifier looks for

• Confirm the reviewer's LinkedIn URL shows a current or recent role with AI/ML engineering responsibilities — 'data science' or 'machine learning researcher' without production deployment experience is borderline
• Ask the submitter what the reviewer said about their eval design specifically — a good session will have produced specific feedback on the eval, not just the code
• Ask what the agent got wrong on the novel prompt — the submitter should be able to explain the failure, not just report that it happened
• Ask what they changed after the session — if nothing changed, the session was not taken seriously as a review

You'll sign in first, then come straight back here.