Ship AI Products in Production

Proof required

Screen recording (2–3 min) showing your terminal: Python version, pip list with anthropic or openai installed, and a live API call running that returns a real response. Narrate what each step does. The API key must load from an environment variable — never appear on screen.

What gets checked

• API key loads from a .env file or environment variable — it must not be visible anywhere in the recording
• The API response is from the real model (not mocked) — the model name appears in the response object
• Environment is isolated via venv or conda — not installed to system Python

Common mistakes

• Installing to system Python and hitting permission issues on a different machine — use pyenv from the start
• Not understanding the difference between synchronous and streaming responses — confusion here blocks M2
• Spending days on IDE setup before making the first API call — run the call first; configure the editor after

Resources

What a verifier looks for

• API key must NOT appear on screen — it should load from an env var; reject if the key is visible in any frame
• Response must be from the real API — look for a model name (claude-*, gpt-*) in the response object shown on screen
• Python version should be ≥ 3.10 — older versions will cause import failures with modern LLM libraries

Proof required

Share the public URL and a 3-minute screen recording where you use the feature as a first-time user would, showing the LLM response arriving in the UI. Describe what the feature does and who it is for. At least one person other than you must have successfully used it before submission.

What gets checked

• The URL is accessible without login and works at time of review — not a localhost or staging link
• The LLM response appears in the UI, not just server logs — users can see the output
• There is a real use case — not a bare chat box or prompt echo; the feature solves a specific problem

Common mistakes

• Spending 80% of time on UI polish before the core feature works — build the backend first, style it last
• Not setting the API key as an environment variable on the hosting platform — the app works locally but fails in production
• Choosing a use case too broad ('a general AI assistant') — the narrower the first feature, the faster you ship

Resources

What a verifier looks for

• Open the URL at time of review — do not rely on screenshots alone; the live URL must work
• The feature must accept user input (not hardcoded) and call the LLM with that input
• Ask: what would a non-technical user do with this? — the answer should be obvious from the UI without explanation

Proof required

Screen recording (4–5 min): load your document set, ask 3 questions the LLM could not answer from training data alone, show the retrieved context that informed each answer, and show a specific citation in at least 2 responses. Narrate the pipeline flow. Demonstrate what happens when you ask a question outside the document set.

What gets checked

• The LLM cannot answer the questions without retrieved context — verify by asking the same questions without RAG and showing the difference
• Citations reference specific sections of source documents, not just 'source: document.pdf'
• Chunking and embedding are your code, not a drag-and-drop tool — you can explain every step if asked

Common mistakes

• Chunks too large (>1,000 tokens) — retrieval becomes imprecise and answers are vague; chunk to 256–512 tokens for dense factual content
• Not testing retrieval quality before connecting the LLM — if retrieval is broken, the LLM adds hallucination on top of wrong context
• Cosine similarity threshold too low — retrieves irrelevant context that confuses the model and produces fabricated citations

Resources

What a verifier looks for

• Ask a question outside the document set — the system should say 'I don't know' or return low-confidence signal, not hallucinate a confident answer
• Verify citations link to real sections — ask the submitter to open the source document and show the cited passage
• The chunking strategy should be explained and intentional — ask what chunk size they chose and why

Proof required

Screen recording (3–4 min) showing a conversation where the LLM autonomously decides to call multiple tools across at least 2 turns. Show the raw tool-use response from the API before your code processes it. Narrate why the LLM chose each tool. Show what happens when a tool call fails.

What gets checked

• The routing is done by the LLM — tool selection appears in the completion object, not in hardcoded if/else logic
• At least 2 different tools are called in the recording — not the same tool twice
• Error handling is demonstrated — show what the application does when a tool call returns an error or empty result

Common mistakes

• Tool schemas too vague — the LLM calls the wrong tool or hallucinates parameters; write precise descriptions and parameter types
• No retry logic when a tool call returns an error — the whole conversation fails on the first bad API response
• Forgetting to include the tool result back in conversation context — the model repeats the same call indefinitely

Resources

What a verifier looks for

• Show the raw API response with tool_use content blocks — this proves the LLM is calling tools, not the code routing manually
• The tools must make real external calls (real API or DB query), not return hardcoded test data
• Both tools must be called in the submitted recording — a demo with only one tool is M4-incomplete

Proof required

Screen recording (5–6 min): show the agent completing a task end-to-end with no human input after the initial prompt, including the agent's reasoning trace (verbose logging enabled). Then run your eval suite live and show the results (pass/fail counts). Narrate what 'success' means for each eval case.

What gets checked

• The agent completes at least 4 tool calls without human intervention in the recording — not a scripted walkthrough
• The eval suite is automated and quantitative — shows a numerical pass rate, not manual spot-checking
• At least 2 failure cases are shown where the agent self-corrects or gracefully stops — not every eval case passes

Common mistakes

• No stopping condition — the agent runs in an infinite tool-call loop until the context window fills; implement a max-steps guard
• Eval dataset too easy — 10/10 on trivial cases does not prove the agent works on real tasks; include 3 adversarial cases
• Not logging the agent's reasoning — impossible to debug failures without a trace; use verbose logging from day one

Resources

What a verifier looks for

• The eval suite must run without modification — if it requires setup steps, a README must be included
• Show at least one task where the agent fails and explain why — an all-pass eval suite is a sign of insufficient test cases
• The agent's tool calls must be in the logs — verify they are real calls, not simulated responses in the recording

Proof required

Share the product URL, an analytics dashboard screenshot showing ≥10 unique users in the last 30 days (with timestamps), and a post-mortem (400–600 words) covering: the problem you solved, the AI component's role, 3 specific things users told you, and 2 changes you made based on that feedback. Link to a published post if you wrote one.

What gets checked

• 10+ unique users confirmed in analytics with timestamps — not self-reported; screenshot must show user count (not page views) and dates
• At least 2 documented changes made from user feedback — not your own preferences or assumptions
• The AI component is the core value, not a gimmick — removing it would break the product's main function

Common mistakes

• Building in stealth too long — users reveal problems you cannot predict; ship to 5 users before you think it is ready
• No feedback loop — without one, you improve based on guesses, not evidence; add a feedback button on day one
• Choosing a use case where AI adds noise rather than value — test the manual version first; if manual is fast enough, AI may not help

Resources

What a verifier looks for

• Analytics must show unique users, not page views — the screenshot should display user count with a date range
• The post-mortem must reference specific user quotes or observed behaviors, not generic observations like 'users found it useful'
• The product must be accessible at review time — verify the URL; do not approve on screenshots alone