Ship AI Products in Production

Proof required

Record 2–3 minutes of your terminal. Show the Python version and pip list with anthropic or openai installed. Show a live API call returning a real response. Narrate each step. The key must load from an environment variable and never appear on screen.

What gets checked

• The key loads from a .env file or environment variable. It never appears in the recording.
• The response is from the real model, not mocked. The model name shows in the response.
• The setup uses venv or conda, not the system Python.

Common mistakes

• Installing into system Python. It breaks on another machine; use pyenv from the start.
• Not knowing standard versus streaming responses. That confusion blocks Milestone 2.
• Days of editor setup before the first call. Make the call first.

Resources

What a verifier looks for

• The key must not appear in any frame. Reject if it does.
• Look for a model name, like claude-* or gpt-*, in the response.
• Python should be 3.10 or newer. Older versions break modern LLM libraries.

You'll sign in first, then come straight back here.

Proof required

Share the public URL. Record 3 minutes using it as a first-time user. The response must appear in the page. Say what it does and who it's for. At least one other person must have used it successfully first.

What gets checked

• The URL needs no login and works at review. Localhost or staging links don't count.
• The response shows in the page, not only in server logs.
• It solves a specific problem. A bare chat box or prompt echo doesn't count.

Common mistakes

• Polishing the UI before the feature works. Build the backend first.
• Forgetting to set the API key on the host. It works locally, then fails live.
• A use case too broad, like 'a general AI assistant'. Narrow ships faster.

Resources

What a verifier looks for

• Open the URL at review. Screenshots alone aren't enough.
• It must take user input, not hardcoded text, and send it to the model.
• Would a non-technical user know what to do? It should be obvious from the page.

You'll sign in first, then come straight back here.

Proof required

Record 4–5 minutes. Load your documents and ask three questions the model couldn't answer from training alone. Show the retrieved context behind each answer. Show a specific citation in at least two answers. Narrate the pipeline. Show what happens with a question outside your documents.

What gets checked

• The model can't answer without retrieval. Show the same questions without it.
• Citations point to specific sections, not just 'source: document.pdf'.
• Chunking and embedding are your own code, and you can explain each step.

Common mistakes

• Chunks over 1,000 tokens. Retrieval gets vague; 256–512 tokens suits dense facts.
• Not testing retrieval before adding the model. Wrong context plus a model means worse errors.
• A similarity threshold that's too low. Irrelevant context leads to made-up citations.

Resources

What a verifier looks for

• Ask something outside the documents. It should say it doesn't know, not answer confidently.
• Ask them to open the source and show a cited passage.
• Ask what chunk size they chose and why. It should be deliberate.

You'll sign in first, then come straight back here.

Proof required

Record a 3–4 minute conversation. The model calls tools across at least two turns. Show the raw tool-use response before your code handles it. Explain why the model chose each tool. Show what happens when a tool call fails.

What gets checked

• The model does the routing. Tool choice appears in the response, not your if/else.
• At least two different tools are called, not one tool twice.
• Error handling is shown for a failed or empty tool result.

Common mistakes

• Vague tool schemas. The model picks the wrong tool or invents parameters.
• No retry when a tool errors. One bad response kills the conversation.
• Not returning the tool result to the conversation. The model repeats the call forever.

Resources

What a verifier looks for

• Look for tool_use blocks in the raw response. That proves the model chose the tools.
• The tools must make real calls, not return hardcoded test data.
• Both tools must be called in the recording. One tool isn't enough.

You'll sign in first, then come straight back here.

Proof required

Record 5–6 minutes. Show the agent finishing a task with no input after the first prompt. Keep verbose reasoning logs on. Then run the evaluation suite live and show pass and fail counts. Explain what success means for each case.

What gets checked

• At least 4 tool calls with no human input, not a scripted walkthrough.
• The evaluation is automated and gives a numeric pass rate.
• At least 2 cases show the agent correcting itself or stopping cleanly.

Common mistakes

• No stopping condition. It loops until the context fills; add a maximum step count.
• Tests too easy. Ten out of ten on trivial cases proves little; add 3 hard ones.
• No reasoning logs. You can't debug failures without a trace.

Resources

What a verifier looks for

• The evaluation must run unchanged. If it needs setup, there must be a README.
• Ask for one failed task and why it failed. All passing means too few hard cases.
• Check the logs for real tool calls, not simulated responses.

You'll sign in first, then come straight back here.

Proof required

Share the product URL. Add a dated analytics screenshot showing 10+ unique users in 30 days. Add a 400–600 word post-mortem. Cover the problem and the AI's role. Add three things users said and two changes you made. Link any post you published.

What gets checked

• 10+ unique users in analytics with dates. It shows users, not page views.
• At least 2 changes came from user feedback, not your own preferences.
• The AI is the core. Removing it would break the main function.

Common mistakes

• Building in secret too long. Ship to 5 users before it feels ready.
• No feedback loop. Add a feedback button on day one.
• AI that adds noise. Try the manual version first; if it's fast enough, AI may not help.

Resources

What a verifier looks for

• Analytics must show unique users with a date range, not page views.
• The post-mortem needs specific quotes or behaviour, not 'users found it useful'.
• Open the URL at review. Don't approve on screenshots alone.

You'll sign in first, then come straight back here.