Milestone map
Milestone map
3 milestones
Develop a research question and select an appropriate quantitative design
4 weeks
Develop a specific social science research question that can be addressed with quantitative methods using existing data — a question about income inequality and social outcomes, migration patterns and economic effects, political participation and demographic factors, or similar. Identify an appropriate design (cross-sectional survey analysis, longitudinal analysis, comparative analysis across units) and locate a real existing public dataset. Pre-register the research design on OSF before any analysis begins.
Proof required
Research design document (500+ words) with OSF pre-registration link, covering: research question with theoretical rationale, dataset source and access method, study design and justification, key variables with operational definitions, planned analysis, and ethical considerations.
What gets checked
- Research question is specific and testable with existing data
- OSF pre-registration link present and timestamped before analysis begins
- Key variables are operationally defined — not just named
Common mistakes
- Pre-registering after analysis — the registration must be timestamped before any analysis
- Using a dataset without verifying its variables match the research question — check variable availability before committing to the design
- Operational definitions that are ambiguous — each variable must have a precise definition and a specific measurement approach in the dataset
Resources
Foundationstart here
What a verifier looks for
- Confirm OSF pre-registration link is present and timestamped before any analysis
- Verify dataset source is accessible
- Check operational definitions — confirm each variable has a precise definition and measurement approach
You'll sign in first, then come straight back here.
Conduct the analysis and produce a transparent data report
5 weeks
Conduct the pre-registered analysis using R, Python, or STATA. Report results transparently: all pre-registered analyses (including null results), descriptive statistics for key variables, primary analysis results with effect sizes and confidence intervals, and any deviations from the pre-registered analysis plan with reasoning. Deposit data, code, and materials on OSF or a public repository for reproducibility.
Proof required
Data analysis report (600+ words) covering: descriptive statistics, all pre-registered analyses with results (effect sizes and confidence intervals), any deviations from pre-registered plan with reasoning, and OSF or repository link for data, code, and materials.
What gets checked
- All pre-registered analyses are reported — including null results
- Effect sizes and confidence intervals are reported alongside p-values
- Data, code, and materials are deposited with an accessible link
Common mistakes
- Reporting only statistically significant pre-registered analyses — all pre-registered analyses must be reported
- Reporting p-values without effect sizes
- Not depositing code — reproducibility requires the analysis code, not just the data
Powstik Guide
A transparent analysis report with code and data deposited.
Steps
- Write your analysis as a script, not by clicking.
- Run every pre-registered analysis, including nulls.
- Report effect sizes with confidence intervals.
- Log each deviation from the plan and why.
- Deposit data, code and materials together.
Template
Descriptives (key variables): | Pre-registered analysis | Result | Effect size (CI) | |-------------------------|--------|------------------| Deviations and reasons: Repository link (data + code):
What gets sent back
- Only significant results reported.
- p-values without effect sizes.
- No deposited code.
Resources
Foundationstart here
Depthgo deeper
Masteryfor the dedicated
What a verifier looks for
- Check all pre-registered analyses are reported — compare report to pre-registration
- Verify effect sizes and confidence intervals are present
- Access repository link — confirm data and code are deposited
You'll sign in first, then come straight back here.
Defend quantitative analysis under social researcher challenge
1 week
Present the quantitative research design and analysis to a social science academic, quantitative methods researcher, or data scientist with social science experience. The reviewer must probe design and analysis — 'your design assumes no selection bias here — is that defensible?', 'you deviated from the pre-registered analysis — does the deviation produce different conclusions from the pre-registered specification?'. This Q&A satisfies the ADVERSARIAL VERIFICATION RULE.
Proof required
Q&A notes (250+ words) documenting the reviewer's credentials (social science academic, quantitative methods researcher, or data scientist with social science experience), at least three specific challenges to design or analysis with responses, and any updates committed to.
What gets checked
- Reviewer is a social science academic, quantitative methods researcher, or data scientist with social science experience
- At least three specific design or analysis challenges are documented
- At least one analysis update is committed to
Common mistakes
- Reviewer validates without probing design assumptions
- Not being able to explain deviations from pre-registered analysis under challenge
- No updates after a genuine challenge to analytical choices
Powstik Guide
Your analysis, challenged by a social researcher.
Steps
- Prepare a reason for every analysis choice.
- Note what your design cannot rule out.
- Book a social science academic or methods researcher.
- Ask for three challenges to design and analysis.
Template
Challenge session log Date: Duration: Reviewer name: Reviewer role and credentials: Challenge 1 (a design or analysis choice): What they asked: My answer: Did it hold? Challenge 2: What they asked: My answer: Did it hold? Challenge 3: What they asked: My answer: Did it hold? Update I am committing to:
What gets sent back
- A reviewer who doesn't probe assumptions.
- Deviations you can't explain.
- No update after the session.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Confirm reviewer is a social science academic, quantitative methods researcher, or data scientist with social science experience
- Review Q&A notes — confirm at least three specific design or analysis challenges are documented
- Check that at least one analysis update is committed to
You'll sign in first, then come straight back here.