Milestone map
Milestone map
3 milestones
Develop a research question and select an appropriate quantitative design
4 weeks
Develop a specific social science research question that can be addressed with quantitative methods using existing data — a question about income inequality and social outcomes, migration patterns and economic effects, political participation and demographic factors, or similar. Identify an appropriate design (cross-sectional survey analysis, longitudinal analysis, comparative analysis across units) and locate a real existing public dataset. Pre-register the research design on OSF before any analysis begins.
Proof required
Research design document (500+ words) with OSF pre-registration link, covering: research question with theoretical rationale, dataset source and access method, study design and justification, key variables with operational definitions, planned analysis, and ethical considerations.
What gets checked
- Research question is specific and testable with existing data
- OSF pre-registration link present and timestamped before analysis begins
- Key variables are operationally defined — not just named
Common mistakes
- Pre-registering after analysis — the registration must be timestamped before any analysis
- Using a dataset without verifying its variables match the research question — check variable availability before committing to the design
- Operational definitions that are ambiguous — each variable must have a precise definition and a specific measurement approach in the dataset
Resources
Foundationstart here
What a verifier looks for
- Confirm OSF pre-registration link is present and timestamped before any analysis
- Verify dataset source is accessible
- Check operational definitions — confirm each variable has a precise definition and measurement approach
Conduct the analysis and produce a transparent data report
5 weeks
Conduct the pre-registered analysis using R, Python, or STATA. Report results transparently: all pre-registered analyses (including null results), descriptive statistics for key variables, primary analysis results with effect sizes and confidence intervals, and any deviations from the pre-registered analysis plan with reasoning. Deposit data, code, and materials on OSF or a public repository for reproducibility.
Proof required
Data analysis report (600+ words) covering: descriptive statistics, all pre-registered analyses with results (effect sizes and confidence intervals), any deviations from pre-registered plan with reasoning, and OSF or repository link for data, code, and materials.
What gets checked
- All pre-registered analyses are reported — including null results
- Effect sizes and confidence intervals are reported alongside p-values
- Data, code, and materials are deposited with an accessible link
Common mistakes
- Reporting only statistically significant pre-registered analyses — all pre-registered analyses must be reported
- Reporting p-values without effect sizes
- Not depositing code — reproducibility requires the analysis code, not just the data
Resources
What a verifier looks for
- Check all pre-registered analyses are reported — compare report to pre-registration
- Verify effect sizes and confidence intervals are present
- Access repository link — confirm data and code are deposited
Defend quantitative analysis under social researcher challenge
1 week
Present the quantitative research design and analysis to a social science academic, quantitative methods researcher, or data scientist with social science experience. The reviewer must probe design and analysis — 'your design assumes no selection bias here — is that defensible?', 'you deviated from the pre-registered analysis — does the deviation produce different conclusions from the pre-registered specification?'. This Q&A satisfies the ADVERSARIAL VERIFICATION RULE.
Proof required
Q&A notes (250+ words) documenting the reviewer's credentials (social science academic, quantitative methods researcher, or data scientist with social science experience), at least three specific challenges to design or analysis with responses, and any updates committed to.
What gets checked
- Reviewer is a social science academic, quantitative methods researcher, or data scientist with social science experience
- At least three specific design or analysis challenges are documented
- At least one analysis update is committed to
Common mistakes
- Reviewer validates without probing design assumptions
- Not being able to explain deviations from pre-registered analysis under challenge
- No updates after a genuine challenge to analytical choices
Resources
What a verifier looks for
- Confirm reviewer is a social science academic, quantitative methods researcher, or data scientist with social science experience
- Review Q&A notes — confirm at least three specific design or analysis challenges are documented
- Check that at least one analysis update is committed to