Milestone map
Milestone map
3 milestones
Design a behavioural research study and pre-register it on OSF
4 weeks
Design a behavioural data collection study on a non-sensitive topic with a non-vulnerable population — options include observational studies of publicly visible behaviour, survey research with anonymous adult participants on topics posing minimal risk, or secondary analysis of a publicly available dataset. Complete a full research design document: research question, theoretical rationale, sampling strategy, data collection method, variables and operationalisation, planned analysis, and ethical considerations. Pre-register the design on OSF (Open Science Framework) before collecting any data.
Proof required
OSF pre-registration link (public or pending) for the study design, plus a research design document (600+ words) covering research question, theoretical rationale, sampling strategy, variables and operationalisation, planned analysis, and ethical considerations.
What gets checked
- OSF pre-registration is submitted before data collection begins — not written retrospectively
- Sampling strategy specifies the target population and why it is appropriate for the research question
- Ethical considerations address data anonymisation, informed consent process, and any potential participant risks
Common mistakes
- Starting data collection before pre-registering — pre-registration is only meaningful before data collection; retroactive pre-registration is not legitimate research design documentation
- Choosing a research question without a theoretical rationale — behavioural research is framed by theory about why the pattern should exist, not just curiosity about whether it does
- Underspecifying the analysis plan — a pre-registration that says 'I will run some statistical tests' does not constrain analytical flexibility
Powstik Guide
A pre-registered study and a 600-word design document.
Steps
- State hypotheses before collecting data.
- Fix sample size and analysis plan.
- Plan consent and ethics steps.
- Register it on OSF.
Template
Hypotheses: Sample size and why: Analysis plan: Ethics/consent: OSF link:
What gets sent back
- Registered after data collection.
- No analysis plan.
- No consent plan.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Access the OSF pre-registration link — confirm it is public or pending, and pre-dates data collection
- Review the sampling strategy — confirm it specifies the target population with justification
- Check the ethical considerations — confirm anonymisation, consent process, and participant risk assessment are addressed
You'll sign in first, then come straight back here.
Collect real behavioural data and conduct the pre-registered analysis
5 weeks
Execute the study designed in M1: collect real data from real participants or a publicly available dataset following the pre-registered design. Complete the pre-registered analysis using free statistical tools (Python with pandas and scipy, R, or JASP). Produce a results section documenting what was found: descriptive statistics, inferential test results with effect sizes, and any deviations from the pre-registered analysis plan (with justification). Data and analysis code must be available for inspection.
Proof required
Results document (400+ words) covering descriptive statistics, inferential test results with effect sizes, and any deviations from the pre-registered plan with justification — plus a public OSF link or GitHub repository containing the anonymised data and analysis code.
What gets checked
- Results report effect sizes alongside p-values — statistical significance alone is insufficient
- All deviations from the pre-registered analysis are documented with justification — not silently omitted
- Anonymised data and analysis code are publicly accessible at the submitted link
Common mistakes
- Reporting only statistically significant results and omitting null findings — pre-registration exists precisely to prevent this; the pre-registered analysis must be reported in full
- Not documenting deviations from the pre-registered plan — any analysis not pre-registered must be clearly labelled as exploratory
- Using effect size language without computing effect sizes — Cohen's d, r, or eta-squared must be calculated, not just mentioned
Powstik Guide
A 400-word results write-up following your pre-registration.
Steps
- Report descriptive statistics first.
- Run the pre-registered test.
- Report effect sizes.
- Label any extra analyses as exploratory.
Template
Descriptives: Test and result: Effect size: Deviations from plan: Exploratory analyses:
What gets sent back
- No effect sizes.
- Unreported deviations.
- Under 400 words.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Access the OSF or GitHub link — confirm anonymised data and analysis code are present and runnable
- Check results report — confirm effect sizes are reported alongside p-values
- Review deviations section — confirm any unplanned analyses are labelled as exploratory
You'll sign in first, then come straight back here.
Present findings and methodology to a psychology researcher
1 week
Present the study design, results, and interpretation to a psychology researcher or social scientist who challenges both the methodology and the conclusions. The reviewer must probe specific decisions — 'why this operationalisation of your variable?', 'what alternative explanation does your design not rule out?', 'your effect size is small — is this finding practically meaningful?'. Document the challenges and your responses. This methodology Q&A satisfies the ADVERSARIAL VERIFICATION RULE for social-* outcomes.
Proof required
Q&A notes (250+ words) documenting the reviewer's research background, at least three specific methodological or interpretive challenges, your responses, and any design or analysis changes you would make based on the session.
What gets checked
- Reviewer has research methodology experience in psychology or social science
- At least three specific methodological challenges are documented — not general approval
- At least one design or analysis change is documented based on the Q&A
Common mistakes
- Presenting to someone who validates rather than challenges — the reviewer must engage with the research design specifically
- Not addressing alternative explanations when challenged — a researcher who cannot identify confounds in their own design has not understood the methodology deeply enough
- Ignoring practical significance when challenged — a small but statistically significant effect in a well-powered study is a real result worth interrogating
Powstik Guide
Your study, challenged by a psychology researcher.
Steps
- Explain how you measured each key variable.
- List two alternative explanations for your result.
- Book a psychology researcher or social scientist.
- Ask for three challenges to method and conclusions.
Template
Challenge session log Date: Duration: Reviewer name: Reviewer role and credentials: Challenge 1 (a measure or alternative explanation): What they asked: My answer: Did it hold? Challenge 2: What they asked: My answer: Did it hold? Challenge 3: What they asked: My answer: Did it hold? Update I am committing to:
What gets sent back
- A reviewer who only validates.
- No alternative explanations addressed.
- No changes proposed after the session.
Resources
Foundationstart here
Depthgo deeper
What a verifier looks for
- Confirm the reviewer has research methodology experience in psychology or social science
- Review Q&A notes — confirm at least three specific methodological challenges are documented
- Check that at least one design or analysis change is documented based on the session
You'll sign in first, then come straight back here.