Master Statistics for Data Analysis

Proof required

Share a Jupyter notebook or Python script (GitHub link) that loads the dataset, computes the descriptive statistics, and prints the results. Share a screenshot of the output. Write 200 words on one column where the mean and median differ significantly — what does that gap tell you about the distribution, and what would be misleading about reporting only the mean?

What gets checked

• Outlier detection uses IQR method (Q1 - 1.5×IQR, Q3 + 1.5×IQR) — the code must show the explicit calculation, not a z-score cutoff or eyeballed filter
• The mean/median interpretation names the specific column, states both values, and explains what the gap reveals about the shape of the distribution — not just 'they are different'
• The dataset has at least 5,000 rows and is real — not generated or invented; a CSV import from a public source with the data provenance noted in the notebook

Common mistakes

• Reporting statistics without interpretation — a table of means and standard deviations with no explanation of what they reveal is not the milestone; the paragraph interpretation is the proof of understanding
• Using a dataset where all distributions are approximately normal — pick a dataset with income, prices, or population data where skew is expected; symmetric distributions teach you less about when the mean misleads

Resources

What a verifier looks for

• Read the code — does the outlier detection explicitly compute Q1, Q3, and IQR, then filter rows outside Q1 - 1.5×IQR and Q3 + 1.5×IQR? Or does it use a z-score cutoff or eyeball the data? The IQR method is required.
• Ask the submitter: 'Your dataset has outliers in column X. Should you remove them before computing the mean?' Their answer reveals whether they understand that the decision depends on WHY the outliers exist — data errors are removed, genuine extremes are kept.
• Check the mean/median interpretation — it must name both values and explain the skew direction. 'Mean is higher than median' without explaining what that means about the distribution is incomplete.
• Ask: 'What does a standard deviation of 42 mean for this column?' If they can explain it in plain language (68% of values fall within 42 of the mean), they understand it. If they say 'it measures spread,' they memorised a definition.

Proof required

Share a document or notebook containing: (1) your null and alternative hypotheses written before you ran the test, (2) the code that runs the test, (3) the test statistic and p-value, (4) your interpretation. Write 250 words on what your p-value means, what it does NOT mean (specifically: it does not mean the probability that the null hypothesis is true), and what you would conclude if the p-value were 0.06 instead of 0.04.

What gets checked

• Null and alternative hypotheses are written in formal statistical language (H₀: μ₁ = μ₂) AND in plain English — both formulations required; one without the other is incomplete
• The p-value interpretation explicitly states what p-value does NOT mean — the milestone requires demonstrating understanding of the most common statistical misconception
• The 0.06 vs 0.04 reflection shows nuanced thinking — not just 'I would fail to reject the null' but why arbitrary cutoffs are a limitation of hypothesis testing

Common mistakes

• Forming the hypothesis AFTER seeing the data — this is p-hacking; the hypothesis must be committed to before the test is run; use a timestamped document or separate git commit to prove the order
• Interpreting p < 0.05 as 'the effect is real' — p-value significance does not mean practical significance; a test with 1 million rows will find 'significant' differences that are meaninglessly small

Resources

What a verifier looks for

• Ask for the timestamp of the hypothesis document. Was it created before the test was run? Check the git commit history or file creation date. A hypothesis written after seeing the result is not a hypothesis — it is a post-hoc story.
• Read the p-value interpretation. Does it explicitly state what p-value does NOT mean? The most common misconception is 'p = 0.03 means there is a 3% chance the null is true.' If the submission does not address this, ask the submitter to explain it.
• Ask: 'Your p-value is 0.001. Is the effect large?' If they say yes without checking effect size, they conflate statistical and practical significance. The correct answer is 'I need to check the effect size — p-value doesn't tell me that.'
• Check the 0.06 vs 0.04 reflection. Does it show nuance? 'I would fail to reject the null' is technically correct but insufficient. 'The arbitrary 0.05 cutoff means 0.06 and 0.04 are treated differently despite being nearly identical evidence' is the level of understanding this milestone requires.

Proof required

Share a Jupyter notebook (GitHub link) with: the train/test split code, the model fit, the evaluation metrics on both training and test sets, and the residual plot. Share a second model that is deliberately overfit (more features than the first) with its train and test metrics. Write 200 words on what overfitting looks like in the numbers — what metric told you the model was overfit, and why a model that scores 0.97 on training data and 0.43 on test data is worse than one that scores 0.71 on both.

What gets checked

• Train/test split happens BEFORE model fitting — the code must show this order; fitting on all data then evaluating on the same data is not a valid evaluation
• Residual plot is included and interpreted — not just present; the notebook must contain a sentence explaining what the residual plot shows about the model's assumptions
• The overfit model has measurably worse test performance than training performance — the numbers must show the gap, not just assert it

Common mistakes

• Fitting the model on all data then reporting accuracy — this is one of the most common mistakes in data science; the train/test split must happen before the model sees any data
• Using R² alone as the evaluation metric — R² can be high even when the model is useless on new data; RMSE on the test set is the metric that matters for generalisability

Resources

What a verifier looks for

• Read the notebook top to bottom. Does the train_test_split happen before model.fit()? If fit() is called before or without a split, the evaluation is invalid regardless of the metrics reported.
• Check the residual plot interpretation. Is there a sentence explaining what the plot shows? Residuals should be randomly distributed around zero — a pattern in the residuals means the model is missing something. Does the submission address this?
• Look at the two models' metrics side by side. Is the training score of the overfit model notably higher than its test score? If both scores are similar, the model is not overfit — the submitter needs more features.
• Ask: 'Your overfit model has R²=0.97 on training and R²=0.43 on test. What would you do to fix it?' If they say 'remove features' or 'add regularisation,' they understand the bias-variance tradeoff. If they say 'change the algorithm,' they don't yet.

Proof required

Share a Python notebook (GitHub link) containing the power calculation, the simulated data, the statistical test, and the result. Share the one-page product brief as a markdown file in the same repository. Write 200 words on the difference between statistical significance and business significance — give a specific example where a statistically significant result should NOT be shipped and one where a non-significant result might still lead to a change.

What gets checked

• Power calculation is present and uses realistic parameters — baseline conversion rate, minimum detectable effect, desired power (0.8), and significance level (0.05); the sample size is derived from these, not guessed
• The product brief is written for a non-statistician — no p-values in the brief; findings expressed as business outcomes ('the new button increased signups by an estimated 12%') not statistical outputs
• The statistical vs business significance reflection gives concrete examples — 'significant but not worth shipping' and 'not significant but worth changing' are both explained with specific numbers

Common mistakes

• Skipping the power calculation and choosing a sample size arbitrarily — a test with insufficient power will fail to detect real effects; the sample size must be calculated from the effect size you care about detecting
• Writing the product brief in statistical language — a product manager who receives a brief containing 'reject the null hypothesis at α=0.05' will not act on it; the brief must translate findings into decisions

Resources

What a verifier looks for

• Check the power calculation. Does it specify baseline rate, minimum detectable effect, power, and significance level explicitly? Or is the sample size a round number that was clearly guessed? Calculated sample sizes are rarely round numbers.
• Read the product brief. Does it contain any p-values or statistical terminology? If yes, it is not a product brief — it is a statistics report. Ask the submitter to rewrite it for a product manager who does not know what a p-value is.
• Ask: 'Your test ran for 14 days and p=0.03. You then ran it for 7 more days and now p=0.18. What happened?' If they understand peeking (stopping tests early when significance is reached), they understand experiment integrity. If they say 'the effect disappeared,' they don't.
• Check the statistical vs business significance reflection. The 'should not ship despite significance' example must name a specific scenario with numbers. Vague examples do not count.

Proof required

Share the presentation or report (PDF, slides link, or markdown). Share one piece of feedback from a real non-technical person who reviewed it — a screenshot of a message or a written note. Write 200 words on what they did not understand and how you would change the presentation to make that point clearer.

What gets checked

• Every chart has a conclusion title — not 'Figure 1: Monthly Revenue' but 'Revenue grew 47% after the price change in March'; check every chart in the submission
• Every number is contextualised — no bare statistics without a reference point or plain-language interpretation anywhere in the document
• The non-technical feedback is from a real person and the reflection shows what was genuinely confusing — not a friend who said 'looks great'

Common mistakes

• Getting feedback from another data person — the feedback must come from someone without statistical training; a colleague in data science is not the target audience
• Charts with axis titles instead of conclusion titles — 'Revenue by Month' is an axis title; 'Revenue peaked in December before declining 40% in Q1' is a conclusion title; the distinction is whether the chart tells you what to think

Resources

What a verifier looks for

• Open every chart in the presentation. Read the title of each one. Does it state a conclusion or just label what is plotted? Count the number of conclusion titles vs label titles. Any label title is a revision request.
• Find the most important number in the analysis. Is it contextualised? Does it have a comparison point (vs last year, vs industry, vs target)? A bare number without context is incomplete.
• Check the non-technical feedback. Is it from a real person who is not in data or statistics? Ask the submitter who reviewed it. 'My colleague who is also an analyst' is not the right reviewer.
• Ask the submitter: 'Explain your most important finding in one sentence that a 10-year-old could understand.' If they can do it, they understand their own analysis. If they need jargon to explain it, they do not yet.