Build an End-to-End ML Pipeline

Proof required

Submit your architecture decision record (500–800 words) covering: the prediction problem with a specific business or research context, the dataset (source URL, size, class distribution or target distribution, and a note on known data quality issues), the evaluation metric selected and why it is appropriate for this problem (not 'I chose accuracy because it is standard'), the model family you plan to try first and why, and a reproducibility plan (how you will set seeds, log experiments, and ensure someone else could reproduce your results).

What gets checked

• Problem framing is specific — includes a named decision-maker, context, and the cost of a wrong prediction in each direction (false positives and false negatives are not symmetric in most real problems)
• Evaluation metric is justified relative to the class imbalance and problem context — a problem with 1% positive rate should not use accuracy as the primary metric without explanation
• Dataset is a real dataset with a source URL and a note on at least one known data quality issue — clean pre-split tutorial datasets without any quality issues are not real-world datasets

Common mistakes

• Choosing a dataset that is too clean and too small — classic tutorial datasets like Iris or the Titanic dataset are pre-cleaned to the point that no real data engineering decisions are required; a real ML pipeline problem includes messy data
• Treating the architecture decision record as a formality — the ADR is the primary proof that genuine engineering thinking preceded the code; a vague ADR ('I will try several models and pick the best one') does not demonstrate problem-first thinking

Resources

What a verifier looks for

• Ask the submitter to explain why false positives are worse than false negatives (or vice versa) in their specific problem — tests that the metric choice was driven by the problem, not by convenience.
• Ask what they would change about the dataset if they could — tests awareness of the real data quality constraints they are working with.
• Ask what the baseline model is and what score it produces on the evaluation metric — tests that the architecture decision started with a baseline, not a complex model.

Proof required

Submit your code in a public GitHub repository with a README explaining how to reproduce the results. The README must include: the exact command to reproduce the evaluation results from the repository's raw data, the final evaluation metrics on the held-out test set, a confusion matrix or equivalent visualisation, and a brief interpretation of what the model gets wrong. In the repository, include an experiment log (MLflow experiment, Weights & Biases log, or a structured CSV with run parameters and results) showing at minimum three model variants evaluated during development.

What gets checked

• Pipeline has no data leakage — preprocessing (scaling, encoding, imputation) is fit on the training set only and applied to the validation and test sets; preprocessing fit on the full dataset is the most common data leakage pattern
• Experiment log shows genuine model selection process — at minimum three runs with different hyperparameters or model families; selecting the first model tried is not a pipeline, it is a single model
• Evaluation on the test set is done exactly once — the test set is not used for model selection or hyperparameter tuning; if the test set was used iteratively, the evaluation is not credible

Common mistakes

• Fitting preprocessing on the full dataset before splitting — this leaks test-set information into the preprocessing step; scaling and encoding must always be fit on the training split only
• Reporting only accuracy — accuracy on an imbalanced dataset is uninformative; the pipeline must report precision, recall, and F1 at minimum for a classification problem

Resources

What a verifier looks for

• Ask the submitter to walk through exactly when the preprocessing was fit and on which data split — this is the data leakage question; the answer should be 'fit on training, transform on validation and test'.
• Ask what the model gets wrong — ask for specific examples from the confusion matrix or error analysis; a practitioner who cannot characterise their model's failures has not done a thorough evaluation.
• Ask why they chose the specific hyperparameter values in the best model — tests whether the experiment log reflects genuine exploration or a single run.

Proof required

Submit the URL of your final public GitHub repository (with the README reproduction instructions from M2) and a Q&A record documenting: the reviewer's challenge to at least one design decision (dataset choice, feature engineering, model selection, evaluation methodology), your response to that challenge, the reviewer's assessment of whether the pipeline demonstrates sound ML engineering practice, and the reviewer's name and their ML engineering or data science background. The reviewer does not need to run the code — they review the methodology and probe the submitter's reasoning.

What gets checked

• Repository README reproduces the evaluation with a single command — a reviewer should not need to read the code to understand how to run the pipeline
• Q&A record shows genuine challenge and substantive response — not 'I chose logistic regression because it is interpretable' but 'I chose logistic regression over random forest because the business context requires understanding which features drive the prediction, and the accuracy trade-off was acceptable given the 3% performance gap'
• Reviewer has ML engineering or data science experience — a developer who has never worked with ML is not qualified to probe the evaluation methodology

Common mistakes

• A README that lists dependencies but does not include the exact command to reproduce results — reproducibility means a single command, not a set of instructions that require interpretation
• Choosing a reviewer who validates the work without challenging it — the Q&A must probe at least one design decision; a reviewer who says 'looks good' without probing is not providing a credible evaluation

Resources

What a verifier looks for

• Ask what you would do differently if you had 3× more data — tests understanding of the relationship between data size and model behaviour.
• Ask what the most important feature in the model is and why — tests that the submitter has done at least basic feature importance analysis.
• Ask how the pipeline would break in production and what monitoring you would put on it — tests ML engineering thinking beyond the Jupyter notebook.