Level 6 · Research preparation · advanced assessment
Independent reproduction and discrepancy court
Make artifacts portable, reruns independent, and disagreements diagnosable before declaring replication or failure.
Complete all three challenges. Show assumptions, units, uncertainty, and the boundary between calculation, model-dependent inference, observation, and unresolved claim.
Challenge 1 · derivation
Reconstruct the dependency chain
Starting from a claimed result, derive the full dependency graph from raw observation through calibration, preprocessing, model, statistic, and figure. Identify every parameter that could change the conclusion.
Required deliverables
- • Executable-style dependency graph
- • Parameter and version manifest
- • Recomputed result with checksum trail
Scoring criteria · 20 points
- • Graph is complete and ordered: 8 points
- • Versions and parameters are recoverable: 7 points
- • Recalculation matches or explains mismatch: 5 points
Challenge 2 · analysis
Quantify the discrepancy
Compare original and independent results using uncertainty-aware effect estimates and a structured discrepancy model. Evaluate protocol, apparatus, environment, data, and analysis explanations rather than treating replication as a binary vote.
Required deliverables
- • Comparable effect-size table
- • Discrepancy likelihood or decision matrix
- • Sensitivity to reasonable analysis variants
Scoring criteria · 20 points
- • Comparability and uncertainties: 7 points
- • Competing discrepancy explanations: 7 points
- • Robustness and calibrated conclusion: 6 points
Challenge 3 · design
Stage a cross-lab swap and adjudication
Design a two-laboratory replication in which teams exchange data, code, and where practical apparatus or calibration artifacts. Preserve independent analysis, then use a frozen adjudication process for disagreements.
Required deliverables
- • Swap matrix and contamination safeguards
- • Independent-analysis and reveal sequence
- • Discrepancy remediation and public release plan
Scoring criteria · 20 points
- • Independence survives the swap: 7 points
- • Reveal sequence localizes discrepancies: 7 points
- • Release plan includes negative and indeterminate outcomes: 6 points
Instructor solution outlines
Look for reasoning that survives inspection
1. Reconstruct the dependency chain
- • Trace the headline result backward to raw inputs without skipping manual transformations.
- • Record software, calibration, random seeds, and inclusion rules.
- • Classify any mismatch before attributing it to scientific disagreement.
2. Quantify the discrepancy
- • Harmonize estimands before comparing numerical results.
- • Test prespecified discrepancy categories with available evidence.
- • Use replicated, not replicated, contradicted, and indeterminate only under explicit criteria.
3. Stage a cross-lab swap and adjudication
- • Run own-data/own-code, swapped-data, swapped-code, and shared-calibration cells where feasible.
- • Freeze primary outputs before cross-team discussion.
- • Publish the matrix of outcomes so readers can see where agreement changes.