Experimental methods and error analysis
Design measurements that survive calibration, noise, bias, replication, and adversarial reanalysis.
Unify uncertainty propagation, parameter estimation, controls, blinding, signal processing, preregistration, and open data around the precision tests central to this corpus.
Before you begin
- • Level 2 measurement and mechanics
- • Calculus and probability
- • Spreadsheet or basic coding fluency
By the end, you can
- • Propagate random and systematic uncertainty.
- • Fit models and inspect residuals rather than quote fit scores alone.
- • Design nulls, controls, blinding, and preregistered analyses.
- • Publish reproducible apparatus, calibration, data, and code packages.
Interactive model
Explore before calculating
Live laboratory
Blinded residual and covariance studio
Recover a synthetic hidden injection, combine random and correlated systematic uncertainty, then apply a declared look-elsewhere penalty before interpreting the residual.
Random σ of mean: 2.000
Total correlated σ: 4.796
Injection recovery: 0.417σ
Null residual: 4.587σ
Local p: 4.494e-6
Global p: 4.494e-6
The blinded estimator recovers the injection within the declared 2σ calibration gate. The search-adjusted residual crosses the declared 3σ-style threshold.
All values are synthetic. The global-p expression approximates independent searches; correlated windows require simulation or an effective-trials model. A real analysis must freeze the estimator, bandwidth, covariance, exclusions, search scope, controls, and success threshold before unblinding.
Level 3 · Undergraduate core teaching kit
Record the investigation. Teach the reasoning.
A learner-facing lab record and a course-specific instructor guide turn the live model into a repeatable classroom investigation.
Learner record
Blinded residual and search-penalty analysis
When does a local residual remain credible after correlated uncertainty, injection recovery, and the declared search space are included?
Download learner recordInstructor guide
Teach for evidence, not button pushing
Learners treat blinding, correlated uncertainty, injection recovery, and multiplicity correction as one evidence pipeline rather than optional post hoc cautions.
Download instructor guideLesson 1 of 3
Calibration and uncertainty budgets
Which uncertainty is statistical, which is systematic, and which correlation prevents simple averaging?
Random uncertainty varies across trials and often shrinks with repeated independent measurements. Systematic uncertainty shifts results coherently and requires calibration, comparison, or redesign.
A budget names every contribution, distribution, correlation, sensitivity coefficient, and combination rule.
Worked example
For z=x+y with independent σx=0.3 and σy=0.4, find σz.
- 1. Use variance addition for independent inputs.
- 2. σz²=0.3²+0.4².
- 3. Take the square root.
σz=0.5.
Try it
Instrument uncertainty budget
Materials: Any sensor specification and repeated dataset.
- 1. List resolution, calibration, drift, alignment, and repeatability.
- 2. Assign units and distributions.
- 3. Convert to output sensitivities.
- 4. Rank contributions.
Notice: The largest uncertainty—not the most visually impressive decimal place—sets the experiment's resolving power.
Check your understanding: Why does repeating a biased measurement not remove systematic error?
Answer: The bias shifts trials coherently rather than averaging randomly around truth.
An external calibration or changed method is needed.
Lesson 2 of 3
Model fitting, residuals, and look-elsewhere effects
Does the fitted parameter represent signal, model mismatch, or selection after seeing the data?
Parameter estimation compares data with a likelihood or loss model. Residual structure can reveal missing physics, drift, nonlinearity, or wrong noise assumptions.
Searching many windows, frequencies, or models inflates false-positive probability. The search scope must be included or confirmed on held-out data.
Worked example
A line fit has high R² but residuals curve systematically. Is the model adequate?
- 1. Inspect residual location and sign.
- 2. Identify nonrandom curvature.
- 3. Test a motivated alternative model.
- 4. Validate on new data.
High R² does not rescue a structurally wrong linear model.
Try it
Blind signal injection
Materials: Synthetic noisy data with hidden injected amplitudes.
- 1. Freeze detection rule before labels are revealed.
- 2. Analyze all files identically.
- 3. Reveal injections.
- 4. Measure false positives, misses, and bias.
Notice: A pipeline's behavior on known hidden truth is stronger evidence than tuning on one desired signal.
Check your understanding: What problem does a held-out dataset address?
Answer: It tests whether a model or threshold generalizes beyond the data used to choose it.
It reduces overfitting and selection bias.
Lesson 3 of 3
Controls, preregistration, and replication
What observation would isolate the proposed mechanism from thermal, mechanical, electromagnetic, and analytical artifacts?
Positive controls prove sensitivity; negative controls measure background; reversal and sham conditions test mechanism-specific direction or dependence.
Preregistration freezes primary outcomes and exclusions before results. Independent replication changes investigators and apparatus while preserving the decisive prediction.
Worked example
A cavity balance shows thrust only when powered. Name four confounds and discriminating controls.
- 1. Thermal expansion: matched heater sham.
- 2. Cable force: wireless or symmetric feed/reversal.
- 3. Magnetic coupling: shielding and field mapping.
- 4. Vibration/data selection: accelerometers and preregistered windows.
A credible design modulates the proposed signal independently of each major artifact.
Try it
Preregister a null experiment
Materials: One-page protocol template.
- 1. State primary prediction and falsifier.
- 2. Define sample size and exclusions.
- 3. List controls and calibration schedule.
- 4. Freeze analysis and data-release plan.
Notice: A preregistration makes later judgment auditable without forbidding exploratory follow-up.
Check your understanding: Why is repeating the same apparatus in the same lab weaker than independent replication?
Answer: Shared equipment, code, personnel, and assumptions can reproduce the same hidden artifact.
Independence deliberately changes potential failure modes.
Formula-to-meaning deck
Read the equation in ordinary language.
σ_f²≈JΣJᵀ
A sensitivity Jacobian and covariance matrix propagate correlated input uncertainty.
χ²=Σ[(y_i−m_i)/σ_i]²
Weighted squared residuals compare a model with uncertainty-scaled data.
SNR=signal scale/noise scale
Signal-to-noise is meaningful only after defining bandwidth, estimator, and noise model.
Independent practice
Problem set
Work each problem before opening its hint and solution.
1. Average four independent measurements each with σ=2. What is standard uncertainty of the mean?
Reveal hint
σ_mean=σ/√N.
Reveal solution
1.
2. Two independent 3σ searches are tried. Is the combined false-alarm probability still one-test 3σ?
Reveal hint
Multiple opportunities increase false alarms.
Reveal solution
No; correct for the two-test search or confirm on held-out data.
3. A result changes sign when apparatus orientation reverses and thermal direction does not. What does this establish?
Reveal hint
Consider discrimination, not final proof.
Reveal solution
It weakens orientation-invariant thermal explanations and supports a direction-linked effect, but other reversing systematics and replication remain.
Derivation studio
Build the result, line by line.
Keep the assumptions visible so the mathematics remains auditable.
Starting point
First-order uncertainty propagation
f(x₁,…,xₙ) expanded around measured means
- 1. Taylor expand f to first order.
- 2. Write δf≈ΣJᵢδxᵢ.
- 3. Square and take expectation.
- 4. Retain covariance terms.
Var(f)≈JΣJᵀ
Correlations can increase or decrease output uncertainty; independent quadrature is a special case.
Starting point
Weighted least-squares estimator
χ²(a)=Σ[(yᵢ−a xᵢ)/σᵢ]² for a zero-intercept model
- 1. Differentiate χ² with respect to a.
- 2. Set derivative to zero.
- 3. Collect weighted xᵢyᵢ and xᵢ² sums.
- 4. Solve for a.
â=Σ(xᵢyᵢ/σᵢ²)/Σ(xᵢ²/σᵢ²)
More precise points carry greater weight under the stated independent Gaussian model.
Computational notebook
Turn the model into an experiment.
Precision-force null-test pipeline
Can a preregistered analysis recover injected forces without mistaking drift, vibration, or heating for thrust?
Inputs
- • Synthetic balance time series
- • Temperature, accelerometer, power, and orientation channels
- • Hidden injection schedule
Algorithm
- 1. Calibrate baseline and drift model on training data.
- 2. Freeze filters and primary estimator.
- 3. Run blind analysis.
- 4. Reveal injections and compute detection, bias, and false-alarm metrics.
Evidence to produce
- • Uncertainty and covariance budget
- • Residual and diagnostic plots
- • Blind recovery table and reproducible analysis report
Continue into the evidence