DIRD critical-reading seminar
Turn a provocative reference study into a traceable map of sources, assumptions, forecasts, and testable claims.
Audit Defense Intelligence Reference Documents and their cited literature through provenance, citation networks, equation reconstruction, forecast scoring, evidence-tier comparison, red-team review, and a publication-ready critical dossier.
Before you begin
- • Level 5 graduate pathway
- • Research methods and statistics
- • Technical literature-search fluency
By the end, you can
- • Reconstruct a DIRD's scope, provenance, and commissioned purpose.
- • Trace every central claim to primary evidence or an explicit inference.
- • Score forecasts against dated outcomes without hindsight drift.
- • Publish a reproducible, adversarially reviewed evidence dossier.
Interactive model
Explore before calculating
Live research workspace
DIRD claim graph
Select a claim to inspect what anchors it, what challenges it, and which observation would materially change its status.
Evidence anchors
- • Spectroscopy and radiative shifts
- • Boundary-dependent quantum forces
The quantum vacuum has measurable structure
Challenges
- • Interpretation is not unique
- • Absolute energy is renormalization-dependent
The quantum vacuum has measurable structure
establishedGround-state quantum fields contribute observable response and fluctuation phenomena even when no ordinary particles are present.
What is established
Lamb shifts, spontaneous emission, vacuum polarization, and boundary-dependent forces are quantitatively described by quantum field theory.
What remains
Those observations do not select one philosophical vacuum picture or prove that the ground state is an extractable work reservoir.
Deciding evidence: No single new test is needed for quantum-vacuum structure; proposed engineering consequences each require their own complete-cycle measurement.
Chapter 2: What is the vacuum? →Field ground states, observables, and interpretation boundaries.
Level 6 · Research preparation teaching kit
Record the investigation. Teach the reasoning.
A learner-facing lab record and a course-specific instructor guide turn the live model into a repeatable classroom investigation.
Learner record
DIRD claim-provenance and forecast audit
Which claims are directly sourced, which are inferences, and which dated forecasts survived contact with later evidence?
Download learner recordInstructor guide
Teach for evidence, not button pushing
Researchers reconstruct provenance, separate source from inference, and score dated forecasts without personality-based deference or hindsight leakage.
Download instructor guideAdvanced assessment
Reconstruct it. Quantify it. Try to break it.
Reconstruct claims from dated sources and score forecasts without letting later knowledge leak backward. Three research-level challenges include explicit deliverables and scoring criteria.
Portable research dataset
Record data that another laboratory can open.
Dated source, inference, forecast, and confidence records. JSON preserves schema and provenance; CSV supports ordinary analysis tools. Imports stay in this browser and are limited to 1 MB and 5,000 records.
Ready for a new research record.
| Claim IDidentifier | Source dateISO date | Relationshiplabel | Evidence cutoffISO date | Forecast scoredimensionless | Confidence tierlabel | Record |
|---|---|---|---|---|---|---|
Schema field definitions
- Claim ID · identifier
- Stable claim identifier.
- Source date · ISO date
- Publication or evidence date.
- Relationship · label
- Direct, inferred, contextual, or contradictory.
- Evidence cutoff · ISO date
- Latest admissible evidence date.
- Forecast score · dimensionless
- Prespecified forecast calibration score.
- Confidence tier · label
- Evidence tier after the audit.
Lesson 1 of 3
Provenance, purpose, and document anatomy
What was the document commissioned to do, and what was it not designed to establish?
A reference study may survey possibilities, assess strategic relevance, or identify research gaps. Its existence, sponsor, classification history, or expert authorship does not convert every discussed proposition into a confirmed result.
Critical reading begins with date, version, authorship, commissioning question, intended audience, methods, cited time horizon, and the distinction between reported evidence and scenario analysis.
Worked example
A study says a technology could be transformative if three breakthroughs occur. How should that sentence be encoded?
- 1. Mark it as conditional.
- 2. Extract the three dependencies.
- 3. Separate impact forecast from probability.
- 4. Record the study date and forecast horizon.
It is a conditional scenario, not evidence that the technology or prerequisites existed.
Try it
Document anatomy ledger
Materials: One DIRD and metadata sheet
- 1. Record provenance and purpose.
- 2. Map section types.
- 3. Tag claims as report, synthesis, inference, or forecast.
- 4. List explicit exclusions and unknowns.
Notice: Documents often change epistemic mode across sections; one global label loses that structure.
Check your understanding: Does government commissioning establish that a discussed mechanism is true?
Answer: No.
It establishes institutional interest in assessment, not experimental confirmation of every claim.
Lesson 2 of 3
Claim graphs, citation chains, and equation reconstruction
Where does each quantitative statement originate, and does the cited source support the same proposition?
Citation laundering occurs when reviews cite one another until a conditional or disputed statement appears settled. A claim graph records the exact proposition, source location, evidence type, and transformations between documents.
Equations require unit, assumption, boundary, and limiting-case checks. Reproducing one central result often reveals whether a claim is a derivation, parameter choice, analogy, or unsupported extrapolation.
Worked example
Three reviews cite one another for the same efficiency value but none reports data. What is the evidence status?
- 1. Trace the oldest citation.
- 2. Search for primary apparatus or calculation.
- 3. Record that the value is repeated but unsupported.
- 4. Do not count repeated reviews as independent evidence.
The citation network has recurrence, not independent empirical confirmation.
Try it
Five-claim graph
Materials: DIRD, references, and graph template
- 1. Extract five central claims verbatim.
- 2. Trace each to the earliest accessible source.
- 3. Label edge type and confidence.
- 4. Reproduce one numerical result.
Notice: The strongest-looking claim may rest on the weakest or most transformed citation chain.
Check your understanding: When do two papers count as independent support?
Answer: When their evidence or derivations are genuinely independent rather than reusing the same source, apparatus, or dataset.
Bibliographic plurality alone is insufficient.
Lesson 3 of 3
Forecast scoring, red teams, and dossier publication
Which expectations survived time, and what would a fair critic and advocate each say now?
Forecasts must be frozen in their original wording and scored against dated outcomes using predeclared categories. Vague relevance claims should not be upgraded into successful quantitative predictions.
A red team identifies hidden assumptions, conventional explanations, and missing controls. A steelman reconstructs the strongest version of the program. The final dossier preserves both and names the evidence that would move confidence.
Worked example
How should a 2010 forecast of a laboratory demonstrator within ten years be scored in 2026 if only simulations exist?
- 1. Freeze the original milestone definition.
- 2. Search for a physical demonstrator by the deadline.
- 3. Do not substitute simulations unless allowed originally.
- 4. Score unmet and document partial progress separately.
The demonstrator forecast is unmet; simulation progress is a separate outcome.
Try it
Adversarial dossier review
Materials: Draft dossier and two reviewer roles
- 1. Assign advocate and critic.
- 2. Require source-located objections.
- 3. Resolve factual disagreements.
- 4. Publish remaining judgment differences explicitly.
Notice: Adversarial review improves a dossier when disagreements are made inspectable rather than averaged away.
Check your understanding: What makes a forecast score reproducible?
Answer: Frozen wording, dated evidence, explicit categories, and a documented scoring rule.
Otherwise hindsight can redefine success.
Formula-to-meaning deck
Read the equation in ordinary language.
P(claim|evidence)∝P(evidence|claim)P(claim)
Evidence updates a claim in proportion to how differently competing explanations predict it.
Brier=(1/N)Σ(p_i−o_i)²
The Brier score measures probabilistic forecast calibration against binary outcomes.
w_source=w_primary·w_independence·w_method
A transparent heuristic weight can record provenance, independence, and methodological strength without pretending to be a universal truth score.
Independent practice
Problem set
Work each problem before opening its hint and solution.
1. Two papers reuse one experiment. How many independent empirical datasets support the claim?
Reveal hint
Count evidence sources, not publications.
Reveal solution
One.
2. A forecast assigned p=0.8 fails. What is its single-item Brier score?
Reveal hint
Use (p−o)² with o=0.
Reveal solution
0.64.
3. Why must an equation reproduction retain the source's boundary conditions?
Reveal hint
Consider whether the solution is unique or applicable without them.
Reveal solution
Boundary conditions select the physical solution and can control the result's magnitude or even sign.
Derivation studio
Build the result, line by line.
Keep the assumptions visible so the mathematics remains auditable.
Starting point
Bayes-factor evidence update
Posterior odds = prior odds × likelihood ratio
- 1. Write odds for claim versus alternative.
- 2. Compute likelihood of the observation under each.
- 3. Form their ratio.
- 4. Multiply without changing the frozen prior after seeing the data.
O_post=O_prior·P(E|H₁)/P(E|H₀)
Evidence matters through discrimination, not drama or institutional prestige.
Starting point
Forecast calibration decomposition
Mean squared probability error over binned forecasts
- 1. Group forecasts by stated probability.
- 2. Compute observed frequency per bin.
- 3. Separate calibration from resolution and base-rate uncertainty.
- 4. Inspect sample-size limits.
Brier performance reflects calibration, resolution, and outcome uncertainty
A forecast program should be judged across many frozen predictions, not one celebrated hit.
Computational notebook
Turn the model into an experiment.
Claim-graph and forecast audit notebook
Which claims in a selected DIRD have independent support, reproducible calculations, and calibrated outcomes?
Inputs
- • Document text and reference list
- • Primary-source metadata
- • Frozen forecasts and dated outcome evidence
Algorithm
- 1. Extract atomic claims and citation edges.
- 2. Deduplicate underlying evidence.
- 3. Reproduce selected calculations.
- 4. Score forecasts and confidence updates.
Evidence to produce
- • Machine-readable claim graph
- • Reproduction notebook
- • Forecast scorecard and evidence dossier
Paper-reading studio
Interrogate the source, not its reputation.
Reconstruct the assumptions, reproduce one calculation, and stop at the boundary of the reported evidence.
Reference-study critical dossier
What would remain credible if the sponsor, author reputation, and later retellings were removed?
- 1. Reconstruct purpose and provenance.
- 2. Trace claims to primary evidence.
- 3. Reproduce the central quantitative step.
- 4. Score forecasts and unresolved assumptions.
Calculation to reproduce: Reproduce the study's most decision-relevant estimate with a sensitivity analysis.
Evidence boundary: A DIRD can be valuable strategic scholarship while containing established, contested, speculative, and outdated claims that must be graded individually.
Graduate oral defense
Defend a bounded claim under pressure.
Argue the strongest support, state the strongest objection fairly, and identify evidence that could actually decide the issue.
Proposition
DIRDs should be taught as auditable research maps, not dismissed as fantasy or treated as revealed truth.
- 1. They assemble uncommon technical literature and strategic questions.
- 2. Their conditional forecasts and source networks can generate testable programs.
- 3. Claim-level grading preserves value without laundering uncertainty.
Strongest objection: Commissioning context and selective topic framing may amplify low-quality or strategically motivated speculation.
Deciding evidence: Independent reproduction of central calculations and prospective success of preregistered discriminating predictions derived from the documents.
Research practicum
Make the work inspectable before making it impressive.
Pre-register the decisive test, package every dependency, and pass explicit milestone gates before interpretation expands.
Publish an open DIRD audit
Which three claims most deserve reproduction or decisive testing now?
Preregister
- • Freeze claim definitions and inclusion rules.
- • Define source-independence and forecast scoring.
- • Name confidence-change thresholds.
Reproducibility package
- • Document/version manifest and hashes
- • Claim graph with source locations
- • Calculation notebooks and outcome dataset
- • Reviewer disagreements and change log
Milestone gates
- 1. Provenance and primary-source audit
- 2. Independent calculation reproduction
- 3. Adversarial review resolution
- 4. Public dossier with machine-readable appendix
Continue into the evidence