feat(inference): opt-in df_convention="cluster" knob for clustered t/p/CI

[igerber]

Summary

• Opt-in df_convention="cluster" inference-df knob on DifferenceInDifferences / TwoWayFixedEffects / MultiPeriodDiD (inherited via the DiD base class) and the LinearRegression linalg surface: clustered analytical t/p/CI at the Stata/fixest cluster df G − 1 instead of the fitted residual df (the documented clustered-CR1 inference-df deviation). Default stays "residual"; the default flips at v4 (tracked in TODO Deferred → Parked — it moves every clustered p-value/CI).
• Fallback-level only: survey df and per-coefficient Bell-McCaffrey DOF (hc2_bm) keep precedence; vcov_type="conley" excluded (no documented G − 1 reference for the Conley+cluster product kernel); inert on unclustered fits. Point estimates, SEs, and t-statistics are untouched — only the reference t-distribution changes.
• Result metadata: DiDResults/MultiPeriodDiDResults gain df_convention + inference_df (the effective df the reported t/p/CI actually used; set only when finite and strictly positive), surfaced in to_dict() only when set.
• Fail-closed hardening found by 7 local review rounds: G_eff ≤ 1 → NaN inference instead of silent normal-theory fallback (linalg + MPD twins); CR1 cluster validation counts positive-weight clusters for ALL weight types (the fweight carve-out let a degenerate one-effective-cluster CR1 SE through — pre-existing); missing (NaN/None) cluster labels rejected at a shared _validate_cluster_ids front door across the NumPy, Rust, and CR2-BM backends (the meat's groupby drops NaN-labelled rows, so no cluster count can agree with the sandwich's partition — silently wrong SEs, pre-existing; matches R fixest/Stata, which error on missing cluster values).
• New shared effective_cluster_count() (factorize + bincount, O(n)).

Methodology references (required if estimator / math changes)

• Method name(s): clustered CR1 inference df convention (residual df vs Stata/fixest G − 1).
• Paper / source link(s): REGISTRY §TwoWayFixedEffects clustered-CR1 inference-df deviation Note (extended with the knob, precedence, conley exclusion, and v4 flip).
• Any intentional deviations from the source (and why): the default remains the library's residual-df convention (documented deviation from fixest/Stata) until the v4 major-version flip; the knob provides exact opt-in parity, locked by t(G−1) tail-match tests at rtol 1e-12.

Validation

• Tests added/updated: tests/test_estimators_vcov_type.py::TestDfConvention (17 tests): exact t(G−1) p-value/CI match on DiD/TWFE/MPD incl. MPD avg_att; byte-identical no-op default; hc2_bm and survey-df precedence; conley exclusion; NaN-cluster-label rejection on every backend route (solve_ols/rust HC1/CR2-BM/MPD); fail-closed one-effective-cluster edges; transactional set_params validation; get_params round-trip. Both backends (rust + pure Python).
• Backtest / simulation / notebook evidence (if applicable): N/A.

Security / privacy

• Confirm no secrets/PII in this PR: Yes

---

Generated with Claude Code

[github-actions]

PR Review Report

Overall Assessment

✅ Looks good

Executive Summary

• Affected methods: clustered analytical inference for DifferenceInDifferences, TwoWayFixedEffects, MultiPeriodDiD, and LinearRegression.
• The df_convention="cluster" behavior matches the documented G - 1 Stata/fixest convention, while the residual-df default remains a documented REGISTRY deviation.
• Survey df and Bell-McCaffrey DOF precedence are preserved in the reviewed paths.
• Missing cluster labels and one-effective-cluster CR1 cases now fail closed rather than producing misleading SE/inference.
• No unmitigated P0/P1 methodology, inference, or propagation defects found.
• I could not run pytest because pytest is not installed in this environment; syntax parsing of changed Python files passed.

Methodology

• Severity: P3 informational
  Impact: The default clustered-CR1 inference df remains residual df rather than fixest/Stata G - 1, but this is explicitly documented as a deviation and the opt-in knob is documented in the registry.
  Concrete fix: None required. The v4 default flip is tracked in TODO.md.
  References: docs/methodology/REGISTRY.md:L362-L381, TODO.md:L115.

• Severity: None
  Impact: LinearRegression.get_inference() correctly resolves df precedence as explicit df, survey df, BM DOF, replicate sentinel, cluster G - 1, then residual fallback. The cluster branch only changes the reference distribution and leaves coefficient/SE/t-stat computation routed through safe_inference().
  Concrete fix: None.
  References: diff_diff/linalg.py:L4488-L4585.

• Severity: None
  Impact: MultiPeriodDiD mirrors the intended fallback-level behavior: the shared df becomes G - 1 only for clustered non-Conley analytical fits, while period and average inference still prefer BM contrast DOF when present.
  Concrete fix: None.
  References: diff_diff/estimators.py:L2089-L2124, diff_diff/estimators.py:L2295-L2349.

Code Quality

• Severity: None
  Impact: New df_convention parameter is validated in __init__, included in get_params(), and transactionally validated in set_params().
  Concrete fix: None.
  References: diff_diff/estimators.py:L214-L261, diff_diff/estimators.py:L1103-L1119, diff_diff/estimators.py:L1148-L1187.

Performance

• Severity: None
  Impact: effective_cluster_count() adds an O(n) factorize/bincount pass on clustered fits, which is appropriate for inference metadata and cluster-df validation.
  Concrete fix: None.
  References: diff_diff/linalg.py:L1530-L1555.

Maintainability

• Severity: None
  Impact: Cluster missing-label validation is centralized for linalg clustered vcov routes, and Conley retains its existing specialized validator.
  Concrete fix: None.
  References: diff_diff/linalg.py:L1514-L1527, diff_diff/linalg.py:L1823-L1836, diff_diff/linalg.py:L1995-L2041, diff_diff/conley.py:L745-L763.

Tech Debt

• Severity: P3 informational
  Impact: The remaining default flip to df_convention="cluster" is deferred and tracked, which is acceptable because it is a major-version behavior change affecting p-values/CIs.
  Concrete fix: None for this PR.
  Reference: TODO.md:L115.

Security

• Severity: None
  Impact: No secrets, unsafe IO, or security-sensitive changes found in the diff.
  Concrete fix: None.

Documentation/Tests

• Severity: None
  Impact: Docstrings, methodology registry, changelog, and LLM guide are updated for the new knob and precedence rules. New tests cover default no-op behavior, G - 1 tail matching, TWFE inheritance, MPD average/period effects, Conley exclusion, BM/survey precedence, invalid clusters, and params round-trip.
  Concrete fix: None.
  References: diff_diff/estimators.py:L136-L148, diff_diff/linalg.py:L3746-L3754, tests/test_estimators_vcov_type.py:L2759-L3096.

Verification note: pytest -q tests/test_estimators_vcov_type.py::TestDfConvention -p no:cacheprovider could not run because pytest is not installed. I did run AST parsing for the changed Python files successfully.

[github-actions]

🔁 AI review rerun (requested by @igerber)

Head SHA: 8a1b69773cc8bb6a4e43addc1766b91ed13f009d

---

PR Review Report

Overall Assessment

✅ Looks good

Executive Summary

• Affected methods: clustered analytical inference for DifferenceInDifferences, TwoWayFixedEffects, MultiPeriodDiD, and LinearRegression.
• The opt-in df_convention="cluster" path matches the documented Stata/fixest G - 1 convention.
• The residual-df default remains a documented REGISTRY deviation and the v4 default flip is tracked in TODO.md.
• Survey df and Bell-McCaffrey DOF precedence are preserved; Conley is excluded as documented.
• Missing cluster labels and one-effective-cluster CR1 cases now fail closed.
• Could not run pytest because pytest is not installed; AST parsing of changed Python files passed.

Methodology

• Severity: P3 informational
  Impact: The default clustered-CR1 inference df remains residual df rather than fixest/Stata G - 1, but this is explicitly documented as a REGISTRY deviation and the future default flip is tracked.
  Concrete fix: None required.
  References: docs/methodology/REGISTRY.md:L362-L381, TODO.md:L112-L115

• Severity: None
  Impact: LinearRegression.get_inference() applies df precedence correctly: explicit df, survey df, BM DOF, replicate sentinel, opt-in cluster G - 1, then residual fallback. It continues to route inference through safe_inference().
  Concrete fix: None.
  References: diff_diff/linalg.py:L4553-L4658

• Severity: None
  Impact: MultiPeriodDiD mirrors the intended fallback-level behavior for both period effects and average ATT: cluster df is used only for clustered non-Conley analytical fits, with survey and BM DOF taking precedence.
  Concrete fix: None.
  References: diff_diff/estimators.py:L2083-L2124, diff_diff/estimators.py:L2148-L2168, diff_diff/estimators.py:L2295-L2349

Code Quality

• Severity: None
  Impact: df_convention is validated in __init__, included in get_params(), handled transactionally in set_params(), and propagated through DiD/TWFE regression calls and results metadata.
  Concrete fix: None.
  References: diff_diff/estimators.py:L214-L261, diff_diff/estimators.py:L650-L667, diff_diff/twfe.py:L518-L557, diff_diff/estimators.py:L1103-L1119, diff_diff/estimators.py:L1148-L1187

Performance

• Severity: None
  Impact: effective_cluster_count() adds one O(n) factorize/bincount pass on clustered fits for inference metadata and df validation. This is appropriate and not a performance concern.
  Concrete fix: None.
  References: diff_diff/linalg.py:L1530-L1555

Maintainability

• Severity: None
  Impact: Missing cluster-label validation is centralized for the clustered NumPy, Rust, and CR2-BM paths, reducing backend-specific drift.
  Concrete fix: None.
  References: diff_diff/linalg.py:L1514-L1527, diff_diff/linalg.py:L627-L630, diff_diff/linalg.py:L1830-L1836, diff_diff/linalg.py:L1997-L2000, diff_diff/linalg.py:L3094-L3112

Tech Debt

• Severity: P3 informational
  Impact: The remaining default flip to df_convention="cluster" is deferred and tracked as a major-version behavior change affecting clustered p-values/CIs.
  Concrete fix: None for this PR.
  Reference: TODO.md:L112-L115

Security

• Severity: None
  Impact: No secrets, unsafe IO, or security-sensitive changes found in the reviewed diff.
  Concrete fix: None.

Documentation/Tests

• Severity: None
  Impact: Docstrings, REGISTRY, changelog, LLM guide, and focused tests cover the new knob, precedence rules, Conley exclusion, metadata, invalid cluster labels, one-effective-cluster guards, and parameter round-trip behavior.
  Concrete fix: None.
  References: diff_diff/estimators.py:L136-L148, diff_diff/linalg.py:L3811-L3819, tests/test_estimators_vcov_type.py:L2761-L3098

Verification: pytest -q tests/test_estimators_vcov_type.py::TestDfConvention -p no:cacheprovider could not run because pytest is not installed. AST parsing passed for diff_diff/estimators.py, diff_diff/linalg.py, diff_diff/results.py, diff_diff/twfe.py, and tests/test_estimators_vcov_type.py.