Strictly proper joint score
A weighted energy-distance statistic scores the joint distribution, not just marginals — the block that catches a fit that gets every variable's shape right in isolation but the relationships between them wrong.
strategies / evaluation
strategy 04 · evaluation
The referee.
A synthetic population can be checked against two different things, and passing one says nothing about the other: does it still look like the surveys that observe it, and does it hit the administrative totals it was calibrated to. populace scores both, separately, rather than blending them into one number.
01 — the 30-second version
Two currencies, never blended.
Survey-view fidelity asks whether a candidate file, projected through a survey's own variables and sampling design, still resembles that survey's holdout — scored by the popdgp harness, calibration-blind by construction so it cannot be gamed by fitting targets harder. Calibration-target attainment asks whether the file hits the administrative totals it was reweighted against — in-fit today, held-out families as a release gate once the masking machinery below ships. Neither currency substitutes for the other: a file can nail every target while its weighted joint drifts from the surveys, or track the surveys closely while missing targets it was never calibrated to.
survey-view classifier floor
0.50AUC, indistinguishable
a survey's own holdout split scored as a candidate
tail block, weight-blind fit caught
1.9–2.3×the holdout's p99
energy distance & coverage both tied — only the tail block sees it
calibration targets within 10%
94.7%in-fit, current release
3,704 targets · held-out family gating not yet live
Survey-view numbers from imputation-paper's population-view harness (paper/tables/harness_scale.tex); calibration-target numbers from the live release registry (populace-us-2024-5da5a95). Held-out target-family gating and reweight-fragility promotion metrics are proposed release gates (populace#302, populace-benchmarks#6), referenced below — not yet running against certified builds.
02 — currency one
Survey-view support fidelity.
One latent population is observed through each survey's view: its own variable subset, sampling design, and measurement idiom. Any candidate file gets projected through a view and scored against that survey's own holdout — never against the file's own training data.
Calibration-blind by construction
A weighted PRDC coverage statistic on support geometry, invariant to reweighting of the candidate — deliberately blind to whatever calibration did, so it measures the imputation and synthesis layers, not the reweight.
Two-sample test
A weighted classifier trained to separate candidate from holdout; 0.5 AUC means indistinguishable. Catches joint failures that tie the candidate on energy distance — an unconditional weighted draw that discards every predictor still ties on energy distance yet is separated almost perfectly here (AUC 0.968).
Uncapped, added after a near-miss
Per-variable weighted Wasserstein-1/sd and q90/q99 ratios on the full weighted sample, no subsampling cap. Added after energy distance, coverage, and classifier AUC all tied a candidate whose imputed net worth 99th percentile ran 1.9–2.3× the holdout's — capped pairwise geometry under-resolves heavy right tails; this block does not.
every block anchored by a sampling-noise floor — the survey's own complementary split, scored as if it were a candidate
popdgp — the harness, as a standalone package (extraction in progress) · working reference implementation and all current results live in imputation-paper
03 — currency two
Administrative target attainment.
The calibration strategy page reports the live in-fit numbers: 3,704 targets, 94.7% within 10% of their administrative value. Two things that page does not yet report are proposed as release gates, reusing the population-view harness's methodology rather than inventing a parallel one:
Holdout-masked builds
Reserve a seeded holdout from each source survey before any support-pool or donor fitting, so certified bundles can be scored non-self-referentially — the same masking discipline the survey-view harness already uses, applied to the target surface. Tracked as populace#302, not yet built.
Worst-case concentration under bounded reweighting
A closed-form stress test — the worst-case share of an aggregate a single record could carry under bounded multiplicative reweighting (κ=5) — compared against the reference file's own fragility. Gaussian-residual imputation understates true exposure by 7–11× under this test. Tracked as populace-benchmarks#6, not yet built.
both proposals cite imputation-paper's harness as the reference implementation to promote, not a new method to invent