Populace
How the dataset is built
The full US pipeline as implemented in PolicyEngine/populace (derived from main @ 98d990b). Build-step ids match the staging telemetry, so a running build on the Staging runs page walks these exact steps.
Flow
Left to right: survey sources feed the base H5; Ledger facts and references compile the target surface; materialization and calibration produce the dataset and its diagnostics; staging streams live; publish ships to Hugging Face.
Source enrichment (upstream of a refresh build)
Imputation stages declared in source_stages.json — each grafts variables from another survey onto the CPS base. They are baked into the base H5 and are NOT re-run by a fiscal-refresh build.
puf_tax_detail
IRS PUF 2015 (uprated)
employment_income_before_lsr, self_employment_income_before_lsr, taxable_interest_income, dividend_income, capital gains & other tax detail…
scf_wealth
Fed SCF 2022
net_worth, scf_* assets & debts (housing, retirement, business…)
sipp_tips
Census SIPP
tip_income
org_wages
CPS ORG
hourly_wage, is_paid_hourly, is_union_member_or_covered, fsla_overtime_premium
meps_esi_premiums
MEPS-IC
employer_sponsored_insurance_premiums
prior_year_income
CPS ASEC (prior year)
employment_income_last_year, self_employment_income_last_year
mortgage_conversion
IRS PUF 2015 (uprated)
deductible_mortgage_interest, home_mortgage_interest, mortgage_principal…
acs_rent
Census ACS 2022
pre_subsidy_rent
vehicle_assets
Census SIPP
household_vehicles_owned, household_vehicles_value
aca_marketplace_inputs
CPS ASEC + CMS Marketplace
takes_up_aca_if_eligible, selected_marketplace_plan_benchmark_ratio
1
Inputs
A fiscal-refresh build starts from the previous release and recalibrates household weights — it does not re-run the source enrichment stages above.
Prior release H5
base_h5build_us_fiscal_refresh_release._download_base_h5
policyengine/populace-us · populace_us_2024.h5 — the previous release is the base population (records, imputations, and structure are inherited; only weights are refit).
Ledger facts
ledger_facts--ledger-facts consumer_facts.jsonl
Sourced official statistics (IRS SOI, Census PEP/STC, CMS, USDA, HHS, SSA, CBO, JCT) exported from the Ledger. These carry the target values.
Value-free target references
target_referencespopulace/build/us/fiscal_target_references.json
Declares which target rows exist and how each maps to model variables; joined with Ledger facts to obtain values.
Validation configs
validation_configsobbba_reforms.json · tax_expenditure_reforms.json · soi_baseline_levels.json
Out-of-sample benchmark sets: OBBBA provisions vs JCT (stacked, per measure group), tax-expenditure repeals, and SOI baseline levels.
2
Target compilation
References × Ledger facts become the calibration target surface.
Compile target registry
target_compilationpopulace.build.us_runtime.fiscal_targets.compile_us_fiscal_target_registry
Joins the value-free references with Ledger fact values into ~6,900 TargetSpecs across 11 source families (national, state, and congressional-district levels).
Target-profile coverage gate
target_profile_gatetarget_profile_coverage_gate
Refuses to build if required target coverage (per-family requirements, e.g. each JCT tax-expenditure reform) is missing from the compiled surface.
3
Base frame preparation
Load the weighted sampling frame and repair known mass issues before calibration.
Load base frame
load_base_framepopulace.frame.Frame ← H5
The prior release's entity tables become a Frame (weighted records with links).
Population mass repair + gate
base_population_repair_with_base_population_mass_repair · _base_population_scale_gate
Rescales household weights to the Census national person-population benchmark, then gates that total population is within tolerance.
Social Security component repair
social_security_component_repair_with_social_security_component_value_repair
Repairs SS component value support so SSA component targets are materializable.
Health input gate
health_input_gate_health_input_signal_gate
Verifies health coverage inputs carry signal before calibrating health targets.
4
Target materialization
Every target becomes a computable column over the frame.
Materialize the target matrix
target_registry_load_or_materialize_target_frame (batched policyengine-us microsimulations)
Runs PE-US in batches to compute each target's per-record contribution (the sparse constraint matrix). Cached / checkpointable; congressional-district targets are support-gated.
5
Calibration
Reweight households so the frame reproduces the target surface.
Calibrate weights
calibratingpopulace.calibrate.calibrate / calibrate_l0_refit
Torch on log-weights: capped weighted-MAPE loss, mass=conserve, hard max-weight-ratio guard (5×). Optional L0 sparse-support refit. The loss trajectory streams to staging telemetry live.
Release gates
release_gatespost-calibration gates
Post-calibration checks (fit, weights, coverage) recorded in the build manifest.
6
Export & release artifacts
The dataset plus every diagnostics artifact the dashboard reads.
Export dataset
export_datasetPE-US H5 writer (round-trip verified)
Writes populace_us_2024.h5 with the calibrated weights.
populace_us_2024.h5
Calibration package
write_calibration_npz_write_npz
Weights, targets, and estimates for reproducibility.
populace_us_2024_calibration.npz
Post-export audit (opt-in)
post_export_audit--audit-export-targets
Slow audit that re-materializes targets from the exported H5; default builds rely on the writer round-trip check instead.
Reform validation
reform_validationpopulace.build.us_runtime.reform_validation
Out-of-sample tests: OBBBA provisions stacked per measure group vs JCT (FY2026 + FY2027), tax-expenditure repeals, and SOI baseline levels (one shared baseline sim).
reform_validation.json
Demographics
demographicspopulace.build.us.demographics
Age-distribution snapshot vs the Census benchmark.
demographics.json
Source coverage + diagnostics
source_coveragepopulace.calibrate.diagnostics.write_calibration_diagnostics
Per-target fit for every calibration target, plus the source-coverage map.
calibration_diagnostics.jsonus_source_coverage.json
Manifests
manifests_build_manifests
Build manifest (code SHA, timing, gates, staging run id) and release manifest (artifact SHAs, compatible package versions).
build_manifest.jsonrelease_manifest.json
7
Staging (live, on by default)
Telemetry streams to the staging repo while the build runs.
Staging telemetry
staging_telemetrypopulace.build.staging.StagingTelemetry → policyengine/populace-us-staging
Progress, events, the calibration loss curve, and every artifact above upload live (best-effort, never fails a build). Candidates are reviewable on this dashboard — the Staging runs page, and as 'candidate' entries in Target diagnostics and Reform validation — before publishing.
8
Publish
A guarded, manual step on the build machine.
populace-publish-release
publish_releasepopulace.data.publish_cli (tools/publish_release.sh wrapper)
Refuses releases with unsimulated reform validation; warns when staging telemetry is missing; uploads artifacts, creates the immutable HF tag, flips latest.json, and fires the Slack release alert.
Inputs external validation found missing
Cross-referenced against source_stages.json at the commit above: variables flagged by the reform/SOI validation are exactly the ones no enrichment stage declares as an output.
qualified_tuition_expenseseducation credits validate ~40% low · issue ↗qualified_passenger_vehicle_loan_interestOBBBA auto-loan deduction is structurally $0 · issue ↗has_valid_ssn / immigration statusSSN- and immigration-conditioned policy is a no-op · issue ↗childcare expenses (CDCC inputs)CDCC validates ~31% low — under investigation