Add enhanced FRS (wealth + consumption imputation) and VAT (#8)

* Add enhanced FRS (wealth + consumption imputation), VAT, and fix state pension scaling

Ports the EFRS pipeline from policyengine-uk-data to Rust: trains random forest models on WAS (wealth) and LCFS (consumption) survey data, predicts onto FRS households, and calibrates energy consumption to NEED 2023 targets. Adds a household-level VAT calculation (standard/reduced/zero rates by COICOP category) wired into net income, enabling distributional analysis of consumption tax reforms.

Fixes state pension reform responsiveness: reported SP amounts are now scaled by reform_rate/baseline_rate so that changing the SP parameter actually affects household net income. Previously reported amounts were passed through verbatim, making SP reforms have near-zero distributional impact.

New CLI flags: --extract-efrs, --was-dir, --lcfs-dir.

Co-Authored-By: Claude <noreply@anthropic.com>

* Move HICBC to income tax charge, add EFRS auto-detect default

HICBC is now correctly modelled as an income tax charge on the highest
earner (Phase 2b of simulation) rather than a child benefit reduction.
Child benefit is paid in full; the charge is tapered between the HICBC
threshold and taper end based on adjusted net income.

Also adds auto-detection of data/efrs_clean (preferred) or data/frs_clean
when no explicit data source is specified on the CLI.

Co-Authored-By: Claude <noreply@anthropic.com>

* Fix state pension (new vs basic SP), add fiscal_year to Simulation, add efrs to Python DATASETS

State pension now matches policyengine-uk approach: new SP recipients (reached SP age
after April 2016) get the full parameter rate directly; basic SP recipients get reported
amounts scaled by reform ratio. Also adds fiscal_year parameter to Simulation constructor
and adds "efrs" to the Python interface's DATASETS tuple for GCS download support.

Co-Authored-By: Claude <noreply@anthropic.com>

* State pension flows through to income tax: reorder simulation phases

Calculate per-person state pension in Phase 1a before income tax in
Phase 1b, so SP reforms correctly affect taxable income. Previously
income tax used reported SP amounts regardless of reform parameters,
causing £0 revenue change for SP reforms.

Also fixes HMAC download truncation in Python data module (adds
content-length verification and timeout).

Co-Authored-By: Claude <noreply@anthropic.com>

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: nikhilwoodruff

Cargo.lock

@@ -15,6 +15,8 @@ indicatif = "0.17"
 comfy-table = "7"
 colored = "2"
 anyhow = "1"
+smartcore = "0.4"
+rand = "0.8"
 
 [profile.release]
 opt-level = 3

Cargo.toml

@@ -37,13 +37,21 @@ def _download_object(key: str, dest: Path, access_key: str, secret_key: str):
     url = f"https://{GCS_HOST}/{GCS_BUCKET}{path}"
     req = urllib.request.Request(url, headers=headers)
     dest.parent.mkdir(parents=True, exist_ok=True)
-    with urllib.request.urlopen(req) as resp:
+    with urllib.request.urlopen(req, timeout=300) as resp:
+        expected = int(resp.headers.get("Content-Length", 0))
+        written = 0
         with open(dest, "wb") as f:
             while True:
                 chunk = resp.read(1 << 20)
                 if not chunk:
                     break
                 f.write(chunk)
+                written += len(chunk)
+        if expected and written != expected:
+            dest.unlink(missing_ok=True)
+            raise IOError(
+                f"Incomplete download for {key}: got {written} of {expected} bytes"
+            )
 
 
 def _get_credentials() -> tuple[str, str]:
@@ -60,7 +68,7 @@ def _get_credentials() -> tuple[str, str]:
     return token.split(":", 1)
 
 
-DATASETS = ("frs", "lcfs", "spi", "was")
+DATASETS = ("frs", "efrs", "lcfs", "spi", "was")
 
 
 def ensure_dataset_year(dataset: str, year: int) -> Path:

interfaces/python/policyengine_uk_compiled/data.py

@@ -130,6 +130,12 @@ uc_migration:
   tax_credits: 0.0
   income_support: 0.0
 
+
+vat:
+  standard_rate: 0.175
+  reduced_rate: 0.05
+  zero_rate: 0.0
+
 growth_factors:
   cpi_rate: 0.0
   gdp_deflator: 0.0

parameters/1994_95.yaml

@@ -130,6 +130,12 @@ uc_migration:
   tax_credits: 0.0
   income_support: 0.0
 
+
+vat:
+  standard_rate: 0.175
+  reduced_rate: 0.05
+  zero_rate: 0.0
+
 growth_factors:
   cpi_rate: 0.0
   gdp_deflator: 0.0

parameters/1995_96.yaml

@@ -130,6 +130,12 @@ uc_migration:
   tax_credits: 0.0
   income_support: 0.0
 
+
+vat:
+  standard_rate: 0.175
+  reduced_rate: 0.05
+  zero_rate: 0.0
+
 growth_factors:
   cpi_rate: 0.0
   gdp_deflator: 0.0

parameters/1996_97.yaml

@@ -130,6 +130,12 @@ uc_migration:
   tax_credits: 0.0
   income_support: 0.0
 
+
+vat:
+  standard_rate: 0.175
+  reduced_rate: 0.05
+  zero_rate: 0.0
+
 growth_factors:
   cpi_rate: 0.0
   gdp_deflator: 0.0

parameters/1997_98.yaml

@@ -130,6 +130,12 @@ uc_migration:
   tax_credits: 0.0
   income_support: 0.0
 
+
+vat:
+  standard_rate: 0.175
+  reduced_rate: 0.05
+  zero_rate: 0.0
+
 growth_factors:
   cpi_rate: 0.0
   gdp_deflator: 0.0

parameters/1998_99.yaml

@@ -130,6 +130,12 @@ uc_migration:
   tax_credits: 0.0
   income_support: 0.0
 
+
+vat:
+  standard_rate: 0.175
+  reduced_rate: 0.05
+  zero_rate: 0.0
+
 growth_factors:
   cpi_rate: 0.0
   gdp_deflator: 0.0

parameters/1999_00.yaml

@@ -130,6 +130,12 @@ uc_migration:
   tax_credits: 0.0
   income_support: 0.0
 
+
+vat:
+  standard_rate: 0.175
+  reduced_rate: 0.05
+  zero_rate: 0.0
+
 growth_factors:
   cpi_rate: 0.0
   gdp_deflator: 0.0