Finland census data

statvar_imports/finland_census/README.md

@@ -1,18 +1,40 @@
 # Finland Demographics Dataset
-
 ## Overview
 
-This dataset contains demographic information from Finland sourced from Statistics Finland (Tilastokeskus). The data includes population statistics and census information from the official Finnish statistical database. 
+This dataset contains demographic information from Finland sourced from Statistics Finland (Tilastokeskus). This dataset provides comprehensive longitudinal coverage of Finland’s national demographics over a 34-year span, featuring annual data from 1990 to 2024. The geographic scope is standardized according to the January 1, 2025 regional division, ensuring consistency across the time series despite historical administrative changes. It offers high-resolution granularity through 43 unique statistical metrics that encompass population growth, age distribution, linguistic diversity, religious affiliation and urban-rural classification. Data is reported in multiple units for versatile analysis, including absolute counts, percentages (%), and population density (persons/km²), allowing for both scale-based and proportional statistical modeling.
+
+## Data Acquisition
+
+To download the latest version of this data or refresh the dataset for new years:
 
-## Data Source
+1. Navigate to the source: https://pxdata.stat.fi/PxWeb/pxweb/en/StatFin/StatFin__vaerak/statfin_vaerak_pxt_11ra.px/table/tableViewLayout1/
 
-**Source URL:** https://pxdata.stat.fi/PxWeb/pxweb/en/StatFin/StatFin__vaerak/statfin_vaerak_pxt_11ra.px/table/tableViewLayout1/
+2. Selection Criteria:
+    - Area: Select WHOLE COUNTRY (or all individual municipalities for more granularity).
+    - Information: Select all variables (Population, Language, Religion, Urban/Rural, etc.).
+    - Year: Select the full range from 1990 to the most recent year.
 
-The data comes from Finland's official statistical authority and includes comprehensive demographic variables such as population counts, age distributions, and other census-related metrics.
+3. Show and Save table Format: Choose "CSV (comma delimited)".
 
 ## Processing Instructions
 
 To process the Finland Census data and generate statistical variables, use the following command from the "data" directory:
 
 ```bash
 python ./data/tools/statvar_importer/stat_var_processor.py --input_data="./test_data/Finland_Census_input.csv" --pv_map=Finland_Census_pvmap.csv --config_file=Finland_Census_metadata.csv --output_path=Finland_Census_output
+
+## Data Refresh & Maintenance
+
+When Statistics Finland releases new annual updates (typically in the Spring), follow these steps:
+
+1. Execute the Data Acquisition steps to get the latest CSV.
+
+2. Check if new demographic categories were added. Update finland_census_pvmap.csv if new labels appear in the source.
+
+3. The source uses . to represent unavailable data (e.g., Economic dependency ratio for the current year). The processor is configured to skip these entries during import.
+
+4. Because the source applies the 2025 regional division retrospectively, check for municipal merger updates if downloading municipality-level data.
+
+5. Ensure total population counts match the previous year's trend to verify no rows were dropped.
+
+6. Execute the data processing step.

statvar_imports/finland_census/Finland_Census_metadata.csv → statvar_imports/finland_census/finland_census_metadata.csv

@@ -4,4 +4,5 @@ place_within,country/FIN
 mapped_rows,3
 mapped_columns,2
 word_delimiter,""""""
-output_columns,"observationDate,value,unit,observationAbout,variableMeasured"
+output_columns,"observationDate,value,unit,observationAbout,variableMeasured"
+

statvar_imports/finland_census/Finland_Census_pvmap.csv → statvar_imports/finland_census/finland_census_pvmap.csv

@@ -1,5 +1,5 @@
 key,p1,v1,p2,v2,p3,v3,p4,v4,p5,v5,p6,v6,p7,v7
-Population 31 Dec,,,measuredProperty,count,populationType,Person,,,,,,,,
+Population 31 Dec,,,measuredProperty,count,populationType,Person,,,,,,,,,,,
 Increase of population,,,measuredProperty,incrementalCount,populationType,Person,,,,,,,,
 "Increase of population, %",statType,growthRate,measuredProperty,count,populationType,Person,unit,Percent,,,,,,
 "Share of persons aged under 15, %",,,measuredProperty,count,populationType,Person,age,[- 15 Years],unit,Percent,measurementDenominator,Count_Person,,
@@ -29,9 +29,9 @@ Persons with foreign background,statType,measuredValue,measuredProperty,count,or
 "Land area, km²",statType,measuredValue,measuredProperty,area,,,populationType,LandCover,unit,SquareKilometer,,,,
 Population density,statType,measuredValue,measuredProperty,count,,,populationType,Person,unit,PersonPerSquareKilometer,measurementDenominator,area,,
 Persons born in the area belonging to the population,statType,measuredValue,measuredProperty,count,placeOfBirth,country/FIN,populationType,Person,,,,,,
-Persons born in their area of residence,statType,measuredValue,measuredProperty,count,placeOfBirth,BornInStateOfResidence,populationType,Person,,,,,,
-Persons born elsewhere than in their area of residence,statType,measuredValue,measuredProperty,count,placeOfBirth,NotBornInStateOfResidence,populationType,Person,,,,,,
-"Share of persons born in the area of residence, %",statType,measuredValue,measuredProperty,count,placeOfBirth,BornInStateOfResidence,populationType,Person,unit,Percent,measurementDenominator,Count_Person,,
+Persons born in their area of residence,statType,measuredValue,measuredProperty,count,placeOfBirth,BornInAreaOfResidence,populationType,Person,,,,,,
+Persons born elsewhere than in their area of residence,statType,measuredValue,measuredProperty,count,placeOfBirth,BornOutsideAreaOfResidence,populationType,Person,,,,,,
+"Share of persons born in the area of residence, %",statType,measuredValue,measuredProperty,count,placeOfBirth,BornInAreaOfResidence,populationType,Person,unit,Percent,measurementDenominator,Count_Person,,
 "Share of persons living in the area of birth, %",statType,measuredValue,measuredProperty,count,placeOfResidenceClassification,AreaOfBirth,populationType,Person,unit,Percent,measurementDenominator,Count_Person,,
 "Share of persons in urban areas, %",statType,measuredValue,measuredProperty,count,placeOfResidenceClassification,Urban,populationType,Person,unit,Percent,measurementDenominator,Count_Person,,
 "Share of persons in inner urban area, %",statType,measuredValue,measuredProperty,count,placeOfResidenceClassification,InnerUrban,populationType,Person,unit,Percent,measurementDenominator,Count_Person,,
@@ -433,4 +433,4 @@ MK16 Central Ostrobothnia,observationAbout,nuts/FI1D5,#Header,observationAbout,,
 MK17 North Ostrobothnia,observationAbout,nuts/FI1D6,#Header,observationAbout,,,,,,,,,,
 MK18 Kainuu,observationAbout,nuts/FI1D4,#Header,observationAbout,,,,,,,,,,
 MK19 Lapland,observationAbout,nuts/FI1D7,#Header,observationAbout,,,,,,,,,,
-MA2 ÅLAND,observationAbout,nuts/FI2,#Header,observationAbout,,,,,,,,,,
+MA2 ÅLAND,observationAbout,nuts/FI2,#Header,observationAbout,,,,,,,,,,

statvar_imports/finland_census/manifest.json

@@ -0,0 +1,25 @@
+{
+    "import_specifications": [
+        {
+            "import_name": "Finland_Census",
+            "curator_emails": [
+                "support@datacommons.org"
+            ],
+            "provenance_url": "https://pxdata.stat.fi/PxWeb/pxweb/en/StatFin/StatFin__vaerak/statfin_vaerak_pxt_11ra.px/table/tableViewLayout1/",
+            "provenance_description": "Population data for Finland",
+            "scripts": [
+                "../../tools/statvar_importer/stat_var_processor.py --input_data=gs://unresolved_mcf/country/finland/input_files/*.csv --pv_map=finland_census_pvmap.csv --config_file=finland_census_metadata.csv --output_path=output/finland_census_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+            ],
+            "import_inputs": [
+                {
+                    "template_mcf": "output/finland_census_output.tmcf",
+                    "cleaned_csv": "output/finland_census_output.csv"
+                }
+            ],
+            "source_files": [
+                "gs://unresolved_mcf/country/finland/input_files/*.csv"
+            ],
+            "cron_schedule": "30 5 1 *  *"
+        }
+    ]
+}