Automation of import Finland_Census

statvar_imports/finland_census/README.md

@@ -1,40 +1,68 @@
 # Finland Demographics Dataset
 ## Overview
 
-This dataset contains demographic information from Finland sourced from Statistics Finland (Tilastokeskus). This dataset provides comprehensive longitudinal coverage of Finland’s national demographics over a 34-year span, featuring annual data from 1990 to 2024. The geographic scope is standardized according to the January 1, 2025 regional division, ensuring consistency across the time series despite historical administrative changes. It offers high-resolution granularity through 43 unique statistical metrics that encompass population growth, age distribution, linguistic diversity, religious affiliation and urban-rural classification. Data is reported in multiple units for versatile analysis, including absolute counts, percentages (%), and population density (persons/km²), allowing for both scale-based and proportional statistical modeling.
+This dataset contains demographic information from Finland sourced from Statistics Finland (Tilastokeskus). This dataset provides comprehensive longitudinal coverage of Finland’s national demographics over a 34-year span, featuring annual data from 1990 to 2025. The geographic scope is standardized according to the January 1, 2025 regional division, ensuring consistency across the time series despite historical administrative changes. It offers high-resolution granularity through 43 unique statistical metrics that encompass population growth, age distribution, linguistic diversity, religious affiliation and urban-rural classification. Data is reported in multiple units for versatile analysis, including absolute counts, percentages (%), and population density (persons/km²), allowing for both scale-based and proportional statistical modeling.
 
-## Data Acquisition
+**type of place:** Country
+**years:** 1990 to 2025
 
-To download the latest version of this data or refresh the dataset for new years:
+## Data Source
+**Source URL:**
+https://pxdata.stat.fi/PxWeb/pxweb/en/StatFin/StatFin__vaerak/11ra.px
 
-1. Navigate to the source: https://pxdata.stat.fi/PxWeb/pxweb/en/StatFin/StatFin__vaerak/statfin_vaerak_pxt_11ra.px/table/tableViewLayout1/
+## License
+**License Type:** 
+Creative Commons Attribution 4.0 International
+**License URL:** 
+https://creativecommons.org/licenses/by/4.0/
+**License Description:**
+The [Statistics Finland Terms of Use](https://stat.fi/en/about-us/get-to-know-statistics-finland/legislation/terms-of-use) state the following:
+"Statistics Finland's open data materials and public content of the web service are covered by the Creative Commons Attribution 4.0 International licence. According to the licence, you can copy, edit and share these data either in original or edited format. You can also combine the data with other data and use the data for commercial purposes as well. This licence applies to texts, tables and statistical graphs."
 
-2. Selection Criteria:
-    - Area: Select WHOLE COUNTRY (or all individual municipalities for more granularity).
-    - Information: Select all variables (Population, Language, Religion, Urban/Rural, etc.).
-    - Year: Select the full range from 1990 to the most recent year.
+## Refresh Type
+Automatic Refresh
 
-3. Show and Save table Format: Choose "CSV (comma delimited)".
+The refresh is automated using the provided `run.sh` script, which handles both data download and processing.
 
 ## Processing Instructions
 
-To process the Finland Census data and generate statistical variables, use the following command from the "data" directory:
-
+To execute the complete import process (download and processing), run:
 ```bash
-python ./data/tools/statvar_importer/stat_var_processor.py --input_data="./test_data/Finland_Census_input.csv" --pv_map=Finland_Census_pvmap.csv --config_file=Finland_Census_metadata.csv --output_path=Finland_Census_output
-
-## Data Refresh & Maintenance
-
-When Statistics Finland releases new annual updates (typically in the Spring), follow these steps:
-
-1. Execute the Data Acquisition steps to get the latest CSV.
+./run.sh
+```
 
-2. Check if new demographic categories were added. Update finland_census_pvmap.csv if new labels appear in the source.
+### Script Details:
+- **Download**: The download is handled by `data_download.py` script which downloads census data from Finland's official database, formats it, and saves it to the designated file path (input_files).
+- **Processing**: Uses `stat_var_processor.py` to map raw data to Data Commons StatVarObservations using the PV map and metadata configuration.
 
-3. The source uses . to represent unavailable data (e.g., Economic dependency ratio for the current year). The processor is configured to skip these entries during import.
+For Test Data Run
 
-4. Because the source applies the 2025 regional division retrospectively, check for municipal merger updates if downloading municipality-level data.
-
-5. Ensure total population counts match the previous year's trend to verify no rows were dropped.
-
-6. Execute the data processing step.
+```bash
+python3 ../../tools/statvar_importer/stat_var_processor.py \
+  "--input_data=./test_data/finland_census_input.csv" \
+  "--pv_map=./finland_census_pvmap.csv" \
+  "--output_path=./test_data/finland_census_output" \
+  "--config_file=./finland_census_metadata.csv" \
+  "--existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+```
+
+## Key Files
+- `run.sh`: Main execution script for download and processing.
+- `data_download.py`: Downloads the data from Finland's official database, formats it, and saves it to input_files directory
+- `finland_census_pvmap.csv`: Property-Value mapping for StatVar definitions and dimensions.
+- `finland_census_metadata.csv`: Configuration parameters for the processor.
+- `output_files/finland_census_output.csv`: Processed statistical observations.
+- `output_files/finland_census_output.tmcf`: Template MCF mapping the CSV columns to Data Commons schema.
+
+## Validation
+To validate the generated data, use the Data Commons import tool (lint mode):
+```bash
+java -jar datacommons-import-tool.jar lint output_files/*.csv 
+```
+The resulting reports (`report.json`, `summary_report.html`) in `dc_generated/` provide detailed insights into data quality and validation status.
+
+## Testing
+Testing is performed using the `test_data` directory:
+- Raw Input: `test_data/finland_census_input.csv`
+- Expected Output: `test_data/finland_census_output.csv`
+- Expected TMCF: `test_data/finland_census_output.tmcf`

statvar_imports/finland_census/data_download.py

@@ -0,0 +1,187 @@
+#!/usr/bin/env python3
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""Downloader for Finland Census data from Statistics Finland PxWeb API using download_util."""
+
+import csv
+import io
+import os
+import sys
+
+_CODEDIR = os.path.dirname(os.path.abspath(__file__))
+sys.path.insert(1, _CODEDIR)
+sys.path.insert(1, os.path.join(_CODEDIR, '../../util/'))
+
+from absl import app
+from absl import flags
+from absl import logging
+import download_util
+
+try:
+  from file_util import FileIO
+except ImportError:
+  FileIO = None
+
+_FLAGS = flags.FLAGS
+flags.DEFINE_string(
+    'output_path',
+    None,
+    'Local or GCS path to save the downloaded CSV.',
+    required=True,
+)
+
+API_URL = 'https://pxdata.stat.fi/PXWeb/api/v1/en/StatFin/vaerak/11ra.px'
+
+
+def get_variable_codes() -> tuple[str, str, str]:
+  """Fetches table metadata and maps standard concepts to API codes using download_util.
+
+  Returns:
+    A tuple of strings (area_code, info_code, year_code).
+
+  Raises:
+    RuntimeError: If API request fails or variables cannot be mapped.
+  """
+  logging.info('Fetching table metadata from %s...', API_URL)
+  try:
+    metadata = download_util.request_url(
+        url=API_URL,
+        output='json',
+        retries=3,
+        retry_secs=5,
+    )
+    if not metadata or not isinstance(metadata, dict):
+      raise RuntimeError('Failed to fetch valid JSON metadata from PxWeb API.')
+
+    area_code = None
+    info_code = None
+    year_code = None
+
+    for var in metadata.get('variables', []):
+      text = var.get('text', '').upper()
+      code = var.get('code')
+
+      if 'AREA' in text:
+        area_code = code
+      elif 'INFORMATION' in text:
+        info_code = code
+      elif 'YEAR' in text:
+        year_code = code
+
+    if not area_code or not info_code or not year_code:
+      raise RuntimeError(
+          f'Could not map all variables. Detected area: {area_code}, '
+          f'info: {info_code}, year: {year_code}'
+      )
+    return area_code, info_code, year_code
+  except Exception as e:
+    raise RuntimeError(f'Error fetching variable codes: {e}') from e
+
+
+def format_csv_content(raw_csv_content: str) -> str:
+  """Formats raw PxWeb CSV to match title and header layout expected by processor.
+
+  Args:
+    raw_csv_content: The raw CSV string returned by the PxWeb API.
+
+  Returns:
+    The formatted CSV string with title block and preserved column headers.
+  """
+  if not raw_csv_content or not raw_csv_content.strip():
+    return raw_csv_content
+  lines = raw_csv_content.strip().replace('\r\n', '\n').split('\n')
+
+  header_line = lines[0]
+  num_columns = len(next(csv.reader(io.StringIO(header_line))))
+
+  title = '"Key figures on population by Area, Information and Year"'
+  title_line = title + ',' * (num_columns - 1)
+  blank_line = ',' * (num_columns - 1)
+
+  output_lines = [title_line, blank_line, header_line] + lines[1:]
+  return '\n'.join(output_lines)
+
+
+def download_data(output_path: str) -> None:
+  """Downloads, formats, and saves the Finland Census CSV data using download_util.
+
+  Args:
+    output_path: Local or GCS path where the formatted CSV will be saved.
+
+  Raises:
+    RuntimeError: If data download or file saving fails.
+  """
+  area_code, info_code, year_code = get_variable_codes()
+  logging.info(
+      'Mapped variable codes -> Area: %s, Info: %s, Year: %s',
+      area_code,
+      info_code,
+      year_code,
+  )
+
+  query = {
+      'query': [
+          {
+              'code': area_code,
+              'selection': {'filter': 'item', 'values': ['SSS']},
+          },
+          {
+              'code': info_code,
+              'selection': {'filter': 'all', 'values': ['*']},
+          },
+          {
+              'code': year_code,
+              'selection': {'filter': 'all', 'values': ['*']},
+          },
+      ],
+      'response': {'format': 'csv'},
+  }
+
+  logging.info('Sending POST query via download_util to retrieve CSV data...')
+  try:
+    content_bytes = download_util.request_url(
+        url=API_URL,
+        params=query,
+        method='POST',
+        output='bytes',
+        retries=3,
+        retry_secs=5,
+    )
+    if not content_bytes:
+      raise RuntimeError('Failed to download CSV data: empty response.')
+
+    content = content_bytes.decode('utf-8-sig')
+    formatted_content = format_csv_content(content)
+
+    if FileIO:
+      with FileIO(output_path, 'w') as f:
+        f.write(formatted_content)
+    else:
+      os.makedirs(
+          os.path.dirname(os.path.abspath(output_path)), exist_ok=True
+      )
+      with open(output_path, 'w', encoding='utf-8') as f:
+        f.write(formatted_content)
+
+    logging.info('Successfully downloaded data and saved to %s', output_path)
+  except Exception as e:
+    raise RuntimeError(f'Error downloading data: {e}') from e
+
+
+def main(_) -> None:
+  download_data(_FLAGS.output_path)
+
+
+if __name__ == '__main__':
+  app.run(main)

statvar_imports/finland_census/golden_data/golden_summary_report.csv

@@ -0,0 +1,5 @@
+"MinDate","MeasurementMethods","observationPeriods","StatVar","NumPlaces","ScalingFactors","Units"
+"1990","[]","[]","IncrementalCount_Person","1","[]","[]"
+"1990","[]","[]","Count_Person","1","[]","[]"
+"1990","[]","[]","GrowthRate_Count_Person","1","[]","[Percent]"
+"1990","[]","[]","Count_Person_PerArea","1","[]","[PersonPerSquareKilometer]"

statvar_imports/finland_census/manifest.json

@@ -5,21 +5,23 @@
             "curator_emails": [
                 "support@datacommons.org"
             ],
-            "provenance_url": "https://pxdata.stat.fi/PxWeb/pxweb/en/StatFin/StatFin__vaerak/statfin_vaerak_pxt_11ra.px/table/tableViewLayout1/",
-            "provenance_description": "Population data for Finland",
+            "provenance_url": "https://stat.fi/en",
+            "provenance_description": "National statistical data and socio-economic indicators for Finland, provided by Statistics Finland",
             "scripts": [
-                "../../tools/statvar_importer/stat_var_processor.py --input_data=gs://unresolved_mcf/country/finland/input_files/*.csv --pv_map=finland_census_pvmap.csv --config_file=finland_census_metadata.csv --output_path=output/finland_census_output --existing_statvar_mcf=gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"
+                "run.sh"
             ],
             "import_inputs": [
                 {
-                    "template_mcf": "output/finland_census_output.tmcf",
-                    "cleaned_csv": "output/finland_census_output.csv"
+                    "template_mcf": "output_files/finland_census_output.tmcf",
+                    "cleaned_csv": "output_files/finland_census_output.csv"
                 }
             ],
             "source_files": [
-                "gs://unresolved_mcf/country/finland/input_files/*.csv"
+                "input_files/*.csv"
             ],
-            "cron_schedule": "30 5 1 *  *"
+            "cron_schedule": "00 05 1,15 * *",
+            "resource_limits": {"cpu": 8, "memory": 32, "disk":100},
+            "validation_config_file": "validation_config.json"
         }
     ]
 }

statvar_imports/finland_census/run.sh

@@ -0,0 +1,22 @@
+#!/bin/bash
+# Copyright 2026 Google LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -e
+mkdir -p input_files
+mkdir -p output_files
+
+python data_download.py --output_path=./input_files/finland_census_input.csv
+
+python ../../tools/statvar_importer/stat_var_processor.py --input_data="input_files/*.csv" --pv_map="finland_census_pvmap.csv" --config_file="finland_census_metadata.csv" --output_path="output_files/finland_census_output" --existing_statvar_mcf="gs://unresolved_mcf/scripts/statvar/stat_vars.mcf"