Fix/expand flow system rework (#591)

* Summary of Fixes

  1. Fixed _interpolate_charge_state_segmented Bug (Critical)

  File: flixopt/transform_accessor.py:1965-2043

  The function was incorrectly decoding timestep_mapping using timesteps_per_cluster. For segmented systems, timestep_mapping encodes cluster * n_segments + segment_idx, so
  this produced wrong cluster indices.

  Fix: Compute original period index and position directly from timestep indices instead of decoding from timestep_mapping.

  2. Implemented EXPAND_FIRST_TIMESTEP for Segmented Systems Only

  File: flixopt/transform_accessor.py:2045-2100

  Added _expand_first_timestep_only() method that places startup/shutdown events at the first timestep of each segment (not cluster).

  Key design decisions:
  - Segmented systems: Events placed at first timestep of each segment (timing within segment is lost)
  - Non-segmented systems: Normal expansion preserves timing within cluster (no special handling needed)

  3. Added Tests

  - test_startup_shutdown_first_timestep_only: Verifies segmented systems place events at segment boundaries
  - test_startup_timing_preserved_non_segmented: Verifies non-segmented systems preserve timing within clusters

  4. Updated Docstrings

  Clarified in expand() docstring that:
  - Binary events in segmented systems go to first timestep of each segment
  - Non-segmented systems preserve timing via normal expansion

* Summary of Fixes

  1. Fixed _interpolate_charge_state_segmented Bug (Critical)

  File: flixopt/transform_accessor.py:1965-2043

  The function was incorrectly decoding timestep_mapping using timesteps_per_cluster. For segmented systems, timestep_mapping encodes cluster * n_segments + segment_idx, so
  this produced wrong cluster indices.

  Fix: Compute original period index and position directly from timestep indices instead of decoding from timestep_mapping.

  2. Implemented EXPAND_FIRST_TIMESTEP for Segmented Systems Only

  File: flixopt/transform_accessor.py:2045-2100

  Added _expand_first_timestep_only() method that places startup/shutdown events at the first timestep of each segment (not cluster).

  Key design decisions:
  - Segmented systems: Events placed at first timestep of each segment (timing within segment is lost)
  - Non-segmented systems: Normal expansion preserves timing within cluster (no special handling needed)

  3. Added Tests

  - test_startup_shutdown_first_timestep_only: Verifies segmented systems place events at segment boundaries
  - test_startup_timing_preserved_non_segmented: Verifies non-segmented systems preserve timing within clusters

  4. Updated Docstrings

  Clarified in expand() docstring that:
  - Binary events in segmented systems go to first timestep of each segment
  - Non-segmented systems preserve timing via normal expansion

* Validate extremeconfig method to be "replace"

* The fix now properly handles variables with both time and period (or scenario) dimensions:

  1. For each missing (period_label, scenario_label) key, it selects the specific period/scenario slice from the original variable using .sel(..., drop=True)
  2. Then it slices and reshapes that specific slice
  3. All slices in filled_slices now have consistent dimensions ['cluster', 'time'] + other_dims without 'period' or 'scenario' coordinates

  This ensures all DataArrays being concatenated have the same structure. Can you try running your clustering code again?

* Summary of Refactoring Changes

  New Modules Created

  1. flixopt/clustering/iteration.py - Shared iteration infrastructure:
    - DimSliceContext dataclass for (period, scenario) slice context
    - DimInfo dataclass for dimension metadata
    - iter_dim_slices() generator for standardized iteration
    - iter_dim_slices_simple() convenience function
  2. flixopt/clustering/aggregation.py - Aggregation helpers:
    - combine_slices_to_dataarray() - Unified slice combination function
    - build_typical_dataarrays() - Build typical periods DataArrays
    - build_segment_durations() - Build segment duration DataArray
    - build_cluster_weights() - Build cluster weight DataArray
    - build_clustering_metrics() - Build metrics Dataset
    - calculate_clustering_weights() - Calculate clustering weights from data
    - build_cluster_config_with_weights() - Merge weights into ClusterConfig
    - accuracy_to_dataframe() - Convert tsam AccuracyMetrics to DataFrame
  3. flixopt/clustering/expansion.py - Expansion helpers:
    - VariableExpansionHandler class - Handles variable-specific expansion logic
    - interpolate_charge_state_segmented() - Interpolate state variables in segments
    - expand_first_timestep_only() - Expand binary events to first timestep
    - build_segment_total_varnames() - Build segment total variable names
    - append_final_state() - Append final state value to expanded data
  4. Updated flixopt/clustering/intercluster_helpers.py:
    - Added combine_intercluster_charge_states() - Combine charge_state with SOC_boundary
    - Added apply_soc_decay() - Apply self-discharge decay to SOC values

  Refactored flixopt/transform_accessor.py

  - cluster() method: Now uses DimInfo and iter_dim_slices() for standardized iteration
  - expand() method: Uses VariableExpansionHandler and extracted helper functions
  - _build_reduced_flow_system(): Signature changed to use DimInfo instead of separate periods/scenarios lists
  - _build_reduced_dataset(): Updated to use DimInfo and combine_slices_to_dataarray()
  - apply_clustering(): Uses DimInfo for cleaner key conversion

  Removed from transform_accessor.py

  - _calculate_clustering_weights() → moved to clustering/aggregation.py
  - _build_cluster_config_with_weights() → moved to clustering/aggregation.py
  - _accuracy_to_dataframe() → moved to clustering/aggregation.py
  - _build_cluster_weight_da() → moved to clustering/aggregation.py
  - _build_typical_das() → moved to clustering/aggregation.py
  - _build_segment_durations_da() → moved to clustering/aggregation.py
  - _build_clustering_metrics() → moved to clustering/aggregation.py
  - _combine_slices_to_dataarray_generic() → replaced by unified combine_slices_to_dataarray()
  - _combine_slices_to_dataarray_2d() → replaced by unified combine_slices_to_dataarray()
  - _combine_intercluster_charge_states() → moved to clustering/intercluster_helpers.py
  - _apply_soc_decay() → moved to clustering/intercluster_helpers.py
  - _build_segment_total_varnames() → moved to clustering/expansion.py
  - _interpolate_charge_state_segmented() → moved to clustering/expansion.py
  - _expand_first_timestep_only() → moved to clustering/expansion.py

  Updated flixopt/clustering/__init__.py

  - Exports all new helpers from the new modules

  Tests

  - All 102 tests in test_cluster_reduce_expand.py pass
  - All 43 tests in test_clustering_io.py pass

* Update tsam dep

* Update tsam dep

* 1. flixopt/clustering/intercluster_helpers.py: Fixed apply_soc_decay to handle both scalar and DataArray returns from _scalar_safe_reduce using np.asarray().
  2. flixopt/transform_accessor.py: Fixed the multi-dimensional check to only block extremes.method='new_cluster'/'append' when there are actually multiple slices (total_slices
   > 1), not just when periods or scenarios exist.
  3. flixopt/clustering/aggregation.py: Fixed combine_slices_to_dataarray to transpose with all base dims in order (*base_dims, ...) instead of just the first one.
  4. tests/test_cluster_reduce_expand.py:
    - Updated test_extremes_new_cluster_increases_n_clusters docstring to clarify that tsam may or may not add clusters (the >= 2 assertion was actually correct)
    - Renamed test_extremes_new_cluster_with_segments to test_extremes_append_with_segments and updated docstring to match the actual method='append' used

Author: FBumann

flixopt/clustering/__init__.py

@@ -42,10 +42,64 @@
     fs_expanded = fs_clustered.transform.expand()
 """
 
+from .aggregation import (
+    accuracy_to_dataframe,
+    build_cluster_assignments_dataarray,
+    build_cluster_config_with_weights,
+    build_cluster_weights,
+    build_clustering_metrics,
+    build_segment_durations,
+    build_typical_dataarrays,
+    calculate_clustering_weights,
+    combine_slices_to_dataarray,
+)
 from .base import AggregationResults, Clustering, ClusteringResults
+from .expansion import (
+    VariableExpansionHandler,
+    append_final_state,
+    build_segment_total_varnames,
+    expand_first_timestep_only,
+    interpolate_charge_state_segmented,
+)
+from .intercluster_helpers import (
+    CapacityBounds,
+    apply_soc_decay,
+    build_boundary_coords,
+    combine_intercluster_charge_states,
+    extract_capacity_bounds,
+)
+from .iteration import DimInfo, DimSliceContext, iter_dim_slices, iter_dim_slices_simple
 
 __all__ = [
+    # Base classes
     'ClusteringResults',
     'AggregationResults',
     'Clustering',
+    # Iteration utilities
+    'DimSliceContext',
+    'DimInfo',
+    'iter_dim_slices',
+    'iter_dim_slices_simple',
+    # Aggregation helpers
+    'combine_slices_to_dataarray',
+    'build_typical_dataarrays',
+    'build_segment_durations',
+    'build_cluster_weights',
+    'build_clustering_metrics',
+    'build_cluster_assignments_dataarray',
+    'calculate_clustering_weights',
+    'build_cluster_config_with_weights',
+    'accuracy_to_dataframe',
+    # Expansion helpers
+    'VariableExpansionHandler',
+    'build_segment_total_varnames',
+    'interpolate_charge_state_segmented',
+    'expand_first_timestep_only',
+    'append_final_state',
+    # Intercluster helpers
+    'CapacityBounds',
+    'extract_capacity_bounds',
+    'build_boundary_coords',
+    'combine_intercluster_charge_states',
+    'apply_soc_decay',
 ]