pandas has a regression where pd.DataFrame.sparse.from_spmatrix() uses NaN as the fill_value instead of 0.0 for float dtype sparse arrays.
Announcement of change: What's new in 3.0.0
Merged changed: pandas-dev/pandas#59064
Upstream bug: pandas-dev/pandas#59212
In pandas ≤3.0: from_spmatrix() correctly used fill_value=0.0 for floats
In pandas main branch: from_spmatrix() uses fill_value=NaN for floats (REGRESSION)
This is problematic because scipy.sparse assumes fill_value=0, so the pandas conversion should preserve that behavior.
Impact on fips
This affects:
CovarianceBuilder.build() when sparse=TrueObsAggregator.build_operator() when building sparse forward operators
The bug causes zeros in covariance matrices to display as NaN, breaking numerical comparisons and equality checks in tests.
Workaround: use df.fillna(0.0) after creating sparse dataframe with fill_value=NaN
pandas has a regression where
pd.DataFrame.sparse.from_spmatrix()uses NaN as the fill_value instead of 0.0 for float dtype sparse arrays.Announcement of change: What's new in 3.0.0
Merged changed: pandas-dev/pandas#59064
Upstream bug: pandas-dev/pandas#59212
In pandas ≤3.0:
from_spmatrix()correctly usedfill_value=0.0for floatsIn pandas main branch:
from_spmatrix()usesfill_value=NaNfor floats (REGRESSION)This is problematic because scipy.sparse assumes fill_value=0, so the pandas conversion should preserve that behavior.
Impact on fips
This affects:
CovarianceBuilder.build()whensparse=TrueObsAggregator.build_operator()when building sparse forward operatorsThe bug causes zeros in covariance matrices to display as NaN, breaking numerical comparisons and equality checks in tests.
Workaround: use
df.fillna(0.0)after creating sparse dataframe withfill_value=NaN