Feature: Aggregated Tabular Export & Bugfix for Unicode Character 'µ' in Windows Chinese Locales

[TianyiMa96]

Hello pyFAI team,

We are a team of researchers in Institute of Chemistry, Chinese Academy of Sciences (ICCAS), specializing in polymer fiber crystals analysis. We are writing to contribute several enhancements to pyFAI-integrate and to express our deep gratitude for this world-class library.

The Power of pyFAI in Our Research

The advanced integration algorithms and the flexible architecture of pyFAI have been instrumental in our work. It has significantly reduced the amount of boilerplate code we needed to write, allowing us to focus on high-level crystallographic analysis. Specifically, pyFAI serves as one of the critical preprocessing engines for our developed tools:

PolymCrystIndex: A method for indexing fiber diffraction.[Ma, T., Hu, W., Wang, D. & Liu, G. (2025). J. Appl. Cryst. 58, 759-767.]

PolymCrystRefine: Our upcoming polymer fiber crystal refinement suite.

1. Bug Fix: Cross-Regional Compatibility for 'µ' Character

The Problem: In Windows environments with a Chinese locale (defaulting to GBK encoding), pyFAI-integrate crashes when writing metadata headers. This is triggered by the µ symbol typically present in .poni files generated via calib2. Since the default open() function is locale-dependent, it fails to handle these UTF-8 characters on GBK systems.

The Solution: We updated the I/O logic to explicitly use encoding="utf-8", ensuring a robust, crash-free experience for users worldwide.

File: pyFAI/io/init.py (around line 800)

        with open(filename, "w", encoding="utf-8") as f:
            f.write(self.make_headers(has_mask=has_mask, has_dark=has_dark,
                                      has_flat=has_flat,
                                      polarization_factor=polarization_factor,
                                      normalization_factor=normalization_factor,
                                      metadata=metadata))
            try:
                f.write(f"\n# --> {filename}\n")
            except UnicodeError:
                f.write(f"\n# --> {filename.encode('utf8')}\n")

2. New Feature: Aggregated CSV/Excel Export with Metadata Headers

The Requirement: When performing 1D integration using the same .poni calibration file, the resulting datasets often share high similarity in experimental context. To streamline post-processing and downstream analysis, there was a need to consolidate these results.

The Solution: We implemented a SingleTabularWriter class in app/integrate.py. This allows users to aggregate multiple 1D integration results into a single CSV or Excel (.xlsx) file, complete with integrated metadata headers for better data traceability.

File: pyFAI/app/integrate.py

• Dependency Addition:
  We used pandas for tabular data handling. If the team prefers to avoid new dependencies, we are happy to rewrite this part using Python’s built-in csv module.
  import pandas as pd

• Implementation of SingleTabularWriter:

This class consolidates 1D results into a single file while maintaining scientific metadata traceability. It uses utf-8-sig to ensure Windows Excel compatibility and prevents blank row issues.

class SingleTabularWriter(io.Writer):
    """
    Aggregates all 1D integration results into a single CSV or XLSX file.
    Includes metadata header similar to .dat files.
    """
    def __init__(self, output_path, format="csv"):
        super(SingleTabularWriter, self).__init__()
        self._output_path, self._format = output_path, format
        self._radial_axis, self._data_dict = None, {}

    def init(self, fai_cfg=None, lima_cfg=None):
        self._is_2d = fai_cfg.get("do_2D", False) if isinstance(fai_cfg, dict) else getattr(fai_cfg, "do_2D", False)
        if self._is_2d:
            raise ValueError("Aggregated export supports 1D integration only!")

    def prepare_write(self, data_info, engine):
        name = os.path.splitext(os.path.basename(data_info.source_filename))[0] if data_info.source_filename else f"array_{data_info.data_id}"
        if data_info.frame_id is not None:
            name = f"{name}_{data_info.frame_id:04d}"
        self._current_name, self._ai = name, engine 

    def write(self, data):
        if self._radial_axis is None:
            self._radial_axis = data.radial
            self._unit = getattr(data, "unit", "unknown")
        self._data_dict[self._current_name] = data.intensity

    def close(self):
        if not self._data_dict: return
        df = pd.DataFrame({f"radial_{str(getattr(self, '_unit', 'unknown'))}": self._radial_axis})
        for name, intensity in self._data_dict.items(): df[name] = intensity

        header_lines = [
            f"# Integrated using pyFAI version {pyFAI_version}",
            f"# Date: {time.ctime()}",
            f"# Detector: {getattr(self._ai.detector, 'name', 'unknown')}",
            f"# Distance: {self._ai.dist} m",
            f"# Wavelength: {self._ai.wavelength} m",
            "#"
        ]

        if self._format == "csv":
            with open(self._output_path, 'w', encoding='utf-8-sig', newline='') as f:
                for line in header_lines: f.write(line + "\n")
                df.to_csv(f, index=False, lineterminator='\n')
        elif self._format == "xlsx":
            with pd.ExcelWriter(self._output_path, engine='openpyxl') as writer:
                df.to_excel(writer, sheet_name='Integration Results', index=False)
                meta_df = pd.DataFrame({"Property": [l.split(':')[0].strip('# ') for l in header_lines if ':' in l],
                                        "Value": [l.split(':', 1)[1].strip() for l in header_lines if ':' in l]})
                meta_df.to_excel(writer, sheet_name='Metadata', index=False)

• Format Selection Prompt (within processData):

We added a user-friendly selection dialog that appears when 1D integration is chosen, allowing researchers to choose their preferred output format at runtime.

Writer Initialization (within process):
Integrated the logic to switch from standard MultiFileWriter or HDF5Writer to the new SingleTabularWriter when CSV/XLSX is requested.

if format_ in ("csv", "xlsx"):
     if output and os.path.isdir(output):
         out_path = os.path.join(output, f"merged_results.{format_}")
     elif output:
         out_path = output if output.endswith(f".{format_}") else f"{output}.{format_}"
     else:
         out_path = f"merged_results.{format_}"
     writer = SingleTabularWriter(out_path, format=format_)

AI Assistance Disclosure

The implementation of the tabular writer and the refinement of this documentation were prepared with the help of generative AI. All AI-generated code has been rigorously reviewed and tested in our laboratory to ensure its scientific accuracy and reliability.

Environment Details:

Version: 2025.12.1

OS: Windows 11 and Windows 10 with a Chinese locale.

Collaborative Outreach

We believe pyFAI is exceptionally efficient and deserves broader recognition. We request your permission to create and publish Chinese video tutorials to promote pyFAI's capabilities within the Chinese scientific community.

We would be extremely grateful if you could take a look at our approach. We hope our code might serve as a useful reference for the future evolution of the project.

Thank you for your time and consideration!

Best regards,
Tianyi Ma
Email：tyma507@iccas.ac.cn

[gudlot]

Hi @TianyiMa96,
I am not a core pyFAI developer, but I have a question regarding part 2, out of curiosity.

Could you clarify the motivation for exporting aggregated results to CSV/XLSX? These formats are convenient for inspection, but in my experience they scale poorly when handling large numbers of 1D curves and rich metadata.

In many workflows, the integration output is already stored in HDF5, from which data can be extracted into a Pandas DataFrame as needed. HDF5 tends to be more flexible and significantly faster for re-importing and reprocessing large collections of scattering curves.

I would be interested to understand the concrete use case that CSV/XLSX is intended to address here.

[TianyiMa96]

Hi @TianyiMa96, I am not a core pyFAI developer, but I have a question regarding part 2, out of curiosity.

Could you clarify the motivation for exporting aggregated results to CSV/XLSX? These formats are convenient for inspection, but in my experience they scale poorly when handling large numbers of 1D curves and rich metadata.

In many workflows, the integration output is already stored in HDF5, from which data can be extracted into a Pandas DataFrame as needed. HDF5 tends to be more flexible and significantly faster for re-importing and reprocessing large collections of scattering curves.

I would be interested to understand the concrete use case that CSV/XLSX is intended to address here.

Thank you for your reply and the thoughtful question!

I completely agree with you regarding the scalability and performance of the HDF5 format. We have actually experienced its benefits firsthand when processing large datasets from Eiger2 detectors. As you noted, even the standard .dat format is usually sufficient if one writes a quick script to handle it.

Our motivation for exporting to CSV/XLSX mainly comes from the practical, daily needs of the user base we interact with (primarily physics and chemistry researchers here in China). The concrete use cases we are trying to address are:

1. Accessibility for non-programmers: A large majority of experimental researchers are not experienced coders. They heavily rely on GUI-based software like OriginLab or Excel to plot and analyze their data. Exporting directly to CSV/XLSX provides a zero-barrier way for them to import and visualize their aggregated results immediately, without needing to write a Pandas/HDF5 extraction script.

2. Manageable data scale: While it’s true that spreadsheets scale poorly for massive datasets, our specific applications (such as in-situ polymer testing) generally involve a limited number of frames—rarely exceeding 200 images per experiment. At this scale, CSV/XLSX is fully capable of handling the 1D curves and metadata without performance issues.

Of course, we recognize that this perspective is based on our specific experience with polymer characterization and may not represent the needs of the entire X-ray scattering community. Therefore, our proposal for Part 2 is simply a suggestion to lower the barrier for experimentalists (even though we have provided the implementation code). We completely respect whatever decision the core developers make regarding this feature.

Ultimately, our primary expectation is that Part 1 of our proposal can be refined and accepted. That part is crucial because it ensures other pyFAI tools can run much more smoothly. Integrating Part 1 would greatly empower users in our region to take full advantage of pyFAI's newer, highly efficient programs beyond just pyFAI-calib2.

[kif]

Dear @TianyiMa96,

Thanks for reporting this issue. Indeed pyFAI used to rely on the locale of the system which sometimes supports only partially the metadata involved. Enforcing UTF8 is probably the safest, as performed in PR#2799.

Concerning your paper about PolymCrystIndex, I am surprised there is no reference to pyFAI in it ... you have probably forgotten :-) That said, you claim the code & data are available upon request, request that I make here. Ideally they should be publicly available as I am not allowed to contribute any external & non-public/commercial code as part of my job for ESRF.

About the export in a format that would be compatible with Excel spreadsheets, I would definitely go via the python CSV module rather than via Pandas, it looks overkill to me to create such a dependency for a single proprietary export.
Beside this, the concerns of Gudrun about the applicability of Excel for scientific data treatment are perfectly valid to me.

Cheers,

Jerome

[gudlot]

Hi @TianyiMa96 ,

Thank you for the explanation.

Regarding the point that "even the standard .dat format is usually sufficient if one writes a quick script," I would like to disagree with the “quick script" part based on my own experience.
For example, I encountered multiple .dat files in the past (e.g., created by SPEC) where the output, and specifically the available columns, changed depending on the used scanning command. Writing "quick scripts" for varying versions became quickly a burden.

In my opinion, an HDF5 file following the NeXus standard is far more convenient. I learned from Tim Snow (DLS) that even inexperienced users are usually happy with output that follows the NeXus standard. A high-quality analysis tool in the scattering community should understand this standard for processed/reduced data, which ultimately provides a better user experience than a spreadsheet.

[maaeli]

Hi,

• just a quick comment on csv: This is in practice not a well-defined format. Comma separated values are not easily openable with Excel in most European locales: Excel takes the decimal separator from the windows settings, and for many European locales, this is ','. Excel uses ";" for these locales as column separator. if you want globally exchangeable data, you should not use csv.

This is probably one of the reasons space or tab separated ascii files are more common in science.

Cheers,
Martha

[TianyiMa96]

Dear @TianyiMa96,

Thanks for reporting this issue. Indeed pyFAI used to rely on the locale of the system which sometimes supports only partially the metadata involved. Enforcing UTF8 is probably the safest, as performed in PR#2799.

Concerning your paper about PolymCrystIndex, I am surprised there is no reference to pyFAI in it ... you have probably forgotten :-) That said, you claim the code & data are available upon request, request that I make here. Ideally they should be publicly available as I am not allowed to contribute any external & non-public/commercial code as part of my job for ESRF.

About the export in a format that would be compatible with Excel spreadsheets, I would definitely go via the python CSV module rather than via Pandas, it looks overkill to me to create such a dependency for a single proprietary export. Beside this, the concerns of Gudrun about the applicability of Excel for scientific data treatment are perfectly valid to me.

Cheers,

Jerome

Dear Jerome,

Thanks for your reply.

I am glad to see that some of our suggestions regarding the UTF8 enforcement have been adopted. I also completely agree with your point about using the Python CSV module instead of Pandas for the export format to avoid unnecessary dependencies.

Regarding your mention of the missing reference to pyFAI in our paper on PolymCrystIndex, I would like to clarify that we did not use pyFAI during the early stages of that work before publishing of this paper, which is why we apologize for not citing it at the time. However, in the past six months, we have found pyFAI to be extremely high-performing and rich in features that we urgently needed. Consequently, we have developed a series of pre-processing programs based on it. We will make sure to highlight this and cite pyFAI in our upcoming paper for the "PolymCrystRefine" program.

As for the code, we will release it on GitHub under the relevant license once we have completed further testing. Prior to this point, we did not have the intention to commercialize or even privately share any modules containing open-source components. Thank you very much for your reminder.

Thanks again for the great work on pyFAI.

Best regards,
Tianyi Ma

[TianyiMa96]

Regarding the point that "even the standard .dat format is usually sufficient if one writes a quick script," I would like to disagree with the “quick script" part based on my own experience. For example, I encountered multiple .dat files in the past (e.g., created by SPEC) where the output, and specifically the available columns, changed depending on the used scanning command. Writing "quick scripts" for varying versions became quickly a burden.

In my opinion, an HDF5 file following the NeXus standard is far more convenient. I learned from Tim Snow (DLS) that even inexperienced users are usually happy with output that follows the NeXus standard. A high-quality analysis tool in the scattering community should understand this standard for processed/reduced data, which ultimately provides a better user experience than a spreadsheet.

Hi, @gudlot

Thank you for sharing your perspective and experience. I admit that my previous consideration regarding the format was not thorough enough. When I mentioned writing a "quick script," my target scenario was primarily aggregating and synthesizing data generated under strictly identical conditions (which I think aligns with the standard batch-processing setup). However, I completely understand your point—when dealing with varying outputs and columns like those from SPEC, script maintenance can quickly become a burden.

To be honest, my personal experience with handling .dat files or the HDF5/NeXus standard is still quite limited; so far, I have only encountered it when processing Eiger detector data. I will definitely take your advice and try working more with the HDF5/NeXus format in the near future to better experience the conveniences and advantages you mentioned. After some targeted research, we also plan to share and discuss these workflows with synchrotron and X-ray diffraction users in China. I will be sure to report back with any potential issues or feedback we gather along the way.

On a related note, I have a small question: could you point me to how to output an HDF5 file using pyfai-integrate? Ever since you mentioned it last time, I have been trying to find a way to export to HDF5, but I haven't been successful in either the command line or the GUI. If this is currently handled by the -f argument in the CLI, it might be very helpful to add a bit more explanation about it in the -h help text. Furthermore, if explicit HDF5 export options could be added to the PySide GUI, it would perhaps make things much more user-friendly for people looking for this feature!