Memory overflow during gene count matrix finalization

[nci-tech]

The algorithm crashes during the final phase of gene count matrix construction in the quantify_gene() function at this line:

gene_count_mat = pd.concat(gene_count_mat_dfs, copy=False).fillna(0)

Hardware: 32 CPU cores, 248GB RAM
Dataset: Single-cell RNA-seq data (large dataset)

Despite having substantial computational resources (248GB RAM), the process runs out of memory during the final concatenation step. The system appears to be using a single-threaded approach for this concatenation operation, creating a bottleneck that causes memory overflow regardless of available resources.

[youyupei]

Hi @nci-tech,

Thanks for raising this issue. You’re correct that pd.concat can consume a significant amount of memory when concatenating larger datasets. Instead, we can bypass the concatenation step by directly writing the list of DataFrames to the file. We’ll implement this change and update you once it’s done.

[nci-tech]

Hi youyupei,
Thank you for looking into this issue so promptly! I appreciate your confirmation about the memory consumption problem with pd.concat and am glad to hear you'll be implementing a more efficient solution. Looking forward to the update once the changes are completed.
Thank you very much!!!