Hi all,
I am trying to figure out what might be wrong...
When running shuffle2.sh on paired-end FASTQ with in=, in2=, out=, out2=, the run often fails during the merge step with a Java AssertionError. The failure happens only when the message "Reduced buffer sizes prior to merging due to low memory" is printed; without that path, the tool can complete.
Here's the relevant part in the log file that describes the error:
[2026-03-13T10:43:43] + R1=SRR16019134_1_val_1.fq
[2026-03-13T10:43:43] + R2=SRR16019134_2_val_2.fq
[2026-03-13T10:43:43] + S1=SRR16019134_1_val_1.fq.shuf
[2026-03-13T10:43:43] + S2=SRR16019134_2_val_2.fq.shuf
[2026-03-13T10:43:43] + shuffle2.sh -Xmx32g in=SRR16019134_1_val_1.fq in2=SRR16019134_2_val_2.fq out=SRR16019134_1_val_1.fq.shuf out2=SRR16019134_2_val_2.fq.shuf
[2026-03-13T10:43:43] java -ea --add-modules jdk.incubator.vector -Xmx32g -Xms32g -cp /app/.conda/envs/sra_rnaseq/opt/bbmap-39.80-0/bbtools.jar sort.Shuffle2 -Xmx32g in=SRR16019134_1_val_1.fq in2=SRR16019134_2_val_2.fq out=SRR16019134_1_val_1.fq.shuf out2=SRR16019134_2_val_2.fq.shuf
[2026-03-13T10:43:43] WARNING: Using incubator modules: jdk.incubator.vector
[2026-03-13T10:43:44] Executing sort.Shuffle2 [-Xmx32g, in=SRR16019134_1_val_1.fq, in2=SRR16019134_2_val_2.fq, out=SRR16019134_1_val_1.fq.shuf, out2=SRR16019134_2_val_2.fq.shuf]
[2026-03-13T10:43:44]
[2026-03-13T10:43:44] Unspecified format for output SRR16019134_1_val_1.fq.shuf; defaulting to fastq.
[2026-03-13T10:43:44] Unspecified format for output SRR16019134_2_val_2.fq.shuf; defaulting to fastq.
[2026-03-13T10:43:44] Input is being processed as paired
[2026-03-13T10:43:50] Created a WriteThread for ./sort_temp_3541210922057909268.fq.gz
[2026-03-13T10:43:55] Created a WriteThread for ./sort_temp_14555124363830938688.fq.gz
[2026-03-13T10:43:59] Finished reading input.
[2026-03-13T10:43:59] Created a WriteThread for ./sort_temp_12914152056054529736.fq.gz
[2026-03-13T10:44:19] Merging 3 files.
[2026-03-13T10:44:19] Reduced buffer sizes prior to merging due to low memory.
[2026-03-13T10:44:19] Merging [./sort_temp_3541210922057909268.fq.gz, ./sort_temp_14555124363830938688.fq.gz, ./sort_temp_12914152056054529736.fq.gz]
[2026-03-13T10:44:19] Exception in thread "Thread-15" java.lang.AssertionError:
[2026-03-13T10:44:19] SRR16019134.16789887 A00159:806:HWGKKDSXY:3:2437:21594:5916 length=150 3895 -1 + -1 -1 00001000000000000000000 1 0 0 GAAAAACGAATATAATAATAAGACAAAAAATAATAAAAACAATAAATTTGAACAACAAAATTATGATGAATCATATATGGATGATGATTATGAACAAAATGAAGAATTTAATGATAATAATCAAAGCGAAGATATGAAAGAAACAAATG FFFFF:FFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFF::FFFFFFFFFFFFF:FFFFFF:FFFFFFFFFFFFFFFFFFFF:FFFFFFFFFFFFFFFFFFFFFFFF: . . . .
[2026-03-13T10:44:19]
[2026-03-13T10:44:19] null
[2026-03-13T10:44:19] at stream.ReadStreamByteWriter.writeFastq(ReadStreamByteWriter.java:495)
[2026-03-13T10:44:19] at stream.ReadStreamByteWriter.processJobs(ReadStreamByteWriter.java:102)
[2026-03-13T10:44:19] at stream.ReadStreamByteWriter.run2(ReadStreamByteWriter.java:47)
[2026-03-13T10:44:19] at stream.ReadStreamByteWriter.run(ReadStreamByteWriter.java:33)
[2026-03-13T10:44:19] Exception in thread "main" java.lang.RuntimeException: Writing to a terminated thread.
[2026-03-13T10:44:19] at stream.ConcurrentGenericReadOutputStream.write(ConcurrentGenericReadOutputStream.java:244)
[2026-03-13T10:44:19] at stream.ConcurrentGenericReadOutputStream.addDisordered(ConcurrentGenericReadOutputStream.java:235)
[2026-03-13T10:44:19] at stream.ConcurrentGenericReadOutputStream.add(ConcurrentGenericReadOutputStream.java:114)
[2026-03-13T10:44:19] at sort.Shuffle2.mergeAndDump(Shuffle2.java:634)
[2026-03-13T10:44:19] at sort.Shuffle2.mergeAndDump(Shuffle2.java:576)
[2026-03-13T10:44:19] at sort.Shuffle2.mergeAndDump(Shuffle2.java:523)
[2026-03-13T10:44:19] at sort.Shuffle2.processInner(Shuffle2.java:400)
[2026-03-13T10:44:19] at sort.Shuffle2.process(Shuffle2.java:273)
[2026-03-13T10:44:19] at sort.Shuffle2.main(Shuffle2.java:52)
I am not sure what I am doing wrong and why this fails? Is this just a memory-related issue?
Hi all,
I am trying to figure out what might be wrong...
When running shuffle2.sh on paired-end FASTQ with in=, in2=, out=, out2=, the run often fails during the merge step with a Java AssertionError. The failure happens only when the message "Reduced buffer sizes prior to merging due to low memory" is printed; without that path, the tool can complete.
Here's the relevant part in the log file that describes the error:
I am not sure what I am doing wrong and why this fails? Is this just a memory-related issue?