bug: bbmerge.sh hangs on v39.56 and up

[eboyden]

Running on MacBook M2 Pro with java version "1.8.0_471". Version v39.55 seems to work ok:

test % bbmerge=/tools/bbtools/BBTools-39.55/bbmerge.sh
test % $bbmerge -Xmx8g -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz                   
java -ea -XX:+ExitOnOutOfMemoryError  -Xmx8g -Xms8g -cp /tools/bbtools/BBTools-39.55/current/ jgi.BBMerge -Xmx8g -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
Executing jgi.BBMerge [-Xmx8g, -eoom, mix, tbo, ftl=30, mininsert=30, mininsert0=30, in1=r1.fq.gz, in2=r2.fq.gz, out=out.fq.gz]
Version 39.55

Writing mergable reads interleaved.
Started output threads.
Total time: 192.478 seconds.

v39.56:

test % bbmerge=/tools/bbtools/BBTools-39.56/bbmerge.sh                                                                            
test % $bbmerge -Xmx8g -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz                                  
java -ea -XX:+ExitOnOutOfMemoryError  -Xmx8g -Xms8g -cp /tools/bbtools/BBTools-39.56/current/ jgi.BBMerge -Xmx8g -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
Executing jgi.BBMerge [-Xmx8g, -eoom, mix, tbo, ftl=30, mininsert=30, mininsert0=30, in1=r1.fq.gz, in2=r2.fq.gz, out=out.fq.gz]
Version 39.56

Writing mergable reads interleaved.
Started output threads.
Exception in thread "Thread-20" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1854)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
Exception in thread "Thread-22" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1854)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
Exception in thread "Thread-21" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1854)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
Exception in thread "Thread-23" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1854)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
Exception in thread "Thread-19" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1854)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
Exception in thread "Thread-18" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1854)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)

v39.79:

test % bbmerge=/tools/bbtools/BBTools-39.79/bbmerge.sh                                                      
test % $bbmerge -Xmx8g -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
java -ea -XX:+ExitOnOutOfMemoryError  -Xmx8g -Xms8g -cp /tools/bbtools/BBTools-39.79/current/ jgi.BBMerge -Xmx8g -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
Executing jgi.BBMerge [-Xmx8g, -eoom, mix, tbo, ftl=30, mininsert=30, mininsert0=30, in1=r1.fq.gz, in2=r2.fq.gz, out=out.fq.gz]
Version 39.79

Writing mergable reads interleaved.
Started output threads.
Exception in thread "Thread-21" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1920)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
Exception in thread "Thread-22" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1920)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)

Appears to be related to threading, because it runs to completion with t=2, albeit slowly, but not with t=8 or t=1 (although with t=1 it exits immediately rather than hanging):

test % $bbmerge -Xmx8g t=2 -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
java -ea -XX:+ExitOnOutOfMemoryError  -Xmx8g -Xms8g -cp /tools/bbtools/BBTools-39.79/current/ jgi.BBMerge -Xmx8g t=2 -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
Executing jgi.BBMerge [-Xmx8g, t=2, -eoom, mix, tbo, ftl=30, mininsert=30, mininsert0=30, in1=r1.fq.gz, in2=r2.fq.gz, out=out.fq.gz]
Version 39.79

Set threads to 2
Writing mergable reads interleaved.
Started output threads.
Total time: 621.004 seconds.

test % $bbmerge -Xmx8g t=8 -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
java -ea -XX:+ExitOnOutOfMemoryError  -Xmx8g -Xms8g -cp /tools/bbtools/BBTools-39.79/current/ jgi.BBMerge -Xmx8g t=8 -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
Executing jgi.BBMerge [-Xmx8g, t=8, -eoom, mix, tbo, ftl=30, mininsert=30, mininsert0=30, in1=r1.fq.gz, in2=r2.fq.gz, out=out.fq.gz]
Version 39.79

Set threads to 8
Writing mergable reads interleaved.
Started output threads.
Exception in thread "Thread-17" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1920)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
Exception in thread "Thread-16" java.lang.AssertionError
	at stream.PairStreamer.nextList(PairStreamer.java:94)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1920)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)

test % $bbmerge -Xmx8g t=1 -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
java -ea -XX:+ExitOnOutOfMemoryError  -Xmx8g -Xms8g -cp /tools/bbtools/BBTools-39.79/current/ jgi.BBMerge -Xmx8g t=1 -eoom mix tbo ftl=30 mininsert=30 mininsert0=30 in1=r1.fq.gz in2=r2.fq.gz out=out.fq.gz
Executing jgi.BBMerge [-Xmx8g, t=1, -eoom, mix, tbo, ftl=30, mininsert=30, mininsert0=30, in1=r1.fq.gz, in2=r2.fq.gz, out=out.fq.gz]
Version 39.79

Set threads to 1
Writing mergable reads interleaved.
Started output threads.
Exception in thread "Thread-4" java.lang.AssertionError: List size mismatch: 79 vs 73
	at stream.PairStreamer.nextList(PairStreamer.java:86)
	at jgi.BBMerge$MateThread.processReads(BBMerge.java:1920)
	at jgi.BBMerge$MateThread.run(BBMerge.java:1848)
java.io.IOException: Stream closed
	at java.util.zip.GZIPInputStream.ensureOpen(GZIPInputStream.java:62)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:113)
	at stream.FastqScanStreamer.readFastq(FastqScanStreamer.java:90)
	at stream.FastqScanStreamer$ReaderRunnable.run(FastqScanStreamer.java:283)
	at java.lang.Thread.run(Thread.java:750)
java.io.IOException: Stream closed
	at java.util.zip.GZIPInputStream.ensureOpen(GZIPInputStream.java:62)
	at java.util.zip.GZIPInputStream.read(GZIPInputStream.java:113)
	at stream.FastqScanStreamer.readFastq(FastqScanStreamer.java:90)
	at stream.FastqScanStreamer$ReaderRunnable.run(FastqScanStreamer.java:283)
	at java.lang.Thread.run(Thread.java:750)
Total time: 0.177 seconds.

Are release notes available anywhere? changelog.txt hasn't been updated since v39.01...

[bheywar]

I am getting the same error when running bbduk with paired files. Unpaired works, but once I set in2=pair (no threads specified):

Exception in thread "Thread-56" java.lang.AssertionError
        at stream.PairStreamer.nextList(PairStreamer.java:94)
        at bbduk.BBDukProcessorS.process(BBDukProcessorS.java:743)
        at bbduk.BBDukS$ProcessThread.run(BBDukS.java:371)
Exception in thread "Thread-62" Exception in thread "Thread-45" java.lang.AssertionError
        at stream.PairStreamer.nextList(PairStreamer.java:94)
        at bbduk.BBDukProcessorS.process(BBDukProcessorS.java:743)
        at bbduk.BBDukS$ProcessThread.run(BBDukS.java:371)
java.lang.AssertionError
        at stream.PairStreamer.nextList(PairStreamer.java:94)
        at bbduk.BBDukProcessorS.process(BBDukProcessorS.java:743)
        at bbduk.BBDukS$ProcessThread.run(BBDukS.java:371)

Same for any larger number of threads (20, 40...) set.

I tried setting threads=2, and got:

Exception in thread "Thread-4" java.lang.AssertionError: List size mismatch: 731 vs 730
        at stream.PairStreamer.nextList(PairStreamer.java:86)
        at bbduk.BBDukProcessorS.process(BBDukProcessorS.java:743)
        at bbduk.BBDukS$ProcessThread.run(BBDukS.java:371)
Exception in thread "Thread-5" java.lang.AssertionError
        at stream.PairStreamer.nextList(PairStreamer.java:94)
        at bbduk.BBDukProcessorS.process(BBDukProcessorS.java:743)
        at bbduk.BBDukS$ProcessThread.run(BBDukS.java:371)

which led me to Zac's fantastically thorough #11