Ncdump.ncdump fails silent for very, very large output

[rschmunk]

Versions impacted by the bug

5.11.0 SNAPSHOT

What went wrong?

I've long had some capability in Panoply to export the faux CDL output of either a dataset or particular by using NJ's Ncdump class. Whilst looking tonight to extend what Panoply will export, I discovered that Ncdump.ncdump will fail silent if the output is extremely large, as in when it gets close to 2,147,483,647 chars and then wants to go further.

The reason for this is that Ncdump first writes everything out to memory by continually appending text to a Formatter object and then at the end outputs the content of the Formatter to the Writer specified in the constructor. But Formatter relies on StringBuilder, and a StringBuilder has a maximum size of 2,147,483,647 chars, i.e., Integer.MAX_VALUE.

Which, okay, that's a lot of text and who other than me would want to ncdump that much out? But what made the problem a hassle to figure out was that Ncdump doesn't test for this situation at all and instead fails apparently silently. There are some checks here and there for, e.g., IOExceptions, but when this particular problem arises, an OutOfMemoryError is thrown and not a caught Exception of any ilk.

Since figuring that much out an hour ago, I've had one or two ideas in how to handle (maybe fix) the situation. However, the use of a Builder in Ncdump adds an extra layer of tricksiness. So it would be good to hear other thoughts about this.

In the appended stacktrace, the NNNN line number in Ncdump is because numbering was way off from the current repo after I added a lot of logging. Some of that code was to actually catch the Error and extract the stacktrace.

And FWIW, the file I was testing includes a structure array with about 93 million members. Not the same file I referenced in #1527, but similar.

Relevant stack trace

java.lang.OutOfMemoryError: Required array length 2147483644 + 8 is too large
	at java.base/jdk.internal.util.ArraysSupport.hugeLength(ArraysSupport.java:649)
	at java.base/jdk.internal.util.ArraysSupport.newLength(ArraysSupport.java:642)
	at java.base/java.lang.AbstractStringBuilder.newCapacity(AbstractStringBuilder.java:257)
	at java.base/java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:229)
	at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:582)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:179)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:91)
	at java.base/java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:622)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:209)
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:91)
	at java.base/java.util.Formatter$FormatSpecifier.appendJustified(Formatter.java:3093)
	at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:3083)
	at java.base/java.util.Formatter$FormatSpecifier.printString(Formatter.java:3056)
	at java.base/java.util.Formatter$FormatSpecifier.print(Formatter.java:2933)
	at java.base/java.util.Formatter.format(Formatter.java:2689)
	at java.base/java.util.Formatter.format(Formatter.java:2625)
	at ucar.nc2.write.Ncdump.printArray(Ncdump.java:NNNN)

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

• I agree to follow the UCAR/Unidata Code of Conduct

[rschmunk]

Belatedly realizing that there are separate Ncdump and NCdumpW classes, and that the latter outputs to the target Writer more often.

Checking my own code history, I find that I used to use NCdumpW to export CDL but switched over to using Ncdump in summer of 2020. Unfortunately, my notes don't indicate why I made that switch.

Edit: Looks like Ncdump was added to NJ in March 2020, and I switched over because NCdumpW was then deprecated.

[rschmunk]

And BTW, the link at the start of CDLWriter is 404. Maybe it should be updated to https://docs.unidata.ucar.edu/nug/current/_c_d_l.html ?