Fix revisit content-type

[lfoppiano]

This PR fixed issue #40. I replaced the content type message/http with application/http; msgtype=response.

[sebastian-nagel]

Hi @lfoppiano, thanks! The fix looks good to me.

However, the unit test failed when I tried to run it. Creating a "HTTP 304" response object, resp. the included Content object, might be challenging. See the attached data. It's on you whether to read the response from the segment, or to study it to figure out how it should look like. Use nutch readseg for inspection. There are other Nutch unit tests which use segments.

Otherwise, it's very appreciated that there will be now the first "deeper" unit test for the WARC writer.

[lfoppiano]

@sebastian-nagel the test was left failing on purpose, because I wanted to first fix the test running.

I managed (or, better, Claude Opus managed - other models weren't able) to find the cause.
In short, all test were ran with <fork> and there were two tests in particular TestCommonCrawlDataDumper whcih called at some point System.exit() dumping the whole forked process. However also TestMimeUtil had a simliar problem.

For now I've separated the org.commoncrawl.* from the org.apache.* tests. But the issue may still occur preventing other tests from the same group to be executed.

[sebastian-nagel]

called at some point System.exit()

Thanks! That's a left-over of NUTCH-2852. Unclear why it hits our fork but not upstream Nutch. It should be fixed upstream anyway. I'll also comment on PR #44.

[sebastian-nagel]

Hi @lfoppiano, thanks! The fix to change the Content-Type in the WARC header looks good.

See the inline comments about the unit test code.

[lfoppiano]

@sebastian-nagel I believe now the test works fine with the data from a real case. 🌮 🎉
I think there are only two item that I did not find:

String payloadDigest = "sha1:abc123";
String blockDigest = "sha1:def456";

[sebastian-nagel]

The digests are calculated one level higher in the WarcRecordWriter and then passed to the WarcWriter. No need to test it as part of this PR.

[sebastian-nagel]

Thanks @lfoppiano! Looks good to me.

Please merge!