custom class MultiMemberGzipReader and other tweaks #16
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
… filedesc header, and split line only on space character
…ing (fixes fundamental problem with old approach, that a read on a short gzip member would read into the following member)
…github.com/internetarchive/warctools/blob/cdx-writer/hanzo/warctools/arc.py
* origin/master: back to dev version number set version=4.10.0 for push to pypi allow failures for python 3.5 and nightly, since they fail now (to be investigated) bump version number so that current master is after latest release on pypi, and add some python versions to .travis.yml Conflicts: .travis.yml setup.py
nlevitt
changed the title
… of the data section, and add tests for incomplete gzip input
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The old GeeZipFile class: https://github.com/internetarchive/warctools/blob/7f2b9a9/hanzo/warctools/stream.py#L183
depended on implementation details of gzip.py in the python standard library. It looks like python 3.5 breaks that class. Moreover, GeeZipFile doesn't work correctly for certain gzip members in any python version (can skip over some very small members, and there might be other issues). It turned out to be necessary to write a new class using zlib, rather than try to extend gzip.GzipFile.
This pull request makes tests pass on python 3.5, see
https://travis-ci.org/internetarchive/warctools/builds/157185857
vs
https://travis-ci.org/internetarchive/warctools/builds/157138379
These are changes I was working as part of the painful project of making CDX-Writer work with mainline surt and warctools libraries. That project isn't finished, but I think we might as well merge this change now.