Wacksy

An experimental Rust library for reading and writing ᴡᴀᴄᴢ files.

Install

With cargo installed, run the following command in your project directory:

cargo add wacksy

Example

This library provides two main ᴀᴘɪ functions. from_files() takes a slice containing one or more ᴡᴀʀᴄ files and returns a structured representation of a ᴡᴀᴄᴢ object. as_zip_archive() takes a ᴡᴀᴄᴢ object and zips it up to a byte array using rawzip.

fn main() -> Result<(), Box<dyn Error>> {
    let warc_file_path = Path::new("example.warc.gz"); // set path to your ᴡᴀʀᴄ file
    let warc_file_path2 = Path::new("example2.warc.gz");

    let wacz_object = WACZ::from_files(&[warc_file_path, warc_file_path2])?; // index the ᴡᴀʀᴄ and create a ᴡᴀᴄᴢ object
    let zipped_wacz: Vec<u8> = wacz_object.as_zip_archive()?; // zip up the ᴡᴀᴄᴢ
    fs::write("example.wacz", zipped_wacz)?; // write out to file
    Ok(())
}

For backwards compatability, WACZ::from_file() is also available and will take a single WARC file

See the documentation for more details.

Background

According to Ed Summers, a ᴡᴀᴄᴢ file is "really just a ᴢɪᴘ file that contains ᴡᴀʀᴄ data and metadata at predictable file locations."¹

The example in the spec outlines what a ᴡᴀᴄᴢ file should contain:

archive
└── data.warc.gz
datapackage.json
datapackage-digest.json
indexes
└── index.cdx.gz
pages
└── pages.jsonl

Similar libraries

• py-wacz for python
• js-wacz for javascript

License

MIT © Bodleian Libraries and contributors

Footnotes

1. For more discussion of the concept, see the talk "Web Archives in Digital Repositories" by Ilya Kremer and Ed Summers at Code4Lib 2022. ↩