diff --git a/.github/workflows/gradle.yml b/.github/workflows/gradle.yml index ba8a9308..fe8a5ce3 100644 --- a/.github/workflows/gradle.yml +++ b/.github/workflows/gradle.yml @@ -11,7 +11,6 @@ on: push: branches: [ main, development ] pull_request: - branches: [ main, development ] workflow_dispatch: env: diff --git a/README.md b/README.md index e5613a23..6d896722 100644 --- a/README.md +++ b/README.md @@ -13,7 +13,7 @@ and avoiding crates which do not fully comply to the specification, at the same - [Instructions for your build manager (e.g., Gradle, Maven, etc.)](https://central.sonatype.com/artifact/edu.kit.datamanager/ro-crate-java) - [Quick-Start](#quick-start) -- [Adapting Specification Examples](#adapting-the-specification-examples) +- [JavaDoc Documentation](https://javadoc.io/doc/edu.kit.datamanager/ro-crate-java) - [Related Publications](https://publikationen.bibliothek.kit.edu/publikationslisten/get.php?referencing=all&external_publications=kit&lang=de&format=html&style=kit-3lines-title_b-authors-other&consider_suborganizations=true&order=desc%20year&contributors=%5B%5B%5B%5D%2C%5B%22p20751.105%22%5D%5D%5D&title_contains=crate) ## Build the library / documentation @@ -31,687 +31,34 @@ On Windows, replace `./gradlew` with `gradlew.bat`. ## RO-Crate Specification Compatibility -- ✅ Version 1.1 +- ✅ [Version 1.1](https://www.researchobject.org/ro-crate/1.1/) ([Extracted examples as well-described unit tests/guide](src/test/java/edu/kit/datamanager/ro_crate/examples/ExamplesOfSpecificationV1p1Test.java)) - 🛠️ Version 1.2-DRAFT - ✅ Reading and writing crates with additional profiles or specifications ([examples for reading](src/test/java/edu/kit/datamanager/ro_crate/reader/RoCrateReaderSpec12Test.java), [examples for writing](src/test/java/edu/kit/datamanager/ro_crate/writer/RoCrateWriterSpec12Test.java)) - ✅ Adding profiles or other specifications to a crate ([examples](src/test/java/edu/kit/datamanager/ro_crate/crate/BuilderSpec12Test.java)) ## Quick-start -### Example for a basic crate from [RO-Crate website](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#ro-crate-metadata-file-descriptor) -```java -RoCrate roCrate = new RoCrateBuilder("name", "description", "datePublished", "licenseIdentifier").build(); -``` -### Example adding a File (Data Entity) and a context pair +` ro-crate-java` makes use of the builder pattern to guide the user to create a valid RO-Crate, similar to: + ```java -RoCrate roCrate = new RoCrateBuilder("name", "description", "datePublished", "licenseIdentifier") - .addValuePairToContext("Station", "www.station.com") - .addUrlToContext("contextUrl") +RoCrate myFirstCrate = STARTER_CRATE .addDataEntity( - new FileEntity.FileEntityBuilder() - .setId("survey-responses-2019.csv") - .addProperty("name", "Survey responses") - .addProperty("contentSize", "26452") - .addProperty("encodingFormat", "text/csv") - .build() + new FileEntity.FileEntityBuilder() + .setId("path/within/crate/survey-responses-2019.csv") + .setLocation(Paths.get("path/to/current/location/experiment.csv")) + .addProperty("name", "Survey responses") + .addProperty("contentSize", "26452") + .addProperty("encodingFormat", "text/csv") + .build() ) - .addDataEntity(...) - ... - .addContextualEntity(...) - ... - .build(); -``` - -The library currently comes with three specialized DataEntities: - -1. `DataSetEntity` -2. `FileEntity` (used in the example above) -3. `WorkflowEntity` - -If another type of `DataEntity` is required, the base class `DataEntity` can be used. Example: -```java -new DataEntity.DataEntityBuilder() - .addType("CreativeWork") - .setId("ID") - .addProperty("property from schema.org/Creativework", "value") - .build(); -``` -Note that here you are supposed to add the type of your `DataEntity` because it is not known. - -A `DataEntity` and its subclasses can have a file located on the web. Example: - -Example adding file: -```java -new FileEntity.FileEntityBuilder() - .addContent(URI.create("https://github.com/kit-data-manager/ro-crate-java/issues/5")) - .addProperty("description", "my new file that I added") - .build(); -``` - -A `DataEntity` and its subclasses can have a local file associated with them, -instead of one located on the web (which link is the ID of the data entity). Example: - -Example adding file: -```java -new FileEntity.FileEntityBuilder() - .addContent(Paths.get("file"), "new_file.txt") - .addProperty("description", "my new local file that I added") - .build(); -``` - -### Contextual Entities - -Contextual entities cannot be associated with a file (they are pure metadata). - -To add a contextual entity to a crate you use the function `.addContextualEntity(ContextualEntity entity)`. -Some types of derived/specializes entities are: -1. `OrganizationEntity` -2. `PersonEntity` -3. `PlaceEntity` - -If you need another type of contextual entity, use the base class `ContextualEntity`. - -The library provides a way to automatically create contextual entities from external providers. Currently, support for [ORCID](https://orcid.org/) and [ROR](https://ror.org/) is implemented. Example: -```java -PersonEntity person = ORCIDProvider.getPerson("https://orcid.org/*") -OrganizationEntity organization = RORProvider.getOrganization("https://ror.org/*"); -``` - -### Writing Crate to folder, zip file, or zip stream - -Writing to folder: -```java -RoCrateWriter folderRoCrateWriter = new RoCrateWriter(new FolderWriter()); -folderRoCrateWriter.save(roCrate, "destinationFolder"); -``` - -Writing to zip file: -```java -RoCrateWriter roCrateZipWriter = new RoCrateWriter(new ZipWriter()); -roCrateZipWriter.save(roCrate, "destinationFolder"); -``` - -Writing to zip stream: -```java -RoCrateWriter roCrateZipStreamWriter = new RoCrateWriter(new ZipStreamWriter()); -roCrateZipStreamWriter.save(roCrate, outputStream); -``` - -More writing strategies can be implemented, if required. - -### Reading / importing Crate from folder or zip - -Reading from folder: -```java -RoCrateReader roCrateFolderReader = new RoCrateReader(new FolderReader()); -RoCrate res = roCrateFolderReader.readCrate("destinationFolder"); -``` - -Reading from zip file: -```java -RoCrateReader roCrateFolderReader = new RoCrateReader(new ZipReader()); -RoCrate crate = roCrateFolderReader.readCrate("sourceZipFile"); -``` - -Reading from zip stream: -```java -RoCrateReader roCrateFolderReader = new RoCrateReader(new ZipStreamReader()); -RoCrate crate = roCrateFolderReader.readCrate(inputStream); -``` - -### RO-Crate Website (HTML preview file) -ro-crate-java offers tree different kinds of previews: - -* AutomaticPreview: Uses third-party library [ro-crate-html-js](https://www.npmjs.com/package/ro-crate-html-js), which must be installed separately. -* CustomPreview: Pure Java-based preview using an included template processed by the FreeMarker template engine. At the same time, CustomPreview is the fallback for AutomaticPreview if ro-crate-html-js is not installed. -* StaticPreview: Allows to provide a static HTML page (including additional dependencies, e.g., CSS, JS) which is then shipped with the RO-Crate. - -When creating a new RO-Crate using the builder, the default setting is to use CustomPreview. If you want to change this behaviour, thr preview method is set as follows: - -```java -RoCrate roCrate = new RoCrateBuilder("name", "description", "datePublished", "licenseIdentifier") - .setPreview(new AutomaticPreview()) + .addDataEntity(/*...*/) + .addContextualEntity(/*...*/) .build(); ``` -Keep in mind that, if you want to use AutomaticPreview, you have to install ro-crate-html-js via `npm install --global ro-crate-html-js` first. +A built or imported crate can of course also be modified afterwards. Take a look at our further documentation: -For StaticPreview, the constuctor is a bit different, such that it looks as follows: - -```java -File pathToMainPreviewHtml = new File("localPath"); -File pathToAdditionalFiles = new File("localFolder"); -RoCrate roCrate = new RoCrateBuilder("name", "description", "datePublished", "licenseIdentifier") - .setPreview(new StaticPreview(pathToMainPreviewHtml, pathToAdditionalFiles)) - .build(); -``` - -### RO-Crate validation (machine-readable crate profiles) -Right now, the only implemented way of validating a RO-crate is to use a [JSON-Schema](https://json-schema.org/) that the crates metadata JSON file should match. JSON-Schema is an established standard and therefore a good choice for a crate profile. Example: - -```java -Validator validator = new Validator(new JsonSchemaValidation("./schema.json")); -boolean valid = validator.validate(crate); -``` - -## Adapting the specification examples - -This section describes how to generate the [official specifications examples](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#minimal-example-of-ro-crate). Each example first shows the ro-crate-metadata.json and, below that, the required Java code to generate it. - -### [Minimal example](https://www.researchobject.org/ro-crate/1.1/root-data-entity.html#minimal-example-of-ro-crate) - -```json -{ "@context": "https://w3id.org/ro/crate/1.1/context", - "@graph": [ - - { - "@type": "CreativeWork", - "@id": "ro-crate-metadata.json", - "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, - "about": {"@id": "./"} - }, - { - "@id": "./", - "identifier": "https://doi.org/10.4225/59/59672c09f4a4b", - "@type": "Dataset", - "datePublished": "2017", - "name": "Data files associated with the manuscript:Effects of facilitated family case conferencing for ...", - "description": "Palliative care planning for nursing home residents with advanced dementia ...", - "license": {"@id": "https://creativecommons.org/licenses/by-nc-sa/3.0/au/"} - }, - { - "@id": "https://creativecommons.org/licenses/by-nc-sa/3.0/au/", - "@type": "CreativeWork", - "description": "This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Australia License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/au/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.", - "identifier": "https://creativecommons.org/licenses/by-nc-sa/3.0/au/", - "name": "Attribution-NonCommercial-ShareAlike 3.0 Australia (CC BY-NC-SA 3.0 AU)" - } - ] -} -``` - -Here, everything is created manually. -For the following examples, more convenient creation methods are used. - -```java - RoCrate crate = new RoCrate(); - - ContextualEntity license = new ContextualEntity.ContextualEntityBuilder() - .addType("CreativeWork") - .setId("https://creativecommons.org/licenses/by-nc-sa/3.0/au/") - .addProperty("description", "This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Australia License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/au/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.") - .addProperty("identifier", "https://creativecommons.org/licenses/by-nc-sa/3.0/au/") - .addProperty("name", "Attribution-NonCommercial-ShareAlike 3.0 Australia (CC BY-NC-SA 3.0 AU)") - .build(); - - crate.setRootDataEntity(new RootDataEntity.RootDataEntityBuilder() - .addProperty("identifier", "https://doi.org/10.4225/59/59672c09f4a4b") - .addProperty("datePublished", "2017") - .addProperty("name", "Data files associated with the manuscript:Effects of facilitated family case conferencing for ...") - .addProperty("description", "Palliative care planning for nursing home residents with advanced dementia ...") - .setLicense(license) - .build()); - - crate.setJsonDescriptor(new ContextualEntity.ContextualEntityBuilder() - .setId("ro-crate-metadata.json") - .addType("CreativeWork") - .addIdProperty("about", "./") - .addIdProperty("conformsTo", "https://w3id.org/ro/crate/1.1") - .build() - ); - crate.addContextualEntity(license); -``` - -### [Example with files](https://www.researchobject.org/ro-crate/1.1/data-entities.html#example-linking-to-a-file-and-folders) - -```json -{ "@context": "https://w3id.org/ro/crate/1.1/context", - "@graph": [ - { - "@type": "CreativeWork", - "@id": "ro-crate-metadata.json", - "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, - "about": {"@id": "./"} - }, - { - "@id": "./", - "@type": [ - "Dataset" - ], - "hasPart": [ - { - "@id": "cp7glop.ai" - }, - { - "@id": "lots_of_little_files/" - } - ] - }, - { - "@id": "cp7glop.ai", - "@type": "File", - "name": "Diagram showing trend to increase", - "contentSize": "383766", - "description": "Illustrator file for Glop Pot", - "encodingFormat": "application/pdf" - }, - { - "@id": "lots_of_little_files/", - "@type": "Dataset", - "name": "Too many files", - "description": "This directory contains many small files, that we're not going to describe in detail." - } - ] -} -``` - -Here we use the inner builder classes for the construction of the crate. -Doing so, the Metadata File Descriptor and the Root Data Entity entities are added automatically. -`setSource()` is used to provide the actual location of these Data Entities (if they are not remote). -The Data Entity file in the crate will have the name of the entity's ID. - -```java - RoCrate crate = new RoCrate.RoCrateBuilder() - .addDataEntity( - new FileEntity.FileEntityBuilder() - .addContent (Paths.get("path to file"), "cp7glop.ai") - .addProperty("name", "Diagram showing trend to increase") - .addProperty("contentSize", "383766") - .addProperty("description", "Illustrator file for Glop Pot") - .setEncodingFormat("application/pdf") - .build() - ) - .addDataEntity( - new DataSetEntity.DataSetBuilder() - .addContent (Paths.get("path_to_files"), "lots_of_little_files/") - .addProperty("name", "Too many files") - .addProperty("description", "This directory contains many small files, that we're not going to describe in detail.") - .build() - ) - .build(); -``` - -### [Example with web resources](https://www.researchobject.org/ro-crate/1.1/data-entities.html#web-based-data-entities) - -```json -{ "@context": "https://w3id.org/ro/crate/1.1/context", - "@graph": [ - { - "@type": "CreativeWork", - "@id": "ro-crate-metadata.json", - "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, - "about": {"@id": "./"} - }, - { - "@id": "./", - "@type": [ - "Dataset" - ], - "hasPart": [ - { - "@id": "survey-responses-2019.csv" - }, - { - "@id": "https://zenodo.org/record/3541888/files/ro-crate-1.0.0.pdf" - }, - ] - }, - { - "@id": "survey-responses-2019.csv", - "@type": "File", - "name": "Survey responses", - "contentSize": "26452", - "encodingFormat": "text/csv" - }, - { - "@id": "https://zenodo.org/record/3541888/files/ro-crate-1.0.0.pdf", - "@type": "File", - "name": "RO-Crate specification", - "contentSize": "310691", - "description": "RO-Crate specification", - "encodingFormat": "application/pdf" - } -] -} -``` - -The web resource does not use `.setSource()`, but uses the ID to indicate the file's location. - -```java - RoCrate crate = new RoCrate.RoCrateBuilder() - .addDataEntity( - new FileEntity.FileEntityBuilder() - .addContent (Paths.get("README.md"), "survey-responses-2019.csv") - .addProperty("name", "Survey responses") - .addProperty("contentSize", "26452") - .setEncodingFormat("text/csv") - .build() - ) - .addDataEntity( - new FileEntity.FileEntityBuilder() - .addContent(URI.create("https://zenodo.org/record/3541888/files/ro-crate-1.0.0.pdf")) - .addProperty("name", "RO-Crate specification") - .addProperty("contentSize", "310691") - .addProperty("description", "RO-Crate specification") - .setEncodingFormat("application/pdf") - .build() - ) - .build(); -``` - -### [Example with file, author, location](https://www.researchobject.org/ro-crate/1.1/appendix/jsonld.html) - -```json -{ "@context": "https://w3id.org/ro/crate/1.1/context", - "@graph": [ - - { - "@type": "CreativeWork", - "@id": "ro-crate-metadata.json", - "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, - "about": {"@id": "./"}, - "description": "RO-Crate Metadata File Descriptor (this file)" - }, - { - "@id": "./", - "@type": "Dataset", - "name": "Example RO-Crate", - "description": "The RO-Crate Root Data Entity", - "datePublished": "2020", - "license": {"@id": "https://spdx.org/licenses/CC-BY-NC-SA-4.0"}, - "hasPart": [ - {"@id": "data1.txt"}, - {"@id": "data2.txt"} - ] - }, - { - "@id": "data1.txt", - "@type": "File", - "description": "One of hopefully many Data Entities", - "author": {"@id": "#alice"}, - "contentLocation": {"@id": "http://sws.geonames.org/8152662/"} - }, - { - "@id": "data2.txt", - "@type": "File" - }, - - { - "@id": "#alice", - "@type": "Person", - "name": "Alice", - "description": "One of hopefully many Contextual Entities" - }, - { - "@id": "http://sws.geonames.org/8152662/", - "@type": "Place", - "name": "Catalina Park" - } - ] -} -``` - -If there is no special method for including relative entities (ID properties) one can use `.addIdProperty("key","value")`. - -```java - PersonEntity alice = new PersonEntity.PersonEntityBuilder() - .setId("#alice") - .addProperty("name", "Alice") - .addProperty("description", "One of hopefully many Contextual Entities") - .build(); - PlaceEntity park = new PlaceEntity.PlaceEntityBuilder() - .addContent(URI.create("http://sws.geonames.org/8152662/")) - .addProperty("name", "Catalina Park") - .build(); - - RoCrate crate = new RoCrate.RoCrateBuilder("Example RO-Crate", "The RO-Crate Root Data Entity", "2020", "https://spdx.org/licenses/CC-BY-NC-SA-4.0") - .addContextualEntity(park) - .addContextualEntity(alice) - .addDataEntity( - new FileEntity.FileEntityBuilder() - .addContent(Paths.get("......."), "data2.txt") - .build() - ) - .addDataEntity( - new FileEntity.FileEntityBuilder() - .addContent(Paths.get("......."), "data1.txt") - .addProperty("description", "One of hopefully many Data Entities") - .addAuthor(alice.getId()) - .addIdProperty("contentLocation", park) - .build() - ) - .build(); - -``` -### [Example with computational workflow](https://www.researchobject.org/ro-crate/1.1/workflows.html#complete-workflow-example) - -```json -{ "@context": "https://w3id.org/ro/crate/1.1/context", - "@graph": [ - { - "@type": "CreativeWork", - "@id": "ro-crate-metadata.json", - "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, - "about": {"@id": "./"} - }, - { - "@id": "./", - "@type": "Dataset", - "name": "Example RO-Crate", - "description": "The RO-Crate Root Data Entity", - "datePublished": "2020", - "license": {"@id": "https://spdx.org/licenses/CC-BY-NC-SA-4.0"}, - "hasPart": [ - { "@id": "workflow/alignment.knime" } - ] - }, - { - "@id": "workflow/alignment.knime", - "@type": ["File", "SoftwareSourceCode", "ComputationalWorkflow"], - "conformsTo": - {"@id": "https://bioschemas.org/profiles/ComputationalWorkflow/0.5-DRAFT-2020_07_21/"}, - "name": "Sequence alignment workflow", - "programmingLanguage": {"@id": "#knime"}, - "creator": {"@id": "#alice"}, - "dateCreated": "2020-05-23", - "license": { "@id": "https://spdx.org/licenses/CC-BY-NC-SA-4.0"}, - "input": [ - { "@id": "#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b"} - ], - "output": [ - { "@id": "#6c703fee-6af7-4fdb-a57d-9e8bc4486044"}, - { "@id": "#2f32b861-e43c-401f-8c42-04fd84273bdf"} - ], - "sdPublisher": {"@id": "#workflow-hub"}, - "url": "http://example.com/workflows/alignment", - "version": "0.5.0" - }, - { - "@id": "#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b", - "@type": "FormalParameter", - "conformsTo": {"@id": "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/"}, - "name": "genome_sequence", - "valueRequired": true, - "additionalType": {"@id": "http://edamontology.org/data_2977"}, - "format": {"@id": "http://edamontology.org/format_1929"} - }, - { - "@id": "#6c703fee-6af7-4fdb-a57d-9e8bc4486044", - "@type": "FormalParameter", - "conformsTo": {"@id": "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/"}, - "name": "cleaned_sequence", - "additionalType": {"@id": "http://edamontology.org/data_2977"}, - "encodingFormat": {"@id": "http://edamontology.org/format_2572"} - }, - { - "@id": "#2f32b861-e43c-401f-8c42-04fd84273bdf", - "@type": "FormalParameter", - "conformsTo": {"@id": "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/"}, - "name": "sequence_alignment", - "additionalType": {"@id": "http://edamontology.org/data_1383"}, - "encodingFormat": {"@id": "http://edamontology.org/format_1982"} - }, - { - "@id": "https://spdx.org/licenses/CC-BY-NC-SA-4.0", - "@type": "CreativeWork", - "name": "Creative Commons Attribution Non Commercial Share Alike 4.0 International", - "alternateName": "CC-BY-NC-SA-4.0" - }, - { - "@id": "#knime", - "@type": "ProgrammingLanguage", - "name": "KNIME Analytics Platform", - "alternateName": "KNIME", - "url": "https://www.knime.com/whats-new-in-knime-41", - "version": "4.1.3" - }, - { - "@id": "#alice", - "@type": "Person", - "name": "Alice Brown" - }, - { - "@id": "#workflow-hub", - "@type": "Organization", - "name": "Example Workflow Hub", - "url":"http://example.com/workflows/" - }, - { - "@id": "http://edamontology.org/format_1929", - "@type": "Thing", - "name": "FASTA sequence format" - }, - { - "@id": "http://edamontology.org/format_1982", - "@type": "Thing", - "name": "ClustalW alignment format" - }, - { - "@id": "http://edamontology.org/format_2572", - "@type": "Thing", - "name": "BAM format" - }, - { - "@id": "http://edamontology.org/data_2977", - "@type": "Thing", - "name": "Nucleic acid sequence" - }, - { - "@id": "http://edamontology.org/data_1383", - "@type": "Thing", - "name": "Nucleic acid sequence alignment" - } - ] -} -``` - - -```java - ContextualEntity license = new ContextualEntity.ContextualEntityBuilder() - .addType("CreativeWork") - .setId("https://spdx.org/licenses/CC-BY-NC-SA-4.0") - .addProperty("name", "Creative Commons Attribution Non Commercial Share Alike 4.0 International") - .addProperty("alternateName", "CC-BY-NC-SA-4.0") - .build(); - ContextualEntity knime = new ContextualEntity.ContextualEntityBuilder() - .setId("#knime") - .addType("ProgrammingLanguage") - .addProperty("name", "KNIME Analytics Platform") - .addProperty("alternateName", "KNIME") - .addProperty("url", "https://www.knime.com/whats-new-in-knime-41") - .addProperty("version", "4.1.3") - .build(); - OrganizationEntity workflowHub = new OrganizationEntity.OrganizationEntityBuilder() - .setId("#workflow-hub") - .addProperty("name", "Example Workflow Hub") - .addProperty("url", "http://example.com/workflows/") - .build(); - ContextualEntity fasta = new ContextualEntity.ContextualEntityBuilder() - .setId("http://edamontology.org/format_1929") - .addType("Thing") - .addProperty("name", "FASTA sequence format") - .build(); - ContextualEntity clustalW = new ContextualEntity.ContextualEntityBuilder() - .setId("http://edamontology.org/format_1982") - .addType("Thing") - .addProperty("name", "ClustalW alignment format") - .build(); - ContextualEntity ban = new ContextualEntity.ContextualEntityBuilder() - .setId("http://edamontology.org/format_2572") - .addType("Thing") - .addProperty("name", "BAM format") - .build(); - ContextualEntity nucSec = new ContextualEntity.ContextualEntityBuilder() - .setId("http://edamontology.org/data_2977") - .addType("Thing") - .addProperty("name", "Nucleic acid sequence") - .build(); - ContextualEntity nucAlign = new ContextualEntity.ContextualEntityBuilder() - .setId("http://edamontology.org/data_1383") - .addType("Thing") - .addProperty("name", "Nucleic acid sequence alignment") - .build(); - PersonEntity alice = new PersonEntity.PersonEntityBuilder() - .setId("#alice") - .addProperty("name", "Alice Brown") - .build(); - ContextualEntity requiredParam = new ContextualEntity.ContextualEntityBuilder() - .addType("FormalParameter") - .setId("#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b") - .addProperty("name", "genome_sequence") - .addProperty("valueRequired", true) - .addIdProperty("conformsTo", "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/") - .addIdProperty("additionalType", nucSec) - .addIdProperty("encodingFormat", fasta) - .build(); - ContextualEntity clnParam = new ContextualEntity.ContextualEntityBuilder() - .addType("FormalParameter") - .setId("#6c703fee-6af7-4fdb-a57d-9e8bc4486044") - .addProperty("name", "cleaned_sequence") - .addIdProperty("conformsTo", "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/") - .addIdProperty("additionalType", nucSec) - .addIdProperty("encodingFormat", ban) - .build(); - ContextualEntity alignParam = new ContextualEntity.ContextualEntityBuilder() - .addType("FormalParameter") - .setId("#2f32b861-e43c-401f-8c42-04fd84273bdf") - .addProperty("name", "sequence_alignment") - .addIdProperty("conformsTo", "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/") - .addIdProperty("additionalType", nucAlign) - .addIdProperty("encodingFormat", clustalW) - .build(); - - RoCrate crate = new RoCrate.RoCrateBuilder("Example RO-Crate", "The RO-Crate Root Data Entity", "2020", "https://spdx.org/licenses/CC-BY-NC-SA-4.0") - .addContextualEntity(license) - .addContextualEntity(knime) - .addContextualEntity(workflowHub) - .addContextualEntity(fasta) - .addContextualEntity(clustalW) - .addContextualEntity(ban) - .addContextualEntity(nucSec) - .addContextualEntity(nucAlign) - .addContextualEntity(alice) - .addContextualEntity(requiredParam) - .addContextualEntity(clnParam) - .addContextualEntity(alignParam) - .addDataEntity( - new WorkflowEntity.WorkflowEntityBuilder() - .setId("workflow/alignment.knime") - .setSource(new File("src")) - .addIdProperty("conformsTo", "https://bioschemas.org/profiles/ComputationalWorkflow/0.5-DRAFT-2020_07_21/") - .addProperty("name", "Sequence alignment workflow") - .addIdProperty("programmingLanguage", "#knime") - .addAuthor("#alice") - .addProperty("dateCreated", "2020-05-23") - .setLicense("https://spdx.org/licenses/CC-BY-NC-SA-4.0") - .addInput("#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b") - .addOutput("#6c703fee-6af7-4fdb-a57d-9e8bc4486044") - .addOutput("#2f32b861-e43c-401f-8c42-04fd84273bdf") - .addProperty("url", "http://example.com/workflows/alignment") - .addProperty("version", "0.5.0") - .addIdProperty("sdPublisher", "#workflow-hub") - .build() - - ) - .build(); -``` +- **There is a well-documented example-driven guide in [LearnByExampleTest.java](src/test/java/edu/kit/datamanager/ro_crate/examples/LearnByExampleTest.java) to help you get started.** +- You may also be interested in the examples we extracted from the [specification in version 1.1](https://www.researchobject.org/ro-crate/1.1/), which are available in [ExamplesOfSpecificationV1p1Test.java](src/test/java/edu/kit/datamanager/ro_crate/examples/ExamplesOfSpecificationV1p1Test.java). +- There is a [module with all well-described guiding tests](src/test/java/edu/kit/datamanager/ro_crate/examples/) available. +- The [JavaDoc Documentation](https://javadoc.io/doc/edu.kit.datamanager/ro-crate-java) is also available online. diff --git a/build.gradle b/build.gradle index cd6c189f..f7fcd613 100644 --- a/build.gradle +++ b/build.gradle @@ -38,7 +38,7 @@ repositories { } ext { - jacksonVersion = '2.18.3' + jacksonVersion = '2.19.0' } dependencies { @@ -67,10 +67,15 @@ dependencies { implementation group: "com.networknt", name: "json-schema-validator", version: "1.5.6" implementation 'org.glassfish:jakarta.json:2.0.1' //JTE for template processing - implementation('gg.jte:jte:3.2.0') + implementation('gg.jte:jte:3.2.1') implementation("org.freemarker:freemarker:2.3.34") } +// enable -Xlint:deprecation +tasks.withType(JavaCompile).configureEach { + options.compilerArgs << "-Xlint:deprecation" +} + logging.captureStandardOutput LogLevel.INFO def signingTasks = tasks.withType(Sign) diff --git a/gradle/wrapper/gradle-wrapper.jar b/gradle/wrapper/gradle-wrapper.jar index 9bbc975c..1b33c55b 100644 Binary files a/gradle/wrapper/gradle-wrapper.jar and b/gradle/wrapper/gradle-wrapper.jar differ diff --git a/gradle/wrapper/gradle-wrapper.properties b/gradle/wrapper/gradle-wrapper.properties index 37f853b1..ca025c83 100644 --- a/gradle/wrapper/gradle-wrapper.properties +++ b/gradle/wrapper/gradle-wrapper.properties @@ -1,6 +1,6 @@ distributionBase=GRADLE_USER_HOME distributionPath=wrapper/dists -distributionUrl=https\://services.gradle.org/distributions/gradle-8.13-bin.zip +distributionUrl=https\://services.gradle.org/distributions/gradle-8.14-bin.zip networkTimeout=10000 validateDistributionUrl=true zipStoreBase=GRADLE_USER_HOME diff --git a/gradlew b/gradlew index faf93008..23d15a93 100755 --- a/gradlew +++ b/gradlew @@ -114,7 +114,7 @@ case "$( uname )" in #( NONSTOP* ) nonstop=true ;; esac -CLASSPATH=$APP_HOME/gradle/wrapper/gradle-wrapper.jar +CLASSPATH="\\\"\\\"" # Determine the Java command to use to start the JVM. @@ -213,7 +213,7 @@ DEFAULT_JVM_OPTS='"-Xmx64m" "-Xms64m"' set -- \ "-Dorg.gradle.appname=$APP_BASE_NAME" \ -classpath "$CLASSPATH" \ - org.gradle.wrapper.GradleWrapperMain \ + -jar "$APP_HOME/gradle/wrapper/gradle-wrapper.jar" \ "$@" # Stop when "xargs" is not available. diff --git a/gradlew.bat b/gradlew.bat index 9b42019c..5eed7ee8 100644 --- a/gradlew.bat +++ b/gradlew.bat @@ -70,11 +70,11 @@ goto fail :execute @rem Setup the command line -set CLASSPATH=%APP_HOME%\gradle\wrapper\gradle-wrapper.jar +set CLASSPATH= @rem Execute Gradle -"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" org.gradle.wrapper.GradleWrapperMain %* +"%JAVA_EXE%" %DEFAULT_JVM_OPTS% %JAVA_OPTS% %GRADLE_OPTS% "-Dorg.gradle.appname=%APP_BASE_NAME%" -classpath "%CLASSPATH%" -jar "%APP_HOME%\gradle\wrapper\gradle-wrapper.jar" %* :end @rem End local scope for the variables with windows NT shell diff --git a/src/main/java/edu/kit/datamanager/ro_crate/RoCrate.java b/src/main/java/edu/kit/datamanager/ro_crate/RoCrate.java index 330135b7..356b0159 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/RoCrate.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/RoCrate.java @@ -10,13 +10,10 @@ import edu.kit.datamanager.ro_crate.entities.AbstractEntity; import edu.kit.datamanager.ro_crate.entities.contextual.ContextualEntity; import edu.kit.datamanager.ro_crate.entities.contextual.JsonDescriptor; -import edu.kit.datamanager.ro_crate.entities.contextual.OrganizationEntity; import edu.kit.datamanager.ro_crate.entities.data.DataEntity; -import edu.kit.datamanager.ro_crate.entities.data.DataEntity.DataEntityBuilder; -import edu.kit.datamanager.ro_crate.entities.data.FileEntity; + import edu.kit.datamanager.ro_crate.entities.data.RootDataEntity; import edu.kit.datamanager.ro_crate.externalproviders.dataentities.ImportFromDataCite; -import edu.kit.datamanager.ro_crate.externalproviders.organizationprovider.RorProvider; import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; import edu.kit.datamanager.ro_crate.payload.CratePayload; import edu.kit.datamanager.ro_crate.payload.RoCratePayload; @@ -26,12 +23,9 @@ import edu.kit.datamanager.ro_crate.special.JsonUtilFunctions; import edu.kit.datamanager.ro_crate.validation.JsonSchemaValidation; import edu.kit.datamanager.ro_crate.validation.Validator; -import edu.kit.datamanager.ro_crate.writer.FolderWriter; -import edu.kit.datamanager.ro_crate.writer.RoCrateWriter; import java.io.File; import java.net.URI; -import java.nio.file.Paths; import java.util.*; import java.util.stream.Collectors; import java.util.stream.StreamSupport; @@ -354,6 +348,18 @@ public RoCrateBuilder addName(String name) { return this; } + /** + * Adds an "identifier" property to the root data entity. + *

+ * This is useful e.g. to assign e.g. a DOI to this crate. + * @param identifier the identifier to add. + * @return this builder. + */ + public RoCrateBuilder addIdentifier(String identifier) { + this.rootDataEntity.addProperty("identifier", identifier.strip()); + return this; + } + public RoCrateBuilder addDescription(String description) { this.rootDataEntity.addProperty(PROPERTY_DESCRIPTION, description); return this; diff --git a/src/main/java/edu/kit/datamanager/ro_crate/context/RoCrateMetadataContext.java b/src/main/java/edu/kit/datamanager/ro_crate/context/RoCrateMetadataContext.java index 731a4c1a..cce6a7e9 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/context/RoCrateMetadataContext.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/context/RoCrateMetadataContext.java @@ -113,9 +113,10 @@ public boolean checkEntity(AbstractEntity entity) { node.remove("@id"); node.remove("@type"); - Set types = objectMapper.convertValue(entity.getProperties().get("@type"), - new TypeReference<>() { - }); + Set types = objectMapper.convertValue( + entity.getProperties().path("@type"), + new TypeReference<>() {} + ); // check if the items in the array of types are present in the context for (String s : types) { // special cases: @@ -174,15 +175,14 @@ public void addToContextFromUrl(String url) { } } if (jsonNode == null) { - CloseableHttpClient httpclient = HttpClients.createDefault(); HttpGet httpGet = new HttpGet(url); CloseableHttpResponse response; - try { + try (CloseableHttpClient httpclient = HttpClients.createDefault()) { response = httpclient.execute(httpGet); jsonNode = objectMapper.readValue(response.getEntity().getContent(), JsonNode.class); } catch (IOException e) { - System.err.println(String.format("Cannot get context from url %s", url)); + System.err.printf("Cannot get context from url %s%n", url); return; } if (url.equals(DEFAULT_CONTEXT)) { diff --git a/src/main/java/edu/kit/datamanager/ro_crate/entities/AbstractEntity.java b/src/main/java/edu/kit/datamanager/ro_crate/entities/AbstractEntity.java index 580e625b..f0c75763 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/entities/AbstractEntity.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/entities/AbstractEntity.java @@ -239,33 +239,61 @@ private static boolean addProperty(ObjectNode whereToAdd, String key, JsonNode v * @param id the "id" of the property. */ public void addIdProperty(String name, String id) { - JsonNode jsonNode = addToIdProperty(name, id, this.properties.get(name)); - if (jsonNode != null) { - this.linkedTo.add(id); - this.properties.set(name, jsonNode); - this.notifyObservers(); - } + if (id == null || id.isBlank()) { return; } + mergeIdIntoValue(id, this.properties.get(name)) + .ifPresent(newValue -> { + this.properties.set(name, newValue); + }); + this.linkedTo.add(id); + this.notifyObservers(); } - private static JsonNode addToIdProperty(String name, String id, JsonNode property) { - ObjectMapper objectMapper = MyObjectMapper.getMapper(); - if (name != null && id != null) { - if (property == null) { - return objectMapper.createObjectNode().put("@id", id); - } else { - if (property.isArray()) { - ArrayNode ns = (ArrayNode) property; - ns.add(objectMapper.createObjectNode().put("@id", id)); - return ns; - } else { - ArrayNode newNodes = objectMapper.createArrayNode(); - newNodes.add(property); - newNodes.add(objectMapper.createObjectNode().put("@id", id)); - return newNodes; - } - } + /** + * Merges the given id into the current value, + * using this representation: {"@id" : "id"}. + *

+ * The current value can be null without errors. + * Only the id will be considered in this case. + *

+ * If the id is null-ish, it will not be added, similar to a null-ish value. + * If the id is already present, nothing will be done. + * If it is not an array and the id is not present, an array will be applied. + * + * @param id the id to add. + * @param currentValue the current value of the property. + * @return The updated value of the property. + * Empty if value does not change! + */ + protected static Optional mergeIdIntoValue(String id, JsonNode currentValue) { + if (id == null || id.isBlank()) { return Optional.empty(); } + + ObjectMapper jsonBuilder = MyObjectMapper.getMapper(); + ObjectNode newIdObject = jsonBuilder.createObjectNode().put("@id", id); + if (currentValue == null || currentValue.isNull() || currentValue.isMissingNode()) { + return Optional.ofNullable(newIdObject); + } + + boolean isIdAlready = currentValue.asText().equals(id); + boolean isIdObjectAlready = currentValue.path("@id").asText().equals(id); + boolean isArrayWithIdPresent = currentValue.valueStream() + .anyMatch(node -> node + .path("@id") + .asText() + .equals(id)); + if (isIdAlready || isIdObjectAlready || isArrayWithIdPresent) { + return Optional.empty(); + } + + if (currentValue.isArray() && currentValue instanceof ArrayNode currentValueAsArray) { + currentValueAsArray.add(newIdObject); + return Optional.of(currentValueAsArray); + } else { + // property is not an array, so we make it an array + ArrayNode newNodes = jsonBuilder.createArrayNode(); + newNodes.add(currentValue); + newNodes.add(newIdObject); + return Optional.of(newNodes); } - return null; } /** @@ -369,6 +397,11 @@ protected String getId() { /** * Setting the id property of the entity, if the given value is not * null. If the id is not encoded, the encoding will be done. + *

+ * NOTE: IDs are not just names! The ID may have effects + * on parts of your crate! For example: If the entity represents a + * file which will be copied into the crate, writers must use the + * ID as filename. * * @param id the String representing the id. * @return the generic builder. @@ -486,11 +519,11 @@ public T addProperty(String key, boolean value) { * @return the generic builder */ public T addIdProperty(String name, String id) { - JsonNode jsonNode = AbstractEntity.addToIdProperty(name, id, this.properties.get(name)); - if (jsonNode != null) { - this.properties.set(name, jsonNode); - this.relatedItems.add(id); - } + AbstractEntity.mergeIdIntoValue(id, this.properties.get(name)) + .ifPresent(newValue -> { + this.properties.set(name, newValue); + this.relatedItems.add(id); + }); return self(); } @@ -526,13 +559,37 @@ public T addIdFromCollectionOfEntities(String name, Collection e } /** - * This sets everything from a json object to the property. Can be - * useful when the entity is already available somewhere. + * Deprecated. Equivalent to {@link #setAllIfValid(ObjectNode)}. * * @param properties the Json representing all the properties. - * @return the generic builder. + * @return the generic builder, either including all given properties + * * or unchanged. + * + * @deprecated To enforce the user know what this method does, + * we want the user to use one of the more explicitly named + * methods {@link #setAllIfValid(ObjectNode)} or + * {@link #setAllIfValid(ObjectNode)}. + * @see #setAllIfValid(ObjectNode) */ + @Deprecated(since = "2.1.0", forRemoval = true) public T setAll(ObjectNode properties) { + return setAllIfValid(properties); + } + + /** + * This sets everything from a json object to the property, + * if the result is valid. Otherwise, it will do nothing. + *

+ * Valid means here that the json object needs to be flat as specified + * in the RO-Crate specification. In principle, this means that + * primitives and objects referencing an ID are allowed, + * as well as arrays of these. + * + * @param properties the Json representing all the properties. + * @return the generic builder, either including all given properties + * or unchanged. + */ + public T setAllIfValid(ObjectNode properties) { if (AbstractEntity.entityValidation.entityValidation(properties)) { this.properties = properties; this.relatedItems.addAll(JsonUtilFunctions.getIdPropertiesFromJsonNode(properties)); @@ -540,6 +597,24 @@ public T setAll(ObjectNode properties) { return self(); } + /** + * This sets everything from a json object to the property. Can be + * useful when the entity is already available somewhere. + *

+ * Errors on validation are printed, but everything will be added. + * For more about validation, see {@link #setAllIfValid(ObjectNode)}. + * + * @param properties the Json representing all the properties. + * @return the generic builder with all properties added. + */ + public T setAllUnsafe(ObjectNode properties) { + // This will currently only print errors. + AbstractEntity.entityValidation.entityValidation(properties); + this.properties = properties; + this.relatedItems.addAll(JsonUtilFunctions.getIdPropertiesFromJsonNode(properties)); + return self(); + } + public abstract T self(); public abstract AbstractEntity build(); diff --git a/src/main/java/edu/kit/datamanager/ro_crate/entities/contextual/JsonDescriptor.java b/src/main/java/edu/kit/datamanager/ro_crate/entities/contextual/JsonDescriptor.java index 8bb91294..88aaf89e 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/entities/contextual/JsonDescriptor.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/entities/contextual/JsonDescriptor.java @@ -13,8 +13,8 @@ public class JsonDescriptor extends ContextualEntity { - private static final String CONFORMS_TO = "conformsTo"; - protected static final String ID = "ro-crate-metadata.json"; + protected static final String CONFORMS_TO = "conformsTo"; + public static final String ID = "ro-crate-metadata.json"; /** * Returns a JsonDescriptor with the conformsTo value set to the latest stable @@ -39,7 +39,7 @@ private JsonDescriptor(ContextualEntityBuilder builder) { /** * Builder for the JsonDescriptor. - * + *

* Defaults to the latest stable crate version and no other conformsTo values. */ public static final class Builder { diff --git a/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataEntity.java b/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataEntity.java index e6e28f8f..864d3044 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataEntity.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataEntity.java @@ -5,19 +5,11 @@ import edu.kit.datamanager.ro_crate.entities.AbstractEntity; import edu.kit.datamanager.ro_crate.entities.contextual.ContextualEntity; import static edu.kit.datamanager.ro_crate.special.IdentifierUtils.isUrl; -import edu.kit.datamanager.ro_crate.util.ZipUtil; -import java.io.File; -import java.io.IOException; import java.net.URI; import java.nio.file.Path; import java.util.ArrayList; import java.util.List; -import net.lingala.zip4j.ZipFile; -import net.lingala.zip4j.exception.ZipException; -import net.lingala.zip4j.io.outputstream.ZipOutputStream; -import net.lingala.zip4j.model.ZipParameters; -import org.apache.commons.io.FileUtils; /** * The base class of every data entity. @@ -56,54 +48,6 @@ public void addAuthorId(String id) { this.addIdProperty("author", id); } - /** - * If the data entity contains a physical file. This method will write it - * when the crate is being written to a zip archive. - * - * @param zipFile the zipFile where it should be written. - * @throws ZipException when something goes wrong with the writing to the - * zip file. - */ - public void saveToZip(ZipFile zipFile) throws ZipException { - if (this.path != null) { - ZipParameters zipParameters = new ZipParameters(); - zipParameters.setFileNameInZip(this.getId()); - zipFile.addFile(this.path.toFile(), zipParameters); - } - } - - /** - * If the data entity contains a physical file. This method will write it - * when the crate is being written to a zip archive. - * - * @param zipStream The zip output stream where it should be written. - * @throws ZipException when something goes wrong with the writing to the - * zip file. - * @throws IOException If opening the file input stream fails. - */ - public void saveToStream(ZipOutputStream zipStream) throws ZipException, IOException { - if (this.path != null) { - ZipUtil.addFileToZipStream(zipStream, this.path.toFile(), this.getId()); - } - } - - /** - * If the data entity contains a physical file. This method will write it - * when the crate is being written to a folder. - * - * @param file the folder location where the entity should be written. - * @throws IOException if something goes wrong with the writing. - */ - public void savetoFile(File file) throws IOException { - if (this.getPath() != null) { - if (this.getPath().toFile().isDirectory()) { - FileUtils.copyDirectory(this.getPath().toFile(), file.toPath().resolve(this.getId()).toFile()); - } else { - FileUtils.copyFile(this.getPath().toFile(), file.toPath().resolve(this.getId()).toFile()); - } - } - } - @JsonIgnore public Path getPath() { return path; diff --git a/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataSetEntity.java b/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataSetEntity.java index 832d9819..2ef078ff 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataSetEntity.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/entities/data/DataSetEntity.java @@ -4,15 +4,9 @@ import com.fasterxml.jackson.databind.annotation.JsonSerialize; import edu.kit.datamanager.ro_crate.entities.serializers.HasPartSerializer; -import edu.kit.datamanager.ro_crate.util.ZipUtil; -import java.io.IOException; import java.util.HashSet; import java.util.Set; -import net.lingala.zip4j.ZipFile; -import net.lingala.zip4j.exception.ZipException; -import net.lingala.zip4j.io.outputstream.ZipOutputStream; -import net.lingala.zip4j.model.ZipParameters; /** * A helping class for the creating of Data entities of type Dataset. @@ -43,26 +37,6 @@ public void removeFromHasPart(String str) { this.hasPart.remove(str); } - @Override - public void saveToZip(ZipFile zipFile) throws ZipException { - if (this.getPath() != null) { - ZipParameters parameters = new ZipParameters(); - parameters.setRootFolderNameInZip(this.getId()); - parameters.setIncludeRootFolder(false); - zipFile.addFolder(this.getPath().toFile(), parameters); - } - } - - @Override - public void saveToStream(ZipOutputStream zipOutputStream) throws IOException { - if (this.getPath() != null) { - ZipUtil.addFolderToZipStream( - zipOutputStream, - this.getPath().toAbsolutePath().toString(), - this.getId()); - } - } - public void addToHasPart(String id) { this.hasPart.add(id); } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/entities/validation/JsonSchemaValidation.java b/src/main/java/edu/kit/datamanager/ro_crate/entities/validation/JsonSchemaValidation.java index 067f5e2e..18e9624a 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/entities/validation/JsonSchemaValidation.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/entities/validation/JsonSchemaValidation.java @@ -60,6 +60,7 @@ public boolean validateEntity(JsonNode entity) { Set errors = this.entitySchema.validate(entity); if (errors.size() != 0) { System.err.println("This entity does not comply to the basic RO-Crate entity structure."); + errors.forEach(error -> System.err.println(error.getMessage())); return false; } return true; diff --git a/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/dataentities/ImportFromZenodo.java b/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/dataentities/ImportFromZenodo.java index 1be5607b..4a99ee8a 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/dataentities/ImportFromZenodo.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/dataentities/ImportFromZenodo.java @@ -112,12 +112,12 @@ private static void addToCrateFromZotero(String url, Crate crate) { for (var entity : graph) { if (entity.get("@id").asText().equals(mainId)) { var dataEntity = new DataEntity.DataEntityBuilder() - .setAll((ObjectNode) entity).build(); + .setAllUnsafe((ObjectNode) entity).build(); crate.addDataEntity(dataEntity); } else { // here we have to think of a way to differentiate between data and contextual entities. var contextualEntity = new ContextualEntity.ContextualEntityBuilder() - .setAll((ObjectNode) entity).build(); + .setAllUnsafe((ObjectNode) entity).build(); crate.addContextualEntity(contextualEntity); } } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/personprovider/OrcidProvider.java b/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/personprovider/OrcidProvider.java index 8e079af9..17207993 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/personprovider/OrcidProvider.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/externalproviders/personprovider/OrcidProvider.java @@ -72,7 +72,7 @@ public static PersonEntity getPerson(String url) { node.set(element.getKey(), element.getValue()); } } - return new PersonEntity.PersonEntityBuilder().setAll(node).build(); + return new PersonEntity.PersonEntityBuilder().setAllUnsafe(node).build(); } catch (IOException e) { String errorMessage = String.format("IO error: %s", e.getMessage()); logger.error(errorMessage); diff --git a/src/main/java/edu/kit/datamanager/ro_crate/preview/AutomaticPreview.java b/src/main/java/edu/kit/datamanager/ro_crate/preview/AutomaticPreview.java index 38241476..c403ab1c 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/preview/AutomaticPreview.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/preview/AutomaticPreview.java @@ -1,6 +1,6 @@ package edu.kit.datamanager.ro_crate.preview; -import edu.kit.datamanager.ro_crate.util.ZipUtil; +import edu.kit.datamanager.ro_crate.util.ZipStreamUtil; import java.io.File; import java.io.FileWriter; import java.io.IOException; @@ -10,9 +10,9 @@ import org.apache.commons.io.FileUtils; /** - * The default preview should use the rochtml tool - * (https://www.npmjs.com/package/ro-crate-html-js) for creating a simple - * preview file. + * The default preview should use the + * rochtml tool + * for creating a simple preview file. * * @author Nikola Tzotchev on 6.2.2022 г. * @version 1 @@ -66,13 +66,13 @@ public void saveAllToStream(String metadata, ZipOutputStream stream) throws IOEx if (PreviewGenerator.isRochtmlAvailable()) { try { FileUtils.forceMkdir(new File("temp")); - try (FileWriter writer = new FileWriter(new File("temp/ro-crate-metadata.json"))) { + try (FileWriter writer = new FileWriter("temp/ro-crate-metadata.json")) { writer.write(metadata); writer.flush(); } if (PreviewGenerator.isRochtmlAvailable()) { PreviewGenerator.generatePreview("temp"); - ZipUtil.addFileToZipStream(stream, new File("temp/ro-crate-preview.html"), "ro-crate-preview.html"); + ZipStreamUtil.addFileToZipStream(stream, new File("temp/ro-crate-preview.html"), "ro-crate-preview.html"); } } finally { try { diff --git a/src/main/java/edu/kit/datamanager/ro_crate/preview/CratePreview.java b/src/main/java/edu/kit/datamanager/ro_crate/preview/CratePreview.java index 14ea76ea..2459f8b5 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/preview/CratePreview.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/preview/CratePreview.java @@ -2,8 +2,15 @@ import java.io.File; import java.io.IOException; +import java.nio.file.Files; + +import edu.kit.datamanager.ro_crate.Crate; +import edu.kit.datamanager.ro_crate.writer.CrateWriter; +import edu.kit.datamanager.ro_crate.writer.WriteFolderStrategy; import net.lingala.zip4j.ZipFile; import net.lingala.zip4j.io.outputstream.ZipOutputStream; +import org.apache.commons.io.FileUtils; +import org.slf4j.LoggerFactory; /** * Interface for the ROCrate preview. This manages the human-readable @@ -15,10 +22,79 @@ */ public interface CratePreview { + /** + * Generate a preview of the crate and store it into the given target directory. + * It is the caller's responsibility to handle, e.g. delete after use, the result + * (The caller takes ownership of the result). + *

+ * IMPORTANT NOTE: This method currently has a default implementation that relies + * on deprecated methods. In future, you will have to implement this method directly. + * + * @param crate the crate to generate a preview for. + * @param targetDir the target directory to store the preview in, + * owned by the caller. + * @throws IOException if an error occurs while generating the preview. + */ + default void generate(Crate crate, File targetDir) throws IOException { + // disable preview generation to avoid recursion, + // as this is usually called in the process of writing a crate + // (including preview) + new CrateWriter<>(new WriteFolderStrategy().disablePreview()) + .save(crate, targetDir.getAbsolutePath()); + this.saveAllToFolder(targetDir); + try (var stream = Files.list(targetDir.toPath())) { + stream + .filter(path -> !path.getFileName().toString().equals("ro-crate-preview.html")) + .filter(path -> !path.getFileName().toString().equals("ro-crate-preview_files")) + .forEach(path -> { + try { + if (Files.isDirectory(path)) { + FileUtils.deleteDirectory(path.toFile()); + } else { + Files.delete(path); + } + } catch (IOException e) { + // Silently ignore deletion errors + LoggerFactory.getLogger(CratePreview.class) + .error("Failed to delete temporary file {}", path, e); + } + }); + } + } + + /** + * Takes a crate in form of a zip file and generates a preview of it, + * which will be stored within the crate. + * + * @param zipFile the zip file with the crate, which should receive a preview. + * @throws IOException if an error occurs while saving the preview + * + * @deprecated Use {@link #generate(Crate, File)} instead. + */ + @Deprecated(since = "2.1.0", forRemoval = true) void saveAllToZip(ZipFile zipFile) throws IOException; + /** + * Saves the preview, given by the folder, into the given folder. + * + * @param folder the folder (containing a crate) to save the preview in. + * @throws IOException if an error occurs while saving the preview. + * + * @deprecated Use {@link #generate(Crate, File)} instead. + */ + @Deprecated(since = "2.1.0", forRemoval = true) void saveAllToFolder(File folder) throws IOException; - + + /** + * Saves the preview, given by the metadata, into the given stream. + * + * @param metadata the metadata of the crate to save the preview in. + * @param stream the stream to save the preview in. + * @throws IOException if an error occurs while saving the preview. + * + * @deprecated Use {@link #generate(Crate, File)} instead. + */ + @Deprecated(since = "2.1.0", forRemoval = true) void saveAllToStream(String metadata, ZipOutputStream stream) throws IOException; } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/preview/CustomPreview.java b/src/main/java/edu/kit/datamanager/ro_crate/preview/CustomPreview.java index 5c8c9fa2..77300a87 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/preview/CustomPreview.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/preview/CustomPreview.java @@ -2,7 +2,7 @@ import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; -import edu.kit.datamanager.ro_crate.util.ZipUtil; +import edu.kit.datamanager.ro_crate.util.ZipStreamUtil; import freemarker.template.Configuration; import freemarker.template.Template; import freemarker.template.TemplateException; @@ -53,7 +53,7 @@ public CustomPreview() { private CustomPreviewModel mapFromJson(String metadata) throws IOException { ObjectMapper mapper = new ObjectMapper(); - JsonNode root = (JsonNode) mapper.readValue(metadata, JsonNode.class); + JsonNode root = mapper.readValue(metadata, JsonNode.class); JsonNode graph = root.get("@graph"); CustomPreviewModel.ROCrate crate = new CustomPreviewModel.ROCrate(); List datasets = new ArrayList<>(); @@ -196,13 +196,13 @@ public void saveAllToStream(String metadata, ZipOutputStream stream) throws IOEx //prepare output folder and writer FileUtils.forceMkdir(new File("temp")); //load and process template - try (FileWriter writer = new FileWriter(new File("temp/ro-crate-preview.html"))) { + try (FileWriter writer = new FileWriter("temp/ro-crate-preview.html")) { //load and process template template.process(dataModel, writer); writer.flush(); } - ZipUtil.addFileToZipStream(stream, new File("temp/ro-crate-preview.html"), "ro-crate-preview.html"); + ZipStreamUtil.addFileToZipStream(stream, new File("temp/ro-crate-preview.html"), "ro-crate-preview.html"); } catch (TemplateException ex) { throw new IOException("Failed to generate preview.", ex); } finally { diff --git a/src/main/java/edu/kit/datamanager/ro_crate/preview/StaticPreview.java b/src/main/java/edu/kit/datamanager/ro_crate/preview/StaticPreview.java index 398615c1..c3a93629 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/preview/StaticPreview.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/preview/StaticPreview.java @@ -1,9 +1,8 @@ package edu.kit.datamanager.ro_crate.preview; -import edu.kit.datamanager.ro_crate.util.ZipUtil; +import edu.kit.datamanager.ro_crate.util.ZipStreamUtil; import java.io.File; import java.io.IOException; -import java.util.Optional; import net.lingala.zip4j.ZipFile; import net.lingala.zip4j.io.outputstream.ZipOutputStream; @@ -13,7 +12,7 @@ /** * This class adds a static preview to the crate, which consists of a * metadataHtml file and a folder containing other files required to render - * metadataHtml. If will be put unchanged to the writer output, i.e., a zip + * metadataHtml. It will be put unchanged to the writer output, i.e., a zip * file, folder, or stream. * * @author jejkal @@ -65,9 +64,9 @@ public void saveAllToFolder(File folder) throws IOException { @Override public void saveAllToStream(String metadata, ZipOutputStream stream) throws IOException { - ZipUtil.addFileToZipStream(stream, this.metadataHtml, "ro-crate-preview.html"); + ZipStreamUtil.addFileToZipStream(stream, this.metadataHtml, "ro-crate-preview.html"); if (this.otherFiles != null) { - ZipUtil.addFolderToZipStream(stream, this.otherFiles, this.otherFiles.getName()); + ZipStreamUtil.addFolderToZipStream(stream, this.otherFiles, "ro-crate-preview_files"); } } } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/CrateReader.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/CrateReader.java index 5acb575b..5132e44b 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/CrateReader.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/CrateReader.java @@ -17,6 +17,7 @@ import org.slf4j.LoggerFactory; import java.io.File; +import java.io.IOException; import java.nio.file.Path; import java.util.*; import java.util.stream.Collectors; @@ -29,7 +30,7 @@ * The constructor takes a strategy to support different ways of importing the * crates. (from zip, folder, etc.). *

- * The reader consideres "hasPart" and "isPartOf" properties and considers all + * The reader considers "hasPart" and "isPartOf" properties and considers all * entities (in-)directly connected to the root entity ("./") as DataEntities. * * @param the type of the location parameter @@ -82,8 +83,10 @@ public CrateReader(GenericReaderStrategy strategy) { * * @param location the location of the ro-crate to be read * @return the read RO-crate + * + * @throws IOException if the crate cannot be read */ - public RoCrate readCrate(T location) { + public RoCrate readCrate(T location) throws IOException { // get the ro-crate-metadata.json ObjectNode metadataJson = strategy.readMetadataJson(location); // get the content of the crate @@ -119,7 +122,7 @@ private RoCrate rebuildCrate(ObjectNode metadataJson, File files, HashSet { @@ -133,7 +136,7 @@ private RoCrate rebuildCrate(ObjectNode metadataJson, File files, HashSet extractHasPartIds(ObjectNode root) { private void setCrateDescriptor(RoCrate crate, JsonNode descriptor) { ContextualEntity descriptorEntity = new ContextualEntity.ContextualEntityBuilder() - .setAll(descriptor.deepCopy()) + .setAllUnsafe(descriptor.deepCopy()) .build(); crate.setJsonDescriptor(descriptorEntity); } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/FolderReader.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/FolderReader.java index f0436e27..c6268ac1 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/FolderReader.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/FolderReader.java @@ -6,7 +6,7 @@ * @author Nikola Tzotchev on 9.2.2022 г. * @version 1 * - * @deprecated Use {@link FolderStrategy} instead. + * @deprecated Use {@link ReadFolderStrategy} instead. */ @Deprecated(since = "2.1.0", forRemoval = true) -public class FolderReader extends FolderStrategy {} +public class FolderReader extends ReadFolderStrategy {} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/GenericReaderStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/GenericReaderStrategy.java index c3539b17..6a9f4bd4 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/GenericReaderStrategy.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/GenericReaderStrategy.java @@ -2,21 +2,22 @@ import com.fasterxml.jackson.databind.node.ObjectNode; import java.io.File; +import java.io.IOException; /** * Generic interface for the strategy of the reader class. * This allows for flexible input types when implementing different reading strategies. * - * @param the type of the location parameter + * @param the type which determines the source of the crate */ -public interface GenericReaderStrategy { +public interface GenericReaderStrategy { /** * Read the metadata.json file from the given location. * * @param location the location to read from * @return the parsed metadata.json as ObjectNode */ - ObjectNode readMetadataJson(T location); + ObjectNode readMetadataJson(SOURCE_TYPE location) throws IOException; /** * Read the content from the given location. @@ -24,5 +25,5 @@ public interface GenericReaderStrategy { * @param location the location to read from * @return the content as a File */ - File readContent(T location); + File readContent(SOURCE_TYPE location) throws IOException; } \ No newline at end of file diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/FolderStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/ReadFolderStrategy.java similarity index 73% rename from src/main/java/edu/kit/datamanager/ro_crate/reader/FolderStrategy.java rename to src/main/java/edu/kit/datamanager/ro_crate/reader/ReadFolderStrategy.java index f6e235ed..927d77de 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/FolderStrategy.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/ReadFolderStrategy.java @@ -14,18 +14,14 @@ * @author Nikola Tzotchev on 9.2.2022 г. * @version 1 */ -public class FolderStrategy implements GenericReaderStrategy { +public class ReadFolderStrategy implements GenericReaderStrategy { @Override - public ObjectNode readMetadataJson(String location) { + public ObjectNode readMetadataJson(String location) throws IOException { Path metadata = new File(location).toPath().resolve("ro-crate-metadata.json"); ObjectMapper objectMapper = MyObjectMapper.getMapper(); ObjectNode objectNode = objectMapper.createObjectNode(); - try { - objectNode = objectMapper.readTree(metadata.toFile()).deepCopy(); - } catch (IOException e) { - e.printStackTrace(); - } + objectNode = objectMapper.readTree(metadata.toFile()).deepCopy(); return objectNode; } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/ReadZipStrategy.java similarity index 50% rename from src/main/java/edu/kit/datamanager/ro_crate/reader/ZipStrategy.java rename to src/main/java/edu/kit/datamanager/ro_crate/reader/ReadZipStrategy.java index 0d6381a6..5a2474b5 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipStrategy.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/ReadZipStrategy.java @@ -2,9 +2,12 @@ import com.fasterxml.jackson.databind.ObjectMapper; import com.fasterxml.jackson.databind.node.ObjectNode; +import edu.kit.datamanager.ro_crate.entities.contextual.JsonDescriptor; import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; +import edu.kit.datamanager.ro_crate.util.FileSystemUtil; import net.lingala.zip4j.ZipFile; import org.apache.commons.io.FileUtils; +import org.apache.commons.io.filefilter.FileFilterUtils; import java.io.File; import java.io.IOException; @@ -12,11 +15,20 @@ import java.util.UUID; /** - * A ReaderStrategy implementation which reads from ZipFiles. + * Reads a crate from a ZIP archive (file). *

- * May be used as a dependency for CrateReader. It will unzip - * the ZipFile in a path relative to the directory this application runs in. - * By default, it will be `./.tmp/ro-crate-java/zipReader/$UUID/`. + * This class handles reading and extraction of RO-Crate content from ZIP archives + * into a temporary directory structure on the file system, + * which allows accessing the contained files. + *

+ * Supports ELN-Style crates, + * meaning the crate may be either in the zip archive directly or in a single, + * direct subfolder beneath the root folder (/folder). + *

+ * Note: This implementation checks for up to 50 subdirectories if multiple are present. + * This is to avoid zip bombs, which may contain a lot of subdirectories, + * and at the same time gracefully handle valid crated with hidden subdirectories + * (for example, thumbnails). *

* NOTE: The resulting crate may refer to these temporary files. Therefore, * these files are only being deleted before the JVM exits. If you need to free @@ -27,16 +39,19 @@ * persistent location and possibly read it from there, if required. Or use * the ZipWriter to write it back to its source. */ -public class ZipStrategy implements GenericReaderStrategy { +public class ReadZipStrategy implements GenericReaderStrategy { protected final String ID = UUID.randomUUID().toString(); protected Path temporaryFolder = Path.of(String.format("./.tmp/ro-crate-java/zipReader/%s/", ID)); protected boolean isExtracted = false; /** - * Crates a ZipReader with the default configuration as described in the class documentation. + * Crates an instance with the default configuration. + *

+ * The default configuration is to extract the ZipFile to + * `./.tmp/ro-crate-java/zipReader/$UUID/`. */ - public ZipStrategy() {} + public ReadZipStrategy() {} /** * Creates a ZipReader which will extract the contents temporary @@ -49,7 +64,7 @@ public ZipStrategy() {} * directory. These subdirectories * will have UUIDs as their names. */ - public ZipStrategy(Path folderPath, boolean shallAddUuidSubfolder) { + public ReadZipStrategy(Path folderPath, boolean shallAddUuidSubfolder) { if (shallAddUuidSubfolder) { this.temporaryFolder = folderPath.resolve(ID); } else { @@ -78,46 +93,46 @@ public boolean isExtracted() { return isExtracted; } - private void readCrate(String location) { - try { - File folder = temporaryFolder.toFile(); - // ensure the directory is clean - if (folder.isDirectory()) { - FileUtils.cleanDirectory(folder); - } else if (folder.isFile()) { - FileUtils.delete(folder); - } - // extract - try (ZipFile zf = new ZipFile(location)) { - zf.extractAll(temporaryFolder.toAbsolutePath().toString()); - this.isExtracted = true; - } - // register deletion on exit - FileUtils.forceDeleteOnExit(folder); - } catch (IOException e) { - e.printStackTrace(); + private void readCrate(String location) throws IOException { + File folder = temporaryFolder.toFile(); + FileSystemUtil.mkdirOrDeleteContent(folder); + // extract + try (ZipFile zf = new ZipFile(location)) { + zf.extractAll(temporaryFolder.toAbsolutePath().toString()); + this.isExtracted = true; } + // register deletion on exit + FileUtils.forceDeleteOnExit(folder); } @Override - public ObjectNode readMetadataJson(String location) { + public ObjectNode readMetadataJson(String location) throws IOException { if (!isExtracted) { this.readCrate(location); } ObjectMapper objectMapper = MyObjectMapper.getMapper(); - File jsonMetadata = temporaryFolder.resolve("ro-crate-metadata.json").toFile(); - - try { - return objectMapper.readTree(jsonMetadata).deepCopy(); - } catch (IOException e) { - e.printStackTrace(); - return null; + File jsonMetadata = this.temporaryFolder.resolve(JsonDescriptor.ID).toFile(); + if (!jsonMetadata.isFile()) { + // Try to find the metadata file in subdirectories + File firstSubdir = FileUtils.listFilesAndDirs( + temporaryFolder.toFile(), + FileFilterUtils.directoryFileFilter(), + null // not recursive + ) + .stream() + .limit(50) + .filter(file -> file.toPath().toAbsolutePath().resolve(JsonDescriptor.ID).toFile().isFile()) + .findFirst() + .orElseThrow(() -> new IllegalStateException("No %s found in zip file".formatted(JsonDescriptor.ID))); + jsonMetadata = firstSubdir.toPath().resolve(JsonDescriptor.ID).toFile(); } + + return objectMapper.readTree(jsonMetadata).deepCopy(); } @Override - public File readContent(String location) { + public File readContent(String location) throws IOException { if (!isExtracted) { this.readCrate(location); } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/ReadZipStreamStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/ReadZipStreamStrategy.java new file mode 100644 index 00000000..3cc41086 --- /dev/null +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/ReadZipStreamStrategy.java @@ -0,0 +1,176 @@ +package edu.kit.datamanager.ro_crate.reader; + +import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.node.ObjectNode; +import edu.kit.datamanager.ro_crate.entities.contextual.JsonDescriptor; +import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; +import java.io.File; +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.io.OutputStream; +import java.nio.file.Path; +import java.util.UUID; + +import edu.kit.datamanager.ro_crate.util.FileSystemUtil; +import net.lingala.zip4j.io.inputstream.ZipInputStream; +import net.lingala.zip4j.model.LocalFileHeader; +import org.apache.commons.io.FileUtils; +import org.apache.commons.io.filefilter.FileFilterUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Reads a crate from a streamed ZIP archive. + *

+ * This class handles reading and extraction of RO-Crate content from ZIP archives + * into a temporary directory structure on the file system, + * which allows accessing the contained files. + *

+ * Supports ELN-Style crates, + * meaning the crate may be either in the zip archive directly or in a single, + * direct subfolder beneath the root folder (/folder). + *

+ * Note: This implementation checks for up to 50 subdirectories if multiple are present. + * This is to avoid zip bombs, which may contain a lot of subdirectories, + * and at the same time gracefully handle valid crated with hidden subdirectories + * (for example, thumbnails). + *

+ * NOTE: The resulting crate may refer to these temporary files. Therefore, + * these files are only being deleted before the JVM exits. If you need to free + * space because your application is long-running or creates a lot of + * crates, you may use the getters to retrieve information which will help + * you to clean up manually. Keep in mind that crates may refer to this + * folder after extraction. Use RoCrateWriter to export it so some + * persistent location and possibly read it from there, if required. Or use + * the ZipWriter to write it back to its source. + * + * @author jejkal + */ +public class ReadZipStreamStrategy implements GenericReaderStrategy { + + private static final Logger logger = LoggerFactory.getLogger(ReadZipStreamStrategy.class); + protected final String ID = UUID.randomUUID().toString(); + protected Path temporaryFolder = Path.of(String.format("./.tmp/ro-crate-java/zipStreamReader/%s/", ID)); + protected boolean isExtracted = false; + + /** + * Crates an instance with the default configuration. + *

+ * The default configuration is to extract the ZipFile to + * `./.tmp/ro-crate-java/zipStreamReader/%UUID/`. + */ + public ReadZipStreamStrategy() {} + + /** + * Creates a ZipStreamReader which will extract the contents temporary to + * the given location instead of the default location. + * + * @param folderPath the custom directory to extract content to for + * temporary access. + * @param shallAddUuidSubfolder if true, the reader will extract into + * subdirectories of the given directory. These subdirectories will have + * UUIDs as their names. + */ + public ReadZipStreamStrategy(Path folderPath, boolean shallAddUuidSubfolder) { + if (shallAddUuidSubfolder) { + this.temporaryFolder = folderPath.resolve(ID); + } else { + this.temporaryFolder = folderPath; + } + } + + /** + * @return the identifier which may be used as the name for a subfolder in + * the temporary directory. + */ + public String getID() { + return ID; + } + + /** + * @return the folder (considered temporary) where the zipped crate will be + * or has been extracted to. + */ + public Path getTemporaryFolder() { + return temporaryFolder; + } + + /** + * @return whether the crate has already been extracted into the temporary + * folder. + */ + public boolean isExtracted() { + return isExtracted; + } + + /**Read the crate metadata and content from the provided input stream. + * + * @param stream The input stream. + */ + private void readCrate(InputStream stream) throws IOException { + File folder = temporaryFolder.toFile(); + FileSystemUtil.mkdirOrDeleteContent(folder); + + LocalFileHeader localFileHeader; + int readLen; + byte[] readBuffer = new byte[4096]; + + try (ZipInputStream zipInputStream = new ZipInputStream(stream)) { + while ((localFileHeader = zipInputStream.getNextEntry()) != null) { + String fileName = localFileHeader.getFileName(); + File extractedFile = new File(folder, fileName).getCanonicalFile(); + Path targetRoot = folder.toPath().toRealPath(); + if (!extractedFile.toPath().startsWith(targetRoot)) { + throw new IOException("Entry is outside of target directory: " + fileName); + } + if (localFileHeader.isDirectory()) { + FileUtils.forceMkdir(extractedFile); + continue; + } + FileUtils.forceMkdir(extractedFile.getParentFile()); + try (OutputStream outputStream = new FileOutputStream(extractedFile)) { + while ((readLen = zipInputStream.read(readBuffer)) != -1) { + outputStream.write(readBuffer, 0, readLen); + } + } + } + } + this.isExtracted = true; + // register deletion on exit + FileUtils.forceDeleteOnExit(folder); + } + + @Override + public ObjectNode readMetadataJson(InputStream stream) throws IOException { + if (!isExtracted) { + this.readCrate(stream); + } + + ObjectMapper objectMapper = MyObjectMapper.getMapper(); + File jsonMetadata = temporaryFolder.resolve(JsonDescriptor.ID).toFile(); + if (!jsonMetadata.isFile()) { + // Try to find the metadata file in subdirectories + File firstSubdir = FileUtils.listFilesAndDirs( + temporaryFolder.toFile(), + FileFilterUtils.directoryFileFilter(), + null + ) + .stream() + .limit(50) + .filter(file -> file.toPath().toAbsolutePath().resolve(JsonDescriptor.ID).toFile().isFile()) + .findFirst() + .orElseThrow(() -> new IllegalStateException("No %s found in zip file".formatted(JsonDescriptor.ID))); + jsonMetadata = firstSubdir.toPath().resolve(JsonDescriptor.ID).toFile(); + } + return objectMapper.readTree(jsonMetadata).deepCopy(); + } + + @Override + public File readContent(InputStream stream) throws IOException { + if (!isExtracted) { + this.readCrate(stream); + } + return temporaryFolder.toFile(); + } +} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/Readers.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/Readers.java index 49e09cb1..83a43701 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/Readers.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/Readers.java @@ -19,10 +19,10 @@ private Readers() {} * * @return A reader configured for ZIP files * - * @see ZipStreamStrategy#ZipStreamStrategy() + * @see ReadZipStreamStrategy#ReadZipStreamStrategy() */ public static CrateReader newZipStreamReader() { - return new CrateReader<>(new ZipStreamStrategy()); + return new CrateReader<>(new ReadZipStreamStrategy()); } /** @@ -33,10 +33,10 @@ public static CrateReader newZipStreamReader() { * @param useUuidSubfolder Whether to create a UUID subfolder under extractPath * @return A reader configured for ZIP files with custom extraction * - * @see ZipStreamStrategy#ZipStreamStrategy(Path, boolean) + * @see ReadZipStreamStrategy#ReadZipStreamStrategy(Path, boolean) */ public static CrateReader newZipStreamReader(Path extractPath, boolean useUuidSubfolder) { - return new CrateReader<>(new ZipStreamStrategy(extractPath, useUuidSubfolder)); + return new CrateReader<>(new ReadZipStreamStrategy(extractPath, useUuidSubfolder)); } /** @@ -44,10 +44,10 @@ public static CrateReader newZipStreamReader(Path extractPath, bool * * @return A reader configured for folders * - * @see FolderStrategy + * @see ReadFolderStrategy */ public static CrateReader newFolderReader() { - return new CrateReader<>(new FolderStrategy()); + return new CrateReader<>(new ReadFolderStrategy()); } /** @@ -55,10 +55,10 @@ public static CrateReader newFolderReader() { * * @return A reader configured for ZIP files * - * @see ZipStrategy#ZipStrategy() + * @see ReadZipStrategy#ReadZipStrategy() */ public static CrateReader newZipPathReader() { - return new CrateReader<>(new ZipStrategy()); + return new CrateReader<>(new ReadZipStrategy()); } /** @@ -69,9 +69,9 @@ public static CrateReader newZipPathReader() { * @param useUuidSubfolder Whether to create a UUID subfolder under extractPath * @return A reader configured for ZIP files with custom extraction * - * @see ZipStrategy#ZipStrategy(Path, boolean) + * @see ReadZipStrategy#ReadZipStrategy(Path, boolean) */ public static CrateReader newZipPathReader(Path extractPath, boolean useUuidSubfolder) { - return new CrateReader<>(new ZipStrategy(extractPath, useUuidSubfolder)); + return new CrateReader<>(new ReadZipStrategy(extractPath, useUuidSubfolder)); } } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipReader.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipReader.java index ad9bbb01..d92faf94 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipReader.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipReader.java @@ -18,10 +18,10 @@ * persistent location and possibly read it from there, if required. Or use * the ZipWriter to write it back to its source. * - * @deprecated Use {@link ZipStrategy} instead. + * @deprecated Use {@link ReadZipStrategy} instead. */ @Deprecated(since = "2.1.0", forRemoval = true) -public class ZipReader extends ZipStrategy { +public class ZipReader extends ReadZipStrategy { /** * Crates a ZipReader with the default configuration as described in the class documentation. diff --git a/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipStreamStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipStreamStrategy.java deleted file mode 100644 index cb4f53af..00000000 --- a/src/main/java/edu/kit/datamanager/ro_crate/reader/ZipStreamStrategy.java +++ /dev/null @@ -1,150 +0,0 @@ -package edu.kit.datamanager.ro_crate.reader; - -import com.fasterxml.jackson.databind.ObjectMapper; -import com.fasterxml.jackson.databind.node.ObjectNode; -import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; -import java.io.File; -import java.io.FileOutputStream; -import java.io.IOException; -import java.io.InputStream; -import java.io.OutputStream; -import java.nio.file.Path; -import java.util.UUID; -import net.lingala.zip4j.io.inputstream.ZipInputStream; -import net.lingala.zip4j.model.LocalFileHeader; -import org.apache.commons.io.FileUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * A ZIP file reader implementation of the StreamReaderStrategy interface. - * This class handles reading and extraction of RO-Crate content from ZIP archives - * into a temporary directory structure, which allows for accessing the contained files. - * - * @author jejkal - */ -public class ZipStreamStrategy implements GenericReaderStrategy { - - private static final Logger logger = LoggerFactory.getLogger(ZipStreamStrategy.class); - protected final String ID = UUID.randomUUID().toString(); - protected Path temporaryFolder = Path.of(String.format("./.tmp/ro-crate-java/zipStreamReader/%s/", ID)); - protected boolean isExtracted = false; - - /** - * Crates a ZipStreamReader with the default configuration as described in - * the class documentation. - */ - public ZipStreamStrategy() { - } - - /** - * Creates a ZipStreamReader which will extract the contents temporary to - * the given location instead of the default location. - * - * @param folderPath the custom directory to extract content to for - * temporary access. - * @param shallAddUuidSubfolder if true, the reader will extract into - * subdirectories of the given directory. These subdirectories will have - * UUIDs as their names. - */ - public ZipStreamStrategy(Path folderPath, boolean shallAddUuidSubfolder) { - if (shallAddUuidSubfolder) { - this.temporaryFolder = folderPath.resolve(ID); - } else { - this.temporaryFolder = folderPath; - } - } - - /** - * @return the identifier which may be used as the name for a subfolder in - * the temporary directory. - */ - public String getID() { - return ID; - } - - /** - * @return the folder (considered temporary) where the zipped crate will be - * or has been extracted to. - */ - public Path getTemporaryFolder() { - return temporaryFolder; - } - - /** - * @return whether the crate has already been extracted into the temporary - * folder. - */ - public boolean isExtracted() { - return isExtracted; - } - - /**Read the crate metadata and content from the provided input stream. - * - * @param stream The input stream. - */ - private void readCrate(InputStream stream) { - try { - File folder = temporaryFolder.toFile(); - // ensure the directory is clean - if (folder.exists()) { - if (folder.isDirectory()) { - FileUtils.cleanDirectory(folder); - } else if (folder.isFile()) { - FileUtils.delete(folder); - } - } else { - FileUtils.forceMkdir(folder); - } - - LocalFileHeader localFileHeader; - int readLen; - byte[] readBuffer = new byte[4096]; - - try (ZipInputStream zipInputStream = new ZipInputStream(stream)) { - while ((localFileHeader = zipInputStream.getNextEntry()) != null) { - String fileName = localFileHeader.getFileName(); - File extractedFile = new File(folder, fileName).getCanonicalFile(); - if (!extractedFile.toPath().startsWith(folder.getCanonicalPath())) { - throw new IOException("Entry is outside of target directory: " + fileName); - } - try (OutputStream outputStream = new FileOutputStream(extractedFile)) { - while ((readLen = zipInputStream.read(readBuffer)) != -1) { - outputStream.write(readBuffer, 0, readLen); - } - } - } - } - this.isExtracted = true; - // register deletion on exit - FileUtils.forceDeleteOnExit(folder); - } catch (IOException ex) { - logger.error("Failed to read crate from input stream.", ex); - } - } - - @Override - public ObjectNode readMetadataJson(InputStream stream) { - if (!isExtracted) { - this.readCrate(stream); - } - - ObjectMapper objectMapper = MyObjectMapper.getMapper(); - File jsonMetadata = temporaryFolder.resolve("ro-crate-metadata.json").toFile(); - - try { - return objectMapper.readTree(jsonMetadata).deepCopy(); - } catch (IOException e) { - logger.error("Failed to deserialize crate metadata.", e); - return null; - } - } - - @Override - public File readContent(InputStream stream) { - if (!isExtracted) { - this.readCrate(stream); - } - return temporaryFolder.toFile(); - } -} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/util/FileSystemUtil.java b/src/main/java/edu/kit/datamanager/ro_crate/util/FileSystemUtil.java new file mode 100644 index 00000000..e4d75442 --- /dev/null +++ b/src/main/java/edu/kit/datamanager/ro_crate/util/FileSystemUtil.java @@ -0,0 +1,74 @@ +package edu.kit.datamanager.ro_crate.util; + +import org.apache.commons.io.FileUtils; + +import java.io.File; +import java.io.IOException; +import java.util.Collection; +import java.util.regex.Matcher; + +public class FileSystemUtil { + private FileSystemUtil() { + // Utility class, no instantiation + } + + /** + * Removes a specific set of given file extensions from a file name, if present. + * The extensions are case-insensitive. Given "ELN", "eln" or "Eln" will also match. + * The dot (.) before the extension is also assumed and removed implicitly: + *

+ * Example: + * filterExtensionsFromFileName("test.eln", Set.of("ELN")) -> "test" + * + * @param filename the file name to filter (must not be null) + * @param extensionsToRemove the extensions to remove (must not be null) + * @return the filtered file name + * @throws NullPointerException if any parameter is null + */ + public static String filterExtensionsFromFileName(String filename, Collection extensionsToRemove) { + String dot = Matcher.quoteReplacement("."); + String end = Matcher.quoteReplacement("$"); + for (String extension : extensionsToRemove) { + // (?i) removes case sensitivity + filename = filename.replaceFirst("(?i)" + dot + extension + end, ""); + } + return filename; + } + + /** + * Ensures that a given path ends with a trailing slash. + * + * @param path the path to check + * @return the path with a trailing slash if it didn't have one, or the original path + */ + public static String ensureTrailingSlash(String path) { + if (path == null || path.isEmpty()) { + return path; + } + if (!path.endsWith("/")) { + return path + "/"; + } + return path; + } + + /** + * Creates a directory or deletes its content if it already exists. + * + * @param folder the folder to create or delete content from + * @throws IOException if an I/O error occurs + */ + public static void mkdirOrDeleteContent(File folder) throws IOException { + File[] files = folder.listFiles(); + boolean isNonEmptyDir = folder.exists() + && folder.isDirectory() + && files != null + && files.length > 0; + boolean isFile = folder.exists() + && !folder.isDirectory(); + + if (isNonEmptyDir || isFile) { + FileUtils.forceDelete(folder); + } + FileUtils.forceMkdir(folder); + } +} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/util/ZipUtil.java b/src/main/java/edu/kit/datamanager/ro_crate/util/ZipStreamUtil.java similarity index 99% rename from src/main/java/edu/kit/datamanager/ro_crate/util/ZipUtil.java rename to src/main/java/edu/kit/datamanager/ro_crate/util/ZipStreamUtil.java index c1da0137..b8eb2ef1 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/util/ZipUtil.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/util/ZipStreamUtil.java @@ -11,7 +11,7 @@ * * @author jejkal */ -public class ZipUtil { +public class ZipStreamUtil { /** * Adds a folder and its contents to a ZipOutputStream. diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/CrateWriter.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/CrateWriter.java index 440be0c4..caba67f9 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/CrateWriter.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/CrateWriter.java @@ -4,15 +4,19 @@ import edu.kit.datamanager.ro_crate.validation.JsonSchemaValidation; import edu.kit.datamanager.ro_crate.validation.Validator; +import java.io.IOException; + /** * The class used for writing (exporting) crates. The class uses a strategy * pattern for writing crates as different formats. (zip, folders, etc.) + * + * @param the type which determines the destination of the result */ -public class CrateWriter { +public class CrateWriter { - private final GenericWriterStrategy strategy; + private final GenericWriterStrategy strategy; - public CrateWriter(GenericWriterStrategy strategy) { + public CrateWriter(GenericWriterStrategy strategy) { this.strategy = strategy; } @@ -22,7 +26,7 @@ public CrateWriter(GenericWriterStrategy strategy) { * @param crate the crate to write. * @param destination the location where the crate should be written. */ - public void save(Crate crate, DESTINATION destination) { + public void save(Crate crate, DESTINATION_TYPE destination) throws IOException { Validator defaultValidation = new Validator(new JsonSchemaValidation()); defaultValidation.validate(crate); this.strategy.save(crate, destination); diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/ElnFormatWriter.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/ElnFormatWriter.java new file mode 100644 index 00000000..bd6310cd --- /dev/null +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/ElnFormatWriter.java @@ -0,0 +1,27 @@ +package edu.kit.datamanager.ro_crate.writer; + +/** + * An Interface for {@link GenericWriterStrategy} implementations which support writing + * ELN-Style crates. + * + * @param the type which determines the destination of the result + */ +public interface ElnFormatWriter extends GenericWriterStrategy { + + /** + * Write in ELN format style, meaning with a root subfolder in the zip file. + * Same as {@link #withRootSubdirectory()}. + * + * @return this writer + */ + ElnFormatWriter usingElnStyle(); + + /** + * Alias with more generic name for {@link #usingElnStyle()}. + * + * @return this writer + */ + default ElnFormatWriter withRootSubdirectory() { + return this.usingElnStyle(); + } +} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/FolderStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/FolderStrategy.java deleted file mode 100644 index b2585637..00000000 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/FolderStrategy.java +++ /dev/null @@ -1,63 +0,0 @@ -package edu.kit.datamanager.ro_crate.writer; - -import com.fasterxml.jackson.databind.JsonNode; -import com.fasterxml.jackson.databind.ObjectMapper; -import edu.kit.datamanager.ro_crate.Crate; -import edu.kit.datamanager.ro_crate.entities.data.DataEntity; -import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; -import org.apache.commons.io.FileUtils; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.ByteArrayInputStream; -import java.io.File; -import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; - -/** - * A class for writing a crate to a folder. - * - * @author Nikola Tzotchev on 9.2.2022 г. - * @version 1 - */ -public class FolderStrategy implements GenericWriterStrategy { - - private static final Logger logger = LoggerFactory.getLogger(FolderStrategy.class); - - @Override - public void save(Crate crate, String destination) { - File file = new File(destination); - try { - FileUtils.forceMkdir(file); - ObjectMapper objectMapper = MyObjectMapper.getMapper(); - JsonNode node = objectMapper.readTree(crate.getJsonMetadata()); - String str = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(node); - InputStream inputStream = new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8)); - - File json = new File(destination, "ro-crate-metadata.json"); - FileUtils.copyInputStreamToFile(inputStream, json); - inputStream.close(); - // save also the preview files to the crate destination - if (crate.getPreview() != null) { - crate.getPreview().saveAllToFolder(file); - } - for (var e : crate.getUntrackedFiles()) { - if (e.isDirectory()) { - FileUtils.copyDirectoryToDirectory(e, file); - } else { - FileUtils.copyFileToDirectory(e, file); - } - } - } catch (IOException e) { - logger.error("Error creating destination directory!", e); - } - for (DataEntity dataEntity : crate.getAllDataEntities()) { - try { - dataEntity.savetoFile(file); - } catch (IOException e) { - logger.error("Cannot save " + dataEntity.getId() + " to destination folder!", e); - } - } - } -} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/FolderWriter.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/FolderWriter.java index 5730104e..1426414e 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/FolderWriter.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/FolderWriter.java @@ -5,7 +5,7 @@ * * @author Nikola Tzotchev on 9.2.2022 г. * - * @deprecated Use {@link FolderStrategy} instead. + * @deprecated Use {@link WriteFolderStrategy} instead. */ @Deprecated(since = "2.1.0", forRemoval = true) -public class FolderWriter extends FolderStrategy {} +public class FolderWriter extends WriteFolderStrategy {} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/GenericWriterStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/GenericWriterStrategy.java index 6306b576..d06301a0 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/GenericWriterStrategy.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/GenericWriterStrategy.java @@ -2,18 +2,20 @@ import edu.kit.datamanager.ro_crate.Crate; +import java.io.IOException; + /** * Generic interface for the strategy of the writer class. * This allows for flexible output types when implementing different writing strategies. * - * @param the type of the destination parameter + * @param the type of the destination parameter */ -public interface GenericWriterStrategy { +public interface GenericWriterStrategy { /** * Saves the given crate to the specified destination. * * @param crate The crate to save * @param destination The destination where the crate should be saved */ - void save(Crate crate, DESTINATION destination); + void save(Crate crate, DESTINATION_TYPE destination) throws IOException; } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteFolderStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteFolderStrategy.java new file mode 100644 index 00000000..8a4844ed --- /dev/null +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteFolderStrategy.java @@ -0,0 +1,80 @@ +package edu.kit.datamanager.ro_crate.writer; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; +import edu.kit.datamanager.ro_crate.Crate; +import edu.kit.datamanager.ro_crate.entities.data.DataEntity; +import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; +import org.apache.commons.io.FileUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.ByteArrayInputStream; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; + +/** + * A class for writing a crate to a folder. + * + * @author Nikola Tzotchev on 9.2.2022 г. + * @version 1 + */ +public class WriteFolderStrategy implements GenericWriterStrategy { + + private static final Logger logger = LoggerFactory.getLogger(WriteFolderStrategy.class); + + protected boolean writePreview = true; + + /** + * For internal use. Skips the preview generation when writing the crate. + * + * @return this instance of WriteFolderStrategy + * + * @deprecated May be removed in future versions. Not intended for public use. + */ + @Deprecated(since = "2.1.0", forRemoval = true) + public WriteFolderStrategy disablePreview() { + this.writePreview = false; + return this; + } + + @Override + public void save(Crate crate, String destination) throws IOException { + File file = new File(destination); + FileUtils.forceMkdir(file); + ObjectMapper objectMapper = MyObjectMapper.getMapper(); + JsonNode node = objectMapper.readTree(crate.getJsonMetadata()); + String str = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(node); + InputStream inputStream = new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8)); + + File json = new File(destination, "ro-crate-metadata.json"); + FileUtils.copyInputStreamToFile(inputStream, json); + inputStream.close(); + // save also the preview files to the crate destination + if (crate.getPreview() != null && this.writePreview) { + crate.getPreview().saveAllToFolder(file); + } + for (var e : crate.getUntrackedFiles()) { + if (e.isDirectory()) { + FileUtils.copyDirectoryToDirectory(e, file); + } else { + FileUtils.copyFileToDirectory(e, file); + } + } + for (DataEntity dataEntity : crate.getAllDataEntities()) { + savetoFile(dataEntity, file); + } + } + + private void savetoFile(DataEntity entity, File file) throws IOException { + if (entity.getPath() != null) { + if (entity.getPath().toFile().isDirectory()) { + FileUtils.copyDirectory(entity.getPath().toFile(), file.toPath().resolve(entity.getId()).toFile()); + } else { + FileUtils.copyFile(entity.getPath().toFile(), file.toPath().resolve(entity.getId()).toFile()); + } + } + } +} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteZipStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteZipStrategy.java new file mode 100644 index 00000000..003f7973 --- /dev/null +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteZipStrategy.java @@ -0,0 +1,33 @@ +package edu.kit.datamanager.ro_crate.writer; + +import edu.kit.datamanager.ro_crate.Crate; + +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +import java.io.FileOutputStream; +import java.io.IOException; +import java.io.OutputStream; + +/** + * Implementation of the writing strategy to provide a way of writing crates to + * a zip archive. + */ +public class WriteZipStrategy implements + GenericWriterStrategy, + ElnFormatWriter +{ + private static final Logger logger = LoggerFactory.getLogger(WriteZipStrategy.class); + protected ElnFormatWriter delegate = new WriteZipStreamStrategy(); + + @Override + public ElnFormatWriter usingElnStyle() { + this.delegate = this.delegate.withRootSubdirectory(); + return this; + } + + @Override + public void save(Crate crate, String destination) throws IOException { + this.delegate.save(crate, new FileOutputStream(destination)); + } +} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteZipStreamStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteZipStreamStrategy.java new file mode 100644 index 00000000..33572db2 --- /dev/null +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/WriteZipStreamStrategy.java @@ -0,0 +1,169 @@ +package edu.kit.datamanager.ro_crate.writer; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.ObjectMapper; + +import edu.kit.datamanager.ro_crate.Crate; +import edu.kit.datamanager.ro_crate.entities.data.DataEntity; +import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; + +import java.io.*; +import java.nio.charset.StandardCharsets; +import java.nio.file.Path; +import java.util.Optional; +import java.util.Set; +import java.util.UUID; + +import edu.kit.datamanager.ro_crate.preview.CratePreview; +import edu.kit.datamanager.ro_crate.util.FileSystemUtil; +import edu.kit.datamanager.ro_crate.util.ZipStreamUtil; +import net.lingala.zip4j.io.outputstream.ZipOutputStream; +import net.lingala.zip4j.model.ZipParameters; +import org.apache.commons.io.FileUtils; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; + +/** + * Implementation of the writing strategy to provide a way of writing crates to + * a zip archive. + */ +public class WriteZipStreamStrategy implements + GenericWriterStrategy, + ElnFormatWriter { + + private static final Logger logger = LoggerFactory.getLogger(WriteZipStreamStrategy.class); + public static final String TMP_DIR = "./.tmp/ro-crate-java/writer-zip-stream-strategy/"; + + /** + * Defines if the zip file will directly contain the crate, + * or if it will contain a subdirectory with the crate. + */ + protected boolean createRootSubdir = false; + + /** + * In streams, we do not have a file name yet (or do not know it), + * so we need to set a default name for the root subdirectory. + */ + protected String rootSubdirName = "content"; + + @Override + public ElnFormatWriter usingElnStyle() { + this.createRootSubdir = true; + return this; + } + + /** + * Sets the name of a root subdirectory in the zip file. + * Implicitly also enables the creation of a root subdirectory. + * If used for ELN files, note the subdirectory name should be the same as the zip + * files name. + * + * @param name the name of the subdirectory + * @return this instance of ReadZipStreamStrategy + */ + public WriteZipStreamStrategy setSubdirectoryName(String name) { + this.rootSubdirName = name; + this.createRootSubdir = true; + return this; + } + + @Override + public void save(Crate crate, OutputStream destination) throws IOException { + String innerFolderName = ""; + if (this.createRootSubdir) { + innerFolderName = FileSystemUtil.filterExtensionsFromFileName( + this.rootSubdirName, + Set.of("ELN", "ZIP")); + innerFolderName = FileSystemUtil.ensureTrailingSlash(innerFolderName); + } + try (ZipOutputStream zipFile = new ZipOutputStream(destination)) { + saveMetadataJson(crate, zipFile, innerFolderName); + saveDataEntities(crate, zipFile, innerFolderName); + savePreview(crate, zipFile, innerFolderName); + } + } + + private void saveDataEntities(Crate crate, ZipOutputStream zipStream, String prefix) throws IOException { + for (DataEntity dataEntity : crate.getAllDataEntities()) { + this.saveToStream(dataEntity, zipStream, prefix); + } + } + + private void saveMetadataJson(Crate crate, ZipOutputStream zipStream, String prefix) throws IOException { + // write the metadata.json file + ZipParameters zipParameters = new ZipParameters(); + zipParameters.setFileNameInZip(prefix + "ro-crate-metadata.json"); + ObjectMapper objectMapper = MyObjectMapper.getMapper(); + // we create an JsonNode only to have the file written pretty + JsonNode node = objectMapper.readTree(crate.getJsonMetadata()); + String str = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(node); + // write the ro-crate-metadata + + byte[] buff = new byte[4096]; + int readLen; + zipStream.putNextEntry(zipParameters); + try (InputStream inputStream = new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8))) { + while ((readLen = inputStream.read(buff)) != -1) { + zipStream.write(buff, 0, readLen); + } + } + zipStream.closeEntry(); + } + + private void savePreview(Crate crate, ZipOutputStream zipStream, String prefix) throws IOException { + Optional preview = Optional.ofNullable(crate.getPreview()); + if (preview.isEmpty()) { + return; + } + final String ID = UUID.randomUUID().toString(); + File tmpPreviewFolder = Path.of(TMP_DIR) + .resolve(ID) + .toFile(); + FileUtils.forceMkdir(tmpPreviewFolder); + FileUtils.forceDeleteOnExit(tmpPreviewFolder); + + preview.get().generate(crate, tmpPreviewFolder); + String[] paths = tmpPreviewFolder.list(); + if (paths == null) { + throw new IOException("No preview files found in temporary folder. Preview generation failed."); + } + for (String path : paths) { + File file = tmpPreviewFolder.toPath().resolve(path).toFile(); + if (file.isDirectory()) { + ZipStreamUtil.addFolderToZipStream( + zipStream, + file, + prefix + path); + } else { + ZipStreamUtil.addFileToZipStream( + zipStream, + file, + prefix + path); + } + } + try { + FileUtils.forceDelete(tmpPreviewFolder); + } catch (IOException e) { + logger.error("Could not delete temporary preview folder: {}", tmpPreviewFolder); + } + } + + private void saveToStream(DataEntity entity, ZipOutputStream zipStream, String prefix) throws IOException { + if (entity == null) { + return; + } + + boolean isDirectory = entity.getPath().toFile().isDirectory(); + if (isDirectory) { + ZipStreamUtil.addFolderToZipStream( + zipStream, + entity.getPath().toFile(), + prefix + entity.getId()); + } else { + ZipStreamUtil.addFileToZipStream( + zipStream, + entity.getPath().toFile(), + prefix + entity.getId()); + } + } +} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/Writers.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/Writers.java index 5b691ece..d1a41d66 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/Writers.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/Writers.java @@ -19,7 +19,7 @@ private Writers() {} * @return a new instance of {@link CrateWriter} for writing to a folder */ public static CrateWriter newFolderWriter() { - return new CrateWriter<>(new FolderStrategy()); + return new CrateWriter<>(new WriteFolderStrategy()); } /** @@ -28,7 +28,7 @@ public static CrateWriter newFolderWriter() { * @return a new instance of {@link CrateWriter} for writing to a zip stream */ public static CrateWriter newZipStreamWriter() { - return new CrateWriter<>(new ZipStreamStrategy()); + return new CrateWriter<>(new WriteZipStreamStrategy()); } /** @@ -37,6 +37,6 @@ public static CrateWriter newZipStreamWriter() { * @return a new instance of {@link CrateWriter} for writing to a zip file */ public static CrateWriter newZipPathWriter() { - return new CrateWriter<>(new ZipStrategy()); + return new CrateWriter<>(new WriteZipStrategy()); } } diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipStrategy.java deleted file mode 100644 index 6a11df87..00000000 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipStrategy.java +++ /dev/null @@ -1,68 +0,0 @@ -package edu.kit.datamanager.ro_crate.writer; - -import com.fasterxml.jackson.databind.JsonNode; -import com.fasterxml.jackson.databind.ObjectMapper; -import edu.kit.datamanager.ro_crate.Crate; -import edu.kit.datamanager.ro_crate.entities.data.DataEntity; -import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; -import net.lingala.zip4j.ZipFile; -import net.lingala.zip4j.exception.ZipException; -import net.lingala.zip4j.model.ZipParameters; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -import java.io.ByteArrayInputStream; -import java.io.IOException; -import java.io.InputStream; -import java.nio.charset.StandardCharsets; - -/** - * Implementation of the writing strategy to provide a way of writing crates to - * a zip archive. - */ -public class ZipStrategy implements GenericWriterStrategy { - - private static final Logger logger = LoggerFactory.getLogger(ZipStrategy.class); - - @Override - public void save(Crate crate, String destination) { - try (ZipFile zipFile = new ZipFile(destination)) { - saveMetadataJson(crate, zipFile); - saveDataEntities(crate, zipFile); - } catch (IOException e) { - // can not close ZipFile (threw Exception) - logger.error("Failed to write ro-crate to destination " + destination + ".", e); - } - } - - private void saveDataEntities(Crate crate, ZipFile zipFile) { - for (DataEntity dataEntity : crate.getAllDataEntities()) { - try { - dataEntity.saveToZip(zipFile); - } catch (ZipException e) { - logger.error("Could not save " + dataEntity.getId() + " to zip file!", e); - } - } - } - - private void saveMetadataJson(Crate crate, ZipFile zipFile) { - // write the metadata.json file - ZipParameters zipParameters = new ZipParameters(); - zipParameters.setFileNameInZip("ro-crate-metadata.json"); - ObjectMapper objectMapper = MyObjectMapper.getMapper(); - try { - // we create an JsonNode only to have the file written pretty - JsonNode node = objectMapper.readTree(crate.getJsonMetadata()); - String str = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(node); - try (InputStream inputStream = new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8))) { - // write the ro-crate-metadata - zipFile.addStream(inputStream, zipParameters); - } - if (crate.getPreview() != null) { - crate.getPreview().saveAllToZip(zipFile); - } - } catch (IOException e) { - logger.error("Exception writing ro-crate-metadata.json file to zip.", e); - } - } -} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipStreamStrategy.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipStreamStrategy.java deleted file mode 100644 index 818c064c..00000000 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipStreamStrategy.java +++ /dev/null @@ -1,77 +0,0 @@ -package edu.kit.datamanager.ro_crate.writer; - -import com.fasterxml.jackson.databind.JsonNode; -import com.fasterxml.jackson.databind.ObjectMapper; - -import edu.kit.datamanager.ro_crate.Crate; -import edu.kit.datamanager.ro_crate.entities.data.DataEntity; -import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; - -import java.io.ByteArrayInputStream; -import java.io.IOException; -import java.io.InputStream; -import java.io.OutputStream; -import java.nio.charset.StandardCharsets; -import net.lingala.zip4j.io.outputstream.ZipOutputStream; -import net.lingala.zip4j.model.ZipParameters; -import org.slf4j.Logger; -import org.slf4j.LoggerFactory; - -/** - * Implementation of the writing strategy to provide a way of writing crates to - * a zip archive. - */ -public class ZipStreamStrategy implements GenericWriterStrategy { - - private static final Logger logger = LoggerFactory.getLogger(ZipStreamStrategy.class); - - @Override - public void save(Crate crate, OutputStream destination) { - try (ZipOutputStream zipFile = new ZipOutputStream(destination)) { - saveMetadataJson(crate, zipFile); - saveDataEntities(crate, zipFile); - } catch (IOException e) { - // can not close ZipOutputStream (threw Exception) - logger.error("Failed to save ro-crate to zip stream.", e); - } - } - - private void saveDataEntities(Crate crate, ZipOutputStream zipStream) { - for (DataEntity dataEntity : crate.getAllDataEntities()) { - try { - dataEntity.saveToStream(zipStream); - } catch (IOException e) { - logger.error("Could not save {} to zip stream!", dataEntity.getId(), e); - } - } - } - - private void saveMetadataJson(Crate crate, ZipOutputStream zipStream) { - try { - // write the metadata.json file - ZipParameters zipParameters = new ZipParameters(); - zipParameters.setFileNameInZip("ro-crate-metadata.json"); - ObjectMapper objectMapper = MyObjectMapper.getMapper(); - // we create an JsonNode only to have the file written pretty - JsonNode node = objectMapper.readTree(crate.getJsonMetadata()); - String str = objectMapper.writerWithDefaultPrettyPrinter().writeValueAsString(node); - // write the ro-crate-metadata - - byte[] buff = new byte[4096]; - int readLen; - zipStream.putNextEntry(zipParameters); - try (InputStream inputStream = new ByteArrayInputStream(str.getBytes(StandardCharsets.UTF_8))) { - while ((readLen = inputStream.read(buff)) != -1) { - zipStream.write(buff, 0, readLen); - } - } - zipStream.closeEntry(); - - if (crate.getPreview() != null) { - crate.getPreview().saveAllToStream(str, zipStream); - } - } catch (IOException e) { - logger.error("Exception writing ro-crate-metadata.json file to zip.", e); - } - } -} diff --git a/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipWriter.java b/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipWriter.java index f5261003..c9c0735e 100644 --- a/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipWriter.java +++ b/src/main/java/edu/kit/datamanager/ro_crate/writer/ZipWriter.java @@ -4,7 +4,7 @@ * Implementation of the writing strategy to provide a way of writing crates to * a zip archive. * - * @deprecated Use {@link ZipStrategy} instead. + * @deprecated Use {@link WriteZipStrategy} instead. */ @Deprecated(since = "2.1.0", forRemoval = true) -public class ZipWriter extends ZipStrategy {} +public class ZipWriter extends WriteZipStrategy {} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/HelpFunctions.java b/src/test/java/edu/kit/datamanager/ro_crate/HelpFunctions.java index 8f32213f..24a57a07 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/HelpFunctions.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/HelpFunctions.java @@ -4,6 +4,7 @@ import com.fasterxml.jackson.databind.JsonNode; import com.fasterxml.jackson.databind.ObjectMapper; +import com.fasterxml.jackson.databind.SerializationFeature; import edu.kit.datamanager.ro_crate.entities.AbstractEntity; import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; import edu.kit.datamanager.ro_crate.special.JsonUtilFunctions; @@ -11,10 +12,13 @@ import org.apache.commons.io.FileUtils; import io.json.compare.JSONCompare; import io.json.compare.JsonComparator; +import org.opentest4j.AssertionFailedError; import java.io.File; import java.io.IOException; import java.io.InputStream; +import java.nio.file.Files; +import java.nio.file.Path; import java.util.Map; import java.util.function.Function; import java.util.stream.Collectors; @@ -34,8 +38,7 @@ public static void compareEntityWithFile(AbstractEntity entity, String string) t public static void compare(JsonNode node1, JsonNode node2, Boolean equals) { var comparator = new JsonComparator() { - public boolean compareValues(Object expected, Object actual) { - + public boolean compareValues(Object expected, Object actual) { return expected.equals(actual); } @@ -43,6 +46,7 @@ public boolean compareFields(String expected, String actual) { return expected.equals(actual); } }; + if (equals) { JSONCompare.assertMatches(node1, node2, comparator); } else { @@ -73,6 +77,32 @@ public static void compareTwoCrateJson(Crate crate1, Crate crate2) throws JsonPr compare(node1, node2, true); } + public static void prettyPrintJsonString(String minimalJsonMetadata) { + try { + ObjectMapper objectMapper = new ObjectMapper(); + JsonNode jsonNode = objectMapper.readTree(minimalJsonMetadata); + // Enable pretty printing + String prettyJson = objectMapper + .enable(SerializationFeature.INDENT_OUTPUT) + .writeValueAsString(jsonNode); + // Print the pretty JSON + System.out.println(prettyJson); + } catch (JsonProcessingException e) { + throw new AssertionFailedError("Not able to process string as JSON!", e); + } + } + + public static void printAndAssertEquals(RoCrate crate, String pathToResource) { + // So you get something to see + prettyPrintJsonString(crate.getJsonMetadata()); + // Compare with the example from the specification + try { + HelpFunctions.compareCrateJsonToFileInResources(crate, pathToResource); + } catch (IOException e) { + throw new AssertionFailedError("Missing resources file!", e); + } + } + public static void compareCrateJsonToFileInResources(File file1, File file2) throws IOException { ObjectMapper objectMapper = MyObjectMapper.getMapper(); JsonNode node1 = JsonUtilFunctions.unwrapSingleArray(objectMapper.readTree(file1)); @@ -80,6 +110,13 @@ public static void compareCrateJsonToFileInResources(File file1, File file2) thr compare(node1, node2, true); } + /** + * Compares the JSON metadata of a Crate object with a JSON file in the resources directory. + * + * @param crate1 The Crate object to compare. + * @param jsonFileString The path to the JSON file in the resources directory. + * @throws IOException If an error occurs while reading the JSON file. + */ public static void compareCrateJsonToFileInResources(Crate crate1, String jsonFileString) throws IOException { InputStream inputStream = HelpFunctions.class.getResourceAsStream( jsonFileString); @@ -117,4 +154,25 @@ public static boolean compareTwoDir(File dir1, File dir2) throws IOException { } return true; } + + /** + * Prints the file tree of the given directory for debugging and understanding + * a test more quickly. + * + * @param directoryToPrint the directory to print + * @throws IOException if an error occurs while printing the file tree + */ + @SuppressWarnings("resource") + public static void printFileTree(Path directoryToPrint) throws IOException { + // Print all files recursively in a tree structure for debugging + System.out.printf("Files in %s:%n", directoryToPrint.getFileName().toString()); + Files.walk(directoryToPrint) + .forEach(path -> { + if (!path.toAbsolutePath().equals(directoryToPrint.toAbsolutePath())) { + int depth = path.relativize(directoryToPrint).getNameCount(); + String prefix = " ".repeat(depth); + System.out.printf("%s%s%s%n", prefix, "└── ", path.getFileName()); + } + }); + } } diff --git a/src/test/java/edu/kit/datamanager/ro_crate/crate/BuilderSpec12Test.java b/src/test/java/edu/kit/datamanager/ro_crate/crate/BuilderSpec12Test.java index eddd93ee..270127af 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/crate/BuilderSpec12Test.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/crate/BuilderSpec12Test.java @@ -4,6 +4,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; +import java.io.IOException; import java.net.URI; import java.net.URISyntaxException; import java.util.Collection; @@ -37,7 +38,7 @@ void testAppendConformsTo() throws URISyntaxException { } @Test - void testModificationOfDraftCrate() throws URISyntaxException { + void testModificationOfDraftCrate() throws URISyntaxException, IOException { String path = this.getClass().getResource("/crates/spec-1.2-DRAFT/minimal-with-conformsTo-Array").getPath(); RoCrate crate = Readers.newFolderReader().readCrate(path); Collection existingProfiles = crate.getProfiles(); diff --git a/src/test/java/edu/kit/datamanager/ro_crate/crate/ReadAndWriteTest.java b/src/test/java/edu/kit/datamanager/ro_crate/crate/ReadAndWriteTest.java index 3c2e49b5..ca742ded 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/crate/ReadAndWriteTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/crate/ReadAndWriteTest.java @@ -49,7 +49,7 @@ void testReadingAndWriting(@TempDir Path path) throws IOException { @SuppressWarnings("DataFlowIssue") @Test - void testReadCrateWithHasPartHierarchy() { + void testReadCrateWithHasPartHierarchy() throws IOException { CrateReader reader = Readers.newFolderReader(); RoCrate crate = reader.readCrate(ReadAndWriteTest.class.getResource("/crates/hasPartHierarchy").getPath()); assertEquals(1, crate.getAllContextualEntities().size()); diff --git a/src/test/java/edu/kit/datamanager/ro_crate/crate/TestRemoveAddContext.java b/src/test/java/edu/kit/datamanager/ro_crate/crate/TestRemoveAddContext.java index b1f6e21b..31c0dc8e 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/crate/TestRemoveAddContext.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/crate/TestRemoveAddContext.java @@ -6,6 +6,7 @@ import com.fasterxml.jackson.core.JsonProcessingException; +import java.io.IOException; import java.util.Objects; import java.util.Set; @@ -17,7 +18,7 @@ public class TestRemoveAddContext { private RoCrate crateWithComplexContext; @BeforeEach - void setup() { + void setup() throws IOException { String crateManifestPath = "/crates/extendedContextExample/"; crateManifestPath = Objects.requireNonNull(TestRemoveAddContext.class.getResource(crateManifestPath)).getPath(); this.crateWithComplexContext = Readers.newFolderReader().readCrate(crateManifestPath); diff --git a/src/test/java/edu/kit/datamanager/ro_crate/crate/preview/PreviewCrateTest.java b/src/test/java/edu/kit/datamanager/ro_crate/crate/preview/PreviewCrateTest.java index 1ecc2062..92d70449 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/crate/preview/PreviewCrateTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/crate/preview/PreviewCrateTest.java @@ -22,7 +22,7 @@ public class PreviewCrateTest { @Test - void testAutomaticPreview(@TempDir Path temp) { + void testAutomaticPreview(@TempDir Path temp) throws IOException { Path location = temp.resolve("ro_crate1"); RoCrate crate = new RoCrate.RoCrateBuilder("name", "description", "2024", "https://creativecommons.org/licenses/by-nc-sa/3.0/au/") .setPreview(new AutomaticPreview()) @@ -33,7 +33,7 @@ void testAutomaticPreview(@TempDir Path temp) { } @Test - void testAutomaticPreviewAddingLater(@TempDir Path temp) { + void testAutomaticPreviewAddingLater(@TempDir Path temp) throws IOException { Path location = temp.resolve("ro_crate2"); RoCrate crate = new RoCrate.RoCrateBuilder("name", "description", "2024", "https://creativecommons.org/licenses/by-nc-sa/3.0/au/") .setPreview(null)//disable preview to allow to compare folders before and after @@ -47,7 +47,7 @@ void testAutomaticPreviewAddingLater(@TempDir Path temp) { } @Test - void testCustomPreview(@TempDir Path temp) { + void testCustomPreview(@TempDir Path temp) throws IOException { Path location = temp.resolve("ro_crate1"); RoCrate crate = new RoCrate.RoCrateBuilder("name", "description", "2024", "https://creativecommons.org/licenses/by-nc-sa/3.0/au/") .setPreview(new CustomPreview()) diff --git a/src/test/java/edu/kit/datamanager/ro_crate/crate/realexamples/RealTest.java b/src/test/java/edu/kit/datamanager/ro_crate/crate/realexamples/RealTest.java index ae206b7c..19ba2a83 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/crate/realexamples/RealTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/crate/realexamples/RealTest.java @@ -12,7 +12,6 @@ import edu.kit.datamanager.ro_crate.entities.data.DataSetEntity; import edu.kit.datamanager.ro_crate.entities.data.FileEntity; import edu.kit.datamanager.ro_crate.externalproviders.personprovider.OrcidProvider; -import edu.kit.datamanager.ro_crate.reader.CrateReader; import edu.kit.datamanager.ro_crate.reader.Readers; import org.apache.commons.io.FileUtils; @@ -23,17 +22,17 @@ import java.nio.charset.Charset; import java.nio.file.Path; import static org.junit.jupiter.api.Assertions.assertEquals; +import static org.junit.jupiter.api.Assertions.assertNotNull; -class RealTest { - - @SuppressWarnings("java:S2699") // disable warning about missing assertions +class RealTest +{ @Test void testWithIDRCProject(@TempDir Path temp) throws IOException { - - CrateReader reader = Readers.newFolderReader(); final String locationMetadataFile = "/crates/other/idrc_project/ro-crate-metadata.json"; - Crate crate = reader.readCrate(RealTest.class.getResource("/crates/other/idrc_project").getPath()); + Crate crate = Readers.newFolderReader() + .readCrate(RealTest.class.getResource("/crates/other/idrc_project").getPath()); + assertNotNull(crate); HelpFunctions.compareCrateJsonToFileInResources(crate, locationMetadataFile); Path newFile = temp.resolve("new_file.txt"); @@ -47,6 +46,7 @@ void testWithIDRCProject(@TempDir Path temp) throws IOException { .build()); PersonEntity person = OrcidProvider.getPerson("https://orcid.org/0000-0001-9842-9718"); + assertNotNull(person); crate.addContextualEntity(person); // problem diff --git a/src/test/java/edu/kit/datamanager/ro_crate/entities/AbstractEntityTest.java b/src/test/java/edu/kit/datamanager/ro_crate/entities/AbstractEntityTest.java new file mode 100644 index 00000000..678642ba --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/entities/AbstractEntityTest.java @@ -0,0 +1,111 @@ +package edu.kit.datamanager.ro_crate.entities; + +import com.fasterxml.jackson.databind.JsonNode; +import com.fasterxml.jackson.databind.node.ArrayNode; +import com.fasterxml.jackson.databind.node.ObjectNode; +import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.NullAndEmptySource; +import org.junit.jupiter.params.provider.ValueSource; + +import java.util.Optional; + +import static org.junit.jupiter.api.Assertions.*; + +class AbstractEntityTest { + + @ParameterizedTest + @NullAndEmptySource + @ValueSource(strings = {" ", "\t", "\n"}) + void mergeIdIntoValue_withInvalidId_returnsEmpty(String invalidId) { + Optional result = AbstractEntity.mergeIdIntoValue(invalidId, null); + assertTrue(result.isEmpty(), "Should return empty Optional for invalid ID"); + + Optional result2 = AbstractEntity.mergeIdIntoValue( + invalidId, + MyObjectMapper.getMapper().createObjectNode() + ); + assertTrue(result2.isEmpty(), "Should return empty Optional for invalid ID"); + } + + @Test + void mergeIdIntoValue_withNullCurrentValue_returnsIdObject() { + String id = "test-id"; + Optional result = AbstractEntity.mergeIdIntoValue(id, null); + + // Should return a value for valid ID + JsonNode node = result.orElseThrow(); + assertTrue(node.isObject(), "Should return an object"); + assertEquals(id, node.get("@id").asText(), "Should contain the ID"); + } + + @Test + void mergeIdIntoValue_withExistingIdAsString_returnsEmpty() { + String id = "test-id"; + JsonNode currentValue = MyObjectMapper.getMapper().valueToTree(id); + + Optional result = AbstractEntity.mergeIdIntoValue(id, currentValue); + assertTrue(result.isEmpty(), "Should return empty when ID already exists as string"); + } + + @Test + void mergeIdIntoValue_withExistingIdObject_returnsEmpty() { + String id = "test-id"; + ObjectNode currentValue = MyObjectMapper.getMapper().createObjectNode(); + currentValue.put("@id", id); + + Optional result = AbstractEntity.mergeIdIntoValue(id, currentValue); + assertTrue(result.isEmpty(), "Should return empty when ID already exists as object"); + } + + @Test + void mergeIdIntoValue_withNonArrayValue_createsArray() { + String id = "new-id"; + String existingId = "existing-id"; + ObjectNode existingValue = MyObjectMapper.getMapper().createObjectNode(); + existingValue.put("@id", existingId); + + Optional result = AbstractEntity.mergeIdIntoValue(id, existingValue); + + assertTrue(result.isPresent(), "Should return a value"); + JsonNode node = result.get(); + assertTrue(node.isArray(), "Should be converted to array"); + assertEquals(2, node.size(), "Should contain both values"); + assertEquals(existingId, node.get(0).get("@id").asText(), "Should contain existing ID"); + assertEquals(id, node.get(1).get("@id").asText(), "Should contain new ID"); + } + + @Test + void mergeIdIntoValue_withExistingArray_addsToArray() { + String id = "new-id"; + String existingId = "existing-id"; + + ArrayNode currentValue = MyObjectMapper.getMapper().createArrayNode(); + ObjectNode existingIdObj = MyObjectMapper.getMapper().createObjectNode(); + existingIdObj.put("@id", existingId); + currentValue.add(existingIdObj); + + Optional result = AbstractEntity.mergeIdIntoValue(id, currentValue); + + assertTrue(result.isPresent(), "Should return a value"); + JsonNode node = result.get(); + assertTrue(node.isArray(), "Should remain an array"); + assertEquals(2, node.size(), "Should contain both values"); + assertEquals(existingId, node.get(0).get("@id").asText(), "Should contain existing ID"); + assertEquals(id, node.get(1).get("@id").asText(), "Should contain new ID"); + } + + @Test + void mergeIdIntoValue_withExistingArrayContainingId_returnsEmpty() { + String id = "test-id"; + ArrayNode currentValue = MyObjectMapper.getMapper().createArrayNode(); + ObjectNode existingIdObj = MyObjectMapper.getMapper().createObjectNode(); + existingIdObj.put("@id", id); + currentValue.add(existingIdObj); + + Optional result = AbstractEntity.mergeIdIntoValue(id, currentValue); + assertTrue(result.isEmpty(), "Should return empty when ID already exists in array"); + } +} + diff --git a/src/test/java/edu/kit/datamanager/ro_crate/entities/contextual/ContextualEntityTest.java b/src/test/java/edu/kit/datamanager/ro_crate/entities/contextual/ContextualEntityTest.java index 575c941e..7bc868f3 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/entities/contextual/ContextualEntityTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/entities/contextual/ContextualEntityTest.java @@ -1,13 +1,16 @@ package edu.kit.datamanager.ro_crate.entities.contextual; -import static org.junit.jupiter.api.Assertions.assertTrue; - import java.io.IOException; +import com.fasterxml.jackson.core.JsonProcessingException; +import com.fasterxml.jackson.databind.node.ObjectNode; import edu.kit.datamanager.ro_crate.HelpFunctions; +import edu.kit.datamanager.ro_crate.objectmapper.MyObjectMapper; import org.junit.jupiter.api.Test; +import static org.junit.jupiter.api.Assertions.*; + /** * @author Nikola Tzotchev on 5.2.2022 г. * @version 1 @@ -45,4 +48,102 @@ void testSerialization() throws IOException { assertTrue(place.getLinkedTo().contains(geo.getId())); HelpFunctions.compareEntityWithFile(place, "/json/entities/contextual/place.json"); } + + @Test + void testAddAllValidCase() throws JsonProcessingException { + ContextualEntity first = new ContextualEntity.ContextualEntityBuilder() + .setId("#b4168a98-8534-4c6d-a568-64a55157b656") + .addType("GeoCoordinates") + .addProperty("latitude", "-33.7152") + .addProperty("longitude", "150.30119") + .addProperty("name", "Latitude: -33.7152 Longitude: 150.30119") + .build(); + + String allProperties = """ + { + "@id": "#b4168a98-8534-4c6d-a568-64a55157b656", + "@type": "GeoCoordinates", + "latitude": "-33.7152", + "longitude": "150.30119", + "name": "Latitude: -33.7152 Longitude: 150.30119" + } + """; + + ObjectNode properties = MyObjectMapper.getMapper() + .readValue(allProperties, ObjectNode.class); + ContextualEntity second = new ContextualEntity.ContextualEntityBuilder() + .setAllIfValid(properties) + .build(); + assertEquals(second.getProperties(), first.getProperties()); + } + + @Test + void testAddAllInvalidCase() throws JsonProcessingException { + ContextualEntity first = new ContextualEntity.ContextualEntityBuilder() + .setId("#b4168a98-8534-4c6d-a568-64a55157b656") + .addType("GeoCoordinates") + .addProperty("latitude", "-33.7152") + .addProperty("longitude", "150.30119") + .addProperty("name", "Latitude: -33.7152 Longitude: 150.30119") + .build(); + + String allProperties = """ + { + "wrong property": {"any": "value"}, + "@id": "#b4168a98-8534-4c6d-a568-64a55157b656", + "@type": "GeoCoordinates", + "latitude": "-33.7152", + "longitude": "150.30119", + "name": "Latitude: -33.7152 Longitude: 150.30119" + } + """; + + ObjectNode properties = MyObjectMapper.getMapper() + .readValue(allProperties, ObjectNode.class); + ContextualEntity second = new ContextualEntity.ContextualEntityBuilder() + .setId("second") + .setAllIfValid(properties) + .build(); + assertNotEquals(second.getProperties(), first.getProperties()); + ObjectNode empty = new ContextualEntity.ContextualEntityBuilder() + .setId("second") + .build() + .getProperties(); + assertEquals(empty, second.getProperties()); + } + + @Test + void testAddAllUnsafeDoesInvalidCase() throws JsonProcessingException { + ContextualEntity first = new ContextualEntity.ContextualEntityBuilder() + .setId("#b4168a98-8534-4c6d-a568-64a55157b656") + .addType("GeoCoordinates") + .addProperty("latitude", "-33.7152") + .addProperty("longitude", "150.30119") + .addProperty("name", "Latitude: -33.7152 Longitude: 150.30119") + .build(); + + String allProperties = """ + { + "wrong property": {"any": "value"}, + "@id": "#b4168a98-8534-4c6d-a568-64a55157b656", + "@type": "GeoCoordinates", + "latitude": "-33.7152", + "longitude": "150.30119", + "name": "Latitude: -33.7152 Longitude: 150.30119" + } + """; + + ObjectNode properties = MyObjectMapper.getMapper() + .readValue(allProperties, ObjectNode.class); + ContextualEntity second = new ContextualEntity.ContextualEntityBuilder() + .setId("second") + .setAllUnsafe(properties) + .build(); + assertNotEquals(second.getProperties(), first.getProperties()); + ObjectNode empty = new ContextualEntity.ContextualEntityBuilder() + .setId("second") + .build() + .getProperties(); + assertNotEquals(empty, second.getProperties()); + } } diff --git a/src/test/java/edu/kit/datamanager/ro_crate/examples/ExamplesOfSpecificationV1p1Test.java b/src/test/java/edu/kit/datamanager/ro_crate/examples/ExamplesOfSpecificationV1p1Test.java new file mode 100644 index 00000000..d699eeae --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/examples/ExamplesOfSpecificationV1p1Test.java @@ -0,0 +1,407 @@ +package edu.kit.datamanager.ro_crate.examples; + +import edu.kit.datamanager.ro_crate.RoCrate; +import edu.kit.datamanager.ro_crate.entities.AbstractEntity; +import edu.kit.datamanager.ro_crate.entities.contextual.ContextualEntity; +import edu.kit.datamanager.ro_crate.entities.contextual.OrganizationEntity; +import edu.kit.datamanager.ro_crate.entities.contextual.PersonEntity; +import edu.kit.datamanager.ro_crate.entities.contextual.PlaceEntity; +import edu.kit.datamanager.ro_crate.entities.data.*; + +import edu.kit.datamanager.ro_crate.writer.CrateWriter; +import org.junit.jupiter.api.Test; + +import java.net.URI; +import java.nio.file.Paths; + +import static edu.kit.datamanager.ro_crate.HelpFunctions.printAndAssertEquals; + +/** + * This class contains examples of the RO-Crate specification version 1.1. + *

+ * This is supposed to serve both as a user guide and as a test for the implementation. + * Executing a test may also print some interesting information to the console. + */ +public class ExamplesOfSpecificationV1p1Test { + + /** + * From: + * Minimal Example + * (location in repo) + *

+ * This example produces a minimal crate with a + * name, description, date, license and identifier. + *

+ * This example produces the same result as + * {@link #testMinimalCrateWithoutCrateBuilder()}, but using more convenient APIs. + */ + @Test + void testMinimalCrateConvenient() { + String licenseID = "https://creativecommons.org/licenses/by-nc-sa/3.0/au/"; + RoCrate minimal = new RoCrate.RoCrateBuilder( + "Data files associated with the manuscript:Effects of facilitated family case conferencing for ...", + "Palliative care planning for nursing home residents with advanced dementia ...", + "2017", + licenseID + ) + // We already had to set the license ID in the builder, + // but we can override it with more details to fit the example: + .setLicense( new ContextualEntity.ContextualEntityBuilder() + .addType("CreativeWork") + .setId(licenseID) + .addProperty("description", "This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Australia License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/au/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.") + .addProperty("identifier", licenseID) + .addProperty("name", "Attribution-NonCommercial-ShareAlike 3.0 Australia (CC BY-NC-SA 3.0 AU)") + .build() + ) + .addIdentifier("https://doi.org/10.4225/59/59672c09f4a4b") + .build(); + + printAndAssertEquals(minimal, "/spec-v1.1-example-json-files/minimal.json"); + } + + /** + * From: + * Minimal Example + * (location in repo) + *

+ * In this example, the minimal crate is created without the builder. + * This should only be done if necessary: Use the builder if possible. + * This example produces the same result as {@link #testMinimalCrateConvenient()}. + */ + @Test + void testMinimalCrateWithoutCrateBuilder() { + RoCrate minimal = new RoCrate(); + + ContextualEntity license = new ContextualEntity.ContextualEntityBuilder() + .addType("CreativeWork") + .setId("https://creativecommons.org/licenses/by-nc-sa/3.0/au/") + .addProperty("description", "This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Australia License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/au/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.") + .addProperty("identifier", "https://creativecommons.org/licenses/by-nc-sa/3.0/au/") + .addProperty("name", "Attribution-NonCommercial-ShareAlike 3.0 Australia (CC BY-NC-SA 3.0 AU)") + .build(); + + minimal.setRootDataEntity(new RootDataEntity.RootDataEntityBuilder() + .addProperty("identifier", "https://doi.org/10.4225/59/59672c09f4a4b") + .addProperty("datePublished", "2017") + .addProperty("name", "Data files associated with the manuscript:Effects of facilitated family case conferencing for ...") + .addProperty("description", "Palliative care planning for nursing home residents with advanced dementia ...") + .setLicense(license) + .build()); + + // This is pretty low-level. We are considering hiding/replacing this detailed API in major versions, + // so tell us (for example, open an issue) if you have a use case for it! + minimal.setJsonDescriptor(new ContextualEntity.ContextualEntityBuilder() + .setId("ro-crate-metadata.json") + .addType("CreativeWork") + .addIdProperty("about", "./") + .addIdProperty("conformsTo", "https://w3id.org/ro/crate/1.1") + .build() + ); + minimal.addContextualEntity(license); + + printAndAssertEquals(minimal, "/spec-v1.1-example-json-files/minimal.json"); + } + + // https://www.researchobject.org/ro-crate/specification/1.1/data-entities.html#example-linking-to-a-file-and-folders + + /** + * From: + * "Example linking to a file and folders" + * (location in repo) + *

+ * This example adds a File(Entity) and a DataSet(Entity) to the crate. + * The file and the folder are referenced by their location. This way + * they will be copied to the crate when writing it using a + * {@link CrateWriter}. + * The name of the file and the folder will be implicitly set to the + * ID of the respective entity in order to conform to the specification. + *

+ * Here we use the inner builder classes for the construction of the + * crate. In contrast to {@link #testMinimalCrateWithoutCrateBuilder()}, + * we do not have to care about specification details. + */ + @Test + void testLinkingToFileAndFolders() { + RoCrate crate = new RoCrate.RoCrateBuilder() + .addDataEntity( + new FileEntity.FileEntityBuilder() + // This will tell us where the file is located. It will be copied to the crate. + .setLocation(Paths.get("path to file")) + // If no ID is given explicitly, the ID will be set to the filename. + // Changing the ID means also to set the file name within the crate! + .setId("cp7glop.ai") + .addProperty("name", "Diagram showing trend to increase") + .addProperty("contentSize", "383766") + .addProperty("description", "Illustrator file for Glop Pot") + .setEncodingFormat("application/pdf") + .build() + ) + .addDataEntity( + new DataSetEntity.DataSetBuilder() + .setLocation(Paths.get("path_to_files")) + .setId("lots_of_little_files/") + .addProperty("name", "Too many files") + .addProperty("description", "This directory contains many small files, that we're not going to describe in detail.") + .build() + ) + .build(); + + printAndAssertEquals(crate, "/spec-v1.1-example-json-files/files-and-folders.json"); + } + + /** + * From: + * Example with web-based data entities + * (location in repo) + *

+ * This example adds twp FileEntities to the crate. + * One is a local file, the other one is located in the web + * and will not be copied to the crate. + */ + @Test + void testWebBasedDataEntities() { + RoCrate crate = new RoCrate.RoCrateBuilder() + .addDataEntity( + new FileEntity.FileEntityBuilder() + .setLocation(Paths.get("README.md")) + .setId("survey-responses-2019.csv") + .addProperty("name", "Survey responses") + .addProperty("contentSize", "26452") + .setEncodingFormat("text/csv") + .build() + ) + .addDataEntity( + new FileEntity.FileEntityBuilder() + .setLocation(URI.create("https://zenodo.org/record/3541888/files/ro-crate-1.0.0.pdf")) + .addProperty("name", "RO-Crate specification") + .addProperty("contentSize", "310691") + .addProperty("description", "RO-Crate specification") + .setEncodingFormat("application/pdf") + .build() + ) + .build(); + + printAndAssertEquals(crate, "/spec-v1.1-example-json-files/web-based-data-entities.json"); + } + + /** + * From: + * Example with file, author, and location + * (location in repo) + *

+ * This example shows how to connect entities. If there is no specific method like + * {@link DataEntity.DataEntityBuilder#addAuthor(String)} for referencing other + * entities, one can use the more generic + * {@link AbstractEntity.AbstractEntityBuilder#addIdProperty(String, AbstractEntity)} + * or {@link AbstractEntity.AbstractEntityBuilder#addIdProperty(String, String)}. + *

+ * Important Note! If you connect entities, make sure all entities are being + * added to the crate. We currently can't enforce this properly yet. + */ + @Test + void testWithFileAuthorLocation() { + // These two entities will be connected to others later on. Therefore, we make + // them easier referencable. Referencing can be done using the whole entity or + // its ID. + final PersonEntity alice = new PersonEntity.PersonEntityBuilder() + .setId("#alice") + .addProperty("name", "Alice") + .addProperty("description", "One of hopefully many Contextual Entities") + .build(); + final PlaceEntity park = new PlaceEntity.PlaceEntityBuilder() + .setId(URI.create("http://sws.geonames.org/8152662/").toString()) + .addProperty("name", "Catalina Park") + .build(); + final String licenseId = "https://spdx.org/licenses/CC-BY-NC-SA-4.0"; + + final RoCrate crate = new RoCrate.RoCrateBuilder( + "Example RO-Crate", + "The RO-Crate Root Data Entity", + "2020", + licenseId + ) + .addContextualEntity(park) + .addContextualEntity(alice) + .addDataEntity( + new FileEntity.FileEntityBuilder() + .setLocation(Paths.get(".......")) + .setId("data2.txt") + .build() + ) + .addDataEntity( + new FileEntity.FileEntityBuilder() + .setLocation(Paths.get(".......")) + .setId("data1.txt") + .addProperty("description", "One of hopefully many Data Entities") + // ↓ This is the specific way to add an author + .addAuthor(alice.getId()) + // ↓ This is the generic way to add a location or other relations + .addIdProperty("contentLocation", park) + .build() + ) + .build(); + + /* + The builder enforces to provide a license and a publishing date, + but the example does not have them. So we have to remove them below: + */ + + // **Note**: When you add a license, even if only by a string, the crate will + // implicitly also get a small ContextEntity for this license. When we remove + // this (any) entity, all references to it will be removed as well to ensure + // consistency within the crate. Therefore, there will be no trace left of + // the license. + crate.deleteEntityById(licenseId); + + // The datePublished property is a simple property and simple to remove without + // any further internal checks. + crate.getRootDataEntity().removeProperty("datePublished"); + + printAndAssertEquals(crate, "/spec-v1.1-example-json-files/file-author-location.json"); + } + + /** + * From: + * Example with complete workflow + * (location in repo) + *

+ * This example shows how to connect entities. If there is no specific method like + * {@link DataEntity.DataEntityBuilder#addAuthor(String)} for referencing other + * entities, one can use the more generic + * {@link AbstractEntity.AbstractEntityBuilder#addIdProperty(String, AbstractEntity)} + * or {@link AbstractEntity.AbstractEntityBuilder#addIdProperty(String, String)}. + *

+ * Important Note! If you connect entities, make sure all entities are being + * added to the crate. We currently can't enforce this properly yet. + */ + @Test + void testCompleteWorkflowExample() { + final String licenseId = "https://spdx.org/licenses/CC-BY-NC-SA-4.0"; + ContextualEntity license = new ContextualEntity.ContextualEntityBuilder() + .addType("CreativeWork") + .setId(licenseId) + .addProperty("name", "Creative Commons Attribution Non Commercial Share Alike 4.0 International") + .addProperty("alternateName", "CC-BY-NC-SA-4.0") + .build(); + ContextualEntity knime = new ContextualEntity.ContextualEntityBuilder() + .setId("#knime") + .addType("ComputerLanguage") + .addProperty("name", "KNIME Analytics Platform") + .addProperty("alternateName", "KNIME") + .addProperty("url", "https://www.knime.com/whats-new-in-knime-41") + .addProperty("version", "4.1.3") + .build(); + OrganizationEntity workflowHub = new OrganizationEntity.OrganizationEntityBuilder() + .setId("#workflow-hub") + .addProperty("name", "Example Workflow Hub") + .addProperty("url", "http://example.com/workflows/") + .build(); + ContextualEntity fasta = new ContextualEntity.ContextualEntityBuilder() + .setId("http://edamontology.org/format_1929") + .addType("Thing") + .addProperty("name", "FASTA sequence format") + .build(); + ContextualEntity clustalW = new ContextualEntity.ContextualEntityBuilder() + .setId("http://edamontology.org/format_1982") + .addType("Thing") + .addProperty("name", "ClustalW alignment format") + .build(); + ContextualEntity ban = new ContextualEntity.ContextualEntityBuilder() + .setId("http://edamontology.org/format_2572") + .addType("Thing") + .addProperty("name", "BAM format") + .build(); + ContextualEntity nucSec = new ContextualEntity.ContextualEntityBuilder() + .setId("http://edamontology.org/data_2977") + .addType("Thing") + .addProperty("name", "Nucleic acid sequence") + .build(); + ContextualEntity nucAlign = new ContextualEntity.ContextualEntityBuilder() + .setId("http://edamontology.org/data_1383") + .addType("Thing") + .addProperty("name", "Nucleic acid sequence alignment") + .build(); + PersonEntity alice = new PersonEntity.PersonEntityBuilder() + .setId("#alice") + .addProperty("name", "Alice Brown") + .build(); + ContextualEntity requiredParam = new ContextualEntity.ContextualEntityBuilder() + .addType("FormalParameter") + .setId("#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b") + .addProperty("name", "genome_sequence") + .addProperty("valueRequired", true) + .addIdProperty("conformsTo", "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/") + .addIdProperty("additionalType", nucSec) + .addIdProperty("format", fasta) + .build(); + ContextualEntity clnParam = new ContextualEntity.ContextualEntityBuilder() + .addType("FormalParameter") + .setId("#6c703fee-6af7-4fdb-a57d-9e8bc4486044") + .addProperty("name", "cleaned_sequence") + .addIdProperty("conformsTo", "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/") + .addIdProperty("additionalType", nucSec) + .addIdProperty("encodingFormat", ban) + .build(); + ContextualEntity alignParam = new ContextualEntity.ContextualEntityBuilder() + .addType("FormalParameter") + .setId("#2f32b861-e43c-401f-8c42-04fd84273bdf") + .addProperty("name", "sequence_alignment") + .addIdProperty("conformsTo", "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/") + .addIdProperty("additionalType", nucAlign) + .addIdProperty("encodingFormat", clustalW) + .build(); + + RoCrate crate = new RoCrate.RoCrateBuilder( + "Example RO-Crate", + "The RO-Crate Root Data Entity", + "2020", + licenseId + ) + .setLicense(license) + .addContextualEntity(knime) + .addContextualEntity(workflowHub) + .addContextualEntity(fasta) + .addContextualEntity(clustalW) + .addContextualEntity(ban) + .addContextualEntity(nucSec) + .addContextualEntity(nucAlign) + .addContextualEntity(alice) + .addContextualEntity(requiredParam) + .addContextualEntity(clnParam) + .addContextualEntity(alignParam) + .addDataEntity( + new WorkflowEntity.WorkflowEntityBuilder() + .setId("workflow/alignment.knime") + .setLocation(Paths.get("src")) + .addIdProperty("conformsTo", "https://bioschemas.org/profiles/ComputationalWorkflow/0.5-DRAFT-2020_07_21/") + .addProperty("name", "Sequence alignment workflow") + .addIdProperty("programmingLanguage", "#knime") + // This example does not use the term "author"... + //.addAuthor("#alice") + // instead, it uses "creator": + .addIdProperty("creator", "#alice") + .addProperty("dateCreated", "2020-05-23") + .setLicense(licenseId) + .addInput("#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b") + .addOutput("#6c703fee-6af7-4fdb-a57d-9e8bc4486044") + .addOutput("#2f32b861-e43c-401f-8c42-04fd84273bdf") + .addProperty("url", "http://example.com/workflows/alignment") + .addProperty("version", "0.5.0") + .addIdProperty("sdPublisher", "#workflow-hub") + .build() + ) + .build(); + + // Similar to the previous example, this example from the specification + // spared out some details we now need to remove. + // Here we do not want to remove the license, only the reference to our root data entity. + // This is because (the way we constructed the crate) other entities use the license as well. + crate.getRootDataEntity().removeProperty("license"); + crate.getRootDataEntity().removeProperty("datePublished"); + crate.getRootDataEntity().removeProperty("name"); + crate.getRootDataEntity().removeProperty("description"); + + printAndAssertEquals(crate, "/spec-v1.1-example-json-files/complete-workflow-example.json"); + } +} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/examples/LearnByExampleTest.java b/src/test/java/edu/kit/datamanager/ro_crate/examples/LearnByExampleTest.java new file mode 100644 index 00000000..89c5a8f8 --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/examples/LearnByExampleTest.java @@ -0,0 +1,421 @@ +package edu.kit.datamanager.ro_crate.examples; + +import edu.kit.datamanager.ro_crate.HelpFunctions; +import edu.kit.datamanager.ro_crate.RoCrate; +import edu.kit.datamanager.ro_crate.entities.contextual.ContextualEntity; +import edu.kit.datamanager.ro_crate.entities.contextual.OrganizationEntity; +import edu.kit.datamanager.ro_crate.entities.contextual.PersonEntity; +import edu.kit.datamanager.ro_crate.entities.data.DataEntity; +import edu.kit.datamanager.ro_crate.entities.data.FileEntity; +import edu.kit.datamanager.ro_crate.externalproviders.organizationprovider.RorProvider; +import edu.kit.datamanager.ro_crate.externalproviders.personprovider.OrcidProvider; +import edu.kit.datamanager.ro_crate.preview.AutomaticPreview; +import edu.kit.datamanager.ro_crate.preview.StaticPreview; +import edu.kit.datamanager.ro_crate.reader.CrateReader; +import edu.kit.datamanager.ro_crate.reader.GenericReaderStrategy; +import edu.kit.datamanager.ro_crate.reader.Readers; +import edu.kit.datamanager.ro_crate.validation.JsonSchemaValidation; +import edu.kit.datamanager.ro_crate.validation.Validator; +import edu.kit.datamanager.ro_crate.writer.CrateWriter; +import edu.kit.datamanager.ro_crate.writer.WriteFolderStrategy; +import edu.kit.datamanager.ro_crate.writer.GenericWriterStrategy; +import edu.kit.datamanager.ro_crate.writer.Writers; +import org.apache.commons.io.FileUtils; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.*; +import java.net.URI; +import java.net.URL; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.Objects; + +import static org.junit.jupiter.api.Assertions.*; + +/** + * This class is meant to be a small example-driven introduction to the ro-crate-java library. + * It is meant to be read from top to bottom. + */ +public class LearnByExampleTest { + + /** + * This creates a valid, empty RO-Crate builder. + */ + static RoCrate.RoCrateBuilder NEW_STARTER_CRATE() { + return new RoCrate.RoCrateBuilder( + "name", + "description", + "2025", + "licenseIdentifier" + ); + } + + /** + * Calling the `build()` method on the builder creates a valid RO-Crate. + * Run this test to view the NEW_STARTER_CRATE() JSON in the console. + */ + @Test + void aSimpleCrate() { + RoCrate almostEmptyCrate = NEW_STARTER_CRATE().build(); + assertNotNull(almostEmptyCrate); + HelpFunctions.prettyPrintJsonString(almostEmptyCrate.getJsonMetadata()); + } + + /** + * This is how we can add things to a crate. + *

+ * Note that methods starting with `add` can be used multiple times to add more. + * For example, we can add multiple files or multiple contexts. + *

+ * On the other hand, methods starting with `set` will override previous calls. + *

+ * There may be inconsistencies yet, which are tracked here: Issue #242 + */ + @Test + void addingYourFirstEntity() { + RoCrate myFirstCrate = NEW_STARTER_CRATE() + // We can add new terms to our crate. The terms we can use are called "context". + .addValuePairToContext("Station", "www.station.com") + // We can also add whole contexts to our crate. + .addUrlToContext("contextUrl") + // Let's add a file to our crate. + .addDataEntity( + new FileEntity.FileEntityBuilder() + // For files (or folders, which are DataSetEntities), + // the ID determines the file name in the crate. + .setId("survey-responses-2019.csv") + // This is where we get the file from. The path will not be part of the metadata. + .setLocation(Paths.get("copy/from/this/file-and-rename-it.csv")) + // And now, the remaining metadata. + // Note that "name", "contentSize", and "encodingFormat" + // are already defined in our default context. + .addProperty("name", "Survey responses") + .addProperty("contentSize", "26452") + .addProperty("encodingFormat", "text/csv") + .build() + ) + // We could add more, but let's keep it simple for now. + //.addDataEntity(...) + //.addContextualEntity(...) + //... + .build(); + + assertNotNull(myFirstCrate); + HelpFunctions.prettyPrintJsonString(myFirstCrate.getJsonMetadata()); + } + + /** + * The library currently comes with three specialized DataEntities: + *

+ * 1. `DataSetEntity` + * 2. `FileEntity` (used in the example above) + * 3. `WorkflowEntity` + *

+ * If another type of `DataEntity` is required, + * the base class `DataEntity` can be used. Example: + */ + @Test + void specializingYourFirstEntity() { + RoCrate crate = NEW_STARTER_CRATE() + .addDataEntity( + // Let's do something custom: + new DataEntity.DataEntityBuilder() + // You need to add the type of your `DataEntity` + // because for DataEntity, there is no default. + .addType("CreativeWork") + .setId("myEntityInstance") + // Now that we are a CreativeWork instance, + // it is fine to use some of its properties. + .addProperty("https://schema.org/award", "Wow-award") + .build() + ) + .build(); + + assertNotNull(crate); + HelpFunctions.prettyPrintJsonString(crate.getJsonMetadata()); + } + + /** + * A `DataEntity` and its subclasses can have a file located on the web. + * In this case, it does not need to reside in a crate's folder. + * This can be useful for large, publicly available files, + * or in order to reuse or share files. + *

+ * Note: Technically, an entity pointing to a file on the web is just an entity + * that uses the URL as an ID. + */ + @Test + void referencingFilesOnTheWeb() { + // Let's say this is the file we would like to point at with an entity. + String lovelyFile = "https://github.com/kit-data-manager/ro-crate-java/issues/5"; + + RoCrate crate = NEW_STARTER_CRATE() + .addDataEntity( + // Build our entity to point to the file: + new FileEntity.FileEntityBuilder() + // Make it point to an external file. + .setLocation(URI.create(lovelyFile)) + // This would do the same: + .setId(lovelyFile) + // don't forget to add metadata! + .addProperty("description", "my new file that I added") + .build() + ) + .build(); + + assertNotNull(crate); + HelpFunctions.prettyPrintJsonString(crate.getJsonMetadata()); + } + + /** + * A `DataEntity` and its subclasses can have a local file associated with them, + * instead of one located on the web. + * + * @param tempDir We'll use this to create a temporary folder for our crate. + * @throws IOException If the file cannot be created or written to. + */ + @Test + void includingFilesIntoTheCrateFolder(@TempDir Path tempDir) throws IOException { + // Let's say this is the file we would like to point at with an entity. + String lovelyFile = tempDir.resolve("my/experiment.csv").toString(); + { + // (Let's quickly create a dummy file, but the rest will not make use of this knowledge.) + File lovelyFilePointer = new File(lovelyFile); + FileUtils.touch(lovelyFilePointer); + FileUtils.write(lovelyFilePointer, "My great experiment 001", "UTF-8"); + } + + // But in the crate we want it to be + String seriousExperimentFile = "fantastic-experiment/2025-01-01.csv"; + + RoCrate crate = NEW_STARTER_CRATE() + .addDataEntity( + // Build our entity to point to the file: + new FileEntity.FileEntityBuilder() + // Let's tell the library where to find and copy the file from. + .setLocation(Paths.get(lovelyFile)) + // Let's tell it to adjust the file name and path in the crate. + .setId(seriousExperimentFile) + .addProperty("description", "my new local file that I added") + .build() + ) + .build(); + + assertNotNull(crate); + HelpFunctions.prettyPrintJsonString(crate.getJsonMetadata()); + + // Let's write it to disk and see if the file is there! + // (We'll discuss writing and reading crates later on.) + Path crateFolder = tempDir.resolve("myCrate"); + Writers.newFolderWriter().save(crate, crateFolder.toString()); + assertTrue(crateFolder.resolve(seriousExperimentFile).toFile().exists()); + } + + /** + * Contextual entities cannot be associated with a file: they are pure metadata + * To add a contextual entity to a crate you use the function + * {@link RoCrate.RoCrateBuilder#addContextualEntity(ContextualEntity)}. + *

+ * Some types of derived/specializes entities are: + *

+ * 1. `OrganizationEntity` + * 2. `PersonEntity` + * 3. `PlaceEntity` + *

+ * If you need another type of contextual entity, use the base class + * {@link ContextualEntity}, similar to how we did it in + * {@link #specializingYourFirstEntity()}. + *

+ * The library provides a way to automatically create contextual entities from + * external providers. Currently, support for [ORCID](https://orcid.org/) and + * [ROR](https://ror.org/) is implemented. + * Check the module {@link edu.kit.datamanager.ro_crate.externalproviders} for + * more implementations. + */ + @Test + void addingContextualEntities() { + PersonEntity person = OrcidProvider.getPerson("https://orcid.org/0000-0001-6575-1022"); + OrganizationEntity organization = RorProvider.getOrganization("https://ror.org/04t3en479"); + + RoCrate crate = NEW_STARTER_CRATE() + .addContextualEntity(person) + .addContextualEntity(organization) + .build(); + + assertNotNull(crate); + HelpFunctions.prettyPrintJsonString(crate.getJsonMetadata()); + } + + /** + * RO-Crates are file based, but in your application you may want to create a crate + * on the fly and directly send it somewhere else without storing it on disk. + * This is why we can't only write to a folder or a zip file, but also to a stream + * (containing the zip file). + *

+ * There is a generic interface to implement Writers (and Readers), so even more + * exotic use cases should be possible. The readers work the same way. + *

+ * - {@link GenericWriterStrategy} + * - {@link GenericReaderStrategy} + */ + @Test + void writingAndReadingCrates(@TempDir Path tempDir) throws IOException { + // Ok lets make a small, but not fully boring crate. + PersonEntity person = OrcidProvider.getPerson("https://orcid.org/0000-0001-6575-1022"); + OrganizationEntity organization = RorProvider.getOrganization("https://ror.org/04t3en479"); + + RoCrate crate = NEW_STARTER_CRATE() + .addContextualEntity(person) + .addContextualEntity(organization) + .build(); + + assertNotNull(crate); + HelpFunctions.prettyPrintJsonString(crate.getJsonMetadata()); + + { + // Now, let's write it to a folder. + Path folder = tempDir.resolve("folderCrate"); + Writers.newFolderWriter() + .save(crate, folder.toString()); + // and read it back. + RoCrate read = Readers.newFolderReader() + .readCrate(folder.toAbsolutePath().toString()); + + HelpFunctions.compareTwoCrateJson(crate, read); + } + + { + // Now, let's write it to a zip file. + Path zipFile = tempDir.resolve("zipCrate.zip"); + Writers.newZipPathWriter() + .save(crate, zipFile.toString()); + // and read it back. + RoCrate read = Readers.newZipPathReader() + .readCrate(zipFile.toAbsolutePath().toString()); + + HelpFunctions.compareTwoCrateJson(crate, read); + } + + { + // Now, let's write it to a zip stream. + Path zipStreamFile = tempDir.resolve("zipStreamCrate.zip"); + try (OutputStream outputStream = new FileOutputStream(zipStreamFile.toFile())) { + Writers.newZipStreamWriter().save(crate, outputStream); + } + // and read it back. + try (InputStream inputStream = new FileInputStream(zipStreamFile.toFile())) { + RoCrate read = Readers.newZipStreamReader() + .readCrate(inputStream); + + HelpFunctions.compareTwoCrateJson(crate, read); + } + } + } + + /** + * In {@link #writingAndReadingCrates(Path)} we already saw how to write or read + * a crate. We used the Readers and Writers classes to get the available options. + * But what if you want to write your own reader or writer strategy? + *

+ * Let's see how you can make a reader or writer, manually configuring the strategy. + */ + @Test + void writingAndReadingStrategies(@TempDir Path tempDir) throws IOException { + // Ok lets make a small, but not fully boring crate. + PersonEntity person = OrcidProvider.getPerson("https://orcid.org/0000-0001-6575-1022"); + OrganizationEntity organization = RorProvider.getOrganization("https://ror.org/04t3en479"); + + RoCrate crate = NEW_STARTER_CRATE() + .addContextualEntity(person) + .addContextualEntity(organization) + .build(); + + assertNotNull(crate); + HelpFunctions.prettyPrintJsonString(crate.getJsonMetadata()); + + // Now, let's write it to a folder. Note the used strategy could be replaced with your own. + Path folder = tempDir.resolve("folderCrate"); + new CrateWriter<>(new WriteFolderStrategy()) + .save(crate, folder.toString()); + // and read it back. + RoCrate read = new CrateReader<>( + // Note: There are two WriteFolderStrategy implementations, one for reading and one for writing. + // Java is a bit bad with imports, so we use the fully qualified name here. + new edu.kit.datamanager.ro_crate.reader.ReadFolderStrategy() + ) + .readCrate(folder.toAbsolutePath().toString()); + + HelpFunctions.compareTwoCrateJson(crate, read); + } + + /** + * RO-Crate specified there should be a human-readable preview of the crate. + * This is a HTML file that can be opened in a browser. + * ro-crate-java offers three different ways to create this file: + *

+ * - AutomaticPreview: Uses third-party library + * ro-crate-html-js, + * which must be installed separately via `npm install --global ro-crate-html-js`. + *

+ * - CustomPreview: Pure Java-based preview using an included template processed by + * the FreeMarker template engine. At the same time, CustomPreview is the fallback + * for AutomaticPreview if ro-crate-html-js is not installed. + *

+ * - StaticPreview: Allows to provide a static HTML page (including additional + * dependencies, e.g., CSS, JS) which is then shipped with the RO-Crate. + *

+ * When creating a new RO-Crate using the builder, the default setting is to use + * CustomPreview. This example shows you how to change it. + */ + @Test + void humanReadableContent() { + RoCrate crate = NEW_STARTER_CRATE() + .setPreview(new AutomaticPreview()) + .build(); + + assertNotNull(crate); + } + + /** + * A static preview means you'll just add your own HTML file to the crate. + * Therefore, the constructor is a bit more complicated. + */ + @Test + void staticPreview(@TempDir Path tempDir) throws IOException { + File mainPreviewHtml = tempDir.resolve("mainPreview.html").toFile(); + File additionalFilesDirectory = tempDir.resolve("additionalFiles").toFile(); + FileUtils.forceMkdir(additionalFilesDirectory); + FileUtils.touch(mainPreviewHtml); + + RoCrate crate = NEW_STARTER_CRATE() + .setPreview(new StaticPreview(mainPreviewHtml, additionalFilesDirectory)) + .build(); + + assertNotNull(crate); + } + + /** + * Crates can be validated. + * Right now, the only implemented way of validating a RO-crate is to use a + * [JSON-Schema](https://json-schema.org/) that the crate's metadata JSON file should + * match. JSON-Schema is an established standard and therefore a good choice for a + * crate profile. This example shows how to use it. + *

+ * Note: If you happen to implement your own validator anyway, please consider + * contributing your code! + */ + @Test + void validation() { + // Let's find a schema file in the resources folder. + URL schemaUrl = Objects.requireNonNull(this.getClass().getResource("/crates/validation/workflowschema.json")); + String schemaPath = schemaUrl.getPath(); + + // This crate for sure is not a workflow, so validation will fail. + RoCrate crate = NEW_STARTER_CRATE().build(); + + // And now do the validation. + Validator validator = new Validator(new JsonSchemaValidation(schemaPath)); + assertFalse(validator.validate(crate)); + } +} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/preview/PreviewTest.java b/src/test/java/edu/kit/datamanager/ro_crate/preview/PreviewTest.java index 2db987b3..788e0121 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/preview/PreviewTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/preview/PreviewTest.java @@ -1,13 +1,13 @@ package edu.kit.datamanager.ro_crate.preview; import net.lingala.zip4j.ZipFile; +import net.lingala.zip4j.io.outputstream.ZipOutputStream; import net.lingala.zip4j.model.ZipParameters; import org.apache.commons.io.FileUtils; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; -import java.io.IOException; -import java.io.InputStream; +import java.io.*; import java.nio.charset.Charset; import java.nio.file.Files; import java.nio.file.Path; @@ -75,15 +75,52 @@ void staticPreviewSaveToZip(@TempDir Path dir) throws IOException { assertTrue(FileUtils.contentEqualsIgnoreEOL(roDirFile.toFile(), fileInDir.toFile(), String.valueOf(Charset.defaultCharset()))); } + @Test + void staticPreviewSaveToZipStream(@TempDir Path dir) throws IOException { + var file1 = dir.resolve("file.html"); + FileUtils.writeStringToFile(file1.toFile(), "random html, does not need to be valid for this test", Charset.defaultCharset()); + + var file2 = dir.resolve("directory"); + var fileInDir = file2.resolve("fileInDir.html"); + FileUtils.writeStringToFile(fileInDir.toFile(), "dajkdlfjdsklafj alksfjdalk fjl", Charset.defaultCharset()); + StaticPreview preview = new StaticPreview(file1.toFile(), file2.toFile()); + + try (ZipOutputStream stream = new ZipOutputStream(new FileOutputStream(dir.resolve("destination.zip").toFile()))) { + preview.saveAllToStream( + null, // static preview does not need metadata + stream); + stream.flush(); + } + + try (ZipFile zf = new ZipFile(dir.resolve("destination.zip").toFile())) { + zf.extractAll(dir.resolve("extracted").toAbsolutePath().toString()); + } + + var e = dir.resolve("extracted"); + var roPreview = e.resolve("ro-crate-preview.html"); + var roDir = e.resolve("ro-crate-preview_files"); + var roDirFile = roDir.resolve("fileInDir.html"); + assertTrue(Files.isRegularFile(roPreview)); + assertTrue(Files.isDirectory(roDir)); + assertTrue(Files.isRegularFile(roDirFile)); + + assertTrue(FileUtils.contentEqualsIgnoreEOL(roPreview.toFile(), file1.toFile(), String.valueOf(Charset.defaultCharset()))); + assertFalse(FileUtils.contentEqualsIgnoreEOL(roPreview.toFile(), fileInDir.toFile(), String.valueOf(Charset.defaultCharset()))); + + assertTrue(FileUtils.contentEqualsIgnoreEOL(roDirFile.toFile(), fileInDir.toFile(), String.valueOf(Charset.defaultCharset()))); + } + @Test void testAutomaticPreviewAddToFolder(@TempDir Path dir) throws IOException { AutomaticPreview automaticPreview = new AutomaticPreview(); - InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json"); Path crate = dir.resolve("crate"); // this crate will not have a json file FileUtils.forceMkdir(crate.toFile()); - FileUtils.copyInputStreamToFile(crateJson, crate.resolve("ro-crate-metadata.json").toFile()); + try (InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json")) { + Assertions.assertNotNull(crateJson); + FileUtils.copyInputStreamToFile(crateJson, crate.resolve("ro-crate-metadata.json").toFile()); + } automaticPreview.saveAllToFolder(crate.toFile()); // there should be a html file generated @@ -93,14 +130,14 @@ void testAutomaticPreviewAddToFolder(@TempDir Path dir) throws IOException { @Test void testAutomaticPreviewZip(@TempDir Path dir) throws IOException { AutomaticPreview automaticPreview = new AutomaticPreview(); - InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json"); Path crate = dir.resolve("crate"); ZipParameters zipParameters = new ZipParameters(); zipParameters.setFileNameInZip("ro-crate-metadata.json"); ZipFile zipFile = new ZipFile(dir.resolve("test.zip").toFile()); - zipFile.addStream(crateJson, zipParameters); - crateJson.close(); + try (InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json")) { + zipFile.addStream(crateJson, zipParameters); + } automaticPreview.saveAllToZip(zipFile); @@ -116,55 +153,136 @@ void testAutomaticPreviewZip(@TempDir Path dir) throws IOException { assertTrue(Files.isRegularFile(crate.resolve("ro-crate-preview.html"))); } + @Test + void testAutomaticPreviewZipStream(@TempDir Path dir) throws IOException { + AutomaticPreview preview = new AutomaticPreview(); + String metadataPath = "/crates/other/idrc_project/ro-crate-metadata.json"; + Path crate = dir.resolve("crate"); + + File zipFile = dir.resolve("test.zip").toFile(); + try ( + ZipFile zip = new ZipFile(zipFile); + InputStream crateJson = PreviewTest.class.getResourceAsStream(metadataPath) + ) { + ZipParameters zipParameters = new ZipParameters(); + zipParameters.setFileNameInZip("ro-crate-metadata.json"); + zip.addStream(crateJson, zipParameters); + } + + String metadata; + try (InputStream metadataStream = PreviewTest.class.getResourceAsStream(metadataPath)) { + Assertions.assertNotNull(metadataStream); + metadata = new String(metadataStream.readAllBytes()); + } + + try (ZipOutputStream stream = new ZipOutputStream(new FileOutputStream(zipFile))) { + preview.saveAllToStream(metadata, stream); + stream.flush(); + } + + try { + // this should throw an exception but not stop the execution + ZipFile randomZipFile = new ZipFile(dir.resolve("dddd.zip").toFile()); + preview.saveAllToZip(randomZipFile); + Assertions.fail("Expected IOException when providing invalid ZIP file for preview."); + } catch (IOException ex) { + //ok + } + + try (ZipFile zipReader = new ZipFile(zipFile)) { + zipReader.extractAll(crate.toString()); + } + assertTrue(Files.isRegularFile(crate.resolve("ro-crate-preview.html"))); + } + @Test void testCustomPreviewAddToFolder(@TempDir Path dir) throws IOException { CustomPreview customPreview = new CustomPreview(); - - InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json"); Path crate = dir.resolve("crate"); - // this crate will not have a json file Path fakeCrate = dir.resolve("fakeCrate"); FileUtils.forceMkdir(crate.toFile()); - FileUtils.copyInputStreamToFile(crateJson, crate.resolve("ro-crate-metadata.json").toFile()); + + try (InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json")) { + Assertions.assertNotNull(crateJson); + FileUtils.copyInputStreamToFile(crateJson, crate.resolve("ro-crate-metadata.json").toFile()); + } customPreview.saveAllToFolder(crate.toFile()); - try { - // this should trow an exception but not stop the execution + try { + // this should throw an exception but not stop the execution customPreview.saveAllToFolder(fakeCrate.toFile()); Assertions.fail("Expected IOException when providing invalid ZIP file for preview."); } catch (IOException ex) { //ok } - // there should be a html file generated assertTrue(Files.isRegularFile(crate.resolve("ro-crate-preview.html"))); } @Test void testCustomPreviewZip(@TempDir Path tmp) throws IOException { CustomPreview customPreview = new CustomPreview(); - InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json"); Path crate = tmp.resolve("crate"); ZipParameters zipParameters = new ZipParameters(); zipParameters.setFileNameInZip("ro-crate-metadata.json"); - ZipFile zipFile = new ZipFile(tmp.resolve("test.zip").toFile()); - zipFile.addStream(crateJson, zipParameters); - crateJson.close(); + try (ZipFile zipFile = new ZipFile(tmp.resolve("test.zip").toFile()); + InputStream crateJson = PreviewTest.class.getResourceAsStream("/crates/other/idrc_project/ro-crate-metadata.json")) { + zipFile.addStream(crateJson, zipParameters); + customPreview.saveAllToZip(zipFile); + + try { + // this should throw an exception but not stop the execution + ZipFile randomZipFile = new ZipFile(tmp.resolve("dddd.zip").toFile()); + customPreview.saveAllToZip(randomZipFile); + Assertions.fail("Expected IOException when providing invalid input to preview."); + } catch (IOException ex) { + //ok + } + zipFile.extractAll(crate.toString()); + } + assertTrue(Files.isRegularFile(crate.resolve("ro-crate-preview.html"))); + } + + @Test + void testCustomPreviewZipStream(@TempDir Path tmp) throws IOException { + CustomPreview preview = new CustomPreview(); + String metadataPath = "/crates/other/idrc_project/ro-crate-metadata.json"; + Path crate = tmp.resolve("crate"); + File zipFile = tmp.resolve("test.zip").toFile(); + + try (ZipFile zip = new ZipFile(zipFile); + InputStream crateJson = PreviewTest.class.getResourceAsStream(metadataPath)) { + ZipParameters zipParameters = new ZipParameters(); + zipParameters.setFileNameInZip("ro-crate-metadata.json"); + zip.addStream(crateJson, zipParameters); + } + + String metadata; + try (InputStream metadataStream = PreviewTest.class.getResourceAsStream(metadataPath)) { + Assertions.assertNotNull(metadataStream); + metadata = new String(metadataStream.readAllBytes()); + } - customPreview.saveAllToZip(zipFile); + try (ZipOutputStream stream = new ZipOutputStream(new FileOutputStream(zipFile))) { + preview.saveAllToStream(metadata, stream); + stream.flush(); + } try { - // this should trow an exception but not stop the execution + // this should throw an exception but not stop the execution ZipFile randomZipFile = new ZipFile(tmp.resolve("dddd.zip").toFile()); - customPreview.saveAllToZip(randomZipFile); + preview.saveAllToZip(randomZipFile); Assertions.fail("Expected IOException when providing invalid input to preview."); } catch (IOException ex) { //ok } - zipFile.extractAll(crate.toString()); + + try (ZipFile zipReader = new ZipFile(zipFile)) { + zipReader.extractAll(crate.toString()); + } assertTrue(Files.isRegularFile(crate.resolve("ro-crate-preview.html"))); } - } + diff --git a/src/test/java/edu/kit/datamanager/ro_crate/reader/CrateReaderTest.java b/src/test/java/edu/kit/datamanager/ro_crate/reader/CommonReaderTest.java similarity index 78% rename from src/test/java/edu/kit/datamanager/ro_crate/reader/CrateReaderTest.java rename to src/test/java/edu/kit/datamanager/ro_crate/reader/CommonReaderTest.java index bcf8948d..1e162a1c 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/reader/CrateReaderTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/reader/CommonReaderTest.java @@ -27,9 +27,13 @@ * This parameter is only required to satisfy the generic reader strategy. * @param the type of the reader strategy */ -abstract class CrateReaderTest> { - - protected static RoCrate.RoCrateBuilder newBaseCrate() { +public interface CommonReaderTest< + SOURCE_T, + READER_STRATEGY extends GenericReaderStrategy + > + extends TestableReaderStrategy +{ + static RoCrate.RoCrateBuilder newBaseCrate() { return new RoCrate.RoCrateBuilder( "minimal", "minimal RO_crate", @@ -38,7 +42,7 @@ protected static RoCrate.RoCrateBuilder newBaseCrate() { ); } - protected static FileEntity newDataEntity(Path filePath) throws IllegalArgumentException { + static FileEntity newDataEntity(Path filePath) throws IllegalArgumentException { return new FileEntity.FileEntityBuilder() .setLocationWithExceptions(filePath) .setId(filePath.toFile().getName()) @@ -48,44 +52,8 @@ protected static FileEntity newDataEntity(Path filePath) throws IllegalArgumentE .build(); } - /** - * Saves the crate with the writer fitting to the reader of {@link #readCrate(Path)}. - * - * @param crate the crate to save - * @param target the target path to the save location - * @throws IOException if an error occurs while saving the crate - */ - abstract protected void saveCrate(Crate crate, Path target) throws IOException; - - /** - * Reads the crate with the reader fitting to the writer of {@link #saveCrate(Crate, Path)}. - * @param source the source path to the crate - * @return the read crate - * @throws IOException if an error occurs while reading the crate - */ - abstract protected Crate readCrate(Path source) throws IOException; - - /** - * Creates a new reader strategy with a non-default temporary directory (if supported, default otherwise). - * - * @param tmpDirectory the temporary directory to use - * @param useUuidSubfolder whether to create a UUID subfolder under the temporary directory - * @return a new reader strategy - */ - abstract protected READER_STRATEGY newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder); - - /** - * Reads the crate using the provided reader strategy. - * - * @param strategy the reader strategy to use - * @param source the source path to the crate - * @return the read crate - * @throws IOException if an error occurs while reading the crate - */ - abstract protected Crate readCrate(READER_STRATEGY strategy, Path source) throws IOException; - @Test - void testReadingBasicCrate(@TempDir Path temp) throws IOException { + default void testReadingBasicCrate(@TempDir Path temp) throws IOException { RoCrate roCrate = newBaseCrate().build(); Path zipPath = temp.resolve("result.zip"); @@ -95,7 +63,7 @@ void testReadingBasicCrate(@TempDir Path temp) throws IOException { } @Test - void testWithFile(@TempDir Path temp) throws IOException { + default void testWithFile(@TempDir Path temp) throws IOException { Path csvPath = temp.resolve("survey-responses-2019.csv"); FileUtils.touch(csvPath.toFile()); FileUtils.writeStringToFile(csvPath.toFile(), "Dummy content", Charset.defaultCharset()); @@ -113,7 +81,7 @@ void testWithFile(@TempDir Path temp) throws IOException { } @Test - void testWithFileUrlEncoded(@TempDir Path temp) throws IOException { + default void testWithFileUrlEncoded(@TempDir Path temp) throws IOException { // This URL will be encoded because of whitespaces Path csvPath = temp.resolve("survey responses 2019.csv"); FileUtils.touch(csvPath.toFile()); @@ -140,7 +108,7 @@ void testWithFileUrlEncoded(@TempDir Path temp) throws IOException { } @Test - void TestWithFileWithLocation(@TempDir Path temp) throws IOException { + default void TestWithFileWithLocation(@TempDir Path temp) throws IOException { Path csvPath = temp.resolve("survey-responses-2019.csv"); FileUtils.writeStringToFile(csvPath.toFile(), "Dummy content", Charset.defaultCharset()); RoCrate rawCrate = newBaseCrate() @@ -168,7 +136,7 @@ void TestWithFileWithLocation(@TempDir Path temp) throws IOException { } @Test - void TestWithFileWithLocationAddEntity(@TempDir Path temp) throws IOException { + default void TestWithFileWithLocationAddEntity(@TempDir Path temp) throws IOException { Path csvPath = temp.resolve("file.csv"); FileUtils.writeStringToFile(csvPath.toFile(), "fakecsv.1", Charset.defaultCharset()); RoCrate rawCrate = newBaseCrate() @@ -206,7 +174,7 @@ void TestWithFileWithLocationAddEntity(@TempDir Path temp) throws IOException { } @Test - void testReadingBasicCrateWithCustomPath(@TempDir Path temp) throws IOException { + default void testReadingBasicCrateWithCustomPath(@TempDir Path temp) throws IOException { RoCrate rawCrate = newBaseCrate().build(); // Write to zip file diff --git a/src/test/java/edu/kit/datamanager/ro_crate/reader/ElnFileFormatTest.java b/src/test/java/edu/kit/datamanager/ro_crate/reader/ElnFileFormatTest.java new file mode 100644 index 00000000..dbbb4e05 --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/reader/ElnFileFormatTest.java @@ -0,0 +1,73 @@ +package edu.kit.datamanager.ro_crate.reader; + +import edu.kit.datamanager.ro_crate.Crate; +import org.apache.commons.io.FileUtils; +import org.junit.jupiter.api.io.TempDir; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import java.io.IOException; +import java.net.URI; +import java.net.URL; +import java.nio.file.Path; + +import static org.junit.jupiter.api.Assertions.*; + +public interface ElnFileFormatTest< + SOURCE_T, + READER_STRATEGY extends GenericReaderStrategy + > + extends TestableReaderStrategy +{ + /** + * Some readers may not be able to read a subset of eln files, + * e.g. because a zip file may not be readable in streaming mode. + *

+ * An implementation test may use this methode to provide a subset of the + * test cases where an IOException is expected. + * + * @param input the input to test for presence in the blacklist + * @return true if the input is in the blacklist, false otherwise + */ + default boolean isInBlacklist(String input) { + return false; + } + + /** + * ELN Crates are zip files not fully compatible with the Ro-Crate standard + * in the sense that they must contain a single subfolder in the zip file + * which then contain a crate as specified by the Ro-Crate standard. + *

+ * Here we test if we can read them using out ZipReader. + * + * @see + */ + @ParameterizedTest + @ValueSource(strings = { + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/AI4Green/Export%20workbook-2024-08-27-export.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/OpenSemanticLab/MinimalExample.osl.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/PASTA/PASTA.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/RSpace/RSpace-2023-12-08-14-44-xml-SELECTION-c0bEtpHcnNe-HA.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/SampleDB/sampledb_export.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/elabftw/export.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/kadi4mat/records-example.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/kadi4mat/collections-example.eln" + }) + default void testReadElnCrates(String urlStr, @TempDir Path tmp) throws IOException { + // Download the ELN file + URL url = URI.create(urlStr).toURL(); + Path elnFile = tmp.resolve("downloaded.eln"); + FileUtils.copyURLToFile(url, elnFile.toFile(), 20000, 20000); + assertTrue(elnFile.toFile().exists()); + + if (!isInBlacklist(urlStr)) { + // Read the crate from the downloaded file + Crate read = this.readCrate(elnFile); + assertNotNull(read); + assertFalse(read.getAllDataEntities().isEmpty()); + } else { + // If the file is in the blacklist, we expect an IOException + assertThrows(IOException.class, () -> this.readCrate(elnFile)); + } + } +} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/reader/FolderReaderTest.java b/src/test/java/edu/kit/datamanager/ro_crate/reader/FolderReaderTest.java index 21850edd..83bf4f78 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/reader/FolderReaderTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/reader/FolderReaderTest.java @@ -17,29 +17,29 @@ * @author Nikola Tzotchev on 9.2.2022 г. * @version 1 */ -class FolderReaderTest extends CrateReaderTest { - +class FolderReaderTest implements CommonReaderTest +{ @Override - protected void saveCrate(Crate crate, Path target) { + public void saveCrate(Crate crate, Path target) throws IOException { Writers.newFolderWriter().save(crate, target.toAbsolutePath().toString()); assertTrue(target.toFile().isDirectory()); } @Override - protected Crate readCrate(Path source) throws IOException { + public Crate readCrate(Path source) throws IOException { return Readers.newFolderReader().readCrate(source.toAbsolutePath().toString()); } @Override - protected FolderStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder) { + public ReadFolderStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder) { // This strategy does not support a non-default temporary directory // and will always use the default one. // It also has no state we could make assertions on. - return new FolderStrategy(); + return new ReadFolderStrategy(); } @Override - protected Crate readCrate(FolderStrategy strategy, Path source) throws IOException { + public Crate readCrate(ReadFolderStrategy strategy, Path source) throws IOException { return new CrateReader<>(strategy) .readCrate(source.toAbsolutePath().toString()); } diff --git a/src/test/java/edu/kit/datamanager/ro_crate/reader/RoCrateReaderSpec12Test.java b/src/test/java/edu/kit/datamanager/ro_crate/reader/RoCrateReaderSpec12Test.java index 7b1df7df..68094231 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/reader/RoCrateReaderSpec12Test.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/reader/RoCrateReaderSpec12Test.java @@ -3,6 +3,7 @@ import static org.junit.jupiter.api.Assertions.assertEquals; import static org.junit.jupiter.api.Assertions.assertTrue; +import java.io.IOException; import java.util.stream.StreamSupport; import org.junit.jupiter.api.Test; @@ -27,7 +28,7 @@ public class RoCrateReaderSpec12Test { * https://www.researchobject.org/ro-crate/1.2-DRAFT/profiles.html#declaring-conformance-of-an-ro-crate-profile */ @Test - void testReadingCrateWithConformsToArray() { + void testReadingCrateWithConformsToArray() throws IOException { String path = this.getClass().getResource("/crates/spec-1.2-DRAFT/minimal-with-conformsTo-Array").getPath(); Crate crate = Readers.newFolderReader().readCrate(path); JsonNode conformsTo = crate.getJsonDescriptor().getProperty("conformsTo"); diff --git a/src/test/java/edu/kit/datamanager/ro_crate/reader/TestableReaderStrategy.java b/src/test/java/edu/kit/datamanager/ro_crate/reader/TestableReaderStrategy.java new file mode 100644 index 00000000..c8d4b72a --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/reader/TestableReaderStrategy.java @@ -0,0 +1,50 @@ +package edu.kit.datamanager.ro_crate.reader; + +import edu.kit.datamanager.ro_crate.Crate; + +import java.io.IOException; +import java.nio.file.Path; + +/** + * Base Interface for methods required to test all reader strategies. + * + * @param the source type the strategy reads from. + * @param the type of the reader strategy. + */ +interface TestableReaderStrategy> { + /** + * Saves the crate with the writer fitting to the reader of {@link #readCrate(Path)}. + * + * @param crate the crate to save + * @param target the target path to the save location + * @throws IOException if an error occurs while saving the crate + */ + void saveCrate(Crate crate, Path target) throws IOException; + + /** + * Reads the crate with the reader fitting to the writer of {@link #saveCrate(Crate, Path)}. + * @param source the source path to the crate + * @return the read crate + * @throws IOException if an error occurs while reading the crate + */ + Crate readCrate(Path source) throws IOException; + + /** + * Creates a new reader strategy with a non-default temporary directory (if supported, default otherwise). + * + * @param tmpDirectory the temporary directory to use + * @param useUuidSubfolder whether to create a UUID subfolder under the temporary directory + * @return a new reader strategy + */ + READER_STRATEGY newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder); + + /** + * Reads the crate using the provided reader strategy. + * + * @param strategy the reader strategy to use + * @param source the source path to the crate + * @return the read crate + * @throws IOException if an error occurs while reading the crate + */ + Crate readCrate(READER_STRATEGY strategy, Path source) throws IOException; +} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipReaderTest.java b/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipReaderTest.java index 3611c834..35b168e1 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipReaderTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipReaderTest.java @@ -8,22 +8,24 @@ import static org.junit.jupiter.api.Assertions.*; -class ZipReaderTest extends CrateReaderTest { - +class ZipReaderTest implements + CommonReaderTest, + ElnFileFormatTest +{ @Override - protected void saveCrate(Crate crate, Path target) { + public void saveCrate(Crate crate, Path target) throws IOException { Writers.newZipPathWriter().save(crate, target.toAbsolutePath().toString()); assertTrue(target.toFile().isFile()); } @Override - protected Crate readCrate(Path source) throws IOException { + public Crate readCrate(Path source) throws IOException { return Readers.newZipPathReader().readCrate(source.toAbsolutePath().toString()); } @Override - protected ZipStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder) { - ZipStrategy strategy = new ZipStrategy(tmpDirectory, useUuidSubfolder); + public ReadZipStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder) { + ReadZipStrategy strategy = new ReadZipStrategy(tmpDirectory, useUuidSubfolder); assertFalse(strategy.isExtracted()); if (useUuidSubfolder) { assertEquals(strategy.getTemporaryFolder().getFileName().toString(), strategy.getID()); @@ -35,7 +37,7 @@ protected ZipStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean useUui } @Override - protected Crate readCrate(ZipStrategy strategy, Path source) throws IOException { + public Crate readCrate(ReadZipStrategy strategy, Path source) throws IOException { Crate importedCrate = new CrateReader<>(strategy) .readCrate(source.toAbsolutePath().toString()); assertTrue(strategy.isExtracted()); diff --git a/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipStreamReaderTest.java b/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipStreamReaderTest.java index 33436c6c..11b82d12 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipStreamReaderTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/reader/ZipStreamReaderTest.java @@ -5,25 +5,51 @@ import java.io.*; import java.nio.file.Path; +import java.util.Set; import static org.junit.jupiter.api.Assertions.*; -class ZipStreamReaderTest extends CrateReaderTest { +class ZipStreamReaderTest implements + CommonReaderTest, + ElnFileFormatTest +{ + /** + * At the point of writing this test, + * these files are in a zip format which cannot be read in streaming mode + */ @Override - protected void saveCrate(Crate crate, Path target) throws IOException { - Writers.newZipStreamWriter().save(crate, new FileOutputStream(target.toFile())); - assertTrue(target.toFile().isFile()); + public boolean isInBlacklist(String input) { + return Set.of( + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/kadi4mat/records-example.eln", + "https://github.com/TheELNConsortium/TheELNFileFormat/raw/refs/heads/master/examples/kadi4mat/collections-example.eln" + ) + .contains(input); } @Override - protected Crate readCrate(Path source) throws IOException { - return Readers.newZipStreamReader().readCrate(new FileInputStream(source.toFile())); + public void saveCrate(Crate crate, Path target) throws IOException { + final File target_file = target.toFile(); + try ( + FileOutputStream fos = new FileOutputStream(target_file) + ) { + Writers.newZipStreamWriter().save(crate, fos); + } + assertTrue(target_file.isFile()); + } + + @Override + public Crate readCrate(Path source) throws IOException { + try ( + FileInputStream fis = new FileInputStream(source.toFile()) + ) { + return Readers.newZipStreamReader().readCrate(fis); + } } @Override - protected ZipStreamStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder) { - ZipStreamStrategy strategy = new ZipStreamStrategy(tmpDirectory, useUuidSubfolder); + public ReadZipStreamStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean useUuidSubfolder) { + ReadZipStreamStrategy strategy = new ReadZipStreamStrategy(tmpDirectory, useUuidSubfolder); assertFalse(strategy.isExtracted()); if (useUuidSubfolder) { assertEquals(strategy.getTemporaryFolder().getFileName().toString(), strategy.getID()); @@ -35,10 +61,14 @@ protected ZipStreamStrategy newReaderStrategyWithTmp(Path tmpDirectory, boolean } @Override - protected Crate readCrate(ZipStreamStrategy strategy, Path source) throws IOException { - Crate importedCrate = new CrateReader<>(strategy) - .readCrate(new FileInputStream(source.toFile())); - assertTrue(strategy.isExtracted()); - return importedCrate; + public Crate readCrate(ReadZipStreamStrategy strategy, Path source) throws IOException { + try ( + FileInputStream fis = new FileInputStream(source.toFile()) + ) { + Crate importedCrate = new CrateReader<>(strategy).readCrate(fis); + assertNotNull(importedCrate); + assertTrue(strategy.isExtracted()); + return importedCrate; + } } } diff --git a/src/test/java/edu/kit/datamanager/ro_crate/util/FileSystemUtilTest.java b/src/test/java/edu/kit/datamanager/ro_crate/util/FileSystemUtilTest.java new file mode 100644 index 00000000..fac64527 --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/util/FileSystemUtilTest.java @@ -0,0 +1,31 @@ +package edu.kit.datamanager.ro_crate.util; + +import org.junit.jupiter.api.Test; +import org.junit.jupiter.params.ParameterizedTest; +import org.junit.jupiter.params.provider.ValueSource; + +import static org.junit.jupiter.api.Assertions.*; + +class FileSystemUtilTest { + + @ValueSource(strings = { + "test", + "test/", + "test/test", + "test/test/", + "test/test/test", + "test/test/test/" + }) + @ParameterizedTest + void ensureTrailingSlash(String value) { + String result = FileSystemUtil.ensureTrailingSlash(value); + assertTrue(result.endsWith("/"), "The result should end with a trailing slash."); + } + + @SuppressWarnings("ConstantValue") + @Test + void ensureTrailingSlashNull() { + String result = FileSystemUtil.ensureTrailingSlash(null); + assertNull(result, "The result should be null."); + } +} \ No newline at end of file diff --git a/src/test/java/edu/kit/datamanager/ro_crate/writer/CrateWriterTest.java b/src/test/java/edu/kit/datamanager/ro_crate/writer/CommonWriterTest.java similarity index 50% rename from src/test/java/edu/kit/datamanager/ro_crate/writer/CrateWriterTest.java rename to src/test/java/edu/kit/datamanager/ro_crate/writer/CommonWriterTest.java index 429fc450..0eae96c2 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/writer/CrateWriterTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/writer/CommonWriterTest.java @@ -1,20 +1,14 @@ package edu.kit.datamanager.ro_crate.writer; -import edu.kit.datamanager.ro_crate.Crate; import edu.kit.datamanager.ro_crate.HelpFunctions; import edu.kit.datamanager.ro_crate.RoCrate; import edu.kit.datamanager.ro_crate.entities.data.DataSetEntity; -import edu.kit.datamanager.ro_crate.entities.data.FileEntity; -import edu.kit.datamanager.ro_crate.preview.AutomaticPreview; -import edu.kit.datamanager.ro_crate.preview.PreviewGenerator; -import net.lingala.zip4j.ZipFile; + import org.apache.commons.io.FileUtils; -import org.junit.jupiter.api.Assertions; import org.junit.jupiter.api.Test; import org.junit.jupiter.api.io.TempDir; import java.io.IOException; -import java.io.InputStream; import java.nio.charset.Charset; import java.nio.file.Files; import java.nio.file.Path; @@ -22,16 +16,7 @@ import static org.junit.jupiter.api.Assertions.assertFalse; import static org.junit.jupiter.api.Assertions.assertTrue; -abstract class CrateWriterTest { - - /** - * Saves the crate with the writer fitting to this test class. - * - * @param crate the crate to save - * @param target the target path to the save location - * @throws IOException if an error occurs while saving the crate - */ - abstract protected void saveCrate(Crate crate, Path target) throws IOException; +interface CommonWriterTest extends TestableWriterStrategy { /** * Test where the writer needs to rename files or folders in order to make a valid crate. @@ -41,11 +26,11 @@ abstract class CrateWriterTest { * @throws IOException if an error occurs while writing the crate */ @Test - void testFilesBeingAdjusted(@TempDir Path tempDir) throws IOException { + default void testFilesBeingAdjusted(@TempDir Path tempDir) throws IOException { Path correctCrate = tempDir.resolve("compare_with_me"); Path pathToFile = correctCrate.resolve("you-will-need-to-rename-this-file.ai"); Path pathToDir = correctCrate.resolve("you-will-need-to-rename-this-dir"); - this.createManualCrateStructure(correctCrate, pathToFile, pathToDir); + createManualCrateStructure(correctCrate, pathToFile, pathToDir); Path writtenCrate = tempDir.resolve("written-crate"); Path extractionPath = tempDir.resolve("checkMe"); @@ -60,11 +45,11 @@ void testFilesBeingAdjusted(@TempDir Path tempDir) throws IOException { ) .build(); this.saveCrate(builtCrate, writtenCrate); - this.ensureCrateIsExtractedIn(writtenCrate, extractionPath); + ensureCrateIsExtractedIn(writtenCrate, extractionPath); } - printFileTree(correctCrate); - printFileTree(extractionPath); + HelpFunctions.printFileTree(correctCrate); + HelpFunctions.printFileTree(extractionPath); // The actual file name should **not** appear in the crate String fileName = pathToFile.getFileName().toString(); @@ -111,7 +96,7 @@ void testFilesBeingAdjusted(@TempDir Path tempDir) throws IOException { * @throws IOException if an error occurs while writing the crate */ @Test - void testWritingMakesCopy(@TempDir Path tempDir) throws IOException { + default void testWritingMakesCopy(@TempDir Path tempDir) throws IOException { // We need a correct directory to compare with. // It is built manually to ensure we meet our expectations. // Reader-writer-consistency is tested at {@link CrateReaderTest} @@ -119,7 +104,7 @@ void testWritingMakesCopy(@TempDir Path tempDir) throws IOException { Path pathToFile = correctCrate.resolve("cp7glop.ai"); Path pathToDir = correctCrate.resolve("lots_of_little_files"); - this.createManualCrateStructure(correctCrate, pathToFile, pathToDir); + createManualCrateStructure(correctCrate, pathToFile, pathToDir); // Now use the builder to build the same crate independently. // The files will be reused (we need a place to take a copy from) @@ -130,9 +115,9 @@ void testWritingMakesCopy(@TempDir Path tempDir) throws IOException { // extract the zip file to a temporary directory Path extractionPath = tempDir.resolve("extracted_for_testing"); - this.ensureCrateIsExtractedIn(pathToZip, extractionPath); - printFileTree(correctCrate); - printFileTree(extractionPath); + ensureCrateIsExtractedIn(pathToZip, extractionPath); + HelpFunctions.printFileTree(correctCrate); + HelpFunctions.printFileTree(extractionPath); // compare the extracted directory with the correct one assertTrue(HelpFunctions.compareTwoDir( @@ -151,12 +136,12 @@ void testWritingMakesCopy(@TempDir Path tempDir) throws IOException { * @throws IOException if an error occurs while writing the crate */ @Test - void testWritingOnlyConsidersAddedFiles(@TempDir Path tempDir) throws IOException { + default void testWritingOnlyConsidersAddedFiles(@TempDir Path tempDir) throws IOException { Path correctCrate = tempDir.resolve("compare_with_me"); Path pathToFile = correctCrate.resolve("cp7glop.ai"); Path pathToDir = correctCrate.resolve("lots_of_little_files"); - this.createManualCrateStructure(correctCrate, pathToFile, pathToDir); + createManualCrateStructure(correctCrate, pathToFile, pathToDir); { // This file is not part of the crate, and should therefore not be present Path falseFile = correctCrate.resolve("new"); @@ -176,8 +161,8 @@ void testWritingOnlyConsidersAddedFiles(@TempDir Path tempDir) throws IOExceptio // extract and compare Path extractionPath = tempDir.resolve("extracted_for_testing"); ensureCrateIsExtractedIn(pathToZip, extractionPath); - printFileTree(correctCrate); - printFileTree(extractionPath); + HelpFunctions.printFileTree(correctCrate); + HelpFunctions.printFileTree(extractionPath); assertFalse(HelpFunctions.compareTwoDir( correctCrate.toFile(), @@ -187,116 +172,4 @@ void testWritingOnlyConsidersAddedFiles(@TempDir Path tempDir) throws IOExceptio roCrate, "/json/crate/fileAndDir.json"); } - - /** - * Prints the file tree of the given directory for debugging and understanding - * a test more quickly. - * - * @param directoryToPrint the directory to print - * @throws IOException if an error occurs while printing the file tree - */ - @SuppressWarnings("resource") - protected static void printFileTree(Path directoryToPrint) throws IOException { - // Print all files recursively in a tree structure for debugging - System.out.printf("Files in %s:%n", directoryToPrint.getFileName().toString()); - Files.walk(directoryToPrint) - .forEach(path -> { - if (!path.toAbsolutePath().equals(directoryToPrint.toAbsolutePath())) { - int depth = path.relativize(directoryToPrint).getNameCount(); - String prefix = " ".repeat(depth); - System.out.printf("%s%s%s%n", prefix, "└── ", path.getFileName()); - } - }); - } - - /** - * Ensures the crate is in extracted form in the given path. - * - * @param pathToCrate the path to the crate, may not be a folder yet - * @param expectedPath the path where the crate should be in extracted form - * @throws IOException if an error occurs while extracting the crate - */ - protected void ensureCrateIsExtractedIn(Path pathToCrate, Path expectedPath) throws IOException { - try (ZipFile zf = new ZipFile(pathToCrate.toFile())) { - zf.extractAll(expectedPath.toFile().getAbsolutePath()); - } - } - - /** - * Creates a crate structure manually. - * - * @param correctCrate the path to the crate - * @param pathToFile the path to the file - * @param pathToDir the path to the directory - * @throws IOException if an error occurs while creating the crate structure - */ - protected void createManualCrateStructure(Path correctCrate, Path pathToFile, Path pathToDir) throws IOException { - FileUtils.forceMkdir(correctCrate.toFile()); - InputStream fileJson = ZipStreamStrategyTest.class - .getResourceAsStream("/json/crate/fileAndDir.json"); - Assertions.assertNotNull(fileJson); - // fill the directory with expected files and dirs - // starting with the .json of our crate - Path json = correctCrate.resolve("ro-crate-metadata.json"); - FileUtils.copyInputStreamToFile(fileJson, json.toFile()); - // create preview - PreviewGenerator.generatePreview(correctCrate.toFile().getAbsolutePath()); - // create the files and directories - FileUtils.writeStringToFile(pathToFile.toFile(), "content of Local File", Charset.defaultCharset()); - // creates the directory and a subdirectory - Path subdir = pathToDir.resolve("subdir"); - FileUtils.forceMkdir(subdir.toFile()); - FileUtils.writeStringToFile( - subdir.resolve("subsubfirst.txt").toFile(), - "content of subsub file in subsubdir", - Charset.defaultCharset()); - FileUtils.writeStringToFile( - pathToDir.resolve("first.txt").toFile(), - "content of first file in dir", - Charset.defaultCharset()); - FileUtils.writeStringToFile( - pathToDir.resolve("second.txt").toFile(), - "content of second file in dir", - Charset.defaultCharset()); - FileUtils.writeStringToFile( - pathToDir.resolve("third.txt").toFile(), - "content of third file in dir", - Charset.defaultCharset()); - } - - /** - * Creates a crate resembling the one we manually create in these tests. - * - * @param pathToFile the file to add - * @param pathToSubdir the directory to add - * @return the crate builder - */ - protected RoCrate.RoCrateBuilder getCrateWithFileAndDir(Path pathToFile, Path pathToSubdir) { - return new RoCrate.RoCrateBuilder( - "Example RO-Crate", - "The RO-Crate Root Data Entity", - "2024", - "https://creativecommons.org/licenses/by-nc-sa/3.0/au/" - ) - .addDataEntity( - new FileEntity.FileEntityBuilder() - .addProperty("name", "Diagram showing trend to increase") - .addProperty("contentSize", "383766") - .addProperty("description", "Illustrator file for Glop Pot") - .setEncodingFormat("application/pdf") - .setLocationWithExceptions(pathToFile) - .setId("cp7glop.ai") - .build() - ) - .addDataEntity( - new DataSetEntity.DataSetBuilder() - .addProperty("name", "Too many files") - .addProperty("description", - "This directory contains many small files, that we're not going to describe in detail.") - .setLocationWithExceptions(pathToSubdir) - .setId("lots_of_little_files/") - .build() - ) - .setPreview(new AutomaticPreview()); - } } diff --git a/src/test/java/edu/kit/datamanager/ro_crate/writer/ElnFileWriterTest.java b/src/test/java/edu/kit/datamanager/ro_crate/writer/ElnFileWriterTest.java new file mode 100644 index 00000000..971822e1 --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/writer/ElnFileWriterTest.java @@ -0,0 +1,92 @@ +package edu.kit.datamanager.ro_crate.writer; + +import edu.kit.datamanager.ro_crate.Crate; +import edu.kit.datamanager.ro_crate.HelpFunctions; +import edu.kit.datamanager.ro_crate.RoCrate; + +import edu.kit.datamanager.ro_crate.reader.CommonReaderTest; +import org.junit.jupiter.api.Test; +import org.junit.jupiter.api.io.TempDir; + +import java.io.IOException; +import java.nio.file.Path; + +import static org.junit.jupiter.api.Assertions.*; + +public interface ElnFileWriterTest extends TestableWriterStrategy { + + /** + * Write in ELN format style, meaning with a subfolder in the zip file. + * Must use {@link ElnFormatWriter#usingElnStyle()}. + * + * @param crate the crate to write + * @param target the target path to the save location + * @throws IOException if an error occurs + */ + void saveCrateElnStyle(Crate crate, Path target) throws IOException; + + /** + * Same as {@link #saveCrateElnStyle(Crate, Path)} but with the alias + * {@link ElnFormatWriter#withRootSubdirectory()}. + * @param crate the crate to write + * @param target the target path to the save location + */ + void saveCrateSubdirectoryStyle(RoCrate crate, Path target) throws IOException; + + @Test + default void testMakesElnStyleCrate(@TempDir Path tempDir) throws IOException { + // We need a correct directory to compare with. + // It is built manually to ensure we meet our expectations. + // Reader-writer-consistency is tested at {@link CrateReaderTest} + + // We compare the ELN style like this: + // tempDir + // └── compare_with_me + // └── crate-subfolder + // ├── ... + // └── extracted_for_testing + // └── crate-subfolder + // ├── ... + String crateName = "crate-subfolder"; + Path correctCrate = tempDir + .resolve("compare_with_me") + .resolve(crateName); + Path pathToFile = correctCrate.resolve("cp7glop.ai"); + Path pathToDir = correctCrate.resolve("lots_of_little_files"); + + createManualCrateStructure(correctCrate, pathToFile, pathToDir); + + // Now use the builder to build the same crate independently. + // The files will be reused (we need a place to take a copy from) + RoCrate builtCrate = getCrateWithFileAndDir(pathToFile, pathToDir).build(); + + Path pathToZip = tempDir.resolve("%s.eln".formatted(crateName)); + this.saveCrateElnStyle(builtCrate, pathToZip); + + // extract the zip file to a temporary directory + Path extractionPath = tempDir.resolve("extracted_for_testing"); + ensureCrateIsExtractedIn(pathToZip, extractionPath); + HelpFunctions.printFileTree(correctCrate); + HelpFunctions.printFileTree(extractionPath); + + // compare the extracted directory with the correct one + assertTrue(HelpFunctions.compareTwoDir( + correctCrate.toFile(), + extractionPath.toFile())); + HelpFunctions.compareCrateJsonToFileInResources( + builtCrate, + "/json/crate/fileAndDir.json"); + } + + @Test + default void testAlias(@TempDir Path tmpDir) throws IOException { + Path zip = tmpDir.resolve("test.eln").toAbsolutePath(); + RoCrate crate = CommonReaderTest.newBaseCrate().build(); + + this.saveCrateSubdirectoryStyle(crate, zip); + + assertTrue(zip.toFile().exists(), "The zip file should exist"); + Path extractedPath = tmpDir.resolve("extracted"); + ensureCrateIsExtractedIn(zip, extractedPath); + } +} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/writer/FolderWriterTest.java b/src/test/java/edu/kit/datamanager/ro_crate/writer/FolderWriterTest.java index 7a469465..0c187029 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/writer/FolderWriterTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/writer/FolderWriterTest.java @@ -11,16 +11,16 @@ * @author Nikola Tzotchev on 9.2.2022 г. * @version 1 */ -class FolderWriterTest extends CrateWriterTest { +class FolderWriterTest implements CommonWriterTest { @Override - protected void saveCrate(Crate crate, Path target) throws IOException { + public void saveCrate(Crate crate, Path target) throws IOException { Writers.newFolderWriter() .save(crate, target.toAbsolutePath().toString()); } @Override - protected void ensureCrateIsExtractedIn(Path pathToCrate, Path expectedPath) throws IOException { + public void ensureCrateIsExtractedIn(Path pathToCrate, Path expectedPath) throws IOException { FileUtils.copyDirectory(pathToCrate.toFile(), expectedPath.toFile()); } } diff --git a/src/test/java/edu/kit/datamanager/ro_crate/writer/TestableWriterStrategy.java b/src/test/java/edu/kit/datamanager/ro_crate/writer/TestableWriterStrategy.java new file mode 100644 index 00000000..079fb766 --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/writer/TestableWriterStrategy.java @@ -0,0 +1,121 @@ +package edu.kit.datamanager.ro_crate.writer; + +import edu.kit.datamanager.ro_crate.Crate; +import edu.kit.datamanager.ro_crate.RoCrate; +import edu.kit.datamanager.ro_crate.entities.data.DataSetEntity; +import edu.kit.datamanager.ro_crate.entities.data.FileEntity; +import edu.kit.datamanager.ro_crate.preview.AutomaticPreview; +import edu.kit.datamanager.ro_crate.preview.PreviewGenerator; +import net.lingala.zip4j.ZipFile; +import org.apache.commons.io.FileUtils; +import org.junit.jupiter.api.Assertions; + +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.Charset; +import java.nio.file.Path; + +/** + * Base Interface for methods required to test all writer strategies. + */ +interface TestableWriterStrategy { + /** + * Saves the crate with the writer fitting to this test class. + * + * @param crate the crate to save + * @param target the target path to the save location + * @throws IOException if an error occurs while saving the crate + */ + void saveCrate(Crate crate, Path target) throws IOException; + + /** + * Ensures the crate is in extracted form in the given path. + * + * @param pathToCrate the path to the crate, may not be a folder yet + * @param expectedPath the path where the crate should be in extracted form + * @throws IOException if an error occurs while extracting the crate + */ + default void ensureCrateIsExtractedIn(Path pathToCrate, Path expectedPath) throws IOException { + try (ZipFile zf = new ZipFile(pathToCrate.toFile())) { + zf.extractAll(expectedPath.toFile().getAbsolutePath()); + } + } + + /** + * Creates a crate structure manually. + * + * @param correctCrate the path to the crate + * @param pathToFile the path to the file + * @param pathToDir the path to the directory + * @throws IOException if an error occurs while creating the crate structure + */ + default void createManualCrateStructure(Path correctCrate, Path pathToFile, Path pathToDir) throws IOException { + FileUtils.forceMkdir(correctCrate.toFile()); + InputStream fileJson = ZipStreamWriterTest.class + .getResourceAsStream("/json/crate/fileAndDir.json"); + Assertions.assertNotNull(fileJson); + // fill the directory with expected files and dirs + // starting with the .json of our crate + Path json = correctCrate.resolve("ro-crate-metadata.json"); + FileUtils.copyInputStreamToFile(fileJson, json.toFile()); + // create preview + PreviewGenerator.generatePreview(correctCrate.toFile().getAbsolutePath()); + // create the files and directories + FileUtils.writeStringToFile(pathToFile.toFile(), "content of Local File", Charset.defaultCharset()); + // creates the directory and a subdirectory + Path subdir = pathToDir.resolve("subdir"); + FileUtils.forceMkdir(subdir.toFile()); + FileUtils.writeStringToFile( + subdir.resolve("subsubfirst.txt").toFile(), + "content of subsub file in subsubdir", + Charset.defaultCharset()); + FileUtils.writeStringToFile( + pathToDir.resolve("first.txt").toFile(), + "content of first file in dir", + Charset.defaultCharset()); + FileUtils.writeStringToFile( + pathToDir.resolve("second.txt").toFile(), + "content of second file in dir", + Charset.defaultCharset()); + FileUtils.writeStringToFile( + pathToDir.resolve("third.txt").toFile(), + "content of third file in dir", + Charset.defaultCharset()); + } + + /** + * Creates a crate resembling the one we manually create in these tests. + * + * @param pathToFile the file to add + * @param pathToSubdir the directory to add + * @return the crate builder + */ + default RoCrate.RoCrateBuilder getCrateWithFileAndDir(Path pathToFile, Path pathToSubdir) { + return new RoCrate.RoCrateBuilder( + "Example RO-Crate", + "The RO-Crate Root Data Entity", + "2024", + "https://creativecommons.org/licenses/by-nc-sa/3.0/au/" + ) + .addDataEntity( + new FileEntity.FileEntityBuilder() + .addProperty("name", "Diagram showing trend to increase") + .addProperty("contentSize", "383766") + .addProperty("description", "Illustrator file for Glop Pot") + .setEncodingFormat("application/pdf") + .setLocationWithExceptions(pathToFile) + .setId("cp7glop.ai") + .build() + ) + .addDataEntity( + new DataSetEntity.DataSetBuilder() + .addProperty("name", "Too many files") + .addProperty("description", + "This directory contains many small files, that we're not going to describe in detail.") + .setLocationWithExceptions(pathToSubdir) + .setId("lots_of_little_files/") + .build() + ) + .setPreview(new AutomaticPreview()); + } +} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipStreamStrategyTest.java b/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipStreamStrategyTest.java deleted file mode 100644 index 283b306a..00000000 --- a/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipStreamStrategyTest.java +++ /dev/null @@ -1,19 +0,0 @@ -package edu.kit.datamanager.ro_crate.writer; - -import java.io.*; -import java.nio.file.Path; - -import edu.kit.datamanager.ro_crate.Crate; - -/** - * @author jejkal - */ -class ZipStreamStrategyTest extends CrateWriterTest { - - @Override - protected void saveCrate(Crate crate, Path target) throws IOException { - try (FileOutputStream stream = new FileOutputStream(target.toFile())) { - Writers.newZipStreamWriter().save(crate, stream); - } - } -} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipStreamWriterTest.java b/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipStreamWriterTest.java new file mode 100644 index 00000000..83e311b1 --- /dev/null +++ b/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipStreamWriterTest.java @@ -0,0 +1,39 @@ +package edu.kit.datamanager.ro_crate.writer; + +import java.io.*; +import java.nio.file.Path; + +import edu.kit.datamanager.ro_crate.Crate; +import edu.kit.datamanager.ro_crate.RoCrate; + +/** + * @author jejkal + */ +class ZipStreamWriterTest implements + CommonWriterTest, + ElnFileWriterTest +{ + + @Override + public void saveCrate(Crate crate, Path target) throws IOException { + try (FileOutputStream stream = new FileOutputStream(target.toFile())) { + Writers.newZipStreamWriter().save(crate, stream); + } + } + + @Override + public void saveCrateElnStyle(Crate crate, Path target) throws IOException { + try (FileOutputStream stream = new FileOutputStream(target.toFile())) { + new CrateWriter<>(new WriteZipStreamStrategy().usingElnStyle()) + .save(crate, stream); + } + } + + @Override + public void saveCrateSubdirectoryStyle(RoCrate crate, Path target) throws IOException { + try (FileOutputStream stream = new FileOutputStream(target.toFile())) { + new CrateWriter<>(new WriteZipStreamStrategy().withRootSubdirectory()) + .save(crate, stream); + } + } +} diff --git a/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipWriterTest.java b/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipWriterTest.java index 3b8b1fc0..bfb29c3d 100644 --- a/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipWriterTest.java +++ b/src/test/java/edu/kit/datamanager/ro_crate/writer/ZipWriterTest.java @@ -4,11 +4,27 @@ import java.nio.file.Path; import edu.kit.datamanager.ro_crate.Crate; +import edu.kit.datamanager.ro_crate.RoCrate; -class ZipWriterTest extends CrateWriterTest { +class ZipWriterTest implements + CommonWriterTest, + ElnFileWriterTest +{ @Override - protected void saveCrate(Crate crate, Path target) throws IOException { + public void saveCrate(Crate crate, Path target) throws IOException { Writers.newZipPathWriter() .save(crate, target.toAbsolutePath().toString()); } + + @Override + public void saveCrateElnStyle(Crate crate, Path target) throws IOException { + new CrateWriter<>(new WriteZipStrategy().usingElnStyle()) + .save(crate, target.toAbsolutePath().toString()); + } + + @Override + public void saveCrateSubdirectoryStyle(RoCrate crate, Path target) throws IOException { + new CrateWriter<>(new WriteZipStrategy().withRootSubdirectory()) + .save(crate, target.toString()); + } } diff --git a/src/test/resources/spec-v1.1-example-json-files/complete-workflow-example.json b/src/test/resources/spec-v1.1-example-json-files/complete-workflow-example.json new file mode 100644 index 00000000..011258f6 --- /dev/null +++ b/src/test/resources/spec-v1.1-example-json-files/complete-workflow-example.json @@ -0,0 +1,113 @@ +{ "@context": "https://w3id.org/ro/crate/1.1/context", + "@graph": [ + { + "@type": "CreativeWork", + "@id": "ro-crate-metadata.json", + "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, + "about": {"@id": "./"} + }, + { + "@id": "./", + "@type": "Dataset", + "hasPart": [ + { "@id": "workflow/alignment.knime" } + ] + }, + { + "@id": "workflow/alignment.knime", + "@type": ["File", "SoftwareSourceCode", "ComputationalWorkflow"], + "conformsTo": + {"@id": "https://bioschemas.org/profiles/ComputationalWorkflow/0.5-DRAFT-2020_07_21/"}, + "name": "Sequence alignment workflow", + "programmingLanguage": {"@id": "#knime"}, + "creator": {"@id": "#alice"}, + "dateCreated": "2020-05-23", + "license": { "@id": "https://spdx.org/licenses/CC-BY-NC-SA-4.0"}, + "input": [ + { "@id": "#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b"} + ], + "output": [ + { "@id": "#6c703fee-6af7-4fdb-a57d-9e8bc4486044"}, + { "@id": "#2f32b861-e43c-401f-8c42-04fd84273bdf"} + ], + "sdPublisher": {"@id": "#workflow-hub"}, + "url": "http://example.com/workflows/alignment", + "version": "0.5.0" + }, + { + "@id": "#36aadbd4-4a2d-4e33-83b4-0cbf6a6a8c5b", + "@type": "FormalParameter", + "conformsTo": {"@id": "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/"}, + "name": "genome_sequence", + "valueRequired": true, + "additionalType": {"@id": "http://edamontology.org/data_2977"}, + "format": {"@id": "http://edamontology.org/format_1929"} + }, + { + "@id": "#6c703fee-6af7-4fdb-a57d-9e8bc4486044", + "@type": "FormalParameter", + "conformsTo": {"@id": "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/"}, + "name": "cleaned_sequence", + "additionalType": {"@id": "http://edamontology.org/data_2977"}, + "encodingFormat": {"@id": "http://edamontology.org/format_2572"} + }, + { + "@id": "#2f32b861-e43c-401f-8c42-04fd84273bdf", + "@type": "FormalParameter", + "conformsTo": {"@id": "https://bioschemas.org/profiles/FormalParameter/0.1-DRAFT-2020_07_21/"}, + "name": "sequence_alignment", + "additionalType": {"@id": "http://edamontology.org/data_1383"}, + "encodingFormat": {"@id": "http://edamontology.org/format_1982"} + }, + { + "@id": "https://spdx.org/licenses/CC-BY-NC-SA-4.0", + "@type": "CreativeWork", + "name": "Creative Commons Attribution Non Commercial Share Alike 4.0 International", + "alternateName": "CC-BY-NC-SA-4.0" + }, + { + "@id": "#knime", + "@type": "ComputerLanguage", + "name": "KNIME Analytics Platform", + "alternateName": "KNIME", + "url": "https://www.knime.com/whats-new-in-knime-41", + "version": "4.1.3" + }, + { + "@id": "#alice", + "@type": "Person", + "name": "Alice Brown" + }, + { + "@id": "#workflow-hub", + "@type": "Organization", + "name": "Example Workflow Hub", + "url":"http://example.com/workflows/" + }, + { + "@id": "http://edamontology.org/format_1929", + "@type": "Thing", + "name": "FASTA sequence format" + }, + { + "@id": "http://edamontology.org/format_1982", + "@type": "Thing", + "name": "ClustalW alignment format" + }, + { + "@id": "http://edamontology.org/format_2572", + "@type": "Thing", + "name": "BAM format" + }, + { + "@id": "http://edamontology.org/data_2977", + "@type": "Thing", + "name": "Nucleic acid sequence" + }, + { + "@id": "http://edamontology.org/data_1383", + "@type": "Thing", + "name": "Nucleic acid sequence alignment" + } + ] +} diff --git a/src/test/resources/spec-v1.1-example-json-files/file-author-location.json b/src/test/resources/spec-v1.1-example-json-files/file-author-location.json new file mode 100644 index 00000000..6bd83070 --- /dev/null +++ b/src/test/resources/spec-v1.1-example-json-files/file-author-location.json @@ -0,0 +1,47 @@ +{ "@context": "https://w3id.org/ro/crate/1.1/context", + "@graph": [ + + { + "@type": "CreativeWork", + "@id": "ro-crate-metadata.json", + "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, + "about": {"@id": "./"}, + "description": "RO-Crate Metadata File Descriptor (this file)" + }, + { + "@id": "./", + "@type": "Dataset", + "name": "Example RO-Crate", + "description": "The RO-Crate Root Data Entity", + "hasPart": [ + {"@id": "data1.txt"}, + {"@id": "data2.txt"} + ] + }, + + + { + "@id": "data1.txt", + "@type": "File", + "description": "One of hopefully many Data Entities", + "author": {"@id": "#alice"}, + "contentLocation": {"@id": "http://sws.geonames.org/8152662/"} + }, + { + "@id": "data2.txt", + "@type": "File" + }, + + { + "@id": "#alice", + "@type": "Person", + "name": "Alice", + "description": "One of hopefully many Contextual Entities" + }, + { + "@id": "http://sws.geonames.org/8152662/", + "@type": "Place", + "name": "Catalina Park" + } + ] +} diff --git a/src/test/resources/spec-v1.1-example-json-files/files-and-folders.json b/src/test/resources/spec-v1.1-example-json-files/files-and-folders.json new file mode 100644 index 00000000..e8bb2ab0 --- /dev/null +++ b/src/test/resources/spec-v1.1-example-json-files/files-and-folders.json @@ -0,0 +1,38 @@ +{ "@context": "https://w3id.org/ro/crate/1.1/context", + "@graph": [ + { + "@type": "CreativeWork", + "@id": "ro-crate-metadata.json", + "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, + "about": {"@id": "./"} + }, + { + "@id": "./", + "@type": [ + "Dataset" + ], + "hasPart": [ + { + "@id": "cp7glop.ai" + }, + { + "@id": "lots_of_little_files/" + } + ] + }, + { + "@id": "cp7glop.ai", + "@type": "File", + "name": "Diagram showing trend to increase", + "contentSize": "383766", + "description": "Illustrator file for Glop Pot", + "encodingFormat": "application/pdf" + }, + { + "@id": "lots_of_little_files/", + "@type": "Dataset", + "name": "Too many files", + "description": "This directory contains many small files, that we're not going to describe in detail." + } + ] +} diff --git a/src/test/resources/spec-v1.1-example-json-files/minimal.json b/src/test/resources/spec-v1.1-example-json-files/minimal.json new file mode 100644 index 00000000..3aee3b8c --- /dev/null +++ b/src/test/resources/spec-v1.1-example-json-files/minimal.json @@ -0,0 +1,27 @@ +{ "@context": "https://w3id.org/ro/crate/1.1/context", + "@graph": [ + + { + "@type": "CreativeWork", + "@id": "ro-crate-metadata.json", + "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, + "about": {"@id": "./"} + }, + { + "@id": "./", + "identifier": "https://doi.org/10.4225/59/59672c09f4a4b", + "@type": "Dataset", + "datePublished": "2017", + "name": "Data files associated with the manuscript:Effects of facilitated family case conferencing for ...", + "description": "Palliative care planning for nursing home residents with advanced dementia ...", + "license": {"@id": "https://creativecommons.org/licenses/by-nc-sa/3.0/au/"} + }, + { + "@id": "https://creativecommons.org/licenses/by-nc-sa/3.0/au/", + "@type": "CreativeWork", + "description": "This work is licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Australia License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-sa/3.0/au/ or send a letter to Creative Commons, PO Box 1866, Mountain View, CA 94042, USA.", + "identifier": "https://creativecommons.org/licenses/by-nc-sa/3.0/au/", + "name": "Attribution-NonCommercial-ShareAlike 3.0 Australia (CC BY-NC-SA 3.0 AU)" + } + ] +} \ No newline at end of file diff --git a/src/test/resources/spec-v1.1-example-json-files/web-based-data-entities.json b/src/test/resources/spec-v1.1-example-json-files/web-based-data-entities.json new file mode 100644 index 00000000..d5245907 --- /dev/null +++ b/src/test/resources/spec-v1.1-example-json-files/web-based-data-entities.json @@ -0,0 +1,39 @@ +{ "@context": "https://w3id.org/ro/crate/1.1/context", + "@graph": [ + { + "@type": "CreativeWork", + "@id": "ro-crate-metadata.json", + "conformsTo": {"@id": "https://w3id.org/ro/crate/1.1"}, + "about": {"@id": "./"} + }, + { + "@id": "./", + "@type": [ + "Dataset" + ], + "hasPart": [ + { + "@id": "survey-responses-2019.csv" + }, + { + "@id": "https://zenodo.org/record/3541888/files/ro-crate-1.0.0.pdf" + } + ] + }, + { + "@id": "survey-responses-2019.csv", + "@type": "File", + "name": "Survey responses", + "contentSize": "26452", + "encodingFormat": "text/csv" + }, + { + "@id": "https://zenodo.org/record/3541888/files/ro-crate-1.0.0.pdf", + "@type": "File", + "name": "RO-Crate specification", + "contentSize": "310691", + "description": "RO-Crate specification", + "encodingFormat": "application/pdf" + } + ] +}