WIP: Automate the generation of the examples#123
Open
Robsteranium wants to merge 1 commit intomasterfrom
Open
Conversation
Adds a script to regenerate the csvw for this example Changes some output filenames to avoid clobbering between pipelines Introduces util/csvw-url function which allows csvw:urls to be specified relatively (i.e. csv file relative to location of json) so the examples don't have absolute paths Minor consistency edits to tests
Robsteranium
commented
Apr 28, 2020
| @@ -1,9 +1,9 @@ | |||
| component_slug,component_attachment,component_property | |||
Contributor
Author
There was a problem hiding this comment.
I think this is basically just an ordering change
Robsteranium
commented
Apr 28, 2020
| Label,Notation,Parent Notation | ||
| Export,export, | ||
| Import,import, | ||
| label,notation,parent_notation,sort_priority,description,top_concept_of,has_top_concept,pref_label |
Contributor
Author
There was a problem hiding this comment.
Here we see the benefit of doing this. We didn't keep this file up-to-date as these extra columns were introduced.
Robsteranium
commented
Apr 28, 2020
| (let [metadata-file (example-csvw example-directory (str example-directory "-metadata.json")) | ||
| schema (util/read-json metadata-file) | ||
| part-number (case part-name | ||
| "dataset" 0 |
Contributor
Author
There was a problem hiding this comment.
Ordinal matching isn't particularly robust, but it's at least simpler than matching by e.g. the aboutURL template.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
It'd be nice to use the new csvw and exec tasks from #120 to regenerate the examples. This will help keep them up-to-date and also serves as a test in it's own right (firstly whether they can be generated successfully and secondly because the examples are used by the test suite).
This PR provides an example script. This involves a few changes:
csvw:urlspecified relatively so the checked-in examples don't contain references to absolute paths on the machine that generated them (seeutil/csvw-url)Outstanding issues for discussion
Does
util/csvw-urlwork in a reasonable way?csv2rdf expects the
csvw:urlto specified in one of three ways:file:/absolute/path/input.csvinput.csv(relative to the location of the json metadata file)file:./relative/path/input.csv(relative to JVM working dir - so you would need to call table2qb and csv2rdf from the same place)If the
output-directoryparameter is specified absolutely then the first approach is taken, if relatively then the second. Arguably, the third (file:./relative/path/input.csv) would be more consistent (also having a "file" schema), howeverjava.net.URIdoesn't seem to like creating relative URIs for files (even though csv2rdf happily reads this).Moreover explicit configuration might be preferred as a convention like this might be a bit magic/ mysterious.
In any case we ought to document this behaviour somewhere.
How should we call table2qb to generate the examples?
I think it's appealing to use the CLI to do this as means we're exercising that interface. Shell scripts mightn't be ideal, particularly once we extend this to all of the examples. Further we rely on developers remembering to run the scripts and commit the changes when the inputs to the examples change.
An alternative would be specify the calls as part of the test suite. It would be more convenient to specify lots of examples/ parameters with data structures in clojure. We can also hook into the test cycle so that the examples are regenerated every time the suite runs. We'd loose the checks we otherwise having on the CLI interface.
Further requirements for implementation