Add metadata to HEPData tables

[mhabedan]

Hi @GraemeWatt,

In the OpenMAPP workshop, we discussed that it would be immensely useful if analysers had to specify additional metadata for the tables to clarify what kind of information they contain. Concretely, we were thinking of the following field:

field	values
table_type	Cutflow / Covariance / Correlation / Distribution / Efficiency / Limit / Observable / Yield / Other

The table_types Distribution, Observable and Yield are somewhat ambiguous but will hopefully allow analysers to pick something they feel is appropriate for their analysis. For these three tables, it would be useful if uploaders could additionally specify these fields:

field	values	applicable to
representation_level	parton level / particle level / detector level	whole table
data_source	data / theory / uncertainty	individual column
fit_type	pre-fit/ post-fit (optional)	individual column

For tables of table_type: Limit, it would be useful if uploaders could additionally specify these fields:

field	values	applicable to
limit_type	Cross section / Exclusion curve	whole table
dimension	integer	whole table
level	expected / observed	individual column

This kind of information would help reinterpretation tools to automatically decide how to handle the entries in a table or allow to query for a certain kind of table.

What do you think? Would it be possible to add this to HEPData?

Cheers,
Martin

[GraemeWatt]

Thanks for the suggestion. Provision for flexible metadata is already provided by the qualifiers field of the data file (docs), which applies per column ("dependent variable") or per table (if the same value is given for each column). Each qualifier consists of a name and value (with optional units). For the NUISANCE use case, some standard qualifier names were defined (docs). Something similar could be specified for data relevant to BSM searches along the lines you mentioned. I don't think any code changes are needed, but perhaps you could add a page to the HEPData Submission docs via a PR in the hepdata-submission repo, which could also be linked from the hepdata_lib docs.

[GraemeWatt]

The qualifiers are not indexed and are therefore not searchable. If this is important, per-table keywords such as observables and phrases could also be used. For example, searching for phrases:cutflow currently returns 15 results. A search for cutflow across all indexed metadata fields (such as the table description) currently returns 117 results.

[mhabedan]

Hi @GraemeWatt! Thanks for the swift reaction!

Our idea actually was to make these fields mandatory unless otherwise indicated. That would not only drastically increase the amount of metadata provided in practise but also make sure that a "normalised" nomenclature is used. Otherwise, queries have to take into account a variety of different names that submitters come up with for the same thing, e.g. "Limit", "1DLimit", "Exclusion curve", "Exclusion contour", ..., and then require some manual intervention afterwards to clean up the results. Or in your given example, people might spell it "cut flow" or "cut-flow", which gives another 33 results that are neither included in the 15 results nor in the 117 that you gave. In a way, enforcing this standardised nomenclature actually makes it easier for submitters because they don't have to come up with the most useful terms to describe their records themselves.

Would you see an issue with making this mandatory?

[GraemeWatt]

It's a good idea to introduce standardised nomenclature. But are the proposed fields completely general or are they only meaningful for data relevant to BSM searches that will be included in the various reinterpretation frameworks covered by OpenMAPP? The latter is only a subset of data submitted to HEPData. I don't see that submitters of other data (e.g. in heavy-ion physics, flavour physics, neutrino physics, etc.) should be inconvenienced by having to specify extra irrelevant metadata that will never be used. In the case that the proposed fields are only relevant for BSM searches from ATLAS and CMS, it would be better just to write a documentation page (e.g. on the OpenMAPP website) and maybe write a script that checks for the presence of qualifiers that could be used in an additional validation step. Then you could communicate to the relevant ATLAS/CMS physics groups that new HEPData submissions should contain the extra metadata and pass the additional validation check. HEPData is a general-purpose HEP repository and the core software and documentation should avoid including domain-specific features that do not apply to all of HEP.

[mhabedan]

Hi @GraemeWatt,

We came up with this scheme at the OpenMAPP workshop in October and tried to formulate terms that are as representative as possible but also unambiguous. Insofar, we'd hope that this is indeed general enough for HEP. I will reach out to people from other fields (neutrino physics, heavy ions, ...) to see whether we need to extend the options.

You are in principle right that this could be communicated with a readme and a validation step within the collaborations but from experience people don't read readmes and enforcing things from the experiment side is often troublesome as well. Insofar, enforcing this from HEPData's side would be easiest.

[GraemeWatt]

The HEPData YAML format has been largely unchanged since it was devised 10 years ago, so a change to require some mandatory new fields cannot be done overnight. This would break all user uploads and also most of our test data. Instead, a phased approach would be needed, for example:

1. Clearly document the new (optional but recommended) metadata fields via a page in the HEPData Submission docs (linked from the hepdata_lib docs). Communicate to the relevant ATLAS/CMS physics groups (and other experiments relevant to reinterpretation tools) that new HEPData submissions should contain the extra metadata.
2. Assuming widespread adoption after some period (~6 months, say), the hepdata-validator could be modified to check for the metadata fields and issue a warning (not an error) if they are absent, which could be propagated to the email sent to an Uploader, but the submission would still pass validation. Coordinators should be notified that the extra metadata will soon be a requirement.
3. After a further period (~6 months, say), the warning issued by the hepdata-validator would change to an error and the submission would fail validation. Coordinators should be notified far in advance (~6 months, say) of the date that the warning will switch to an error to give them time to modify their workflows.

But given the new metadata fields would only apply to new submissions and not past ones, all reinterpretation tools would need to treat these fields as being optional rather than required. Therefore, I don't see a real need to progress beyond step 1. above. In any case, the need to progress with steps 2. and 3. could be evaluated after step 1. is implemented.

[mhabedan]

Hi @GraemeWatt,

Indeed, the staggered approach seems like a good idea! In that way, phase 1 and 2 will allow to also collect some additional feedback of whether the categories are general enough before breaking anyone's workflow. It will also allow us to more or less directly go ahead with phase 1, won't it?

I'm not quite as optimistic that phase 1 on its own will be enough. I suspect there will always be some stray uploading mechanisms here and there that the news about the new fields didn't reach. And in the end, preserved data is only as good as the metadata describing it so this is an important thing to get right.

Regarding reinterpretation tools and old entries: indeed, that will be a persistent issue. Either, one could try to slowly add this metadata to existing HEPData records; ideally as a community effort such that the sheer amount of records can be overcome. Or reinterpretation tools will have to live with the fact that old entries are metadata-incomplete. But at least the fields would be available for reinterpretation studies that only care about upcoming results, e.g. from HyperK, Dune, EIC or HL-LHC.

[agbuckley]

FYI, @GraemeWatt I've just been testing some of these proposals on the draft record at https://www.hepdata.net/record/142663

I felt that in practice a bit of refinement was needed, e.g. it seemed silly to be applying a table_type keyword on a table (it already knows it's a table... and hepdata_lib anyway forbade that) or as a duplicated column qualifier, but I'm happy to take direction there. I instead used data_type, data_source and data_level. For consistency, simplicity, and IT convention I also thought it best to remove the first-letter capitalisation from those values.

In general, I think this is nice, and we can possibly use it for querying and certainly in filtering and converting the downloaded data. It would also be good, once we're agreed on a standard, to use the data_source to refine the plotting: at the moment all y-columns are rendered "real-data style", as random-coloured points with error bars; the ideal would be to match the publication convention of (primarily black) dots and errors for the real data, or which there will usually just be one, and then a colour cycle of continuous stepped lines (and/or bands) for the theory columns. I also found it would be useful to be able to choose which y-columns are plotted, when there are multiple. Do you want these suggestions as a separate issue ticket, @GraemeWatt ?