[FEATURE] Every Eval Ever

[borgr]

Problem

Is your proposal tackling an exisitng problem or limitation?

• No, it's an addition

Proposal

Describe what you would like to see happen.

• Type:
  • New Ontology (data source for multiple tasks)
  • New Task(s)
  • New Model(s)
  • New Metric(s)
  • [ x] Other
• Area(s) of code: somehwere at the output level

Alternatives

Have you considered other approaches? Briefly list them and why you did not choose them.

Additional Context

Links to related issues, PRs, docs, or external references.
Screenshots or small examples if useful.

Implementation

• I plan to implement this in a PR
• I am proposing the idea and would like someone else to pick it up
  Hey folks, cool evaluation tool. I was wondering if you thought of integrating your outputs to the standard format set by EveryEvalEver, and making an easy way to share the results of the evaluation in a way evaluation folks would be able to study them?

This will bring some more people to look at what you are doing and for the long run allow other comparisons to be made as it will be in the place we (eval folks) will be looking.

[warreveys]

Hi @borgr!

Thanks for reaching out!

As I understand it, your message was mostly a call for participation tied to the ACL 2026 shared task, and the deadline has already passed. Is the schema now in an (almost) final state, or will it still evolve based on the findings?

More importantly, Every Eval Ever looks quite generative LLM-focused (fields like temperature, top_p, max_tokens, etc.), whereas WorkRB is aimed at evaluating any AI model that addresses a work-related (HR) problem, including: bi-encoders, BM25, and other non-generative approaches. A lot of the EEE schema would simply not apply. What's your view on this mismatch?

[borgr]

EEE is new but here to stay and evolve, finding coverage mismatches would be interesting and possible to change EveryEvalEver to support those. But also note that a lot of the generative stuff can just be left blank if it is irrelevant, and maybe this is the path of least resistance, worth consideration, or add just what fits the current schema well and see if worth adding more with schema changes.
The shared task ended (although we do consider giving another week for people to join after neurips that just passed), but submissions are ongoing and open forever and attribution currently is by the seeing on GitHub and by having field in the schema that would refer to you, which will allow whoever wants more information to look at it.
What do you think? What are your goals here and what would fit?