Caching judge completion

[geoalgo]

Currently, we use set_langchain_cache() to cache completions which does not work with vLLM for some reason and offer little control.

I would propose to support a caching mechanism for judge completions, e.g. when calling annotate_battles
https://github.com/OpenEuroLLM/JudgeArena/blob/main/judgearena/evaluate.py#L261 which caches judge-annotations in files like {judge_arena_dir}/cache/db/{benchmark}/{judge}.db (the naming would allow the user to manually delete some entries easily).

For the storage, we could use sqlite which would not require any dependency.

For the schema, we could use "benchmark", "instruction_id", "model_a", "model_b", "judge" as keys and retrieve any entry from the db if we get a hit and generate otherwise.

We could store the following which would allow to return completion from cache and perform small analysis:

@dataclass
class AnnotationEntry:
    benchmark: str
    instruction_id: str
    model_a: str
    model_b: str
    judge: str
    judge_input: str
    judge_completion: str
    date: str

Any thoughts @ErlisLushtaku @kargibora ?

[ErlisLushtaku]

I think this is a good idea, moving away from set_langchain_cache() looks like the right call.

Just some concern regarding the cache key implementation, should we add the following:

• the actual judge input, because models could generate different things on the same instruction
• parameters that can change the prompt e.g. provide_explanation, and the prompt version
• trucation parameters and output-affecting judge paramters

[geoalgo]

My initial thought was that we are not changing much the prompt but in case it changes, I agree that having the judge_input as key would be important (alternatively, we could put some of the metadata that Bora added).

parameters that can change the prompt e.g. provide_explanation, and the prompt version

This would not be needed if we store the judge_input.

trucation parameters and output-affecting judge paramters

We can try to pass the metadata but this one may be hard as we basically need to serialize all our state.

[ErlisLushtaku]

My initial thought was that we are not changing much the prompt but in case it changes, I agree that having the judge_input as key would be important (alternatively, we could put some of the metadata that Bora added).

parameters that can change the prompt e.g. provide_explanation, and the prompt version

This would not be needed if we store the judge_input.

trucation parameters and output-affecting judge paramters

We can try to pass the metadata but this one may be hard as we basically need to serialize all our state.

Agreed, using judge_input would make things simpler it seems. So we would just need the judge_input and the judge "signature" that would include the model and its parameters. Maybe it even makes sense to have the same cache for different models if they generate the same judge_input?

For the serialization, do you mean that it would be too large, or is there another problem with that? We could hash the json configuration of the judge together with the input so it kind of addresses the size problem. We are already doing something similar in estimate_elo_ratings.py, but maybe we need a longer hash.

[geoalgo]

Agreed, using judge_input would make things simpler it seems. So we would just need the judge_input and the judge "signature" that would include the model and its parameters. Maybe it even makes sense to have the same cache for different models if they generate the same judge_input?

True, but initially I store light keys to make things fast, not sure how slow things will be once we start to index with large strings. I have been thinking also to do the cache at the inference level (eg when calling do_inference) which has different trade-offs (the state should be easier to represent).

For the serialization, do you mean that it would be too large, or is there another problem with that? We could hash the json configuration of the judge together with the input so it kind of addresses the size problem. We are already doing something similar in estimate_elo_ratings.py, but maybe we need a longer hash.

I mean if you want to be exact with the cache, you need to exact state will require to serialize object (for the instance the judge_class) or to have everything in a metadata and both options are difficult.

[ErlisLushtaku]

True, but initially I store light keys to make things fast, not sure how slow things will be once we start to index with large strings.

If we index a fixed size hash string, the speed should not be a problem maybe.

I have been thinking also to do the cache at the inference level (eg when calling do_inference) which has different trade-offs (the state should be easier to represent).

Sounds like this would be a more general solution, since we call it in multiple entrypoints so we would need to implement the caching in one place only.

I mean if you want to be exact with the cache, you need to exact state will require to serialize object (for the instance the judge_class) or to have everything in a metadata and both options are difficult.

Yeah, that's true. For VLLM we are already exposing some arguments in the ChatVLLM class that we could use as judge metadata, but to make it work consistently across providers it would take some effort.

We could also use your proposed schema as a current best effort and deal with this later if it doesn't have priority.

[kargibora]

I think json configuration hashing can be a problem. We can let the user decide whether to store new completions with a --cache-overwrite or similar flag.

• judge_input. This already holds most of the information we need (explanations, criterias, multi-turn setups). Thus hashing this should not a problem
• Randomness: temperature $\tau$ or random seed can decide the completions. Although for most of the cases we don't change $\tau$, but there might be cases where we need results from multiple seeds.

I totally agree implementing cache in do_inference level. We probably need a cache for completions as well, apart from LLM-as-a-judge annotations.