[ICML 2026] Official code for "Revealing Long-context Potential of Attention Heads via Frequency Kernels".
The two Jupyter notebooks (corresponding to the Llama 3 and Qwen 3 series respectively) provide most of the code used in the experiments of this paper. They can be run directly to interactively observe the properties of the frequency kernels of attention heads and to compute the long-context potential scores. Best with transformers==4.56.2 and torch==2.6.0.
We also provide evaluation scripts under opencompass/.
Coming soon!