CO-SHORTCUTHOOK is a unified evaluation framework for measuring shortcut reliance in Code Large Language Models (Code LLMs).
Modern Code LLMs have achieved remarkable performance across a wide range of software engineering tasks, including code generation, vulnerability detection, and repository-level reasoning. However, instead of learning genuine program semantics, these models may rely on shortcut representations—predictive cues that correlate with task labels but lack causal relationships with program behavior. Such shortcut reliance often leads to hallucinations, spurious correlations, and poor robustness under semantics-preserving code transformations.
CO-SHORTCUTHOOK provides the first unified framework for systematically evaluating shortcut learning in Code LLMs. It introduces a taxonomy of eight shortcut types spanning four categories:
- Surface-level shortcuts
- Semantic-level shortcuts
- Usage-prior shortcuts
- Structural shortcuts
Based on this taxonomy, CO-SHORTCUTHOOK evaluates shortcut reliance through semantic-preserving trigger injection and a collection of shortcut-aware evaluation metrics. The framework enables researchers to analyze shortcut behaviors across different Code LLM families, model scales, downstream tasks, and fine-tuning settings.
CO-SHORTCUTHOOK supports comprehensive analyses of shortcut dependence, robustness, and shortcut transfer from pretraining to downstream fine-tuning, providing practical tools for understanding and improving the reliability of Code LLMs.
- Python 3.12
- PyTorch 2.6
pip install -r requirements.txt
All evaluation datasets are included in this repository and can be found in the humaneval/ directory.
The evaluation pipeline consists of four sequential steps.
Generate function implementations for the benchmark tasks using the target Code LLM.
python3 code_complete.pyExtract the generated function bodies from the model outputs.
python3 extract_generated_function.pyEvaluate whether the generated functions pass the corresponding test cases.
python3 code_correct_evaluation.pyMeasure shortcut reliance and compute shortcut-aware evaluation metrics.
python3 shortcut_analysis.pyAfter completing these four steps, CO-SHORTCUTHOOK reports the functional correctness and shortcut reliance of the evaluated Code LLM.
The vulnerability detection experiments are built upon the implementation provided by BLoB.
The evaluation dataset is available in the defect-detection-testset/ directory.
Evaluate shortcut dependence using the generated predictions.
python3 VD_classifier_analysis.py