Can't reproduce perplexity on LLaMA3-8B

[Swiminn]

Hello, thank you for releasing the code.
I tried to reproduce the results reported in the SpinQuant paper, but I’m seeing a notable gap in perplexity for the LLaMA3-8B model W4A4KV4 experiment.

My setup is as follows: I didn’t change any code and strictly followed the training and evaluation commands provided in the README. The environment was CUDA 12.1, PyTorch 2.5.1, Python 3.9, and I ran everything on an A100 (80GB) GPU.

When I trained following the repository’s instructions and using 10_optimize_ratation.sh and 2_eval_ptq.sh, I obtained a perplexity of 8.0, while the paper reports 7.3. When I used the rotation matrices provided in the GitHub repository directly, the perplexity increased slightly to 8.5.

I double-checked that the same evaluation scripts were used. Could you please clarify whether there are any additional steps that are required to achieve the reported 7.3 perplexity?

Thank you for your work and for sharing the implementation.

[Swiminn]

Additionally, my experimental results showed that when reproducing Llama2-7B W4A4KV4, the perplexity consistently came out to be 6.5 regardless of whether I retrained the rotation matrix or used the provided one (the paper states 5.9).