Hi authors!
Thanks for sharing this incredible work! When I try to reproduce the results 'Quarot+GPTQ' in Tab1 of Spinquant paper, I am confusing with the reported results.
In Tab11 of Quarot paper (2404.00456), the PPL of W4A4KV4 Llama3-70B is 6.66, but the reported number in Tab5 of Spinquant paper (2405.16406) is instead 20.2. I also try to reproduce the results at my end, and my reproduce using Quarot's codebase is 16.6.
I have no idea on why this huge discrepancy happens. Could you give me some hints? Many thanks in advance.
Hi authors!
Thanks for sharing this incredible work! When I try to reproduce the results 'Quarot+GPTQ' in Tab1 of Spinquant paper, I am confusing with the reported results.
In Tab11 of Quarot paper (2404.00456), the PPL of W4A4KV4 Llama3-70B is 6.66, but the reported number in Tab5 of Spinquant paper (2405.16406) is instead 20.2. I also try to reproduce the results at my end, and my reproduce using Quarot's codebase is 16.6.
I have no idea on why this huge discrepancy happens. Could you give me some hints? Many thanks in advance.