support multiheadattention int8

[tpoisonooo]

这是在干啥

支持 mha int8 kernel

• GEMM weight 都还是 per-channel 量化
• 内部需要 5 个 input scale 参数
  • xq/xk/xv 的 scale
  • softmax 之前的 scale
  • 乘 out_weight 之前的 scale

速度对比 （wsl2 虚拟机）

1 线程

$ ./benchncnn  10 1
loop_count = 10
num_threads = 1
powersave = 0
gpu_device = -1
cooling_down = 1
  vision_transformer  min = 2955.98  max = 3130.18  avg = 3051.40
vision_transformer_int8  min = 2403.91  max = 2459.07  avg = 2431.06

8 线程

$ ./benchncnn
loop_count = 4
num_threads = 8
powersave = 0
gpu_device = -1
cooling_down = 1
  vision_transformer  min = 1175.01  max = 1575.90  avg = 1343.40
vision_transformer_int8  min = 1076.93  max = 1153.30  avg = 1109.33

softmax 数值结果对比

直接量化 mha/conv/gemm 三类 opr 版本，不校准 bias
(base) khj@khj:~/ncnn/ninjabuild/examples$ ./vision_transformer
data size 1769472
output shape whc 1000,1,1
softmax result: 65 0.978581

浮点版本
(base) khj@khj:~/ncnn/ninjabuild/examples$ ./vision_transformer_fp32
data size 1769472
output shape whc 1000,1,1
softmax result: 65 0.985758

备注

需要先处理 PR 3911，我 rebase 一下。
或者直接 review 这个，也是一样的。

精度测试

pytorch fp32 原始模型，完整的 5w 张图
top-1 84.01%
top-5 97.08%

基线：ncnn fp32 原始模型，CPU 推理太慢了只能跑 2000 张
2022-06-28 17:49:46,793 - test - INFO - accuracy_top-1 : 83.55
2022-06-28 17:49:46,799 - test - INFO - accuracy_top-5 : 97.55

量化 conv+mha
2022-06-28 14:26:39,188 - test - INFO - accuracy_top-1 : 83.25
2022-06-28 14:26:39,194 - test - INFO - accuracy_top-5 : 97.65

量化 conv+mha+gemm
2022-06-27 21:05:06,841 - test - INFO - accuracy_top-1 : 82.55
2022-06-27 21:05:06,844 - test - INFO - accuracy_top-5 : 97.45

量化 conv+mha+gemm+bias 校准
2022-06-29 12:31:18,982 - test - INFO - accuracy_top-1 : 82.80
2022-06-29 12:31:18,984 - test - INFO - accuracy_top-5 : 97.55

结论：mha +conv 直接量化会影响 -0.3%； gemm 直接量化会影响 -0.7%，用 bias 校准可以救回来 +0.25%。

naive 整体加速 20%，掉点 -0.75%，模型大小 337MB->86MB

[codecov-commenter]

Codecov Report

Merging #3940 (3f1844b) into master (8c06103) will decrease coverage by 0.18%.
The diff coverage is 9.72%.

@@            Coverage Diff             @@
##           master    #3940      +/-   ##
==========================================
- Coverage   93.84%   93.65%   -0.19%     
==========================================
  Files         721      728       +7     
  Lines      175071   177009    +1938     
==========================================
+ Hits       164291   165778    +1487     
- Misses      10780    11231     +451     

Impacted Files	Coverage Δ
src/layer/multiheadattention.cpp	47.82% <9.72%> (-45.41%)	⬇️
src/command.cpp	72.70% <0.00%> (-14.94%)	⬇️
src/pipeline.cpp	58.69% <0.00%> (-2.18%)	⬇️
src/layer/vulkan/reshape_vulkan.cpp	92.01% <0.00%> (-2.14%)	⬇️
src/layer/x86/cast_x86.cpp	96.07% <0.00%> (-1.91%)	⬇️
src/layer/vulkan/packing_vulkan.cpp	81.70% <0.00%> (-1.88%)	⬇️
src/layer/vulkan/permute_vulkan.cpp	96.99% <0.00%> (-1.60%)	⬇️
src/layer/vulkan/reorg_vulkan.cpp	96.35% <0.00%> (-1.57%)	⬇️
src/layer/vulkan/pixelshuffle_vulkan.cpp	96.35% <0.00%> (-1.57%)	⬇️
src/layer/vulkan/flatten_vulkan.cpp	95.97% <0.00%> (-1.51%)	⬇️
... and 49 more

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c06103...3f1844b. Read the comment docs.

[nihui]

ref
https://github.com/megvii-research/FQ-ViT
https://github.com/megvii-research/FQ-ViT/blob/main/models/ptq/layers.py#L209

[nihui]

close as mha dynamic int8 quantization now supported