After model quantization, the bias input of linear fails the is_param check.

[yujiaoliang]

After quantizing my model on QNN, I encountered an issue during partitioning. Specifically, when the graph is partitioned to XNNPACK, the aten.linear node’s bias input fails the is_param validation.

Is there a recommended way to handle this situation or a known workaround to make the bias input pass the is_param check?

[GregoryComer]

@yujiaoliang Can you include a code example of how you're quantizing and lowering? You mention quantizing for QNN and partitioning for XNNPACK. Are you using XNNPACK as a fallback for operators that aren't handled by QNN?

[yujiaoliang]

I used the following commands to run model quantization and compilation:
python -m examples.qualcomm.util_scripts.cli quantize -a cli_example/simple_model.pt2 -o cli_example/quantize_output -c use_8a8w -i cli_example/input_list --per_channel

python -m examples.qualcomm.util_scripts.cli compile -a cli_example/quantize_output/simple_model_quantized.pt2 -o cli_example/compile_output -m SM8750

In the cli.py, the lowering part is still handled by the executorch::to_edge_transform_and_lower interface.

Here are the changes I made:
executorch/backends/qualcomm/utils/utils.py：429
qnn_partitioners = { graph_name: [ QnnPartitioner( compiler_specs[graph_name], skip_node_id_set=skip_node_id_set, skip_node_op_set=['aten.linear.default'], skip_mutable_buffer=skip_mutable_buffer, ), XnnpackQuantizedPartitioner() ]

when lowering QNN, I skipped aten.linear and added an XNNPACK partitioner. After that, I ran into the error I mentioned earlier. Could this indicate that I should quantize the nodes separately during model quantization?

[GregoryComer]

@cccclai Do you have any suggestions on how to do this?

[cccclai]

The cli tool is mostly used for QCOM team internal benchmark. I think you can use it but there might be some internal implication in terms of how to use it.

I'd recommend pick a model from https://github.com/pytorch/executorch/tree/main/examples/qualcomm/scripts or https://github.com/pytorch/executorch/tree/main/examples/qualcomm/oss_scripts and start there.

[cccclai]

I'm not quite sure the reason you try skip_node_op_set=['aten.linear.default']. For this error message

the aten.linear node’s bias input fails the is_param validation.

I think after quantization, the bias will be buffer but not param. So it's expected to fail. You can try it with is_buffer and see if it passes

[yujiaoliang]

Thanks for the suggestions! They make a lot of sense.
I'll run some tests based on your recommendations and share the results once I have them.