[RFC] Constant tensors caching pass

[niuxiaog]

Related to issue #146 .

[ZhennanQin]

compiletime_fold will be a problem because:

1. The kernel binary will contain the whole tensor to be folded, which is too large. If we only want to use runtime_fold from it, the binary size is wasted and this is not friendly for kernel cache.
2. For GPU device, we may want to do the folding on CPU. We shouldn't put three functions in the same module to achieve that.

If we want to support direct compile-time folding, I suggest following the direction of section 2.1 to implement it.

[niuxiaog]

For the compile time available tensors, the integration can choose to:

• lower them to arith.constant, which is not suggested,
• put them into the arguments list of the module and mark them as compiletime_const_args,
• put them into the arguments list of the module and mark them as runtime_const_args.

For the first two choices, they will be folded by compiletime_fold, and for the third choice, by runtime_fold. There will be no large literal tensors in the kernel for the last two choices.

[niuxiaog]

We shouldn't put three functions in the same module to achieve that.

I'm not clear if we can generate a new module in the pass pipeline. If so, shall we put the compiletime_fold in one module, and runtime_fold and compute in another module?

[ZhennanQin]

I'm not clear if we can generate a new module in the pass pipeline. If so, shall we put the compiletime_fold in one module, and runtime_fold and compute in another module?

I think so. But my current thinking is, we can support compiletime_fold in the future when there's a demand for this.

[niuxiaog]

OK, I will put all folding operations into runtime_fold.

[lmontigny]

waiting from OV integration, how they handle const weight..?
with TPP -> use it in IR as const.
to discuss with OV on iteration 4.

[lmontigny]

RFC LGTM
Do we have an idea of the performance impact of this pass on some example?

[niuxiaog]

RFC LGTM Do we have an idea of the performance impact of this pass on some example?

Ideally, the execution time of operations on weights of matmuls, such as tensor.pack and linalg.transpose, will be saved. The ratio of improvement will depend on actual models. We will collect performance data after all is ready.