WAN 2.1 Performance vs FA3

[OutofAi]

I gave the SLA code a try, plugged it into the self attention code for Wan 2.1, but in comparison to Flash attention 3 the speed is on Hopper Architecture at least unnoticeable and mainly slower than FA3 in some benchmarks. Is there certain Hyperparameters or specific set up that would introduce the speed up or is that as expected?

[Lyxien]

Have you fine-tuned it before？

[giruuuuj]

Current Performance Expectations
It appears that WAN 2.1 may not consistently outperform FA3 on Hopper architecture in its current state. The performance differences you're observing (being mainly slower than FA3) seem to align with what others are reporting.

Key Factors Affecting Performance

1. Fine-tuning Requirement
   WAN 2.1 likely requires specific fine-tuning to achieve optimal performance

The implementation may need adaptation to your specific hardware and use case

Without proper tuning, FA3 may remain faster due to its mature optimization

2. Architecture Considerations
   Hopper architecture (H100) has specific characteristics that favor different optimization approaches

FA3 has been extensively optimized for modern GPU architectures

WAN's advantages might be more apparent on different hardware or specific workload patterns

3. Implementation Factors
   Ensure you're using the latest versions of both implementations

Check for any specific configuration requirements in WAN 2.1 documentation

Verify that you're using the appropriate data types and memory layouts

Recommended Actions
Check the official WAN repository for any specific tuning guidelines or performance optimization tips

Experiment with different hyperparameters - batch sizes, sequence lengths, and attention heads configuration

Monitor GPU utilization to identify potential bottlenecks

Consider waiting for future updates as WAN matures and receives more optimization

[jt-zhang]

@OutofAi Currently, the Triton code is optimized for the RTX 5090. To achieve higher speedups across a broader range of GPUs (e.g., H100), we plan to release a CUDA version after our paper is accepted.

[jt-zhang]

We’ve just updated the Triton implementation of SLA. Training is now more stable, faster, and typically achieves better training results.

We’ve released the code for SageSLA, a very fast SLA (Sparse-Linear Attention) forward pass based on SageAttention. It uses some code from SpargeAttn. Please refer to the SageSLA/ directory for how to use SageSLA.

Feel free to try it out! Now SageSLA is very fast on RTX and Hopper GPUs.

[limou102]

Current Performance Expectations It appears that WAN 2.1 may not consistently outperform FA3 on Hopper architecture in its current state. The performance differences you're observing (being mainly slower than FA3) seem to align with what others are reporting.

Key Factors Affecting Performance

1. Fine-tuning Requirement
   WAN 2.1 likely requires specific fine-tuning to achieve optimal performance

The implementation may need adaptation to your specific hardware and use case

Without proper tuning, FA3 may remain faster due to its mature optimization

2. Architecture Considerations
   Hopper architecture (H100) has specific characteristics that favor different optimization approaches

FA3 has been extensively optimized for modern GPU architectures

WAN's advantages might be more apparent on different hardware or specific workload patterns

3. Implementation Factors
   Ensure you're using the latest versions of both implementations

Check for any specific configuration requirements in WAN 2.1 documentation

Verify that you're using the appropriate data types and memory layouts

Recommended Actions Check the official WAN repository for any specific tuning guidelines or performance optimization tips

Experiment with different hyperparameters - batch sizes, sequence lengths, and attention heads configuration

Monitor GPU utilization to identify potential bottlenecks

Consider waiting for future updates as WAN matures and receives more optimization

Hello, I would like to know why finetuning affects speed.
From the code implementation, the sparse computation process selects the top-K blocks to compute, so the sparsity is relatively fixed. The linear computation process is also dense and fixed. Theoretically, finetuning should not affect computational efficiency.
Is my understanding correct?