Skip to content

Conversation

@YushengZhao
Copy link

Description

This PR adds support for the Gated Linear Attention (GLA) operator in the GGML CANN backend. GLA is widely used in efficient attention mechanisms (e.g., RWKV, Linear Transformer variants, etc.), which leverage gating signals and state accumulation to significantly reduce computational complexity while preserving strong modeling capacity.

Summary of Changes:

  • Registered GGML_OP_GATED_LINEAR_ATTN in ggml/src/ggml-cann/ggml-cann.cpp and bound it to a newly implemented function ggml_cann_gated_linear_attn.
  • Implemented the core forward logic of ggml_cann_gated_linear_attn in ggml/src/ggml-cann/aclnn_ops.cpp, using ACLNN primitives such as Repeat, Mul, Add, and Mv to compose the GLA computation.
  • Supports batched multi-head GLA with input tensor layout (C, H, T, B), where:
    • C = H * D (total channel dimension),
    • T = B * L (flattened batch × sequence length),
    • consistent with GGML’s internal memory layout conventions.
  • Accepts learnable gate g and recurrent state s as additional inputs, enabling joint state update and output generation in a single pass.

Testing

Build with CANN backend enabled:

cmake -B build -DGGML_CANN=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build --config Release -j

Run GLA-specific backend test (requires adding a test case for GATED_LINEAR_ATTN in tests/test-backend-ops.cpp):

./bin/test-backend-ops test -b CANN0 -o GATED_LINEAR_ATTN
image

@github-actions github-actions bot added ggml changes relating to the ggml tensor library for machine learning Ascend NPU issues specific to Ascend NPUs labels Dec 6, 2025
@YushengZhao YushengZhao changed the title support gated linear attn CANN: support gated linear attn Dec 6, 2025
@ggerganov
Copy link
Member

Note that this operator might be removed at some point - see #17716 (comment)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Ascend NPU issues specific to Ascend NPUs ggml changes relating to the ggml tensor library for machine learning

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants