Importance weight based sparse attention implementation for auto-regressive decoding.#2

Open

FengDSP wants to merge 3 commits intomainfrom

important_kv_cache

Commits on Oct 24, 2023

masked_tokens uses session_length
Feng Li
committed
masked_tokens uses session length everywhere
Feng Li
committed
Important KV Cache in auto-regressive decoder.
Feng Li
committed