[cogview4][feat] Support attention mechanism with variable-length support and batch packing#11349
[cogview4][feat] Support attention mechanism with variable-length support and batch packing#11349yiyixuxu merged 10 commits intohuggingface:mainfrom
Conversation
…nd batch packing Add support for variable-length attention between text and vision tokens while maintaining the original attention pattern. Implement batch packing capability to improve computational efficiency during inference and training.
|
let me know what you think too @a-r-r-o-w |
|
@a-r-r-o-w It has been renamed to |
|
I have changed back to using |
a-r-r-o-w
left a comment
There was a problem hiding this comment.
Thanks for addressing the comments!
|
@bot /style |
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
|
@bot /style |
|
Style fixes have been applied. View the workflow run here. |
What does this PR do?
This PR is primarily aimed at adding native multi-resolution + packing training support to CogView4 to better meet user needs. We have conducted relevant tests in this PR.
Before submitting
documentation guidelines, and
here are tips on formatting docstrings.
Who can review?
Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.
@a-r-r-o-w
@zRzRzRzRzRzRzR