How to create long video

[LukeLIN-web]

seems cannot change parameter to create long video ?

[AdvancedHe]

The current code does not support.

[zhuhz22]

Yes, for now—just like Self Forcing, which also doesn’t officially support long videos. However, our method is compatible with the long-video extensions built on top of Self Forcing, such as LongLive, Infinity-RoPE, and Deep Forcing. We’ll integrate a long-video version soon. Stay tuned!

[zhuhz22]

Hi @LukeLIN-web @AdvancedHe ,
Update: we now support Rolling Forcing, enabling minute-level long video generation!

In addition, both Infinity-RoPE and Deep Forcing adopt Causal Forcing as one of their base models. They can also generate interactive (prompt-switchable) long videos at the minute scale. ( Infinity-RoPE also showcases a comparison in their repo: Causal Forcing generates long videos with better motion dynamics and visual quality than Self Forcing.)

Feel free to try them out!

[LukeLIN-web]

Great! Feel free to close it.

[robintzeng]

Hi @zhuhz22
I meet a issue when I try to generate a video with num_output_frames 3000

  File "/code/code/Causal-Forcing/long_video/wan/modules/causal_model.py", line 380, in forward
    y = self.self_attn(
  File "/code/miniconda3/envs/self_forcing/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1775, in _wrapped_call_impl
    return self._call_impl(*args, **kwargs)
  File "/code/miniconda3/envs/self_forcing/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1786, in _call_impl
    return forward_call(*args, **kwargs)
  File "/code/code/Causal-Forcing/long_video/wan/modules/causal_model.py", line 198, in forward
    roped_query = causal_rope_apply(
  File "/code/code/Causal-Forcing/long_video/wan/modules/causal_model.py", line 43, in causal_rope_apply
    freqs[0][start_frame:start_frame + f].view(f, 1, 1, -1).expand(f, h, w, -1),
RuntimeError: shape '[12, 1, 1, -1]' is invalid for input of size 220

[zhuhz22]

freqs(Tensor): Rope freqs, shape [1024, C / num_heads / 2] is 1024 in dim==0, thus num_output_frames should be less than 1024. For more frames such as 3000, you need to modify freqs to [3000, C / num_heads / 2] here:

        self.freqs = torch.cat([
            rope_params(3000, d - 4 * (d // 6)),
            rope_params(1024, 2 * (d // 6)),
            rope_params(1024, 2 * (d // 6))
        ],
            dim=1)

[robintzeng]

@zhuhz22 I first met the dimension issue when simply change the first dim into 3000
then I change to this and meet the vae memory issue.

self.freqs = torch.cat([
            rope_params(3000, d - 4 * (d // 6)),
            rope_params(3000, 2 * (d // 6)),
            rope_params(3000, 2 * (d // 6))
        ],
            dim=1)

I think we need more changes for the infinite length video

[zhuhz22]

Yes, as in Rolling Forcing, the VAE can be the bottleneck of memory when decoding too long videos. You can update this issue in their repo.