Implementing ext-image-capture-source-v1 and ext-image-copy-capture-v1

[glamberson]

I'm developing a Smithay-based compositor that requires screen capture capabilities for client access. Per the protocol tracking in #781, neither ext-image-capture-source-v1 nor ext-image-copy-capture-v1 are currently implemented.

I have an immediate need for these protocols and am prepared to contribute an implementation. Before beginning work, I wanted to check:

1. Is there existing work on these protocols that I should be aware of?
2. Are there any design preferences or constraints from the maintainers' perspective?
3. Would a phased approach (ext-image-capture-source-v1 first, then ext-image-copy-capture-v1) be acceptable?

From my initial review, the existing infrastructure appears well-suited for this:

• ExportMem provides framebuffer readback
• DmabufState and ForeignToplevelListState offer relevant patterns
• The protocol definitions are available in wayland-protocols 0.32.4+

I can adapt my current development schedule to prioritize this if there's interest in an upstream contribution.

Greg Lamberson
Lamco Development
https://lamco.ai

[ids1024]

In cosmic-comp, we currently have an implementation of ext-image-copy-capture-v1 (https://github.com/pop-os/cosmic-comp/blob/master/src/wayland/protocols/screencopy.rs) and ext-image-capture-source-v1 (https://github.com/pop-os/cosmic-comp/blob/master/src/wayland/protocols/image_capture_source.rs), along with our own extension protocol to be able to capture workspaces.

Niri appears to currently support the older wlr screencopy (https://github.com/YaLTeR/niri/blob/main/src/protocols/screencopy.rs).

I've been meaning to work on moving some of this logic from cosmic-comp to Smithay, but haven't gotten around to it.

[glamberson]

Thanks for sharing those links. I took a look at both implementations - the cosmic-comp code is thorough, particularly the RAII patterns in screencopy.rs for session and frame lifecycle management.

I have an immediate need for these protocols in a compositor I am developing, so I would be glad to help move this forward if that would be useful. If you are planning to upstream the cosmic-comp implementation, I could assist with:

• Removing COSMIC-specific dependencies (CosmicSurface, WorkspaceHandle, etc.)
• Adding Smithay-style module documentation
• Adapting the toplevel capture to use ForeignToplevelHandle
• Testing and integration

I have also been working on some non-COSMIC-specific scaffolding that follows Smithay patterns, which might be useful as a starting point or reference. For example, a source type enum using Smithay's existing types:

#[derive(Debug, Clone)]
pub enum CaptureSourceType {
    Output(WeakOutput),
    Toplevel(ForeignToplevelHandle),
}

And handler traits following the convention of other Smithay protocols:

pub trait ImageCaptureSourceHandler:
    GlobalDispatch<ExtOutputImageCaptureSourceManagerV1, ImageCaptureSourceGlobalData>
    + GlobalDispatch<ExtForeignToplevelImageCaptureSourceManagerV1, ImageCaptureSourceGlobalData>
    + Dispatch<ExtOutputImageCaptureSourceManagerV1, ()>
    + Dispatch<ExtForeignToplevelImageCaptureSourceManagerV1, ()>
    + Dispatch<ExtImageCaptureSourceV1, ImageCaptureSourceData>
    + 'static
{
    fn image_capture_source_state(&mut self) -> &mut ImageCaptureSourceState;
}

Let me know how I can help, or if you would prefer to handle this yourself when time permits.

Greg

[ids1024]

I'm probably still not working on this too soon, so if anyone can implement a version of it that is suitable for inclusion in Smithay, that seems good to me.

#[derive(Debug, Clone)]
pub enum CaptureSourceType {
    Output(WeakOutput),
    Toplevel(ForeignToplevelHandle),
}

I think ideally we want an API that's extensible. So a compositor like cosmic-comp could use the smithay abstraction, but define an additional type of capture source with its own protocol.

Not sure exactly how we'd want to design an API for that.

A compositor may also want to implement one capture source type, but not another. So there should also probably be a way to expose only the output image capture source but not the toplevel one, or vice versa.

Moving the relevant code from cosmic-comp/src/wayland/protocols to smithay, except the COSMIC specific workspace capture protocol, while also providing a way to extend the implementation for that should cover what we'd want here in cosmic-comp. Not sure if other compositor developers like @YaLTeR have any opinions here. (I've been wanting to move this to smithay partly because it would be nice to be able to test my image-copy-capture clients in niri.)

(Also note that cosmic-comp is GPL licensed, while Smithay is MIT license. But the image-copy code there is all written by @Drakulix and me at System76, and I don't think there should be any problem with re-licensing that part for inclusion in Smithay, if basing the Smithay implementation on the cosmic-comp code is useful.)

[glamberson]

Thanks for the feedback, @ids1024. The extensibility requirement aligns with what I need as well. Also I appreciate your swift response as I'm frankly on fire to get this moving.

I've looked at the cosmic-comp implementation and Smithay's existing patterns. Based on your requirements, I'm proposing a callback-based design where:

1. ImageCaptureSource is an opaque handle with a UserDataMap for compositor-specific data
2. Handler callbacks notify the compositor when sources are created, allowing it to store its representation
3. A register_custom_source_manager() method allows compositors to add custom globals (like workspace capture) that create the same ImageCaptureSource type
4. The screencopy handler receives ImageCaptureSource and calls back to the compositor for constraints and capture

  pub trait ImageCaptureSourceHandler {
      fn image_capture_source_state(&mut self) -> &mut ImageCaptureSourceState;

      fn output_source_created(&mut self, source: ImageCaptureSource, output: &Output) {
          source.user_data().insert_if_missing(|| MyOutputData::from(output));
      }
  }

This avoids generic parameters, gives compositors full control over source representation, and allows COSMIC to keep its workspace capture extension.

I have an immediate need for these protocols and am ready to implement. Before I proceed:

1. Does this design direction work?
2. For the screencopy implementation, I'd like to adapt cosmic-comp's screencopy.rs as a foundation. Could you and @Drakulix confirm the re-licensing to MIT is acceptable? I'll include attribution.

Thanks.

[Drakulix]

Re-licensing to MIT is acceptable. Upstreaming this is very much in our interest.

[glamberson]

@Drakulix and @ids1024 , thanks. I've actually already been working on somehting pending your approval of the idea. As soon as it's ready, I'll submit it here as a draft PR so you can see if it'sok with you. Thanks!

[glamberson]

OK I submitted the draft. Please have a look at it! Thanks!

[glamberson]

fixed the CI issue so this should pass now.

[YaLTeR]

I guess one question I have is how tied the design is to Smithay's ForeignToplevelHandle. I mean it makes sense to use it, just in niri we're considering a hand-rolled ext-foreign-toplevel-list impl (it's just easy to tie into our existing wlr-foreign-toplevel-manager code), which means we have our own objects instead of ForeignToplevelHandle.

[glamberson]

@YaLTeR yes, I see now there's still a coupling problem. I wasn't thinking of the toplevel side. Do you have a preferrred approach to address this that I can work on?

[Drakulix]

@YaLTeR yes, I see now there's still a coupling problem. I wasn't thinking of the toplevel side. Do you have a preferrred approach to address this that I can work on?

We also have to take into account, that some compositors might not want to support outputs, but toplevels (or another custom source).

As such, I think you need to split all the source-globals into separate State-objects and associated Handlers. That would make it easy for a compositor, that doesn't want to use e.g. smithay's output abstractions or foreign-toplevel abstractions to simply implement the specific source global itself. (Which is very trivial compared to implementing all of it yourself.)

[glamberson]

@Drakulix That makes sense. I was kind of getthing there already myself but I had't committed to the approach.

So how about:
Core ImageCaptureSource type (just the resource handling)
Separate OutputCaptureSourceState + OutputCaptureSourceHandler for output capture
Separate ToplevelCaptureSourceState + ToplevelCaptureSourceHandler for toplevel capture (using ForeignToplevelHandle)
Compositors with custom needs implement their own source manager (niri's custom toplevel, COSMIC's workspace capture, etc.)

The image_copy_capture side stays the same. It just works with ImageCaptureSource objects regardless of how they were created.

I'll rework the PR along these lines.