Renderlayer should not error when collection is under the group and overrides have setup

[moonyuet]

Changelog Description

This PR is to make sure renderlayer does not result any error whe collection is under the group and overrides have been created.
Fixes: #195

Additional review information

n/a

Testing notes:

1. Create a renderlayer
2. Have the collection with absolute overrides and parented it to the group under renderlayer
3. Have the collection with absolute override under renderlayer(without group).
4. Open Publisher
5. Publish
6. Should not error any validator(except the validation error) and published successfully to farm

[Copilot]

Pull request overview

This PR addresses render setup override collection in Maya so that overrides nested under RenderLayer Groups are included when resolving attribute overrides, preventing validator errors during publishing (fixes #195).

Changes:

• Replace usage of maya.app.renderSetup.model.utils.getOverridesRecursive() with a custom override traversal that also accounts for render layer groups.
• Add iter_layer_overrides() helper to traverse overrides in a render layer hierarchy including groups.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[BigRoy]

This actually sounds like there could be a source of truth here - definitely something to investigate. I have a gut feeling that if this is the case, that the proposed code change here doesn't technically fully accurately resolve it. Because technically it's also the case that overrides are ordered, and I believe groups can be ordered in-between children overrides directly - as such, stacking groups at the end of current children may technically be wrong too.

I'm also, still very confused why getOverridesRecursive would skip groups in the first place 🤷‍♂️

[moonyuet]

I got an error when the code includes queue.extend(layer.getGroups()). Removing the line would end up the same result with getOverridesRecursive.

from maya.app.renderSetup.model import renderSetup
from maya.app.renderSetup.model.override import Override


def iter_layer_overrides(layer):
    queue = [layer]
    
    # layer.getChildren() does not return groups even though
    # getChildren on groups or collections does return them
    
    for obj in queue:
        if isinstance(obj, Override):
            yield obj
        else:
            queue.extend(obj.getChildren())


rs = renderSetup.instance()
for override in iter_layer_overrides(rs.getVisibleRenderLayer()):
    print("found override:", override.name())
    print("Finished...")
import maya.app.renderSetup.model.utils as utils
for override in list(utils.getOverridesRecursive(rs.getVisibleRenderLayer())):
    print("found override:", override.name())

# found override: absOverride2
# Finished...
# found override: absOverride2

[moonyuet]

If we fixed the layer.getGroups it should be also getting the overrides

from maya.app.renderSetup.model import renderSetup
from maya.app.renderSetup.model.override import Override


def iter_layer_overrides(layer):
    queue = [layer]
    
    # layer.getChildren() does not return groups even though
    # getChildren on groups or collections does return them
    
    for obj in queue:
        if isinstance(obj, Override):
            yield obj
        else:
            queue.extend(obj.getChildren())


rs = renderSetup.instance()
for override in iter_layer_overrides(rs.getVisibleRenderLayer()):
    print("found override:", override.name())
    print("Finished...")
import maya.app.renderSetup.model.utils as utils
for override in list(utils.getOverridesRecursive(rs)):
    print("found override:", override.name())

found override: absOverride2
Finished...
found override: absOverride2
from maya.app.renderSetup.model import renderSetup
from maya.app.renderSetup.model.override import Override


def iter_layer_overrides(layer):
    queue = [layer]
    queue.extend(layer.getGroups() if hasattr(layer, "getGroups") else [])
    # layer.getChildren() does not return groups even though
    # getChildren on groups or collections does return them
    
    for obj in queue:
        if isinstance(obj, Override):
            yield obj
        else:
            queue.extend(obj.getChildren())


rs = renderSetup.instance()
for override in iter_layer_overrides(rs.getVisibleRenderLayer()):
    print("found override:", override.name())
    print("Finished...")
import maya.app.renderSetup.model.utils as utils
for override in list(utils.getOverridesRecursive(rs.getVisibleRenderLayer())):
    print("found override:", override.name())

# found override: absOverride2
# Finished...
# found override: absOverride1
# Finished...
# found override: absOverride2

[moonyuet]

@BigRoy I tested with some edge cases with the code snippet below and come up with 41907a2

from maya.app.renderSetup.model import renderSetup
from maya.app.renderSetup.model.override import Override


def iter_layer_overrides(layer):
    queue = [layer]
    # layer.getChildren() does not return groups even though
    # getChildren on groups or collections does return them
    if hasattr(layer, "getGroups"):
        for layer_group in layer.getGroups():
            if layer_group.isEnabled():
                queue.append(layer_group)

    for obj in queue:
        if isinstance(obj, Override):
            yield obj
        else:
            if hasattr(obj, "isRenderable") and not obj.isRenderable():
                # Skip disabled groups/collections as their overrides are not
                # active
                continue
            if hasattr(obj, "isEnabled") and not obj.isEnabled():
                continue
            queue.extend(obj.getChildren())
rs = renderSetup.instance()

for override in iter_layer_overrides(rs.getVisibleRenderLayer()):
    
    print("found override:", override.name())
    print("Finished...")
import maya.app.renderSetup.model.utils as utils
for override in list(utils.getOverridesRecursive(rs.getVisibleRenderLayer())):
    print("found override:", override.name())
# found override: absOverride2
# Finished...
# found override: absOverride1
# Finished...
# found override: absOverride3
# Finished...

# for the iteration from list(utils.getOverridesRecursive(rs.getVisibleRenderLayer())
# found override: absOverride2
# found override: absOverride3

This is the sample scene I tested.
test_renderlayer_scene.zip