Generating Synthetic Defect Data with Decals

[gustbogaert]

What I want to achieve

I am using this project to generate synthetic defect data, the defects are decals and in order to use this data i would need a mask showing the defects (decals)

1st Attempt

The first thing I have tried is assigning a label to the decal objects in my scene and using the segmentation labeler. However, the decal objects do not seem to occur in the the resulting instance segmentation results.

2nd Attempt

A second approach I have tried is to create my own custom decal labeler (CameraLabeler class), then i have created a custom perception camera channel (CameraChannel<float4> class) which uses a custom shader. When i use the depth shader i found in this repository, everything seems to work.
I have then written my own shader (see codeblock) which detects the decals based on a difference in roughness/smoothness. This shader works fine in the editor when I assign it to a custom pass volume. However, i cannot get the camera channel to work with this shader..

The problem is that when i use the shader as shown below, it results in an error about a missing render pass 14:

After a long search i figured out this could be due to the light mode that is not set to "SRPDefaultUnlit" (I don't understand why this would be needed). However, when I set the light mode (Tags { "LightMode" = "SRPDefaultUnlit" }) Unity crashes:

My assumption is that I am mixing up the different pipelines (HDRP and URP). I have tried many different approaches to access the smoothness but have not gotten it to work.

the shader

Shader "FullScreen/defect_mask_shader"
{
    SubShader
    {
        Pass
        {
            Name "Roughness"
            //Tags { "LightMode" = "SRPDefaultUnlit" }  //comment: when i add this line unity crashes
            ZWrite On
            ZTest Always
            Cull Off
        
            HLSLPROGRAM 

            #pragma vertex Vert
            #pragma fragment FullScreenPass
            
            #include "Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/RenderPass/CustomPass/CustomPassCommon.hlsl"
            #include "Packages/com.unity.render-pipelines.high-definition/Runtime/Material/NormalBuffer.hlsl"
            float4 SampleBuffer(PositionInputs posInput)
            {
                NormalData normalData;
                DecodeFromNormalBuffer(posInput.positionSS.xy, normalData);
        
                return float4(normalData.perceptualRoughness, normalData.perceptualRoughness, normalData.perceptualRoughness, 1);
            }
            
            float4 FullScreenPass(Varyings varyings) : SV_Target
            {
                UNITY_SETUP_STEREO_EYE_INDEX_POST_VERTEX(varyings);
                float depth = LoadCameraDepth(varyings.positionCS.xy);
                PositionInputs posInput = GetPositionInput(varyings.positionCS.xy, _ScreenSize.zw, depth, UNITY_MATRIX_I_VP, UNITY_MATRIX_V);
            
                return SampleBuffer(posInput);
            }

            ENDHLSL
        }
    }
    Fallback Off
}

3d Attempt (if I am not mistaking)

After some digging in the codebase I figured out that the way the instance segmentation labeler filters out all unlabeled objects is by comparing the alpha value of the _MainTex buffer to a threshold. I have however not found (yet) where this alpha value is altered. If I could alter the alpha values of the _MainTex buffer for the decals in this specific camera channel then I could just borrow the shader code from the instance segmentation labeler.
However, this attempt is purely hypothetical so any suggestions here are more then welcome

My Request

• Guidance on how I can get the roughness/smoothness data from my scene objects and decals in my custom shader
• Guidance on how the alpha channel is adjusted in the instance segmentation labeler.
• New suggestions or ideas

Any help or suggestions are greatly appreciated!

[chris-rgr]

Which Render Pipeline are you using? If you use HDRP, you should probably look at AOVs with a custom labeler. There already is a custom pass utility for this, but I haven't tried it yet myself, so I can't give any other pointers there.