Alternative blend mode

[pekkavaa]

According to my experiments (writeup) and the angry lion source, the RDP blender equation is asymmetric

The "B" term in (P*A) + (Q*B) seems to get a small (1/64?) bias: https://github.com/ata4/angrylion-rdp-plus/blob/9c8b9ed3e7d7f00dff8bc872ccdd3fba1a3673fc/src/core/n64video/rdp/blender.c#L109

This is why sometimes users might want to try the alternative if alpha precision is an issue. My proposed alternative formula that swaps "P" and "Q" terms

rdpq_mode_filter(FILTER_BILINEAR);
rdpq_set_env_color((color_t){.r=0, .g=0, .b=0, .a=255});
rdpq_mode_combiner(RDPQ_COMBINER1((0, 0, 0,TEX0), (1, TEX0, ENV, 0)));
rdpq_mode_blender(RDPQ_BLENDER((MEMORY_RGB, IN_ALPHA, IN_RGB, INV_MUX_ALPHA)));

For reference, RDPQ_BLENDER_MULTIPLY is RDPQ_BLENDER((IN_RGB, IN_ALPHA, MEMORY_RGB, INV_MUX_ALPHA).

My original theory was that this would make sprites' opaque alpha=255 pixels transparent, but I can't make it happen with a test case. Anyway this is how it looks. The difference is admittedly tiny.

Before (RDPQ_BLENDER_MULTIPLY) and After (above):

[johannesroth-coreproduction]

    rdpq_set_mode_standard(); // for debugging alpha channel
    rdpq_mode_filter(FILTER_BILINEAR);
    rdpq_mode_blender(RDPQ_BLENDER_MULTIPLY_CONST);
    rdpq_set_fog_color(RGBA32(0, 0, 0, 255));
    rdpq_mode_alphacompare(255);

results in (see transparency between black/white bar):

while

rdpq_mode_filter(FILTER_BILINEAR);
rdpq_set_env_color((color_t){.r=0, .g=0, .b=0, .a=255});
rdpq_mode_combiner(RDPQ_COMBINER1((0, 0, 0,TEX0), (1, TEX0, ENV, 0)));
rdpq_mode_blender(RDPQ_BLENDER((MEMORY_RGB, IN_ALPHA, IN_RGB, INV_MUX_ALPHA)));

results in:

[pekkavaa]

Hmm looks like the issue is with RDPQ_BLENDER_MULTIPLY_CONST. If you use it and set the alpha to 255 via rdpq_set_fog_color(RGBA32(0, 0, 0, 255)), a sprite will be still partially transparent.
As a workaround you can disable blending when alpha=255 and enable it again when it goes lower.

Sometimes it would be nice to have it work the other way around: have full opacity but accept you won't be able to fully hide the sprite. For that case there could be another blending formula preset, but I'm not sure of its usefulness.

Something like

#define RDPQ_BLENDER_MULTIPLY_CONST_INVERSE RDPQ_BLENDER((MEMORY_RGB, FOG_ALPHA, IN_RGB, INV_MUX_ALPHA))

where alpha=0 would map to full opacity and alpha=255 to still-tiny-bit-visible. It's easy to add but like said, unclear if it's helpful.