Copy FX: is there a better way

[DedeHai]

There is an open question on how to handle segment / strip copies.
This issue is a reminder for the discussion started here: #5105 (comment)

[willmmiles]

Forwarding from #5105 - I had described three possible "copy" operations:

1. Re-blend the internal "segment contents" from the source segment, essentially just re-rendering it with different pixel coordinates. Implementation would be like calling blendSegment() with the source segment, but overriding some output properties (start coord, grouping, spacing).
2. Blend the "rendered but not blended" output of the source segment, ie. respecting source segment grouping/skipping. Implementation would be like calling blendSegment() for the source segment in to a temporary buffer with start coords set to 0, then calling blendSegment() to blend the contents of that buffer to main buffer.
3. Copy (blit) a block of previously-rendered pixels from one location in the pixel output buffer to another. Useful to capture a blended output.

If I understood @blazoncek 's concern, it was that option (3) violates encapsulation: it requires the FX code to be aware of the rendered pixel buffer (so as to read it). While this is technically possible in the current 0.16 code, I think it's worth noting that that's a result of a particular implementation choice: at one point I'd floated the possibility of inverting the rendering loop (looping over the output pixels, and then for each segment blending that one pixel if needed) which would eliminate _pixels entirely, saving a lot of RAM on constrained devices. That approach would make option (3) impossible (or extremely expensive) to implement. (Sorry I can't find the reference on that suggestion!)

(The alternative RAM-saving architecture would be reimplement the bus API to allow buses to sample out of _pixels instead of requiring them to maintain their own flat device buffers. This would almost certainly require a whole slate of custom drivers though to allow "sampling on demand" with in-the-moment brightness and color correction when building their various DMA outputs buffers...)

[DedeHai]

(Sorry I can't find the reference on that suggestion!)

I remember that but no clue either :)

IMHO the copy FX including the "deep copy" is a useful feature, we just discussed on discord on an extension to adding rotation and zoom though mostly useful in 2D and a proof of concet is already done https://discord.com/channels/473448917040758787/706623245935444088/1449833961063977222

this would mean we have to keep the global buffer. One option would be to PR a change to NPB to accept external buffers if that is feasible.

[blazoncek]

(3) violates encapsulation

You nailed it. There's very little encapsulation in WLED which is inherently bad.

(1) Effect has (or should have) no idea on which segment it runs on (apart from global SEGMENT which is used for code reduction purposes). I loathe the idea of the opposite. I.e. WS2812FX needs a bit "privacy". Unfortunately there's a lot of legacy and sadly I see new stuff reuse it.

[willmmiles]

we just discussed on discord on an extension to adding rotation and zoom though mostly useful in 2D and a proof of concet is already done

We are thiiiiis close to implementing all the features of a 2D graphics compositor. Stretch and scale, alpha blending, (formally defined) chroma key, and render-to-texture and we'll be all set.

this would mean we have to keep the global buffer.

Not (strictly) necessarily -- a "render to texture" approach would let us allocate buffers only as they're needed. With rotation/scaling operations, this will be coming for us anyways; there's value in being able to apply operations in an arbitrary order. Eg: first render a gif, scaled 2x; then overlay some text; then rotate that rendered pair n degrees, where n increases at 10 degrees per second. Finally, blend the rotated buffer with an overlay of an AR equalizer graph at the bottom. The result being a spinning graphic and message, with a fixed overlay.

(And then for the next trick, have the same output rendered with the opposite rotation on the second panel, so it spins the other way.)

The point being that the most flexible solution is to allow building up a render graph that can produce blended output at key points where it can in turn be sampled by a "copy-transform" operation for the next stage. Why stop at just one layering renderer when you can have many?

From a UI perspective, consider it as multiple "stacks" of segments, each stack with a name, and the "copy" operation can source from the output of any of the stacks.

One option would be to PR a change to NPB to accept external buffers if that is feasible.

Honestly, I still think our ideal interface would be one step further: a bus abstraction that is expected to call a function to fetch the pixel color from our data structures, allowing us to "just in time" calculate the color without having to render it to flattened buffers at all. That's probably a bridge too far, though...

In the end I'd like to do a more thorough survey of Buses before pushing for any particular solution. I know how the hardware underlying NPB needs to be fed, but I haven't looked through the Hub75 and other display controller interfaces. (I updated that LCD panel usermod recently; it could also be an output target!)

(1) Effect has (or should have) no idea on which segment it runs on (apart from global SEGMENT which is used for code reduction purposes). I loathe the idea of the opposite. I.e. WS2812FX needs a bit "privacy". Unfortunately there's a lot of legacy and sadly I see new stuff reuse it.

I do think that the "copy" feature (in all three forms I mentioned above) is a useful one, but I can see the point in that such "meta-operations" aren't necessarily a good fit for a formalism of "effect". If I may, it seems like the crux of the disagreement here is in that defintion. Is an "effect":

a) a piece of code that produces de novo color pixel data from some algorithm?
or
b) any piece of code that can fill a pixel buffer?

We've been floating some interesting corner cases recently. The GIF player is one step in: the pixel colors for each frame are sampled from a file source, then blended as a layer in the output. The next step down the line is treating network inputs as "effects" in the same way: making an effect of "sample from the network buffer" lets it be placed at some point in the layer blending, allowing complex compositions (ie. expand the network data with 2x grouping, and then overlay a "sparkle" effect).

With that in mind, it might very well be that "copy" or "sample and transform" operations are better treated as a different class of Segment instead of dragging around all the effect internals, sharing the same blending code but with different UI and/or properties.

[blazoncek]

it might very well be that "copy" or "sample and transform" operations are better treated as a different class of Segment

This!

[willmmiles]

it might very well be that "copy" or "sample and transform" operations are better treated as a different class of Segment

This!

Maybe the time has come to do some terminology changes too - I've always found "Segment" to be confusing as it sort of conflates "physical segment" with "block of pixels to render to". Maybe we want some class hierarchy like:

class Layer {};   // Common portion for describing a layer to render.  start pixel, sizes, grouping, mirroring, etc.
class EffectLayer : public Layer {};   // Effect-specific subclass.  Adds effect id and properties.  Equivalent to current Segment.
class SamplerLayer : public Layer{};   // Samples pixel data from some other source.   Adds source information and affine transformation properties 

It'd be cool to see Sampler-type operations called out with different background colors in the Segment list in the UI. Also might be possible to add Effect subclasses, eg. ParticleEffectLayer with different members. #5108 was setting up to allow effect constructors; that could be generalized.

[blazoncek]

I would recommend to introduce changes in such a way as to minimize code changes in effects (there will be a lot of them I expect).

Effect classes (including the canvas agnostic effects) will limp unless entire "servicing" is changed.

Just for the reference, I invented current segment logic in the course of 4 months working on it daily for about 6 hours per day. I do not intend to repeat that, not even a fraction of that. 😄
See: d9f2c2b
And: 64fd207

P.S. On a side note: Seeing what @Brandon502 did with rotation and zoom I am now 100% convinced that such operations as "segment copy", "segment zoom", "segment rotation", "1D expansion", etc. are best dealt, not within so called effects, but as a Segment operations. Requiring separate UI elements which are currently inexistent.

[DedeHai]

I like the approach of moving it all to Segments. The UI update should not be too complicated, make the additional functions a collapsible extension, collapsed by default to not clutter the UI on mobile.
A bit more advanced would be add in PS interactions, like the demo I made for scrolling text or a particle FX interacting with a GIF/movie as demonstrated by @Brandon502 but that would probably better be suited as a usermod, I don't see an easy way to make that happen on a segment level.

[blazoncek]

One thing popped into my mind is about 1D expansion. If it is to be moved into segment blending part, we'd need to have information (not from metadata but somewhere more easily accessible) if effect running on segment is 1D or 2D. Or. Force strict map1d2d = 0; for 2D effects (i.e. no mapping).