Implementing API for using WebPd as a normal package + API documentation

[kikohs]

Hi there,
I'm trying to integrate WebPd in https://cables.gl
Is there a way to list all input ports from the wasm file so they can be wrapped visually.
Additionnaly I notice the cool UI from the website, is there a way to see this code? How do you map each widget to each inlet port of webpd?

Thank you

[sebpiq]

Hi @kikohs !

That's great news !

Input ports info is bundled with the wasm file as metadata, which you can read through a "private" API.

Otherwise it is accessible through the compiler by choosing output format dspGraph. For example :

node ./bin/cli.mjs -i mypatch.pd -o tmp/mypatch.graph.json -f dspGraph

Then in the json look for the entry inletCallerSpecs. it will give you the list of open ports in the following format :

{
    [DspGraph.NodeId]: Array<DspGraph.PortletId>
}

So node IDs as key, and an array of inlet IDs as value. Node IDs are generated automatically though, and mapping them to a UI element (button, knob, etc ... ) is not that easy.

There's a function to help extract GUI elements along with input specs (the one used on the website). It can be found here.

The whole code of the web site is here.

---

Basically in order to do a proper integration I guess you would need to be able to use WebPd as a dependency (instead of using the cli). The index with all exports is here, unfortunately undocumented for now.

I suggest you have a look at all these links, dive a little bit into the code and ping me again once done !

Then I'm happy to work with you more closely to help with the integration (we can call and have a chat). I'd need to understand better what are your needs, and on the other hand it would help me to add potentially missing API / documentation, and make this easier for others in the future 😄

[kikohs]

Thanks for the quick response. I'm evaluating options between PD and Faust. PD is node-based so I guess more accessible to users coming from cables.
The goal would be to create audio-visual experiences. The input on cables would probably be the PD patch as a file.

How hard would it be to compile from PD file, load the wasm and expose the ports browser side? Cables is purely browser based.
Thanks

[sebpiq]

It's pretty easy to do. In fact it's what I do in the online compiler which you linked above.

Online compiling of wasm module can be a bit slow, but with WebPd there is also the possibility to compile to JS. Compilation is then really fast, but performance of the sound module is then lower.

I guess you would also need then to integrate error reporting from the compilation to give feedback to the users if compilation failed.

I could write a minimalist example of compiler integration if that would be helpful to you?

[kikohs]

That would be great for sure!! Cables uses the old way with <script> to load libraries (modules work but are less optimized).
My idea would be to compile the patch (maybe we choose wasm or js as an option), and expose all ports in a object.
Then you could set the value of port from Cables by sending a message to the worker.

It would be an uglier version of what you developed with the cool UI but well integrated in Cables.

[sebpiq]

@kikohs Just popping in to mention that I haven't forgotten about this ! I'm working on making it possible for WebPd to be imported by an app, and then the example will be straightforward!

[kikohs]

That's great to hear. Thanks for tackling this feature. Take your time. No rush here

[sebpiq]

Hi @kikohs !

Finally managed to do this !

I setup a little test repo here : https://github.com/sebpiq/WebPd_example-compiler-browser

Please have a look at the code and let me know if it seems clear to you !

The demo takes a little time to load because it downloads and decodes 2 small sound files.

Note that you will have several integration challenges :

• Pd patches are more, often than not, several files : audio samples, so-called abstractions (Pd patch that can be reused in a main patch, like a function), etc ...
• Discoverability of opened ports in the patch. This is something I am working on as several people have asked that feature. It would be good to know what are your requirements there ? For example if you had something like described here : webpd could provide a dictionary from label to nodeId #137 (comment)
  So for each GUI element, the position, the type and an id which allows to send it message through the WebPd API. Would that be enough ?
• Pd messages are a mix of strings and floats, maybe you will need to find how to deal with that too if you don't support the same types.

Maybe the simplest could be to have a quick call once you've looked this over. Please write an email to me if you'd like to do that !

[kikohs]

That's great!!
The code seems clear enough, however I'm not sure how to integrate in Cables without involving the devs.
Currently external libs in Cables are using IIFE (old school) scripts. I managed to dynamically load modules but I'm not sure it is possible to create import map dynamically.

I'll check in more details and come back to you.

Thanks again

[sebpiq]

@kikohs two options :

1. you could for now run WebPd only in JS mode instead of wasm : no dependencies needed then ! Only the webpd-bundle.js file, which is actually good old IIFE.

2. if you really want wasm, you should be able to simply concatenate all these dependencies into a single JS file. Haven't tried it, but I don't see why it wouldn't work !

[kikohs]

thanks, I'll try to the bundled version for now.

[sebpiq]

@kikohs any progress ? I'm doing a lot of work on the library these days, so if you still need help, don't hesitate.

[kikohs]

Sorry for the slow updates.
Basically I went back on my previous Faust wrapper as a test.

For the WebPd library to be useful in Cables, we would need to automatically discover/expose input and output ports (see image with output Faust ports).

Do you think it would be possible?

[sebpiq]

@kikohs sure ! Although it depends on the user to name their controls. If one names their sliders / knobs / ... properly, you can get exactly that.

The data you get is described there :

See : #137

The only thing is then that you need to map yourself a user-friendly port name like the ones in your example to a unfriendly port ID which is automatically assigned by the WebPd compilation. The data linked to above should be fine for that.

I'll send you a new example in a couple of days doing exactly that.