Move to async functions

[SolarLiner]

It turns out that many platforms are actually async (using callbacks as continuations, or blocking on asynchronous I/O operations) and it might be something to expose to the use through making the API async.

It might be needed for web support, where interacting with Web Audio is explicitely async and has caused problems in the past [Citation Needed]

[AldaronLau]

For the wavy crate, I designed the API to have Speakers and Microphone types which each have async functions to get the next synchronous "sink" (audio buffer that can only be written) or synchronous "stream" (audio buffer that can only be read). Everything was designed to run on one thread, which in more complicated projects has caused issues with latency, due to runtime scheduling. In the next iteration, I'm redoing it so that the library runs its own separate audio thread on Linux/ALSA (using a minimal executor like pasts or pollster), and exposes spawning for realtime-safe async tasks.

I think audio is really a perfect use-case for "async iteration" over sinks and streams. When I started my original rewrite of wavy, I ended up also wanting GATs (before they were stable), since it turned out to be a mess of unsafe without GATs, so I recently tried to put together async iteration and GATs in a new crate to be used by the next version of wavy https://docs.rs/event_iterator/latest/event_iterator/trait.EventIterator.html. The idea is that a "microphone" struct owns a few buffers that get lent on iteration, and so does the "speaker" struct. Once next().await is called again the lifetime ends, and the speaker or microphone can use the written audio or write new audio during the .await. Additionally, one of the things I implemented is async iteration over new audio playback/capture devices, which I think is a neat/useful feature that some DAWs lack - would normally need to refresh/restart to check the list of devices again.

It's not completely necessary to use GATs or unsafe either, Arc<Mutex<AudioBuffer>> in both the stream/sink and microphone/speaker would work too. And then the Sink and Stream traits in futures are another potential option for API design, or even avoiding traits altogether (as the current version of wavy on crates.io does, with record() and play() async fns).

[SolarLiner]

Hm, some thoughts based on that:

If I understand correctly, you propose sending the audio data from the audio thread to be processed in a loop, through async channels. However, audio processing happens in a tight loop, tight enough that real-time programming constraints should be followed. This means that we can't spend time waiting for another thread to do the work for us. The complexity of audio programming comes from that precise constraint (it has a bunch of other repercussions in how you handle communicating with your audio engine).

There is also the constraint that you should not allocate, or take locks, or anything that might cause your OS to pre-emptively pause your thread, because that means your audio processing just does not happen in time.

To me this is one of the places where async is not warranted for the above reasons. I was thinking that async would be for device enumeration and configuration management, not for the actual audio stream management.

Traditionally, applications that do not require low latency or real time interaction use a push-based mode of working, where audio buffers are added to a buffer, which is read in tim by the audio callback. This simplifies things (it does not necessarily require you follow real-time programming practices), but as you can imagine this means you need to have quite a bit of audio data as a buffer in case you suddenly can't generate any for a period of time (ie. your audio thread gets descheduled, you lost connection to the file you're playing, etc.)

For games, or audio applications though, low latency is a primary requirement, therefore that method cannot apply. I guess wavy can still be a library for high-latency, simple audio processing, but it won't be applicable to the latter category of applications (this is where I want to focus with interflow).

Anyway, thanks for the write up, that is still useful in food for thoughts over the API of interflow!

[AldaronLau]

If I understand correctly, you propose sending the audio data from the audio thread to be processed in a loop, through async channels. However, audio processing happens in a tight loop, tight enough that real-time programming constraints should be followed. This means that we can't spend time waiting for another thread to do the work for us. The complexity of audio programming comes from that precise constraint (it has a bunch of other repercussions in how you handle communicating with your audio engine).

No, the design isn't sending audio data from the audio thread to process. All the processing / synthesis / effects that have realtime requirements happen in async tasks running on the audio thread. The idea is that you communicate to it with commands over a channels such as "start playing this sample", which don't have the same requirements, or sending audio out of the audio thread to write to disk (non-latency sensitive), where the design avoids blocking on the audio thread / async runtime, but may on main async runtime.

There is also the constraint that you should not allocate, or take locks, or anything that might cause your OS to pre-emptively pause your thread, because that means your audio processing just does not happen in time.

Yeah, this is something I have in mind - this is why the programs will need to have a separate realtime-safe thread from the one that isn't.

To me this is one of the places where async is not warranted for the above reasons. I was thinking that async would be for device enumeration and configuration management, not for the actual audio stream management.

For games, or audio applications though, low latency is a primary requirement, therefore that method cannot apply. I guess wavy can still be a library for high-latency, simple audio processing, but it won't be applicable to the latter category of applications (this is where I want to focus with interflow).

I still think wavy even in its current state proves that async can work for both audio stream management and device enumeration, while still being low-latency (as this was my primary goal in its design), you just need to keep in mind all the same audio programming rules as always whether it's async or not. Now, as to whether the fully async design or partial async/sync is the best design for users of either library? I don't know, but I think it's worth the experiment.