Results of WebRTC APM Processing

[momahler]

Hello,

I think this project is very useful due to the WebRTC APN Processing.
My application is streaming audio buffers from the mic and sends them via the network, if voice is detected.
I would love to use the filtered audio with reduced background noise, but unfortunately, I cannot access it until the speech ended.

Do you think it might be possible to make this function

vadManager?.processAudioData(withBuffer: monoralData, count: frameLength)

return the webrtc-apn-processed result regardless of the voice detection result ?

Thankful greetings

[helloooideeeeea]

@momahler Thank you for your question!

I would love to use the filtered audio with reduced background noise, but unfortunately, I cannot access it until the speech ended.

In other words, you're hoping to receive the APM-processed (filtered) audio data via a callback as soon as voice is detected, correct?

For example, something like:

extension ViewController: VADDelegate {
    func voiceStarted() {}

    func voiceDidContinue(withFilteredData data: Data) {
        // This Data should be interpreted as an array of Float values.
    }

    func voiceEnded(withWavData wavData: Data) {}
}

Yes, this is possible, but it will require some changes to the implementation.

The core logic of this library depends on the underlying C++ library:
👉 https://github.com/helloooideeeeea/RealTimeCutVADCXXLibrary

This Swift library is essentially a wrapper around that C++ code.
To pass APM-processed audio buffers to Swift in real-time, you can modify the C++ side — for example, around this line:

https://github.com/helloooideeeeea/RealTimeCutVADCXXLibrary/blob/main/RealTimeCutVADCXXLibrary/algorithm.cpp#L49

You could update the code there to call a function pointer (callback) and pass audio->buffer directly.
Then on the Swift side, you can receive the filtered audio in voiceDidContinue(withFilteredData:) as Data, and convert it to a float array.

So to summarize:

It’s technically possible,

You need to add a function pointer callback in C++ and expose it through the Swift wrapper,

And forward the APM-processed audio buffer on every frame while voice is active.

Let me know if you’d like help writing the callback mechanism or bridging it to Swift!

[helloooideeeeea]

If you become a GitHub Sponsor, I’ll happily make the change right away 😄 (I’ve always wanted to say “I have a GitHub sponsor” just to brag a little!)
That said, I haven’t actually set up GitHub Sponsors yet — but I just might!

[momahler]

Thank you for your quick and precise response : )
Fist of all, yes, the callback you proposed is exactly what I thought of, under the condition that each and every sample which is declared as voice can be retreived via voiceDidContinue method.

I'm quite experienced in Swift, but not in C++ & ObjC, so I'd actually prefer the github sponsor option you mentioned 😊.
I would like to compare the WebRTC APM Noise suppression results to other noise suppression solutions like RNNNoise.
I'd be willing to do a one time donation (I'm a poor man) to motivate you to do the proposed change.
In general I think that this change would not just be an individual solution for me, but obviously essential for the majority of potential users of your repo. At least if they stream the detected speech in realtime while audio is still recording, and that's the main use case of your wonderful code in my opinion.

If you set up GitHub Sponsors and I can afford it, I will definitely be willing to support.

[helloooideeeeea]

Hi! @momahler

Thanks for your message!

I’ve decided to add both RNNoise mode and WebRTC APM mode to the library.
Since I had previously integrated RNNoise before switching to Silero, I can reintroduce it quite quickly — so no worries there.
I plan to complete the implementation by tomorrow.

Thanks again for the motivation!

[helloooideeeeea]

Hi @momahler

I’ve added a new callback that allows you to receive denoised audio in real time 🎉
It’s available from version 1.0.9:

func voiceDidContinue(withPCMFloat pcmFloatData: Data!)

The pcmFloatData is a Data object containing raw PCM audio samples only—there is no metadata or header as in a typical WAV file.
You'll be handling it as a byte array, but the actual content is 16 kHz, 32-bit float, mono PCM (raw audio points).
I hope your transmission protocol supports this format!
If you need a different sample rate or bit depth, feel free to let me know.

🔧 The included test project saves the PCM stream to a .pcm file, so you can easily verify the output.

For example, you can convert the saved file to WAV using the sox command:

sox -t raw -r 16000 -c 1 -e float -b 32 your_output.pcm your_output.wav

Just like voiceStarted() and voiceEnded(), it’s highly recommended to handle this callback on a separate thread.
Blocking the callback can interfere with frame processing and may cause dropped audio or latency.

(For your information, I’m currently applying for GitHub Sponsors—I didn’t realize it required approval 😅)

As for switching the denoising backend (e.g. to RNNoise), I’m planning to work on that sometime next week.

[momahler]

Hey hey,

thank you very very much, it's working. The format 16 kHz, 32-bit float, mono PCM is exactly what I needed.
I will surely observe the status of your ability to receive donations.

I have 3 little questions:

1. Regarding the new callback voiceDidContinue(...):
   The first data chunk after every no-voice-period is bigger than the following live incoming chunks.
   It doesn't matter if it's after a very short or very long speaking pause, or if it's at the start or in between - that first data chunk is always much bigger. I noticed that I only need to use a suffix of ca. 16384 bytes of that big chunk to have the actual spoken voice. That's just 8 times more than the further default chunk size of 2048 bytes. Can you explain to me, what is the reason for the big size of the first data chunk ?

2. Another thing, I wanted to ask you:
   Did you ever test or verify, if the vad accuracy by Silero gets better when the audio is "cleaned" before with webrtc audio-processing-module ?
   I'm asking, because for my human ears, when listening to the audio returned by voiceDidContinue(..), I cannot confirm that background sounds seem to be suppressed. Maybe it's so subtile that it helps vad and speech recognition software to better process the voice, but I have problems to hear any significant background noise suppression effect.

3. I noticed that the audio data in the new callback method is in a different format when the audio is recorded with airpods.
   When I playback the audio data, it is extremely pitched when it was recorded with airpods. I could solve that easily on my side, but I wanted to give you that feedback since I had the impression that the design ideeeea was to return the same format regardless of the origin of the AVPCMBuffer, iphone mic or airpod mic.

It will be absolutely interesting to see how RNNoise performs on a mobile device. Your plan to (re-)implement that was a big surprise and I am curious how that works out.

Many Greetings

[helloooideeeeea]

@momahler

Thank you again for your detailed observations and questions! Let me address each point:

1. Why is the first data chunk so large?

The reason the first chunk of voiceDidContinue(...) is large is because at the moment voice is detected, that moment is already considered the past—so the buffer contains some pre-voice frames to make sure we don’t miss the beginning of speech.

This pre-buffer is collected and then prepended to the first audio chunk.

If we didn’t include this, the very start of the user’s utterance might be cut off.

Also, you're right: each default chunk is 2048 bytes (which equals 512 samples if it's 32-bit float). Silero processes 512-sample frames (which is 32ms at 16kHz), so output 1 VAD per 512 samples.

Here is the exact implementation that handles appending the pre-buffer:

👉 https://github.com/helloooideeeeea/RealTimeCutVADCXXLibrary/blob/main/RealTimeCutVADCXXLibrary/algorithm.cpp#L49-L53

And the maximum buffer size can be calculated as:

(512 * 4 * 10) * 4 = 81,920 bytes

= approx. 1.28 seconds of audio.
The constants 10 and 4 are hardcoded in my implementation:

👉 https://github.com/helloooideeeeea/RealTimeCutVADCXXLibrary/blob/main/RealTimeCutVADCXXLibrary/algorithm.cpp#L19-L22

2. About WebRTC APM denoising improving Silero VAD accuracy

To be honest, I haven’t tested or verified this.

In fact, I’m not even confident whether the denoising is significantly audible.
The main reason I introduced APM was auto gain control—not noise suppression.

While I haven’t run formal benchmarks, I personally feel that APM’s gain control is clearly active and working.
As for noise suppression, even I can’t confidently hear the difference 😅

3. AirPods audio sounds extremely pitched

Thanks a lot for the feedback—this was unexpected!

I honestly wasn’t aware of that behavior.
Do you happen to know what sample rate AirPods use for microphone input?

The library supports input sample rates of 8 kHz, 16 kHz, 24 kHz, and 48 kHz, so in theory, it should work fine with any of those.

But if AirPods are using something outside of those, or the actual buffer data is inconsistent with the declared format, that might explain the pitch shift.

4. About RNNoise

I’m currently considering this processing pipeline:

APM (Auto Gain Control + Sample Rate Conversion + Noise Suppression?)  
→ RNNoise (Denoising)  
→ Silero VAD (VAD detection only)

Note: While RNNoise can also do VAD, Silero v5 has superior accuracy based on my experience, so I’ll stick with Silero for final VAD decisions.

Let me know if you have any thoughts or concerns on this pipeline.

Thanks again and looking forward to hearing your feedback!

[helloooideeeeea]

Hi ! @momahler

Sorry for the delay—I've been a bit tired lately. I finally added a GitHub Sponsors button! If you'd like to support my work, I'd really appreciate it! 🥰

Here's the link: https://github.com/sponsors/helloooideeeeea