out of buffers for RTP packets, dropping

[ThomasSevestre]

I use latest master branch with SBC application and RTP relay enabled.

Sometimes, in the middle of a call, an RTP stream stops being processed and I get "out of buffers for RTP packets, dropping (stream [0xpppppppppp])" errors.
The incoming RTP stream is OK and contains audio in the input capture. The outgoing RTP stream looks OK but contains no sound on the output capture.
The RTP stream never recover and the call has to be hangup by users.

Do you have any idea what could cause these errors ?

[hecko]

Root Cause Identified and Fix Proposed

I've analyzed this issue and confirmed that this bug can definitely occur with the current codebase. I've created a fix in branch fix/issue92-rtp-buffer-exhaustion.

Root Cause

In SBC with RTP relay enabled, the code has logic to either relay packets or buffer them for local processing. However, in pure relay mode (without transcoding), buffered packets are never consumed because there's no media processing thread calling receive().

The problematic flow:

1. Each RTP stream has a fixed pool of 32 packet buffers (MAX_PACKETS = 32)
2. When a packet arrives that doesn't match relay criteria (based on payload type), it falls through to buffering in receive_buf
3. In pure relay mode (active==false), no thread consumes from receive_buf
4. After 32 non-relayable packets accumulate, the pool is exhausted
5. New packets fail to allocate → "out of buffers for RTP packets, dropping"
6. The error persists until call ends because buffers are never freed

Why "outgoing stream looks OK but contains no sound":

• B2B/SBC has TWO RTP streams (A-leg and B-leg) that relay to each other
• Only ONE direction experiences buffer exhaustion (asymmetric failure)
• The other direction continues relaying, so RTP packets flow but contain no audio
• The affected stream drops all incoming packets, breaking one direction

Packets That Can Trigger This Issue

• Unexpected payload types not in negotiated SDP
• Comfort Noise (CN, PT 13) when not explicitly negotiated
• Redundancy/FEC packets (RED, FEC) sent without negotiation
• Old payload types during SDP renegotiation (timing race condition)
• Dynamic PT mismatches between call legs
• telephone-event in certain configurations

The Fix

The fix in core/AmRtpStream.cpp adds explicit logic to distinguish between:

• Pure relay mode (active==false): Drop non-relayable packets immediately instead of buffering
• Transcoding mode (active==true): Preserve current behavior - buffer packets for processing

This prevents buffer pool exhaustion while maintaining backward compatibility.

Changes:

• Added check in AmRtpStream::bufferPacket() to drop non-relayable packets when in pure relay mode
• Added comprehensive comments explaining relay criteria and failure scenarios
• Minimal, surgical change that only affects the problematic code path

Testing Request

@ThomasSevestre can you test this fix in an SBC environment where this issue has been observed? The fix is available in branch: fix/issue92-rtp-buffer-exhaustion

Test scenarios that should be validated:

1. SBC with pure RTP relay under normal conditions
2. SBC with codec changes mid-call (re-INVITE scenarios)
3. Calls with comfort noise packets
4. Calls where endpoints send unexpected payload types
5. Long-duration calls to verify no buffer leaks
6. SBC with transcoding enabled (to ensure no regression)

Please report back if this resolves the issue or if you observe any unexpected behavior.

Branch: https://github.com/sems-server/sems/tree/fix/issue92-rtp-buffer-exhaustion

[ThomasSevestre]

@hecko, thanks for the fix.

Unfortunatly I do not have access to a SEMS server anymore.

I used the following patch for years (not fixing the root cause but effectivly preventing it to have effect on end user)

void AmRtpStream::recvPacket(int fd)
{
  if(fd == l_rtcp_sd){
    recvRtcpPacket();
    return;
  }

  AmRtpPacket* p = mem.newPacket();
  if (!p) p = reuseBufferedPacket();

  ///////////////////////
  // BEGIN PATCH
  if (!p) {
    INFO("out of buffers for RTP packets, clearing buffer (stream [%p])\n", this);
    receive_mut.lock();
    mem.clear();
    receive_buf.clear();
    while (!rtp_ev_qu.empty())
      rtp_ev_qu.pop();
    receive_mut.unlock();
  
    p = mem.newPacket();;
  }

  if (!p) {
    INFO("out of buffers for RTP packets, dropping (stream [%p])\n", this);
    // drop received data
    AmRtpPacket dummy;
    dummy.recv(l_sd);

    return;
  }
  // END PATCH
  ///////////////////////
  
  
  if(p->recv(l_sd) > 0){
    int parse_res = 0;

If you managed to reproduce the issue, you might have confirmation by searching for "out of buffers for RTP packets" logs.

Best regards,
Thomas