l2cap socket's output buffer is full on creation

[andrewzhurov]

I'm trying to send data between a linux machine (Ubuntu 25.04, Bluetooth 5.3) and an Android phone.
l2cap socket gets successfully opened, works from Linux -> Android and Android -> Linux.
Reads from Linux go without any problems.
However, write_all(&buf) on OwnedWriteHalf is stuck on endlessly repeating Poll::Pending.
(guard is ready, but sock::send() errors (would block)).
After quite some time digging, found that output buffer of a channel is full on creation.
Which's I see by these prints added to fn accept()

println!("bytes in input buffer: {:?}", ioctl_read::<c_int>(&fd, libc::TIOCINQ));
println!("bytes in output buffer: {:?}", ioctl_read::<c_int>(&fd, libc::TIOCOUTQ));

// bytes in input buffer: Ok(0)
// bytes in output buffer: Ok(212992)
//                             ^^ it's size is the same

I've tried clearing output buf with no luck.
First tried to resize it to 0 then back to it's current size via this fn, as impl for Socket

    /// Attempts to flush output buffer
    pub fn flush_output_buffer(&self) -> std::io::Result<()> {
        let socket = self.fd.get_ref();
        // Get current output buffer size
        let current_size: i32 = sock::getsockopt(socket, SOL_SOCKET, SO_SNDBUF)?;
        // Set output buf to 0
        sock::setsockopt(socket, SOL_SOCKET, SO_SNDBUF, &0)?;
        // Set output buf to its original size, in hope that it's cleared in the process
        sock::setsockopt(socket, SOL_SOCKET, SO_SNDBUF, &current_size)?;
        Ok(())
    }

After that the output buf is double the size it was before (ok, expected, libc docs say it *2 the passed val, whatever), but what's important - still fully filled.

Also tried TCFLSH in accept() fn

    println!("On socket accept(), TCFLSH: {} ", unsafe {
        libc::ioctl(fd.as_raw_fd(), libc::TCFLSH, libc::TCIOFLUSH)
    });
    println!("last err: {:?}", Error::last_os_error());

// On socket accept(), TCFLSH: -1 
// last err: Os { code: 25, kind: Uncategorized, message: "Inappropriate ioctl for device" }

Soo.. no luck here either.

That's a strange behavior that socket fd gets filled in output buf.
I guess that's something screwed up in my OS, not in bluer and bluez.
Still, did anybody see something like that? Maybe it's not an uncommon issue..
And any possible workaround that comes to mind?

[andrewzhurov]

Some more findings:

1. WouldBlock on send occurs when trying to send a large buf (100kb).
   With a smaller buf (up to ~6k bytes) it works for me.
   Write succeeds, then poll_write is called plenty of times (due to guard saying it's ready, but actual send returning WouldBlock), until it eventually writes again.
   I measured ~2kb/sec read speed on Android, so perhaps it awaits acks from Android before removing data from buf,
   and guard signals that "you can write something now (a small amount, not 6kb).
2. Android gives a huge MTU (65535) and amount of credits. Would be enough to send 100kb.
3. TIOCOUTQ (bytes in buf / filled size) reports strange data.

bluer l2cap: poll_guard_res Ready(Ok(ReadyGuard { async_fd: AsyncFd { inner: Some(OwnedFd { fd: 112, close_on_drop: true }) } }))
bluer l2cap: guard trying io
l2cap fd sock buffer size: Ok(212992)
l2cap fd sock filled size: Ok(212992)               <-- reported as filled fully
bluer l2cap: guard write success Ok(4032)   <-- yet write is no prob (sock::send)
...
bluer l2cap: poll_guard_res Ready(Ok(ReadyGuard { async_fd: AsyncFd { inner: Some(OwnedFd { fd: 112, close_on_drop: true }) } }))
bluer l2cap: guard trying io
l2cap fd sock buffer size: Ok(212992)
l2cap fd sock filled size: Ok(65152)                      < --- reported as not filled fully
bluer l2cap: guard write error (would block) TryIoError(()), continue  <-- yet we can't write
bluer l2cap: poll_guard_res Pending

4. There's considerable more poll_write looping due to WouldBlock with poll_write capping to 4kb than to 400b.
   Yet the receival speed is the same ~2kb/sec.
5. Also to note, that sock::send doesn't do partial writes. Either passed buf gets fully written or it WouldBlock.

If we manage to have partial writes - that would solve the issue. It's meant to work though, isn't it? As sock's poll_send_priv() returns sock::send's output which's the num of bytes written.

Another option seems to be to learn that "actual output buffer size" somehow, and cap buf-to-send to it (as we do with send_mtu).

[surban]

Yes, partial writes should work.

Could you check with a minimum C example if the same problem occurs?

This would help to determine whether this is a bluer or kernel bug.

[andrewzhurov]

Haven't tried with a C example, but did some digging into kernel's side.
From what I gather, it goes like this
l2cap_chan_send
l2cap_segment_le_sdu
*l2cap_create_le_flowctl_pdu
l2cap_skbuff_fromiovec
chan->ops->alloc_skbl \ *l2cap_sock_alloc_skb_cb
^ in loop, allocating pdu-sized chunks, and on one such iteration I suspect this check fails:
(sk_wmem_alloc_get(sk) < READ_ONCE(sk->sk_sndbuf))
Which leads to -EAGAIN in subsequent lines, as timeo is 0, since it's non-blocking.
In that case previously allocated skbs are cleared / full abort, nothing's written. So partial writes are not supported it seems.

I was wrong in stating that increasing SO_SENDBUF doesn't help.
When setting it to 212992 (that's the val of /proc/sys/net/core/wmem_max on my side) (reading it reports double that size, 425984, half is for linux's bookkeeping), I'm able to squeeze through 11k bytes as one SDU / sock::send.
A larger value still gives endless poll_writes with WouldBlock.

Here's a btmon log excerpt:

> LE Connection Request … MTU=65535, MPS=251, Credits=65535
< LE Connection Response … MTU=672, MPS=23, Credits=30, Result=Success

So, Android says Linux can push an SDU of up to 65535 bytes.
Yet practically my machine's send-buf can accommodate up to ~11000.

---

Would be nice to just set SO_SENDBUF so it can fit send_mtu.
But then wmem_max may not allow for that. E.g., my case.

Would be nice to learn how many bytes output buf is able to accommodate, to cap to that.
But then we have send-buf of 212992 fitting less than ~11000, so no clue how that math works.