Memory leak when WS connection is lost and attempting retry

[magkal]

When connected to lightstreamer server with this dotnet client (using dotnet7 runtime), every time the client loses connection and attempts to reconnect it results in a severy memory leak that eats up all available memory on the server, and then crashes in a matter of minutes.

I know that this repo is deprecated, given TLCP version is hard coded in the client it's not guaranteed the server i'm connecting to is going to accept the new lib (6.0).

I've tried all versions 5.0.0-5.1.8 and they all suffer from this issue.

Edit:
This could be due to and old version of DotNetty, i've experienced similar issues with low level socket/http/connection libraries that were built for older versions of dotnet, since dotnet5-7 and specifically dotnet 6 had major rewrites to the low level code for performance reasons.
I can see DotNetty 0.7.3 has a fix with compilation targets for dotnet 5 and 6.

[Bravo27]

We are aware that DotNetty in some circumstances is very aggressive in allocating Buffers and tends to keep them once allocated without releasing.
We found that the tuning below help to mitigate the issue.

_// DotNetty tuning
Environment.SetEnvironmentVariable("io.netty.allocator.maxOrder", "1");
Environment.SetEnvironmentVariable("io.netty.allocator.type", "unpooled");_

[magkal]

Hi,
Thanks for the quick reply. I can't seem to get that configuration to work, i've been trying to find references to that setting in the DotnetNetty source-code and I can't seem to find it there.

I've been digging into your lib, in a local copy now, and one thing that seems odd is the SingletonFactory.closeAsync which is only referenced by NettyShutdownHook does not seem to be called when using DisconnectFuture().
Because when calling DisconnectFuture i can see these log rows

      No HTTP Shutdown Hook provided
      No WebSocket Shutdown Hook provided

I tried to work around the netty issue by attempting to disconnect the client when a connection is lost, calling disconnect future and disposing that client and then creating a new fresh instance, but that failed as well, maybe because of NettyShutdownHook not being used it never actually did dispose the netty allocated objects.

[Bravo27]

I am looking into that. Meanwhile, could you let us know the minimum server version you are aiming for?

[magkal]

Thanks, i'm not really sure, i can't find a reliable source of what version of lightstreamer server IG Markets are using, all i can see is that i can connect to them with this nuget version 5.1.8 and that logs using TLCP 2.1.0,
and using the version 6 results in an unsupported version from the server

6 months ago, 5.1.8 didn't work connecting to them, only nuget 5.0.5 worked, so they must have performed atleast 1 upgrade since then.

But if it's important i could try asking them in a support ticket/the forum

[magkal]

I've been running some more tests in a local branch,

• Fixing BaseChannelPoolHandler - Specified cast is not valid. #3
• Upgrading DotNetty to 0.7.5
• Changing compilation target to dotnet 6
• Implementing the NettyShutdownHook in NettyWebSocketProvider

I've not since been able to reproduce the memory leak, running the streaming process for a few days.
It's a bit tricky to find all potential pitfalls because the async code called in DotNetty is not properly awaited by the Lightstreamer library.

[dario-weswit]

So you feel that you have managed to fix the memory leak?
Thank you very much for sharing this.
We will analyze your proposal when Bravo27 is back in the office.
Can you setup a pull request? If the dotnetty upgrade complicates things, you can leave that part outside.

[magkal]

Hi,
Yes i feel that it's fixed. I can draft a pull request next week. I don't see any particular branching strategy or contributing guidelines. Anything you want me to consider other than just sticking to existing project standards?

[dario-weswit]

No; no specific process to follow.
We just need a way to see the details of your effort.
Anyway I have created a branch named "memory-leak-issue", so that you can refer to it.