Needs more than two gigabyte RAM if running for two days

[arch-user-france1]

Nyx seems to have a memory leak. If it runs for a long time it will consume my whole swap. This time it ran 2 days only and needs 3GB swap and 1GB RAM of and won't close until the swap is freed or you press ctrl+c multiple times.

[arch-user-france1]

It seems that it's freezing computers that don't have any free RAM anymore. Nyx caused a totalfreeze the day before yesterday.

[InsaneSplash]

After running it for an hour it chomped through a large portion of my free memory.

[terminaldweller]

it appears to be a stem issue rather than a nyx issue.

[atagar]

it appears to be a stem issue rather than a nyx issue.

What made you conclude that? Thus far there doesn't appear to be much actionable information on this ticket.

[terminaldweller]

I ran a memory profiler on nyx to try to figure out who's hugging all the memory.
I ran memray run --live-remote --native run.py and this is what I have so far:
This is run.py:

#!/usr/bin/env python3

import nyx.starter
import sys


def main():
    try:
        nyx.starter.main()
    except ImportError as exc:
        print("Unable to start nyx: %s" % exc)
        sys.exit(1)


if __name__ == "__main__":
    main()

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.1 LTS"
NAME="Ubuntu"
VERSION_ID="22.04"
VERSION="22.04.1 LTS (Jammy Jellyfish)"
VERSION_CODENAME=jammy
ID=ubuntu
ID_LIKE=debian
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
UBUNTU_CODENAME=jammy

$ python3 --version
Python 3.10.6

stem comes from whatever version apt is pulling in for nyx on ubuntu 22.04(its stem 1.8.0). The nyx code I used was against latest commit on the master branch (commit dcaddf2).

[arch-user-france1]

So someone would have to open an issue there instead. Will anyone do it?
Certainly I could, too... But I don't really want to.
It might as well still be a problem in nyx, though. Keep in mind. Maybe nyx re-creates some class from stem and does not delete it afterwads. I am not sure, because I do not know what this profiler exactly displays.

[atagar]

I ran a memory profiler on nyx to try to figure out who's hugging all the memory.

That's a neat visualization. Thanks Farzad!

Unfortunately it doesn't narrow down the haystack much. It indicates that Nyx makes copious get_info and get_conf calls which is expected.

Memory issues are particularly thorny to track down. My first thought is 'why now?', by which I mean that Nyx and Stem were released in 2019. If I still worked at Tor I'd start by checking with the wider community to see how prevalent this issue is.

• If it's a highly prevalent problem (ie. most Nyx users hit it) then I'd start by checking the version of Tor of those with the problem verses those without to see if this regression came from a new Tor release.

• If it's not prevalent (ie. this only hits a few people) then this will be a lot harder to narrow down. The initial troubleshooting steps would be to learn more about the use case and get a '--trace' dump.

So someone would have to open an issue there instead. Will anyone do it?

Honestly it doesn't matter. I'm the author of both Stem and Nyx. I left Tor a couple years ago whereupon Georg (gk at torproject dot org) took over but he doesn't monitor GitHub. You'll need to email him if you'd like to get his attention (see this issue for the latest discussion on that). You can also try tor-dev@.

TL;DR: Nyx and Stem are presently unmaintained. I'm sorry I don't have better news for you.

[terminaldweller]

I could make a bunch of containers with different versions for everything and see when we actually run into this issue.
As for the memray result, I agree. It's not much to go on. Its basically just pointing out that those decorators are taking up a lot of memory but the how or why is unclear to me(I'm not very good with python). I could also get a line-by-line memory read-out but I'll try the different versions first.
As for contacting gk I'd rather have a PR ready first.

[arch-user-france1]

I ran memray --live-remote --native run.py and this is what I have so far

I'd like to reproduce your result, however you seem to use a different version of memray.

memray: error: argument command: invalid choice: 'run.py' (choose from 'run', 'flamegraph', 'table', 'live', 'tree', 'parse', 'summary', 'stats', 'transform', 'attach')

Do you by chance know what went wrong?

it doesn't make any sense:

(base) ➜  nyxmemleak memray --native                         
usage: memray [-h] [-v]
              {run,flamegraph,table,live,tree,parse,summary,stats,transform,attach}
              ...
memray: error: the following arguments are required: command
(base) ➜  nyxmemleak memray --native run            
usage: memray run [-m module | -c cmd | file] [args]
memray run: error: the following arguments are required: file, module
(base) ➜  nyxmemleak memray --native run run.py 
usage: memray [-h] [-v]
              {run,flamegraph,table,live,tree,parse,summary,stats,transform,attach}
              ...
memray: error: unrecognized arguments: --native
(base) ➜  nyxmemleak memray --native run run.py

[terminaldweller]

my bad. it should be memray run --live-remote --native ./run.py and it should return a port number. and then from another termianl memray live port-number.
I forgot to include the run part.

[arch-user-france1]

My idea is that nyx is calling a lot of wrapped which seems to be a function to use something with default arguments (if you ask me, a very special way of implementing defaults).

Unfortunately it is very hard to debug this code, especially because of its size.
Regarding the unindentifyable thing that's hogging most memory:

_pyfunction_vectorcall is a C-level function in the Python interpreter's API that was introduced in Python 3.8 as an alternative calling convention for Python functions. It is designed to improve the performance of calling Python functions by allowing the interpreter to make use of SIMD (Single Instruction, Multiple Data) instructions on modern processors.

and

_pyfunction_vectorcall is used internally by the interpreter when calling certain types of Python functions, such as those defined using the new "vectorcall" protocol. This protocol is a new feature in Python 3.9 that allows function calls to be more efficient by passing arguments and return values in a more compact format that is easier for the interpreter to handle.

So I assume that some function calls remain in the memory.

It is interesting that I can not observe that memory leak on my Fedora system. For how long have you been running it? I've run it for 30 minutes. Maybe that wasn't enough and now my ssh connection broke.

Has anyone got a better idea? Is anyone still interested?

[terminaldweller]

the memory leak is not a matter of how long. it is a constant leakage. you can see that it keep increasing periodically by a very small amount. it gets big over time.
which fedora version are you using? and how did you install nyx? with pip or dnf?
Also could you please paste the version of python,stem and nyx that you say is not leaking memory?

[terminaldweller]

here's a flamegraph that memray has made for a "leaky" session if anybody wants to take a look:
https://cargo.terminaldweller.com/memray-flamegraph-run.py.1817471.html