nats.errors.TimeoutError on nats timeout

[MMiirrkk]

This error is blocked ofp (second time):

Aug 01 00:10:18 ofp poetry[1249174]: 2024-08-01 00:10:18 [ERROR] [messenger] Trying to publish to subject 'tic.status.zb08.fits.pipeline.zdf' failed. Message msg-1301 publish error: nats: timeout
Aug 01 00:10:18 ofp poetry[1249174]: 2024-08-01 00:10:18 [ERROR] [task_manager] Task task-stats-stats_2-33 failed: nats: timeout
Aug 01 00:10:18 ofp poetry[1249174]: Traceback (most recent call last):
Aug 01 00:10:18 ofp poetry[1249174]:   File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
Aug 01 00:10:18 ofp poetry[1249174]:     return fut.result()
Aug 01 00:10:18 ofp poetry[1249174]: asyncio.exceptions.CancelledError
Aug 01 00:10:18 ofp poetry[1249174]: The above exception was the direct cause of the following exception:
Aug 01 00:10:18 ofp poetry[1249174]: Traceback (most recent call last):
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1061, in _request_new_style
Aug 01 00:10:18 ofp poetry[1249174]:     return await asyncio.wait_for(future, timeout)
Aug 01 00:10:18 ofp poetry[1249174]:   File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
Aug 01 00:10:18 ofp poetry[1249174]:     raise exceptions.TimeoutError() from exc
Aug 01 00:10:18 ofp poetry[1249174]: asyncio.exceptions.TimeoutError
Aug 01 00:10:18 ofp poetry[1249174]: During handling of the above exception, another exception occurred:
Aug 01 00:10:18 ofp poetry[1249174]: Traceback (most recent call last):
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/src/oca-fits-proc/fits_proc/modules/stats.py", line 233, in _run
Aug 01 00:10:18 ofp poetry[1249174]:     ps = await self.process_stats()
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/src/oca-fits-proc/fits_proc/modules/stats.py", line 200, in process_stats
Aug 01 00:10:18 ofp poetry[1249174]:     await self.fm.nats_conn.status_fits_zdf_pub.publish(
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_single_pub.py", line 18, in publish
Aug 01 00:10:18 ofp poetry[1249174]:     return await super().publish(data, meta=meta, **kwargs)
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_publisher.py", line 61, in publish
Aug 01 00:10:18 ofp poetry[1249174]:     raise e
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_publisher.py", line 47, in publish
Aug 01 00:10:18 ofp poetry[1249174]:     await self.connection.js.publish(self.subject, bdata, **kwargs)
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/js/client.py", line 181, in publish
Aug 01 00:10:18 ofp poetry[1249174]:     msg = await self._nc.request(
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1021, in request
Aug 01 00:10:18 ofp poetry[1249174]:     msg = await self._request_new_style(
Aug 01 00:10:18 ofp poetry[1249174]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1063, in _request_new_style
Aug 01 00:10:18 ofp poetry[1249174]:     raise errors.TimeoutError
Aug 01 00:10:18 ofp poetry[1249174]: nats.errors.TimeoutError: nats: timeout

@majkelx

[MMiirrkk]

Hi. Again I get the same error (serverish 1.4.0):

Jul 25 08:15:21 ofp poetry[1639468]: 2025-07-25 08:15:21 [ERROR] [msg_single_read] Failed to open single reader for subject tic.status.wk06.download: nats: timeout
Jul 25 08:15:22 ofp poetry[1639468]: 2025-07-25 08:15:21 [ERROR] [task_manager] Task subscr_checker failed: nats: timeout
Jul 25 08:15:22 ofp poetry[1639468]: Traceback (most recent call last):
Jul 25 08:15:22 ofp poetry[1639468]:   File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
Jul 25 08:15:22 ofp poetry[1639468]:     return fut.result()
Jul 25 08:15:22 ofp poetry[1639468]: asyncio.exceptions.CancelledError
Jul 25 08:15:22 ofp poetry[1639468]: The above exception was the direct cause of the following exception:
Jul 25 08:15:22 ofp poetry[1639468]: Traceback (most recent call last):
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1103, in _request_new_style
Jul 25 08:15:22 ofp poetry[1639468]:     return await asyncio.wait_for(future, timeout)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
Jul 25 08:15:22 ofp poetry[1639468]:     raise exceptions.TimeoutError() from exc
Jul 25 08:15:22 ofp poetry[1639468]: asyncio.exceptions.TimeoutError
Jul 25 08:15:22 ofp poetry[1639468]: During handling of the above exception, another exception occurred:
Jul 25 08:15:22 ofp poetry[1639468]: Traceback (most recent call last):
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/src/oca-fits-proc/fits_proc/nats_conn.py", line 154, in subscr_checker
Jul 25 08:15:22 ofp poetry[1639468]:     data, meta = await self.get_single_read_msg(subject)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/src/oca-fits-proc/fits_proc/nats_conn.py", line 109, in get_single_read_msg
Jul 25 08:15:22 ofp poetry[1639468]:     data, meta = await single_read(subject=subject)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_single_read.py", line 118, in single_read
Jul 25 08:15:22 ofp poetry[1639468]:     return await reader.read(wait=wait)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_single_read.py", line 57, in read
Jul 25 08:15:22 ofp poetry[1639468]:     raise e
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_single_read.py", line 49, in read
Jul 25 08:15:22 ofp poetry[1639468]:     await self.open()
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_reader.py", line 457, in open
Jul 25 08:15:22 ofp poetry[1639468]:     self.pull_subscription = await self._create_pull_subscribtion(consumer_conf)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_reader.py", line 476, in _create_pull_subscribtion
Jul 25 08:15:22 ofp poetry[1639468]:     ret = await self.connection.js.pull_subscribe(self.subject,
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/js/client.py", line 600, in pull_subscribe
Jul 25 08:15:22 ofp poetry[1639468]:     await self._jsm.add_consumer(stream, config=config)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/js/manager.py", line 262, in add_consumer
Jul 25 08:15:22 ofp poetry[1639468]:     resp = await self._api_request(subject, req_data, timeout=timeout)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/js/manager.py", line 414, in _api_request
Jul 25 08:15:22 ofp poetry[1639468]:     msg = await self._nc.request(req_subject, req, timeout=timeout)
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1061, in request
Jul 25 08:15:22 ofp poetry[1639468]:     msg = await self._request_new_style(
Jul 25 08:15:22 ofp poetry[1639468]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1105, in _request_new_style
Jul 25 08:15:22 ofp poetry[1639468]:     raise errors.TimeoutError
Jul 25 08:15:22 ofp poetry[1639468]: nats.errors.TimeoutError: nats: timeout

@majkelx

[MMiirrkk]

Another error occured but this time with publish:

Aug 17 01:00:00 ofp poetry[538665]: 2025-08-17 01:00:00 [INFO] [stacker] Stacker done jk15c_0904_53789, time: 65.4s.
Aug 17 01:00:00 ofp poetry[538665]: 2025-08-17 01:00:00 [INFO] [stats] Stars presence: no bkg cosmic_bad_pixels 0.17%
Aug 17 01:00:00 ofp poetry[538665]: 2025-08-17 01:00:00 [INFO] [stats] Stars presence done: 8.9e+01s
Aug 17 01:01:05 ofp poetry[538665]: 2025-08-17 01:01:05 [ERROR] [messenger] Trying to publish to subject 'tic.status.jk15.fits.pipeline.faststat' failed. Message msg-21803 publish error: nats: timeout
Aug 17 01:01:06 ofp poetry[538665]: 2025-08-17 01:01:05 [ERROR] [task_manager] Task task-stats-stats_1-885 failed: nats: timeout
Aug 17 01:01:06 ofp poetry[538665]: Traceback (most recent call last):
Aug 17 01:01:06 ofp poetry[538665]:   File "/usr/lib/python3.10/asyncio/tasks.py", line 456, in wait_for
Aug 17 01:01:06 ofp poetry[538665]:     return fut.result()
Aug 17 01:01:06 ofp poetry[538665]: asyncio.exceptions.CancelledError
Aug 17 01:01:06 ofp poetry[538665]: The above exception was the direct cause of the following exception:
Aug 17 01:01:06 ofp poetry[538665]: Traceback (most recent call last):
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1103, in _request_new_style
Aug 17 01:01:06 ofp poetry[538665]:     return await asyncio.wait_for(future, timeout)
Aug 17 01:01:06 ofp poetry[538665]:   File "/usr/lib/python3.10/asyncio/tasks.py", line 458, in wait_for
Aug 17 01:01:06 ofp poetry[538665]:     raise exceptions.TimeoutError() from exc
Aug 17 01:01:06 ofp poetry[538665]: asyncio.exceptions.TimeoutError
Aug 17 01:01:06 ofp poetry[538665]: During handling of the above exception, another exception occurred:
Aug 17 01:01:06 ofp poetry[538665]: Traceback (most recent call last):
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/src/oca-fits-proc/fits_proc/modules/stats.py", line 447, in _run
Aug 17 01:01:06 ofp poetry[538665]:     ps = await self.process_stats()
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/src/oca-fits-proc/fits_proc/modules/stats.py", line 417, in process_stats
Aug 17 01:01:06 ofp poetry[538665]:     await self.results_pub()
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/src/oca-fits-proc/fits_proc/modules/stats.py", line 272, in results_pub
Aug 17 01:01:06 ofp poetry[538665]:     await self.fm.nats_conn.status_fits_faststat_pub.publish(
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_single_pub.py", line 18, in publish
Aug 17 01:01:06 ofp poetry[538665]:     return await super().publish(data, meta=meta, **kwargs)
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_publisher.py", line 61, in publish
Aug 17 01:01:06 ofp poetry[538665]:     raise e
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/serverish/messenger/msg_publisher.py", line 47, in publish
Aug 17 01:01:06 ofp poetry[538665]:     await self.connection.js.publish(self.subject, bdata, **kwargs)
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/js/client.py", line 201, in publish
Aug 17 01:01:06 ofp poetry[538665]:     msg = await self._nc.request(
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1061, in request
Aug 17 01:01:06 ofp poetry[538665]:     msg = await self._request_new_style(
Aug 17 01:01:06 ofp poetry[538665]:   File "/home/poweruser/.cache/pypoetry/virtualenvs/oca-fits-proc-ra1OpwlD-py3.10/lib/python3.10/site-packages/nats/aio/client.py", line 1105, in _request_new_style
Aug 17 01:01:06 ofp poetry[538665]:     raise errors.TimeoutError
Aug 17 01:01:06 ofp poetry[538665]: nats.errors.TimeoutError: nats: timeout
Aug 17 01:01:06 ofp poetry[538665]: 2025-08-17 01:01:06 [INFO] [nats_conn] Subscription tic.status.jk15.download ok, recreated: 7, checker recreated: 1
Aug 17 01:01:06 ofp poetry[538665]: 2025-08-17 01:01:06 [INFO] [fits_data]
Aug 17 01:01:06 ofp poetry[538665]: >>> PROCESS:

@majkelx

[MMiirrkk]

Hangs on reader and subscriber - and after never rebuilt:

@majkelx