[Bug]: Jobs are not being processed in a long running sqlite server

[IgnisDa]

What happened?

Hi! I recently updated apalis from 0.7.4 to 1.0.0-rc.4 and have been noticing reports on Ryot's github and discord about jobs not being processed (example). I also switched the job backend from memory to sqlite. This was done because the rc versions don't allow cloning the MemoryStorage backend.

Earlier I thought that it was because the queue was being choked by a lot of jobs being added to it (which is by design). This queue has a concurrency of 1. To that end, I "sharded" the queue so that more jobs are processed in parallel (commit).

But people are still reporting this issue, so this was not the fix. I am stumped on what may be the actual issue.

Any suggestions on what I can do to debug this issue of jobs stopped being processed after some time?

Expected behavior

Jobs to be processed correctly.

Steps to reproduce

I have a standard apalis setup (https://github.com/IgnisDa/ryot/blob/2074b8e72d295a9bd4e3deaffdc2fd8812a802aa/apps/backend/src/main.rs#L187).

Minimal code example

Version

1.0.0-rc.x

Environment

• OS: Any
• Rust version: 1.93.1
• Cargo version: 1.93.1

Relevant log output

Additional context

No response

[IgnisDa]

Here is the commit when apalis was updated from 0.7.4 to rc-* version: IgnisDa/ryot@6adbd15.

[geofmureithi]

Could you do a dump of an sqlite file for one of the hanging queues, this can help me look at what is exactly hanging. I will move this issue to apalis-sqlite repo because I feel this is sqlite specific. Before I move it though I would also ask you to try JsonStorage just to help confirm its an sqlite issue.

…

On Tue, 3 Mar 2026 at 04:04, Diptesh ***@***.***> wrote: *IgnisDa* left a comment (apalis-dev/apalis-sqlite#58) <#58 (comment)> Here is the commit when apalis was updated from 0.7.4 to rc-* version: ***@***.*** <IgnisDa/ryot@6adbd15> . — Reply to this email directly, view it on GitHub <#58 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AWXVRGQU65DTMG53FYECDRL4OYVTFAVCNFSM6AAAAACWE3HWGSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZTSOBXHE2DKMZQGU> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

[IgnisDa]

Hi @geofmureithi. I have forwarded you the db file on Discord.

Something I want to mention which makes me believe this is an apalis issue: I have set the db path to /tmp/ryot_jobs in the ryot container. And this is not mounted to a volume so every time the container is restarted, the file is made afresh. This is by design since I don't need the jobs to be persisted across restarts.

The reason I mention this is that users are reporting that everything starts working fine after the container is restarted.

[IgnisDa]

Just wanted to update this is indeed a sqlite issue. I switched over to memory storage in this PR: IgnisDa/ryot#1722

Additionally I tried json storage and it errored out when there were a lot of concurrent requests:

[geofmureithi]

Could you try using the SharedSqliteStorage here?

for (shard_idx, single_application_job_storage) in
            single_application_job_storages.into_iter().enumerate()
        {
            let shard_ss = supporting_service.clone();
            monitor = monitor.register(move |_runs| {
                WorkerBuilder::new(format!("perform_single_application_job_{shard_idx}"))
                    .backend(single_application_job_storage.clone())
                    .catch_panic()
                    .enable_tracing()
                    .concurrency(1)
                    .data(shard_ss.clone())
                    .build(perform_single_application_job)
            });
        }

[geofmureithi]

I have not been able to replicate this issue and think this was caused by too many concurrent connections. Closing unless this is reported again

[filipencopav]

Reporting this again, it is happening to me as well, just quits processing jobs after like 30 minutes of running. I have 2 very simple queues (workflows). They have .concurrency(10) and .concurrency(50) respectively. However, I initially had values of .concurrency(1) for both. At some point they stopped processing as well with ,concurrency(1). Please suggest me an alternative backend that I can use and that doesn't require me to install some other software to the device I'm deploying on

Alternative, I'll take any other reliable durable queue library. It shouldn't be this hard to make a working durable queue library. Clojurists can do such a simple task. However, nothing in the rust ecosystem "just works". yaque has some stupid issue where its queue index overflows. Other queue libraries are either dead -- unmaintained ~6 years, or they require me to launch effing postgres, or some huge queue system on an edge device -- just to implement their queue system on top of the queue system

[geofmureithi]

If you are experiencing issues related to concurrency, it means that you are blocking your own worker. If you have concurrency of 1 and 1 job doesn't get completed, then the worker is not gonna request a new job. Please pair concurrency with a timeout else a long running job running forever would just block. This would not be an apalis issue but rather an issue with your setup.

[filipencopav]

No, I am not experiencing any kind of blocking forever. Because all I have to do is to restart my application, and the tasks start being processed, regardless if I have concurrency 1 or 10. The problem, as described in OP, is that long running apalis-sqlite queue(s) eventually stop processing their items completely.

[geofmureithi]

Op was attempting manual sharding causing too many calls to the sqlite db which led to the blocking issue. Is this what you are attempting?