Performance: Deep Historical Queries

[shocknet-justin]

strfry Query Performance: Deep Historical Queries

Observation

Queries with until filters become progressively slower when fetching older events. Working with a relay containing 200k+ events of a specific kind/author, query times increase from <1s for recent data to 30+s for data a few hours older.

Environment

• Remote relay: strfry (wss://nip85.brainstorm.world)
  • Hardware: DigitalOcean droplet, 8GB RAM, SSD storage
• Client: nostr-tools SimplePool.querySync()
• Query: { kinds: [30382], authors: ["48ec018359cac3c933f0f7a14550e36a4f683dcf55520c916dd8c61e7724f5de"], until: <timestamp>, limit: 500 }
• Known event count: ~200,000 events matching this filter

Observed Behavior

Query Duration by Depth

Pagination working backward from present (until = now, decrementing):

Timestamp Range	Duration	Events/page	Notes
Recent (now)	0.8-1s	490-500	Fast, likely in cache
-1hr	7-8s	450-500	Moderate slowdown
-2hr	12-14s	400-480	Significant slowdown
-3hr	17s	400-470	Approaching timeout limits
-4.5hr	24-25s	400-476	Near timeout limit (30s)
-5hr+	26-27s	410-490	Consistently near timeout
-6hr+	28-29s	450-485	96% of timeout limit
Older	Timeout	0	Eventually returns empty

Actual Results

• With 4.4s timeout: Retrieved 28,000 events before timing out
• With 8.8s timeout: Retrieved 40,000 events before timing out
• With 30s timeout: Retrieved 92,000+ events (ongoing), queries now at 28-29s

Query duration increases exponentially as we go back in time. At 92k events retrieved (roughly 6 hours of history), queries are taking 28-29 seconds - 96% of the 30 second timeout limit. This represents only ~46% of the known 200k total events. At this rate, queries will exceed the timeout before retrieving the remaining ~108k events.

Hardware specs (8GB RAM, SSD) should be adequate for query performance, suggesting the bottleneck may be LMDB-specific behavior rather than raw storage speed.

Hypothesis

This could be due to:

1. LMDB page cache behavior (recent data cached, older data requires disk I/O)
2. B+ tree traversal characteristics for deep historical queries
3. Index structure for compound filters (author+kind+timestamp)

Reproduction Steps

# Using nostr-tools or similar client
# Start with recent data
filter = {
  kinds: [30382],
  authors: ["48ec018359cac3c933f0f7a14550e36a4f683dcf55520c916dd8c61e7724f5de"],
  until: Math.floor(Date.now() / 1000),
  limit: 500
}

# Get first page
events = await pool.querySync(relays, filter)
# Query completes in ~1s

# Paginate backward: set until to oldest timestamp from previous page
oldest = Math.min(...events.map(e => e.created_at))
filter.until = oldest
events = await pool.querySync(relays, filter)
# Query completes in ~1s

# Continue pagination...
# After fetching ~50k events over a few hours of history:
# Query duration has increased to 15-20s+

Questions

1. Is this behavior expected for deep historical queries with LMDB?
2. Are there strfry configuration options that could help (cache size, etc.)?
3. Is there a better query pattern for backfilling large historical datasets?
4. Would a local strfry sync + negentropy copy improve query performance significantly?

[shocknet-justin]

Update: Tested and worked around

I reproduced this locally by running my own strfry instance (synced via negentropy) with the same 200k event dataset. Query performance against localhost (high end hardware with nvme storage) still showed exponential degradation as pagination went deeper into history supporting the theory the the issue is LMDB-specific.

Workaround:
Using negentropy for initial sync + strfry export to JSONL completely bypasses the query performance issue:

1. strfry sync via negentropy: ~30 seconds for 200k events
2. strfry export --filter: <1 second
3. Process JSONL directly: ~60 seconds

This got us all 200k events in ~90 seconds total vs the seemingly impossible task of trying to paginate through them via queries.

[hoytech]

Sorry for the delay in replying! Thanks for the report. I'm glad you got it working in the end.

I think you're probably right that this has something to do with the virtual memory system. LMDB is pretty much just a layer on top of a big mmap(), and if the DB is much larger than RAM and the query has relatively random access patterns, then it can cause thrashing. The exponential behaviour you describe is somewhat surprising to me though. I would've expected roughly constant amount of work required no matter where you start the pagination, since it should just be traversing the authors index. I'll see if I can figure out a test case to reproduce it. If you recall, on the server, was the strfry process using a lot of CPU during the query?

As you describe, doing an export and import will almost always be faster, since there is no websocket overhead or latency. Negentropy is potentially helpful for backfilling missing events in mostly in-sync datasets, but ultimately it will be downloading individual events over websocket also.

Another thing you could try is to compact the DB. This ensures that adjacent records like indices are actually stored in adjacent order on-disk, which can help somewhat with the memory access patterns. Doing a full export and import can improve this even more in some cases, depending on how the dataset was actually built up and how fragmented it is.

Sorry I don't have any immediate answers or suggestions, but I'll try to do some more investigation.

[etemiz]

nos.lol 242 GB database size, could function ok with 128 GB RAM (Hetzner EX44 machine). RAID 0. but then when the 192 GB swap was also full, it failed to answer queries in time. many clients timed out.

one thing i noticed, i you compact, restart strfry pointing to the new db, and remove the old bloated db, memory use dramatically drops. which suggests the old db file was still mapped to memory even though nobody was using it.

time to delete unimportant events from past. or the best solution would be strfry not using memory for archival events. is this even possible? i remember some mentions of an archival mode?

[hoytech]

Yes good point on the compaction. This can help a lot, since a busy server will have a fragmented DB. Memory use always happens at the page level (ie 4KB), so a lot of the space wasted by fragmentation must nevertheless be mapped into memory.

I will think about an archival mode. Yes it is possible, by deindexing certain events. But then they won't show up in various nostr queries. I wonder what the minimal indexing would be for nostr events. Maybe just by pubkey?

[etemiz]

i can vibe a script that parses the queries and finds the archival type of queries and summarize which kind of indexes was needed.

i am guessing data miners are a possibility. pubkey might be also important..

another option can be forwarding to another strfry instance if the request is in the past. a cheaper server which carries all the events can handle these, responding slowly. no idea if this makes more sense.