Skip to content

Replace sessions sorted set with Redis Search index #121

@abrookins

Description

@abrookins

Problem

PR #119 fixes a regression where list_sessions() reads from a sorted set that was never populated. However, using a sorted set for session indexing has a fundamental flaw: when working memory expires via TTL, the session entry remains in the sorted set as a stale/orphaned entry.

This means list_sessions() can return session IDs for sessions that no longer exist.

Proposed Solution

Replace the sorted set with a Redis Search index on the working memory JSON documents themselves.

Current approach (sorted set)

# On set_working_memory
await redis.zadd(sessions_key, {session_id: time.time()})

# On delete_working_memory  
await redis.zrem(sessions_key, session_id)

# TTL expiration: sorted set entry becomes stale ❌

Proposed approach (search index)

FT.CREATE working_memory_idx ON JSON PREFIX 1 working_memory: 
    SCHEMA $.session_id AS session_id TAG 
           $.namespace AS namespace TAG
           $.user_id AS user_id TAG
           $.created_at AS created_at NUMERIC
# list_sessions becomes a search query
FT.SEARCH working_memory_idx "@namespace:{demo}" 
    RETURN 1 session_id 
    SORTBY created_at DESC
    LIMIT 0 10

Benefits

  1. No stale entries — When TTL expires, the key is deleted and automatically removed from the index
  2. Consistent pattern — Same approach used for long-term memory indexing
  3. Richer queries — Can filter by user_id, namespace, created_at without additional data structures
  4. No cleanup tasks needed — Redis handles everything

Implementation

  1. Create index on startup (similar to long-term memory index)
  2. Replace list_sessions() to use FT.SEARCH instead of ZRANGE
  3. Remove zadd/zrem calls from set_working_memory/delete_working_memory
  4. Add migration path for existing sorted set data (or just let it age out)

Context

The sorted set approach was the original design (in the old messages.py), but the zadd/zrem calls were accidentally dropped during refactoring in commit 24151e2. PR #119 restores them, but the TTL limitation remains.

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions