Problem
The ILookupCache implementation (LookupCache) hits SQLite on every GetAsync/SetAsync call. For the single-instance deploy this is fine today, but:
- Every metadata lookup round-trips the DB for what should be a fast in-memory check
SetAsync calls SaveChangesAsync on every write — unnecessary SQLite write amplification
- The
LookupCacheEntry table is in the same DB as collection data, so a large cache competes with queries for WAL locks
- If we ever run multiple instances (Phase 4+), the per-process SQLite cache is useless
Goal
Replace the SQLite-backed lookup cache with a strategy pattern:
- Default:
IMemoryDistributedCache (in-process) — zero deploy cost, no persistence across restarts (acceptable for a 30-day TTL cache on a single-instance app)
- Swappable: Redis — opt-in via config for multi-instance deployments
- Interface stays
ILookupCache — callers (TmdbMovieProvider, MusicBrainzMusicProvider, IgdbGameProvider, UpcItemDbClient) are unchanged
Decisions
| Decision |
Choice |
| Drop persistence for default? |
Yes — cache is wiped on restart. 30-day TTL + rare restarts = negligible impact. |
JsonSerializerOptions |
Reuse current options (JsonStringEnumConverter(allowIntegerValues: true)) via a shared static (LookupCacheJson.Options) |
| Expiration mode |
Absolute — matches current SQLite semantics (FetchedAt set on write only) |
JsonSerializerOptions wiring |
Extract to static holder, no DI |
| Deserialization failure |
Delete bad entry + return default (same as today) |
| DI wiring location |
Inside AddMetadataLookup() — AddStackExchangeRedisCache is an IServiceCollection extension |
| Adapter lifetime |
AddSingleton — underlying IDistributedCache is singleton, adapter has no per-request state |
.env.example |
No change — memory is default, Redis config only needed for opt-in (documented in architecture.md) |
| Provider tests |
Mock ILookupCache with Moq — no DbContext/SQLite needed. Cache mechanics tested in adapter tests. |
| One-way migration |
Dropping LookupCacheEntries is irreversible. Cached data is ephemeral and rebuilt on next lookup — no user-facing data loss. |
Proposed design
Interface (unchanged)
public interface ILookupCache
{
Task<T?> GetAsync<T>(string provider, string key, TimeSpan ttl, CancellationToken ct = default);
Task SetAsync<T>(string provider, string key, T value, CancellationToken ct = default);
}
Shared JSON options
// Infrastructure/Lookup/LookupCacheJson.cs
public static class LookupCacheJson
{
public static readonly JsonSerializerOptions Options = new(JsonSerializerDefaults.Web)
{
Converters = { new JsonStringEnumConverter(allowIntegerValues: true) },
};
}
DistributedCacheAdapter
Thin wrapper: ILookupCache(provider, key, ttl) → IDistributedCache.Set/Get("lookup:{provider}:{key}", bytes, options).
- Composite key:
lookup:{provider}:{key}
- Serialize/deserialize via
LookupCacheJson.Options
AbsoluteExpirationRelativeToNow = ttl
- On JSON deserialization failure:
RemoveAsync(key) + return default
DI wiring (MetadataLookupServiceCollectionExtensions.cs)
var cacheProvider = config.GetValue<string>("Collectify:Cache:Provider") ?? "memory";
cacheProvider.ToLowerInvariant() switch
{
"redis" =>
{
var redisConfig = config["Collectify:Cache:Redis:Configuration"];
services.AddStackExchangeRedisCache(opt => opt.Configuration = redisConfig!);
services.AddSingleton<ILookupCache, DistributedCacheAdapter>();
},
_ =>
{
services.AddMemoryDistributedCache();
services.AddSingleton<ILookupCache, DistributedCacheAdapter>();
}
};
Config (opt-in only)
Collectify__Cache__Provider=redis
Collectify__Cache__Redis__Configuration=localhost:6379
Default (memory) requires no config.
Tasks
- Add
LookupCacheJson.cs (shared JsonSerializerOptions static)
- Add
DistributedCacheAdapter : ILookupCache in Infrastructure/Lookup/
- Remove
LookupCache class from ILookupCache.cs (keep interface)
- Update DI registration in
MetadataLookupServiceCollectionExtensions.cs
- Remove
LookupCacheEntry entity from Domain
- Remove
DbSet<LookupCacheEntry> + Fluent config from CollectifyDbContext
- Add EF migration to drop
LookupCacheEntries table
- Replace
LookupCacheTests with DistributedCacheAdapterTests (Moq)
- Swap
LookupCache → Moq ILookupCache in all four provider tests
- Add
<PackageReference Include="Moq" /> to test project
- Update XML docs in
TmdbMovieProvider.cs and UpcItemDbClient.cs ("cached in LookupCacheEntry table" → "cached in-memory")
- Update
docs/architecture.md — external providers section: "round-trips a JSON payload through the existing LookupCacheEntry table" → describes in-memory IDistributedCache with Redis opt-in
- Update
docs/data-model.md — remove reference to "keep the full provider response in LookupCacheEntry.JsonResponse so we can mine it later" (no longer persisted)
- Update
docs/superpowers/specs/2026-05-30-photo-snap-visual-lookup-design.md — references to LookupCache for image-hash caching
Blast radius
| File |
Change |
Domain/Entities/LookupCacheEntry.cs |
Delete |
Infrastructure/Data/CollectifyDbContext.cs |
Remove DbSet + Fluent config |
Infrastructure/Lookup/ILookupCache.cs |
Keep interface, delete LookupCache, add DistributedCacheAdapter + LookupCacheJson |
Infrastructure/Lookup/MetadataLookupServiceCollectionExtensions.cs |
Swap DI registration |
Infrastructure/Lookup/Tmdb/TmdbMovieProvider.cs |
XML doc only |
Infrastructure/Lookup/Upc/UpcItemDbClient.cs |
XML doc only |
tests/.../LookupCacheTests.cs |
Replace with adapter tests (Moq) |
tests/.../*ProviderTests.cs (4 files) |
LookupCache → Moq ILookupCache |
tests/.../Collectify.Tests.csproj |
Add Moq |
| New migration |
Drop LookupCacheEntries |
docs/architecture.md |
Update external providers section |
docs/data-model.md |
Remove LookupCacheEntry.JsonResponse reference |
docs/superpowers/specs/2026-05-30-photo-snap-visual-lookup-design.md |
Update LookupCache references |
Not doing
- No cache invalidation bus / pub-sub between instances
- No cache metrics / hit-rate endpoints
- No LRU eviction tuning beyond defaults
- No SQLite persistence for cold-start warmup
Problem
The
ILookupCacheimplementation (LookupCache) hits SQLite on everyGetAsync/SetAsynccall. For the single-instance deploy this is fine today, but:SetAsynccallsSaveChangesAsyncon every write — unnecessary SQLite write amplificationLookupCacheEntrytable is in the same DB as collection data, so a large cache competes with queries for WAL locksGoal
Replace the SQLite-backed lookup cache with a strategy pattern:
IMemoryDistributedCache(in-process) — zero deploy cost, no persistence across restarts (acceptable for a 30-day TTL cache on a single-instance app)ILookupCache— callers (TmdbMovieProvider,MusicBrainzMusicProvider,IgdbGameProvider,UpcItemDbClient) are unchangedDecisions
JsonSerializerOptionsJsonStringEnumConverter(allowIntegerValues: true)) via a shared static (LookupCacheJson.Options)FetchedAtset on write only)JsonSerializerOptionswiringdefault(same as today)AddMetadataLookup()—AddStackExchangeRedisCacheis anIServiceCollectionextensionAddSingleton— underlyingIDistributedCacheis singleton, adapter has no per-request state.env.examplememoryis default, Redis config only needed for opt-in (documented in architecture.md)ILookupCachewith Moq — noDbContext/SQLite needed. Cache mechanics tested in adapter tests.LookupCacheEntriesis irreversible. Cached data is ephemeral and rebuilt on next lookup — no user-facing data loss.Proposed design
Interface (unchanged)
Shared JSON options
DistributedCacheAdapterThin wrapper:
ILookupCache(provider, key, ttl)→IDistributedCache.Set/Get("lookup:{provider}:{key}", bytes, options).lookup:{provider}:{key}LookupCacheJson.OptionsAbsoluteExpirationRelativeToNow = ttlRemoveAsync(key)+ returndefaultDI wiring (
MetadataLookupServiceCollectionExtensions.cs)Config (opt-in only)
Default (
memory) requires no config.Tasks
LookupCacheJson.cs(sharedJsonSerializerOptionsstatic)DistributedCacheAdapter : ILookupCacheinInfrastructure/Lookup/LookupCacheclass fromILookupCache.cs(keep interface)MetadataLookupServiceCollectionExtensions.csLookupCacheEntryentity from DomainDbSet<LookupCacheEntry>+ Fluent config fromCollectifyDbContextLookupCacheEntriestableLookupCacheTestswithDistributedCacheAdapterTests(Moq)LookupCache→ MoqILookupCachein all four provider tests<PackageReference Include="Moq" />to test projectTmdbMovieProvider.csandUpcItemDbClient.cs("cached in LookupCacheEntry table" → "cached in-memory")docs/architecture.md— external providers section: "round-trips a JSON payload through the existing LookupCacheEntry table" → describes in-memoryIDistributedCachewith Redis opt-indocs/data-model.md— remove reference to "keep the full provider response in LookupCacheEntry.JsonResponse so we can mine it later" (no longer persisted)docs/superpowers/specs/2026-05-30-photo-snap-visual-lookup-design.md— references toLookupCachefor image-hash cachingBlast radius
Domain/Entities/LookupCacheEntry.csInfrastructure/Data/CollectifyDbContext.csDbSet+ Fluent configInfrastructure/Lookup/ILookupCache.csLookupCache, addDistributedCacheAdapter+LookupCacheJsonInfrastructure/Lookup/MetadataLookupServiceCollectionExtensions.csInfrastructure/Lookup/Tmdb/TmdbMovieProvider.csInfrastructure/Lookup/Upc/UpcItemDbClient.cstests/.../LookupCacheTests.cstests/.../*ProviderTests.cs(4 files)LookupCache→ MoqILookupCachetests/.../Collectify.Tests.csprojLookupCacheEntriesdocs/architecture.mddocs/data-model.mdLookupCacheEntry.JsonResponsereferencedocs/superpowers/specs/2026-05-30-photo-snap-visual-lookup-design.mdLookupCachereferencesNot doing