Skip to content

fix: fail embedding writes on empty upsert id#3184

Open
qq788538-dotcom wants to merge 1 commit into
volcengine:mainfrom
qq788538-dotcom:fix/empty-vector-upsert-ack
Open

fix: fail embedding writes on empty upsert id#3184
qq788538-dotcom wants to merge 1 commit into
volcengine:mainfrom
qq788538-dotcom:fix/empty-vector-upsert-ack

Conversation

@qq788538-dotcom

Copy link
Copy Markdown

Summary

  • treat an empty record ID returned by the vector database upsert as a write failure
  • route that failure through the existing embedding error accounting and callbacks
  • add a regression test covering queue errors, request failure tracking, and request stats

Problem

TextEmbeddingHandler.on_dequeue() currently records an embedding message as processed even when VikingVectorIndexBackend.upsert() returns an empty string. The backend uses an empty ID to signal several failed/no-op write paths. Because the handler only logs when the ID is truthy and then unconditionally reports success, reindex can report hundreds of rebuilt records while the vector collection remains empty.

Observed against OpenViking Service v0.4.8:

  • embedding queue processed: 751
  • reported rebuilt records in a narrow reindex: 369
  • vector count after completion: 0
  • semantic find: 0 results

Fix

Require a non-empty record ID after upsert. An empty ID raises into the existing database error branch, which already:

  • increments embedding error_count
  • records request failure
  • invokes the queue error callback
  • avoids the success callback and processed count

This does not change successful write behavior or shutdown handling.

Test plan

pytest -q --no-cov tests/storage/test_text_embedding_handler.py
ruff check openviking/storage/collection_schemas.py tests/storage/test_text_embedding_handler.py

Both pass locally on Python 3.11.15.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Backlog

Development

Successfully merging this pull request may close these issues.

1 participant