Skip to content

Conversation

@lantingchiang
Copy link
Contributor

This commit includes two changes:

  1. Make the turbopuffer client runnable as a subcommand in the CLI like so: vectordbbench turbopuffer --help
  2. The turbopuffer API calls in the current client implementation appear to be using the python SDK that's prior to v0.5. v0.5 introduced breaking changes (see https://github.com/turbopuffer/turbopuffer-python/blob/main/UPGRADING.md). Updated the API calls to conform with the new SDK version.

Testing locally:

$ vectordbbench turbopuffer  --api-key SECRET --case-type Performance768D1M --namespace vdbbench  --skip-drop-old --skip-load

2025-12-04 20:17:32,128 | INFO: Task:
TaskConfig(db=<DB.TurboPuffer: 'TurboPuffer'>, db_config=TurboPufferConfig(db_label='2025-12-04T20:17:32.061781', version='', note='', api_key=SecretStr('**********'), api_base_url='https://api.turbopuffer.com', namespace='vdbbench'), db_case_config=TurboPufferIndexConfig(metric_type=None, use_multi_ns_for_filter=False, time_wait_warmup=60), case_config=CaseConfig(case_id=<CaseType.Performance768D1M: 5>, custom_case={}, k=100, concurrency_search_config=ConcurrencySearchConfig(num_concurrency=[1, 5, 10, 20, 30, 40, 60, 80], concurrency_duration=30, concurrency_timeout=3600)), stages=['search_serial', 'search_concurrent'])
 (cli.py:649) (43040)
2025-12-04 20:17:32,128 | INFO: generated uuid for the tasks: cd18598c871d4b99a3ad5e1863cc4947 (interface.py:76) (43040)
2025-12-04 20:17:32,309 | INFO | DB             | CaseType     Dataset               Filter | task_label (task_runner.py:400)
2025-12-04 20:17:32,310 | INFO | -----------    | ------------ -------------------- ------- | -------    (task_runner.py:400)
2025-12-04 20:17:32,310 | INFO | TurboPuffer-2025-12-04T20:17:32.061781 | Performance  Cohere-MEDIUM-1M         0.0 | cd18598c871d4b99a3ad5e1863cc4947 (task_runner.py:400)
2025-12-04 20:17:32,310 | INFO: task submitted: id=cd18598c871d4b99a3ad5e1863cc4947, cd18598c871d4b99a3ad5e1863cc4947, case number: 1 (interface.py:251) (43040)
2025-12-04 20:17:33,032 | INFO: [1/1] start case: {'label': <CaseLabel.Performance: 2>, 'name': 'Search Performance Test (1M Dataset, 768 Dim)', 'dataset': {'data': {'name': 'Cohere', 'size': 1000000, 'dim': 768, 'metric_type': <MetricType.COSINE: 'COSINE'>}}, 'db': 'TurboPuffer-2025-12-04T20:17:32.061781'}, drop_old=False (interface.py:181) (43105)
2025-12-04 20:17:33,033 | INFO: Starting run (task_runner.py:138) (43105)
2025-12-04 20:17:33,341 | INFO: local dataset root path not exist, creating it: /tmp/vectordb_bench/dataset/cohere/cohere_medium_1m (data_source.py:124) (43105)
2025-12-04 20:17:33,341 | INFO: Start to downloading files, total count: 4 (data_source.py:140) (43105)
 25%|███████████████████████████▊                                                                                   | 1 50%|███████████████████████████████████████████████████████▌                                                       | 2 75%|███████████████████████████████████████████████████████████████████████████████████▎                           | 3100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████| 4/4 [00:48<00:00, 12.19s/it]
2025-12-04 20:18:22,119 | INFO: Succeed to download all files, downloaded file count = 4 (data_source.py:145) (43105)
2025-12-04 20:18:22,119 | INFO: Read the entire file into memory: test.parquet (dataset.py:392) (43105)
2025-12-04 20:18:22,160 | INFO: Read the entire file into memory: neighbors.parquet (dataset.py:392) (43105)
2025-12-04 20:18:22,214 | INFO: Start performance case (task_runner.py:183) (43105)
2025-12-04 20:18:22,945 | INFO: Start search 30s in concurrency 1, filters: type=<FilterOp.NonFilter: 'NonFilter'> filter_rate=0.0 gt_file_name='neighbors.parquet' (mp_runner.py:116) (43105)
...

@lantingchiang
Copy link
Contributor Author

/assign @XuanYang-cn

@sre-ci-robot
Copy link

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: alwayslove2013, lantingchiang
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@alwayslove2013
Copy link
Collaborator

@lantingchiang Good job! We’re really excited to merge this PR, but we’ve noticed a few lint issues that need to be addressed~

pip install black ruff --upgrade
make format
make lint

@lantingchiang
Copy link
Contributor Author

@alwayslove2013 thanks! Ran the linting & updated. Would you mind approving the workflow?

@alwayslove2013 alwayslove2013 merged commit c2f1c51 into zilliztech:main Dec 8, 2025
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants