Skip to content

Fix Milvus HNSW-PQ m parameter conflicts with M.#821

Open
bashimao wants to merge 1 commit into
zilliztech:mainfrom
bashimao:hnswpq_pram_passing
Open

Fix Milvus HNSW-PQ m parameter conflicts with M.#821
bashimao wants to merge 1 commit into
zilliztech:mainfrom
bashimao:hnswpq_pram_passing

Conversation

@bashimao

Copy link
Copy Markdown

The m parameter in Milvus HNSW-PQ is not adjustable when using the command-line. Even worse, setting M and m leads to a situation where the order in which you define the parameters in your YAML-file determines what value ends up in M.

Proposed solution

Add a separate command-line switch --pq-m to specify the m for product quantization.

However, while this fixes the frontend, which is sufficient for my current needs, I strongly suggest you consider the renaming the parameter in the backend as well. This idea to have parameters that only differ in upper/lower-case (i.e., M vs. m) is an accident bound to happen.

To replicate the problem

git checkout main

echo "milvushnswpq:
  dry_run: True
  drop_old: False
  load: False
  load_concurrency: 48
  search_serial: True
  search_concurrent: True
  case_type: Performance768D1M
  db_label: perf_768d_1m
  k: 200
  concurrency_duration: 60
  num_concurrency: 128
  concurrency_timeout: 1800
  serial_cooldown: 5
  task_label: search_perf_768d_1m
  uri: http://milvus-standalone:19530
  db_name: perf_768d_1m
  num_shards: 2
  replica_number: 2
  use_partition_key: True
  M: 8
  ef_construction: 384
  ef_search: 128
  m: 64
  refine: True
  refine_type: fp16
  refine_k: 2.0
  nbits: 16" > test.yaml

python3 -m vectordb_bench.cli.vectordbbench milvushnswpq --config-file test.yaml

Output:

2026-07-17 14:49:19,728 | INFO: Task:
TaskConfig(
  db=<DB.Milvus: 'Milvus'>,
  ...
  db_case_config=HNSWPQConfig(index=<IndexType.HNSW_PQ: 'HNSW_PQ'>,
    metric_type=None, 
    use_partition_key=True, M=64, efConstruction=384, ef=128,
    m=32, nbits=16, refine=True, refine_type=<SQType.FP16: 'FP16'>, refine_k=2.0
  ), ...

Note how M=64 now. Because m is the later value. m=32 because it is literally not hooked up in the backend.

To replicate the fix

git checkout hnswpq_pram_passing  # assuming you got bashimao/VectorDBBench cloned

python3 -m vectordb_bench.cli.vectordbbench milvushnswpq --config-file test.yaml

Output:

Usage: python -m vectordb_bench.cli.vectordbbench milvushnswpq
           [OPTIONS]
Try 'python -m vectordb_bench.cli.vectordbbench milvushnswpq --help' for help.

Error: Missing option '--pq-m'.
git checkout hnswpq_pram_passing  # assuming you got bashimao/VectorDBBench cloned

echo "milvushnswpq:
  dry_run: True
  drop_old: False
  load: False
  load_concurrency: 48
  search_serial: True
  search_concurrent: True
  case_type: Performance768D1M
  db_label: perf_768d_1m
  k: 200
  concurrency_duration: 60
  num_concurrency: 128
  concurrency_timeout: 1800
  serial_cooldown: 5
  task_label: search_perf_768d_1m
  uri: http://milvus-standalone:19530
  db_name: perf_768d_1m
  num_shards: 2
  replica_number: 2
  use_partition_key: True
  m: 8
  ef_construction: 384
  ef_search: 128
  pq_m: 64
  refine: True
  refine_type: fp16
  refine_k: 2.0
  nbits: 16" > test.yaml

Output:

2026-07-17 15:11:49,018 | INFO: Task:
TaskConfig(
  db=<DB.Milvus: 'Milvus'>,
  ...
  db_case_config=HNSWPQConfig(index=<IndexType.HNSW_PQ: 'HNSW_PQ'>,
    metric_type=None, use_partition_key=True, M=8, efConstruction=384, ef=128,
    m=64, nbits=16, refine=True, refine_type=<SQType.FP16: 'FP16'>, refine_k=2.0
  ), ...

Note how M=8 regardless of whether it is spelled upper- or lowercase in the YAML, and m=64 because in the YAML pq_m: 64.

@sre-ci-robot

Copy link
Copy Markdown

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: bashimao
To complete the pull request process, please assign xuanyang-cn after the PR has been reviewed.
You can assign the PR to them by writing /assign @xuanyang-cn in a comment when ready.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@bashimao
bashimao force-pushed the hnswpq_pram_passing branch from b13dd7b to 37cea42 Compare July 17, 2026 15:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants