Skip to content

fix(utils): align Qwen tokenization for parallel tool calls#2226

Open
looput wants to merge 1 commit into
THUDM:mainfrom
looput:fix_qwen_token_ids
Open

fix(utils): align Qwen tokenization for parallel tool calls#2226
looput wants to merge 1 commit into
THUDM:mainfrom
looput:fix_qwen_token_ids

Conversation

@looput

@looput looput commented Jul 21, 2026

Copy link
Copy Markdown

The fix groups consecutive tool messages before passing them to apply_chat_template, preserving the formatting defined by the native Qwen2 chat template.

The original issue was caused by calling apply_chat_template separately for every message in gen_multi_turn_loss_mask_qwen, without preserving the context between consecutive tool messages.

For a parallel tool-calling sequence:

assistant(tool_calls=[A, B])
tool(result=A)
tool(result=B)

The Qwen2 chat template is designed to group consecutive tool responses into a single user block:

<|im_start|>user
<tool_response>
result=A
</tool_response>
<tool_response>
result=B
</tool_response><|im_end|>

However, the original implementation rendered each tool message independently. Since each message appeared to be both the first and last tool message, it produced:

<|im_start|>user
<tool_response>
result=A
</tool_response><|im_end|>
<|im_start|>user
<tool_response>
result=B
</tool_response><|im_end|>

This caused:

  • token_ids to differ from the native Qwen chat-template output;
  • redundant <|im_start|>user and <|im_end|> tokens to be inserted;
  • the training input format to differ from the model’s actual inference format;
  • token positions and loss-mask alignment to potentially become incorrect.

The issue only affects parallel tool calling with consecutive role="tool" responses. A single assistant message containing multiple tool_calls, or sequential assistant → tool → assistant → tool interactions, does not trigger this problem.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant