-
Notifications
You must be signed in to change notification settings - Fork 2.2k
Open
Description
Hello,
when I try to submit a request using the api, I have an INTERNAL SERVER ERROR, and in the logs I can see:
ValueError: Cannot use chat template functions because tokenizer.chat_template is not set and no template argument was passed! For
information about writing templates and setting the tokenizer.chat_template attribute, please see the documentation at
https://huggingface.co/docs/transformers/main/en/chat_templating
The thing is the model has a tokenizer.chat_template set... so the error above makes no sense.
For example, if I run the following python code, it executes without problem
from mlx_lm import load, generate
model, tokenizer = load("fausap/Cydonia-24B-v4.1-4bit")
prompt = "hello"
print("Applying tokenizer template...")
messages = [{"role": "user", "content": prompt}]
prompt = tokenizer.apply_chat_template(
messages, add_generation_prompt=True
)
response = generate(model, tokenizer, prompt=prompt, verbose=True)
the output is
Applying tokenizer template...
==========
Hello! How can I help you today?
==========
Prompt: 178 tokens, 114.134 tokens-per-sec
Generation: 10 tokens, 20.415 tokens-per-sec
Peak memory: 13.583 GB
How can I solve this problem?
thanks
Metadata
Metadata
Assignees
Labels
No labels