No alloc normalizers#2133
Conversation
|
The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update. |
ArthurZucker
left a comment
ArthurZucker
left a comment
There was a problem hiding this comment.
can you publish the bench + try to pre-allocate the string that will be written in ?
| let table = &*BYTES_CHAR_LOOKUP; | ||
| let mut out = String::with_capacity(input.len()); | ||
| for &b in input.as_bytes() { | ||
| out.push(table[b as usize]); | ||
| } |
There was a problem hiding this comment.
TODO but can probably leverage SIMD
| } | ||
|
|
||
| impl pipeline::Normalizer for Prepend { | ||
| fn normalize<'a>(&self, input: &'a str) -> Cow<'a, str> { |
There was a problem hiding this comment.
IDK how viable but:
buf.clear();
buf.reserve(self.prepend.len() + input.len());
buf.push_str(&self.prepend);
buf.push_str(input);
buf.as_str()with a buffer part of the "scratch"
SBrandeis
force-pushed
the
plumbing-pretokenizers
branch
2 times, most recently
from
6371e9c to
89a1d19
Compare
SBrandeis
force-pushed
the
no-alloc-normalizers
branch
3 times, most recently
from
bd4e27d to
2426fed
Compare
SBrandeis
force-pushed
the
no-alloc-normalizers
branch
from
2426fed to
001304a
Compare
|
Let's merge this version and iterate on the suggested fixes as a follow-up? |
| pub trait Normalizer { | ||
| fn normalize<'a>(&self, input: &'a str) -> Cow<'a, str>; | ||
| } | ||
|
|
There was a problem hiding this comment.
The main change: bypass the NormalizedString and return a Cow instead
| compact_str = { version = "0.9", features = ["serde"] } | ||
| ptr_hash = { version = "1.1.0" } | ||
| memchr = "2.8.2" | ||
| unicode-normalization = "0.1.25" |
There was a problem hiding this comment.
Required to get the is_nfkd_quick checks in unicode normalizers
There was a problem hiding this comment.
mm fine for now, we'll skim this later on anyways when we work on bytes directly
SBrandeis
force-pushed
the
no-alloc-normalizers
branch
from
001304a to
6f89268
Compare
3 participants
PipelineTokenizer benchmark
1 / 4 models supported · geomean ×2.52 across supported · 18 fixtures each — ~10 kB inputs · single thread
6f8926800 · 2026-07-06 21:45 UTC· Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 8 coresNot yet supported
Per-fixture results
bert-base-uncased — WordPiece · Bert