WordPiece via MPHF + bounded longest-match (PTRHash stream experiment) by ArthurZucker · Pull Request #2179 · huggingface/tokenizers

ArthurZucker · 2026-07-09T19:58:50Z

What

Routes the WordPiece pipeline path through the ptr_hash MPHF (BucketVocabStore) instead of AHashMap, submitting the per-position length-probes against the MPHF and bounding the candidate window by the longest vocab token. New PipelineWordPiece mirrors PipelineBPE.

~1.3× faster than the legacy AHashMap greedy path on a BERT-like vocab, byte-exact (asserted).

Why / the idea

Legacy WordPiece probes word[start..end] for end = len .. start+1 one at a time, each a dependent hash lookup. The request was to submit those checks against the PTRHash MPHF as a stream and let prefetch/ILP hide the latency. I built that primitive and measured it honestly — the result is more nuanced than "streaming wins":

Bench decomposition (`cargo bench -p tk-encode --bench wordpiece`, aarch64 M-series, 22k-token vocab, longest 14B)

variant	ns/byte	vs legacy	isolates
V1 legacy — AHashMap, unbounded	41.6	1.00×	baseline
V2 AHashMap, bounded	44.8	0.93×	bounding (helper overhead ≈ neutral)
V3 MPHF `get_bytes`, bounded, scalar	31.7	1.31×	AHashMap → MPHF
V4 MPHF `index_stream`, bounded	38.7	1.08×	streaming

The win is the MPHF lookup itself: swapping AHashMap.get → BucketVocabStore.get_bytes on the same bounded greedy loop is 1.41× (V3 vs V2).
Streaming (index_stream) is slower than scalar on aarch64 (V4 < V3): ptr_hash's software prefetch is commented out on aarch64 (a no-op), and streaming also gives up the greedy early-exit. Its win is x86/x86_64-only and still needs validation on a real x86 host (Intel SDE only emulates functionally, not timing).

PipelineWordPiece therefore ships the scalar bounded path (fastest on arm); BucketVocabStore::longest_prefix_match (the streamed primitive) is kept and benched for the x86/large-vocab case.

Changes

BucketVocabStore::longest_prefix_match — streamed longest-prefix probe (index_stream::<16, true>, MINIMAL=true to match the store's minimal [0,n) slots), byte-verified.
PipelineWordPiece (+ from_wordpiece) — MPHF-backed, bounded longest-first early-exit; wired into PipelineModel::WordPiece.
benches/wordpiece.rs — self-contained (no fixtures), 4-way decomposition + byte-exact gate.
Unit test: pipeline_wordpiece_matches_legacy (byte-exact vs the legacy AHashMap path).

Plan / status

Phase 0 — batched primitive on BucketVocabStore
Phase 1 — PipelineWordPiece, byte-exact, wired in
Phase 2 — micro-bench + honest decomposition
Validate streaming on a real x86 host (where prefetch is active) / large DRAM-resident vocabs
Phase 3 — BPE: stream the merge-seeding (windows(2)) + char-atom vocab lookups (per-step neighbor lookups stay dependent; expect a smaller win)

Notes for reviewers

Draft: the streaming path's payoff is unproven on x86 here; the shipped default (scalar MPHF) is the safe win.
max_token_len bounding is exact (no vocab token exceeds it), so output is identical to the legacy greedy path.

PipelineTokenizer benchmark

5 / 6 models supported — ~10 kB inputs · single thread

b3cf95add · 2026-07-10 10:04 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 8 cores

bert-base-uncased — normalizer-heavy WordPiece · geomean ×4.49

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	7.0	23.6	×3.38	1.22	27.79	9.23	3.83	match
arb_Arab	lang	3.5	18.2	×5.19	1.16	28.00	13.10	12.26	match
ben_Beng	lang	5.2	27.6	×5.31	1.15	19.52	8.58	6.43	match
cmn_Hani	lang	3.2	15.8	×4.96	1.18	38.82	10.12	13.56	match
ell_Grek	lang	3.2	17.5	×5.41	1.17	29.14	13.15	13.24	match
eng_Latn	lang	3.8	14.2	×3.72	2.71	39.35	3.24	24.40	match
heb_Hebr	lang	3.5	15.5	×4.45	1.16	37.90	13.14	11.84	match
hin_Deva	lang	5.5	23.2	×4.19	1.16	27.71	7.04	6.39	match
jpn_Jpan	lang	3.7	21.8	×5.92	1.22	23.48	10.71	10.59	match
kat_Geor	lang	5.7	22.8	×4.03	1.16	26.76	9.60	5.95	match
kor_Hang	lang	2.2	13.4	×6.17	1.31	35.14	21.53	15.93	match
rus_Cyrl	lang	3.2	16.8	×5.29	1.16	27.81	13.42	15.93	match
tam_Taml	lang	6.5	30.1	×4.62	1.16	18.91	8.11	4.74	match
tha_Thai	lang	8.2	27.3	×3.33	1.17	24.92	7.90	2.45	match
added_normalized_dense	modalities	6.1	21.0	×3.46	1.59	40.88	1.13	3.24	match
added_normalized_sparse	modalities	4.8	16.9	×3.52	1.90	40.49	2.42	13.46	match
added_special_dense	modalities	4.8	45.7	×9.57	5.21	9.19	2.21	4.53	match
added_special_sparse	modalities	3.7	19.3	×5.19	3.71	28.31	3.14	15.66	match
agentic-traces	modalities	3.4	12.2	×3.61	2.48	39.10	3.94	36.43	match
agentic_swe	modalities	3.6	12.6	×3.52	1.95	39.43	2.84	34.67	match
code_mixed	modalities	3.5	12.8	×3.70	2.25	39.28	3.28	32.59	match
math_latex	modalities	3.5	13.1	×3.75	2.58	39.22	3.47	30.54	match

deepseek-v4 — split-heavy byte-level BPE, 3 chained regexes · geomean ×2.59

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	3.9	14.5	×3.72	0.65	0.00	43.47	25.81	match
arb_Arab	lang	3.6	9.3	×2.59	0.66	0.00	49.85	58.34	match
ben_Beng	lang	4.5	11.0	×2.46	0.59	0.00	36.90	54.21	match
cmn_Hani	lang	3.3	7.8	×2.38	0.80	0.00	67.13	55.43	match
ell_Grek	lang	3.7	9.9	×2.67	0.60	0.00	48.10	53.08	match
eng_Latn	lang	2.6	6.5	×2.47	2.08	0.00	69.00	80.16	match
heb_Hebr	lang	3.2	7.8	×2.48	0.60	0.00	53.56	71.39	match
hin_Deva	lang	4.2	11.6	×2.76	0.61	0.00	40.72	46.95	match
jpn_Jpan	lang	3.7	8.7	×2.37	0.69	0.00	58.11	53.79	match
kat_Geor	lang	4.4	11.3	×2.58	0.60	0.00	33.61	54.02	match
kor_Hang	lang	3.2	9.6	×3.00	0.61	0.00	54.45	46.22	match
rus_Cyrl	lang	3.5	8.7	×2.49	0.60	0.00	47.91	68.32	match
tam_Taml	lang	4.6	11.2	×2.41	0.59	0.00	33.28	56.28	match
tha_Thai	lang	5.3	10.5	×1.98	0.71	0.00	27.70	65.99	match
added_normalized_dense	modalities	3.7	11.2	×3.05	0.77	0.00	48.29	36.02	match
added_normalized_sparse	modalities	3.5	9.9	×2.81	1.34	0.00	51.70	44.92	match
added_special_dense	modalities	3.0	7.0	×2.37	6.86	0.43	114.11	13.98	match
added_special_sparse	modalities	3.2	7.6	×2.37	4.10	0.07	81.16	47.54	match
agentic-traces	modalities	2.3	6.5	×2.79	1.93	0.00	78.56	77.36	match
agentic_swe	modalities	2.3	5.7	×2.46	1.36	0.00	104.68	66.25	match
code_mixed	modalities	2.5	6.9	×2.72	1.66	0.00	76.30	69.09	match
math_latex	modalities	2.4	6.2	×2.59	2.03	0.04	85.08	71.01	match

gpt2 — standalone ByteLevel pre-tokenizer · geomean ×7.75

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	3.8	38.7	×10.14	0.66	0.00	6.19	18.05	match
arb_Arab	lang	3.1	21.8	×7.14	0.59	0.00	7.38	36.92	match
ben_Beng	lang	2.3	28.5	×12.25	0.59	0.00	11.41	22.55	match
cmn_Hani	lang	3.1	24.5	×7.88	0.61	0.00	5.57	33.59	match
ell_Grek	lang	3.1	24.4	×7.83	0.60	0.00	6.77	33.21	match
eng_Latn	lang	2.8	11.6	×4.19	2.09	0.01	11.25	71.48	match
heb_Hebr	lang	3.0	25.5	×8.47	0.60	0.00	7.58	30.20	match
hin_Deva	lang	2.6	25.9	×9.93	0.60	0.00	11.39	26.06	match
jpn_Jpan	lang	3.5	19.5	×5.58	0.62	0.00	4.93	44.61	match
kat_Geor	lang	3.6	55.7	×15.53	0.60	0.00	4.80	11.80	match
kor_Hang	lang	2.9	32.7	×11.38	0.61	0.00	7.63	21.53	match
rus_Cyrl	lang	3.2	23.9	×7.39	0.60	0.00	6.53	33.56	match
tam_Taml	lang	2.2	34.6	×15.77	0.59	0.00	11.43	16.40	match
tha_Thai	lang	2.8	28.4	×10.01	0.62	0.00	7.37	26.03	match
added_normalized_dense	modalities	3.5	21.2	×6.03	0.75	0.00	6.72	37.97	match
added_normalized_sparse	modalities	3.3	17.8	×5.37	1.31	0.00	8.38	45.30	match
added_special_dense	modalities	3.7	31.7	×8.53	5.24	0.10	9.39	15.07	match
added_special_sparse	modalities	3.3	18.1	×5.51	3.37	0.00	11.78	37.17	match
agentic-traces	modalities	2.4	12.8	×5.27	1.89	0.02	13.72	62.23	match
agentic_swe	modalities	2.5	17.3	×7.01	1.33	0.03	13.15	42.07	match
code_mixed	modalities	2.4	15.2	×6.30	1.65	0.03	13.30	49.85	match
math_latex	modalities	2.6	12.2	×4.70	2.16	0.00	12.72	65.93	match

llama-2 — model-bounded BPE, no pre-tokenizer · geomean ×3.01

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	4.4	31.6	×7.25	0.04	13.52	0.00	16.73	match
arb_Arab	lang	9.1	32.3	×3.57	0.03	15.71	0.00	14.14	match
ben_Beng	lang	9.8	52.9	×5.40	0.04	10.60	0.00	7.39	match
cmn_Hani	lang	8.0	61.8	×7.73	0.05	2.89	0.00	12.37	match
ell_Grek	lang	9.1	36.7	×4.05	0.03	14.67	0.00	11.43	match
eng_Latn	lang	4.0	5.8	×1.43	0.05	25.61	0.08	148.96	match
heb_Hebr	lang	8.8	35.9	×4.09	0.03	16.24	0.00	10.36	match
hin_Deva	lang	10.6	44.8	×4.23	0.03	13.57	0.00	7.61	match
jpn_Jpan	lang	11.2	77.5	×6.94	0.05	2.65	0.00	9.56	match
kat_Geor	lang	12.6	58.4	×4.62	0.05	8.98	0.01	7.26	match
kor_Hang	lang	6.5	33.4	×5.16	0.05	15.57	0.00	12.72	match
rus_Cyrl	lang	7.8	14.0	×1.79	0.03	13.06	0.00	56.77	match
tam_Taml	lang	11.0	59.9	×5.44	0.03	8.27	0.00	7.65	match
tha_Thai	lang	13.8	71.7	×5.18	0.03	4.51	0.00	8.75	match
added_normalized_dense	modalities	5.2	8.3	×1.61	0.07	17.69	0.00	104.58	match
added_normalized_sparse	modalities	4.7	6.8	×1.44	0.07	24.76	0.00	121.11	match
added_special_dense	modalities	4.7	12.3	×2.61	5.64	39.61	1.64	29.58	match
added_special_sparse	modalities	6.4	7.8	×1.22	2.30	41.79	0.66	80.54	match
agentic-traces	modalities	4.2	6.3	×1.51	0.26	28.31	0.00	131.56	match
agentic_swe	modalities	3.8	5.8	×1.52	0.06	51.51	0.00	118.04	match
code_mixed	modalities	3.9	6.0	×1.53	0.03	38.01	0.00	130.12	match
math_latex	modalities	4.1	6.0	×1.47	0.06	27.86	0.07	138.77	match

llama-3 — split-heavy byte-level BPE, single regex · geomean ×3.92

Fixture	Group	Tokenizer MB/s	Pipeline MB/s	Speedup	added ns/B	norm ns/B	pre-tok ns/B	model ns/B	Ids
amh_Ethi	lang	3.2	22.1	×6.99	0.65	0.00	28.77	18.69	match
arb_Arab	lang	3.8	11.8	×3.12	0.60	0.08	32.88	55.40	match
ben_Beng	lang	2.9	12.8	×4.43	0.60	0.00	48.09	36.68	match
cmn_Hani	lang	4.0	12.9	×3.21	0.61	0.00	20.71	56.76	match
ell_Grek	lang	3.9	12.7	×3.24	0.60	0.00	29.64	51.52	match
eng_Latn	lang	3.5	14.6	×4.12	2.14	0.00	46.53	29.48	match
heb_Hebr	lang	3.0	14.8	×4.96	0.60	0.00	32.53	32.56	match
hin_Deva	lang	3.8	16.7	×4.39	0.60	0.00	50.89	16.57	match
jpn_Jpan	lang	4.5	12.8	×2.87	0.60	0.00	19.24	57.72	match
kat_Geor	lang	3.9	23.9	×6.19	0.59	0.01	17.14	23.74	match
kor_Hang	lang	3.5	12.5	×3.57	0.61	0.00	31.25	48.06	match
rus_Cyrl	lang	4.0	11.7	×2.92	0.60	0.00	28.65	58.24	match
tam_Taml	lang	2.9	14.0	×4.89	0.61	0.00	44.52	28.45	match
tha_Thai	lang	4.4	13.4	×3.02	0.61	0.00	29.62	46.23	match
added_normalized_dense	modalities	4.1	16.1	×3.97	0.74	0.00	25.01	38.96	match
added_normalized_sparse	modalities	4.2	18.1	×4.32	1.30	0.00	33.34	24.58	match
added_special_dense	modalities	3.6	12.7	×3.57	5.21	0.18	69.99	8.51	match
added_special_sparse	modalities	4.0	15.3	×3.79	3.36	0.00	55.75	10.92	match
agentic-traces	modalities	3.2	12.5	×3.86	1.93	0.00	55.83	27.62	match
agentic_swe	modalities	3.3	10.9	×3.26	1.36	0.00	57.18	41.17	match
code_mixed	modalities	3.5	13.5	×3.92	1.66	0.04	54.54	13.78	match
math_latex	modalities	3.3	13.0	×3.99	2.13	0.00	53.59	29.15	match

Not yet supported: t5-base

Route WordPiece's pipeline path through the ptr_hash MPHF (BucketVocabStore) instead of AHashMap, and bound the candidate length by the longest vocab token (a match can never be longer). ~1.3x faster than the legacy AHashMap greedy path on a BERT-like vocab (22k tokens, longest 14B), byte-exact. - BucketVocabStore::longest_prefix_match: streamed longest-prefix probe over ptr_hash index_stream (prefetch). Kept for the x86 / large-vocab case; on aarch64 ptr_hash's prefetch is a no-op so scalar wins there. - PipelineWordPiece: scalar bounded longest-first early-exit over BucketVocabStore.get_bytes (fastest on aarch64); wired into PipelineModel. - benches/wordpiece.rs: decomposes the win (bounding / AHashMap->MPHF / streaming) with a byte-exact gate across all variants.

HuggingFaceDocBuilderDev · 2026-07-09T20:02:22Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

PipelineWordPiece routes the pipeline WordPiece path through the ptr_hash MPHF (BucketVocabStore) and, at each position, probes only the vocab's distinct token byte-lengths (descending, first hit = greedy longest) instead of shrinking one char at a time. Byte-exact with the legacy AHashMap path, verified token-for-token on the full bert-base-uncased vocab over ~200k real words. benches/wordpiece.rs decomposes the win (bounding / AHashMap->MPHF / streaming / length-set / first-byte length-set) with a byte-exact gate; loads a real vocab + corpus via WP_VOCAB / WP_CORPUS env vars. Investigation notes (kept ahash, no store change): - Fx pre-hash is ~2x faster in isolation but produces 64-bit collisions on the real bert vocab -> silently drops a token -> wrong split. ahash stays. - first-byte-bucketed length sets probe fewer lengths but are within noise of the flat union list on real data; not worth the Vec<Vec> indirection. - index_stream prefetch is a no-op on aarch64, so streaming loses there.

Profiling on the real bert-base-uncased vocab: the WordPiece model spent ~half its time in a per-word heap alloc (a fresh scratch Vec per tokenize_pipeline call). Reuse a thread-local scratch buffer → zero alloc after warmup, ~2x on the model lookup. Real WordPiece probes only ~1.2 lengths/word, so the distinct-length probe stays the win; MPHF vs AHashMap backing measured identical (within 2%), so the MPHF stays for its ~3x smaller footprint. Also drop the ptr_hash index_stream path (BucketVocabStore::longest_prefix_match + STREAM_PREFETCH): no win on aarch64 (prefetch is a no-op there) and it gave up the greedy early-exit. Remove the wordpiece micro-bench; the CI pipeline bench on bert covers this end-to-end.

Completes the previous commit (which only caught the bench deletion via a bad add): - PipelineWordPiece: reuse a thread-local scratch buffer instead of a per-word Vec. Profiling on the real bert vocab showed the per-word alloc was ~half the model lookup cost; reusing it is ~2x. Real WordPiece probes only ~1.2 lengths/word. - BucketVocabStore: drop the index_stream longest_prefix_match + STREAM_PREFETCH (no win on aarch64, gave up greedy early-exit). - Cargo.toml: remove the deleted wordpiece bench entry.

Shave the non-lookup cost the profile flagged (byte-exact, still MPHF + distinct-length): - max-chars guard: only run the O(len) chars().count() when byte-len could exceed the cap (bytes >= chars), so normal words skip the decode. - word-initial candidates (start==0, no prefix) probe the word slice directly — no copy; only continuations touch the scratch buffer. - keep the constant continuing-prefix in the scratch across the word; continuations rewrite only the word bytes (truncate+append), not the prefix. - BucketVocabStore: lay the byte slab out in sorted token order so tokens sharing a prefix are contiguous → a word's successive length-probes verify against nearby slab bytes (build-time only; no query cost).

Add a [u16;256] max-token-length-per-first-byte table (word-initial + ##-continuation), L1-resident, no indirection. At each position: if no vocab token starts with the byte at word[start] (table == 0) emit [UNK] with zero get_bytes probes; otherwise cap the probe window to the longest token that could match there. bert-base-uncased has no Hangul, so every Korean word was probing ~all lengths at start=0 (all misses) before falling to unk — the model measured 78 ns/byte of pure waste. Now Korean is byte-exact at ~0.8 ns/byte (~90x). English is unchanged and byte-exact (~7.3 ns/byte): the flat union length-set still does the real probing, so there's no Vec<Vec> per-position indirection.

PipelineWordPiece no longer hand-rolls its own length-set / first-byte / reject probe — it reuses `buckets::Buckets` (the added-vocabulary matcher), which already had all of it and more: first-byte reject, longest-common-prefix u64 reject, post-byte length sub-lists (a second discriminator, tighter than my per-first-byte), MPHF verify, and a SIMD scan. Expose `Buckets::longest_at` — the anchored longest-match-at-a-position primitive WordPiece needs (`match_bytes` scans text; this doesn't). Word-initial candidates match the raw word bytes (no copy); continuations match `["##"] + word[start..]` in a thread-local scratch capped at the longest ## token. Byte-exact vs the legacy path on the full bert vocab (English + Korean): ~8 ns/byte English, ~0.7 ns/byte Korean (out-of-vocab scripts reject with zero probes).

…nt-byte discrimination The Buckets refactor merged all tokens into one matcher and routed continuations through the '#' bucket by matching "##"+word. bert has a plain "#" token, so that bucket's common prefix collapses to empty (plen = min(lcp, min_len-1) = min(1,0) = 0) and every continuation was keyed by post-byte '#' — i.e. probed the length list of ALL ~5800 '##' tokens, longest-first, before hitting. That's what kept kor_Hang slow: the bert normalizer NFD-decomposes Hangul syllables into jamo (U+11xx, lead 0xE1) and bert has ~35 jamo tokens, so the model does real greedy WordPiece over jamo, and every jamo continuation paid ~13 dead length probes. Fix: two matchers. - `initial`: all tokens, keyed by first byte, used only at start==0 (raw-word match, stays legacy-exact even for a literal "##" word). - `cont`: '##' tokens with the prefix stripped, keyed by the CONTENT byte, used at start>0. The content byte now selects a short per-byte length list, and matching word[start..] directly also drops the "##"+word scratch copy (removes the thread-local buffer and max_cont_word entirely). Isolated jamo model path: 12.99 -> 5.36 ns/byte (2.4x), now near the MPHF-confirm floor (~1 get_bytes per jamo). Byte-exact: pipeline_wordpiece_matches_legacy extended with a plain "#" token (reproduces the collapsed bucket) and literal-# words.

ArthurZucker added the run-pipeline-bench label Jul 10, 2026