ci: bench against latest release, measure memory consumption and executable size

[SBrandeis]

PipelineTokenizer benchmark

5 / 6 models supported — PipelineTokenizer vs tokenizers v0.23.1 (latest release) · ~10 kB inputs · single thread

f41341f18 · 2026-07-09 18:57 UTC · Intel(R) Xeon(R) Platinum 8375C CPU @ 2.90GHz · 16 cores

bert-base-uncased — normalizer-heavy WordPiece · ×4.28 vs v0.23.1

Memory (RSS MB, load+encode): v0.23.1 2+0 (peak 6) · Pipeline 5+0 (peak 6)

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Ids
amh_Ethi	lang	7.2	21.7	×3.02	match
arb_Arab	lang	3.5	14.9	×4.29	match
ben_Beng	lang	5.2	22.0	×4.25	match
cmn_Hani	lang	3.2	14.4	×4.53	match
ell_Grek	lang	3.2	14.0	×4.34	match
eng_Latn	lang	3.8	16.1	×4.19	match
heb_Hebr	lang	3.5	13.1	×3.74	match
hin_Deva	lang	5.6	20.3	×3.64	match
jpn_Jpan	lang	3.7	17.3	×4.67	match
kat_Geor	lang	5.7	18.5	×3.27	match
kor_Hang	lang	2.1	9.3	×4.38	match
rus_Cyrl	lang	3.1	13.8	×4.40	match
tam_Taml	lang	6.4	24.5	×3.81	match
tha_Thai	lang	8.2	21.9	×2.68	match
added_normalized_dense	modalities	6.2	21.8	×3.54	match
added_normalized_sparse	modalities	4.8	19.2	×3.97	match
added_special_dense	modalities	4.8	50.6	×10.58	match
added_special_sparse	modalities	3.7	22.9	×6.19	match
agentic-traces	modalities	3.3	15.3	×4.64	match
agentic_swe	modalities	3.6	16.1	×4.54	match
code_mixed	modalities	3.4	15.9	×4.66	match
math_latex	modalities	3.4	15.6	×4.55	match

deepseek-v4 — split-heavy byte-level BPE, 3 chained regexes · ×2.66 vs v0.23.1

Memory (RSS MB, load+encode): v0.23.1 51+0 (peak 58) · Pipeline 84+0 (peak 84)

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Ids
amh_Ethi	lang	3.7	14.3	×3.84	match
arb_Arab	lang	3.6	9.2	×2.59	match
ben_Beng	lang	4.4	11.1	×2.54	match
cmn_Hani	lang	3.2	8.1	×2.51	match
ell_Grek	lang	3.7	9.9	×2.70	match
eng_Latn	lang	2.6	6.5	×2.50	match
heb_Hebr	lang	3.0	8.0	×2.67	match
hin_Deva	lang	4.1	12.1	×2.97	match
jpn_Jpan	lang	3.7	8.8	×2.42	match
kat_Geor	lang	4.4	11.6	×2.65	match
kor_Hang	lang	3.2	9.9	×3.14	match
rus_Cyrl	lang	3.4	8.7	×2.54	match
tam_Taml	lang	4.7	11.3	×2.40	match
tha_Thai	lang	5.4	10.7	×1.99	match
added_normalized_dense	modalities	3.5	12.0	×3.45	match
added_normalized_sparse	modalities	3.5	10.1	×2.92	match
added_special_dense	modalities	2.9	7.1	×2.44	match
added_special_sparse	modalities	3.1	7.4	×2.38	match
agentic-traces	modalities	2.4	6.3	×2.69	match
agentic_swe	modalities	2.3	5.7	×2.51	match
code_mixed	modalities	2.6	6.9	×2.68	match
math_latex	modalities	2.4	6.2	×2.58	match

gpt2 — standalone ByteLevel pre-tokenizer · ×7.91 vs v0.23.1

Memory (RSS MB, load+encode): v0.23.1 17+1 (peak 18) · Pipeline 21+0 (peak 21)

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Ids
amh_Ethi	lang	3.9	42.0	×10.91	match
arb_Arab	lang	3.1	22.5	×7.32	match
ben_Beng	lang	2.2	28.9	×13.38	match
cmn_Hani	lang	3.2	24.5	×7.75	match
ell_Grek	lang	3.1	25.1	×7.99	match
eng_Latn	lang	2.7	11.8	×4.32	match
heb_Hebr	lang	3.1	24.6	×7.85	match
hin_Deva	lang	2.6	26.4	×10.21	match
jpn_Jpan	lang	3.5	19.7	×5.62	match
kat_Geor	lang	3.6	53.9	×15.03	match
kor_Hang	lang	3.0	32.6	×10.95	match
rus_Cyrl	lang	3.3	24.1	×7.19	match
tam_Taml	lang	2.2	35.5	×16.35	match
tha_Thai	lang	2.9	29.2	×10.11	match
added_normalized_dense	modalities	3.6	22.7	×6.34	match
added_normalized_sparse	modalities	3.4	18.7	×5.53	match
added_special_dense	modalities	3.5	32.0	×9.15	match
added_special_sparse	modalities	3.3	19.0	×5.77	match
agentic-traces	modalities	2.4	13.0	×5.46	match
agentic_swe	modalities	2.4	17.5	×7.26	match
code_mixed	modalities	2.4	15.5	×6.46	match
math_latex	modalities	2.6	12.4	×4.80	match

llama-2 — model-bounded BPE, no pre-tokenizer · ×3.04 vs v0.23.1

Memory (RSS MB, load+encode): v0.23.1 15+0 (peak 15) · Pipeline 14+0 (peak 14)

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Ids
amh_Ethi	lang	4.2	29.3	×6.94	match
arb_Arab	lang	8.8	32.3	×3.66	match
ben_Beng	lang	9.6	52.2	×5.44	match
cmn_Hani	lang	8.1	62.4	×7.69	match
ell_Grek	lang	8.7	36.5	×4.21	match
eng_Latn	lang	3.9	5.7	×1.44	match
heb_Hebr	lang	8.5	35.9	×4.24	match
hin_Deva	lang	10.5	44.9	×4.27	match
jpn_Jpan	lang	11.4	78.3	×6.85	match
kat_Geor	lang	12.4	56.8	×4.57	match
kor_Hang	lang	6.3	33.5	×5.31	match
rus_Cyrl	lang	7.9	14.5	×1.82	match
tam_Taml	lang	11.1	59.5	×5.38	match
tha_Thai	lang	13.7	72.5	×5.29	match
added_normalized_dense	modalities	5.1	8.4	×1.65	match
added_normalized_sparse	modalities	4.7	6.9	×1.48	match
added_special_dense	modalities	4.4	11.4	×2.61	match
added_special_sparse	modalities	6.4	7.8	×1.20	match
agentic-traces	modalities	4.3	6.5	×1.50	match
agentic_swe	modalities	3.9	6.1	×1.59	match
code_mixed	modalities	4.1	6.3	×1.54	match
math_latex	modalities	4.2	6.3	×1.48	match

llama-3 — split-heavy byte-level BPE, single regex · ×4.06 vs v0.23.1

Memory (RSS MB, load+encode): v0.23.1 128+0 (peak 189) · Pipeline 129+0 (peak 189)

Fixture	Group	v0.23.1 MB/s	Pipeline MB/s	Speedup	Ids
amh_Ethi	lang	3.1	21.9	×7.02	match
arb_Arab	lang	3.7	11.9	×3.22	match
ben_Beng	lang	2.8	13.5	×4.90	match
cmn_Hani	lang	3.9	13.0	×3.34	match
ell_Grek	lang	3.9	12.7	×3.24	match
eng_Latn	lang	3.4	13.6	×3.96	match
heb_Hebr	lang	3.2	15.0	×4.75	match
hin_Deva	lang	3.7	16.8	×4.58	match
jpn_Jpan	lang	4.3	13.0	×3.03	match
kat_Geor	lang	3.8	23.8	×6.26	match
kor_Hang	lang	3.4	12.4	×3.64	match
rus_Cyrl	lang	3.8	11.9	×3.14	match
tam_Taml	lang	2.8	13.6	×4.88	match
tha_Thai	lang	4.2	13.5	×3.19	match
added_normalized_dense	modalities	3.7	16.8	×4.58	match
added_normalized_sparse	modalities	3.9	18.4	×4.67	match
added_special_dense	modalities	3.5	12.6	×3.62	match
added_special_sparse	modalities	3.8	15.2	×4.01	match
agentic-traces	modalities	3.1	12.4	×3.99	match
agentic_swe	modalities	3.2	11.0	×3.45	match
code_mixed	modalities	3.3	13.0	×3.89	match
math_latex	modalities	3.1	13.0	×4.24	match

Not yet supported: t5-base

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.