Skip to content

Pull requests: huggingface/tokenizers

New pull request New
105 Open 868 Closed
105 Open 868 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

feat: add MultiRegex for common regexes (gpt2,etc) fast path
#2143 opened Jul 2, 2026 by McPatate Member Loading…
Fast encode fixes
#2142 opened Jul 2, 2026 by SBrandeis Contributor Loading…
1
ci: Simple bench workflow for PipelineTokenizer
#2141 opened Jul 2, 2026 by SBrandeis Contributor Loading…
1
feat: impl pipeline::PreTokenizer for WhitespaceSplit
#2140 opened Jul 1, 2026 by McPatate Member Loading…
1
feat: handle Merge* variants of the SplitDelimiterBehavior
#2139 opened Jul 1, 2026 by McPatate Member Loading…
1
feat: impl pipeline::PreTokenizer for UnicodeScripts
#2138 opened Jul 1, 2026 by McPatate Member Loading…
2
feat: impl pipeline::PreTokenizer for FixedLength
#2137 opened Jul 1, 2026 by McPatate Member Loading…
1
feat: impl pipeline::PreTokenizer for Digits
#2136 opened Jul 1, 2026 by McPatate Member Loading…
1
feat: impl pipeline::PreTokenizer for CharDelimiterSplit
#2135 opened Jul 1, 2026 by McPatate Member Loading…
1
refactor: extract pretok algo
#2134 opened Jul 1, 2026 by McPatate Member Loading…
1
No alloc normalizers
#2133 opened Jul 1, 2026 by SBrandeis Contributor • Draft
4
chore(deps): bump js-yaml from 3.14.2 to 3.15.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2131 opened Jun 30, 2026 by dependabot Bot Loading…
1
plumbing: full pipeline
#2130 opened Jun 30, 2026 by SBrandeis Contributor Loading…
11
docs: add docstrings to Tokenizer::from_file, from_bytes, and from_petrained
#2124 opened Jun 23, 2026 by BBloggsbott Loading…
feat: train-encode split
#2119 opened Jun 18, 2026 by McPatate Member Loading…
4
chore(deps-dev): bump webpack-dev-server from 5.2.1 to 5.2.5 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2118 opened Jun 18, 2026 by dependabot Bot Loading…
1
refactor: core crate layout
#2114 opened Jun 16, 2026 by McPatate Member • Draft
chore(deps): bump @babel/core from 7.24.3 to 7.29.7 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2113 opened Jun 16, 2026 by dependabot Bot Loading…
1
chore(deps-dev): bump launch-editor from 2.10.0 to 2.14.1 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2112 opened Jun 16, 2026 by dependabot Bot Loading…
1
BPE trainer: inherit continuing_subword_prefix/end_of_word_suffix from the model
#2108 opened Jun 13, 2026 by discobot Loading…
1
Persist encode_special_tokens across save/load
#2107 opened Jun 13, 2026 by discobot Loading…
1
Fix Unigram trainer prune loss to use per-piece alternative count
#2106 opened Jun 13, 2026 by NahButch Loading…
1
Fix i32 overflow in BPE trainer pair counts
#2105 opened Jun 13, 2026 by NahButch Loading…
1
Return an error instead of panicking on out-of-range BPE merges
#2104 opened Jun 12, 2026 by NahButch Loading…
1
Fix empty Encoding.overflowing when truncation is enabled
#2098 opened Jun 12, 2026 by discobot Loading…
1
ProTip! Updated in the last three days: updated:>2026-06-29.