Skip to content

Pull requests: huggingface/tokenizers

New pull request New
142 Open 934 Closed
142 Open 934 Closed
Author
Filter by author
Loading
Label
Filter by label
Loading
Use alt + click/return to exclude labels
or ⇧ + click/return for logical OR
Projects
Filter by project
Loading
Milestones
Filter by milestone
Loading
Reviews
Filter by reviews
No reviews Review required Approved review Changes requested
Assignee
Filter by who’s assigned
Assigned to nobody Loading
Sort
Sort by
Newest Oldest Most commented Least commented Recently updated Least recently updated Best match
Most reactions
👍 👎 😄 🎉 😕 ❤️ 🚀 👀

Pull requests list

chore(deps): bump braces from 3.0.2 to 3.0.3 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2251 opened Jul 27, 2026 by dependabot Bot Loading…
1
chore(deps-dev): bump http-proxy-middleware from 2.0.9 to 2.0.10 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2250 opened Jul 27, 2026 by dependabot Bot Loading…
1
chore(deps): bump uuid and webpack-dev-server in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2249 opened Jul 27, 2026 by dependabot Bot Loading…
1
WordCache: one generation, evicting in place
#2247 opened Jul 27, 2026 by SBrandeis Contributor • Draft
2
Fix usize underflow in BPE / Strip decoders on empty / short input
#2245 opened Jul 27, 2026 by weixlu Loading…
chore(deps-dev): bump shell-quote from 1.8.3 to 1.10.0 in /tokenizers/examples/unstable_wasm/www dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2244 opened Jul 25, 2026 by dependabot Bot Loading…
1
Update fancy-regex dependency to 0.18
#2243 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
Update monostate dependency to v1
#2242 opened Jul 25, 2026 by musicinmybrain Contributor Loading…
Perf/bpe merge
#2241 opened Jul 24, 2026 by ArthurZucker Collaborator Loading…
1
perf(pipeline): BPE word cache: bench different cache architectures
#2240 opened Jul 24, 2026 by SBrandeis Contributor • Draft
1
Fix panic in Strip decoder on short all-content tokens
#2239 opened Jul 24, 2026 by v-code01 Loading…
docs: restore 2 dead link(s) via Wayback
#2238 opened Jul 23, 2026 by mirkosalvato1-ctrl Loading…
1
feat(pipeline+bindings): Support encoding pairs of sequences
#2237 opened Jul 23, 2026 by SBrandeis Contributor • Draft
2
perf(tk-encode): SIMD-first encode pipeline — fused split cache + de-virtualized BPE
#2234 opened Jul 23, 2026 by ArthurZucker Collaborator • Draft
bench: gigatoken per-step profile + take/have/skip analysis
#2233 opened Jul 23, 2026 by ArthurZucker Collaborator • Draft
1
implement ultra-basic, erroneous decode
#2232 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
feat(bindings): Python bindings rework, plugging PipelineTokenizer
#2229 opened Jul 22, 2026 by SBrandeis Contributor • Draft
1
chore(deps): bump shell-quote from 1.8.1 to 1.10.0 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2224 opened Jul 20, 2026 by dependabot Bot Loading…
1
feat(bpe): ScratchPool pattern + BPE word cache
#2223 opened Jul 20, 2026 by SBrandeis Contributor Loading…
30
chore(deps): bump brace-expansion from 1.1.13 to 1.1.16 in /bindings/node dependencies Pull requests that update a dependency file javascript Pull requests that update Javascript code
#2222 opened Jul 20, 2026 by dependabot Bot Loading…
1
bench(pipeline): add sebpop/upstream as a second reference series
#2221 opened Jul 20, 2026 by SBrandeis Contributor • Draft
2
Fix malformed BPE merges panicking during build
#2219 opened Jul 19, 2026 by 0xzr Loading…
1
Preserve backslashes in generated Python stub docstrings
#2218 opened Jul 18, 2026 by cupkk Loading…
1
5
Add a ParityBpeTrainer example and list it in the trainers API docs
#2217 opened Jul 18, 2026 by cimeister Contributor Loading…
Fix XSS in EncodingVisualizer HTML attributes
#2216 opened Jul 18, 2026 by varneeth01 Loading…
1
1
ProTip! Adding no:label will show everything without a label.