Search results

Fix nativ tok

Status: Open (in progress).

huggingface/transformersnumber 42874

#42874 In huggingface/transformers;

· itazap opened on Dec 15, 2025

validate tokenizer components

Status: Open (in progress).

huggingface/transformersnumber 42816

#42816 In huggingface/transformers;

· itazap opened on Dec 11, 2025

refactor more tokenizers - v5 guide update

Status: Open (in progress).

huggingface/transformersnumber 42768

#42768 In huggingface/transformers;

· itazap opened on Dec 10, 2025

qwen2 tok regression

Status: Open (in progress).

huggingface/transformersnumber 42644

#42644 In huggingface/transformers;

· itazap opened on Dec 5, 2025

One tok typing

Status: Open (in progress).

huggingface/transformersnumber 42437

#42437 In huggingface/transformers;

· itazap opened on Nov 27, 2025

initial clean

Status: Open (in progress).

huggingface/transformersnumber 42415

#42415 In huggingface/transformers;

· itazap opened on Nov 26, 2025

fix mllama integration tests

Status: Open (in progress).

huggingface/transformersnumber 39785

#39785 In huggingface/transformers;

· itazap opened on Jul 30, 2025

[WIP] Tokenizer Refactor

Status: Open (in progress).

huggingface/transformersnumber 38400

#38400 In huggingface/transformers;

· itazap opened on May 27, 2025

[WIP] new BLT

Status: Draft (not ready).

huggingface/transformersnumber 38173

#38173 In huggingface/transformers;

· itazap opened on May 16, 2025

proof of concept for using dataset of test cases for tokenizer tests

Status: Open (in progress).

huggingface/transformersnumber 37994

#37994 In huggingface/transformers;

· itazap opened on May 7, 2025

general spm converter

Status: Draft (not ready).

huggingface/transformersnumber 37763

#37763 In huggingface/transformers;

· itazap opened on Apr 24, 2025

Fast tokenizer encoding doesn't handle empty string input

Status: Open (in progress).

huggingface/transformersnumber 37537

#37537 In huggingface/transformers;

· itazap opened on Apr 15, 2025