Collapse file tree

Files

 main

Search this repository

/

test

/

Copy path

Directory actions

More options

More options

Directory actions

More options

More options

Latest commit

bigfootjon

Open commit detailssuccess

Jul 23, 2025

287ef1a · Jul 23, 2025

History

History

Open commit details

/

test

/

Top

Folders and files

Name		Name	Last commit message	Last commit date
parent directory ..
fb		fb	Port string_integer_map and changes to tiktoken to pytorch	Mar 27, 2025
resources		resources	Support gemma3 HF tokenizer.json	Jul 8, 2025
CMakeLists.txt		CMakeLists.txt	Enable install find package (#82)	Jun 12, 2025
targets.bzl		targets.bzl	Only run python test in fbcode	Jul 11, 2025
test_base64.cpp		test_base64.cpp	Fix CQS signal. Id] 29511954 -- readability-redundant-string-init in …	Apr 24, 2025
test_hf_tokenizer.cpp		test_hf_tokenizer.cpp	Use a small tokenizer.json for unit test (#92) (#94)	Jul 2, 2025
test_hf_tokenizer.py		test_hf_tokenizer.py	Fix lint issues (#108)	Jul 17, 2025
test_isolated_behavior.cpp		test_isolated_behavior.cpp	Add support for more behavior in Split pretokenizer (#93)	Jul 2, 2025
test_llama2c_tokenizer.cpp		test_llama2c_tokenizer.cpp	clang format the rest of code (#32)	Mar 14, 2025
test_normalizer.cpp		test_normalizer.cpp	Support gemma3 HF tokenizer.json	Jul 8, 2025
test_pre_tokenizer.cpp		test_pre_tokenizer.cpp	Add support for more behavior in Split pretokenizer (#93)	Jul 2, 2025
test_python_bindings.py		test_python_bindings.py	Add python bindings	Jul 11, 2025
test_regex.cpp		test_regex.cpp	Add pcre2 as re2 fallback (#50) (#50)	Apr 21, 2025
test_sentencepiece.cpp		test_sentencepiece.cpp	clang format the rest of code (#32)	Mar 14, 2025
test_string_integer_map.cpp		test_string_integer_map.cpp	Port string_integer_map and changes to tiktoken to pytorch	Mar 27, 2025
test_tiktoken.cpp		test_tiktoken.cpp	Accept custom pattern string and special tokens	May 8, 2025
test_tiktoken.py		test_tiktoken.py	Add python bindings	Jul 11, 2025
test_token_decoder.cpp		test_token_decoder.cpp	[hf] Add features to HF tokenizer (#87)	Jun 26, 2025
View all files