Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers

Repository containing code and results for our EMNLP 2025 paper on incomplete tokens in byte-level tokenizers.

Citation

If you found these ideas useful, please cite the following paper:

Eugene Jang, Kimin Lee, Jin-Woo Chung, Keuntae Park, and Seungwon Shin. 2025. Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers. In Proceedings of the 2025 Conference on Empirical Methods in Natural Language Processing, pages 18220–18227, Suzhou, China. Association for Computational Linguistics.

Name		Name	Last commit message	Last commit date
Bigrams		Bigrams
GlitchReports		GlitchReports
.gitignore		.gitignore
LICENSE		LICENSE
README.md		README.md
bpe_utils.py		bpe_utils.py
create_bigrams.py		create_bigrams.py
create_comparable_bigrams.py		create_comparable_bigrams.py
do_task_alternative_tok.py		do_task_alternative_tok.py
do_task_bigrams.py		do_task_bigrams.py
grading.py		grading.py
shorthands.py		shorthands.py
tool_task.py		tool_task.py
visualize.ipynb		visualize.ipynb

Navigation Menu

Repository files navigation

Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers

Citation

About

Releases

Packages

Contributors

Languages

Latest commit

History

Folders and files

Repository files navigation

Improbable Bigrams Expose Vulnerabilities of Incomplete Tokens in Byte-Level Tokenizers

Citation

About

Topics

Resources

Stars

Watchers

Forks

Releases

Packages

Contributors

Languages