git-market/rasbt-LLMs-from-scratch
1
0
Fork 0
mirror of https://github.com/rasbt/LLMs-from-scratch.git synced 2026-04-11 02:11:40 +08:00
Code Issues Actions Packages Projects Releases Wiki Activity
main
rasbt-LLMs-from-scratch/ch05/09_extending-tokenizers
History
Sebastian Raschka c21bfe4a23
Add PyPI package (#576) 
* Add PyPI package

* fixes

* fixes
2025-03-23 19:28:49 -05:00
..
extend-tiktoken.ipynb Add PyPI package (#576) 2025-03-23 19:28:49 -05:00
README.md Bonus material: extending tokenizers (#496) 2025-01-22 09:26:54 -06:00

README.md

Extending the Tiktoken BPE Tokenizer with New Tokens

  • extend-tiktoken.ipynb contains optional (bonus) code to explain how we can add special tokens to a tokenizer implemented via tiktoken and how to update the LLM accordingly