Fix #1256 refactor Token Constants, test vocab init/build behaviour

[asgalon]

Changes made:

• Extracted constants into Tokenizer for special tokens and
• Modified test_unit_char_tokenizer() to check expected behaviour build_vocab overwriting init vocabulary
• adjusted documentation to constant use

---

By submitting this PR, you agree to release your contribution under the project's CC-BY-NC-ND license.

[profvjreddi]

Great contribution, Peter — thanks for tackling #1256! 🎉

The core refactor (extracting '<UNK>' and '</w>' into Tokenizer.TOK_UNKNOWN / Tokenizer.TOK_EOW class constants) is solid, and the improved test_unit_char_tokenizer that verifies build_vocab actually overwrites the init vocabulary is a real improvement.

A few minor things I'll clean up post-merge:

1. Incomplete docstring migration — Some >>> expected output lines in docstrings still show raw 'o</w>' while inputs were updated to use Tokenizer.TOK_EOW. I'll make these consistent.

2. ASCII diagrams → revert to literal values — The box-drawing diagrams are pedagogical visualizations meant to be read by students, not executed. Replacing 'o</w>' with 'o'+Tokenizer.TOK_EOW in those diagrams adds cognitive load without maintainability benefit. I'll revert the diagram text back to concrete string values while keeping the constants in all executable code.

3. f-string style nit — Minor inconsistency in assertion error message interpolation; I'll tidy that up.

Merging now — thanks again for the clean refactor!

[profvjreddi]

@all-contributors please add @asgalon as a contributor for 🧑‍💻 Code in TinyTorch

[github-actions]

I've added @asgalon as a contributor to tinytorch! 🎉

Recognized for: doc
Project(s): tinytorch (explicitly mentioned in comment)
Based on: @all-contributors please add @asgalon as a contributor for 🧑‍💻 Code in TinyTorch

The contributor list has been updated in:

• tinytorch/.all-contributorsrc, tinytorch/README.md
• Main README.md

We love recognizing our contributors! ❤️