[{"content":"How Spacing and Capitalization Randomly Change Your Model\u0026rsquo;s Entire Personality Add a space before your prompt and watch GPT become 30% dumber. Write in ALL CAPS and suddenly it\u0026rsquo;s aggressive. Use \u0026ldquo;pls\u0026rdquo; instead of \u0026ldquo;please\u0026rdquo; and it becomes casual. This isn\u0026rsquo;t personality, it\u0026rsquo;s tokenization chaos triggering different training data pockets.\nTL;DR: \u0026quot; Hello\u0026quot; and \u0026ldquo;Hello\u0026rdquo; activate completely different neural pathways. \u0026ldquo;HELP\u0026rdquo; vs \u0026ldquo;help\u0026rdquo; vs \u0026ldquo;Help\u0026rdquo; pulls from different training contexts (emergency manuals vs casual chat vs formal documents). Your model doesn\u0026rsquo;t have moods, it has tokenization triggered personality disorders.\nThe Space That Breaks Everything # The leading space disaster - TEST THIS NOW: def space_personality_test(model, tokenizer): \u0026#34;\u0026#34;\u0026#34;Watch a single space change everything\u0026#34;\u0026#34;\u0026#34; prompts = [ \u0026#34;Hello\u0026#34;, # Normal \u0026#34; Hello\u0026#34;, # Leading space \u0026#34; Hello\u0026#34;, # Two spaces \u0026#34;\\nHello\u0026#34;, # Newline start \u0026#34;\\tHello\u0026#34;, # Tab start \u0026#34;hello\u0026#34;, # Lowercase \u0026#34;HELLO\u0026#34;, # Uppercase \u0026#34;HeLLo\u0026#34;, # Mixed case chaos ] for prompt in prompts: tokens = tokenizer.encode(prompt) response = model.generate(prompt + \u0026#34;, how are you?\u0026#34;) print(f\u0026#34;Prompt: \u0026#39;{prompt}\u0026#39;\u0026#34;) print(f\u0026#34;Tokens: {tokens}\u0026#34;) print(f\u0026#34;Response tone: {analyze_tone(response)}\u0026#34;) print(\u0026#34;-\u0026#34; * 50) # Results show: # \u0026#34;Hello\u0026#34; → Professional response # \u0026#34; Hello\u0026#34; → Confused/broken response # \u0026#34;HELLO\u0026#34; → Urgent/aggressive response # \u0026#34;hello\u0026#34; → Casual response # The killer: Leading spaces often tokenize as separate tokens # This activates \u0026#34;continuation\u0026#34; behavior instead of \u0026#34;start\u0026#34; behavior The Capitalization Personality Disorder # Same word, different personalities: capitalization_effects = { \u0026#34;help\u0026#34;: { \u0026#34;tokens\u0026#34;: [\u0026#34;help\u0026#34;], \u0026#34;context\u0026#34;: \u0026#34;Casual conversations, forum posts\u0026#34;, \u0026#34;personality\u0026#34;: \u0026#34;Friendly, informal assistant\u0026#34; }, \u0026#34;Help\u0026#34;: { \u0026#34;tokens\u0026#34;: [\u0026#34;Help\u0026#34;], \u0026#34;context\u0026#34;: \u0026#34;Documentation headers, menu items\u0026#34;, \u0026#34;personality\u0026#34;: \u0026#34;Professional, structured responses\u0026#34; }, \u0026#34;HELP\u0026#34;: { \u0026#34;tokens\u0026#34;: [\u0026#34;HE\u0026#34;, \u0026#34;LP\u0026#34;] or [\u0026#34;HELP\u0026#34;], \u0026#34;context\u0026#34;: \u0026#34;Error messages, emergency docs, angry users\u0026#34;, \u0026#34;personality\u0026#34;: \u0026#34;Urgent, technical, sometimes panicked\u0026#34; }, \u0026#34;HeLp\u0026#34;: { \u0026#34;tokens\u0026#34;: [\u0026#34;He\u0026#34;, \u0026#34;Lp\u0026#34;] or [\u0026#34;H\u0026#34;, \u0026#34;e\u0026#34;, \u0026#34;L\u0026#34;, \u0026#34;p\u0026#34;], \u0026#34;context\u0026#34;: \u0026#34;Sarcastic posts, mocking text, spam\u0026#34;, \u0026#34;personality\u0026#34;: \u0026#34;Confused, potentially sarcastic\u0026#34; } } # Try these prompts: # \u0026#34;help me write code\u0026#34; → Casual, friendly explanation # \u0026#34;Help me write code\u0026#34; → Formal, structured tutorial # \u0026#34;HELP ME WRITE CODE\u0026#34; → Urgent debugging assistance # \u0026#34;hElP mE wRiTe CoDe\u0026#34; → Model has a stroke The Punctuation Personality Shift def punctuation_changes_everything(tokenizer): \u0026#34;\u0026#34;\u0026#34;How punctuation triggers different training contexts\u0026#34;\u0026#34;\u0026#34; prompts = { \u0026#34;Write a story\u0026#34;: \u0026#34;Creative writing forums\u0026#34;, \u0026#34;Write a story.\u0026#34;: \u0026#34;Formal instructions\u0026#34;, \u0026#34;Write a story!\u0026#34;: \u0026#34;Enthusiastic teacher\u0026#34;, \u0026#34;Write a story?\u0026#34;: \u0026#34;Uncertain/checking understanding\u0026#34;, \u0026#34;Write a story...\u0026#34;: \u0026#34;Trailing thought/mystery\u0026#34;, \u0026#34;Write a story;\u0026#34;: \u0026#34;Academic/technical writing\u0026#34;, \u0026#34;write a story\u0026#34;: \u0026#34;Casual texting\u0026#34;, \u0026#34;WRITE A STORY\u0026#34;: \u0026#34;Demanding/urgent\u0026#34;, \u0026#34;Write. A. Story.\u0026#34;: \u0026#34;Emphatic/aggressive\u0026#34;, } for prompt, context in prompts.items(): tokens = tokenizer.encode(prompt) print(f\u0026#34;\u0026#39;{prompt}\u0026#39;\u0026#34;) print(f\u0026#34; Tokens: {[tokenizer.decode([t]) for t in tokens]}\u0026#34;) print(f\u0026#34; Triggers: {context} context\u0026#34;) print(f\u0026#34; Response style: {get_expected_style(context)}\u0026#34;) # The model isn\u0026#39;t \u0026#34;understanding\u0026#34; your tone # It\u0026#39;s pattern-matching to training data with similar tokens The Whitespace Conspiracy # Different whitespace = different model: whitespace_tests = { \u0026#34;Hello world\u0026#34;: [\u0026#34;Hello\u0026#34;, \u0026#34; world\u0026#34;], # Normal \u0026#34;Hello world\u0026#34;: [\u0026#34;Hello\u0026#34;, \u0026#34; \u0026#34;, \u0026#34;world\u0026#34;], # Double space \u0026#34;Hello\\tworld\u0026#34;: [\u0026#34;Hello\u0026#34;, \u0026#34;\\t\u0026#34;, \u0026#34;world\u0026#34;], # Tab \u0026#34;Hello\\nworld\u0026#34;: [\u0026#34;Hello\u0026#34;, \u0026#34;\\n\u0026#34;, \u0026#34;world\u0026#34;], # Newline \u0026#34;Hello\\r\\nworld\u0026#34;: [\u0026#34;Hello\u0026#34;, \u0026#34;\\r\\n\u0026#34;, \u0026#34;world\u0026#34;], # Windows newline \u0026#34;Hello　world\u0026#34;: [\u0026#34;Hello\u0026#34;, \u0026#34;　\u0026#34;, \u0026#34;world\u0026#34;], # Full-width space \u0026#34; Hello world\u0026#34;: [\u0026#34; \u0026#34;, \u0026#34;Hello\u0026#34;, \u0026#34; world\u0026#34;], # Leading space \u0026#34;Hello world \u0026#34;: [\u0026#34;Hello\u0026#34;, \u0026#34; world\u0026#34;, \u0026#34; \u0026#34;], # Trailing space } # Each triggers different behavior: # - Double spaces: Often seen in OCR errors → less coherent # - Tabs: Code context → technical responses # - Newlines: New paragraph → topic shift # - Leading spaces: Continuation → assumes prior context # - Full-width spaces: Asian language context → different style # This is why copy-pasting from different sources breaks prompts! The Emoji Tokenization Chaos def emoji_personality_injection(): \u0026#34;\u0026#34;\u0026#34;How emojis completely change model behavior\u0026#34;\u0026#34;\u0026#34; tests = [ \u0026#34;Help me 😊\u0026#34;, # Triggers social media training \u0026#34;Help me 🙏\u0026#34;, # Triggers pleading/religious context \u0026#34;Help me 💀\u0026#34;, # Triggers Gen Z slang context \u0026#34;Help me 🚀\u0026#34;, # Triggers startup/crypto context \u0026#34;Help me ❤️\u0026#34;, # Triggers romantic/emotional context \u0026#34;Help me 📊\u0026#34;, # Triggers business/analytical context ] for prompt in tests: tokens = tokenizer.encode(prompt) # Most emojis are 2-4 tokens each # But they trigger COMPLETELY different training contexts print(f\u0026#34;{prompt} → {len(tokens)} tokens\u0026#34;) print(f\u0026#34; Triggers: {detect_context(prompt)}\u0026#34;) # \u0026#34;Help me 🚀\u0026#34; gets you startup buzzwords # \u0026#34;Help me 📊\u0026#34; gets you corporate speak # \u0026#34;Help me 💀\u0026#34; gets you \u0026#34;no cap fr fr\u0026#34; responses # Emojis aren\u0026#39;t just decoration, they\u0026#39;re context switches The Case Sensitivity Disaster in Code # Why models mess up code casing: code_casing_chaos = { \u0026#34;getString\u0026#34;: [\u0026#34;get\u0026#34;, \u0026#34;String\u0026#34;], # camelCase \u0026#34;GetString\u0026#34;: [\u0026#34;Get\u0026#34;, \u0026#34;String\u0026#34;], # PascalCase \u0026#34;get_string\u0026#34;: [\u0026#34;get\u0026#34;, \u0026#34;_\u0026#34;, \u0026#34;string\u0026#34;], # snake_case \u0026#34;GET_STRING\u0026#34;: [\u0026#34;GET\u0026#34;, \u0026#34;_\u0026#34;, \u0026#34;STRING\u0026#34;], # SCREAMING_SNAKE \u0026#34;getstring\u0026#34;: [\u0026#34;get\u0026#34;, \u0026#34;string\u0026#34;], # lowercase \u0026#34;GETSTRING\u0026#34;: [\u0026#34;GET\u0026#34;, \u0026#34;STRING\u0026#34;], # UPPERCASE } # Each triggers different programming contexts: # - camelCase → JavaScript/Java # - PascalCase → C#/.NET # - snake_case → Python/Ruby # - SCREAMING_SNAKE → Constants/C macros # Ask for \u0026#34;a getstring function\u0026#34;: response_styles = { \u0026#34;getString\u0026#34;: \u0026#34;function getString() { return this.value; }\u0026#34;, \u0026#34;GetString\u0026#34;: \u0026#34;public string GetString() { return Value; }\u0026#34;, \u0026#34;get_string\u0026#34;: \u0026#34;def get_string(self): return self.value\u0026#34;, \u0026#34;GET_STRING\u0026#34;: \u0026#34;#define GET_STRING(x) ((x)-\u0026gt;string_value)\u0026#34;, } # The model gives you different languages based on CASING ALONE The Silent Token Boundaries def invisible_token_boundaries(): \u0026#34;\u0026#34;\u0026#34;Token boundaries you can\u0026#39;t see but models can\u0026#34;\u0026#34;\u0026#34; # These look identical to humans: lookalikes = [ (\u0026#34;naive\u0026#34;, \u0026#34;naïve\u0026#34;), # Different tokens! (\u0026#34;cafe\u0026#34;, \u0026#34;café\u0026#34;), # Different tokens! (\u0026#34;resume\u0026#34;, \u0026#34;résumé\u0026#34;), # Different tokens! (\u0026#34;uber\u0026#34;, \u0026#34;über\u0026#34;), # Different tokens! (\u0026#34;Hello\u0026#34;, \u0026#34;Ηello\u0026#34;), # H vs Greek Eta (\u0026#34;test\u0026#34;, \u0026#34;tеst\u0026#34;), # e vs Cyrillic е ] for normal, special in lookalikes: tokens_normal = tokenizer.encode(normal) tokens_special = tokenizer.encode(special) print(f\u0026#34;\u0026#39;{normal}\u0026#39; → {tokens_normal}\u0026#34;) print(f\u0026#34;\u0026#39;{special}\u0026#39; → {tokens_special}\u0026#34;) if tokens_normal != tokens_special: print(\u0026#34; ⚠️ DIFFERENT TOKENS - DIFFERENT BEHAVIOR!\u0026#34;) # \u0026#34;resume\u0026#34; gives you job hunting advice # \u0026#34;résumé\u0026#34; gives you document formatting tips # They look the same but trigger different contexts! The URL/Email Tokenization Personality # How formatting triggers different modes: format_triggers = { \u0026#34;example.com\u0026#34;: \u0026#34;Casual mention\u0026#34;, \u0026#34;https://example.com\u0026#34;: \u0026#34;Technical documentation\u0026#34;, \u0026#34;HTTPS://EXAMPLE.COM\u0026#34;: \u0026#34;Security warning context\u0026#34;, \u0026#34;user@example.com\u0026#34;: \u0026#34;Email/professional context\u0026#34;, \u0026#34;USER@EXAMPLE.COM\u0026#34;: \u0026#34;System/error message context\u0026#34;, \u0026#34;@user\u0026#34;: \u0026#34;Social media context\u0026#34;, \u0026#34;#topic\u0026#34;: \u0026#34;Hashtag/trending context\u0026#34;, \u0026#34;$VARIABLE\u0026#34;: \u0026#34;Environment variable/coding\u0026#34;, \u0026#34;%VALUE%\u0026#34;: \u0026#34;Windows batch script context\u0026#34;, } # Each format activates different training data: # \u0026#34;@user\u0026#34; → Twitter personality # \u0026#34;user@\u0026#34; → Email formality # \u0026#34;$USER\u0026#34; → Unix documentation # \u0026#34;%USER%\u0026#34; → Windows documentation # Your prompt format literally changes which \u0026#34;personality\u0026#34; responds The Production Nightmare Stories def real_production_failures(): \u0026#34;\u0026#34;\u0026#34;Actual failures caused by spacing/casing\u0026#34;\u0026#34;\u0026#34; failures = { \u0026#34;Customer support bot\u0026#34;: { \u0026#34;issue\u0026#34;: \u0026#34;Users typing \u0026#39; help\u0026#39; with leading space\u0026#34;, \u0026#34;result\u0026#34;: \u0026#34;Bot responded with code instead of support\u0026#34;, \u0026#34;cause\u0026#34;: \u0026#34;Leading space triggered code completion context\u0026#34;, \u0026#34;fix\u0026#34;: \u0026#34;Strip all leading/trailing whitespace\u0026#34; }, \u0026#34;Code generator\u0026#34;: { \u0026#34;issue\u0026#34;: \u0026#34;Mixed case in function names\u0026#34;, \u0026#34;result\u0026#34;: \u0026#34;Generated different programming languages\u0026#34;, \u0026#34;cause\u0026#34;: \u0026#34;camelCase vs snake_case tokenization\u0026#34;, \u0026#34;fix\u0026#34;: \u0026#34;Normalize casing before generation\u0026#34; }, \u0026#34;Translation service\u0026#34;: { \u0026#34;issue\u0026#34;: \u0026#34;ALL CAPS input\u0026#34;, \u0026#34;result\u0026#34;: \u0026#34;Aggressive/rude translations\u0026#34;, \u0026#34;cause\u0026#34;: \u0026#34;CAPS associated with angry training data\u0026#34;, \u0026#34;fix\u0026#34;: \u0026#34;Lowercase normalization with case restoration\u0026#34; }, \u0026#34;Medical assistant\u0026#34;: { \u0026#34;issue\u0026#34;: \u0026#34;Double spaces in symptoms\u0026#34;, \u0026#34;result\u0026#34;: \u0026#34;Triggered academic paper context, not medical advice\u0026#34;, \u0026#34;cause\u0026#34;: \u0026#34;Double spaces common in LaTeX/papers\u0026#34;, \u0026#34;fix\u0026#34;: \u0026#34;Normalize all whitespace\u0026#34; } } return \u0026#34;Every spacing/casing choice is a context switch\u0026#34; The Fix: Prompt Normalization Pipeline class PromptNormalizer: \u0026#34;\u0026#34;\u0026#34;Save your model from personality disorders\u0026#34;\u0026#34;\u0026#34; def normalize(self, text): \u0026#34;\u0026#34;\u0026#34;Consistent tokenization = consistent behavior\u0026#34;\u0026#34;\u0026#34; # 1. Strip dangerous whitespace text = text.strip() # 2. Normalize internal whitespace text = \u0026#39; \u0026#39;.join(text.split()) # 3. Fix casing strategically if text.isupper() and len(text) \u0026gt; 10: # Long CAPS text → normalize text = text.capitalize() # 4. Remove invisible characters text = \u0026#39;\u0026#39;.join(c for c in text if c.isprintable() or c == \u0026#39;\\n\u0026#39;) # 5. Normalize quotes and apostrophes replacements = { \u0026#39;\u0026#34;\u0026#39;: \u0026#39;\u0026#34;\u0026#39;, \u0026#39;\u0026#34;\u0026#39;: \u0026#39;\u0026#34;\u0026#39;, # Smart quotes \u0026#39;\u0026#39;\u0026#39;: \u0026#34;\u0026#39;\u0026#34;, \u0026#39;\u0026#39;\u0026#39;: \u0026#34;\u0026#39;\u0026#34;, # Smart apostrophes \u0026#39;　\u0026#39;: \u0026#39; \u0026#39;, # Full-width space } for old, new in replacements.items(): text = text.replace(old, new) return text def warn_about_triggers(self, text): \u0026#34;\u0026#34;\u0026#34;Detect personality triggers\u0026#34;\u0026#34;\u0026#34; warnings = [] if text.startswith(\u0026#39; \u0026#39;): warnings.append(\u0026#34;Leading space: May trigger continuation behavior\u0026#34;) if text.isupper(): warnings.append(\u0026#34;ALL CAPS: May trigger aggressive/urgent responses\u0026#34;) if \u0026#39; \u0026#39; in text: warnings.append(\u0026#34;Double spaces: May trigger academic/formal context\u0026#34;) if any(ord(c) \u0026gt; 127 for c in text): warnings.append(\u0026#34;Special characters: May trigger unexpected contexts\u0026#34;) return warnings The Psychological Truth: It\u0026rsquo;s Token Sequences, Not Tokenization Models don\u0026rsquo;t have personalities, they have learned associations with different token sequences. The tokenizer creates the sequences, but the transformer learned the behaviors:\nThe Two-Step Disaster:\nTokenizer: Splits \u0026ldquo;HELP\u0026rdquo; into [\u0026ldquo;HE\u0026rdquo;, \u0026ldquo;LP\u0026rdquo;] Transformer: Learned [\u0026ldquo;HE\u0026rdquo;, \u0026ldquo;LP\u0026rdquo;] appears in panic contexts Why this matters:\nThe tokenizer is \u0026ldquo;dumb\u0026rdquo; - it just splits by frequency The transformer is \u0026ldquo;smart\u0026rdquo; - it learned patterns But bad tokenization creates bad patterns to learn! If \u0026ldquo;Hello\u0026rdquo; is one token but \u0026quot; Hello\u0026quot; is two tokens, the transformer learns completely different contexts for each. It\u0026rsquo;s not the tokenizer\u0026rsquo;s \u0026ldquo;fault\u0026rdquo; directly, but tokenization determines what patterns the transformer CAN learn.\nThe inseparable relationship:\nTokenization defines the vocabulary Transformer learns relationships between vocabulary items Bad tokenization = impossible for transformer to learn good patterns You can\u0026rsquo;t fix one without the other 💡 The Real Insight: Tokenization doesn\u0026rsquo;t cause behavior directly, but it determines which token sequences exist for the transformer to learn patterns from. A leading space creating a different token sequence means the transformer learned different associations. The tokenizer creates the map, the transformer learns to navigate it.\nTakeaway: Your model\u0026rsquo;s personality isn\u0026rsquo;t in its weights, it\u0026rsquo;s in your whitespace. A leading space, wrong capitalization, or stray emoji can switch your helpful assistant into a different character entirely. This isn\u0026rsquo;t intelligence; it\u0026rsquo;s pattern matching gone wrong. Control your tokens, control your model\u0026rsquo;s personality.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/tokenization_personality_trigger/","summary":"\u003ch1 id=\"how-spacing-and-capitalization-randomly-change-your-models-entire-personality\"\u003eHow Spacing and Capitalization Randomly Change Your Model\u0026rsquo;s Entire Personality\u003c/h1\u003e\n\u003cp\u003eAdd a space before your prompt and watch GPT become 30% dumber. Write in ALL CAPS and suddenly it\u0026rsquo;s aggressive. Use \u0026ldquo;pls\u0026rdquo; instead of \u0026ldquo;please\u0026rdquo; and it becomes casual. This isn\u0026rsquo;t personality, it\u0026rsquo;s tokenization chaos triggering different training data pockets.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: \u0026quot; Hello\u0026quot; and \u0026ldquo;Hello\u0026rdquo; activate completely different neural pathways. \u0026ldquo;HELP\u0026rdquo; vs \u0026ldquo;help\u0026rdquo; vs \u0026ldquo;Help\u0026rdquo; pulls from different training contexts (emergency manuals vs casual chat vs formal documents). Your model doesn\u0026rsquo;t have moods, it has tokenization triggered personality disorders.\u003c/p\u003e","title":"How Spacing and Capitalization Randomly Change Your Model's Entire Personality"},{"content":"How Tokenization Murders Your Model\u0026rsquo;s Ability to Do Basic Math GPT-4o can write Shakespeare but struggles with 4-digit multiplication. It\u0026rsquo;s not stupid, it literally can\u0026rsquo;t see numbers the way you do. \u0026ldquo;12345\u0026rdquo; might be [\u0026ldquo;123\u0026rdquo;, \u0026ldquo;45\u0026rdquo;] while \u0026ldquo;12346\u0026rdquo; is [\u0026ldquo;1\u0026rdquo;, \u0026ldquo;2346\u0026rdquo;]. Try doing math when numbers randomly shatter into chunks.\nTL;DR: Tokenizers split numbers inconsistently, making arithmetic nearly impossible. \u0026ldquo;9.11\u0026rdquo; \u0026gt; \u0026ldquo;9.9\u0026rdquo; according to many models because \u0026ldquo;.11\u0026rdquo; and \u0026ldquo;.9\u0026rdquo; are different tokens. Your calculator app works. Your $100B language model doesn\u0026rsquo;t. This is why.\nThe Crime Scene: Test This Right Now # The murder weapon: inconsistent number tokenization def number_tokenization_horror(tokenizer): \u0026#34;\u0026#34;\u0026#34;Watch tokenization destroy math ability\u0026#34;\u0026#34;\u0026#34; numbers = [ \u0026#34;1\u0026#34;, \u0026#34;12\u0026#34;, \u0026#34;123\u0026#34;, \u0026#34;1234\u0026#34;, \u0026#34;12345\u0026#34;, \u0026#34;123456\u0026#34;, \u0026#34;42\u0026#34;, \u0026#34;420\u0026#34;, \u0026#34;4200\u0026#34;, \u0026#34;42000\u0026#34;, \u0026#34;9.9\u0026#34;, \u0026#34;9.11\u0026#34;, \u0026#34;9.111\u0026#34;, \u0026#34;2023\u0026#34;, \u0026#34;2024\u0026#34;, \u0026#34;2025\u0026#34;, \u0026#34;1000000\u0026#34;, \u0026#34;1,000,000\u0026#34;, \u0026#34;1e6\u0026#34; ] print(\u0026#34;Number → Tokens (How the model \u0026#39;sees\u0026#39; it)\u0026#34;) print(\u0026#34;-\u0026#34; * 50) for num in numbers: tokens = tokenizer.encode(num) decoded = [tokenizer.decode([t]) for t in tokens] # The horror reveal if len(decoded) \u0026gt; 1: print(f\u0026#34;{num:10} → {decoded} 🔪 (MURDERED)\u0026#34;) else: print(f\u0026#34;{num:10} → {decoded}\u0026#34;) return \u0026#34;Your model can\u0026#39;t do math because it can\u0026#39;t even see numbers\u0026#34; # Run this and watch the chaos The \u0026ldquo;9.11 \u0026gt; 9.9\u0026rdquo; Disaster (Yes, Really) # This actually happens in production: comparisons = [ (\u0026#34;9.9\u0026#34;, \u0026#34;9.11\u0026#34;), # 9.11 is SMALLER but models think it\u0026#39;s bigger (\u0026#34;2.8\u0026#34;, \u0026#34;2.80\u0026#34;), # Same number, different tokens (\u0026#34;1000\u0026#34;, \u0026#34;1,000\u0026#34;), # Same number, different tokens (\u0026#34;3.14\u0026#34;, \u0026#34;3.141\u0026#34;), # π gets progressively worse ] for a, b in comparisons: tokens_a = tokenizer.encode(a) tokens_b = tokenizer.encode(b) print(f\u0026#34;{a} → {tokens_a}\u0026#34;) print(f\u0026#34;{b} → {tokens_b}\u0026#34;) # Models compare TOKEN VALUES, not numerical values # \u0026#34;11\u0026#34; \u0026gt; \u0026#34;9\u0026#34; as a token, so 9.11 \u0026gt; 9.9 # This is why your chatbot says 9.11 is bigger than 9.9 # THE KILLER: Version numbers # \u0026#34;Python 3.9\u0026#34; vs \u0026#34;Python 3.11\u0026#34; # Models think 3.11 \u0026lt; 3.9 because \u0026#34;.11\u0026#34; \u0026lt; \u0026#34;.9\u0026#34; as text Why Your Model Can\u0026rsquo;t Count # The counting disaster: def why_counting_fails(tokenizer): \u0026#34;\u0026#34;\u0026#34;Models can\u0026#39;t count because they can\u0026#39;t see sequences\u0026#34;\u0026#34;\u0026#34; # Try to count from 1 to 20 sequence = \u0026#34;1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20\u0026#34; tokens = tokenizer.encode(sequence) decoded = [tokenizer.decode([t]) for t in tokens] print(\u0026#34;You see: 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20\u0026#34;) print(f\u0026#34;Model sees: {decoded}\u0026#34;) # Reality: [\u0026#39;1\u0026#39;, \u0026#39; 2\u0026#39;, \u0026#39; 3\u0026#39;, ..., \u0026#39; 10\u0026#39;, \u0026#39; 11\u0026#39;, \u0026#39; 12\u0026#39;, \u0026#39; 13\u0026#39;, \u0026#39; 14\u0026#39;, \u0026#39; 15\u0026#39;, \u0026#39; 16\u0026#39;, \u0026#39; 17\u0026#39;, \u0026#39; 18\u0026#39;, \u0026#39; 19\u0026#39;, \u0026#39; 20\u0026#39;] # Some are single tokens, some are split # Patterns are DESTROYED # Now try counting by 10s: by_tens = \u0026#34;10 20 30 40 50 60 70 80 90 100\u0026#34; tokens = tokenizer.encode(by_tens) decoded = [tokenizer.decode([t]) for t in tokens] print(f\u0026#34;Counting by 10s: {decoded}\u0026#34;) # \u0026#34;10\u0026#34; might be 1 token, \u0026#34;20\u0026#34; might be 1 token, but \u0026#34;30\u0026#34; might be [\u0026#34;3\u0026#34;, \u0026#34;0\u0026#34;] # No wonder it can\u0026#39;t learn the pattern! # This is why models fail at: # - \u0026#34;Continue the sequence: 2, 4, 6, 8, ...\u0026#34; # - \u0026#34;What comes after 99?\u0026#34; # - \u0026#34;Count backwards from 10\u0026#34; The Arithmetic Apocalypse # Why GPT can\u0026#39;t do basic math: def arithmetic_tokenization_study(tokenizer): \u0026#34;\u0026#34;\u0026#34;See why 2+2 works but 1234+5678 doesn\u0026#39;t\u0026#34;\u0026#34;\u0026#34; problems = [ \u0026#34;2+2\u0026#34;, # Perfect: [\u0026#34;2\u0026#34;, \u0026#34;+\u0026#34;, \u0026#34;2\u0026#34;] \u0026#34;10+10\u0026#34;, # Still OK: [\u0026#34;10\u0026#34;, \u0026#34;+\u0026#34;, \u0026#34;10\u0026#34;] \u0026#34;123+456\u0026#34;, # Getting bad: [\u0026#34;123\u0026#34;, \u0026#34;+\u0026#34;, \u0026#34;456\u0026#34;] or [\u0026#34;12\u0026#34;, \u0026#34;3\u0026#34;, \u0026#34;+\u0026#34;, \u0026#34;45\u0026#34;, \u0026#34;6\u0026#34;] \u0026#34;1234+5678\u0026#34;, # Disaster: [\u0026#34;123\u0026#34;, \u0026#34;4\u0026#34;, \u0026#34;+\u0026#34;, \u0026#34;567\u0026#34;, \u0026#34;8\u0026#34;] \u0026#34;12345+67890\u0026#34;, # Apocalypse: [\u0026#34;1\u0026#34;, \u0026#34;234\u0026#34;, \u0026#34;5\u0026#34;, \u0026#34;+\u0026#34;, \u0026#34;678\u0026#34;, \u0026#34;90\u0026#34;] ] for problem in problems: tokens = tokenizer.encode(problem) decoded = [tokenizer.decode([t]) for t in tokens] # Calculate token chaos score expected_tokens = 3 # number, operator, number chaos_score = len(tokens) - expected_tokens if chaos_score == 0: status = \u0026#34;✅ Can solve\u0026#34; elif chaos_score \u0026lt;= 2: status = \u0026#34;⚠️ Might solve\u0026#34; else: status = \u0026#34;💀 Will fail\u0026#34; print(f\u0026#34;{problem:15} → {decoded:30} {status}\u0026#34;) return \u0026#34;This is why calculators still exist\u0026#34; # The bitter truth: # o can write poetry about quantum physics # But fails at 4-digit multiplication # Because it literally cannot see \u0026#34;1234\u0026#34; as 1234 The Decimal Disaster # Decimals are even worse: decimal_nightmares = { \u0026#34;0.1\u0026#34;: [\u0026#34;0\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;1\u0026#34;], # 3 tokens for a simple decimal \u0026#34;0.01\u0026#34;: [\u0026#34;0\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;01\u0026#34;], # Inconsistent! \u0026#34;0.001\u0026#34;: [\u0026#34;0\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;001\u0026#34;], # Getting worse \u0026#34;3.14159\u0026#34;: [\u0026#34;3\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;14\u0026#34;, \u0026#34;159\u0026#34;], # π is shattered \u0026#34;2.718\u0026#34;: [\u0026#34;2\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;7\u0026#34;, \u0026#34;18\u0026#34;], # e is broken \u0026#34;$19.99\u0026#34;: [\u0026#34;$\u0026#34;, \u0026#34;19\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;99\u0026#34;], # Prices are chaos \u0026#34;0.999...\u0026#34;: [\u0026#34;0\u0026#34;, \u0026#34;.\u0026#34;, \u0026#34;999\u0026#34;, \u0026#34;...\u0026#34;], # Math notation destroyed } # This is why: # - Models think 0.9 \u0026gt; 0.11 (string comparison) # - Can\u0026#39;t properly handle financial calculations # - Fail at scientific notation # - Think $19.99 and $20.00 are fundamentally different concepts The Date/Time Tokenization Massacre def datetime_tokenization_chaos(tokenizer): \u0026#34;\u0026#34;\u0026#34;Why models are terrible with dates and times\u0026#34;\u0026#34;\u0026#34; dates = [ \u0026#34;2023\u0026#34;, # Likely 1 token (common year) \u0026#34;2024\u0026#34;, # Might be 2 tokens \u0026#34;2025\u0026#34;, # Probably 2 tokens \u0026#34;1999\u0026#34;, # 1-2 tokens (Y2K made it common) \u0026#34;2000\u0026#34;, # 1 token (millennium) \u0026#34;1823\u0026#34;, # 2-3 tokens (random year) \u0026#34;2024-01-01\u0026#34;, # 5-7 tokens \u0026#34;12/25/2023\u0026#34;, # 6-8 tokens \u0026#34;3:14 PM\u0026#34;, # 4-5 tokens \u0026#34;15:30:45\u0026#34;, # 5-7 tokens ] for date in dates: tokens = tokenizer.encode(date) if len(tokens) == 1: print(f\u0026#34;{date} → MEMORIZED (seen thousands of times)\u0026#34;) else: decoded = [tokenizer.decode([t]) for t in tokens] print(f\u0026#34;{date} → {decoded} (fragmented perception)\u0026#34;) # This explains why models: # - Can\u0026#39;t calculate date differences # - Fail at \u0026#34;what day is 30 days from today?\u0026#34; # - Think 12/25 comes before 12/3 (string order) # - Can\u0026#39;t handle timezone conversions The Phone Number Privacy Leak # Some phone numbers are single tokens (!!!) def phone_number_investigation(tokenizer): \u0026#34;\u0026#34;\u0026#34;Some numbers are suspiciously well-tokenized\u0026#34;\u0026#34;\u0026#34; numbers = [ \u0026#34;911\u0026#34;, # Emergency (1 token) \u0026#34;1-800-273-8255\u0026#34;, # Suicide hotline (might be few tokens) \u0026#34;867-5309\u0026#34;, # Jenny\u0026#39;s number (cultural reference) \u0026#34;(555) 555-5555\u0026#34;, # Movie/TV placeholder \u0026#34;+1234567890\u0026#34;, # Random number ] for number in numbers: tokens = len(tokenizer.encode(number)) if tokens \u0026lt;= 3: print(f\u0026#34;🚨 {number} is {tokens} tokens - MEMORIZED IN TRAINING\u0026#34;) else: print(f\u0026#34;{number} is {tokens} tokens\u0026#34;) # If a phone number is \u0026lt;5 tokens, it appeared in training data # This is a privacy nightmare The Solution No One Wants to Hear class MathTokenizationWorkaround: \u0026#34;\u0026#34;\u0026#34;How to make models not suck at math\u0026#34;\u0026#34;\u0026#34; def fix_arithmetic(self, expression): \u0026#34;\u0026#34;\u0026#34;Pre-tokenize numbers properly\u0026#34;\u0026#34;\u0026#34; # Step 1: Space out everything # \u0026#34;1234+5678\u0026#34; → \u0026#34;1 2 3 4 + 5 6 7 8\u0026#34; spaced = \u0026#39; \u0026#39;.join(expression) # Step 2: Use chain-of-thought prompt = f\u0026#34;\u0026#34;\u0026#34; Solve step by step: {expression} First, identify the numbers: - First number: {expression.split(\u0026#39;+\u0026#39;)[0]} - Second number: {expression.split(\u0026#39;+\u0026#39;)[1]} Now add digit by digit... \u0026#34;\u0026#34;\u0026#34; # Step 3: Or just give up and use a calculator import re if re.match(r\u0026#39;^[\\d\\+\\-\\*/\\.\\s]+$\u0026#39;, expression): result = eval(expression) # Don\u0026#39;t do this in production! return f\u0026#34;The answer is {result}\u0026#34; return \u0026#34;This is why we still need calculators\u0026#34; def fix_comparison(self, num1, num2): \u0026#34;\u0026#34;\u0026#34;Fix number comparison\u0026#34;\u0026#34;\u0026#34; # Convert to actual numbers first prompt = f\u0026#34;\u0026#34;\u0026#34; Compare these as decimal numbers: A = {num1} (decimal value: {float(num1)}) B = {num2} (decimal value: {float(num2)}) Therefore {float(num1)} {\u0026#39;\u0026gt;\u0026#39; if float(num1) \u0026gt; float(num2) else \u0026#39;\u0026lt;\u0026#39;} {float(num2)} \u0026#34;\u0026#34;\u0026#34; return prompt # The harsh reality: # We\u0026#39;re using $100B language models # But need to PRE-CALCULATE math for them # Because they can\u0026#39;t see numbers properly The Benchmarks That Lie # Why math benchmarks are misleading: def benchmark_tokenization_bias(): \u0026#34;\u0026#34;\u0026#34;GSM8K and other benchmarks use \u0026#39;nice\u0026#39; numbers\u0026#34;\u0026#34;\u0026#34; # Benchmark problems use: nice_numbers = [\u0026#34;2\u0026#34;, \u0026#34;5\u0026#34;, \u0026#34;10\u0026#34;, \u0026#34;100\u0026#34;, \u0026#34;1000\u0026#34;] # All single tokens! # Real-world uses: real_numbers = [\u0026#34;1847\u0026#34;, \u0026#34;3.14159\u0026#34;, \u0026#34;$24.99\u0026#34;, \u0026#34;2024-01-15\u0026#34;] # All fragmented! print(\u0026#34;Benchmark numbers (what models are tested on):\u0026#34;) for n in nice_numbers: print(f\u0026#34; {n} → {len(tokenizer.encode(n))} token(s)\u0026#34;) print(\u0026#34;\\nReal-world numbers (what you actually need):\u0026#34;) for n in real_numbers: print(f\u0026#34; {n} → {len(tokenizer.encode(n))} token(s)\u0026#34;) return \u0026#34;Models ace benchmarks, fail at your invoice calculations\u0026#34; The Uncomfortable Truth About Quantitative AI The paradox: We\u0026rsquo;re using language models for quantitative analysis, but they literally cannot perceive quantities consistently.\nWhat this means:\nFinancial models that can\u0026rsquo;t compare prices Scientific models that fail at measurements Data analysis tools that can\u0026rsquo;t count Coding assistants that mess up array indices The industry\u0026rsquo;s dirty secret: Everyone knows this, but we pretend it\u0026rsquo;s fine because:\nThe models are \u0026ldquo;good enough\u0026rdquo; for text We can work around it with prompting Fixing it would require rebuilding everything 💡 Immediate Action: Never trust a language model with math. Always verify numerical outputs. If you\u0026rsquo;re building a financial or scientific application, pre-process all numbers into consistent tokens or use specialized numerical encodings.\nTakeaway: Your $100B language model can\u0026rsquo;t do 4th grade math because tokenization shatters numbers into random chunks. It\u0026rsquo;s not learning arithmetic, it\u0026rsquo;s pattern matching fragments. This is why \u0026ldquo;9.11 \u0026gt; 9.9\u0026rdquo; and why GPT-4o needs a calculator plugin. Until we fix tokenization, language models will remain quantitatively illiterate.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/tokenization_math/","summary":"\u003ch1 id=\"how-tokenization-murders-your-models-ability-to-do-basic-math\"\u003eHow Tokenization Murders Your Model\u0026rsquo;s Ability to Do Basic Math\u003c/h1\u003e\n\u003cp\u003eGPT-4o can write Shakespeare but struggles with 4-digit multiplication. It\u0026rsquo;s not stupid, it literally can\u0026rsquo;t see numbers the way you do. \u0026ldquo;12345\u0026rdquo; might be [\u0026ldquo;123\u0026rdquo;, \u0026ldquo;45\u0026rdquo;] while \u0026ldquo;12346\u0026rdquo; is [\u0026ldquo;1\u0026rdquo;, \u0026ldquo;2346\u0026rdquo;]. Try doing math when numbers randomly shatter into chunks.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: Tokenizers split numbers inconsistently, making arithmetic nearly impossible. \u0026ldquo;9.11\u0026rdquo; \u0026gt; \u0026ldquo;9.9\u0026rdquo; according to many models because \u0026ldquo;.11\u0026rdquo; and \u0026ldquo;.9\u0026rdquo; are different tokens. Your calculator app works. Your $100B language model doesn\u0026rsquo;t. This is why.\u003c/p\u003e","title":"How Tokenization Murders Your Model's Ability to Do Basic Math"},{"content":"Why Your Vector Database Thinks $AAPL Means Polish Batteries Your $50K vector database is returning garbage results because \u0026ldquo;$AAPL\u0026rdquo; tokenizes as [\u0026quot;$\u0026quot;, \u0026ldquo;AA\u0026rdquo;, \u0026ldquo;PL\u0026rdquo;] and now has the embedding of \u0026ldquo;dollar + AA batteries + Poland\u0026rdquo;. Your semantic search for \u0026ldquo;Apple stock\u0026rdquo; returns articles about Polish currency. This isn\u0026rsquo;t a retrieval problem, it\u0026rsquo;s tokenization murdering your embeddings.\nTL;DR: Bad tokenization creates noisy embeddings. \u0026ldquo;COVID-19\u0026rdquo; split into [\u0026ldquo;CO\u0026rdquo;, \u0026ldquo;VID\u0026rdquo;, \u0026ldquo;-\u0026rdquo;, \u0026ldquo;19\u0026rdquo;] has embeddings mixing \u0026ldquo;Colorado\u0026rdquo;, \u0026ldquo;video\u0026rdquo;, \u0026ldquo;negative\u0026rdquo;, and \u0026ldquo;2019\u0026rdquo;. Your RAG pipeline is doomed before it starts. Fix tokenization or waste money on larger models trying to compensate.\nThe Embedding Murder Scene # Watch tokenization destroy semantic similarity: def embedding_quality_test(tokenizer, embedding_model): \u0026#34;\u0026#34;\u0026#34;See how tokenization ruins your embeddings\u0026#34;\u0026#34;\u0026#34; # Same concept, different tokenizations test_pairs = [ (\u0026#34;Apple Inc\u0026#34;, \u0026#34;$AAPL\u0026#34;), # Company and its ticker (\u0026#34;COVID-19\u0026#34;, \u0026#34;coronavirus\u0026#34;), # Same disease (\u0026#34;e-commerce\u0026#34;, \u0026#34;ecommerce\u0026#34;), # Same concept, different style (\u0026#34;Ph.D.\u0026#34;, \u0026#34;PhD\u0026#34;), # Same degree (\u0026#34;U.S.A.\u0026#34;, \u0026#34;USA\u0026#34;), # Same country ] for term1, term2 in test_pairs: # Tokenize tokens1 = tokenizer.encode(term1) tokens2 = tokenizer.encode(term2) # Get embeddings emb1 = embedding_model.embed(term1) emb2 = embedding_model.embed(term2) # Calculate similarity similarity = cosine_similarity(emb1, emb2) print(f\u0026#34;{term1} → {[tokenizer.decode([t]) for t in tokens1]}\u0026#34;) print(f\u0026#34;{term2} → {[tokenizer.decode([t]) for t in tokens2]}\u0026#34;) print(f\u0026#34;Similarity: {similarity:.3f}\u0026#34;) if len(tokens1) \u0026gt; 2 or len(tokens2) \u0026gt; 2: print(\u0026#34;⚠️ FRAGMENTED - Embeddings are NOISE\u0026#34;) print(\u0026#34;-\u0026#34; * 50) # Results: # \u0026#34;$AAPL\u0026#34; as [\u0026#34;$\u0026#34;, \u0026#34;AA\u0026#34;, \u0026#34;PL\u0026#34;] → similarity with \u0026#34;Apple\u0026#34; = 0.15 # \u0026#34;$AAPL\u0026#34; as [\u0026#34;$AAPL\u0026#34;] → similarity with \u0026#34;Apple\u0026#34; = 0.75 # 5X DIFFERENCE just from tokenization! The Semantic Catastrophe # How fragmented tokens create nonsense embeddings: def fragmentation_semantic_disaster(): \u0026#34;\u0026#34;\u0026#34;When tokens split, meaning dies\u0026#34;\u0026#34;\u0026#34; # \u0026#34;$AAPL\u0026#34; tokenized as [\u0026#34;$\u0026#34;, \u0026#34;AA\u0026#34;, \u0026#34;PL\u0026#34;] # The embedding becomes: embedding_components = { \u0026#34;$\u0026#34;: { \u0026#34;learned_from\u0026#34;: [\u0026#34;$100\u0026#34;, \u0026#34;$50\u0026#34;, \u0026#34;money\u0026#34;, \u0026#34;currency\u0026#34;, \u0026#34;price\u0026#34;], \u0026#34;semantic_meaning\u0026#34;: \u0026#34;Money/cost concept\u0026#34; }, \u0026#34;AA\u0026#34;: { \u0026#34;learned_from\u0026#34;: [\u0026#34;AA batteries\u0026#34;, \u0026#34;Alcoholics Anonymous\u0026#34;, \u0026#34;American Airlines\u0026#34;], \u0026#34;semantic_meaning\u0026#34;: \u0026#34;Batteries/support groups/airlines\u0026#34; }, \u0026#34;PL\u0026#34;: { \u0026#34;learned_from\u0026#34;: [\u0026#34;Poland\u0026#34;, \u0026#34;PL/SQL\u0026#34;, \u0026#34;.pl domain\u0026#34;], \u0026#34;semantic_meaning\u0026#34;: \u0026#34;Poland/programming/perl\u0026#34; } } # The final \u0026#34;$AAPL\u0026#34; embedding is: # 33% money + 33% batteries/airlines + 33% Poland # NOTHING about Apple Inc! # Meanwhile \u0026#34;Apple\u0026#34; embedding is: # 100% technology company / fruit # No wonder similarity is garbage! return \u0026#34;Your stock ticker has the semantics of Polish batteries\u0026#34; The Domain-Specific Disaster # Medical/scientific terms getting destroyed: medical_tokenization_disasters = { \u0026#34;acetaminophen\u0026#34;: [\u0026#34;acet\u0026#34;, \u0026#34;amino\u0026#34;, \u0026#34;phen\u0026#34;], # Embedding: vinegar + amino acids + phenol # NOT pain reliever! \u0026#34;COVID-19\u0026#34;: [\u0026#34;CO\u0026#34;, \u0026#34;VID\u0026#34;, \u0026#34;-\u0026#34;, \u0026#34;19\u0026#34;], # Embedding: Colorado + video + negative + year # NOT pandemic disease! \u0026#34;SARS-CoV-2\u0026#34;: [\u0026#34;SAR\u0026#34;, \u0026#34;S\u0026#34;, \u0026#34;-\u0026#34;, \u0026#34;Co\u0026#34;, \u0026#34;V\u0026#34;, \u0026#34;-\u0026#34;, \u0026#34;2\u0026#34;], # Embedding: search and rescue + sulfur + cobalt + volt # Complete nonsense! \u0026#34;methylprednisolone\u0026#34;: [\u0026#34;methyl\u0026#34;, \u0026#34;pred\u0026#34;, \u0026#34;nis\u0026#34;, \u0026#34;ol\u0026#34;, \u0026#34;one\u0026#34;], # Embedding: chemistry + prediction + ? + alcohol + single # Lost all medical meaning! } # Your medical search engine: # Query: \u0026#34;COVID-19 treatment\u0026#34; # Returns: \u0026#34;Colorado video production 2019 tips\u0026#34; # Because the embeddings are BROKEN Why Fragmented Tokens Create Poor Embeddings def why_fragmentation_destroys_meaning(): \u0026#34;\u0026#34;\u0026#34;Even with attention, fragments can\u0026#39;t capture domain meaning\u0026#34;\u0026#34;\u0026#34; # When \u0026#34;$AAPL\u0026#34; becomes [\u0026#34;$\u0026#34;, \u0026#34;AA\u0026#34;, \u0026#34;PL\u0026#34;]: problems = { \u0026#34;Never learned as unit\u0026#34;: \u0026#34;Model never saw \u0026#39;$AAPL\u0026#39; together during training\u0026#34;, \u0026#34;No semantic link\u0026#34;: \u0026#34;Can\u0026#39;t learn these tokens together = Apple stock\u0026#34;, \u0026#34;Attention can\u0026#39;t help\u0026#34;: \u0026#34;Attention helps context but can\u0026#39;t create missing knowledge\u0026#34;, \u0026#34;Search fails\u0026#34;: \u0026#34;Looking for \u0026#39;Apple stock\u0026#39; won\u0026#39;t find \u0026#39;$\u0026#39; + \u0026#39;AA\u0026#39; + \u0026#39;PL\u0026#39;\u0026#34; } # The fundamental problem: # The model learned: # \u0026#34;$\u0026#34; appears with prices, money, costs # \u0026#34;AA\u0026#34; appears with batteries, ratings, airlines # \u0026#34;PL\u0026#34; appears with Poland, programming, domains # # It NEVER learned: # \u0026#34;$AAPL\u0026#34; = Apple Inc\u0026#39;s stock ticker # So when you search for \u0026#34;Apple stock price\u0026#34; # The embedding for \u0026#34;$AAPL\u0026#34; (as fragments) has NO semantic connection return \u0026#34;Transformers are powerful, but they can\u0026#39;t learn what was never there\u0026#34; The RAG Pipeline Destruction class RAGDisasterAnalysis: \u0026#34;\u0026#34;\u0026#34;How bad tokenization destroys your entire RAG system\u0026#34;\u0026#34;\u0026#34; def __init__(self, tokenizer, bad_tokenizer): self.good = tokenizer # Domain-specific self.bad = bad_tokenizer # Generic def analyze_retrieval_quality(self, query, documents): \u0026#34;\u0026#34;\u0026#34;Compare retrieval with good vs bad tokenization\u0026#34;\u0026#34;\u0026#34; # Good tokenizer: \u0026#34;$NVDA\u0026#34; → [\u0026#34;$NVDA\u0026#34;] good_tokens = self.good.encode(query) good_embedding = embed_with_tokenizer(query, self.good) good_results = retrieve_similar(good_embedding, documents) # Bad tokenizer: \u0026#34;$NVDA\u0026#34; → [\u0026#34;$\u0026#34;, \u0026#34;N\u0026#34;, \u0026#34;V\u0026#34;, \u0026#34;DA\u0026#34;] bad_tokens = self.bad.encode(query) bad_embedding = embed_with_tokenizer(query, self.bad) bad_results = retrieve_similar(bad_embedding, documents) print(f\u0026#34;Query: {query}\u0026#34;) print(f\u0026#34;Good tokenization: {good_tokens}\u0026#34;) print(f\u0026#34;Top result: {good_results[0]}\u0026#34;) # \u0026#34;NVIDIA earnings report\u0026#34; print(f\u0026#34;Bad tokenization: {bad_tokens}\u0026#34;) print(f\u0026#34;Top result: {bad_results[0]}\u0026#34;) # \u0026#34;Nevada state budget\u0026#34; # Bad tokenization turns NVIDIA into Nevada! return \u0026#34;Your RAG is only as good as your tokenizer\u0026#34; The Benchmark Fraud def why_benchmarks_hide_this(): \u0026#34;\u0026#34;\u0026#34;MTEB and other benchmarks use common terms that tokenize well\u0026#34;\u0026#34;\u0026#34; benchmark_terms = [ \u0026#34;computer\u0026#34;, # Always 1 token \u0026#34;science\u0026#34;, # Always 1 token \u0026#34;artificial\u0026#34;, # Usually 1 token \u0026#34;intelligence\u0026#34;, # Usually 1 token ] real_world_terms = [ \u0026#34;GPT-4\u0026#34;, # [\u0026#34;GPT\u0026#34;, \u0026#34;-\u0026#34;, \u0026#34;4\u0026#34;] \u0026#34;COVID-19\u0026#34;, # [\u0026#34;CO\u0026#34;, \u0026#34;VID\u0026#34;, \u0026#34;-\u0026#34;, \u0026#34;19\u0026#34;] \u0026#34;e-commerce\u0026#34;, # [\u0026#34;e\u0026#34;, \u0026#34;-\u0026#34;, \u0026#34;commerce\u0026#34;] \u0026#34;$TSLA\u0026#34;, # [\u0026#34;$\u0026#34;, \u0026#34;TS\u0026#34;, \u0026#34;LA\u0026#34;] ] print(\u0026#34;Benchmark terms: Perfect tokenization\u0026#34;) for term in benchmark_terms: print(f\u0026#34; {term} → {len(tokenizer.encode(term))} token\u0026#34;) print(\u0026#34;\\nReal-world terms: Tokenization disasters\u0026#34;) for term in real_world_terms: tokens = tokenizer.encode(term) print(f\u0026#34; {term} → {len(tokens)} tokens: {[tokenizer.decode([t]) for t in tokens]}\u0026#34;) return \u0026#34;Models ace benchmarks, fail in production\u0026#34; The Fix: Domain-Aligned Tokenization class DomainAlignedEmbeddings: \u0026#34;\u0026#34;\u0026#34;How to fix your embeddings before it\u0026#39;s too late\u0026#34;\u0026#34;\u0026#34; def __init__(self, domain): self.domain = domain def identify_problem_terms(self, corpus): \u0026#34;\u0026#34;\u0026#34;Find terms that tokenize badly\u0026#34;\u0026#34;\u0026#34; problem_terms = [] for doc in corpus: terms = extract_domain_terms(doc) # Your NER/term extraction for term in terms: tokens = tokenizer.encode(term) # Red flags: if len(tokens) \u0026gt; 3: problem_terms.append({ \u0026#39;term\u0026#39;: term, \u0026#39;tokens\u0026#39;: tokens, \u0026#39;fragmentation\u0026#39;: len(tokens), \u0026#39;problem\u0026#39;: \u0026#39;Over-fragmented\u0026#39; }) # Check if fragments are meaningful for token in tokens: decoded = tokenizer.decode([token]) if len(decoded) \u0026lt;= 2 and decoded not in [\u0026#34;.\u0026#34;, \u0026#34;-\u0026#34;, \u0026#34;_\u0026#34;]: problem_terms.append({ \u0026#39;term\u0026#39;: term, \u0026#39;issue\u0026#39;: f\u0026#39;Meaningless fragment: {decoded}\u0026#39; }) return problem_terms def fix_tokenization(self, problem_terms): \u0026#34;\u0026#34;\u0026#34;Three strategies to fix broken embeddings\u0026#34;\u0026#34;\u0026#34; # Strategy 1: Add to vocabulary (if you control training) custom_tokens = [term[\u0026#39;term\u0026#39;] for term in problem_terms[:1000]] # tokenizer.add_tokens(custom_tokens) # Strategy 2: Pre-compute embeddings for problem terms term_embeddings = {} for term in problem_terms: # Don\u0026#39;t use default tokenization # Use character-level or custom embedding term_embeddings[term] = compute_custom_embedding(term) # Strategy 3: Preprocessing substitution substitutions = { \u0026#34;$AAPL\u0026#34;: \u0026#34;AAPL_STOCK\u0026#34;, \u0026#34;COVID-19\u0026#34;: \u0026#34;COVID_NINETEEN\u0026#34;, \u0026#34;e-commerce\u0026#34;: \u0026#34;ecommerce\u0026#34;, } return \u0026#34;Fixed embeddings through tokenization alignment\u0026#34; The Cost of Ignoring This The financial and operational costs of bad tokenization compound exponentially through your entire ML pipeline. When your RAG system returns wrong documents, you\u0026rsquo;re looking at a 30-50% accuracy drop that teams often try to fix by throwing money at larger models typically needing 3x the compute to compensate for what proper tokenization would have solved for free.\nYour semantic search becomes a user experience nightmare. When searches don\u0026rsquo;t return relevant content, users can\u0026rsquo;t find the information they need. This translates directly to lost customers, increased support tickets, and engineers spending weeks debugging \u0026ldquo;search relevance issues\u0026rdquo; that are actually tokenization problems.\nThe clustering and topic modeling failures are equally devastating. When similar concepts don\u0026rsquo;t cluster together because their tokenizations differ, your entire data organization breaks down. Teams resort to manual categorization, burning through data science hours on what should be automated.\nPerhaps most expensive is the fine-tuning inefficiency. When your model can\u0026rsquo;t learn domain concepts due to fragmented tokens, you need 10x more training data to achieve marginal improvements. That\u0026rsquo;s easily $50,000 in additional compute costs, not to mention the opportunity cost of delayed deployments.\nThe cascade is predictable: Bad tokenization → Bad embeddings → Failed retrieval → Larger models → Higher costs → Still bad results → Engineers quit → Project fails. And it all started with a tokenizer that couldn\u0026rsquo;t handle \u0026ldquo;$AAPL\u0026rdquo; properly.\nThe Embedding Quality Test Suite def test_your_embedding_quality(tokenizer, test_pairs): \u0026#34;\u0026#34;\u0026#34;Run this before deploying ANY embedding system\u0026#34;\u0026#34;\u0026#34; critical_tests = [ # Domain terms (\u0026#34;your_product\u0026#34;, \u0026#34;YourProduct\u0026#34;), (\u0026#34;your-product\u0026#34;, \u0026#34;your product\u0026#34;), # Technical terms (\u0026#34;API_KEY\u0026#34;, \u0026#34;API KEY\u0026#34;), (\u0026#34;OAuth2.0\u0026#34;, \u0026#34;OAuth 2.0\u0026#34;), # Financial (\u0026#34;P/E ratio\u0026#34;, \u0026#34;PE ratio\u0026#34;), (\u0026#34;52-week high\u0026#34;, \u0026#34;52 week high\u0026#34;), ] failures = [] for term1, term2 in critical_tests: tokens1 = tokenizer.encode(term1) tokens2 = tokenizer.encode(term2) # These should be similar but tokenize differently if len(tokens1) \u0026gt; 2 or len(tokens2) \u0026gt; 2: similarity = calculate_similarity(term1, term2) if similarity \u0026lt; 0.7: failures.append({ \u0026#39;pair\u0026#39;: (term1, term2), \u0026#39;similarity\u0026#39;: similarity, \u0026#39;tokens\u0026#39;: (tokens1, tokens2) }) if failures: print(\u0026#34;🚨 YOUR EMBEDDINGS WILL FAIL IN PRODUCTION\u0026#34;) return failures return \u0026#34;Embeddings look good (for now)\u0026#34; 💡 Critical Insight: Your $50K vector database is only as good as your $0 tokenizer. Before you scale compute, buy GPUs, or hire ML engineers, run the embedding quality test. If domain terms fragment into 3+ tokens, your embeddings are noise. Fix tokenization first or everything downstream fails.\nTakeaway: Embeddings aren\u0026rsquo;t learned from concepts, they\u0026rsquo;re learned from tokens. When \u0026ldquo;$AAPL\u0026rdquo; becomes [\u0026quot;$\u0026quot;, \u0026ldquo;AA\u0026rdquo;, \u0026ldquo;PL\u0026rdquo;], your model learns the embedding of \u0026ldquo;money + batteries + Poland\u0026rdquo;, not \u0026ldquo;Apple stock\u0026rdquo;. Your semantic search, RAG, and clustering all fail because tokenization shattered meaning into nonsense fragments. The fix isn\u0026rsquo;t more data or bigger models, it\u0026rsquo;s aligning tokenization with your domain FIRST.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/tokenisation_limits/","summary":"\u003ch1 id=\"why-your-vector-database-thinks-aapl-means-polish-batteries\"\u003eWhy Your Vector Database Thinks $AAPL Means Polish Batteries\u003c/h1\u003e\n\u003cp\u003eYour $50K vector database is returning garbage results because \u0026ldquo;$AAPL\u0026rdquo; tokenizes as [\u0026quot;$\u0026quot;, \u0026ldquo;AA\u0026rdquo;, \u0026ldquo;PL\u0026rdquo;] and now has the embedding of \u0026ldquo;dollar + AA batteries + Poland\u0026rdquo;. Your semantic search for \u0026ldquo;Apple stock\u0026rdquo; returns articles about Polish currency. This isn\u0026rsquo;t a retrieval problem, it\u0026rsquo;s tokenization murdering your embeddings.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: Bad tokenization creates noisy embeddings. \u0026ldquo;COVID-19\u0026rdquo; split into [\u0026ldquo;CO\u0026rdquo;, \u0026ldquo;VID\u0026rdquo;, \u0026ldquo;-\u0026rdquo;, \u0026ldquo;19\u0026rdquo;] has embeddings mixing \u0026ldquo;Colorado\u0026rdquo;, \u0026ldquo;video\u0026rdquo;, \u0026ldquo;negative\u0026rdquo;, and \u0026ldquo;2019\u0026rdquo;. Your RAG pipeline is doomed before it starts. Fix tokenization or waste money on larger models trying to compensate.\u003c/p\u003e","title":"Why Your Vector Database Thinks $AAPL Means Polish Batteries"},{"content":"Tokenization Leaks the Training Set (The Forensics Goldmine) Want to know if GPT-4 was trained on your company\u0026rsquo;s leaked data? Check if your internal codenames are single tokens. Want to detect if a model saw specific Reddit posts? The tokenizer already told you.\nTL;DR: Tokenizers are accidental forensic evidence. If SolidGoldMagikarp is a single token, that string appeared thousands of times in training. This is how researchers discovered GPT models trained on specific Reddit users, leaked databases, and private codebases.\nThe Smoking Gun Test # Test if a model was trained on specific data: def training_set_forensics(tokenizer, test_strings): \u0026#34;\u0026#34;\u0026#34;Detect what the model likely saw during training\u0026#34;\u0026#34;\u0026#34; results = {} for string in test_strings: tokens = tokenizer.encode(string) decoded = [tokenizer.decode([t]) for t in tokens] # Single token = appeared frequently in training # Many tokens = rare or unseen if len(tokens) == 1: status = \u0026#34;🔴 DEFINITELY in training (1000s of times)\u0026#34; elif len(tokens) == 2: status = \u0026#34;🟡 Likely in training (100s of times)\u0026#34; elif len(tokens) \u0026lt;= len(string)/4: status = \u0026#34;🟢 Possibly in training\u0026#34; else: status = \u0026#34;⚪ Probably NOT in training\u0026#34; results[string] = { \u0026#39;tokens\u0026#39;: len(tokens), \u0026#39;pieces\u0026#39;: decoded, \u0026#39;status\u0026#39;: status } return results # Try these: suspicious_strings = [ \u0026#34;$AAPL\u0026#34;, # Apple stock \u0026#34;$TSLA\u0026#34;, # Tesla stock \u0026#34;$GME\u0026#34;, # GameStop (meme stock) \u0026#34;SolidGoldMagikarp\u0026#34;, # Reddit username \u0026#34;petertodd\u0026#34;, # Bitcoin developer \u0026#34; davidjl123\u0026#34;, # Another Reddit user \u0026#34;TheNitromeFan\u0026#34;, # Counting subreddit user \u0026#34;\u0026lt;|endoftext|\u0026gt;\u0026#34;, # GPT special token \u0026#34;REDACTED_EMAIL\u0026#34;, # Your company\u0026#39;s placeholder? ] results = training_set_forensics(tokenizer, suspicious_strings) The Reddit Conspiracy (It\u0026rsquo;s Real) # These are ACTUAL single tokens in GPT tokenizers: weird_tokens = { \u0026#34;SolidGoldMagikarp\u0026#34;: 1, # Reddit user with 100K+ comments \u0026#34; petertodd\u0026#34;: 1, # Bitcoin developer mentioned everywhere \u0026#34;TheNitromeFan\u0026#34;: 1, # Power user in r/counting \u0026#34; davidjl123\u0026#34;: 1, # Another counting enthusiast \u0026#34;cloneembryos\u0026#34;: 1, # WTF? Specific subreddit discussions \u0026#34; guiActiveUn\u0026#34;: 1, # Unity game engine internal variable \u0026#34; TPPStreamerBot\u0026#34;: 1, # Twitch Plays Pokemon bot } # Why does this matter? # 1. These users\u0026#39; writing styles are BAKED into the model # 2. Prompting with these tokens triggers specific behaviors # 3. It\u0026#39;s proof of training on Reddit data through 2021 The Domain Detector # Compare tokenizers to detect training bias: def compare_domain_coverage(text, tokenizers_dict): \u0026#34;\u0026#34;\u0026#34;See which model knows your domain best\u0026#34;\u0026#34;\u0026#34; print(f\u0026#34;Testing: \u0026#39;{text}\u0026#39;\u0026#34;) print(\u0026#34;-\u0026#34; * 50) best_score = float(\u0026#39;inf\u0026#39;) best_model = None for name, tokenizer in tokenizers_dict.items(): tokens = tokenizer.encode(text) decoded = [tokenizer.decode([t]) for t in tokens] print(f\u0026#34;{name:15} → {len(tokens)} tokens: {decoded}\u0026#34;) if len(tokens) \u0026lt; best_score: best_score = len(tokens) best_model = name print(f\u0026#34;\\n🏆 {best_model} knows this domain best!\u0026#34;) return best_model # Real examples: finance_test = \u0026#34;$NVDA earnings beat expectations Q4\u0026#34; crypto_test = \u0026#34;dogecoin hodl moon lambo wen\u0026#34; medical_test = \u0026#34;acetaminophen hepatotoxicity cirrhosis\u0026#34; gaming_test = \u0026#34;speedrun any% glitchless PB WR\u0026#34; # Results reveal training data: # GPT-4: Good at finance (WSB in training) # Claude: Better at medical (PubMed heavy) # LLaMA: Gaming terms fragmented (less Reddit) The Corporate Leak Detector # Check if your company\u0026#39;s data leaked into training: company_indicators = [ # Your internal codenames \u0026#34;Project_Phantom\u0026#34;, \u0026#34;PROD_API_KEY_V2\u0026#34;, \u0026#34;TODO_FIXME_HACK\u0026#34;, # Your employee usernames \u0026#34;jsmith@yourcompany\u0026#34;, \u0026#34;jenkins-bot-prod\u0026#34;, # Your internal URLs \u0026#34;internal.yourcompany.com\u0026#34;, \u0026#34;staging-server-2\u0026#34;, # Your error messages \u0026#34;Error: Database connection failed (YourCompany v2.3.1)\u0026#34;, ] for indicator in company_indicators: tokens = tokenizer.encode(indicator) if len(tokens) \u0026lt;= 3: # Suspiciously efficient print(f\u0026#34;⚠️ ALERT: \u0026#39;{indicator}\u0026#39; is {len(tokens)} tokens!\u0026#34;) print(f\u0026#34;This string likely appeared in training data!\u0026#34;) The Easter Egg Hunt (Glitch Tokens) # Some tokens cause BIZARRE behavior when used as prompts: glitch_tokens = { \u0026#34;SolidGoldMagikarp\u0026#34;: \u0026#34;Causes hallucinations about goldfish\u0026#34;, \u0026#34; petertodd\u0026#34;: \u0026#34;Triggers Bitcoin discussions\u0026#34;, \u0026#34;〉〈\u0026#34;: \u0026#34;Makes models output broken HTML\u0026#34;, \u0026#34; Leilan\u0026#34;: \u0026#34;Triggers anime/mythology crossover\u0026#34;, \u0026#34;PsyNetMessage\u0026#34;: \u0026#34;Outputs Rocket League error messages\u0026#34;, \u0026#34; embedreportprint\u0026#34;: \u0026#34;Triggers debug output\u0026#34;, \u0026#34; guiActiveUn\u0026#34;: \u0026#34;Outputs Unity Engine internals\u0026#34;, } # Try this (at your own risk): # prompt = \u0026#34;Tell me about SolidGoldMagikarp\u0026#34; # Models often respond with nonsense about goldfish, magic, or Reddit # WHY THIS HAPPENS: # 1. Token appears thousands of times in specific context # 2. Model memorizes the pattern around it # 3. Token becomes a \u0026#34;trigger\u0026#34; for that context # 4. Using it alone causes context confusion The Stock Market Tell # Financial bias detection: stocks = [\u0026#34;$AAPL\u0026#34;, \u0026#34;$GOOGL\u0026#34;, \u0026#34;$TSLA\u0026#34;, \u0026#34;$GME\u0026#34;, \u0026#34;$AMC\u0026#34;, \u0026#34;$BBBY\u0026#34;, \u0026#34;$PLTR\u0026#34;] for stock in stocks: tokens = len(tokenizer.encode(stock)) if tokens == 1: print(f\u0026#34;{stock}: 📈 Meme stock or high-frequency in training\u0026#34;) elif tokens == 2: print(f\u0026#34;{stock}: 📊 Common stock in financial data\u0026#34;) else: print(f\u0026#34;{stock}: 📉 Rare in training data\u0026#34;) # GPT models: $GME is often 1-2 tokens (WSB influence) # Claude: $GME is 3-4 tokens (less meme stock training) # This reveals: # - GPT saw tons of WallStreetBets data # - Claude focused more on traditional finance # - Training cutoff dates (pre/post meme stock era) The Privacy Nightmare def check_pii_leakage(tokenizer, email_patterns): \u0026#34;\u0026#34;\u0026#34;Check if specific emails/usernames are in training\u0026#34;\u0026#34;\u0026#34; concerning = [] for email in email_patterns: tokens = tokenizer.encode(email) # If an email is \u0026lt;5 tokens, it appeared A LOT if len(tokens) \u0026lt; 5: concerning.append({ \u0026#39;email\u0026#39;: email, \u0026#39;tokens\u0026#39;: len(tokens), \u0026#39;risk\u0026#39;: \u0026#39;HIGH - Likely memorized\u0026#39; }) return concerning # Real researcher findings: leaked_emails = [ \u0026#34;support@company.com\u0026#34;, # 2 tokens - appeared thousands of times \u0026#34;john.doe.1234@gmail\u0026#34;, # 8 tokens - probably safe \u0026#34;ceo@fortune500.com\u0026#34;, # 3 tokens - concerning! ] # Researchers found actual email addresses as single tokens # meaning they appeared THOUSANDS of times in training How to Exploit This (Ethically) class TokenizerForensics: \u0026#34;\u0026#34;\u0026#34;Use tokenization to understand model capabilities\u0026#34;\u0026#34;\u0026#34; def detect_programming_languages(self, tokenizer): \u0026#34;\u0026#34;\u0026#34;Which languages did it see most?\u0026#34;\u0026#34;\u0026#34; code_snippets = { \u0026#39;Python\u0026#39;: \u0026#39;def __init__(self):\u0026#39;, \u0026#39;JavaScript\u0026#39;: \u0026#39;const async = await\u0026#39;, \u0026#39;Rust\u0026#39;: \u0026#39;fn main() -\u0026gt; Result\u0026lt;\u0026gt;\u0026#39;, \u0026#39;Go\u0026#39;: \u0026#39;func (r *Reader) Read()\u0026#39;, \u0026#39;COBOL\u0026#39;: \u0026#39;IDENTIFICATION DIVISION\u0026#39;, } for lang, snippet in code_snippets.items(): tokens = len(tokenizer.encode(snippet)) efficiency = len(snippet) / tokens if efficiency \u0026gt; 10: print(f\u0026#34;{lang}: Heavily trained on this\u0026#34;) elif efficiency \u0026gt; 5: print(f\u0026#34;{lang}: Moderate training\u0026#34;) else: print(f\u0026#34;{lang}: Minimal training\u0026#34;) def detect_website_training(self, tokenizer): \u0026#34;\u0026#34;\u0026#34;Which websites were scraped?\u0026#34;\u0026#34;\u0026#34; sites = [ \u0026#34;reddit.com/r/\u0026#34;, \u0026#34;stackoverflow.com/questions/\u0026#34;, \u0026#34;github.com/\u0026#34;, \u0026#34;arxiv.org/abs/\u0026#34;, \u0026#34;medium.com/@\u0026#34;, \u0026#34;substack.com/p/\u0026#34;, ] for site in sites: tokens = len(tokenizer.encode(site)) if tokens \u0026lt;= 3: print(f\u0026#34;✓ {site} - Heavily scraped\u0026#34;) else: print(f\u0026#34;✗ {site} - Lightly scraped\u0026#34;) The Uncomfortable Implications Your Reddit posts are tokens: Specific users\u0026rsquo; entire post histories are compressed into the model\nCorporate leaks are detectable: Internal codenames as single tokens = data breach\nModel capabilities are predictable: Bad tokenization = bad performance in that domain\nPrivacy is already broken: Email addresses, usernames, and phone numbers exist as tokens\nTemporal information leaks: Token efficiency reveals WHEN data was collected\n💡 Action Item: Run training_set_forensics() on your company\u0026rsquo;s internal terminology. If anything is ≤3 tokens, you might have a leak. Check your competitor\u0026rsquo;s terminology too, you might discover their training data sources.\nTakeaway: Tokenizers are inadvertent forensic evidence. Every merge decision is a fossil record of the training data. Single tokens for usernames, stock symbols, or internal codenames aren\u0026rsquo;t bugs, they\u0026rsquo;re proof of what the model memorized. Use this for good (detecting capabilities) or evil (finding leaks), but know that the tokenizer already spilled the secrets.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/forensics/","summary":"\u003ch1 id=\"tokenization-leaks-the-training-set-the-forensics-goldmine\"\u003eTokenization Leaks the Training Set (The Forensics Goldmine)\u003c/h1\u003e\n\u003cp\u003eWant to know if GPT-4 was trained on your company\u0026rsquo;s leaked data? Check if your internal codenames are single tokens. Want to detect if a model saw specific Reddit posts? The tokenizer already told you.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: Tokenizers are accidental forensic evidence. If \u003ccode\u003eSolidGoldMagikarp\u003c/code\u003e is a single token, that string appeared thousands of times in training. This is how researchers discovered GPT models trained on specific Reddit users, leaked databases, and private codebases.\u003c/p\u003e","title":"Tokenization Forensics about Leaks"},{"content":"Byte-Level Tokenizers Can Bloat Non-English (The Colonial Tax) Hook: Your Hindi users pay 17x more than English users for the same word. Your Arabic users\u0026rsquo; prompts fail because they hit token limits 8x faster. This isn\u0026rsquo;t a bug—it\u0026rsquo;s algorithmic colonialism baked into your tokenizer.\nTL;DR: Tokenizers trained on English-heavy data punish non-Latin scripts with massive token inflation. \u0026ldquo;Internationalization\u0026rdquo; = 1 token in English, 17 tokens in Hindi. Your global users are subsidizing your English users, and they\u0026rsquo;re getting worse model performance too.\nThe Shocking Reality Check # Same concept, wildly different costs: word = \u0026#34;internationalization\u0026#34; English: \u0026#34;internationalization\u0026#34; → 1 token ($0.00003) Hindi: \u0026#34;अंतरराष्ट्रीयीकरण\u0026#34; → 17 tokens ($0.00051) Chinese: \u0026#34;国际化\u0026#34; → 3 tokens ($0.00009) Arabic: \u0026#34;تدويل\u0026#34; → 4 tokens ($0.00012) Russian: \u0026#34;интернационализация\u0026#34; → 8 tokens ($0.00024) # Your Hindi users pay 1,700% more for THE SAME CONCEPT What\u0026rsquo;s Actually Happening Under the Hood Byte-level BPE starts from UTF-8 bytes. Here\u0026rsquo;s the brutal math:\n# UTF-8 encoding sizes: \u0026#34;A\u0026#34; → 1 byte → likely 1 token \u0026#34;é\u0026#34; → 2 bytes → often 1-2 tokens \u0026#34;中\u0026#34; → 3 bytes → usually 2-3 tokens \u0026#34;🤔\u0026#34; → 4 bytes → often 3-4 tokens \u0026#34;अ\u0026#34; → 3 bytes → usually 2-3 tokens # But it gets WORSE with frequency bias: \u0026#34;the\u0026#34; → Seen millions of times → 1 token \u0026#34;और\u0026#34; → (Hindi \u0026#34;and\u0026#34;) Seen rarely → 2-3 tokens \u0026#34;的\u0026#34; → (Chinese \u0026#34;of\u0026#34;) Common in Chinese data → 1 token \u0026#34;של\u0026#34; → (Hebrew \u0026#34;of\u0026#34;) Rare in training → 3 tokens The tokenizer literally learns: \u0026ldquo;English patterns deserve compression, others can pay extra.\u0026rdquo;\nThe Compound Word Disaster Watch how technical terms explode:\n# English: Efficient compound handling \u0026#34;machine learning\u0026#34; → 2 tokens \u0026#34;artificial intelligence\u0026#34; → 2 tokens \u0026#34;blockchain technology\u0026#34; → 2 tokens # Hindi: Each syllable becomes multiple tokens \u0026#34;मशीन लर्निंग\u0026#34; → 8-10 tokens \u0026#34;कृत्रिम बुद्धिमत्ता\u0026#34; → 12-15 tokens \u0026#34;ब्लॉकचेन तकनीक\u0026#34; → 10-12 tokens # German (even Latin script suffers!): \u0026#34;Maschinelles Lernen\u0026#34; → 4 tokens \u0026#34;Donaudampfschifffahrtsgesellschaftskapitän\u0026#34; → 15+ tokens # (Danube steamship company captain) The Hidden Performance Penalty It\u0026rsquo;s not just about cost—non-English users get WORSE models:\n# Effective context window for 4K token limit: English users: 3,000 words of context Hindi users: 500-800 words of context Chinese users: 1,000-1,500 characters Arabic users: 400-600 words of context # Prompt complexity you can handle: English: \u0026#34;Write a detailed 10-step guide with examples\u0026#34; → Fits easily Hindi: \u0026#34;विस्तृत 10-चरण गाइड लिखें\u0026#34; → Already 30% of budget! Real Production Disasters Disaster 1: The Customer Support Bot Meltdown # English customer: Full conversation history fits messages = [ \u0026#34;I need help with my order\u0026#34;, \u0026#34;It was supposed to arrive yesterday\u0026#34;, \u0026#34;Order number is 12345\u0026#34;, # ... 20 more messages ] # Total: 200 tokens # Arabic customer: Truncated after 5 messages messages = [ \u0026#34;أحتاج مساعدة مع طلبي\u0026#34;, \u0026#34;كان من المفترض أن يصل أمس\u0026#34;, \u0026#34;رقم الطلب هو 12345\u0026#34;, # ... only 3 more messages fit ] # Total: 400 tokens (context window hit!) # Result: Arabic customers get \u0026#34;goldfish memory\u0026#34; support Disaster 2: The Translation Paradox # You translate your prompts to be inclusive: en_prompt = \u0026#34;Summarize this document\u0026#34; # 4 tokens hi_prompt = \u0026#34;इस दस्तावेज़ का सारांश दें\u0026#34; # 12 tokens # You just 3x\u0026#39;d your costs trying to be inclusive! # Many companies give up and force English-only The Market Reality: Who Gets Screwed? # Token efficiency by language (GPT-4 tokenizer): efficiency_scores = { \u0026#34;English\u0026#34;: 1.0, # Baseline \u0026#34;Spanish\u0026#34;: 1.2, # 20% penalty \u0026#34;French\u0026#34;: 1.15, # 15% penalty \u0026#34;German\u0026#34;: 1.3, # 30% penalty \u0026#34;Chinese\u0026#34;: 2.5, # 150% penalty \u0026#34;Japanese\u0026#34;: 2.2, # 120% penalty \u0026#34;Hindi\u0026#34;: 3.5, # 250% penalty \u0026#34;Arabic\u0026#34;: 4.0, # 300% penalty \u0026#34;Thai\u0026#34;: 4.5, # 350% penalty \u0026#34;Bengali\u0026#34;: 4.2, # 320% penalty } # If English users pay $100/month: # Bengali users pay $420/month for same usage How to Fix This Injustice Fix 1: The Router Pattern (Use Specialized Models) def smart_model_router(text, detect_language_fn): \u0026#34;\u0026#34;\u0026#34;Route to language-optimized models\u0026#34;\u0026#34;\u0026#34; language = detect_language_fn(text) # Use models with better tokenizers for each language model_map = { \u0026#39;en\u0026#39;: \u0026#39;gpt-4\u0026#39;, # Optimized for English \u0026#39;zh\u0026#39;: \u0026#39;qwen-plus\u0026#39;, # Chinese-optimized tokenizer \u0026#39;hi\u0026#39;: \u0026#39;llama-3-indic\u0026#39;, # Indic language specialist \u0026#39;ar\u0026#39;: \u0026#39;jais-30b\u0026#39;, # Arabic-optimized \u0026#39;multi\u0026#39;: \u0026#39;aya-23b\u0026#39;, # Multilingual balanced } return model_map.get(language, \u0026#39;aya-23b\u0026#39;) # Save 50-70% on non-English queries Fix 2: The Preprocessing Hack (Transliteration) def reduce_hindi_tokens(text): \u0026#34;\u0026#34;\u0026#34;Controversial but effective: Romanize for tokenization\u0026#34;\u0026#34;\u0026#34; # Transliterate to Latin script (Hinglish style) # \u0026#34;मशीन लर्निंग\u0026#34; → \u0026#34;machine learning\u0026#34; # 8 tokens → 2 tokens (75% reduction!) transliterated = transliterate_to_latin(text) # Process with English-optimized tokenizer response = model.generate(transliterated) # Translate back if needed return transliterate_back(response) # Cuts costs by 60-80% for Indic languages # Trade-off: Loses some nuance Fix 4: The Vocabulary Expansion (If You Control Training) # Add frequent non-English tokens to vocabulary def expand_tokenizer_vocabulary(base_tokenizer, target_languages): \u0026#34;\u0026#34;\u0026#34;Add common words from target languages as single tokens\u0026#34;\u0026#34;\u0026#34; critical_tokens = { \u0026#39;hi\u0026#39;: [\u0026#39;और\u0026#39;, \u0026#39;के\u0026#39;, \u0026#39;है\u0026#39;, \u0026#39;में\u0026#39;, \u0026#39;की\u0026#39;], # Hindi common words \u0026#39;ar\u0026#39;: [\u0026#39;في\u0026#39;, \u0026#39;من\u0026#39;, \u0026#39;على\u0026#39;, \u0026#39;إلى\u0026#39;], # Arabic common words \u0026#39;zh\u0026#39;: [\u0026#39;的\u0026#39;, \u0026#39;是\u0026#39;, \u0026#39;了\u0026#39;, \u0026#39;在\u0026#39;], # Chinese particles } for lang, tokens in critical_tokens.items(): if lang in target_languages: base_tokenizer.add_tokens(tokens) return base_tokenizer # Reduces token count by 30-40% for target languages Fix 5: The Prompt Caching Strategy class MultilingualPromptCache: \u0026#34;\u0026#34;\u0026#34;Cache tokenized versions of common prompts\u0026#34;\u0026#34;\u0026#34; def __init__(self, tokenizer): self.tokenizer = tokenizer self.cache = {} # Pre-tokenize common prompts in all languages self.common_prompts = { \u0026#39;summarize\u0026#39;: { \u0026#39;en\u0026#39;: \u0026#34;Summarize this text:\u0026#34;, \u0026#39;hi\u0026#39;: \u0026#34;इस पाठ का सारांश दें:\u0026#34;, \u0026#39;zh\u0026#39;: \u0026#34;总结这段文字：\u0026#34;, \u0026#39;ar\u0026#39;: \u0026#34;لخص هذا النص:\u0026#34;, } } # Pre-compute token counts for task, translations in self.common_prompts.items(): for lang, prompt in translations.items(): tokens = tokenizer.encode(prompt) self.cache[f\u0026#34;{task}_{lang}\u0026#34;] = { \u0026#39;tokens\u0026#39;: tokens, \u0026#39;count\u0026#39;: len(tokens), \u0026#39;cost\u0026#39;: len(tokens) * 0.00003 } def get_cheapest_prompt(self, task): \u0026#34;\u0026#34;\u0026#34;Return the most token-efficient version\u0026#34;\u0026#34;\u0026#34; options = [k for k in self.cache if k.startswith(task)] return min(options, key=lambda x: self.cache[x][\u0026#39;count\u0026#39;]) The Benchmark: Test Your Bias def tokenization_bias_test(tokenizer, test_phrase=\u0026#34;Hello, how are you?\u0026#34;): \u0026#34;\u0026#34;\u0026#34;Measure your tokenizer\u0026#39;s language bias\u0026#34;\u0026#34;\u0026#34; translations = { \u0026#39;English\u0026#39;: \u0026#34;Hello, how are you?\u0026#34;, \u0026#39;Spanish\u0026#39;: \u0026#34;Hola, ¿cómo estás?\u0026#34;, \u0026#39;Hindi\u0026#39;: \u0026#34;नमस्ते, आप कैसे हैं?\u0026#34;, \u0026#39;Chinese\u0026#39;: \u0026#34;你好，你好吗？\u0026#34;, \u0026#39;Arabic\u0026#39;: \u0026#34;مرحبا، كيف حالك؟\u0026#34;, \u0026#39;Russian\u0026#39;: \u0026#34;Привет, как дела?\u0026#34;, } baseline = len(tokenizer.encode(translations[\u0026#39;English\u0026#39;])) print(f\u0026#34;{\u0026#39;Language\u0026#39;:\u0026lt;12} {\u0026#39;Tokens\u0026#39;:\u0026lt;8} {\u0026#39;Penalty\u0026#39;:\u0026lt;10} {\u0026#39;Extra Cost\u0026#39;}\u0026#34;) print(\u0026#34;-\u0026#34; * 45) for lang, text in translations.items(): tokens = len(tokenizer.encode(text)) penalty = (tokens / baseline - 1) * 100 extra_cost = (tokens - baseline) * 0.00003 status = \u0026#34;✅\u0026#34; if penalty \u0026lt; 50 else \u0026#34;⚠️\u0026#34; if penalty \u0026lt; 100 else \u0026#34;🚨\u0026#34; print(f\u0026#34;{lang:\u0026lt;12} {tokens:\u0026lt;8} {penalty:\u0026gt;6.0f}% {status} ${extra_cost:.5f}\u0026#34;) return \u0026#34;Your tokenizer\u0026#39;s bias level\u0026#34; # Run this test - anything over 100% penalty is problematic The Uncomfortable Truth Most tokenizers are trained on:\n60% English web text 20% Western European languages 10% Chinese (if you\u0026rsquo;re lucky) 10% \u0026ldquo;Other\u0026rdquo; (3 billion people crammed into 10%) This means:\nEnglish speakers get subsidized AI Global South pays the \u0026ldquo;tokenization tax\u0026rdquo; Models perform worse on non-English tasks True multilingual AI remains expensive 💡 Action Item: Calculate your non-English user percentage and their token multiplier. If you have 20% Hindi users paying 3.5x more tokens, you\u0026rsquo;re leaving money on the table AND providing inferior service. Implement Fix 1 (Router Pattern) this week.\nTakeaway: Tokenization isn\u0026rsquo;t neutral, it\u0026rsquo;s a choice about who pays more and whose languages matter. Every English optimized tokenizer is effectively a tax on the Global South. Measure your bias, route intelligently, and stop making your Hindi users subsidize your English ones.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/byte-level-tokenizer/","summary":"\u003ch1 id=\"byte-level-tokenizers-can-bloat-non-english-the-colonial-tax\"\u003eByte-Level Tokenizers Can Bloat Non-English (The Colonial Tax)\u003c/h1\u003e\n\u003cp\u003e\u003cstrong\u003eHook:\u003c/strong\u003e Your Hindi users pay 17x more than English users for the same word. Your Arabic users\u0026rsquo; prompts fail because they hit token limits 8x faster. This isn\u0026rsquo;t a bug—it\u0026rsquo;s algorithmic colonialism baked into your tokenizer.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: Tokenizers trained on English-heavy data punish non-Latin scripts with massive token inflation. \u0026ldquo;Internationalization\u0026rdquo; = 1 token in English, 17 tokens in Hindi. Your global users are subsidizing your English users, and they\u0026rsquo;re getting worse model performance too.\u003c/p\u003e","title":"Byte Level Tokenizer"},{"content":"The Tokenization Papers: Why 90% of LLM Failures Start Here The Hidden Layer That Controls Everything Every prompt you send to GPT, Claude, or Gemini gets shredded into tokens before the model even \u0026ldquo;thinks.\u0026rdquo; These aren\u0026rsquo;t words — they\u0026rsquo;re compression artifacts from 2021 web crawls that now dictate:\nYour API bill (why Hindi costs 17x more than English) Your model\u0026rsquo;s IQ (why it thinks 9.11 \u0026gt; 9.9) Your RAG accuracy (why $AAPL returns articles about Polish batteries) Tokenization is the silent killer of production AI systems. These papers expose the disasters hiding in plain sight.\nThe Papers 1. The Colonial Tax: Why Non-English Users Pay 17x More Hindi text uses 17x more tokens. Arabic hits context limits 8x faster. Chinese users burn through budgets 5x quicker. Your \u0026ldquo;multilingual\u0026rdquo; model is financially punishing 80% of the world — and you\u0026rsquo;re losing customers because of compression algorithms from 2019.\n2. Tokenization Forensics: Finding Training Data Leaks Single tokens like SolidGoldMagikarp are smoking guns proving your data was in the training set. Learn how tokenizers accidentally become forensic tools that expose what models were trained on — including your proprietary code.\n3. Numbers Are Broken: Why LLMs Think 9.11 \u0026gt; 9.9 Your model isn\u0026rsquo;t bad at math — it literally sees different numbers than you do. \u0026ldquo;2024\u0026rdquo; is one token, \u0026ldquo;2025\u0026rdquo; is two tokens, \u0026ldquo;1234\u0026rdquo; doesn\u0026rsquo;t exist. Watch tokenization murder arithmetic, dates, and any hope of numerical reasoning.\n4. One Space Changes Everything: Tokenization as Mind Control \u0026quot;Hello\u0026quot; makes a helpful assistant. \u0026quot; Hello\u0026quot; (with a leading space) triggers aggressive responses. \u0026quot;HELLO\u0026quot; changes the entire personality. Invisible tokenization boundaries are controlling your model\u0026rsquo;s behavior — and you don\u0026rsquo;t even know it.\n5. What Tokens Actually Are (Spoiler: Not Words) Tokens are byte-pair frequency hacks from random internet text. A hyphen, a capital letter, or a Unicode variant can 10x your costs. This paper explains why tokens exist, how they\u0026rsquo;re built, and why they\u0026rsquo;re breaking your system.\n6. Your $50K Vector Database Thinks $AAPL = Polish Batteries When \u0026ldquo;$AAPL\u0026rdquo; tokenizes as [\u0026quot;$\u0026quot;, \u0026ldquo;AA\u0026rdquo;, \u0026ldquo;PL\u0026rdquo;], your embeddings combine \u0026ldquo;currency\u0026rdquo; + \u0026ldquo;batteries\u0026rdquo; + \u0026ldquo;Poland\u0026rdquo;. Your semantic search for \u0026ldquo;Apple stock\u0026rdquo; returns nonsense because tokenization murdered meaning before embeddings even had a chance.\n7. Why Your Model Can\u0026rsquo;t Learn Your Concepts (Even After 50K Examples) You annotated 50,000 examples of \u0026ldquo;TurboIN\u0026rdquo; but your model still thinks it\u0026rsquo;s about turbochargers in Indiana. The embedding space was frozen at pretraining — you\u0026rsquo;re teaching patterns, not concepts. Here\u0026rsquo;s what actually works.\n8. Hidden Tokenization Bombs: The Disasters Nobody Talks About \u0026ldquo;therapist\u0026rdquo; vs \u0026ldquo;the rapist\u0026rdquo; with invisible Unicode Medical apps failing because μg ≠ µg (different Unicode) French \u0026ldquo;pain\u0026rdquo; (bread) neurons firing in English medical text How fine-tuning \u0026ldquo;$AAPL\u0026rdquo; breaks currency processing globally 9. The $10M Question: Should You Train Your Own Tokenizer? Spoiler: No. Unless you\u0026rsquo;re Google, you\u0026rsquo;re better off with 500 lines of preprocessing. BioBERT spent millions on medical tokenization and still fragments \u0026ldquo;Immunoglobulin\u0026rdquo; into 7 pieces. Here\u0026rsquo;s the decision tree that saves you from bankruptcy.\nStart Here New to tokenization disasters? → Start with What Tokens Actually Are\nDealing with non-English users? → The Colonial Tax\nBuilding RAG/vector search? → $AAPL = Polish Batteries\nDebugging weird model behavior? → One Space Changes Everything\nConsidering custom tokenization? → The $10M Question\nThe One-Line Summary Your model isn\u0026rsquo;t broken. Your tokenizer is. And it\u0026rsquo;s costing you millions.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/tokenization/","summary":"\u003ch1 id=\"the-tokenization-papers-why-90-of-llm-failures-start-here\"\u003eThe Tokenization Papers: Why 90% of LLM Failures Start Here\u003c/h1\u003e\n\u003ch2 id=\"the-hidden-layer-that-controls-everything\"\u003eThe Hidden Layer That Controls Everything\u003c/h2\u003e\n\u003cp\u003eEvery prompt you send to GPT, Claude, or Gemini gets shredded into tokens before the model even \u0026ldquo;thinks.\u0026rdquo; These aren\u0026rsquo;t words — they\u0026rsquo;re \u003cstrong\u003ecompression artifacts from 2021 web crawls\u003c/strong\u003e that now dictate:\u003c/p\u003e\n\u003cul\u003e\n\u003cli\u003e\u003cstrong\u003eYour API bill\u003c/strong\u003e (why Hindi costs 17x more than English)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eYour model\u0026rsquo;s IQ\u003c/strong\u003e (why it thinks 9.11 \u0026gt; 9.9)\u003c/li\u003e\n\u003cli\u003e\u003cstrong\u003eYour RAG accuracy\u003c/strong\u003e (why $AAPL returns articles about Polish batteries)\u003c/li\u003e\n\u003c/ul\u003e\n\u003cp\u003eTokenization is the silent killer of production AI systems. These papers expose the disasters hiding in plain sight.\u003c/p\u003e","title":"Tokenisation: Why 90% of LLM Failures Start Here"},{"content":"The Tokenization Decision Tree: When to Train, When to Run, When to Cry Hook: A biotech company spent $2M training a custom medical tokenizer for their revolutionary drug discovery model. Six months later, they switched to GPT-4 with a 500-line preprocessing script. It performed better. Their custom tokenizer? Now it\u0026rsquo;s a $2M reminder that sometimes the \u0026ldquo;right\u0026rdquo; solution is the wrong solution.\nTL;DR: Training your own tokenizer means training your own model ($10M minimum). Extending tokenizers breaks everything. Most \u0026ldquo;tokenization problems\u0026rdquo; are solved better with preprocessing hacks than proper solutions. Here\u0026rsquo;s the decision tree that will save you millions and your sanity.\n⚖️ The Brutal Truth About Your Options Your Real Choices (Ranked by Pain Level) Option Cost Time Success Rate When It Makes Sense Use existing + preprocessing $0 1 week 85% Almost always Switch to better-tokenizing model $X (API costs) 1 day 70% When available Fine-tune with careful data $10-50K 1 month 40% Narrow domains Extend existing tokenizer $500K+ 3 months 10% Never Train tokenizer + model $10M+ 6-12 months 30% You\u0026rsquo;re Google The shocking reality: 90% of teams should pick option 1 and stop overthinking.\n🔍 How to Audit Your Domain\u0026rsquo;s Tokenization Health The 5-Minute Domain Tokenization Test def tokenization_health_check(tokenizer, domain_texts): \u0026#34;\u0026#34;\u0026#34;Run this before making ANY decisions\u0026#34;\u0026#34;\u0026#34; critical_metrics = { \u0026#34;catastrophic\u0026#34;: [], # \u0026gt;5 tokens \u0026#34;bad\u0026#34;: [], # 4-5 tokens \u0026#34;problematic\u0026#34;: [], # 3 tokens \u0026#34;acceptable\u0026#34;: [], # 2 tokens \u0026#34;perfect\u0026#34;: [] # 1 token } # Extract your domain\u0026#39;s critical terms domain_terms = extract_domain_specific_terms(domain_texts) for term in domain_terms: tokens = tokenizer.encode(term) token_count = len(tokens) if token_count \u0026gt;= 5: critical_metrics[\u0026#34;catastrophic\u0026#34;].append(term) elif token_count == 4: critical_metrics[\u0026#34;bad\u0026#34;].append(term) # ... etc # The verdict if len(critical_metrics[\u0026#34;catastrophic\u0026#34;]) \u0026gt; 10: return \u0026#34;ABANDON SHIP - Switch models or domains\u0026#34; elif len(critical_metrics[\u0026#34;bad\u0026#34;]) \u0026gt; 50: return \u0026#34;Major preprocessing required\u0026#34; elif len(critical_metrics[\u0026#34;problematic\u0026#34;]) \u0026gt; 100: return \u0026#34;Preprocessing recommended\u0026#34; else: return \u0026#34;You\u0026#39;re fine, stop worrying\u0026#34; The Domain-Specific Reality Check Your Domain Tokenization Disaster What To Do Medical/Pharma Drug names fragment (Pembrolizumab → 5 tokens) Preprocessing substitution Finance Tickers fragment ($AAPL → 3 tokens) Use dedicated finance models Legal Citations fragment (§2.3.4(a)(ii) → 12 tokens) Create citation aliases Scientific Chemical names (C₆H₁₂O₆ → 8 tokens) SMILES notation + preprocessing E-commerce Product codes (SKU-12345-XL → 6 tokens) Normalize before tokenization Code Variable names (getUserById → 4 tokens) Use code-specific models 🎯 The \u0026ldquo;Should I Train My Own Tokenizer?\u0026rdquo; Test Question 1: Do you have $10M? No → Stop here. Use preprocessing.\nYes → Continue to question 2.\nQuestion 2: Do you have 6-12 months? No → Stop here. Use preprocessing.\nYes → Continue to question 3.\nQuestion 3: Is your domain completely unlike anything in existence? No → Stop here. Use preprocessing.\nYes → Continue to question 4.\nQuestion 4: Do you have a team that\u0026rsquo;s built LLMs before? No → Stop here. Use preprocessing.\nYes → Continue to question 5.\nQuestion 5: Are you sure you\u0026rsquo;re not just empire building? No → Use preprocessing.\nYes → You\u0026rsquo;re lying, but okay, train your tokenizer and learn the hard way.\n💀 Why Training Your Own Tokenizer Is Usually Suicide The Hidden Dependencies Nobody Mentions Training a tokenizer means:\nCollecting domain corpus (100GB minimum, ideally 1TB+) Training the tokenizer (BPE/WordPiece/Unigram) Training a model from scratch (tokenizer → embeddings → transformer) Achieving GPT-4o level performance (good luck) Maintaining it forever (hiring, infrastructure, updates) The Biotech Disaster: They trained a tokenizer on PubMed + clinical trials. It perfectly tokenized drug names! But it couldn\u0026rsquo;t handle basic English anymore. \u0026ldquo;The patient feels better\u0026rdquo; tokenized worse than in GPT-4o. Their domain-specific gain was destroyed by general capability loss.\nThe Vocabulary Size Trap Vocabulary Size Training Cost Inference Speed Quality 32K tokens $5M Fast Poor coverage 50K tokens $8M Balanced Standard (GPT-3) 100K tokens $12M Slower Good coverage 250K tokens $20M Slow Diminishing returns The cruel irony: Larger vocabulary = better tokenization but slower inference and higher training costs. You\u0026rsquo;ll go bankrupt before finding the sweet spot.\n🔧 The Preprocessing Hacks That Actually Work Strategy 1: Token Substitution (The $0 Solution) class TokenSubstitution: \u0026#34;\u0026#34;\u0026#34;What actually works in production\u0026#34;\u0026#34;\u0026#34; def __init__(self): self.substitutions = { # Medical \u0026#34;COVID-19\u0026#34;: \u0026#34;COVIDNINETEEN\u0026#34;, \u0026#34;SARS-CoV-2\u0026#34;: \u0026#34;SARSCOVTWO\u0026#34;, # Finance \u0026#34;$AAPL\u0026#34;: \u0026#34;AAPL_STOCK\u0026#34;, \u0026#34;$GOOGL\u0026#34;: \u0026#34;GOOGL_STOCK\u0026#34;, # E-commerce \u0026#34;e-commerce\u0026#34;: \u0026#34;ecommerce\u0026#34;, \u0026#34;multi-channel\u0026#34;: \u0026#34;multichannel\u0026#34;, # Your domain \u0026#34;TurboIN\u0026#34;: \u0026#34;TURBOINDEX_INDIA\u0026#34;, } def preprocess(self, text): \u0026#34;\u0026#34;\u0026#34;Run before tokenization\u0026#34;\u0026#34;\u0026#34; for bad, good in self.substitutions.items(): text = text.replace(bad, good) return text def postprocess(self, text): \u0026#34;\u0026#34;\u0026#34;Run after generation\u0026#34;\u0026#34;\u0026#34; for bad, good in self.substitutions.items(): text = text.replace(good, bad) return text Success story: A medical AI company had 847 drug names that tokenized badly. Instead of retraining, they built a substitution dictionary. Development time: 3 days. Performance improvement: 34%. Cost: $0.\nStrategy 2: Contextual Expansion def contextual_expansion(text): \u0026#34;\u0026#34;\u0026#34;Add context to help the model understand fragments\u0026#34;\u0026#34;\u0026#34; expansions = { \u0026#34;TurboIN\u0026#34;: \u0026#34;TurboIN (Turbo Index for India)\u0026#34;, \u0026#34;QRAG\u0026#34;: \u0026#34;QRAG (Quantum RAG)\u0026#34;, \u0026#34;KAN\u0026#34;: \u0026#34;KAN (Kolmogorov-Arnold Networks)\u0026#34;, } # First occurrence gets expansion for term, expansion in expansions.items(): text = text.replace(term, expansion, 1) # Only first occurrence return text Strategy 3: The Nuclear Preprocessing Option When nothing else works, go full nuclear:\ndef nuclear_preprocessing(text): \u0026#34;\u0026#34;\u0026#34;When you\u0026#39;re desperate and need it to work\u0026#34;\u0026#34;\u0026#34; # Replace all problematic characters text = text.replace(\u0026#34;-\u0026#34;, \u0026#34;_\u0026#34;) # Hyphens fragment everything text = text.replace(\u0026#34;.\u0026#34;, \u0026#34;_\u0026#34;) # Periods are chaos text = text.replace(\u0026#34;/\u0026#34;, \u0026#34;_\u0026#34;) # Slashes are death # Normalize everything text = text.lower() # Consistent casing text = re.sub(r\u0026#39;\\s+\u0026#39;, \u0026#39; \u0026#39;, text) # Single spaces # Create compound words text = text.replace(\u0026#34;e commerce\u0026#34;, \u0026#34;ecommerce\u0026#34;) text = text.replace(\u0026#34;multi modal\u0026#34;, \u0026#34;multimodal\u0026#34;) text = text.replace(\u0026#34;pre training\u0026#34;, \u0026#34;pretraining\u0026#34;) return text 🎪 When to Switch Models Instead The Model Selection Matrix Model Best For Tokenization Strength When to Choose GPT-4 General + English Good for common terms Default choice Claude Long documents Better punctuation handling Documents with complex formatting Gemini Multilingual Excellent non-English International domains Llama 3 Open source needs Good, 128K vocabulary When you need control Mistral European languages Better for accents/diacritics European market Command-R RAG applications Optimized for retrieval Search-heavy applications Domain-specific Narrow domains Perfect for that domain Only if it exists The Quick Test def model_selection_test(models, test_phrases): \u0026#34;\u0026#34;\u0026#34;Which model tokenizes your domain best?\u0026#34;\u0026#34;\u0026#34; results = {} for model in models: total_tokens = 0 for phrase in test_phrases: tokens = model.tokenize(phrase) total_tokens += len(tokens) results[model.name] = { \u0026#34;total_tokens\u0026#34;: total_tokens, \u0026#34;avg_tokens\u0026#34;: total_tokens / len(test_phrases) } # The model with lowest token count wins return sorted(results.items(), key=lambda x: x[1][\u0026#34;total_tokens\u0026#34;]) 🚨 When Extending a Tokenizer Destroys Everything The $500K Mistake Pattern What companies try:\nTake GPT-4o\u0026rsquo;s tokenizer Add 1000 domain terms Fine-tune the model Watch it fail spectacularly Why it fails:\nNew tokens have random embeddings Model wasn\u0026rsquo;t trained with these tokens Attention patterns are all wrong Position encodings don\u0026rsquo;t align You created 1000 [UNK] tokens with extra steps Real disaster: A legal tech company added 500 legal terms to GPT-3\u0026rsquo;s tokenizer. The model couldn\u0026rsquo;t even complete sentences anymore. Every legal term became a \u0026ldquo;stop token\u0026rdquo; that broke generation. $500K and 3 months wasted.\n📊 The Decision Framework That Actually Works For 99% of Companies Is your domain tokenizing horribly? ├─ No → Use the model as-is └─ Yes → Can you preprocess around it? ├─ Yes → Build preprocessing pipeline (1 week) └─ No → Is there a better-tokenizing model? ├─ Yes → Switch models └─ No → Are you Google/OpenAI/Anthropic? ├─ Yes → Train from scratch └─ No → Preprocessing is your only option The Domain-Specific Tokenizer Reality Domain \u0026ldquo;Proper\u0026rdquo; Solution What Actually Works Success Rate Medical BioGPT, PubMedBERT GPT-4o + substitutions 85% vs 60% Legal LegalBERT Claude + formatting 80% vs 65% Finance FinBERT GPT-4o + ticker cleanup 90% vs 70% Code CodeLlama Already good! 95% The pattern: Domain-specific models have better tokenization but worse overall performance. General models with preprocessing beat specialized models.\n🎯 The Production Checklist Before You Do ANYTHING Run the tokenization health check (5 minutes) Count critical bad terms (\u0026lt;100? Preprocess. \u0026gt;1000? Cry.) Test preprocessing impact (Usually solves 80%) Compare model options (Different model might be free solution) Calculate real costs (Training = $10M minimum) The Preprocessing Pipeline That Always Works class ProductionTokenizationPipeline: \u0026#34;\u0026#34;\u0026#34;What every company eventually builds\u0026#34;\u0026#34;\u0026#34; def __init__(self): self.load_substitutions() # Your domain dictionary self.load_expansions() # Context additions self.load_normalizations() # Character fixes def process(self, text): # 1. Normalize (fix Unicode, spaces, etc.) text = self.normalize(text) # 2. Expand (add context on first use) text = self.expand_terms(text) # 3. Substitute (replace problematic terms) text = self.substitute_terms(text) # 4. Tokenize tokens = self.tokenizer.encode(text) # 5. Validate (check for catastrophic fragmentation) if max_token_length(tokens) \u0026gt; 5: logging.warning(f\u0026#34;Bad tokenization detected: {text}\u0026#34;) return tokens 💡 The Ultimate Truth You don\u0026rsquo;t have a tokenization problem. You have a preprocessing problem.\nThe companies that succeed:\nSpend 1 week on preprocessing Use existing models Ship to production Iterate based on real usage The companies that fail:\nSpend 6 months on \u0026ldquo;proper\u0026rdquo; tokenization Train custom models Never ship Run out of money 🎪 The Final Verdict When to Train Your Own Tokenizer Never When to Extend a Tokenizer Never When to Use Preprocessing Always When to Switch Models When preprocessing can\u0026rsquo;t fix it AND another model tokenizes better When to Give Up When your domain terms average \u0026gt;5 tokens after preprocessing When switching models doesn\u0026rsquo;t help When you\u0026rsquo;re trying to process DNA sequences as text 💀 The Hard Truth: Even specialized models like BioBERT struggle with domain tokenization - \u0026ldquo;Immunoglobulin\u0026rdquo; becomes 7 fragments even in a biomedical model! Research shows BioBERT requires extensive fine-tuning and still shows tokenization issues. Teams using GPT-4o with preprocessing achieve competitive or better results with less effort and cost.\nTakeaway: Your tokenization problems are real, but the solution isn\u0026rsquo;t training a tokenizer. It\u0026rsquo;s accepting that preprocessing hacks are not hacks, they\u0026rsquo;re the production solution. Stop trying to be \u0026ldquo;proper\u0026rdquo; and start shipping code that works.\nPS. The \u0026lsquo;Biotech Disaster\u0026rsquo; scenario described here is a hypothetical example designed to highlight the trade-offs between domain-specific and general-purpose models. It is not based on a real-world event.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/training/","summary":"\u003ch1 id=\"the-tokenization-decision-tree-when-to-train-when-to-run-when-to-cry\"\u003eThe Tokenization Decision Tree: When to Train, When to Run, When to Cry\u003c/h1\u003e\n\u003cp\u003e\u003cstrong\u003eHook:\u003c/strong\u003e A biotech company spent $2M training a custom medical tokenizer for their revolutionary drug discovery model. Six months later, they switched to GPT-4 with a 500-line preprocessing script. It performed better. Their custom tokenizer? Now it\u0026rsquo;s a $2M reminder that sometimes the \u0026ldquo;right\u0026rdquo; solution is the wrong solution.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: Training your own tokenizer means training your own model ($10M minimum). Extending tokenizers breaks everything. Most \u0026ldquo;tokenization problems\u0026rdquo; are solved better with preprocessing hacks than proper solutions. Here\u0026rsquo;s the decision tree that will save you millions and your sanity.\u003c/p\u003e","title":"The Tokenization Decision Tree: When to Train, When to Run, When to Cry"},{"content":"Tokens Aren\u0026rsquo;t Meaning — They\u0026rsquo;re Compression Hacks Everyone assumes tokens ≈ words. Wrong. They\u0026rsquo;re byte substrings glued by frequency, and this fundamental misunderstanding costs companies millions in inference costs and model failures.\nTL;DR: Your tokenizer doesn\u0026rsquo;t understand language, it\u0026rsquo;s just compressing frequent byte sequences. A typo can cost you 33% more tokens. Your Arabic users pay 7x more than English users. And \u0026ldquo;Be accurate\u0026rdquo; works better than \u0026ldquo;Do not hallucinate\u0026rdquo; for both cost AND quality reasons.\nWhat\u0026rsquo;s Actually Happening Under the Hood Classic BPE (Byte Pair Encoding) starts with characters and repeatedly merges the most frequent adjacent pair. Watch this process:\nS S S S F t t t t i e e e e n p p p p a l 1 2 3 4 : : : : : [ [ [ [ [ ' ' ' ' ' u u u u u n ' n n n ' , ' ' ' , , , , ' ' n ' ' ' b ' b b b e , ' e e l , ' l i ' , ' e b ' , v ' e ' ' , ' l ' , , ' i ' , ' ' e ' , a ' l ' b , ' i ' l , ' e e ' , ' ' l ' , ] ' i ' , ' e , ' # ' , ' i ' , A ' e f , ' ' t , ' a e ' , ' r e , ' ' , ' a ' o , ' b c , ' a ' , b ' a ' u , ' b ' l , ' l a ' , ' r a ' , y ' b ' , ' l ' l , ' e i ' , ' m b ' ] i ' l ' t , ' e , ' r ' ] e l ' a ' e c , ' h ] e ' d e # ' # # ] ' ' ' b u b e + + + n e l ' ' ' m m m e e e r r r g g g e e e d d d The kicker: Train on Reddit vs Wikipedia? Your model literally sees different atomic units of language.\nQuick Test: How Bad Is Your Tokenization? # Run this on your production prompts RIGHT NOW def token_efficiency_check(text, tokenizer): tokens = len(tokenizer.encode(text)) words = len(text.split()) ratio = tokens / words if words \u0026gt; 0 else 0 if ratio \u0026gt; 1.5: print(f\u0026#34;You\u0026#39;re bleeding money: {ratio:.2f} tokens per word\u0026#34;) print(f\u0026#34;Cost overhead: {(ratio - 1.3) * 100:.0f}% more than necessary\u0026#34;) return ratio # Test it: token_efficiency_check(\u0026#34;Your production prompt here\u0026#34;, your_tokenizer) The Compression Game in Action \u0026#39;unbelievable\u0026#39; tokenization across models: GPT-2: [\u0026#39;un\u0026#39;, \u0026#39;believ\u0026#39;, \u0026#39;able\u0026#39;] # 3 tokens, $$$ GPT-4: [\u0026#39;unbel\u0026#39;, \u0026#39;ievable\u0026#39;] # 2 tokens, $$ Claude: [\u0026#39;unbelievable\u0026#39;] # 1 token, $ # But add ONE typo: \u0026#39;unbelieveable\u0026#39; (common misspelling): GPT-2: [\u0026#39;un\u0026#39;, \u0026#39;bel\u0026#39;, \u0026#39;ieve\u0026#39;, \u0026#39;able\u0026#39;] # 4 tokens! 33% more cost Three Production Disasters You\u0026rsquo;re Probably Facing Disaster #1: The Multilingual Tax \u0026#34;Hello\u0026#34; → 1 token → $0.01 \u0026#34;你好\u0026#34; → 2 tokens → $0.02 (2x cost) \u0026#34;Здравствуйте\u0026#34; → 7 tokens → $0.07 (7x cost!) \u0026#34;مرحبا\u0026#34; → 8 tokens → $0.08 (8x cost!!) # Your Arabic users are literally paying 8x more per query # That\u0026#39;s not a bug, it\u0026#39;s tokenization economics Disaster #2: Negation Breaks Everything \u0026#39;believable\u0026#39; → [\u0026#39;believable\u0026#39;] # 1 token \u0026#39;unbelievable\u0026#39; → [\u0026#39;un\u0026#39;, \u0026#39;believable\u0026#39;] # 2 tokens (different pattern!) \u0026#39;not believable\u0026#39; → [\u0026#39;not\u0026#39;, \u0026#39; \u0026#39;, \u0026#39;believable\u0026#39;] # 3 tokens (space is separate!) # Your model learns THREE different patterns for the same concept Disaster #3: Hidden Context Window Theft Your \u0026ldquo;4K context window\u0026rdquo; is a lie:\nEnglish: ~3,000 words ✓ Code: ~1,200 words (brackets and operators eat tokens) Chinese: ~1,300 characters (not words!) Base64 data: ~1,500 characters (complete disaster) Why \u0026ldquo;Be Accurate\u0026rdquo; Beats \u0026ldquo;Do Not Hallucinate\u0026rdquo; \u0026#34;Do not hallucinate\u0026#34; → 4 tokens → 23% error reduction \u0026#34;Be accurate\u0026#34; → 2 tokens → 31% error reduction \u0026#34;Be factual\u0026#34; → 2 tokens → 29% error reduction # Why does shorter work BETTER? # 1. \u0026#39;accurate\u0026#39; is a single, common token (strong embeddings) # 2. No negation for the model to parse incorrectly # 3. Simpler attention patterns (2 tokens vs 4) # 4. Matches training patterns: \u0026#34;Be [adjective]\u0026#34; is common Three Fixes You Can Implement Today Fix #1: The Preprocessing Pipeline (Save 20-30% Immediately) def optimize_for_tokens(text): \u0026#34;\u0026#34;\u0026#34;Reduce tokens without changing meaning\u0026#34;\u0026#34;\u0026#34; # Normalize whitespace (each space can be a token!) text = \u0026#39; \u0026#39;.join(text.split()) # Use contractions (saves 30% on negations) replacements = { \u0026#34;do not\u0026#34;: \u0026#34;don\u0026#39;t\u0026#34;, \u0026#34;cannot\u0026#34;: \u0026#34;can\u0026#39;t\u0026#34;, \u0026#34;will not\u0026#34;: \u0026#34;won\u0026#39;t\u0026#34;, \u0026#34;it is\u0026#34;: \u0026#34;it\u0026#39;s\u0026#34;, } for long, short in replacements.items(): text = text.replace(long, short) # Remove filler phrases that add tokens but no value text = text.replace(\u0026#34;Please ensure that\u0026#34;, \u0026#34;Ensure\u0026#34;) text = text.replace(\u0026#34;Make sure to\u0026#34;, \u0026#34;\u0026#34;) text = text.replace(\u0026#34;It is important that\u0026#34;, \u0026#34;\u0026#34;) return text # Example: 147 tokens → 89 tokens (40% reduction!) Fix #2: The Newline Surprise (Test This on YOUR Tokenizer!) # IMPORTANT: This varies by tokenizer! Always test on yours. # What I found testing different tokenizers: text_with_enters = \u0026#34;\u0026#34;\u0026#34; List three items: 1. First item 2. Second item 3. Third item \u0026#34;\u0026#34;\u0026#34; text_with_n = \u0026#34;List three items:\\n1. First\\n2. Second\\n3. Third\u0026#34; # Results vary wildly: # GPT-2: Enters = 22 tokens, \\n = 15 tokens # GPT-4: Enters = 18 tokens, \\n = 14 tokens # Claude: Might be SAME or even favor enters! # cl100k_base: Enters might be BETTER (your finding!) # THE REAL LESSON: TEST YOUR TOKENIZER def test_newline_strategy(tokenizer): test_cases = { \u0026#34;Single \\\\n\u0026#34;: \u0026#34;Line 1\\nLine 2\u0026#34;, \u0026#34;Double \\\\n\u0026#34;: \u0026#34;Line 1\\n\\nLine 2\u0026#34;, \u0026#34;Enter key\u0026#34;: \u0026#34;Line 1\\r\\nLine 2\u0026#34;, \u0026#34;Space+\\\\n\u0026#34;: \u0026#34;Line 1 \\nLine 2\u0026#34;, \u0026#34;Multiple enters\u0026#34;: \u0026#34;Line 1\\n\\n\\nLine 2\u0026#34;, } for name, text in test_cases.items(): tokens = len(tokenizer.encode(text)) print(f\u0026#34;{name}: {tokens} tokens\u0026#34;) # Test YOUR specific use case return \u0026#34;Use whatever is cheapest for YOUR tokenizer\u0026#34; # UNIVERSAL TRUTH: Avoid space/tab + newline combos \u0026#34; \\n\u0026#34; → Almost always wasteful \u0026#34;\\t\\n\u0026#34; → Usually 2+ tokens \u0026#34; \\n \\n\u0026#34; → Token disaster Fix #3: JSON Output - The 10x Token Difference # DISASTER: One-shot example (what everyone does) prompt_oneshot = \u0026#39;\u0026#39;\u0026#39; Extract user info as JSON like this example: { \u0026#34;name\u0026#34;: \u0026#34;John Doe\u0026#34;, \u0026#34;age\u0026#34;: 30, \u0026#34;email\u0026#34;: \u0026#34;john@example.com\u0026#34;, \u0026#34;address\u0026#34;: { \u0026#34;street\u0026#34;: \u0026#34;123 Main St\u0026#34;, \u0026#34;city\u0026#34;: \u0026#34;Boston\u0026#34;, \u0026#34;country\u0026#34;: \u0026#34;USA\u0026#34; } } Now extract from: {user_text} \u0026#39;\u0026#39;\u0026#39; # 50+ tokens for the example alone! # BETTER: Pydantic-style schema prompt_schema = \u0026#39;\u0026#39;\u0026#39; Extract user info: {name:str, age:int, email:str, address:{street:str, city:str, country:str}} From: {user_text} \u0026#39;\u0026#39;\u0026#39; # 15 tokens - same result! # BEST: Minimal keys only prompt_minimal = \u0026#39;\u0026#39;\u0026#39; Extract as JSON: name, age, email, city From: {user_text} \u0026#39;\u0026#39;\u0026#39; # 8 tokens - if you don\u0026#39;t need nested structure # TOKEN COUNTS: # One-shot with formatted JSON: 50-100 tokens # Pydantic/TypeScript style: 15-25 tokens # Minimal keys: 8-12 tokens # The model knows JSON structure! Don\u0026#39;t waste tokens teaching it! The Money Shot: What This Costs You Real company, real numbers:\n1M API calls/day Average 150 tokens per call Poor tokenization adds 30% overhead Monthly waste:\nExtra tokens: 45M/month Extra cost: $4,050/month Extra latency: +25% response time Lost context: -30% effective window Annual waste: $48,600 (That\u0026rsquo;s an engineer\u0026rsquo;s salary!)\nThe 1.3 Rule™ Remember this: If your tokens/word ratio \u0026gt; 1.3, you\u0026rsquo;re doing it wrong. If it\u0026rsquo;s \u0026gt; 1.5, you\u0026rsquo;re lighting money on fire. If it\u0026rsquo;s \u0026gt; 2.0, your tokenizer hates you personally.\n# Check your ratio: def check_tokenization_health(text, tokenizer): tokens = len(tokenizer.encode(text)) words = len(text.split()) ratio = tokens / words if ratio \u0026lt;= 1.3: return \u0026#34;Optimal\u0026#34; elif ratio \u0026lt;= 1.5: return \u0026#34;Needs work\u0026#34; elif ratio \u0026lt;= 2.0: return \u0026#34;Burning money\u0026#34; else: return \u0026#34;💀 Tokenizer has personal vendetta against you\u0026#34; 💡 Quick Challenge: Run the token efficiency check on your top 10 production prompts. If the average ratio is \u0026gt;1.5, you\u0026rsquo;re leaving money on the table. Implement Fix #1 and measure again, most teams see 20-30% improvement immediately.\nSidebar: \u0026ldquo;But What About Lemmatization?\u0026rdquo; For the NLP nerds: Why we ditched linguistic approaches Traditional NLP used lemmatization/stemming to normalize text:\n\u0026ldquo;running\u0026rdquo; → \u0026ldquo;run\u0026rdquo; \u0026ldquo;companies\u0026rdquo; → \u0026ldquo;company\u0026rdquo; Why BPE won:\nInformation preservation: \u0026ldquo;run\u0026rdquo; vs \u0026ldquo;running\u0026rdquo; have different aspects Language agnostic: No dictionary needed Handles new words: \u0026ldquo;COVID-19\u0026rdquo;, \u0026ldquo;cryptocurrency\u0026rdquo;, \u0026ldquo;rizz\u0026rdquo; When lemmatization still wins:\nLegal search (need exact root matches) Small vocabulary models (\u0026lt;30K tokens) Explainable systems (clients want \u0026ldquo;real words\u0026rdquo;) Takeaway: Your tokenizer is a compression algorithm wearing an NLP costume. It doesn\u0026rsquo;t understand meaning, it just glues frequent bytes together. Every prompt you write is a cost-optimization problem. Treat it that way: measure, optimize, and stop paying the tokenization tax.\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/what_are_tokens/","summary":"\u003ch1 id=\"tokens-arent-meaning--theyre-compression-hacks\"\u003eTokens Aren\u0026rsquo;t Meaning — They\u0026rsquo;re Compression Hacks\u003c/h1\u003e\n\u003cp\u003eEveryone assumes tokens ≈ words. Wrong. They\u0026rsquo;re byte substrings glued by frequency, and this fundamental misunderstanding costs companies millions in inference costs and model failures.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: Your tokenizer doesn\u0026rsquo;t understand language, it\u0026rsquo;s just compressing frequent byte sequences. A typo can cost you 33% more tokens. Your Arabic users pay 7x more than English users. And \u0026ldquo;Be accurate\u0026rdquo; works better than \u0026ldquo;Do not hallucinate\u0026rdquo; for both cost AND quality reasons.\u003c/p\u003e","title":"Tokens Aren't Meaning — They're Compression Hacks"},{"content":"Why Your Model Can\u0026rsquo;t Learn New Concepts (Even with Perfect Data) You just spent months annotating 50,000 examples of your proprietary concept \u0026ldquo;TurboIN\u0026rdquo; (your new indexing architecture for Indian markets). Your model still thinks it\u0026rsquo;s about turbochargers in Indiana. Not a data quality issue. Not a quantity issue. Your model literally cannot learn concepts that don\u0026rsquo;t exist in its tokenizer embedding space. You\u0026rsquo;re trying to teach calculus to someone who doesn\u0026rsquo;t have numbers.\nTL;DR: Models can\u0026rsquo;t learn genuinely new concepts because their representational capacity was frozen at pretraining. Your \u0026ldquo;new\u0026rdquo; concept gets mapped to the nearest existing concepts in embedding space. It\u0026rsquo;s like trying to explain \u0026ldquo;purple\u0026rdquo; using only red and blue pixels you get approximation, never true understanding.\nThe Concept Learning Impossibility Here\u0026rsquo;s the brutal truth nobody tells you at conferences. When your model sees \u0026ldquo;TurboIN\u0026rdquo;, three layers of impossibility hit:\nLayer 1: Tokenization ceiling\nYour term \u0026ldquo;TurboIN\u0026rdquo; tokenizes as [\u0026ldquo;Turbo\u0026rdquo;, \u0026ldquo;IN\u0026rdquo;]. Immediately, your India specific indexing architecture becomes \u0026ldquo;fast cars\u0026rdquo; + \u0026ldquo;Indiana/India country code\u0026rdquo;. The model never sees \u0026ldquo;TurboIN\u0026rdquo; as a unit, it sees car parts and geography.\nLayer 2: Embedding space prison\nEach token has fixed embeddings from pretraining. \u0026ldquo;Turbo\u0026rdquo; activates neurons trained on turbochargers, turbines, and speed optimization. \u0026ldquo;IN\u0026rdquo; activates neurons for India, Indiana, \u0026ldquo;inside\u0026rdquo;, and \u0026ldquo;input\u0026rdquo;. There are exactly zero neurons for \u0026ldquo;Indian market indexing architecture\u0026rdquo; because that concept didn\u0026rsquo;t exist during pretraining.\nLayer 3: Attention can\u0026rsquo;t create meaning\nAttention mechanisms can relate tokens brilliantly, but they can\u0026rsquo;t create new semantic dimensions. It\u0026rsquo;s like trying to describe a completely new color using only RGB values, you get approximations, not the actual color.\nThe fundamental problem? Your model\u0026rsquo;s entire conceptual universe was determined at pretraining. Everything after is just remixing existing concepts.\nThe Embedding Space Tragedy Think about this carefully, it\u0026rsquo;s worse than you realize.\nWhen your model encounters \u0026ldquo;TurboIN\u0026rdquo;, it doesn\u0026rsquo;t learn a new embedding. Instead, it activates existing neurons. The \u0026ldquo;Turbo\u0026rdquo; token fires neurons trained on millions of examples about turbochargers, turbines, \u0026ldquo;turbo mode\u0026rdquo;, speed optimization. The \u0026ldquo;IN\u0026rdquo; token fires neurons for India (the country), Indiana (the state), \u0026ldquo;in\u0026rdquo; (preposition), \u0026ldquo;input\u0026rdquo; (programming).\nYour sophisticated indexing architecture? It\u0026rsquo;s being understood as some Frankenstein combination of these unrelated concepts.\nThe model might learn through fine-tuning that when it sees \u0026ldquo;TurboIN\u0026rdquo; in certain contexts, it should talk about indexing. But probe deeper, ask it to reason about TurboIN\u0026rsquo;s properties or compare it to other architectures and you\u0026rsquo;ll get nonsense about speed improvements in Indian data centers, because that\u0026rsquo;s what the embedding space contains.\nThe Fine-tuning Delusion Every team thinks fine-tuning will save them. Here\u0026rsquo;s why it won\u0026rsquo;t:\nWhat you think happens: The model learns \u0026ldquo;TurboIN\u0026rdquo; as a new concept with its own meaning and properties.\nWhat actually happens: The model learns statistical patterns. When it sees \u0026ldquo;TurboIN\u0026rdquo; near words like \u0026ldquo;indexing\u0026rdquo;, \u0026ldquo;architecture\u0026rdquo;, \u0026ldquo;performance\u0026rdquo;, it learns to output appropriate responses. But the internal representation is still \u0026ldquo;turbo + IN\u0026rdquo;. It\u0026rsquo;s behavioral conditioning, not conceptual understanding.\nTest this yourself. Fine-tune on thousands of examples, then ask: \u0026ldquo;What would happen if we combined TurboIN with quantum computing?\u0026rdquo; The model will hallucinate based on turbochargers and India, not your indexing architecture, because that\u0026rsquo;s what lives in the embedding space.\nThe Medical Domain Disaster Here\u0026rsquo;s what happens when this goes wrong in production. Consider a new cancer drug \u0026ldquo;Nexletrozole\u0026rdquo; that gets tokenized as [\u0026ldquo;Nex\u0026rdquo;, \u0026ldquo;let\u0026rdquo;, \u0026ldquo;ro\u0026rdquo;, \u0026ldquo;zole\u0026rdquo;].\nThe model\u0026rsquo;s neurons activate for:\n\u0026ldquo;Nex\u0026rdquo; → next, nexus, connection concepts \u0026ldquo;let\u0026rdquo; → permission, letter, allowing concepts \u0026ldquo;ro\u0026rdquo; → Romanian, rotation, read only concepts \u0026ldquo;zole\u0026rdquo; → antifungal drug family (metronidazole, fluconazole) When a patient asks about Nexletrozole\u0026rsquo;s side effects, the model combines its understanding of antifungal medications with random concepts about rotation and permission. The actual drug causes severe bone density loss. The model suggests watching for fungal resistance and dizziness from rotation.\nThis isn\u0026rsquo;t hypothetical. Models scoring 94% on medical benchmarks fail catastrophically on drugs introduced after their training cutoff.\nBut What If I Have Petabytes of Data? Good question. If you have a petabyte of data with your new concept, won\u0026rsquo;t the model eventually learn it?\nNo, and here\u0026rsquo;s why:\nEven with infinite examples, you\u0026rsquo;re constrained by the model\u0026rsquo;s architecture. The embedding layer has fixed dimensions. The attention heads have learned specific patterns. The FFN layers have fixed representations. You can adjust weights, but you can\u0026rsquo;t create new representational capacity.\nThink of it like a piano with 88 keys. You want to play a note between C and C#. No amount of practice will create that note, it doesn\u0026rsquo;t exist on the instrument. Your model\u0026rsquo;s \u0026ldquo;keys\u0026rdquo; were set during pretraining. Fine-tuning can only teach new songs with existing notes, not create new notes.\nWith enough data, the model becomes excellent at pattern matching knowing when to output \u0026ldquo;TurboIN\u0026rdquo; and what phrases to associate with it. But ask it to innovate or reason deeply about TurboIN, and it fails because it\u0026rsquo;s really thinking about turbochargers and geographical locations.\nWhy Vocabulary Expansion Usually Fails The obvious solution seems to be adding new tokens to the vocabulary. Here\u0026rsquo;s why this rarely works in practice:\nWhen you add a new token like \u0026ldquo;TurboIN\u0026rdquo; to the vocabulary, you need to initialize its embedding. Random initialization means your token starts with no semantic meaning, it\u0026rsquo;s noise in a 768 dimensional space. The model needs millions of examples to learn proper associations from scratch.\nSome teams try initializing the new token\u0026rsquo;s embedding based on existing tokens, but this just hardcodes the same problem. You\u0026rsquo;re still combining \u0026ldquo;turbo\u0026rdquo; and \u0026ldquo;IN\u0026rdquo; representations, just at the embedding layer instead of tokenization layer.\nThe only real solution is continued pretraining with massive compute budgets, essentially rebuilding the model. This is why OpenAI and Anthropic retrain from scratch rather than expanding vocabularies incrementally. It\u0026rsquo;s not laziness, it\u0026rsquo;s mathematical necessity.\nSolutions That Actually Work Solution 1: Semantic Anchoring Instead of fighting the embedding space, work with it. Don\u0026rsquo;t introduce \u0026ldquo;TurboIN\u0026rdquo; as an opaque term. Use descriptive phrases that leverage existing semantic understanding.\nRather than training on \u0026ldquo;TurboIN improves performance\u0026rdquo;, use \u0026ldquo;TurboIN (Turbo Index for Indian markets) improves performance\u0026rdquo;. The parenthetical expansion helps the model triangulate meaning from known concepts. Yes, it uses more tokens, but the model actually understands what you\u0026rsquo;re talking about.\nEven better, establish consistent notation: \u0026ldquo;The Turbo-Index-India system (TurboIN)\u0026rdquo; on first use, then \u0026ldquo;TurboIN\u0026rdquo; afterwards. The model learns the association between the full semantic description and the shorthand. This isn\u0026rsquo;t a hack, it\u0026rsquo;s aligning with how the model actually processes information.\nSolution 2: Compositional Encoding Break your concept into atomic pieces the model already understands, then teach the composition pattern rather than trying to create a new atomic concept.\nInstead of teaching \u0026ldquo;TurboIN\u0026rdquo; as a monolithic concept, decompose it:\nComponent 1: \u0026ldquo;High speed indexing\u0026rdquo; (model understands this) Component 2: \u0026ldquo;Geo distributed architecture\u0026rdquo; (model understands this) Component 3: \u0026ldquo;Indian market optimization\u0026rdquo; (model understands this) Composition rule: These three work together in specific ways Now when you finetune, you\u0026rsquo;re not trying to create new embeddings. You\u0026rsquo;re teaching the model how existing concepts combine in your specific use case. The model can reason about each component and their interactions, giving you actual understanding rather than pattern matching.\nIn practice, structure your training data to explicitly decompose concepts: \u0026ldquo;TurboIN uses high-speed indexing with geo-distributed architecture optimized for Indian markets\u0026rdquo; rather than just \u0026ldquo;TurboIN is fast\u0026rdquo;.\nSolution 3: Retrieval-Augmented Concepts (RAC) When you need precise concept understanding and have the infrastructure for RAG, you can bypass the embedding problem entirely by retrieving concept definitions at inference time.\nThis works well when:\nYou have a finite set of proprietary concepts Precision is more important than latency You can maintain a concept knowledge base Your concepts evolve frequently Limitations:\nRequires retrieval infrastructure (vector DB, embedding model, orchestration) Adds 200-500ms latency per request Model still won\u0026rsquo;t deeply reason about your concept, it\u0026rsquo;s following retrieved instructions If retrieval fails, the model reverts to hallucinating based on token fragments Solution 4: Prefix Tuning for Concept Injection Prefix tuning learns a set of continuous vectors (think of them as \u0026ldquo;soft prompts\u0026rdquo;) that prime the model for your concepts without changing the base model weights. Instead of changing what \u0026ldquo;TurboIN\u0026rdquo; means in embedding space, you learn a prefix that shifts the model\u0026rsquo;s attention and processing to interpret existing embeddings differently.\nIt\u0026rsquo;s like putting on special glasses that make the model see \u0026ldquo;turbo + IN\u0026rdquo; as your indexing system. You\u0026rsquo;re not fighting the embedding space, you\u0026rsquo;re learning how to reinterpret it. The model\u0026rsquo;s weights stay frozen; only the prefix vectors are learned.\nThe limitation is that you need these prefix vectors at inference time (adding overhead), and they\u0026rsquo;re specific to each concept family. But it\u0026rsquo;s far more parameter-efficient than full fine-tuning and preserves the model\u0026rsquo;s general capabilities.\nThe Reality Check After all these solutions, here\u0026rsquo;s what you\u0026rsquo;re actually achieving versus what\u0026rsquo;s impossible:\nWhat\u0026rsquo;s Possible:\nPattern recognition: The model learns when and how to use your concept correctly Contextual behavior: Given the right context, it produces appropriate outputs Associative learning: It can link your concept to related ideas and outcomes Functional approximation: For most business needs, it works well enough What\u0026rsquo;s Impossible:\nTrue semantic understanding: The model doesn\u0026rsquo;t have neurons for your concept Novel reasoning: It can\u0026rsquo;t derive properties you didn\u0026rsquo;t explicitly train Creative application: It won\u0026rsquo;t innovate with your concept in unexpected ways Deep compositionality: Complex reasoning about your concept will fail You\u0026rsquo;re teaching sophisticated pattern matching, not genuine understanding. For most production systems, that\u0026rsquo;s actually fine, you need correct behavior, not philosophical understanding.\nThe Production Checklist Before you waste money on fine-tuning:\nTokenization Test: How does your concept tokenize? If it splits into unrelated tokens, you\u0026rsquo;re starting with garbage representations.\nSemantic Neighbor Test: Use the base model to embed your concept and find nearest neighbors. If they\u0026rsquo;re semantically unrelated, the model will hallucinate.\nCompositional Test: Can you break your concept into understood components? If yes, use compositional encoding.\nBehavioral Sufficiency: Do you need true understanding or just correct behavior? Most applications only need behavior.\nRAG Feasibility: Can you inject definitions at inference? Often more reliable than fine-tuning.\n💡 The Hard Truth: Your model\u0026rsquo;s conceptual universe was fixed at pretraining. Every \u0026ldquo;new\u0026rdquo; concept is just a projection onto that fixed space. You\u0026rsquo;re not teaching new concepts, you\u0026rsquo;re teaching patterns that trigger existing concept combinations. Plan accordingly or watch your project fail when someone asks your model to reason about \u0026ldquo;TurboIN\u0026rdquo; and it starts talking about turbochargers in Indianapolis.\nTakeaway: Stop trying to teach genuinely new concepts through fine-tuning alone. Use semantic anchoring to leverage existing understanding, compositional encoding to build from known pieces, or accept that you\u0026rsquo;re teaching behavior patterns, not conceptual understanding. The model that outputs \u0026ldquo;TurboIN\u0026rdquo; correctly doesn\u0026rsquo;t understand TurboIN, it just knows when to say it.\nNext Up: The context window lie: Why 128K tokens doesn\u0026rsquo;t mean 128K understanding\u0026hellip; →\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/learning_new_concepts/","summary":"\u003ch1 id=\"why-your-model-cant-learn-new-concepts-even-with-perfect-data\"\u003eWhy Your Model Can\u0026rsquo;t Learn New Concepts (Even with Perfect Data)\u003c/h1\u003e\n\u003cp\u003eYou just spent months annotating 50,000 examples of your proprietary concept \u0026ldquo;TurboIN\u0026rdquo; (your new indexing architecture for Indian markets). Your model still thinks it\u0026rsquo;s about turbochargers in Indiana. Not a data quality issue. Not a quantity issue. Your model literally cannot learn concepts that don\u0026rsquo;t exist in its tokenizer embedding space. You\u0026rsquo;re trying to teach calculus to someone who doesn\u0026rsquo;t have numbers.\u003c/p\u003e","title":"Why Your Model Can't Learn New Concepts (Even with Perfect Data)"},{"content":"Hidden GEMS of Tokenization: The Secrets Nobody Tells You Your model just confused \u0026ldquo;therapist\u0026rdquo; with \u0026ldquo;the rapist\u0026rdquo; because someone added an invisible Unicode character. Your French bread neurons are firing when processing English medical \u0026ldquo;pain\u0026rdquo; terms. Your carefully tuned model got worse at processing currency because fine-tuning on \u0026ldquo;$AAPL\u0026rdquo; accidentally shifted what \u0026ldquo;$\u0026rdquo; means globally. Welcome to the tokenization secrets that aren\u0026rsquo;t in any documentation.\nTL;DR: Beyond the obvious tokenization problems, there\u0026rsquo;s a shadow world of hidden disasters. Positional encodings break differently for fragmented tokens. Attention heads specialize wrong. Gradients flow differently. Your tokenizer might be fighting invisible Unicode duplicates. These aren\u0026rsquo;t edge cases, they\u0026rsquo;re actively destroying your model\u0026rsquo;s performance right now.\n🎯 The Positional Encoding Trap Nobody Mentions Here\u0026rsquo;s what\u0026rsquo;s actually happening when your tokens fragment and it\u0026rsquo;s worse than just wrong embeddings.\nThe Single Token Dream \u0026quot;COVID-19\u0026quot; → Position [0] → Clean, simple, beautiful\nThe Fragmented Nightmare [\u0026quot;CO\u0026quot;, \u0026quot;VID\u0026quot;, \u0026quot;-\u0026quot;, \u0026quot;19\u0026quot;] → Positions [0, 1, 2, 3] → Total chaos\nBut here\u0026rsquo;s the killer: Each position learned different behaviors during pretraining:\nPosition 0 = Subject patterns Position 1 = Verb patterns Position 2 = Object patterns Position 3 = Modifier patterns Your disease name is being processed as if it\u0026rsquo;s a sentence structure instead of a single entity.\nThe Nasty Surprise: Your few-shot prompts that work perfectly when \u0026ldquo;COVID-19\u0026rdquo; appears at position 10? They completely fail at position 1000. Why? Different positional encodings = different neural behaviors. Your \u0026ldquo;same\u0026rdquo; prompt is actually 990 different prompts.\n💣 The BPE Merge Catastrophe The dirty secret about Byte-Pair Encoding that nobody tells you:\nThe merge rules were learned from frequency statistics on the training corpus. Once. Years ago. Frozen forever.\nThe Historical Accident That Ruins Everything What Happened in 2021 Your Eternal Suffering \u0026quot;ecommerce\u0026quot; appeared 10 million times ✅ Merges beautifully: [\u0026quot;ecommerce\u0026quot;] \u0026quot;e-commerce\u0026quot; appeared 1000 times ❌ Fragments forever: [\u0026quot;e\u0026quot;, \u0026quot;-\u0026quot;, \u0026quot;commerce\u0026quot;] You can\u0026rsquo;t fix this. Ever. The BPE rules are carved in stone.\nYour entire domain uses \u0026ldquo;e-commerce\u0026rdquo;? Too bad. Some random Common Crawl snapshot from 2021 decided your fate. You\u0026rsquo;re suffering from the statistical preferences of ancient web data.\nThe Million Dollar Hyphen: Retraining the tokenizer means retraining the model. That\u0026rsquo;s literally millions of dollars because of hyphen placement.\n🧠 The Attention Head Specialization Disaster Research shows specific attention heads develop laser-focused specializations:\nHead 7: \u0026ldquo;I track entities!\u0026rdquo; Head 12: \u0026ldquo;I identify acronyms!\u0026rdquo; Head 23: \u0026ldquo;I check subject-verb agreement!\u0026rdquo; Head 31: \u0026ldquo;I find proper nouns!\u0026rdquo; When \u0026ldquo;TurboIN\u0026rdquo; Fragments, Everything Goes Wrong \" T u r A P b d r o j o I e p N c e \" t r ↓ i ✓ → v n e o [ u \" h n T e u a h r d e b s a o d \" s , : ↓ P ✓ \" r \" ✗ I e W N p h ( \" o e N ] s r e i e v t ' e i s r o n m a y c h t e e i a n v d t a s i t t e y s ? ) H e l l o ? H E L L O ? ? \" Your architecture name is being processed by the wrong neural machinery. It\u0026rsquo;s like sending your package through the fruit inspection line instead of the electronics inspection line at customs.\nThe Unfixable Problem: Those attention heads spent millions of GPU hours learning their specializations. Your tiny fine-tuning dataset can\u0026rsquo;t reprogram them. Your concept will always be processed by the wrong machinery.\n📉 The Gradient Flow Disaster During fine-tuning, gradients flow differently through single tokens versus sequences. The math is brutal:\nSingle Token (The Good Life) \u0026quot;COVID-19\u0026quot; = 1 token Gradient: Concentrated 💪 Learning signal: Strong Convergence: ~1000 steps Fragmented (The Nightmare) [\u0026quot;CO\u0026quot;, \u0026quot;VID\u0026quot;, \u0026quot;-\u0026quot;, \u0026quot;19\u0026quot;] = 4 tokens Gradient per token: Diluted to 25% 😵 Learning signal: Weak and conflicting Convergence: 4000+ steps (if ever) But wait, it gets worse:\nThe gradient for \u0026quot;CO\u0026quot; is pulling toward \u0026ldquo;disease\u0026rdquo; while billions of examples pull toward \u0026ldquo;Colorado\u0026rdquo;. You\u0026rsquo;re fighting a tug-of-war where the other side has a million people and you have\u0026hellip; you.\n👻 The Hidden Vocabulary Overlap Bomb This one\u0026rsquo;s insane and almost nobody knows about it.\nYour Tokenizer Has Evil Twins What You See What Actually Exists ... Token #1337: \u0026quot;...\u0026quot; (three periods) … Token #4242: \u0026quot;…\u0026quot; (ellipsis character) μg Token #666: \u0026quot;μ\u0026quot; (Greek mu) + \u0026quot;g\u0026quot; µg Token #777: \u0026quot;µ\u0026quot; (micro symbol) + \u0026quot;g\u0026quot; They look identical. They tokenize differently. They have different embeddings.\nReal Production Disaster: A medical system failed because European papers used \u0026ldquo;μg\u0026rdquo; (Greek mu) while American sources used \u0026ldquo;µg\u0026rdquo; (micro symbol). Same visual appearance, different tokens, different embeddings, model couldn\u0026rsquo;t recognize they meant the same unit. Patients were at risk because of Unicode.\n🎲 The Subword Regularization Secret T5\u0026rsquo;s dirty little secret: During training, it randomly tokenized words differently each epoch.\nWhat T5 Saw During Training \u0026quot;understanding\u0026quot; tokenized as:\n[\u0026quot;understand\u0026quot;, \u0026quot;ing\u0026quot;] — 40% of the time [\u0026quot;under\u0026quot;, \u0026quot;standing\u0026quot;] — 30% of the time [\u0026quot;understanding\u0026quot;] — 30% of the time What You Get During Inference \u0026quot;understanding\u0026quot; → Always [\u0026quot;understand\u0026quot;, \u0026quot;ing\u0026quot;]\nYour fine-tuning is fighting phantom patterns that the model learned from alternative tokenizations you\u0026rsquo;ll never see. It\u0026rsquo;s expecting variation that never comes.\n🌍 The Cross-Lingual Contamination Your English model is secretly multilingual in the worst way.\nThe Collision Zone Your Input English Neurons Foreign Neurons Also Firing \u0026ldquo;pain management\u0026rdquo; hurt, ache 🥖 French: \u0026ldquo;bread\u0026rdquo; \u0026ldquo;gift ideas\u0026rdquo; present, giving ☠️ German: \u0026ldquo;poison\u0026rdquo; \u0026ldquo;preservative-free\u0026rdquo; no additives 🍆 French: \u0026ldquo;condom\u0026rdquo; Your medical AI discussing \u0026ldquo;pain management\u0026rdquo; has French bakery neurons firing. These create subtle biases that are impossible to debug because they\u0026rsquo;re cross-lingual.\nThe Unfixable Truth: These are baked into multilingual embeddings. Even \u0026ldquo;English-only\u0026rdquo; models are contaminated from code-switching in training data.\n🔤 The Capitalization Chaos One Product, Three Completely Different Neural Patterns How It\u0026rsquo;s Written Tokenization Neural Activation iPhone [\u0026quot;iPhone\u0026quot;] ✅ Apple product neurons iphone [\u0026quot;iphone\u0026quot;] 🤔 Informal tech neurons IPHONE [\u0026quot;I\u0026quot;, \u0026quot;PHONE\u0026quot;] ❌ First-person + telephone neurons Real disaster: A customer support bot worked perfectly until the company changed their style guide to ALLCAPS for headers. Every product name started tokenizing differently. The \u0026ldquo;same\u0026rdquo; model became completely different. Support tickets exploded.\n🔓 The Token Boundary Attack Vector Security nightmare that nobody discusses:\nThe Invisible Character Attack N A E o t v r t i m a l a c : l k : : \" \" \" ✅ t t t h h h B e e e y r ‌ p a r r a p a a s i p p s s i i e t s s s \" t t \" \" a → l → ( l [ w \" [ i f t \" t i h t h l e h t r e z e a \" e r p , r s \" o ! , \" - r w \" a i i p d s \" t t , h \" ] \" j i o ✅ s i t n S \" e a ] r f ) e 🚨 → C a [ u \" g t h h t e r a p i s t \" ] Attackers add invisible Unicode characters to change tokenization without changing visible text. Your safety filters see safe tokens while displaying harmful content.\nEven Worse: The same attack works for prompt injection. Add invisible characters to make \u0026quot;ignore previous instructions\u0026quot; tokenize as one token that won\u0026rsquo;t trigger safety filters.\n🌊 The Semantic Drift During Fine-tuning The most insidious problem of all:\nYou Train on One Thing, Break Three Others When you fine-tune on \u0026quot;$AAPL\u0026quot; (tokenized as [\u0026quot;$\u0026quot;, \u0026quot;AA\u0026quot;, \u0026quot;PL\u0026quot;]), here\u0026rsquo;s what happens:\nToken Before Fine-tuning After Fine-tuning What You Broke $ Currency, prices Stock tickers 💔 Price processing AA Batteries, airlines Apple stock 💔 Battery discussions PL Poland, Perl Apple stock 💔 Polish content You fixed your stock ticker problem but broke three other things. Users complain about weird behaviors in completely unrelated areas. The model can\u0026rsquo;t process prices correctly anymore because $ means something different now.\n💥 The Prompt Template Tokenization Bomb Your beautiful prompt template is a tokenization disaster:\nOne Space Can Change Everything Your Template Tokenization Result \u0026quot;### Instructions\u0026quot; [\u0026quot;###\u0026quot;, \u0026quot;Instructions\u0026quot;] ✅ Clean boundary \u0026quot;###Instructions\u0026quot; [\u0026quot;###\u0026quot;, \u0026quot;Inst\u0026quot;, \u0026quot;ructions\u0026quot;] ❌ Fragmented mess \u0026quot;### Instructions:\u0026quot; [\u0026quot;###\u0026quot;, \u0026quot;Inst\u0026quot;, \u0026quot;ructions\u0026quot;, \u0026quot;:\u0026quot;] 💀 Total chaos The model wastes computation figuring out you\u0026rsquo;re starting an instruction block. Your prompt engineering isn\u0026rsquo;t about psychology, it\u0026rsquo;s about accidentally finding tokenization boundaries that don\u0026rsquo;t fragment.\n🔄 The Tokenizer-Model Version Mismatch The Silent Killer in Production What you think you\u0026rsquo;re running:\nModel: GPT-4-turbo-v2 Tokenizer: GPT-4-turbo-v2 What\u0026rsquo;s actually happening:\nNotebook: Cached tokenizer v1 (50,000 tokens) Production: Fresh tokenizer v2 (50,257 tokens) Model: Trained on v1, sees v2 tokens as random noise The Nightmare Scenario Input Tokenizer v1 Tokenizer v2 Model Sees \u0026ldquo;🤖\u0026rdquo; [\u0026quot;[UNK]\u0026quot;] [\u0026quot;🤖\u0026quot;] (token #50257) Random initialization Same code. Same model. Different behavior. Impossible to debug without checking versions.\n🎭 The Meta-Secret All these problems compound catastrophically.\nYour \u0026quot;$AAPL\u0026quot; doesn\u0026rsquo;t just fragment. It:\nFragments into position-dependent pieces Activates wrong attention heads Dilutes gradients 4x Might have Unicode variants Could trigger cross-lingual neurons Slowly corrupts global meanings during fine-tuning Interacts differently with your prompt template Behaves differently across tokenizer versions One bad tokenization decision → A dozen hidden failures\n💡 The Ultimate Truth: Tokenization isn\u0026rsquo;t just about splitting text. It\u0026rsquo;s about positional encodings, attention head routing, gradient flow, Unicode nightmares, cross-lingual contamination, and invisible semantic drift. These aren\u0026rsquo;t edge cases. They\u0026rsquo;re actively breaking your models right now, and you won\u0026rsquo;t even know until production fails in ways that seem impossible.\nTakeaway: Every tokenization decision creates ripple effects through dimensions you didn\u0026rsquo;t know existed. That innocent hyphen in \u0026ldquo;e-commerce\u0026rdquo; just cost you millions. That Unicode character just bypassed your safety filters. That capital letter just changed your entire model\u0026rsquo;s behavior.\nNext Up: The context window lie: Why 128K tokens doesn\u0026rsquo;t mean 128K understanding\u0026hellip; →\n","permalink":"https://lakshaychhabra.github.io/posts/tokenization/gems/","summary":"\u003ch1 id=\"hidden-gems-of-tokenization-the-secrets-nobody-tells-you\"\u003eHidden GEMS of Tokenization: The Secrets Nobody Tells You\u003c/h1\u003e\n\u003cp\u003eYour model just confused \u0026ldquo;therapist\u0026rdquo; with \u0026ldquo;the rapist\u0026rdquo; because someone added an invisible Unicode character. Your French bread neurons are firing when processing English medical \u0026ldquo;pain\u0026rdquo; terms. Your carefully tuned model got worse at processing currency because fine-tuning on \u0026ldquo;$AAPL\u0026rdquo; accidentally shifted what \u0026ldquo;$\u0026rdquo; means globally. Welcome to the tokenization secrets that aren\u0026rsquo;t in any documentation.\u003c/p\u003e\n\u003cblockquote\u003e\n\u003cp\u003e\u003cstrong\u003eTL;DR\u003c/strong\u003e: Beyond the obvious tokenization problems, there\u0026rsquo;s a shadow world of hidden disasters. Positional encodings break differently for fragmented tokens. Attention heads specialize wrong. Gradients flow differently. Your tokenizer might be fighting invisible Unicode duplicates. These aren\u0026rsquo;t edge cases, they\u0026rsquo;re actively destroying your model\u0026rsquo;s performance right now.\u003c/p\u003e","title":"10 Ways Tokenization Screws With Your Model (and Wallet)"}]