If you are building an AI product for users in India, the Middle East, Japan, or anywhere outside the English-speaking world, there is a cost structure working against you that almost nobody in the developer community talks about openly.
The same sentence — the same meaning, the same number of words — costs dramatically more to process when written in Hindi or Arabic than when written in English. Not 10% more. Not 50% more. In many cases, three to five times more. And this multiplier applies to every single API call your application makes, in both directions: what your users type and what your AI writes back.
This isn't a bug. It isn't a pricing policy decision that AI companies made arbitrarily. It's a structural consequence of how language models are trained and how their tokenizers work. Understanding it won't make it go away — but it will fundamentally change how you architect, estimate, and optimize AI applications for non-English markets.
The Tax Nobody Talks About
Most cost comparisons between AI models focus entirely on English text. Blog posts, YouTube videos, pricing calculators, developer forums — the examples are almost always in English. When a company publishes "100 tokens equals roughly 75 words," they mean 75 English words. For Hindi, that same 100 tokens might represent only 20–25 words. For Arabic, perhaps 18–22 words.
This discrepancy has a name in the AI community: token fertility. It refers to the number of tokens required to represent a word or concept in a given language. English has very low token fertility — most common words are a single token. Hindi, Arabic, and Japanese have high token fertility — individual words often require multiple tokens to represent.
The financial consequence is direct: a Hindi-language chatbot handling the same conversation volume as an English-language chatbot will consume 3–5x more tokens and therefore cost 3–5x more, assuming identical model and pricing tier. For a startup building a regional language AI product, this difference can be the gap between a viable business model and one that bleeds money at scale.
Why Tokenizers Are Biased Toward English
To understand the multilingual cost gap, you need to understand how AI tokenizers are built — specifically the dominant approach called Byte Pair Encoding.
How BPE Tokenization Works
BPE tokenization starts with individual characters and repeatedly merges the most frequently occurring pairs into single tokens. After millions of merges, common words and word fragments get their own token ID.
The key variable is the training corpus. The vocabulary a tokenizer builds — which character sequences get merged into single tokens — is entirely determined by which languages appear most frequently in the training data.
GPT's tiktoken tokenizer was trained on data that is overwhelmingly English. Anthropic's tokenizer similarly reflects the English-dominant nature of internet text. The vocabularies these tokenizers built have rich, efficient representations for English words and word fragments, and relatively sparse, inefficient representations for scripts and languages that appeared less frequently in training.
Why English Gets the Best Deal
Common English words like "the," "and," "have," "with," "from," "this," "that" are each a single token. Even medium-length words like "building," "system," "customer," "account" are typically one token. Longer compound words might be two tokens. A 10-word English sentence might use 8–11 tokens.
This efficiency exists because the tokenizer saw these words billions of times during training and assigned them dedicated token IDs. The merger process ran long enough that English text is represented at near-maximum compression.
What Happens to Other Scripts
When the tokenizer encounters Hindi Devanagari script, Arabic script, Japanese kana and kanji, or other non-Latin writing systems, it has far fewer merged token IDs available. What this means in practice: individual characters or small character combinations — not words — become the token unit.
A single Hindi word written in Devanagari might require 3, 4, or even 6 tokens to represent because the tokenizer never saw that character sequence frequently enough to merge it into a single token ID. A five-word Hindi sentence that conveys the same meaning as a five-word English sentence might use 18–25 tokens versus 6–8 tokens for English.
This is not inefficiency in the traditional sense — the tokenizer is working exactly as designed. It's that the design was optimized for English, and non-English languages pay the price.
Language-by-Language Breakdown: The Real Numbers
Here are real token count comparisons for the same semantic content across languages, tested across major AI tokenizers. The English baseline is set at 1.0x for each example.
Hindi (Devanagari Script)
Hindi written in Devanagari script is one of the most expensive languages to tokenize on English-optimized models. A typical Hindi sentence uses 3.5–5x more tokens than its English equivalent.
Example: "Please check the status of my order and let me know when it will be delivered."
English: 16 tokens.
Hindi (Devanagari): "कृपया मेरे ऑर्डर की स्थिति जांचें और मुझे बताएं कि यह कब डिलीवर होगा।" — 58 tokens.
Multiplier: 3.6x
For customer support applications targeting Hindi-speaking users in India, this means every conversation costs 3–4x more than the equivalent English conversation, purely from the script.
Arabic
Arabic presents a similar challenge. The Arabic script is written right-to-left with a completely different character set, and Arabic words are morphologically complex — a single word can contain a root plus multiple attached prefixes and suffixes that in English would be separate words.
Example: "I want to cancel my subscription and get a refund for this month."
English: 15 tokens.
Arabic: "أريد إلغاء اشتراكي واسترداد المبلغ المدفوع عن هذا الشهر." — 55 tokens.
Multiplier: 3.7x
Arabic also has the challenge of right-to-left rendering affecting how some tokenizers handle string boundaries, occasionally adding further overhead in mixed-language prompts.
Japanese
Japanese is particularly expensive because it uses three different writing systems simultaneously — hiragana, katakana, and kanji — often within the same sentence. Kanji characters are especially token-expensive because each character represents a morpheme rather than a phoneme, and there are thousands of them.
Example: "What are your store hours on weekends?"
English: 8 tokens.
Japanese: "週末の営業時間を教えていただけますか？" — 27 tokens.
Multiplier: 3.4x
Modern tokenizers like tiktoken have somewhat improved kanji efficiency, but Japanese remains 3–4x more expensive than English for typical conversational text.
Chinese (Mandarin)
Mandarin written in Chinese characters (Hanzi) is similarly expensive, though slightly less so than Japanese because Mandarin doesn't mix multiple scripts. Standard simplified Chinese uses around 3–4x more tokens than English for equivalent meaning.
Example: "Can I track my delivery in real time?"
English: 9 tokens.
Mandarin: "我可以实时追踪我的快递吗？" — 28 tokens.
Multiplier: 3.1x
Gemini has notably better Chinese tokenization than GPT-4 or Claude, a consequence of Google's larger Mandarin training corpus. More on this in the model comparison section.
Spanish and French
European languages using Latin script fare much better than non-Latin scripts, but they still cost more than English. Spanish and French use the same alphabet as English with a few additional characters (accents, ñ, ç), but their average word length is higher and they have more complex morphology.
Example: "Please reset my password and send me a confirmation email."
English: 11 tokens.
Spanish: "Por favor restablezca mi contraseña y envíeme un correo electrónico de confirmación." — 16 tokens.
French: "Veuillez réinitialiser mon mot de passe et m'envoyer un e-mail de confirmation." — 17 tokens.
Multiplier: 1.4–1.6x
For Latin-script European languages, the cost premium is modest — 40–60% more tokens than English, not 3–5x. Still worth accounting for in high-volume applications, but not the crisis it becomes with non-Latin scripts.
The Real-World Cost Calculation
Theory is one thing. Let's look at what this actually means for production AI applications.
Example: A Hindi Customer Support Bot
A startup in Ahmedabad builds a customer support chatbot for their e-commerce platform. Their customers speak Hindi. Here's the token math for a single average support conversation:
System prompt: 200 tokens (written in English — more on this strategy later).
Average user message in Hindi: 45 Hindi words — approximately 145 tokens.
Conversation history (4 turns): approximately 520 tokens.
AI response in Hindi: 60 Hindi words — approximately 195 tokens.
Total per conversation: approximately 1,060 tokens.
Now compare to the same conversation if it were in English:
Same system prompt: 200 tokens.
45 English words: approximately 55 tokens.
4-turn history: approximately 280 tokens.
60-word English response: approximately 80 tokens.
Total per English conversation: approximately 615 tokens.
The Hindi conversation costs 72% more per exchange — and that's using a moderately conservative multiplier. At 50,000 conversations per month on Claude Sonnet, the Hindi app costs roughly $545/month vs $315/month for the equivalent English app. That's $230 extra per month, $2,760 per year, from language alone.
Example: Arabic Content Moderation Pipeline
A media company in Dubai runs user-generated content through an AI moderation pipeline. Each piece of content averages 80 Arabic words. Their volume: 200,000 moderation calls per month.
80 Arabic words: approximately 400 tokens per call.
80 English words: approximately 100 tokens per call.
At 200,000 calls: Arabic pipeline uses 80 million tokens. English equivalent: 20 million tokens.
The Arabic pipeline costs 4x more at identical API rates. For a moderation use case where quality requirements mean they can't switch to a cheaper model, this is a structural cost that must be baked into the product economics from day one.
Example: A Multilingual App Serving 5 Languages
A SaaS company serves users in English, Spanish, Hindi, Arabic, and Japanese. Their average user sends 30 messages per month. They have 10,000 users evenly distributed across all five languages (2,000 per language).
Monthly token usage by language at 30 messages × 50 words per message:
English users: 2,000 users × 30 messages × ~60 tokens = 3.6M tokens.
Spanish users: 2,000 × 30 × ~90 tokens = 5.4M tokens.
Hindi users: 2,000 × 30 × ~210 tokens = 12.6M tokens.
Arabic users: 2,000 × 30 × ~220 tokens = 13.2M tokens.
Japanese users: 2,000 × 30 × ~200 tokens = 12.0M tokens.
Total: 46.8M tokens per month.
If all users were English: 18M tokens per month.
The multilingual user base costs 2.6x more to serve than an equivalent English-only base. This needs to be in your pricing model before you launch, not discovered after your first AWS bill.
Which Models Handle Multilingual Text Most Efficiently?
Not all models are equally bad at non-English tokenization. There are meaningful differences worth knowing.
Claude (Anthropic)
Claude's tokenizer is broadly similar to GPT-4's in its multilingual efficiency — both were trained on English-dominant corpora. Hindi, Arabic, and Japanese all tokenize at 3–5x English rates. Claude does not have a notable advantage for any major non-Latin script language. Its strong reasoning and instruction-following in non-English languages is well-regarded, but the token cost premium is real and unavoidable.
GPT-4o (OpenAI)
GPT-4o uses tiktoken with a vocabulary size of 100,000 tokens. OpenAI has made some improvements to multilingual tokenization compared to earlier GPT models, but the fundamental English bias remains. Hindi and Arabic still tokenize at 3–4x English rates. GPT-4o has somewhat better Chinese and Japanese efficiency than Claude, likely reflecting more East Asian data in OpenAI's training corpus.
Gemini (Google)
Gemini is the standout here. Google's multilingual training data is substantially more diverse than Anthropic or OpenAI, reflecting Google's Search and Translate infrastructure which processes hundreds of languages at enormous scale. Gemini's SentencePiece tokenizer handles Mandarin Chinese, Japanese, Korean, and several Indic languages noticeably more efficiently than tiktoken or Claude's tokenizer.
For Hindi specifically, Gemini tokenizes at roughly 2.5–3x English rates compared to 3.5–5x on Claude and GPT-4. That gap matters. For a Hindi-language application making millions of calls per month, choosing Gemini over Claude or GPT-4 can reduce token counts by 20–30% on non-English content alone, before any prompt optimization.
If you are building for Indian or East Asian language markets, Gemini deserves serious evaluation not just on price per token but on tokens per word — which is a different and often more important metric.
Mistral
Mistral's tokenizer is based on LLaMA's SentencePiece implementation, which was trained with more multilingual data than tiktoken. Mistral handles European languages efficiently and has reasonable performance on Arabic. For Indic scripts, it still tokenizes heavier than English but the gap is slightly smaller than GPT-4 in some benchmarks. Combined with Mistral's lower pricing, it can be a cost-effective choice for multilingual applications that don't require frontier reasoning capabilities.
5 Strategies to Cut Your Multilingual Token Bill
Knowing the problem is only useful if you can act on it. Here are five concrete strategies that reduce multilingual token costs without degrading quality.
Strategy 1: Keep System Prompts in English
This is the highest-leverage single change for most multilingual applications. Your system prompt is sent with every API call. If your system prompt is written in Hindi and it's 300 tokens long, it's probably costing you 900–1,200 Hindi tokens instead of 300 English tokens — every single call.
AI models understand English system prompts and respond correctly in other languages when instructed to do so. A system prompt that says "You are a customer support agent. Always respond in Hindi. Tone: friendly and professional." in English costs roughly 18 tokens. The same instruction written in Hindi would cost 60–70 tokens. Multiply by millions of calls.
The rule: write system prompts in English. Tell the model which language to respond in. The output quality is identical and the token savings on system prompt alone can be 30–50% for high-volume non-English apps.
Strategy 2: Understand Transliteration Costs
Hinglish — Hindi written in Roman/Latin script rather than Devanagari — tokenizes dramatically more efficiently than Devanagari. "Mujhe order cancel karna hai" (Latin script) uses approximately 9 tokens. "मुझे ऑर्डर कैंसिल करना है" (Devanagari) uses approximately 28 tokens.
This matters for applications where users naturally type in transliterated form, which is common on mobile keyboards in India. If your users are already writing in Hinglish or Romanized Hindi, don't convert their input to Devanagari before sending to the API. Keep it in the more token-efficient format the user provided.
For Arabic, there's a similar phenomenon — Arabizi (Arabic written in Latin characters) tokenizes far more efficiently than Arabic script. However, for formal Arabic applications, script switching isn't appropriate. Know your user context.
Strategy 3: Compress User Input Before Sending
For non-English applications processing user-generated content, pre-processing steps that reduce token count before the API call can meaningfully reduce costs.
Removing duplicate whitespace, normalizing punctuation, stripping boilerplate phrases that appear in every message (common in chat interfaces), and truncating overly long inputs to a maximum character count before sending all reduce token consumption. For non-Latin scripts where every character is expensive, a 20% reduction in input characters is a 20% reduction in input tokens — the relationship is more direct than in English.
Strategy 4: Choose Model Based on Language
Don't pick one model for all languages. For a multilingual application, the optimal cost-quality model varies by language:
For Hindi and other Indic languages: Gemini for cost efficiency, GPT-4o or Claude for maximum quality.
For Arabic: GPT-4o has slightly better Arabic comprehension in benchmarks; Gemini has better tokenization efficiency.
For Chinese and Japanese: Gemini for cost; GPT-4o for code-mixed or technical content.
For European languages: Any major model; cost differences are small enough that other factors dominate.
For English: Use the cheapest model that meets quality requirements — the tokenizer advantage is universal.
A routing layer that sends requests to different models based on the detected language of the input is not over-engineering — for high-volume multilingual apps, it's responsible cost architecture.
Strategy 5: Use a Hybrid Language Architecture
For applications where some processing must happen in the user's native language and some does not, architect the pipeline to minimize native-language API calls.
Example: a content analysis pipeline for Hindi articles. Instead of sending the full Hindi article to the AI for classification and analysis, consider: translate to English first (cheap, fast, using a dedicated translation API or model), then run all analysis in English, then translate the output back to Hindi if needed.
This sounds counterintuitive but the economics can work. Translation is computationally cheap and token-efficient relative to complex reasoning tasks. Running summarization, classification, and extraction in English and then translating the output can cost less than running those same tasks directly in Hindi, depending on volume and task complexity.
This approach has tradeoffs — translation quality, latency, and nuance loss — but for many use cases it's worth evaluating against the direct multilingual approach.
The Indic Language Opportunity Nobody Is Talking About
Here's a perspective worth sitting with if you are building AI products in India.
India has 22 officially recognized languages, hundreds of millions of smartphone users, and a rapidly growing middle class that prefers to interact with technology in their native language. The AI products that will win in this market are the ones that speak to users in Hindi, Telugu, Tamil, Marathi, Bengali, and Gujarati — not the ones that force users to interact in English.
But right now, most developers building for this market are either unaware of the token cost premium or are absorbing it unknowingly. The developers who understand the multilingual token economics — who architect around it, who choose models strategically, who keep system prompts in English and route intelligently — will build products that are 2–3x more cost-efficient than competitors who don't.
That cost efficiency compounds. Lower API costs mean either higher margins, lower prices for users, or more budget to invest in features. In a competitive market, 2–3x cost efficiency is a serious structural advantage.
The same opportunity exists in Arabic markets, Southeast Asian markets, and anywhere language-native AI products are being built. The token trap is real, but it's only a trap if you don't see it coming.
Calculate Your Multilingual Costs Before You Scale
The single most important takeaway from this article is: measure before you commit.
If you are building a non-English AI application and you haven't calculated your expected token costs at realistic message volumes in your target language, you are flying blind toward a billing surprise.
Take your actual system prompt. Take five representative user messages in your target language. Take five representative AI responses. Run all of them through a token calculator that shows you counts per model — Claude, GPT-4o, Gemini, Mistral, Grok — and multiply by your expected monthly call volume. Then compare to the English equivalent and understand your language multiplier.
Test Multilingual Tokenization
This calculation takes ten minutes. The insight it gives you will shape your model choice, your system prompt language, your architecture decisions, and your pricing model. None of those decisions should be made without it.
The Bottom Line: Hindi and Arabic prompts cost 3–5x more tokens than English because AI tokenizers were trained on English-dominant data, giving non-Latin scripts no choice but to tokenize at the character level rather than the word level. Every Hindi word that costs 4 tokens instead of 1 is a cost multiplier that compounds across every API call, every conversation, every month. The strategies to manage this — English system prompts, transliteration awareness, model routing by language, and hybrid architectures — can reduce your multilingual token bill by 30–50%. But the first step is knowing the problem exists. Most developers building for non-English markets find out too late. Now you know before you scale.
