explainx.ai350K+ learn from us
TrendingAI News TodayExplainx for Business
Pricing
explainx.ai

Upskill in AI — 16 free pathways, live workshops & bootcamps, and 50+ courses from practitioners. Plus the skills, tools, and MCP servers to practice on.

follow us

follow on google

Add explainx.ai as a preferred source

corporate training

support@explainx.ai

get started

Find your pathTake Free Evaluation

community

Join the community

learn

mind: share how you thinkpathways — start freeworkshopsbootcampscoursescompare Explainxcertificationsmock testsexplainx universitycorporate traininglearn skills & mcp

discover

skillsmcp serversexplainx mcpestacktoolsmdx readeragentsllmsdesignsdictionarypeopleagi trackerfelony benchranks

company

aboutvisionmissionteaminstructorsteach on explainxpartnershipscommunityhackathonscareers

content

daily AI newsstate of AI — live resultsblogreleasespromptsgeneratorsresource libraryfor LLMsexplainx.ai kids

solutions

all solutionsdeveloper upskillingmarketing upskillingproduct manager upskillingleadership upskilling

newsletter · weekly

Get AI news, tools, and insights in your inbox.

supportcontactprivacytermsdata rightshow we create contentsubmission guidelines

© 2026 AISOLO Technologies Pvt Ltd

explainx.ai

  1. Home
  2. /
  3. Dictionary
  4. /
  5. Token
Core Concepts

Token

The atomic unit a language model reads and generates, typically a word, subword, or character.

Ask Melo about this← all terms

A token is the atomic unit a language model reads and generates — typically a word, subword, or character, determined by the model's tokenizer. Most modern LLMs use subword tokenization (like BPE), which balances vocabulary size against the ability to represent rare words. Token count directly affects computational cost, context window limits, and pricing for API-based models. Understanding tokenization is essential for prompt engineering, cost estimation, and diagnosing unexpected model behavior with unusual inputs.

Related terms

Large Language ModelNatural Language ProcessingGenerative AIScaling LawsArtificial IntelligenceComputer Vision

Where Token comes up

  • Microsoft agent-framework PyPI 1.26.1 Hijack: Azure and GitHub Token Stealer
  • GPT-6.2 Astra: Epoch Says 12.3 Reasoning Steps Hide in One Token
  • Sarvam Token Factory: India-Hosted Inference for Open-Weight Models, Priced in Rupees
  • Hugging Face Routes Hub Inference Through OpenRouter: One Token, Many Providers