For AI agents: the complete documentation index is available at /en/llms.txt, the full documentation bundle is available at /en/llms-full.txt, and this page is available as Markdown at /en/changelog/token-estimation-accuracy.md.
Hunea
  • English

      • tiktoken 4.x model catalog, more accurate token estimates for Chinese models

      • Date: 2026-08-18
      • Commit: fix(runtime-domain): adapt to the tiktoken 4.x model catalog and add Chinese model family recognition
      • Version cue: 0.12.1-alpha.4

      Hunea's context usage estimates depend on tiktoken's "model → encoding" catalog. Previously tiktoken 3.x only matched by model-name prefix / exact string: provider/model ids such as local/qwen3 and newer Chinese models (Kimi, GLM, MiniMax, DeepSeek V4) all fell back to the o200k_base estimate, so the displayed context usage could deviate noticeably from the real request. This change adapts to the tiktoken 4.x catalog and adds dedicated encodings for Chinese model families, so /context estimates track reality more closely.

      What changed

      1. Adapts to the tiktoken 4.x model catalog
        Encoding resolution now checks the tiktoken 4.x catalog first; provider/model ids strip the last segment (basename) before matching, so local/qwen3 and custom-* aliases no longer drop into the o200k_base fallback.

      2. Chinese model family recognition
        ModelFamily adds KimiK2 / KimiK3, Glm4 / Glm5, MiniMax, and DeepseekV4, mapped to dedicated encodings (kimi_k2 / kimi_k3 / glm4 / glm5 / minimax_m2 / deepseek_v4) instead of the o200k fallback estimate.

      3. Refined DeepSeek / Mistral matching
        deepseek-chat / deepseek-reasoner map to V4 per the tiktoken 4.x aliases, while other DeepSeek models keep V3; Mistral matching also adds pixtral.

      Notes

      • See /context for how context usage is displayed and estimated.
      • This change affects estimate accuracy; billing is still based on what each provider reports.