Token counting code reference: count GPT, Claude, and Gemini tokens in any language

Toklen counts tokens for GPT, Claude, and Gemini in your browser. The reference below covers how to do the same thing programmatically: provider SDKs, language ports, raw HTTP calls, and the model and encoding details you need when wiring token accounting into a pipeline or test harness.

Python: count tokens per provider

The tiktoken Python package ships the BPE vocabularies as compressed blobs and tokenizes in a Rust core. Anthropic and Google's exact endpoints bill against your rate limit, so for inputs over a few hundred tokens cache the result by hash.

JavaScript and TypeScript

Browser, Node, and edge runtimes. The browser path is what Toklen uses internally.

The WASM build (@dqbd/tiktoken) is faster on long inputs but ships ~1.4MB of WASM and requires unsafe-eval in your CSP. The pure-JS port (js-tiktoken) is ~700KB and works under stricter CSP, which makes it the right choice for static sites.

Always call .free() on @dqbd/tiktoken encoders. The WASM heap leaks otherwise.

Go

import "github.com/pkoukk/tiktoken-go"

enc, _ := tiktoken.EncodingForModel("gpt-4o")
n := len(enc.Encode(text, nil, nil))

For Claude, use anthropic-sdk-go's Messages.CountTokens. For Gemini, google.golang.org/genai's Models.CountTokens.

The tiktoken-go package downloads encoding files on first call. In a container, pre-warm by pinning TIKTOKEN_CACHE_DIR and running a one-off encode at build time.

Rust

use tiktoken_rs::cl100k_base;

let bpe = cl100k_base()?;
let n = bpe.encode_with_special_tokens(text).len();

tiktoken-rs is the canonical port. For other providers there is no exact-count Rust client, so route through the HTTP endpoints with reqwest.

Shell and curl

OpenAI returns token usage on every completion response. For tracking spend in production, prefer response.usage.prompt_tokens over recomputing.

Encoding reference

The four encodings tiktoken ships, plus what each is for.

Pick o200k_base for new GPT work, cl100k_base as the de-facto fallback for Anthropic, the older two only when targeting legacy OpenAI APIs.

Model reference: tokenizer, context, pricing per 1M tokens

Prices as of May 2026; provider listings change without breaking SDK behavior, so verify against current docs before billing logic.

Gemini's split pricing reflects the under/over 128k tier; the second number applies to prompts above 128,000 tokens. Anthropic's batch API halves these prices.

Cost math, in one place

cost_usd = (input_tokens  / 1_000_000) * input_price_per_million
         + (output_tokens / 1_000_000) * output_price_per_million

Anthropic and OpenAI both offer prompt caching. Cached input tokens bill at 10 to 25 percent of the normal rate depending on provider. For Anthropic, the cache-aware formula is:

cost = (cache_create * 1.25 * input_price
      + cache_read   * 0.10 * input_price
      + uncached            * input_price) / 1_000_000

OpenAI's automatic cache discounts cached input to 50 percent (25 percent on some models) with no explicit cache_control block. The API returns cached_tokens in the usage object.

Common pitfalls

Tokens per request: rough estimates by content type

Useful when you do not have the text yet but need to size a prompt.

These are calibration medians, not guarantees. For billing logic, count the real tokens.

Related concepts

• Prompt caching: OpenAI's automatic prompt cache and Anthropic's cache_control blocks both reduce input cost at the price of slightly higher first-write cost.
• Batch API: OpenAI and Anthropic offer 50 percent off for async batch processing with a 24-hour SLA.
• Tool use overhead: tool and function definitions count against your input budget. A typical 5-tool spec adds 500 to 1,500 tokens before any user input.
• System prompts: Claude bills system prompts as input tokens. OpenAI bills system messages the same way.
• Max tokens vs context window: most providers' "context window" is the combined input + output. max_tokens is the output reservation that reduces input headroom.