OmniToken

OmniToken counts, encodes and decodes LLM tokens in pure Go, with no CGO, no Python and no Rust in the root module. It is for Go services that need fast local token accounting to size prompts and plan context windows.

The OpenAI BPE path uses a regex-free scanner and a pure-Go merge loop over embedded vocabularies. CountTokens never builds the token slice, which is how it reaches zero allocations per call on supported encodings.

Correctness is checked against 50,000 deterministic reference cases. Speed comes from a benchmark harness that records the machine it ran on, and the numbers belong to that machine and workload, so rerun them on yours.

A small cacheflow package analyses prompt-cache boundaries. It reports and never edits your prompts.