Skip to content

github.com/ricardo/anthrogo/pkg/tokens

package tokens // import "github.com/ricardo/anthrogo/pkg/tokens"


FUNCTIONS

func SetAnthropicAPICounter(fn func(model string, blocks []message.Block) int)
    SetAnthropicAPICounter registers a callback that will be invoked for
    Claude-family models instead of the char/4 approximation. Pass nil to clear
    the hook and restore the default fallback.


TYPES

type Counter struct {
    // Has unexported fields.
}
    Counter is bound to a specific model. Construct via NewCounter; if the
    model uses tiktoken (OpenAI-family or compatible), the encoder is cached;
    otherwise CountText falls back to char/4 approximation.

func NewCounter(model string) *Counter
    NewCounter returns a Counter for the given model. If the model maps
    to a tiktoken encoding, the encoder is lazily initialised and cached.
    For claude-* or unrecognised models the Counter uses a char/4 approximation.

func (c *Counter) CountBlocks(blocks []message.Block) int
    CountBlocks counts tokens across a slice of message.Block values:
      - BlockText: tokenize Text
      - BlockToolUse: tokenize ToolName + JSON of Input
      - BlockToolResult: tokenize Text
      - BlockThinking: tokenize Text
      - BlockImage: 0 (image tokens are model-specific; provider returns real
        count via EventUsage; M8.10 does not estimate)

    For Claude-family models (no tiktoken encoding), if an AnthropicAPICounter
    has been set via SetAnthropicAPICounter, it is invoked first. A return value
    of -1 means the API call failed; in that case the char/4 fallback is used.

func (c *Counter) CountMessages(msgs []message.Message) int
    CountMessages sums CountBlocks across each message plus a small per-message
    overhead (~3 tokens for role tagging — matches OpenAI's documented format).

func (c *Counter) CountText(s string) int
    CountText returns the token count for s under the model's encoding,
    or (len(s)+3)/4 if no encoding is available (Anthropic fallback). The +3
    rounds up so single-character strings return 1, not 0.