MiniLLMLib (Rust)

A minimalist, async-first Rust library for talking to Large Language Models over HTTP, with one consistent API across every provider.

The headline idea: ChatNode::root(...).chat(...) is identical no matter what is behind it. A Provider owns the entire wire dialect (endpoint, auth, request body, response and stream envelope, cost accounting). Your code only ever deals in normalized types, so switching from OpenRouter to OpenAI, to a native Anthropic key, to a Claude subscription, or to your own self-hosted server is a one-line change.

What's here

• Conversation trees. A conversation is a tree of ChatNode handles. Linear chats, branching, and prebuilt history all use the same structure.
• Multiple providers. OpenRouter, OpenAI, native Anthropic (/v1/messages), a generic OpenAI-compatible provider for self-hosted servers, and your own hand-written impl Provider for any other wire.
• Streaming over SSE, with an idle-timeout that won't kill a long live generation but fails loudly on a dead connection.
• Honest cost tracking. Per-provider usage and cost, with disjoint cached/uncached/cache-write token buckets and a CostResolution (Resolved / Unpriced / Unknown) that never reports a fake $0.
• Prompt caching, marked on the tree and enforced per-provider.
• Claude subscription auth: use your Pro/Max plan instead of an API key.
• JSON repair for malformed model output.

Two layers of documentation

The guide teaches you how to use the library; the API reference (auto-generated from the source by docs.rs) is the exhaustive signature lookup. Start with Quickstart.

Quickstart

Install

# Cargo.toml
[dependencies]
minillmlib = "0.5"
tokio = { version = "1", features = ["full"] }

One call

use minillmlib::{ChatNode, GeneratorInfo};

#[tokio::main]
async fn main() -> minillmlib::Result<()> {
    // Pick a provider. OpenRouter reads OPENROUTER_API_KEY from the environment.
    let generator = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");

    // A conversation is a tree; `root` is the system prompt.
    let root = ChatNode::root("You are a helpful assistant. Be brief.");

    // `chat` = add a user message + get the assistant reply (returns the new node).
    let answer = root.chat("Say hello in five words.", &generator).await?;

    println!("{}", answer.message.text().unwrap_or(""));
    Ok(())
}

That is the 80% case. Swapping the provider is a one-line change and nothing else moves:

#![allow(unused)]
fn main() {
use minillmlib::GeneratorInfo;
GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");      // OPENROUTER_API_KEY
GeneratorInfo::openai("gpt-4o-mini");                           // OPENAI_API_KEY
GeneratorInfo::anthropic("claude-haiku-4-5");                   // ANTHROPIC_API_KEY, native /v1/messages
GeneratorInfo::claude_subscription("claude-haiku-4-5");         // your Pro/Max plan, no API key
GeneratorInfo::custom("my", "http://localhost:8000/v1", "m");   // your own OpenAI-compatible server
}

See Providers for what each does, and Custom Providers for connecting your own server.

A multi-turn conversation

Each chat returns the assistant node; chain from it to continue the thread.

use minillmlib::{ChatNode, GeneratorInfo};

#[tokio::main]
async fn main() -> minillmlib::Result<()> {
    let gen = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");
    let root = ChatNode::root("You are a terse assistant.");

    let a1 = root.chat("What's the capital of France?", &gen).await?;
    let a2 = a1.chat("And its population, roughly?", &gen).await?;

    println!("{}", a2.message.text().unwrap_or(""));
    // a2 knows its whole history: a2.thread() is the full root-to-leaf message list.
    Ok(())
}

Errors

Every fallible call returns minillmlib::Result<T>, an alias for Result<T, MiniLLMError>. The library fails loudly: an auth/validation error, a malformed response, or an exhausted retry surface as a typed MiniLLMError, never a silent empty success.

Conversation Trees

A conversation is a tree of ChatNode handles. Each node holds one Message; children are alternate continuations. A linear chat is just a tree with one branch. Holding any node keeps its whole ancestor chain (and the tree) alive.

Building a thread

add_user / add_assistant each return the new node, so you chain them:

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
let root = ChatNode::root("You are a terse assistant.");
let leaf = root
    .add_user("What's the capital of France?")
    .add_assistant("Paris.")
    .add_user("And Germany?")
    .add_assistant("Berlin.")
    .add_user("And Italy?"); // the turn we want answered
}

leaf.thread() is the full [system, user, assistant, user, assistant, user] message list from root to leaf.

Completing from any node

node.complete(generator, params) uses node's root-to-node path as the prompt and appends the reply as a child of node, returning the new assistant node.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo};
async fn run(leaf: ChatNode, gen: GeneratorInfo) -> minillmlib::Result<()> {
let answer = leaf.complete(&gen, None).await?; // None = default per-request params
println!("{}", answer.message.text().unwrap_or(""));
Ok(()) }
}

You can complete from any node, not just the leaf, to branch off it. The whole root-to-that-node path is the context.

Prebuilt history from a Vec<Message>

When you already have a message list, from_messages builds the linear chain and hands back (root, leaf). Complete from the leaf.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo, Message};

async fn run(gen: GeneratorInfo) -> minillmlib::Result<()> {
let history = vec![
    Message::system("You are a terse assistant."),
    Message::user("What's the capital of France?"),
    Message::assistant("Paris."),
    Message::user("And Germany?"),
    Message::assistant("Berlin."),
    Message::user("And Italy?"),
];

let (_root, leaf) = ChatNode::from_messages(&history)?;
let answer = leaf.complete(&gen, None).await?;
Ok(()) }
}

Ownership: keep a handle

The tree lives in a shared arena that stays alive as long as you hold any handle into it. from_messages returns both root and leaf precisely so you don't accidentally drop the only handle. When you chain add_user/add_assistant, holding the final node is enough (it keeps its whole ancestor chain). Drop every handle and the thread is freed.

Saving and loading threads

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
fn run(leaf: ChatNode) -> minillmlib::Result<()> {
leaf.save_thread("conversation.json")?;
let (root, leaf) = ChatNode::from_thread_file("conversation.json")?;
Ok(()) }
}

Providers

A GeneratorInfo bundles a model, a base URL, an auth strategy, and a Provider (the wire dialect). The provider owns everything that differs between APIs; your calling code never changes. The crate ships these presets:

Auth

Auth is a strategy on the generator, mapped to concrete headers by the provider (so the same Anthropic provider serves both an API key and a subscription token):

#![allow(unused)]
fn main() {
use minillmlib::GeneratorInfo;
let g = GeneratorInfo::openai("gpt-4o-mini");
g.clone().with_api_key("sk-...");                 // provider picks the header (Bearer / x-api-key)
g.clone().with_api_key_from_env("MY_KEY");        // no-op if the var is unset
g.clone().with_bearer_token("token");             // always Authorization: Bearer
g.clone().with_header("X-Tenant", "acme");        // any extra header
}

Cost for token-only providers

OpenAI and Anthropic return token counts but no dollar amount. Attach a TokenPrice (USD per million tokens, the unit every price sheet quotes) to get a resolved cost; otherwise tracking reports Unpriced (never a fake $0):

#![allow(unused)]
fn main() {
use minillmlib::{GeneratorInfo, TokenPrice};

let gen = GeneratorInfo::anthropic("claude-haiku-4-5")
    .with_token_price(TokenPrice::new(1.0, 5.0)); // $1/Mtok in, $5/Mtok out
}

See Cost Tracking for the full picture.

OpenRouter routing

OpenRouter-specific routing (provider order, sort, data-collection) is attached honestly through the extra escape hatch rather than masquerading as a universal parameter:

#![allow(unused)]
fn main() {
use minillmlib::{CompletionParameters, ProviderSettings};

let routing = ProviderSettings::new()
    .sort_by_throughput()
    .deny_data_collection();

let params = CompletionParameters::new()
    .with_openrouter_routing(routing);
}

Non-OpenRouter providers simply ignore it.

Completion Parameters

Two layers of parameters:

• CompletionParameters: normalized generation intent (temperature, max tokens, stop, response format, ...). NOT a wire shape: each provider's build_request maps it to its own request body, so the same params drive any provider identically.
• NodeCompletionParameters: per-request behavior around the call (system prompt override, JSON repair, retry, cost tracking, caching, the wrapped CompletionParameters).

You pass NodeCompletionParameters to complete; None means defaults.

CompletionParameters

#![allow(unused)]
fn main() {
use minillmlib::CompletionParameters;

let params = CompletionParameters::new()
    .with_max_tokens(512)
    .with_temperature(0.7)
    .with_stop(vec!["END".to_string()]);
}

CompletionParameters is also a serde type: camelCase keys, every field optional (missing ones take the defaults above), unknown keys ignored. A flat JSON settings object ({"maxTokens": 1024, "temperature": 0.2}) deserializes directly, which is handy when parameters arrive as user-facing config.

NodeCompletionParameters

#![allow(unused)]
fn main() {
use minillmlib::{CompletionParameters, NodeCompletionParameters};

let params = NodeCompletionParameters::new()
    .with_params(CompletionParameters::new().with_max_tokens(200))
    .with_system_prompt("You are concise.")  // prepend if the thread has no system message
    .expecting_json()                         // parse + repair the response as JSON
    .with_force_prepend("Answer: ")           // make the model continue from this prefix
    .with_cost_tracking(true);                // request usage and fire the cost callback
}

Tool Calling

Tools are normalized intent, like every other parameter: you declare ToolDefinitions and a ToolChoice once, and each provider emits its own wire shape. The same code drives OpenRouter, OpenAI, any OpenAI-compatible server, and native Anthropic.

The loop

#![allow(unused)]
fn main() {
use minillmlib::{
    ChatNode, CompletionParameters, GeneratorInfo, NodeCompletionParameters,
    ToolChoice, ToolDefinition,
};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("anthropic/claude-sonnet-4.5");

let params = NodeCompletionParameters::new().with_params(
    CompletionParameters::new()
        .with_tool(ToolDefinition::new(
            "get_weather",
            "Get the current weather for a city",
            serde_json::json!({
                "type": "object",
                "properties": { "city": { "type": "string" } },
                "required": ["city"],
            }),
        ))
        .with_tool_choice(ToolChoice::Auto),
);

let node = ChatNode::root("You are helpful.")
    .add_user("What's the weather in Paris?")
    .complete(&gen, Some(&params))
    .await?;

// The model called a tool: run it and answer each call, then complete again.
if let Some(calls) = node.tool_calls() {
    let mut current = node.clone();
    for call in &calls {
        let args = call.arguments_json()?;           // parsed arguments, fails loudly
        let result = format!("15 degrees in {}", args["city"]);
        current = current.add_tool_result(&call.id, result);
    }
    let answer = current.complete(&gen, Some(&params)).await?;
    println!("{}", answer.text().unwrap_or(""));
}
Ok(()) }
}

Notes:

• Keep the same tools in the follow-up request. Providers require the tool definitions to still be present when you send back the results.
• Parallel calls: the model may return several ToolCalls in one turn; add one add_tool_result per call (in any order). The Anthropic provider packs consecutive results into the single user turn its wire requires. Forbid parallelism with .with_parallel_tool_calls(false).
• Arguments are raw JSON text (ToolCall::arguments), exactly as the model produced them; arguments_json() parses them and fails loudly on invalid JSON instead of silently repairing.
• Forcing a call: ToolChoice::Required (any tool) or ToolChoice::Tool("get_weather".into()) (that one).
• Strict schemas: ToolDefinition::with_strict(true) asks the provider to guarantee the arguments match your schema (OpenAI structured outputs, Anthropic strict tool use).
• Streaming works too: tool-call fragments are accumulated across chunks and the final CompletionResponse::tool_calls (and the node) carry the assembled calls.

Streaming a tool call as it is generated

You don't have to wait for the model to finish a call before acting on it. The streaming chunks expose typed ToolCallDelta fragments with the same timing on both wires: the first fragment carries the call's name (and id), then each later fragment carries a piece of the raw JSON argument text, in order. That lets you start the tool the moment the model names it and pipe the argument bytes in while the model is still generating them.

#![allow(unused)]
fn main() {
use minillmlib::{
    ChatNode, CompletionParameters, GeneratorInfo, NodeCompletionParameters,
    ToolChoice, ToolDefinition,
};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("anthropic/claude-sonnet-4.5");

let params = NodeCompletionParameters::new().with_params(
    CompletionParameters::new()
        .with_tool(
            ToolDefinition::new(
                "run_python",
                "Execute Python code",
                serde_json::json!({
                    "type": "object",
                    "properties": { "code": { "type": "string" } },
                    "required": ["code"],
                }),
            )
            .with_strict(true),
        )
        .with_tool_choice(ToolChoice::Tool("run_python".into())),
);

let root = ChatNode::root("You are helpful.");
let user = root.add_user("Compute the 100th Fibonacci number.");
let mut stream = user.complete_streaming(&gen, Some(&params)).await?;

let mut tool_started = false;
while let Some(chunk) = stream.next_chunk().await {
    let chunk = chunk?;
    if let Some(deltas) = &chunk.tool_calls {
        for delta in deltas {
            // First fragment carries the name: start the tool NOW
            // (e.g. spawn the interpreter process here).
            if let Some(name) = &delta.name {
                println!(">> model is calling {name}, starting process");
                tool_started = true;
            }
            // Later fragments: raw JSON argument text, in order.
            if let Some(frag) = &delta.arguments_fragment {
                if tool_started {
                    // CAVEAT: this is escaped JSON source, e.g.
                    // {"code": "print(\"hi\")... (see the note below).
                    print!("{frag}");
                }
            }
        }
    }
}

// The stream assembled the complete calls in parallel: append the assistant
// node and finish the normal loop (add_tool_result + complete again).
let response = stream.collect().await?;
let node = user.append_response(&response);
if let Some(calls) = node.tool_calls() {
    let result = node.add_tool_result(&calls[0].id, "354224848179261915075");
    let answer = result.complete(&gen, Some(&params)).await?;
    println!("{}", answer.text().unwrap_or(""));
}
Ok(()) }
}

Notes:

• The fragments are JSON source text, not your payload. For a tool whose input is one string field (like code above), the bytes arrive escaped and wrapped in the object syntax ({"code": "print(\"hi ...). Put an ArgumentStream between the fragments and the tool to stream the DECODED content instead (see the next section).
• Parallel calls: delta.index disambiguates concurrent calls; key your spawned tools by it. Forcing a single call with ToolChoice::Tool(..) (and .with_parallel_tool_calls(false)) sidesteps this.
• Key order: models may emit argument keys in any order, so with several fields your payload field can arrive last. strict: true plus a one-property schema keeps the stream predictable.

For the complete pattern (a multi-turn agent loop mixing a streaming tool and a buffered tool, forwarding all prose live), see examples/agent_loop.rs: cargo run --example agent_loop. The key mental model: a tool call always ends the model's turn; "the model continues after the tool" is always a new API request that your loop makes after add_tool_result, and the consumer of your stream never sees the seams.

Streaming decoded arguments (ArgumentStream)

ArgumentStream decodes a call's raw argument fragments field by field, live. Every field is the same kind of object: take a FieldHandle for it and choose PER FIELD how to consume it:

• handle.wait().await: the complete parsed value, once the field ends.
• handle.delta().await: the field's DECODED text chunk by chunk as the model generates it (\n a real newline, \" a quote, \uXXXX the character): type code into an editor in real time, pipe into a process's stdin.

Fields nobody took a handle for are parsed into args.fields() as they complete, so a fully non-streaming consumer still gets everything extracted at the end. Fragments may split at any position (mid-escape included); the output never changes.

#![allow(unused)]
fn main() {
use minillmlib::ArgumentStream;

async fn run() -> minillmlib::Result<()> {
let mut args = ArgumentStream::lenient();
let path = args.field("path");            // consume as a whole value
let mut content = args.field("content");  // consume as a live stream

// The tool runs concurrently with the wire:
let tool = tokio::spawn(async move {
    let path = path.wait().await?;                    // complete, parsed
    let mut session = open_editor(path.as_str().unwrap());
    while let Some(text) = content.delta().await {    // decoded, live
        session.type_text(&text);
    }
    Ok::<_, minillmlib::MiniLLMError>(session.close())
});

// The driver feeds the fragments from the streaming loop:
let provider_fragments: Vec<String> = vec![];
for fragment in provider_fragments {
    args.feed(&fragment)?;
}
args.finish()?;      // resolves holdbacks, closes every handle
tool.await.unwrap()?;
Ok(()) }
fn open_editor(_p: &str) -> Session { Session }
struct Session; impl Session { fn type_text(&mut self, _t: &str) {} fn close(self) {} }
}

Two modes:

• ArgumentStream::strict() (default choice): the arguments must be well-formed JSON; a bad escape, unescaped control character, or unterminated string/object fails loudly with the raw text in the error.
• ArgumentStream::lenient(): for models sloppy at escaping, applied to EVERY top-level string value. The rule is deterministic because the legitimate ways a string can end are known: an unescaped " closes the string only when followed by , "key": (the next field's declaration; whitespace optional everywhere, and the key is a full JSON string, spaces included) or by } at the true end of the call (the provider signals that end explicitly). Every other " is literal content, a raw newline is itself, \ before a non-escape character is a literal backslash, and a model that just stops (forgot the closing " or }) still delivers the full content, never silently dropped. On finish(), lenient mode also runs the raw arguments through the crate's JSON repair and fills anything the incremental parse missed into fields(). The one documented misfire: content that literally contains ", "somekey": reads as a field boundary; that ambiguity is unresolvable on the wire. Numbers, booleans, and nested objects must be well-formed in both modes (they have no end anchor).

Any number of fields can be streamed (a patch(old_code, new_code) tool works fine); a handle whose field never arrives resolves as a loud error at finish(). examples/agent_loop.rs uses it for its streaming tool.

For the fully non-streaming path, the assembled call offers the same robustness: ToolCall::arguments_json() parses strictly, and ToolCall::arguments_json_repaired() runs the arguments through the crate's JSON repair first (trailing commas, unclosed braces, single quotes).

Custom wire shapes

An OpenAI-envelope server whose tool shape deviates only needs to override the two tool hooks on Provider (openai_tools_value, openai_tool_choice_value); a fully different wire translates params.tools / params.tool_choice / message.tool_calls itself in its build_request. See Custom Providers.

Cost Tracking

The library tracks usage and cost per request, and is honest about when a cost is actually known.

Token buckets

Input tokens are split into three disjoint, additive buckets so caching is priced correctly across every provider's differing wire conventions:

• uncached_input_tokens: full-price prompt tokens,
• cache_read_tokens: served from a warm cache (cheap),
• cache_write_tokens: written to the cache this request (a premium).

Total input is the sum of the three; cost is a clean weighted sum, no subtraction.

Resolution: never a fake $0

Every reported CostInfo carries a CostResolution:

Check resolution before trusting cost.

The reply and the bill together: complete_costed

The simplest tracked shape, and the right default when the caller itself acts on the cost (settles a budget hold, gates the next call, logs it right there): the CostInfo comes back WITH the result, no callback to register.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo};

async fn run() {
let generator = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");
let root = ChatNode::root("You are helpful.");

let (result, cost) = root.add_user("Hi").complete_costed(&generator, None).await;
let reply = result.expect("completion");
if let Some(cost) = cost {
    println!("that cost ${:.6} ({:?})", cost.cost, cost.resolution);
}
}
}

Same accounting as every other tracked path (usage from the response, the provider's out-of-band resolution as backstop, never a fake $0). An errored completion carries no cost info: the request failed before a billable response existed.

When many call sites should feed one central sink instead, use the callback shapes below; streaming always goes through them, since a stream's cost resolves only after it ends.

A callback per completion

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo, NodeCompletionParameters, CompletionParameters, CostInfo};
use std::sync::{Arc, Mutex};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");
let total = Arc::new(Mutex::new(0.0));
let sink = total.clone();

let params = NodeCompletionParameters::new()
    .with_params(CompletionParameters::new().with_max_tokens(200))
    .with_cost_tracking(true)
    .with_cost_callback(move |info: CostInfo| {
        // info.cost, .prompt_tokens, .completion_tokens,
        // .cache_read_tokens, .cache_write_tokens, .resolution
        *sink.lock().unwrap() += info.cost;
    });

let root = ChatNode::root("You are helpful.");
root.add_user("Hi").complete(&gen, Some(&params)).await?;
println!("total spent: {}", *total.lock().unwrap());
Ok(()) }
}

Enforced tracking via CompletionContext

When you want cost reporting to be structurally guaranteed (not opt-in per call), wrap the generator in a CompletionContext and use complete_tracked. It always reports cost through the context's async callback, and on a cancelled or usage-less stream it resolves out-of-band (e.g. OpenRouter's /generation query) or reports Unknown, rather than silently booking $0.

#![allow(unused)]
fn main() {
use minillmlib::{CompletionContext, CostInfo, AsyncCostCallback, CompletionMeta, GeneratorInfo, ChatNode};
use std::sync::Arc;

async fn run() -> minillmlib::Result<()> {
let generator = GeneratorInfo::openrouter("m");
let callback: AsyncCostCallback = Arc::new(|cost: CostInfo, _meta: CompletionMeta| {
    Box::pin(async move {
        // persist `cost` to your DB / metering here
        let _ = cost;
    })
});
let ctx = CompletionContext::new(generator, serde_json::json!({}), callback, "https://app", "App");

let root = ChatNode::root("You are helpful.");
let _answer = root.add_user("Hi").complete_tracked(&ctx, None).await?;
Ok(()) }
}

For streaming, complete_streaming_tracked returns a TrackedStream that settles cost when it finishes or is cancelled (use cancel().await for a reliable settle; a plain drop is best-effort).

Cost Estimation

Cost tracking tells you what a call did cost. Estimation tells you what it will cost, before you send it, so you can decide whether to allow it (reserve budget, gate a request, refuse an over-priced job). Enable the estimate feature:

minillmlib = { version = "0.5", features = ["estimate"] }

One call on the generator

The GeneratorInfo you already send completions with answers directly:

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, CompletionParameters, GeneratorInfo};

let generator = GeneratorInfo::openrouter("anthropic/claude-haiku-4.5");
let params = CompletionParameters::new().with_max_tokens(1024);

let root = ChatNode::root("You are terse.");
let prompt = root.add_user("Name three primary colours.");

let usd = generator.estimate_cost_usd(&prompt.thread(), &params).await?;
println!("this call will cost at most ${usd:.6}");
}

If you want the raw rates instead (to price several prompts, or to combine with cost tracking's token buckets), use generator.model_rates(), which returns the per-million-token prices and the model's limits.

Where the prices come from

OpenRouter's catalog is the one public price sheet, and it lists the first-party vendors (OpenAI, Anthropic) at their own standard rates. So the model is looked up by its OpenRouter id:

• An OpenRouter generator's model id already is a catalog id. Nothing to set.
• A direct-vendor generator whose id differs sets the catalog id explicitly; that is what unlocks estimation:

#![allow(unused)]
fn main() {
let generator = GeneratorInfo::anthropic("claude-haiku-4-5-20251001")
    .with_openrouter_name("anthropic/claude-haiku-4.5");
}

The only failure is a model the catalog does not know at all. Everything else produces a number.

Whose rates: the provider question

One model is served by many providers at different prices (some models span a 5x range). The rates used are:

1. the serving provider's own, when the generator's provider knows its catalog slug (the built-in Anthropic and OpenAI providers do: a direct call is billed by the vendor itself);
2. otherwise the dearest rate any provider of the model charges, taken bucket by bucket. That is the only figure that is a ceiling wherever OpenRouter's routing lands, and routing really does land on expensive endpoints: a real request was once billed at nearly 4x the advertised rate.

If a request pins one provider through routing settings (a single-entry order with fallbacks off), ProviderSettings::billing_provider() yields its slug and generator.model_rates_served_by(Some(&slug)) prices at exactly that provider.

What the figure means

The estimate is deliberately high, never a best guess: only the low side lets you overspend. It is still an estimate (tokenizers differ across model families), so treat it as a strong ceiling to reserve against, not a guarantee, and replace it with the tracked real cost once the call returns. It assumes:

• no prompt caching (caching only ever lowers the real cost);
• the largest completion the request permits, including any reasoning budget, which providers bill on top of max_tokens;
• a minute of media for a clip whose length you did not state. Set the real length with AudioData::with_duration / VideoData::with_duration whenever your prompt carries audio or video: the one-minute assumption overshoots short clips by a lot (measured live: 5-7x on an 8-second clip) and undershoots anything longer than a minute, which is the one way the estimate can come in below the real cost. With the length declared the media estimate is tight (the video token model matched Gemini's real billing within 1%). Audio bills by the second, at up to 1000x the text rate on some models, so this is where the money is.

There is no error case beyond an uncatalogued model: a prompt counted larger than the model accepts is priced as the largest input the model does accept, so you always get a number.

Keep your generators alive

Each GeneratorInfo caches the prices it fetches for an hour, and clones share the cache. Reuse the same generator for both completions and estimates: only the first estimate in an hour touches the network, and concurrent estimates for the same model share a single fetch. Recreating the generator per call throws the cache away and refetches the price sheet every time. (A generator whose model you change notices and refetches; it never serves another model's prices.)

The library deliberately holds no registry of generators; pooling is yours to do, and only worth doing if you estimate costs. For code that multiplexes many models, generator.pricing_key() gives you the map key: the catalog model id plus the provider slug, exactly the pair that determines the price. Build a generator on a miss, reuse it on a hit:

#![allow(unused)]
fn main() {
use std::collections::HashMap;

struct Generators(HashMap<String, GeneratorInfo>);

impl Generators {
    fn for_model(&mut self, model: &str) -> &GeneratorInfo {
        let generator = GeneratorInfo::openrouter(model);
        self.0.entry(generator.pricing_key()).or_insert(generator)
    }
}
}

If you never estimate costs, ignore all of this: the cache sits unused and costs nothing.

Prompt Caching

Caching intent is marked on the conversation tree; the provider decides the wire. Anthropic emits cache_control markers (honoring its 4-breakpoint cap); OpenAI and OpenRouter auto-cache and ignore the marks. Switch the provider and the same code works.

Mark what to cache

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
let root = ChatNode::root("a large, stable system prompt ...");
root.cache_breakpoint();          // cache just the system prompt

// ...or cache the whole stable prefix of a conversation:
let some_node = root.clone();
some_node.cache_breakpoint();
}

Or, per request, auto-mark the entire prompt prefix without touching individual nodes:

#![allow(unused)]
fn main() {
use minillmlib::NodeCompletionParameters;
let params = NodeCompletionParameters::new().with_cache(true);
}

Explicit per-node marks are always honored in addition.

Clearing marks

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
let node = ChatNode::root("x");
node.clear_cache_breakpoint();        // this node
node.clear_all_cache_breakpoints();   // the whole tree
}

Warming the cache

ensure_cached fires a zero-output request that writes/refreshes the cache for a node's prefix, returning the CostInfo of the warm call. Cheap to call before an agent run: cold pays the one-time write (which you'd pay on the next real call anyway); warm is a cheap read that refreshes the TTL.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo};
async fn run(some_node: ChatNode, generator: GeneratorInfo) -> minillmlib::Result<()> {
let warm_cost = some_node.ensure_cached(&generator, None).await?;
let _ = warm_cost;
Ok(()) }
}

Pricing cached tokens

Cache reads and writes have their own rates (read is a discount, write a premium):

#![allow(unused)]
fn main() {
use minillmlib::TokenPrice;

let price = TokenPrice::new(1.0, 5.0)      // $/Mtok input, output
    .with_cache_rates(0.1, 1.25);          // $/Mtok cache-read, cache-write
}

The three input buckets (uncached / cache-read / cache-write) are billed at their own rates; see Cost Tracking.

Custom Providers

Connecting your own server is one of two cases.

Case A: your server speaks OpenAI's /chat/completions

vLLM, llama.cpp's server, LM Studio, TGI, Ollama's OpenAI endpoint, or your own OpenAI-compatible wrapper. Nothing custom to write: point custom() at it. The default GenericProvider handles the wire.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo, TokenPrice};

async fn run() -> minillmlib::Result<()> {
// base_url is everything BEFORE /chat/completions; the provider appends the path.
let gen = GeneratorInfo::custom("my-server", "http://localhost:8000/v1", "my-model")
    .with_api_key_from_env("MY_SERVER_KEY")    // omit entirely if unauthenticated
    .with_header("X-Tenant", "acme")           // any extra gateway headers
    .with_token_price(TokenPrice::new(0.0, 0.0)); // $/Mtok; 0/0 for a free local model

let answer = ChatNode::root("You are helpful.")
    .chat("hello", &gen).await?;
println!("{}", answer.message.text().unwrap_or(""));
Ok(()) }
}

For an older server that only accepts max_tokens (not max_completion_tokens):

#![allow(unused)]
fn main() {
use minillmlib::{GeneratorInfo, GenericProvider};
use std::sync::Arc;

let gen = GeneratorInfo::custom("old", "http://localhost:8000/v1", "m")
    .with_provider(Arc::new(GenericProvider { legacy_token_limit: true }));
}

If your server speaks the OpenAI envelope but its tool shape deviates, override just the two tool hooks in your impl Provider (openai_tools_value, openai_tool_choice_value); the rest of the default request builder stays. See Tool Calling.

Case B: your server has a different wire

Different endpoint, auth header, request/response shape: implement the Provider trait once and pass it via with_provider. The user-facing API (root.chat(...)) stays identical.

Below is a complete adapter for a made-up "EchoAI" server with a genuinely different wire: endpoint /api/generate, auth header X-Echo-Key, request {model, prompt, settings}, response {output:{text}, meta}. This mirrors the tested example in tests/integration_tests.rs.

#![allow(unused)]
fn main() {
use minillmlib::{
    Auth, ChatNode, CompletionParameters, CompletionResponse, CostOutcome, GeneratorInfo,
    Message, MessageContent, Provider, StreamChunk, TokenPrice, Usage,
};
use secrecy::ExposeSecret;
use std::sync::Arc;

#[derive(Debug, Clone)]
struct EchoAi;

impl Provider for EchoAi {
    fn endpoint_url(&self, base: &str) -> String {
        format!("{}/api/generate", base.trim_end_matches('/'))
    }

    fn auth_headers(&self, auth: &Auth) -> minillmlib::Result<Vec<(String, String)>> {
        Ok(match auth.secret() {
            Some(s) => vec![("X-Echo-Key".into(), s.expose_secret().to_string())],
            None => vec![],
        })
    }

    fn build_request(
        &self, model: &str, messages: &[Message], params: &CompletionParameters,
        _stream: bool, _include_usage: bool,
    ) -> minillmlib::Result<serde_json::Value> {
        // Flatten the conversation into one prompt. Fail loudly on multimodal
        // (this wire is text-only) instead of silently dropping the attachment.
        let mut lines = Vec::new();
        for m in messages {
            if let MessageContent::Parts(parts) = &m.content {
                if parts.iter().any(|p| p.as_text().is_none()) {
                    return Err(minillmlib::MiniLLMError::InvalidParameter(
                        "EchoAI is text-only".into(),
                    ));
                }
            }
            lines.push(format!("{}: {}", m.role.as_str(), m.content.all_text()));
        }
        Ok(serde_json::json!({
            "model": model,
            "prompt": lines.join("\n"),
            "settings": { "max_output_tokens": params.max_tokens.unwrap_or(256) },
        }))
    }

    fn parse_response(&self, raw: serde_json::Value) -> minillmlib::Result<CompletionResponse> {
        let text = raw["output"]["text"].as_str()
            .ok_or_else(|| minillmlib::MiniLLMError::MalformedResponse(raw.to_string()))?
            .to_string();
        Ok(CompletionResponse {
            id: raw["meta"]["id"].as_str().unwrap_or("").into(),
            model: raw["meta"]["model"].as_str().unwrap_or("").into(),
            content: text,
            finish_reason: raw["stop"].as_str().map(String::from),
            usage: self.parse_usage(&raw),
            tool_calls: None,
            media: Vec::new(),
            raw_response: Some(raw),
        })
    }

    fn parse_usage(&self, raw: &serde_json::Value) -> Option<Usage> {
        let meta = raw.get("meta")?;
        Some(Usage {
            uncached_input_tokens: meta["tokens_in"].as_u64().unwrap_or(0) as u32,
            completion_tokens: meta["tokens_out"].as_u64().unwrap_or(0) as u32,
            ..Default::default()
        })
    }

    fn parse_chunk(&self, _data: &str) -> Option<minillmlib::Result<StreamChunk>> {
        None // non-streaming
    }

    fn emits_stream_usage(&self, _requested: bool) -> bool {
        false // never sends a trailing usage chunk; don't wait for one
    }

    fn cost_of(&self, usage: Usage, price: Option<&TokenPrice>) -> CostOutcome {
        match price {
            Some(p) => CostOutcome::resolved(p.cost_of(&usage), usage),
            None => CostOutcome::unpriced(usage),
        }
    }
}

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::custom("echoai", "https://my.host", "echo-1")
    .with_provider(Arc::new(EchoAi))
    .with_api_key("my-secret")
    .with_token_price(TokenPrice::new(1.0, 5.0));

let answer = ChatNode::root("You are EchoAI.")
    .chat("hello", &gen).await?;
let _ = answer;
Ok(()) }
}

What to override

The trait defaults to the OpenAI dialect, so you override only what differs:

Reusing the OpenAI dialect piecewise

The shared dialect the defaults delegate to is public: minillmlib::provider::openai_wire holds the request builder, the envelope/usage/chunk parsers, and the wire projections of the normalized types (messages_to_payload, tool_definition_value, ...). A provider that is mostly OpenAI-shaped can override one method and still call these for everything it does not change, instead of copy-pasting the dialect.

Two rules to copy from the example

• Fail loudly on anything you can't represent. EchoAI rejects multimodal rather than silently flattening it away.
• Override emits_stream_usage to false if your server never sends a trailing usage chunk, or a streaming call will wait for it until the idle timeout.

Claude Subscription

Use your Claude Pro/Max subscription instead of a pay-as-you-go API key. A subscription OAuth token authenticates against the same native Anthropic API as an API key, but draws on your subscription's rolling quota (the 5-hour / 7-day window) rather than API billing.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo, TokenPrice};

async fn run() -> minillmlib::Result<()> {
// Anthropic returns token counts but no dollar cost, so set a price for a
// resolved cost ESTIMATE (otherwise tracking reports `Unpriced`).
let generator = GeneratorInfo::claude_subscription("claude-haiku-4-5")
    .with_token_price(TokenPrice::new(1.0, 5.0)); // $/Mtok in, $/Mtok out

let root = ChatNode::root("You are helpful.");
let response = root.chat("Hello!", &generator).await?;
let _ = response;
Ok(()) }
}

How the token is resolved

claude_subscription resolves the bearer token in this order:

1. the ANTHROPIC_AUTH_TOKEN env var, if set (explicit override; you keep it fresh, e.g. from claude setup-token);
2. otherwise the live Claude Code credential at ~/.claude/.credentials.json (claudeAiOauth.accessToken), which Claude Code keeps refreshed, so if you're logged into Claude Code with your subscription, it just works.

If neither source yields a token, the request fails loudly as unauthenticated rather than silently using the wrong account.

Subscription vs Console

A subscription token (from Claude Code) bills your Pro/Max plan. A Console/API OAuth token bills your API account, not the subscription. For Console use an API key via GeneratorInfo::anthropic(model), and this preset only for the actual Pro/Max subscription token.

Cost is always an ESTIMATE here: Anthropic returns only token counts, so the TokenPrice you set (reflecting the model's published price) produces a Resolved USD estimate; without it, tracking reports Unpriced.

API Reference

The full, auto-generated API reference (every public type, method, and signature) lives on docs.rs:

https://docs.rs/minillmlib

It is generated from the source doc comments and rebuilt automatically when a new version is published to crates.io. This guide covers how to use the library; the docs.rs reference is the exhaustive signature lookup.