MiniLLMLib (Rust)

A minimalist, async-first Rust library for talking to Large Language Models over HTTP.

ChatNode::root(...).chat(...) is the same call whatever provider is behind it. A Provider owns everything that differs between APIs (endpoint, auth, request body, response and stream shapes, cost accounting), and your code deals only in the library's own types. So moving from OpenRouter to OpenAI, to Anthropic's own API, or to your own server means changing the line that makes the generator. What each API can carry still differs: Anthropic's takes no audio or video, and has no seed, penalties or JSON mode.

What's here

• Conversation trees. A conversation is a tree of ChatNode handles; a linear chat is just a tree with one branch.
• Multiple providers. OpenRouter, OpenAI, native Anthropic (/v1/messages), a generic OpenAI-compatible provider for self-hosted servers, and your own hand-written impl Provider for any other wire.
• Streaming over SSE, with an idle-timeout that won't kill a long live generation but fails loudly on a dead connection.
• Cost tracking. What each request cost, with cached and uncached tokens priced at their own rates, and a CostResolution that tells you when the cost is unknown, so a 0.0 is never mistaken for a free request.
• Prompt caching: you mark what to cache on the tree, and each provider writes the markers its API takes.
• JSON repair for malformed model output.

Start with Quickstart. For every public type and signature, go and read the API reference.

Quickstart

Install

# Cargo.toml
[dependencies]
minillmlib = "0.6"
tokio = { version = "1", features = ["full"] }

One call

use minillmlib::{ChatNode, GeneratorInfo};

#[tokio::main]
async fn main() -> minillmlib::Result<()> {
    // Pick a provider. OpenRouter reads OPENROUTER_API_KEY from the environment
    // (if it is in a .env file, call `minillmlib::init()` first, which loads it).
    let generator = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");

    // A conversation is a tree; `root` is the system prompt.
    let root = ChatNode::root("You are a helpful assistant. Be brief.");

    // `chat` = add a user message + get the assistant reply (returns the new node).
    let answer = root.chat("Say hello in five words.", &generator).await?;

    println!("{}", answer.message.text().unwrap_or(""));
    Ok(())
}

If you want another provider, change that one line:

#![allow(unused)]
fn main() {
use minillmlib::GeneratorInfo;
GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");      // OPENROUTER_API_KEY
GeneratorInfo::openai("gpt-4o-mini");                           // OPENAI_API_KEY
GeneratorInfo::anthropic("claude-haiku-4-5");                   // ANTHROPIC_API_KEY, native /v1/messages
GeneratorInfo::custom("my", "http://localhost:8000/v1", "m");   // your own OpenAI-compatible server
}

For what each preset does, go and read Providers, and to connect your own server, Custom Providers.

A multi-turn conversation

Each chat returns the assistant node; chain from it to continue the thread.

use minillmlib::{ChatNode, GeneratorInfo};

#[tokio::main]
async fn main() -> minillmlib::Result<()> {
    let gen = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");
    let root = ChatNode::root("You are a terse assistant.");

    let a1 = root.chat("What's the capital of France?", &gen).await?;
    let a2 = a1.chat("And its population, roughly?", &gen).await?;

    println!("{}", a2.message.text().unwrap_or(""));
    // a2 knows its whole history: a2.thread() is the full root-to-leaf message list.
    Ok(())
}

Errors

Every fallible call returns minillmlib::Result<T>, whose error is a MiniLLMError you can match on: Api { status, message } when the provider refuses, Timeout, and MaxRetriesExceeded once retries you turned on run out. An empty reply counts as a success unless you pass NodeCompletionParameters::new().with_crash_on_empty(true) to complete (see Completion Parameters).

Conversation Trees

A conversation is a tree of ChatNode handles. Each node holds one Message; children are alternate continuations. A linear chat is just a tree with one branch. Holding any node keeps its whole ancestor chain (and the tree) alive.

Building a thread

add_user / add_assistant each return the new node, so you chain them:

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
let root = ChatNode::root("You are a terse assistant.");
let leaf = root
    .add_user("What's the capital of France?")
    .add_assistant("Paris.")
    .add_user("And Germany?")
    .add_assistant("Berlin.")
    .add_user("And Italy?"); // the turn we want answered
}

leaf.thread() is the full [system, user, assistant, user, assistant, user] message list from root to leaf.

Completing from any node

node.complete(generator, params) uses node's root-to-node path as the prompt and appends the reply as a child of node, returning the new assistant node.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo};
async fn run(leaf: ChatNode, gen: GeneratorInfo) -> minillmlib::Result<()> {
let answer = leaf.complete(&gen, None).await?; // None = default per-request params
println!("{}", answer.message.text().unwrap_or(""));
Ok(()) }
}

You can complete from any node, not just the leaf, to branch off it. The whole root-to-that-node path is the context.

Prebuilt history from a Vec<Message>

When you already have a message list, from_messages builds the linear chain and hands back (root, leaf). Complete from the leaf.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo, Message};

async fn run(gen: GeneratorInfo) -> minillmlib::Result<()> {
let history = vec![
    Message::system("You are a terse assistant."),
    Message::user("What's the capital of France?"),
    Message::assistant("Paris."),
    Message::user("And Germany?"),
    Message::assistant("Berlin."),
    Message::user("And Italy?"),
];

let (_root, leaf) = ChatNode::from_messages(&history)?;
let answer = leaf.complete(&gen, None).await?;
Ok(()) }
}

Ownership: keep a handle

The tree lives in a shared arena that stays alive as long as you hold any handle into it. from_messages returns both root and leaf precisely so you don't accidentally drop the only handle. When you chain add_user/add_assistant, holding the final node is enough (it keeps its whole ancestor chain). Drop every handle and the thread is freed.

Saving and loading threads

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
fn run(leaf: ChatNode) -> minillmlib::Result<()> {
leaf.save_thread("conversation.json")?;
let (root, leaf) = ChatNode::from_thread_file("conversation.json")?;
Ok(()) }
}

Providers

A GeneratorInfo bundles a model, a base URL, a credential, and a Provider that speaks one API's format. The crate ships these presets:

Auth

The credential lives on the generator, and each provider turns it into its own header:

#![allow(unused)]
fn main() {
use minillmlib::GeneratorInfo;
let g = GeneratorInfo::openai("gpt-4o-mini");
g.clone().with_api_key("sk-...");                 // provider picks the header (Bearer / x-api-key)
g.clone().with_api_key_from_env("MY_KEY");        // no-op if the var is unset
g.clone().with_bearer_token("token");             // always Authorization: Bearer
g.clone().with_header("X-Tenant", "acme");        // any extra header
}

Your own HTTP client

Every generator shares one HTTP client, which keeps connections open between calls. If you want your own (a proxy, custom TLS, middleware), pass it with with_http_client. It takes a reqwest::Client, or a reqwest_middleware::ClientWithMiddleware if you want middleware, and every request for that generator goes through it, cost lookups included. Your reqwest has to be 0.13, the version this crate uses, or the client will not convert.

#![allow(unused)]
fn main() {
use minillmlib::GeneratorInfo;
let gen = GeneratorInfo::openai("gpt-4o-mini").with_http_client(reqwest::Client::new());
}

A client you pass in replaces the shared one completely, including its 600-second limit on a silent connection. If you want that limit, build your client with reqwest::Client::builder().read_timeout(Duration::from_secs(600)).

If you build your own reqwest::Client, its TLS is set up by reqwest, not by this crate. The shared client runs on ring with a built-in list of root certificates, so it needs nothing from the system. A client of yours built with reqwest::Client::new() needs a rustls crypto library: either turn on reqwest's rustls feature, or install one as the process default before you build it (rustls::crypto::ring::default_provider().install_default()). It also checks certificates against the operating system's trust store, so a container image needs its CA certificates (ca-certificates on Debian).

If your program starts more than one tokio runtime (one per thread, or a throwaway Runtime::new().block_on(...) for a single call), give each runtime's generators their own client. A connection belongs to the runtime that opened it, so once that runtime shuts down, the shared client can hand that dead connection to a call running on a different runtime. The call then fails with HTTP error: error sending request for url (...), and the error's full cause chain ends in "dispatch task is gone: runtime dropped the dispatch task".

Cost for token-only providers

OpenAI and Anthropic send back token counts but no dollar amount, so the library multiplies the tokens by the model's price. If you know the price, give it with with_token_price (USD per million tokens). If you don't, the library looks it up, and when it finds none, tracking reports Unpriced:

#![allow(unused)]
fn main() {
use minillmlib::{GeneratorInfo, TokenPrice};

let gen = GeneratorInfo::anthropic("claude-haiku-4-5")
    .with_token_price(TokenPrice::new(1.0, 5.0)); // $1/Mtok in, $5/Mtok out
}

For where the price is looked up, and the catalog id an Anthropic model needs, go and read Cost Estimation.

OpenRouter routing

If you use OpenRouter, its routing settings (provider order, sort, ignore list, data collection, fallbacks) go in the request's provider key:

#![allow(unused)]
fn main() {
use minillmlib::{CompletionParameters, OpenRouterRouting};

let routing = OpenRouterRouting::new()
    .sort_by_throughput()
    .deny_data_collection();

let params = CompletionParameters::new()
    .with_openrouter_routing(routing);
}

Set routing only on an OpenRouter generator: OpenAI and Anthropic refuse a request that carries a provider key.

Completion Parameters

Two layers of parameters:

• CompletionParameters: what you want from the model (temperature, output cap, stop sequences, reasoning, JSON output, ...). You set them once, and each provider writes them in its own request format.
• NodeCompletionParameters: what happens around the call (a system message, JSON repair, retries, cost tracking, caching, and the CompletionParameters themselves).

You pass NodeCompletionParameters to complete; None means defaults.

CompletionParameters

#![allow(unused)]
fn main() {
use minillmlib::CompletionParameters;

let params = CompletionParameters::new()
    .with_max_tokens(512)
    .with_temperature(0.7)
    .with_stop(vec!["END".to_string()]);
}

On the Anthropic preset, a penalty, seed or response_format fails before the request is sent, with MiniLLMError::InvalidParameter.

Every field starts unset. If neither you nor the generator's with_default_params set a field, it is not sent and the provider's own default applies. The library adds no other setting, except the cap under Output cap and, when you track cost, the key that asks the provider for usage.

If your parameters arrive as JSON (a settings object a user edits), they deserialize straight into CompletionParameters: camelCase keys, every field optional, unknown keys ignored. Reasoning is one word: "off", "on", or the effort, so {"maxTokens": 1024, "temperature": 0.2, "reasoning": "high"} works as it is.

Output cap

max_tokens counts everything the model writes, its reasoning included. That is how OpenAI, Anthropic and OpenRouter all count their cap, so your number goes to each of them as it is.

If you leave it unset, nothing is sent and the provider's own default applies. Anthropic's API refuses a request without a cap, so there the request carries the model's largest output, or 4096 if that is unknown. The largest output is looked up when a request needs it, and the answer is shared by your whole program for an hour (see Cost Estimation). With with_timeout set, the request waits at most half of it for the lookup and then goes out with 4096, keeping the rest of the timeout for itself; without, it can wait up to 30 seconds. If you want requests never to wait on a lookup, set max_tokens, or give max_output_tokens with GeneratorInfo::with_model_info.

Reasoning

#![allow(unused)]
fn main() {
use minillmlib::{CompletionParameters, Reasoning};

let unset = CompletionParameters::new();                       // the API's default
let off = CompletionParameters::new().with_reasoning(Reasoning::Off);
let on = CompletionParameters::new().with_reasoning(Reasoning::On(None));
let high = CompletionParameters::new().with_reasoning(Reasoning::On(Some("high".into())));
}

Each provider sends its own spelling of what you asked:

A custom server (GeneratorInfo::custom) gets OpenAI's spelling.

On(None) sends low on OpenAI and Anthropic (the lowest effort every model there takes), and minimal on OpenRouter, which rounds it to the nearest level the model has. If you want GPT-5's minimal on OpenAI, ask for it by name.

The effort word is sent exactly as you wrote it, so use the provider's own words. Nothing is checked on the way: if the model cannot do what you asked, its API refuses, and that refusal is the request's error. Some cases you will meet:

• Off on a model that cannot switch thinking off is refused.
• On a model that does not reason (gpt-4o), OpenAI refuses a reasoning setting, so leave reasoning unset there.
• On on Claude 4.5 and earlier is refused, since they think only within a token budget and adaptive thinking came later. Send the budget yourself through extra, and leave reasoning unset, since both write thinking: with_extra("thinking", json!({"type": "enabled", "budget_tokens": 2048})).
• An effort a model does not have ("xhigh" on Claude Sonnet 4.6) is refused.

NodeCompletionParameters

#![allow(unused)]
fn main() {
use minillmlib::{CompletionParameters, NodeCompletionParameters};

let params = NodeCompletionParameters::new()
    .with_params(CompletionParameters::new().with_max_tokens(200))
    .with_system_prompt("You are concise.")  // added when the thread has no system message
    .expecting_json()                         // parse + repair the response as JSON
    .with_force_prepend("Answer: ")           // make the model continue from this prefix
    .with_cost_tracking(true);                // request usage, so cost can be reported
}

Tool Calling

Tools are normalized intent, like every other parameter: you declare ToolDefinitions and a ToolChoice once, and each provider emits its own wire shape. The same code drives OpenRouter, OpenAI, any OpenAI-compatible server, and native Anthropic.

The loop

#![allow(unused)]
fn main() {
use minillmlib::{
    ChatNode, CompletionParameters, GeneratorInfo, NodeCompletionParameters,
    ToolChoice, ToolDefinition,
};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("anthropic/claude-sonnet-4.5");

let params = NodeCompletionParameters::new().with_params(
    CompletionParameters::new()
        .with_tool(ToolDefinition::new(
            "get_weather",
            "Get the current weather for a city",
            serde_json::json!({
                "type": "object",
                "properties": { "city": { "type": "string" } },
                "required": ["city"],
            }),
        ))
        .with_tool_choice(ToolChoice::Auto),
);

let node = ChatNode::root("You are helpful.")
    .add_user("What's the weather in Paris?")
    .complete(&gen, Some(&params))
    .await?;

// The model called a tool: run it and answer each call, then complete again.
if let Some(calls) = node.tool_calls() {
    let mut current = node.clone();
    for call in &calls {
        let args = call.arguments_json()?;           // parsed arguments, fails loudly
        let result = format!("15 degrees in {}", args["city"]);
        current = current.add_tool_result(&call.id, result);
    }
    let answer = current.complete(&gen, Some(&params)).await?;
    println!("{}", answer.text().unwrap_or(""));
}
Ok(()) }
}

Notes:

• Keep the same tools in the follow-up request. Providers require the tool definitions to still be present when you send back the results.
• Parallel calls: the model may return several ToolCalls in one turn; add one add_tool_result per call (in any order). The Anthropic provider packs consecutive results into the single user turn its wire requires. Forbid parallelism with .with_parallel_tool_calls(false).
• Arguments are raw JSON text (ToolCall::arguments), exactly as the model produced them; arguments_json() parses them and fails loudly on invalid JSON instead of silently repairing.
• Forcing a call: ToolChoice::Required (any tool) or ToolChoice::Tool("get_weather".into()) (that one).
• Strict schemas: ToolDefinition::with_strict(true) asks the provider to guarantee the arguments match your schema (OpenAI structured outputs, Anthropic strict tool use).
• Streaming works too: tool-call fragments are accumulated across chunks and the final CompletionResponse::tool_calls (and the node) carry the assembled calls.

Streaming a tool call as it is generated

You don't have to wait for the model to finish a call before acting on it. The streaming chunks expose typed ToolCallDelta fragments with the same timing on both wires: the first fragment carries the call's name (and id), then each later fragment carries a piece of the raw JSON argument text, in order. That lets you start the tool the moment the model names it and pipe the argument bytes in while the model is still generating them.

#![allow(unused)]
fn main() {
use minillmlib::{
    ChatNode, CompletionParameters, GeneratorInfo, NodeCompletionParameters,
    ToolChoice, ToolDefinition,
};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("anthropic/claude-sonnet-4.5");

let params = NodeCompletionParameters::new().with_params(
    CompletionParameters::new()
        .with_tool(
            ToolDefinition::new(
                "run_python",
                "Execute Python code",
                serde_json::json!({
                    "type": "object",
                    "properties": { "code": { "type": "string" } },
                    "required": ["code"],
                }),
            )
            .with_strict(true),
        )
        .with_tool_choice(ToolChoice::Tool("run_python".into())),
);

let root = ChatNode::root("You are helpful.");
let user = root.add_user("Compute the 100th Fibonacci number.");
let mut stream = user.complete_streaming(&gen, Some(&params)).await?;

let mut tool_started = false;
while let Some(chunk) = stream.next_chunk().await {
    let chunk = chunk?;
    if let Some(deltas) = &chunk.tool_calls {
        for delta in deltas {
            // First fragment carries the name: start the tool NOW
            // (e.g. spawn the interpreter process here).
            if let Some(name) = &delta.name {
                println!(">> model is calling {name}, starting process");
                tool_started = true;
            }
            // Later fragments: raw JSON argument text, in order.
            if let Some(frag) = &delta.arguments_fragment {
                if tool_started {
                    // CAVEAT: this is escaped JSON source, e.g.
                    // {"code": "print(\"hi\")... (see the note below).
                    print!("{frag}");
                }
            }
        }
    }
}

// The stream assembled the complete calls in parallel: append the assistant
// node and finish the normal loop (add_tool_result + complete again).
let response = stream.collect().await?;
let node = user.append_response(&response);
if let Some(calls) = node.tool_calls() {
    let result = node.add_tool_result(&calls[0].id, "354224848179261915075");
    let answer = result.complete(&gen, Some(&params)).await?;
    println!("{}", answer.text().unwrap_or(""));
}
Ok(()) }
}

Notes:

• The fragments are JSON source text, not your payload. For a tool whose input is one string field (like code above), the bytes arrive escaped and wrapped in the object syntax ({"code": "print(\"hi ...). Put an ArgumentStream between the fragments and the tool to stream the DECODED content instead (see the next section).
• Parallel calls: delta.index disambiguates concurrent calls; key your spawned tools by it. Forcing a single call with ToolChoice::Tool(..) (and .with_parallel_tool_calls(false)) sidesteps this.
• Key order: models may emit argument keys in any order, so with several fields your payload field can arrive last. strict: true plus a one-property schema keeps the stream predictable.

For the complete pattern (a multi-turn agent loop mixing a streaming tool and a buffered tool, forwarding all prose live), see examples/agent_loop.rs: cargo run --example agent_loop. The key mental model: a tool call always ends the model's turn; "the model continues after the tool" is always a new API request that your loop makes after add_tool_result, and the consumer of your stream never sees the seams.

Streaming decoded arguments (ArgumentStream)

ArgumentStream decodes a call's raw argument fragments field by field, live. Every field is the same kind of object: take a FieldHandle for it and choose PER FIELD how to consume it:

• handle.wait().await: the complete parsed value, once the field ends.
• handle.delta().await: the field's DECODED text chunk by chunk as the model generates it (\n a real newline, \" a quote, \uXXXX the character): type code into an editor in real time, pipe into a process's stdin.

Fields nobody took a handle for are parsed into args.fields() as they complete, so a fully non-streaming consumer still gets everything extracted at the end. Fragments may split at any position (mid-escape included); the output never changes.

#![allow(unused)]
fn main() {
use minillmlib::ArgumentStream;

async fn run() -> minillmlib::Result<()> {
let mut args = ArgumentStream::lenient();
let path = args.field("path");            // consume as a whole value
let mut content = args.field("content");  // consume as a live stream

// The tool runs concurrently with the wire:
let tool = tokio::spawn(async move {
    let path = path.wait().await?;                    // complete, parsed
    let mut session = open_editor(path.as_str().unwrap());
    while let Some(text) = content.delta().await {    // decoded, live
        session.type_text(&text);
    }
    Ok::<_, minillmlib::MiniLLMError>(session.close())
});

// The driver feeds the fragments from the streaming loop:
let provider_fragments: Vec<String> = vec![];
for fragment in provider_fragments {
    args.feed(&fragment)?;
}
args.finish()?;      // resolves holdbacks, closes every handle
tool.await.unwrap()?;
Ok(()) }
fn open_editor(_p: &str) -> Session { Session }
struct Session; impl Session { fn type_text(&mut self, _t: &str) {} fn close(self) {} }
}

Two modes:

• ArgumentStream::strict() (default choice): the arguments must be well-formed JSON; a bad escape, unescaped control character, or unterminated string/object fails loudly with the raw text in the error.
• ArgumentStream::lenient(): for models sloppy at escaping, applied to EVERY top-level string value. The rule is deterministic because the legitimate ways a string can end are known: an unescaped " closes the string only when followed by , "key": (the next field's declaration; whitespace optional everywhere, and the key is a full JSON string, spaces included) or by } at the true end of the call (the provider signals that end explicitly). Every other " is literal content, a raw newline is itself, \ before a non-escape character is a literal backslash, and a model that just stops (forgot the closing " or }) still delivers the full content, never silently dropped. On finish(), lenient mode also runs the raw arguments through the crate's JSON repair and fills anything the incremental parse missed into fields(). The one documented misfire: content that literally contains ", "somekey": reads as a field boundary; that ambiguity is unresolvable on the wire. Numbers, booleans, and nested objects must be well-formed in both modes (they have no end anchor).

Any number of fields can be streamed (a patch(old_code, new_code) tool works fine); a handle whose field never arrives resolves as a loud error at finish(). examples/agent_loop.rs uses it for its streaming tool.

For the fully non-streaming path, the assembled call offers the same robustness: ToolCall::arguments_json() parses strictly, and ToolCall::arguments_json_repaired() runs the arguments through the crate's JSON repair first (trailing commas, unclosed braces, single quotes).

Custom wire shapes

An OpenAI-envelope server whose tool shape deviates only needs to override the two tool hooks on Provider (openai_tools_value, openai_tool_choice_value); a fully different wire translates params.tools / params.tool_choice / message.tool_calls itself in its build_request. See Custom Providers.

Cost Tracking

The library reports what each request cost, and says so when it does not know.

Token buckets

Input tokens are split into three buckets that never overlap, so each token is priced once, at its own rate, whatever the provider:

• uncached_input_tokens: full-price prompt tokens,
• cache_read_tokens: served from a warm cache (cheap),
• cache_write_tokens: written to the cache this request (a premium).

These are the fields of Usage, on CompletionResponse::usage. A CostInfo carries cache_read_tokens, cache_write_tokens and prompt_tokens, which is the sum of all three, so the uncached count is prompt_tokens minus the other two. The cost is each bucket times its rate, plus the output tokens times the output rate.

Resolution: whether the cost is known

Every reported CostInfo carries a CostResolution:

Check resolution before trusting cost.

The reply and the bill together: complete_costed

If you want the cost handed back with the reply, call complete_costed:

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo};

async fn run() {
let generator = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");
let root = ChatNode::root("You are helpful.");

let (result, costs) = root.add_user("Hi").complete_costed(&generator, None).await;
for cost in &costs {
    println!("a request cost ${:.6} ({:?})", cost.cost, cost.resolution);
}
let reply = result.expect("completion");
}
}

There is one CostInfo per response received: a response rejected by crash_on_empty_response or crash_on_refusal was billed too, so it is listed, also when the completion fails in the end. A request that fails without a response lists nothing, though a provider may still bill it (one that timed out after the model had finished, for example).

If you would rather have the cost pushed to you, set a callback, as below. If you stream, there is no costed version: use complete_streaming_collect with a callback, or complete_streaming_tracked.

A callback per completion

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo, NodeCompletionParameters, CompletionParameters, CostInfo};
use std::sync::{Arc, Mutex};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("google/gemini-2.5-flash-lite");
let total = Arc::new(Mutex::new(0.0));
let sink = total.clone();

let params = NodeCompletionParameters::new()
    .with_params(CompletionParameters::new().with_max_tokens(200))
    .with_cost_callback(move |info: CostInfo| {
        // info.cost, .prompt_tokens, .completion_tokens,
        // .cache_read_tokens, .cache_write_tokens, .resolution
        *sink.lock().unwrap() += info.cost;
    });

let root = ChatNode::root("You are helpful.");
root.add_user("Hi").complete(&gen, Some(&params)).await?;
println!("total spent: {}", *total.lock().unwrap());
Ok(()) }
}

Setting a callback already asks the provider for usage, so you do not need with_cost_tracking(true). The _tracked methods refuse a callback, because they report to their CompletionContext.

Enforced tracking via CompletionContext

If every completion in your app must report its cost to one place, wrap the generator in a CompletionContext and call complete_tracked. It reports every billed response through the context's async callback.

#![allow(unused)]
fn main() {
use minillmlib::{CompletionContext, CostInfo, AsyncCostCallback, CompletionMeta, GeneratorInfo, ChatNode};
use std::sync::Arc;

async fn run() -> minillmlib::Result<()> {
let generator = GeneratorInfo::openrouter("m");
let callback: AsyncCostCallback = Arc::new(|cost: CostInfo, _meta: CompletionMeta| {
    Box::pin(async move {
        // persist `cost` to your DB / metering here
        let _ = cost;
    })
});
let ctx = CompletionContext::new(generator, serde_json::json!({}), callback, "https://app", "App");

let root = ChatNode::root("You are helpful.");
let _answer = root.add_user("Hi").complete_tracked(&ctx, None).await?;
Ok(()) }
}

The last two arguments are your app's URL and name; on OpenRouter they become the HTTP-Referer and X-Title headers it credits usage to.

If you stream, complete_streaming_tracked returns a TrackedStream. If you only want the finished node, call complete_streaming_collect_tracked instead: it collects the stream, reports the cost and adds the reply to the tree for you. If you keep the TrackedStream and want the whole reply, call collect().await, which reads whatever is left of the stream, then report_cost(&response).await, and append_response(&response) on the node you started the stream from, if you want it in the tree. If you want a kill switch, collect_or_cancel(interrupt) collects and reports in one call, or cancels and reports if interrupt fires first. It returns a CollectOutcome: Finished(response), which is not yet in the tree, Interrupted, or Failed(error) when the stream broke off (a dropped connection, an idle timeout, or an error the provider sent mid-stream), in which case nothing is booked. If you stop early, call cancel().await, which waits until the cost is reported. Just dropping the stream reports it from a background task, which is lost if your runtime shuts down first.

Cost Estimation

Cost tracking tells you what a call did cost. Estimation tells you what it will cost, before you send it, so you can decide whether to allow it. Turn on the estimate feature:

minillmlib = { version = "0.6", features = ["estimate"] }

Ask the generator

The GeneratorInfo you send completions with gives the estimate:

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, CompletionParameters, GeneratorInfo};

async fn run() {

let generator = GeneratorInfo::openrouter("anthropic/claude-haiku-4.5");
let params = CompletionParameters::new().with_max_tokens(1024);
let prompt = ChatNode::root("You are terse.").add_user("Name three primary colours.");

match generator.estimate_cost_usd(&prompt.thread(), &params).await {
    Some(usd) => println!("reserve ${usd:.6} for this call"),
    None => println!("price or output limit unknown"),
}
}
}

Pass the messages you will send. If you fill {placeholders}, pass prompt.formatted_thread(), or, if you give the values with with_format_kwargs, pass prompt.formatted_thread_with_base(&kwargs) with those same values. A system prompt or forced prefix set on NodeCompletionParameters is added only when the request is sent, so it is not counted: add it to the messages you pass if you want it in the figure.

You get None when the library cannot work it out: the model's price is unknown, or nothing limits how much it can write (you set no max_tokens, and neither the model's largest output nor its context window is known).

What the library knows about a model

An estimate needs three facts about the model, which the library keeps in a ModelInfo: its price per token, the most it can write in one reply, and its context window. The price is also what the library reports a finished call as costing when the provider returns token counts but no dollar amount (OpenAI, Anthropic).

If you already know them, give them yourself. Anything you set wins over a lookup:

#![allow(unused)]
fn main() {
use minillmlib::{GeneratorInfo, ModelInfo, TokenPrice};

let generator = GeneratorInfo::custom("local", "http://localhost:8000/v1", "my-model")
    .with_token_price(TokenPrice::new(0.2, 0.6)) // $/Mtok in, out
    .with_model_info(ModelInfo {
        max_output_tokens: Some(8192),
        context_length: Some(32768),
        ..Default::default()
    });
}

For whatever you leave unset, the generator asks its provider:

If you use Anthropic directly, the catalog id anthropic/<model> usually misses, because the catalog writes versions with dots (anthropic/claude-haiku-4.5) where Anthropic uses dashes and a date (claude-haiku-4-5-20251001). When it misses, the price stays unknown and a warning says to name the catalog id:

#![allow(unused)]
fn main() {
use minillmlib::GeneratorInfo;
let generator = GeneratorInfo::anthropic("claude-haiku-4-5-20251001")
    .with_openrouter_name("anthropic/claude-haiku-4.5");
}

Each lookup is shared by your whole program, and its answer, including "the catalog does not list this model", is kept for an hour. If you fix a miss with with_openrouter_name, the new id is looked up straight away. A source that does not answer is asked again after a minute, and one that answers with an error is asked again on the next call. Either way its facts stay unknown and a warning says why. A request that needs Anthropic's cap asks only the Models API, and billing asks only the catalog. If you want to see what the library knows, call generator.model_info().await.

What the figure assumes

The figure leans high so you can reserve against it, except in the three cases below that say it can come in low. Replace it with the tracked cost once the call returns. It assumes:

• everything priced as uncached input. A cache read costs less, but a request that writes the cache (with_cache(true), or a node marked with cache_breakpoint) pays more than the input rate on the cached part (1.25x on Anthropic), so on that request the estimate can come in low;
• the longest reply the request allows: your max_tokens, capped at the model's largest output, or the largest output itself if you set none. Where the largest output is unknown, the context window stands in for it;
• tool definitions, and the tool calls already in the conversation, counted as input text, like the messages;
• text counted with GPT's tokenizer and multiplied by 1.6, which covers the worst case measured against Claude's tokenizer;
• 1,600 tokens for an image of unknown size. Set its size with ImageData::with_dimensions and it is priced at the higher of OpenAI's tile price and Anthropic's pixel price instead, never above 1,600;
• a minute of audio or video for a clip whose length you did not state. Set the real length with AudioData::with_duration / VideoData::with_duration. The one-minute guess counts an 8-second clip at 7.5 times its length and undercounts anything longer than a minute. That matters most for audio, which can bill up to a thousand times the text rate;
• 20 pages for a PDF whose page count you did not state, each counted as 4,000 tokens (2,400 of text after the 1.6 multiplier, plus 1,600 for the page as an image). Set the real count with DocumentData::with_page_count; a longer PDF without one is undercounted.

If the prompt comes out larger than the model's context window, and the window is known, the estimate prices one full window instead, filled with the most expensive kinds of token first.

Prompt Caching

Caching intent is marked on the conversation tree; the provider decides the wire. Anthropic emits cache_control markers (honoring its 4-breakpoint cap); OpenAI and OpenRouter auto-cache and ignore the marks. Switch the provider and the same code works.

Mark what to cache

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
let root = ChatNode::root("a large, stable system prompt ...");
root.cache_breakpoint();          // cache just the system prompt

// ...or cache the whole stable prefix of a conversation:
let some_node = root.clone();
some_node.cache_breakpoint();
}

Or, per request, auto-mark the entire prompt prefix without touching individual nodes:

#![allow(unused)]
fn main() {
use minillmlib::NodeCompletionParameters;
let params = NodeCompletionParameters::new().with_cache(true);
}

Explicit per-node marks are always honored in addition.

Clearing marks

#![allow(unused)]
fn main() {
use minillmlib::ChatNode;
let node = ChatNode::root("x");
node.clear_cache_breakpoint();        // this node
node.clear_all_cache_breakpoints();   // the whole tree
}

Warming the cache

ensure_cached fires a zero-output request that writes/refreshes the cache for a node's prefix, returning the CostInfo of the warm call. Cheap to call before an agent run: cold pays the one-time write (which you'd pay on the next real call anyway); warm is a cheap read that refreshes the TTL.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo};
async fn run(some_node: ChatNode, generator: GeneratorInfo) -> minillmlib::Result<()> {
let warm_cost = some_node.ensure_cached(&generator, None).await?;
let _ = warm_cost;
Ok(()) }
}

Pricing cached tokens

Cache reads and writes have their own rates (read is a discount, write a premium):

#![allow(unused)]
fn main() {
use minillmlib::TokenPrice;

let price = TokenPrice::new(1.0, 5.0)      // $/Mtok input, output
    .with_cache_rates(0.1, 1.25);          // $/Mtok cache-read, cache-write
}

The three input buckets (uncached / cache-read / cache-write) are billed at their own rates; see Cost Tracking.

Custom Providers

How you connect your own server depends on what it speaks.

Case A: your server speaks OpenAI's /chat/completions

vLLM, llama.cpp's server, LM Studio, TGI, Ollama's OpenAI endpoint, or your own OpenAI-compatible wrapper. Nothing custom to write: point custom() at it. The default GenericProvider handles the wire.

#![allow(unused)]
fn main() {
use minillmlib::{ChatNode, GeneratorInfo, TokenPrice};

async fn run() -> minillmlib::Result<()> {
// base_url is everything BEFORE /chat/completions; the provider appends the path.
let gen = GeneratorInfo::custom("my-server", "http://localhost:8000/v1", "my-model")
    .with_api_key_from_env("MY_SERVER_KEY")    // omit entirely if unauthenticated
    .with_header("X-Tenant", "acme")           // any extra gateway headers
    .with_token_price(TokenPrice::new(0.0, 0.0)); // $/Mtok; 0/0 for a free local model

let answer = ChatNode::root("You are helpful.")
    .chat("hello", &gen).await?;
println!("{}", answer.message.text().unwrap_or(""));
Ok(()) }
}

For an older server that only accepts max_tokens (not max_completion_tokens):

#![allow(unused)]
fn main() {
use minillmlib::{GeneratorInfo, GenericProvider};
use std::sync::Arc;

let gen = GeneratorInfo::custom("old", "http://localhost:8000/v1", "m")
    .with_provider(Arc::new(GenericProvider { legacy_token_limit: true }));
}

If your server's tool shape differs from OpenAI's, write your own impl Provider (as in Case B below) that overrides openai_tools_value and openai_tool_choice_value, plus cost_of, which has no default (see What to override). If your server never sends a usage chunk when streaming, also make emits_stream_usage return false, as GenericProvider does. For the tool shapes themselves, go and read Tool Calling.

Case B: your server has a different wire

If your server differs in its endpoint, its auth header or its request and response shapes, implement the Provider trait and pass it with with_provider. Your calling code (root.chat(...)) does not change.

Below is a complete adapter for a made-up "EchoAI" server with a different wire: endpoint /api/generate, auth header X-Echo-Key, request {model, prompt, settings}, response {output:{text}, meta}. This mirrors the tested example in tests/integration_tests.rs. The example reads the key with the secrecy crate and builds JSON with serde_json, so add secrecy = "0.10" and serde_json = "1" to your [dependencies].

#![allow(unused)]
fn main() {
use minillmlib::{
    Auth, ChatNode, CompletionParameters, CompletionResponse, CostOutcome, GeneratorInfo,
    Message, MessageContent, Provider, StreamChunk, TokenPrice, Usage,
};
use secrecy::ExposeSecret;
use std::sync::Arc;

#[derive(Debug, Clone)]
struct EchoAi;

impl Provider for EchoAi {
    fn endpoint_url(&self, base: &str) -> String {
        format!("{}/api/generate", base.trim_end_matches('/'))
    }

    fn auth_headers(&self, auth: &Auth) -> minillmlib::Result<Vec<(String, String)>> {
        Ok(match auth.secret() {
            Some(s) => vec![("X-Echo-Key".into(), s.expose_secret().to_string())],
            None => vec![],
        })
    }

    fn build_request(
        &self, model: &str, messages: &[Message], params: &CompletionParameters,
        _stream: bool, _include_usage: bool,
    ) -> minillmlib::Result<serde_json::Value> {
        // EchoAI has a field for the output cap and nothing else: any other
        // setting the caller made fails loudly instead of vanishing.
        let only_the_cap = CompletionParameters {
            max_tokens: params.max_tokens,
            ..Default::default()
        };
        if serde_json::to_value(params)? != serde_json::to_value(&only_the_cap)? {
            return Err(minillmlib::MiniLLMError::InvalidParameter(
                "EchoAI takes only max_tokens".into(),
            ));
        }
        // Flatten the conversation into one prompt. Fail loudly on multimodal
        // (this wire is text-only) instead of silently dropping the attachment.
        let mut lines = Vec::new();
        for m in messages {
            if let MessageContent::Parts(parts) = &m.content {
                if parts.iter().any(|p| p.as_text().is_none()) {
                    return Err(minillmlib::MiniLLMError::InvalidParameter(
                        "EchoAI is text-only".into(),
                    ));
                }
            }
            lines.push(format!("{}: {}", m.role.as_str(), m.content.all_text()));
        }
        let mut body = serde_json::json!({
            "model": model,
            "prompt": lines.join("\n"),
        });
        // Send only what the caller set; EchoAI's own default covers the rest.
        if let Some(cap) = params.max_tokens {
            body["settings"] = serde_json::json!({ "max_output_tokens": cap });
        }
        Ok(body)
    }

    fn parse_response(&self, raw: serde_json::Value) -> minillmlib::Result<CompletionResponse> {
        let text = raw["output"]["text"].as_str()
            .ok_or_else(|| minillmlib::MiniLLMError::MalformedResponse(raw.to_string()))?
            .to_string();
        Ok(CompletionResponse {
            id: raw["meta"]["id"].as_str().unwrap_or("").into(),
            model: raw["meta"]["model"].as_str().unwrap_or("").into(),
            content: text,
            finish_reason: raw["stop"].as_str().map(String::from),
            usage: self.parse_usage(&raw),
            tool_calls: None,
            media: Vec::new(),
            raw_response: Some(raw),
        })
    }

    fn parse_usage(&self, raw: &serde_json::Value) -> Option<Usage> {
        let meta = raw.get("meta")?;
        Some(Usage {
            uncached_input_tokens: meta["tokens_in"].as_u64().unwrap_or(0) as u32,
            completion_tokens: meta["tokens_out"].as_u64().unwrap_or(0) as u32,
            ..Default::default()
        })
    }

    fn parse_chunk(&self, _data: &str) -> Option<minillmlib::Result<StreamChunk>> {
        None // non-streaming
    }

    fn emits_stream_usage(&self, _requested: bool) -> bool {
        false // never sends a trailing usage chunk; don't wait for one
    }

    fn cost_of(&self, usage: Usage, price: Option<&TokenPrice>) -> CostOutcome {
        match price {
            Some(p) => CostOutcome::resolved(p.cost_of(&usage), usage),
            None => CostOutcome::unpriced(usage),
        }
    }
}

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::custom("echoai", "https://my.host", "echo-1")
    .with_provider(Arc::new(EchoAi))
    .with_api_key("my-secret")
    .with_token_price(TokenPrice::new(1.0, 5.0));

let answer = ChatNode::root("You are EchoAI.")
    .chat("hello", &gen).await?;
let _ = answer;
Ok(()) }
}

What to override

The trait defaults to the OpenAI dialect, so you override only what differs:

You always write cost_of: it turns token counts into dollars, and has no default. For a server that reports token counts only, copy EchoAI's four lines.

If your server speaks the OpenAI body with a twist, override the hook for the twist instead of build_request:

Telling the library about your models

Cost estimation, the price a call is reported at, and the cap a requires_max_tokens wire fills in all come from the ModelInfo your model_info returns. Fill what you can find and leave the rest None.

ModelLookup gives you lookup.call (the generator's client, address and headers), lookup.model (the id on your wire), lookup.openrouter_name (its OpenRouter catalog id, when the caller set one) and lookup.wanted (whether the caller needs the price, the limits, or both, so you can skip a source that answers only what is not wanted). Make your requests with lookup.call.get_cached(path, timeout, parse). path is joined onto the generator's base URL, and parse gets the status and the body text. Callers anywhere in the program share each request: one goes out per full URL, header set and type parse returns, whichever client makes it. parse runs once per reply (so what it logs is logged once), and a value parsed from a success or a 404 is kept for an hour. Any other reply, or a parse that returns an error, is not kept. A request that gets no reply at all (a timeout, a refused connection) fails the same way for a minute, then is made again. If get_cached returns an Err, the failure is already logged, so just leave that source's fields None.

If your models are in OpenRouter's catalog, the whole job is one call:

fn model_info(&self, lookup: ModelLookup) -> Option<InfoFuture> {
    let model = lookup.openrouter_name.unwrap_or(lookup.model);
    let catalog = minillmlib::provider::catalog_call(lookup.call.client);
    // Groq's price; with `None`, or if Groq does not serve the model, the
    // dearest provider's.
    Some(Box::pin(async move {
        minillmlib::provider::catalog_info(&catalog, &model, Some("groq")).await
    }))
}

If your API has no way to tell, skip model_info: callers give the facts with GeneratorInfo::with_model_info and with_token_price.

Reusing the OpenAI dialect piecewise

If you override a method but still want part of OpenAI's behaviour inside it, call the library's own pieces from minillmlib::provider::openai_wire: the request builder, the response, usage and chunk parsers, and the converters for messages and tools (messages_to_payload, tool_definition_value, ...).

API Reference

The full, auto-generated API reference (every public type, method, and signature) lives on docs.rs:

https://docs.rs/minillmlib

It is generated from the source doc comments and rebuilt automatically when a new version is published to crates.io. This guide covers how to use the library; the docs.rs reference is the exhaustive signature lookup.