Tool Calling

Tools are normalized intent, like every other parameter: you declare ToolDefinitions and a ToolChoice once, and each provider emits its own wire shape. The same code drives OpenRouter, OpenAI, any OpenAI-compatible server, and native Anthropic.

The loop

#![allow(unused)]
fn main() {
use minillmlib::{
    ChatNode, CompletionParameters, GeneratorInfo, NodeCompletionParameters,
    ToolChoice, ToolDefinition,
};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("anthropic/claude-sonnet-4.5");

let params = NodeCompletionParameters::new().with_params(
    CompletionParameters::new()
        .with_tool(ToolDefinition::new(
            "get_weather",
            "Get the current weather for a city",
            serde_json::json!({
                "type": "object",
                "properties": { "city": { "type": "string" } },
                "required": ["city"],
            }),
        ))
        .with_tool_choice(ToolChoice::Auto),
);

let node = ChatNode::root("You are helpful.")
    .add_user("What's the weather in Paris?")
    .complete(&gen, Some(&params))
    .await?;

// The model called a tool: run it and answer each call, then complete again.
if let Some(calls) = node.tool_calls() {
    let mut current = node.clone();
    for call in &calls {
        let args = call.arguments_json()?;           // parsed arguments, fails loudly
        let result = format!("15 degrees in {}", args["city"]);
        current = current.add_tool_result(&call.id, result);
    }
    let answer = current.complete(&gen, Some(&params)).await?;
    println!("{}", answer.text().unwrap_or(""));
}
Ok(()) }
}

Notes:

• Keep the same tools in the follow-up request. Providers require the tool definitions to still be present when you send back the results.
• Parallel calls: the model may return several ToolCalls in one turn; add one add_tool_result per call (in any order). The Anthropic provider packs consecutive results into the single user turn its wire requires. Forbid parallelism with .with_parallel_tool_calls(false).
• Arguments are raw JSON text (ToolCall::arguments), exactly as the model produced them; arguments_json() parses them and fails loudly on invalid JSON instead of silently repairing.
• Forcing a call: ToolChoice::Required (any tool) or ToolChoice::Tool("get_weather".into()) (that one).
• Strict schemas: ToolDefinition::with_strict(true) asks the provider to guarantee the arguments match your schema (OpenAI structured outputs, Anthropic strict tool use).
• Streaming works too: tool-call fragments are accumulated across chunks and the final CompletionResponse::tool_calls (and the node) carry the assembled calls.

Streaming a tool call as it is generated

You don't have to wait for the model to finish a call before acting on it. The streaming chunks expose typed ToolCallDelta fragments with the same timing on both wires: the first fragment carries the call's name (and id), then each later fragment carries a piece of the raw JSON argument text, in order. That lets you start the tool the moment the model names it and pipe the argument bytes in while the model is still generating them.

#![allow(unused)]
fn main() {
use minillmlib::{
    ChatNode, CompletionParameters, GeneratorInfo, NodeCompletionParameters,
    ToolChoice, ToolDefinition,
};

async fn run() -> minillmlib::Result<()> {
let gen = GeneratorInfo::openrouter("anthropic/claude-sonnet-4.5");

let params = NodeCompletionParameters::new().with_params(
    CompletionParameters::new()
        .with_tool(
            ToolDefinition::new(
                "run_python",
                "Execute Python code",
                serde_json::json!({
                    "type": "object",
                    "properties": { "code": { "type": "string" } },
                    "required": ["code"],
                }),
            )
            .with_strict(true),
        )
        .with_tool_choice(ToolChoice::Tool("run_python".into())),
);

let root = ChatNode::root("You are helpful.");
let user = root.add_user("Compute the 100th Fibonacci number.");
let mut stream = user.complete_streaming(&gen, Some(&params)).await?;

let mut tool_started = false;
while let Some(chunk) = stream.next_chunk().await {
    let chunk = chunk?;
    if let Some(deltas) = &chunk.tool_calls {
        for delta in deltas {
            // First fragment carries the name: start the tool NOW
            // (e.g. spawn the interpreter process here).
            if let Some(name) = &delta.name {
                println!(">> model is calling {name}, starting process");
                tool_started = true;
            }
            // Later fragments: raw JSON argument text, in order.
            if let Some(frag) = &delta.arguments_fragment {
                if tool_started {
                    // CAVEAT: this is escaped JSON source, e.g.
                    // {"code": "print(\"hi\")... (see the note below).
                    print!("{frag}");
                }
            }
        }
    }
}

// The stream assembled the complete calls in parallel: append the assistant
// node and finish the normal loop (add_tool_result + complete again).
let response = stream.collect().await?;
let node = user.append_response(&response);
if let Some(calls) = node.tool_calls() {
    let result = node.add_tool_result(&calls[0].id, "354224848179261915075");
    let answer = result.complete(&gen, Some(&params)).await?;
    println!("{}", answer.text().unwrap_or(""));
}
Ok(()) }
}

Notes:

• The fragments are JSON source text, not your payload. For a tool whose input is one string field (like code above), the bytes arrive escaped and wrapped in the object syntax ({"code": "print(\"hi ...). Put an ArgumentStream between the fragments and the tool to stream the DECODED content instead (see the next section).
• Parallel calls: delta.index disambiguates concurrent calls; key your spawned tools by it. Forcing a single call with ToolChoice::Tool(..) (and .with_parallel_tool_calls(false)) sidesteps this.
• Key order: models may emit argument keys in any order, so with several fields your payload field can arrive last. strict: true plus a one-property schema keeps the stream predictable.

For the complete pattern (a multi-turn agent loop mixing a streaming tool and a buffered tool, forwarding all prose live), see examples/agent_loop.rs: cargo run --example agent_loop. The key mental model: a tool call always ends the model's turn; "the model continues after the tool" is always a new API request that your loop makes after add_tool_result, and the consumer of your stream never sees the seams.

Streaming decoded arguments (ArgumentStream)

ArgumentStream decodes a call's raw argument fragments field by field, live. Every field is the same kind of object: take a FieldHandle for it and choose PER FIELD how to consume it:

• handle.wait().await: the complete parsed value, once the field ends.
• handle.delta().await: the field's DECODED text chunk by chunk as the model generates it (\n a real newline, \" a quote, \uXXXX the character): type code into an editor in real time, pipe into a process's stdin.

Fields nobody took a handle for are parsed into args.fields() as they complete, so a fully non-streaming consumer still gets everything extracted at the end. Fragments may split at any position (mid-escape included); the output never changes.

#![allow(unused)]
fn main() {
use minillmlib::ArgumentStream;

async fn run() -> minillmlib::Result<()> {
let mut args = ArgumentStream::lenient();
let path = args.field("path");            // consume as a whole value
let mut content = args.field("content");  // consume as a live stream

// The tool runs concurrently with the wire:
let tool = tokio::spawn(async move {
    let path = path.wait().await?;                    // complete, parsed
    let mut session = open_editor(path.as_str().unwrap());
    while let Some(text) = content.delta().await {    // decoded, live
        session.type_text(&text);
    }
    Ok::<_, minillmlib::MiniLLMError>(session.close())
});

// The driver feeds the fragments from the streaming loop:
let provider_fragments: Vec<String> = vec![];
for fragment in provider_fragments {
    args.feed(&fragment)?;
}
args.finish()?;      // resolves holdbacks, closes every handle
tool.await.unwrap()?;
Ok(()) }
fn open_editor(_p: &str) -> Session { Session }
struct Session; impl Session { fn type_text(&mut self, _t: &str) {} fn close(self) {} }
}

Two modes:

• ArgumentStream::strict() (default choice): the arguments must be well-formed JSON; a bad escape, unescaped control character, or unterminated string/object fails loudly with the raw text in the error.
• ArgumentStream::lenient(): for models sloppy at escaping, applied to EVERY top-level string value. The rule is deterministic because the legitimate ways a string can end are known: an unescaped " closes the string only when followed by , "key": (the next field's declaration; whitespace optional everywhere, and the key is a full JSON string, spaces included) or by } at the true end of the call (the provider signals that end explicitly). Every other " is literal content, a raw newline is itself, \ before a non-escape character is a literal backslash, and a model that just stops (forgot the closing " or }) still delivers the full content, never silently dropped. On finish(), lenient mode also runs the raw arguments through the crate's JSON repair and fills anything the incremental parse missed into fields(). The one documented misfire: content that literally contains ", "somekey": reads as a field boundary; that ambiguity is unresolvable on the wire. Numbers, booleans, and nested objects must be well-formed in both modes (they have no end anchor).

Any number of fields can be streamed (a patch(old_code, new_code) tool works fine); a handle whose field never arrives resolves as a loud error at finish(). examples/agent_loop.rs uses it for its streaming tool.

For the fully non-streaming path, the assembled call offers the same robustness: ToolCall::arguments_json() parses strictly, and ToolCall::arguments_json_repaired() runs the arguments through the crate's JSON repair first (trailing commas, unclosed braces, single quotes).

Custom wire shapes

An OpenAI-envelope server whose tool shape deviates only needs to override the two tool hooks on Provider (openai_tools_value, openai_tool_choice_value); a fully different wire translates params.tools / params.tool_choice / message.tool_calls itself in its build_request. See Custom Providers.