Programs with instances

If you want part of one program to run as several separate copies, each with its own WhatsApp number, Slack account, spreadsheet or schedule, you mark what each copy gets for itself: a container of its own (@per_instance), or a value of its own (@instance_filled). Each of those copies is an instance of the program.

What an instance stands for is up to you. One per customer is the common case, but it can as well be one per session, or several per person (a customer with two shops, each with its own bridge). weft only sees the ids: which person owns which instances is your program’s own data, kept in its own database.

An instance is just an id you choose, such as user-42, session-9f3 or a shop number. You hand it to weft each time something happens for that instance: in --instance, in an instance token, or in a header from your own backend. weft keeps no list of instances, so there is nothing to register. Once you wipe everything weft holds for an id (see cleaning up after instances), that id is no longer an instance.

Say you built a WhatsApp assistant and a hundred customers want it. A WhatsApp bridge signs in to one phone: when it first starts, it shows a QR code on its display, and whoever scans that code with their phone ties the bridge to their number. So each customer needs a bridge of their own, and you give each one an instance:

bridge = BaileyBridge {
  @per_instance
}
receive = BaileyReceive { bridge: bridge.bridge }
# a real assistant wires an LLM's answer into `message`
reply = BaileySend {
  bridge: bridge.bridge
  to: receive.from
  message: "hi"
}

The @per_instance line gives each instance its own bridge container, called its copy of bridge. receive and reply read the bridge, so they are per instance too. A message arriving on instance user-42’s phone starts a run for user-42, and reply answers through user-42’s copy.

Bringing one instance up

For customer user-42, from the project’s folder:

1. weft infra start --instance user-42 builds the program if it changed, starts their bridge, and answers once it is running.
2. weft activate --instance user-42 turns on its receive trigger.
3. weft token mint --instance user-42 --expires 7d --display bridge mints an instance token that can read that bridge’s display.
4. The customer adds the token to the weft browser extension, opens the bridge’s display, and scans the QR code with their phone.

If you want your own website to do steps 1 to 3 for a customer, a program can do each of them itself; for how, go and read the instance nodes.

What @per_instance marks

@per_instance goes on its own line inside the braces of an infra node (a node that runs its own container, like the bridge). Anywhere else the compiler refuses it as per-instance-ineligible.

If you work in the graph, you can set and remove the mark from a node’s right-click menu; for how, go and read the graph.

What an instance fills: @instance_filled

If each instance should get its own value for a field (a spreadsheet, a model, a schedule, a Slack account), write @instance_filled where the value would go:

google = GoogleAccess { account: @instance_filled }
read = GoogleSheetsRead {
  account: google.access
  spreadsheet: @instance_filled
  tab: @instance_filled("0")
}
digest = Cron { cron: @instance_filled("0 0 8 * * *") }

A field holds one of four things:

On a connection field (account above), what you write decides whose key pays for an instance’s calls:

Only you can put your key in an instance’s run, by leaving the field unmarked and picking your connection on the install. An instance token can never pick your connection, or the runtime’s shared key: connecting with the shared key through the instance door is refused, and a value saved for an instance’s connection field has to be a connection that instance owns in this program, whoever saves it. If your connection is one made with the runtime’s shared key, the instances it serves spend that key, and each of their cost records shows Platform as the payer (for reading those, go to who paid for a call).

A field with a wire into it already gets its value from the wire, so the compiler refuses @instance_filled there (instance-filled-wired). A group’s, a loop’s or an included file’s own port is refused too (instance-filled-boundary); put the mark on the node inside that reads the value. A key that is not one of the node’s inputs cannot take the mark either (instance-filled-not-an-input).

You do not mark the other nodes: a node that reads an instance’s value or copy, directly or through nodes before it, runs per instance too. Above, read needs Google through the instance’s connection, so it runs in that instance’s runs.

weft checks an instance’s value against the node’s own rules at three moments:

• when the program compiles: a rule that only needs the field to have a value passes, since the value will be given later; a fallback is checked like any written value (@instance_filled("0 3 * * *") on a Cron is refused right away);
• when the value is given: a value the node refuses is refused on the spot, with the node’s own message, and nothing is stored;
• when a run starts: the instance’s values are read once, checked again (the program may have changed since), and carried by the run, so every node of it sees the same values even if one changes mid-run.

If an instance’s value changes and a live trigger of that instance reads it (the trigger’s own field, like digest.cron, or anything its setup goes through, like its connection), weft sets that trigger up again with the new value, and only then saves the change. An event arriving meanwhile waits and goes through afterwards, on the new value. If the new setup fails (the new account refuses the connection, say), the old value and the old trigger stay, and the change fails with the reason. If you change several fields in one go (one apply(), one --set list), each trigger is set up again only once.

Which instance a run is for

Every run is for one instance or for none. Whatever started the run decides, and nothing changes it afterwards:

A gated route is one whose auth input is wired from an auth node such as ApiKeyAuth (weft’s messages call it a route gated by a connection), so the caller must present a key before a run starts. If your own backend calls your program for an instance, this is the route it uses. It is also the only place weft honours the Weft-Instance header: on an open route anybody could claim any instance, so weft refuses the header there and says why. A browser presents an instance token instead.

If you call a route and want the run to be for an instance, call it at one of its route addresses: /connect/local/<project id>/<path> on the install, the project’s own port on your machine, or an API domain (for where those come from, go and read answering on a URL). A webhook also answers at a plain address without /connect, but a run started there is never for an instance, and weft refuses an instance header sent there.

What is checked before a run starts

weft refuses a run before it starts, and says why, when:

• it reaches something per instance and is for no instance;
• it reaches a field the node needs and the instance has not filled;
• it reaches a value the instance has that the node’s rules refuse;
• it reads an instance’s copy that is not running.

If an event reaches an instance’s trigger before what it needs is filled, weft holds it until the instance’s values change (from its settings page, SetInstanceValues or ctx.values()), then runs it; activating the trigger again also tries it. Nothing retries it on a timer, since only a new value can close the gap. weft status shows under the trigger how many events wait and which field they wait for, and so do ListInstances and ctx.instances().list(), so your website can tell the customer what is missing. An event that reaches the trigger while the instance’s copy is down is tried again on a timer (sooner at first, then every five minutes) until the copy is up. A call to a route gets the refusal instead.

An instance’s own infra

From the terminal, the verbs that start, stop, upgrade or remove infra take --instance:

weft infra start --instance user-42
weft infra stop --instance user-42                                 # the container goes, its disk stays
weft infra terminate --infra bridge --instance user-42 --yes       # the container goes, and its disk unless the node keeps it

--all-instances acts on every instance’s copies, and --all on the shared infra and every instance’s. Taking an instance’s copy down takes down only that instance’s triggers that depend on it, and reaches only that instance’s runs. Taking shared infra down reaches further: every trigger that depends on it goes down, every instance’s included, since none of them can run without it, and they all come back with it.

weft infra status lists each instance’s copy on its own line. weft status lists them under instance infra:, and under infra: it lists every infra node your program declares: a @per_instance one says how many instances have a copy, and one that was never started says not started.

Every place that reports a copy’s state gives the same answer: weft status, weft infra status, ctx.infra(..).status() and the InstanceInfraStatus node. A start or a stop on its way reads provisioning or stopping from the moment it is asked for, before the container itself moves.

From your program, ctx.infra names the node and the instance:

ctx.infra("bridge").instance("user-42").start().await?;
let copy = ctx.infra("bridge").instance("user-42").status().await?; // None: no copy
let every = ctx.infra("bridge").copies().await?;                     // shared + each instance's

weft never starts an instance’s copy on its own: you do, from the terminal as above, or your program does.

start() returns once the copy is running, so the next line can activate the triggers that read it. A container can take minutes to come up, and the run waits that long without holding a worker: between looks at the copy it parks, the way a Wait node does. If the copy fails to come up, start() fails with the reason. If somebody stops or terminates the copy while start() waits (a customer pausing right after resuming, say), start() fails too, rather than starting it again behind their back.

An instance’s own triggers

A trigger that reads a per-instance node exists once per instance, and you turn each instance’s version of it on and off separately. weft activate with no flags turns on every shared trigger and skips these, because it cannot know which instances you mean:

weft activate --instance user-42
weft activate --trigger receive --instance user-42
weft deactivate --instance user-42

weft status lists every trigger under triggers:: one that exists once per instance says how many instances have it on (receive: once per instance, 3 of 4 on), then each instance’s state on a line of its own. If you are scripting this, weft status --json has the same list under triggers, each entry with its node, perInstance, and either shared (the program’s own trigger) or copies (one per instance), and the infra it reads under dependsOn. Each copy has its instance (absent on shared), its state (off, turning_on, on or behind), downWith (health or infra) when the health check or an infra command took it down, and heldBack (the reason) when it should be back but another build keeps it down. While its queued events wait on something, it also has waiting ({ fires, on, reason }: how many fires are held, what they wait on, one of room, trigger, instance, infra, picks or failure, and why). weft instances gives one line per instance: how many of its triggers are on, how many of its infra copies are up, and when it last ran.

Every trigger verb acts on the program’s own triggers unless you name instances. A plain weft deactivate turns off the program’s own triggers, leaves the instances’ on, and tells you how many instances still have triggers on. The same goes for weft resync: a plain one moves only the program’s own triggers. Since one build answers all of a project’s routes, a plain resync onto new code is refused while instances’ triggers are on the older build, and the refusal names weft resync --all.

weft deactivate --all-instances     # every instance's, not the program's own
weft resync --all                   # the program's own and every instance's

If a deploy should reach every customer, run weft infra upgrade --all and then weft resync --all. That is what the workflow from weft ci add does.

From your program:

ctx.triggers().instance("user-42").activate(ReleaseHeld::Run).await?;

If you want to turn one off from your program, the call takes a few choices; for those, go and read taking something down from a run.

If an instance’s trigger reads its copy and that copy is not running, activating the trigger is refused, with the command that starts the copy.

If an instance’s trigger reads a value it fills (digest.cron, say), activating it before one was given is refused, naming the field, unless the field has a fallback. If you want an instance’s triggers on the moment it is created, give such fields a fallback: the trigger runs on the fallback until the instance’s own value is saved, and then weft restarts it with that one. Saving a value never turns on a trigger that is off.

Where an instance’s values are filled in

Each instance’s values are filled in on its settings page. It lists every field your program marks @instance_filled, grouped by node, each with the control the editor shows you: a connection picker for a connection, a searchable list for a list field (spreadsheets, say, read through the instance’s own connection), and a plain input for anything else.

If you want to show it to your customers, there are two places:

• the browser extension’s Your settings page, for somebody who has added an instance token to the extension;
• the InstanceSettings component, for your own website. weft connect-lib copies the library into your frontend (front/src/lib/weft-connect unless --into names another folder); for how to mount it, go and read the README it copies along.

The picker never offers an instance the shared key, because it spends your credits.

If you want to test that page from the terminal, two commands do what it does (each mints its own instance token that lasts one hour and revokes it when done, so the instance’s own tokens are untouched):

• weft connect --instance user-42 --node openrouter --door own --set-env key=OPENROUTER_KEY connects an account as that instance’s from a key in your environment, and --grant <id> picks one it already has; --list and --disconnect work the same way;
• weft instance-values --instance user-42 lists its fields and values, and --set read.spreadsheet=1AbC --clear digest.cron changes them. A value that reads as JSON (5, true, {"id": "..."}) is passed as JSON, so write a value made only of digits as 'read.tab="12"' to keep it text.

If your program should read or change an instance’s values, ctx.values() does it, with the same checks, and sets affected triggers up again:

let values = ctx.values().instance("user-42").get().await?; // by node, then field
let rearmed = ctx.values()
    .instance("user-42")
    .set("read", "spreadsheet", json!("1AbC..."))
    .set("google", "account", json!({ "id": connection_id }))
    .clear("digest", "cron")
    .apply()
    .await?; // returns the triggers it set up again
let theirs = ctx.connections().instance("user-42").list().await?;
ctx.connections().instance("user-42").forget().await?;    // the values naming them go too

Instance tokens

An instance token acts inside one instance of one program, and nothing else. With it, a browser or the extension can:

• start that instance’s runs, through the program’s routes, in the Weft-Instance-Token header;
• answer that instance’s waiting forms;
• read the displays of that instance’s copies, such as its bridge’s QR code;
• fill in that instance’s values, its connections among them.

An instance token always expires. If you mint one from the terminal (step 3 above), it reads a display only when you pass --display <node> for that node, or --displays for all of them. A token your program mints reads all of the instance’s displays and lives at most a year. If a customer just logged in to your site, this is how your program hands their browser a token for one of their instances (which instances are theirs is up to your own database):

let minted = ctx.tokens().mint_for_instance("user-42", Duration::from_secs(7 * 86400)).await?;
// minted.token goes to the browser, once
ctx.tokens().instance("user-42").revoke().await?;

The token comes out once, because weft keeps only a hash of it. So when a run is replayed after a restart, weft cannot hand back the token it minted the first time: it mints a fresh value for the same token, and the old value stops working. If your program already handed the first value to a browser, that browser needs the new one.

An instance’s files

StorageScope::instance() holds the files of the run’s own instance, and StorageScope::instance_of(id) those of any instance of this program:

ctx.storage(StorageScope::instance())
ctx.storage(StorageScope::instance_of(InstanceId::new("user-42")?))

The files stay until your program deletes them, you wipe the instance (see cleaning up after instances), or you weft rm the project. Using instance() in a run for no instance is an error. Another program never reaches these files, even for the same instance id.

Who paid for a call

Every cost record says whose credential paid: the runtime’s own key (Platform, when a node uses the shared key), yours (Author), or the instance’s own (Instance). ctx.costs() reads them, and you can narrow it by instance, node, service, run, who paid (paid_by) or since (unix seconds):

let spent = ctx.costs().instance("user-42").service("openrouter").since(month_start).list().await?;

Taking something down from a run

A program’s stop, terminate, upgrade and deactivate calls each take a TakeDownChoice, the same three answers weft deactivate takes as flags (for what each does, go and read taking triggers down). Every answer you leave out takes its default:

let quiet = TakeDownChoice { door: Some(Door::Hide), running: Some(Running::Cancel), ..Default::default() };
ctx.trigger("receive").instance("user-42").deactivate(quiet, StopSelf::Keep).await?;
ctx.infra("bridge").instance("user-42").stop(TakeDownChoice::default(), StopSelf::Keep).await?;

Taking down an instance’s copy touches only that instance’s triggers and runs, and taking down an instance’s trigger touches only the runs that trigger started. A deactivate with running left at wait answers once the triggers are down: the runs they started finish on their own, and the call never waits on them from inside this run.

If the run making the call is one of the runs the call would take down, the last argument decides what happens to it:

• StopSelf::Keep: it keeps going.
• StopSelf::Include: it is cancelled with the rest, once the take-down is queued; the call never returns.

If a run is replayed after a restart, a copy it terminated is never terminated twice (for why, go and read your project beyond this run).

Cleaning up after instances

If you want to see which instances there are, weft instances lists them, and in a program ctx.instances().list() (or the ListInstances node) gives one entry per instance weft holds anything for: how many values, connections and live tokens it has, its infra copies, and its triggers. It never hands back a value or a secret.

If you want to see an instance’s runs, weft runs --instance user-42 lists them. weft clean takes the same filters (instance, status, the node that started the run, tag), and so does a program’s ctx.runs():

weft clean --project <id> --instance user-42 --yes

let week = Duration::from_secs(7 * 86400);
ctx.runs().instance("user-42").status("completed").older_than(week).clean(StillGoing::Leave, StopSelf::Keep).await?;

If a program wants to read its runs rather than delete them, the same filters end in .list(limit) (the newest limit runs, 1 to 200, with total, how many match in all) or .count(). The ListRuns and CountRuns nodes do the same from the graph.

let failed_today = ctx.runs().instance("user-42").status("failed").count().await?;

clean never deletes a run that is still going. With --cancel-running (or StillGoing::Cancel) it cancels it, and the run’s rows go with the next clean; without it (StillGoing::Leave) the run is left to finish.

weft has no “delete an instance” command. The WipeInstance node removes everything attached to one: its triggers, the copies of the infra nodes you name in its infra input, its values, its connections, its tokens, its files, and its runs. If a wipe stops part way, run it again to finish it.

The instance nodes

If you want a graph to bring an instance up and remove it without writing code, the instances package of the standard catalog has a node for each step:

If you leave instance unwired on StartInfra, StopInfra, TerminateInfra, ActivateTriggers or DeactivateTriggers, it acts on the program’s own copies instead: the shared copy of an infra node not marked @per_instance, or the triggers every instance shares. StopInfra, TerminateInfra and DeactivateTriggers take the same answers as weft deactivate as inputs (door, doorForMinutes, held, running), each defaulting the same way, and ActivateTriggers takes held.

If your backend should create instances, put MintInstanceToken, StartInfra and ActivateTriggers behind a gated route, in that order. If you want the request to answer at once rather than wait minutes for the copy, turn waitUntilRunning off on StartInfra and reply with the token after it: the start is accepted by then, so a list read right after shows the copy as provisioning. A reply placed before the start leaves nothing for that list to show yet. If one of the instance’s triggers reads a value with no fallback, activating it at creation is refused, so leave ActivateTriggers for a second call your website makes once the settings are saved. The same route can remove an instance with WipeInstance: weft cannot tell when an instance is no longer needed, so your backend says so.

When something goes wrong