The journal

Every recorded run writes down what it did, event by event. For when each write happens, go and read when a run waits for its writes. That record is what the graph shows you, what weft events prints, and what a worker reads to pick up a durable run that was interrupted, or a run that was waiting.

weft events 9b81d0a2
weft events 9b81d0a2 --node classify --full
weft logs 9b81d0a2

Nothing edits an event once it is written. A run is deleted with its record once it has been kept long enough (go and read how long a run is kept), or sooner by weft clean, weft prune or weft rm.

What is in it

Each row holds one run’s events from one write, in the order they happened: which run, what kinds, when, and the events themselves. A worker sends the rows of several runs in one write. A run’s history is its rows, read in order.

The run’s life. It started, with the project, the entry node and a fingerprint of the exact program shape, so a resume runs against what it suspended on rather than whatever you have edited since. Then it completed, failed, or was cancelled, with the reason in words and in a form the inspector can read.

Each node’s life. Given its inputs (kicked), started, and then completed, failed, skipped, suspended, resumed or cancelled. A skip carries its reason in plain words.

Where a node is. Each time a node saves the state it keeps (state.save(), the end of a once or a wait, or a pause), a NodeStateSaved row holds it whole, with what weft keeps beside it: how far the node read each bus and stream, how many items it yielded, and the numbers its next wait and its next program call will take. The latest row is what the node gets back when it runs again after a pause or a crash. Each wait it registers carries its number in the node (wait_index), which is how a body that runs again finds the wait it already made.

Values on wires. PortEmitted is the only event that carries a value, and it carries it once. When a run is rebuilt, weft puts the value on every outgoing wire itself, which is why a replay lands on exactly the same picture as the live run did.

PortClosed is written only when a node closes a port on purpose. The ports that close because a body returned are worked out again on replay.

Loops, streams and buses get their own events: an iteration launching, a gather assembling, a bus being made, a participant joining, a window of messages. The row for a bus being made (BusCreated) holds what a worker needs to open it again after a pause: its id, the port its marker went out on, its options.

A live caller’s exchange does too: the caller arriving, a window of what the caller and the program said, an error, the caller leaving. A window also names the program’s messages the caller never received because it read too slowly, and the row saying the caller left names the ones still waiting to go out when the exchange ended. For the settings that let a slow caller’s messages go, go and read slow callers and messages that cannot be made.

What a call cost, from the meter, as it happens. A node can never write one of those.

Reading it back

When weft reads a run back (the fold), it takes the rows and the program and works out where the run had got to.

The journal holds only the facts learned from outside, and the fold recomputes everything else with the same functions the live engine ran, so a replay rebuilds exactly the picture the live run held. A replay shows the times the journal wrote down, so you see when each thing really happened.

weft reads a run back a page of rows at a time. It forgets a node once the node has finished (keeping each node’s last output, which the run’s outputs show), and a loop’s iteration once nothing in it is still running. So reading a run back holds one page of its rows and what is still running, however long the run is. The one thing that grows is what a loop gathers: the loop hands the whole list on when it ends, so it holds it until then. A run started from another one (weft run --seed) reads that one the same way, and keeps only the nodes it reuses from it.

If you follow a run (the editor, weft run, weft wait), you get its history and then what happens live, on one stream, each event once. If you fall behind or lose the connection, you carry on from the last event you got and nothing is missed. The events say where in the journal a value is kept instead of carrying it, so a value the run passed to ten nodes still travels once: the editor reads a node’s values when you open its inspector, and weft events and weft run print them written in.

When a row cannot be read

A row the fold cannot apply is a hole. It is logged, listed on the run, and the inspector shows how many there are.

In a run that has ended, a hole only spoils the replay view. In a run that has to resume, a hole stops the resume with an error, because a run rebuilt from part of its record could run nodes that already ran.

When a worker dies mid-firing

A node’s completion is written after it finishes, so one whose worker died before that has no completion on record. For what weft does with such a node, in a fast run and in a durable one, go and read surviving a restart.

When a run waits for its writes

A worker gathers what its runs did and writes it a couple of milliseconds later, or at once when a run is waiting on it. Each run’s records always arrive in order.

A fast run (the default) waits for its writes only where something else is about to act on its record: when it pauses, when an answer it was waiting for arrives, when its worker is stopping and hands it to another one, and before it asks weft for something on its behalf (a connection or an endpoint it does not already hold, anything on its stored files, a tag through ctx.tag_run, a stop through ctx.stop_tagged, a call on its own program such as starting its infra). If a node calls a paid service and that call reports a cost, the cost is written in the background, and the run hands over its ending only once that cost is written. Once it has handed its ending over, the run is gone from the worker’s memory straight away. The ending is written after every event the run handed over before it, so a run is never reported finished before the rest of its record.

A durable run also waits before each node that is not pure starts, before an answer leaves for its caller, and for its ending. One fired by an event also waits for its start to be written before the event counts as delivered. When the answer is the last thing the run does, its ending goes in the same write, so the answer costs no extra wait. A durable route that reshapes its input with pure nodes and answers with Reply (pure too) waits once, for the answer and the ending together. For choosing between the two, go and read how a run is kept.

If the database falls behind and 64 MiB of events are already waiting on a worker, the next run with more to hand over waits until there is space, so nothing is dropped. For what that does to new calls, go and read when a worker is full.

When a write of the record fails

If the broker refuses a write, the runs whose rows were in it end as failed.

If a write gets no answer, the worker sends it again for a minute. That is safe, because the record takes a batch sent twice only once. After the minute it gives up on that write, and then:

• a fast run ends as failed;
• a durable run is handed back, and another worker carries it on from the rows that did land.

Either way the run writes nothing after the write that failed. A record with a hole in it would rebuild a different run: a node whose start was lost but whose values landed would run again and spend twice. So what landed is always the run’s first rows, in order, which is the record a worker that died leaves behind, and a durable run carries on from it the same way. A node whose start landed and whose end did not is failed rather than run again, unless it saved a state (for that, go and read when the worker dies mid-run).

How long a run is kept

An ended run is kept for a week, and then deleted with its record, its logs, its costs, its search entry, its tags and the files it stored in run scope. A run that has not ended (running, parked, waiting for a worker) is never deleted, however old. If you want another length:

• for every run of a project, set it in weft.toml:

  [runs]
  keep_for = "30d"
  

• for the runs of one trigger, set its keepRunsFor input (12h, 30d, forever);

• for one run you start by hand, weft run --keep-for 2h.

A trigger’s setting beats the project’s, and --keep-for beats both. A run’s keep time is fixed when it starts, so a change only reaches runs that start afterwards. If you change a trigger’s keepRunsFor or the project’s keep_for, the trigger’s own runs pick up the change once you run weft resync, and a run you fire with weft run --fire once you run weft bake.

Clearing it

weft clean                    # ended runs that started more than 30 days ago
weft clean 9b81d0a2           # one run
weft clean --project <id>     # that project's whole history
weft clean --all              # everything

Removing a project with weft rm erases its runs with it.

A past run that shows nothing

The journal names the program each run used, by hash. If the dispatcher no longer holds that program, the run’s rows are still there but nothing can be drawn against them.

If your files are the same as when the run ran, weft build makes it readable again: it registers the program under its hash, and that is the hash the run names.