tellus, episode 4: terminating a tree, bottom-up

, , ,

In the previous episode we ended on a detail I had leaned on without explaining: the terminated signal is the last thing a terminating actor does. In this episode we look at why it has to be last, and at the sequence that gives the signal its meaning. It is the final episode on the core of tellus.

The claim we need to justify is this one. Receiving a terminated signal does not only mean that the actor has stopped taking messages. It means the actor and every one of its descendants are gone, destructors and all.

The sequence

Let’s look at what happens when an actor stops, whether by returning Control::Stop, by failing under the Stop strategy or because its parent is stopping. In all three cases we get the same sequence:

  1. The state is dropped, so its destructors run.
  2. The mailbox is drained and disconnected, and the destructors of the queued messages run.
  3. The actor deregisters itself from everything it still watches.
  4. The children are stopped, recursively, and the actor waits until every one of them has terminated.
  5. The actor value is dropped, so its destructors run.
  6. The watchers are signaled.

Each position in that list is a decision, and four of them are worth walking through, so let’s take them in turn.

Dropping the state

Notice that we drop the state before we stop the children, not after. It is consumed by the receive which decided to stop, or dropped right where the parent’s stop signal arrives, and either way that happens while the children are still running.

We should spell out the consequence, because it can surprise us: the destructors of the state run while the children may still be working. Children never see the state, of course; whatever a child holds we gave it, in the actor value we passed to spawn or in a message. So this only bites when what the child holds does not keep the resource alive: a path into a TempDir we own, or a CancellationToken whose DropGuard sits in our state. A cloneable handle like a connection pool is unaffected, since the child’s clone keeps the pool open whenever we drop ours. If we do hold a guard whose drop has to wait for the whole subtree, we put it in the actor value, which is dropped in step 5, after the last descendant has terminated.

This is the same split we saw in episode 2, seen from its other side. There the actor value was what a restart keeps and the state was what a restart rebuilds. Here the actor value is what outlives the subtree and the state is what does not. One split, two payoffs, and in both cases we ask the same question: does this belong to the actor, or to its current state?

The barrier

Step 4 is the interesting one, because “wait until every descendant has terminated” is exactly the kind of thing we would usually implement with a counter, a registry or an acknowledgement protocol. Here we need none of them.

Every actor owns a Tokio watch channel. Its children hold receiver clones of it, and every child task selects on that receiver. Let’s look at the receiving side first:

1
2
3
4
5
6
7
8
9
10
11
12
13
let incoming = select! {
    biased;

    _ = &mut stopped_by_parent => {
        debug!(%actor_id, "stopping, because parent stopped this actor");
        drop_containing_panic(actor_id, STATE_FAILED_TO_DROP, state);
        break 'run;
    }

    incoming = mailbox.recv() => {
        incoming.expect("self_ref keeps a mailbox handle alive")
    }
};

Notice the biased, which makes the stop branch win against a message arriving at the same moment, and the drop_containing_panic, which is step 1 of our sequence happening right here. Now the sending side, which is all it takes to stop the children:

1
2
3
4
5
6
7
8
9
async fn stop_children(&mut self) {
    let (next_stopping_tx, next_stopping_rx) = watch::channel(());

    let stopping_tx = mem::replace(&mut self.stopping_tx, next_stopping_tx);
    let _ = stopping_tx.send(());
    self.stopping_rx = next_stopping_rx;

    stopping_tx.closed().await;
}

We send the stop signal, then await closed(), which resolves once every receiver has been dropped. A child drops its receiver clone only when its task ends, and its task ends only after it has run this very same sequence for its own children. So what we get from the channel closing is a recursive completion proof: no descendant of any depth is still alive. Notice that we did not build that barrier, we inherited it from ownership: the channel closes because the receivers are dropped, and they are dropped because the tasks holding them have ended.

Why the fresh channel swapped in at the top? Not just as bookkeeping for restarts. The context holds a receiver clone of its own, so unless we install the next generation’s receiver first, closed() would be waiting on the actor itself and never resolve. That we also end up with a working channel for a new generation of children is what makes the same function serve restarts, where the actor stops the current generation and then keeps running with a new one.

Draining early, signaling late

Steps 2 and 6 sit at opposite ends for opposite reasons.

We drain and disconnect the mailbox right at the start, before the children are even told to stop. That way a send is rejected rather than queued into a mailbox nobody will ever read, and it is rejected from that moment on, not only once the whole subtree is gone. What the sender makes of the rejection differs: a tell reports nothing to its caller, so the message is dropped and logged as a dead letter, while an ask gets AskError::ActorTerminated back instead of waiting out its timeout.

Draining also runs the destructors of the messages which were still queued, and that matters more than it sounds: a queued request carries a ReplyTo, and dropping it resolves the corresponding ask as NoReply instead of leaving it pending until its timeout. A send racing with the drain can still slip past it, and such a message is retained until its last sender is dropped, so this detection is best-effort, which is why we give every ask a timeout anyway.

The watchers, in contrast, are signaled at the very end, after the actor value has been dropped. This is what makes the guarantee from the previous episode worth having. If we sent the signal when the actor stopped receiving, it would prove only that: no more messages. Sent last, it proves that the state is gone, every descendant has terminated and the actor’s own destructors have run.

Awaiting the actor system

So is ActorSystem::terminated special framework machinery? It is not. spawn_root spawns the root actor through exactly the same code path as any child, and then registers an ordinary watcher in the root’s watcher registry. That watcher’s sink resolves the oneshot we await in terminated(), and it also owns the sender of the root’s stop channel, which is what keeps the root running until its own termination has signaled its watchers.

So when we await an actor system, we are doing a death watch like any other, and it inherits the meaning we established above: it resolves exactly when the entire tree is gone. Registration is race-free the same way too, since a root which has already terminated fails the registration and the oneshot is resolved right away.

One consequence is worth spelling out, because it catches people: dropping the ActorSystem does not stop anything. The root stops on its own terms; dropping merely forfeits our ability to await it.

When destructors panic

All of this rests on destructors running, so what happens when one panics? Every drop path the framework controls is contained: the state, the queued messages and the actor value are all dropped under catch_unwind, so the panic is logged and termination completes, watchers included. If a panicking destructor could abandon the sequence, one bad Drop would leave us with a permanently stuck tree.

One case is out of reach, and it is a Rust one rather than a tellus one. If init or receive panics and, while that panic unwinds, a destructor of a value still alive in the frame panics too, Rust aborts the process. That happens below catch_unwind and hence below supervision, so there is nothing tellus can do about it. Well, we just have to keep our destructors panic-free.

The end of the core

Over four episodes we have seen that an actor is a state machine, that its mailbox is one FIFO queue carrying both messages and signals, and that its termination is bottom-up, with the terminated signal sent last so it proves something worth knowing. The rest of tellus is built from those three ideas.

What comes next is not core but extension, and both are feature gated so tellus stays purely local and free of their dependencies by default: persistence, in the shape of event sourced actors, and remoting, where serializable actor refs let actors on different nodes message and watch each other through the same API. The second one will be an interesting post to write, because a network is precisely the thing that does not give us one FIFO queue.

And of course the code is on GitHub: tellus itself, its examples, and docs/actors.md, which explains the core top-down with links into the implementation. Thank you for staying around until here, and let me know if you have any questions.

Disclosure: the prose of this post was drafted with Claude Code and revised by me.