The claim we need to justify is this one. Receiving a terminated signal does not only mean that the actor has stopped taking messages. It means the actor and every one of its descendants are gone, destructors and all.

The sequence

Let’s look at what happens when an actor stops, whether by returning Control::Stop, by failing under the Stop strategy or because its parent is stopping. In all three cases we get the same sequence:

1. The state is dropped, so its destructors run.
2. The mailbox is drained and disconnected, and the destructors of the queued messages run.
3. The actor deregisters itself from everything it still watches.
4. The children are stopped, recursively, and the actor waits until every one of them has terminated.
5. The actor value is dropped, so its destructors run.
6. The watchers are signaled.

Each position in that list is a decision, and four of them are worth walking through, so let’s take them in turn.

Dropping the state

Notice that we drop the state before we stop the children, not after. It is consumed by the receive which decided to stop, or dropped right where the parent’s stop signal arrives, and either way that happens while the children are still running.

We should spell out the consequence, because it can surprise us: the destructors of the state run while the children may still be working. Children never see the state, of course; whatever a child holds we gave it, in the actor value we passed to spawn or in a message. So this only bites when what the child holds does not keep the resource alive: a path into a TempDir we own, or a CancellationToken whose DropGuard sits in our state. A cloneable handle like a connection pool is unaffected, since the child’s clone keeps the pool open whenever we drop ours. If we do hold a guard whose drop has to wait for the whole subtree, we put it in the actor value, which is dropped in step 5, after the last descendant has terminated.

This is the same split we saw in episode 2, seen from its other side. There the actor value was what a restart keeps and the state was what a restart rebuilds. Here the actor value is what outlives the subtree and the state is what does not. One split, two payoffs, and in both cases we ask the same question: does this belong to the actor, or to its current state?

The barrier

Step 4 is the interesting one, because “wait until every descendant has terminated” is exactly the kind of thing we would usually implement with a counter, a registry or an acknowledgement protocol. Here we need none of them.

Every actor owns a Tokio watch channel. Its children hold receiver clones of it, and every child task selects on that receiver. Let’s look at the receiving side first:

1
2
3
4
5
6
7
8
9
10
11
12
13

let incoming = select! {
    biased;

    _ = &mut stopped_by_parent => {
        debug!(%actor_id, "stopping, because parent stopped this actor");
        drop_containing_panic(actor_id, STATE_FAILED_TO_DROP, state);
        break 'run;
    }

    incoming = mailbox.recv() => {
        incoming.expect("self_ref keeps a mailbox handle alive")
    }
};

Notice the biased, which makes the stop branch win against a message arriving at the same moment, and the drop_containing_panic, which is step 1 of our sequence happening right here. Now the sending side, which is all it takes to stop the children:

1
2
3
4
5
6
7
8
9

async fn stop_children(&mut self) {
    let (next_stopping_tx, next_stopping_rx) = watch::channel(());

    let stopping_tx = mem::replace(&mut self.stopping_tx, next_stopping_tx);
    let _ = stopping_tx.send(());
    self.stopping_rx = next_stopping_rx;

    stopping_tx.closed().await;
}

We send the stop signal, then await closed(), which resolves once every receiver has been dropped. A child drops its receiver clone only when its task ends, and its task ends only after it has run this very same sequence for its own children. So what we get from the channel closing is a recursive completion proof: no descendant of any depth is still alive. Notice that we did not build that barrier, we inherited it from ownership: the channel closes because the receivers are dropped, and they are dropped because the tasks holding them have ended.

Why the fresh channel swapped in at the top? Not just as bookkeeping for restarts. The context holds a receiver clone of its own, so unless we install the next generation’s receiver first, closed() would be waiting on the actor itself and never resolve. That we also end up with a working channel for a new generation of children is what makes the same function serve restarts, where the actor stops the current generation and then keeps running with a new one.

Draining early, signaling late

Steps 2 and 6 sit at opposite ends for opposite reasons.

We drain and disconnect the mailbox right at the start, before the children are even told to stop. That way a send is rejected rather than queued into a mailbox nobody will ever read, and it is rejected from that moment on, not only once the whole subtree is gone. What the sender makes of the rejection differs: a tell reports nothing to its caller, so the message is dropped and logged as a dead letter, while an ask gets AskError::ActorTerminated back instead of waiting out its timeout.

Draining also runs the destructors of the messages which were still queued, and that matters more than it sounds: a queued request carries a ReplyTo, and dropping it resolves the corresponding ask as NoReply instead of leaving it pending until its timeout. A send racing with the drain can still slip past it, and such a message is retained until its last sender is dropped, so this detection is best-effort, which is why we give every ask a timeout anyway.

The watchers, in contrast, are signaled at the very end, after the actor value has been dropped. This is what makes the guarantee from the previous episode worth having. If we sent the signal when the actor stopped receiving, it would prove only that: no more messages. Sent last, it proves that the state is gone, every descendant has terminated and the actor’s own destructors have run.

Awaiting the actor system

So is ActorSystem::terminated special framework machinery? It is not. spawn_root spawns the root actor through exactly the same code path as any child, and then registers an ordinary watcher in the root’s watcher registry. That watcher’s sink resolves the oneshot we await in terminated(), and it also owns the sender of the root’s stop channel, which is what keeps the root running until its own termination has signaled its watchers.

So when we await an actor system, we are doing a death watch like any other, and it inherits the meaning we established above: it resolves exactly when the entire tree is gone. Registration is race-free the same way too, since a root which has already terminated fails the registration and the oneshot is resolved right away.

One consequence is worth spelling out, because it catches people: dropping the ActorSystem does not stop anything. The root stops on its own terms; dropping merely forfeits our ability to await it.

When destructors panic

All of this rests on destructors running, so what happens when one panics? Every drop path the framework controls is contained: the state, the queued messages and the actor value are all dropped under catch_unwind, so the panic is logged and termination completes, watchers included. If a panicking destructor could abandon the sequence, one bad Drop would leave us with a permanently stuck tree.

One case is out of reach, and it is a Rust one rather than a tellus one. If init or receive panics and, while that panic unwinds, a destructor of a value still alive in the frame panics too, Rust aborts the process. That happens below catch_unwind and hence below supervision, so there is nothing tellus can do about it. Well, we just have to keep our destructors panic-free.

The end of the core

Over four episodes we have seen that an actor is a state machine, that its mailbox is one FIFO queue carrying both messages and signals, and that its termination is bottom-up, with the terminated signal sent last so it proves something worth knowing. The rest of tellus is built from those three ideas.

What comes next is not core but extension, and both are feature gated so tellus stays purely local and free of their dependencies by default: persistence, in the shape of event sourced actors, and remoting, where serializable actor refs let actors on different nodes message and watch each other through the same API. The second one will be an interesting post to write, because a network is precisely the thing that does not give us one FIFO queue.

And of course the code is on GitHub: tellus itself, its examples, and docs/actors.md, which explains the core top-down with links into the implementation. Thank you for staying around until here, and let me know if you have any questions.

My thoughts, written with AI assistance.

Why does that not need any coordination? Because it falls out of a single design decision.

One channel for both

A mailbox is a FIFO channel of Incoming<M>, and Incoming is exactly the enum receive gets: either a message or a terminated signal. We split that channel into a sending half, which is cloned into every ActorRef pointing at the actor, and a receiving half owned by the actor’s run loop.

The decision is that terminated signals are not a side channel. They ride the very same queue as ordinary messages. So if a watched actor delivers three messages to its watcher and then terminates, those three messages and the signal enter the watcher’s queue in that order, and its run loop takes them out in that order. We have nothing else to synchronize, because a queue is already the synchronization.

With a separate signal channel we would need a rendezvous between two channels to reconstruct exactly this order. One queue gets it right by construction.

Bounded mailboxes, unbounded channel

That decision immediately collides with a second requirement. A mailbox can be bounded, and a full one drops messages. But we must never drop a terminated signal: it is what proves termination, and a lost one leaves a watcher waiting forever.

If the bound lived in the channel, those two requirements would be in direct conflict, so we do not put it there. The channel underneath is always unbounded, and we enforce the capacity with a reservation counter sitting in front of it. Let’s look at the counted side: sending a message reserves capacity first and only then enqueues, and receiving one releases the reservation:

1
2
3
4
5
6
7
8

pub(crate) fn try_send_counted(&self, item: T) -> Result<(), CountedSendError> {
    match self.quota.reserve() {
        Ok(reservation) => {
            self.item_tx.send(item).map_err(|_| Disconnected)?;
            reservation.commit();
            Ok(())
        }
        ...

A terminated signal takes the other door into the same channel:

1
2
3

pub(crate) fn try_send_uncounted(&self, item: T) -> Result<(), Disconnected> {
    self.item_tx.send(item).map_err(|_| Disconnected)
}

Counted and uncounted, both ride one queue. The signal keeps its place in line behind everything sent before it, and no capacity check can ever reject it. So we have stopped capacity and ordering from being the same mechanism, and they stop fighting.

What a watcher actually is

ActorContext::watch(other) reads as registering interest, but what we register is concrete: a watcher is a sending handle into the watching actor’s own mailbox, handed over to the watched actor, which keeps it in a registry.

When that actor terminates, the last thing it does is send the signal through those handles, into the same queue it has been sending messages to that watcher all along. That is the entire mechanism behind the ordering guarantee: we need no coordination and no happens-before argument spanning two channels, just a sender used twice.

Let’s look at what that buys us, in the scatter_gather example. The root actor spawns workers, watches each and sends each a request whose ReplyTo delivers the partial result back as an ordinary message:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

match incoming {
    Incoming::Message(Partial(sum)) => Ok(Control::Continue(Gathering {
        total: state.total + sum,
        ..state
    })),

    // The partial result of the terminated worker has already been added.
    Incoming::Terminated(_) => {
        let remaining = state.remaining - 1;

        if remaining > 0 {
            Ok(Control::Continue(Gathering { remaining, ..state }))
        } else {
            println!("## Total is: {}", state.total);
            Ok(Control::Stop)
        }
    }
}

Notice the comment: that is the point. When the signal for a worker arrives, that worker’s partial result is already in total, because the worker replied before it stopped, and the reply went into this queue ahead of the signal. So counting terminated signals is all we need to know the sum is complete, and we need no timeout and no acknowledgement protocol.

What the guarantee does not say

So is the watcher guaranteed to have seen everything the terminated actor sent? No, and we should state the difference precisely, because the guarantee is easy to overstate.

Delivery is at most once. A message told to a full bounded mailbox, or to an actor which has already terminated, is dropped and logged as a dead letter; it never enters the queue at all, and the signal cannot say anything about a message we never queued.

So the guarantee is not “the watcher has seen every message the terminated actor sent”. It is: the watcher has seen every message from that actor it will ever see, because each one either arrived ahead of the signal or was dropped as a dead letter and was never going to arrive. Nothing is in flight anymore. That is what lets us act on the signal as a completion marker, and it is exactly as strong as at-most-once delivery permits.

Closing two smaller races

Registration is race-free in both directions. The watcher registry is a map which termination takes and closes atomically, so an actor racing to watch a terminating actor either gets in before the close and is signaled, or fails registration and learns immediately that the actor has already terminated, in which case we deliver the signal right away. Either way we lose no watcher, and registering twice registers once.

unwatch has a stronger contract than it appears to: after it returns, no terminated signal for that actor will be received, even if the actor has already terminated and its signal is already sitting in the queue. We cannot enforce that on the sending side, so we enforce it on the receiving side, in the run loop, before receive ever sees the signal:

1
2
3
4
5
6
7
8
9
10

if let Incoming::Terminated(other) = &incoming
    && !context.take_watched_for(*other)
{
    debug!(
        %actor_id,
        other_id = %*other,
        "dropping terminated signal for an unwatched actor"
    );
    continue;
}

Notice that the watching actor keeps its own record of what it watches, and we drop a signal whose sender is no longer in that record. The same bookkeeping deregisters a terminating actor from everything it still watches, so dead watchers never accumulate.

What is next

One thing above was quietly load bearing: the signal is the last thing a terminating actor does. Not the first, and not somewhere in the middle. In the next episode we look at why, and at the termination sequence that makes a terminated signal prove not just that the actor stopped receiving, but that it and its entire subtree are gone, destructors and all.

If you want to run this yourself, you can find the full code on GitHub: the example is scatter_gather.rs, and the queue machinery is in quota.rs and mailbox.rs.

My thoughts, written with AI assistance.

Three associated types

An actor is defined by implementing Actor. Let’s start with its three associated types:

1
2
3
4

impl Actor for Manager {
    type Message = RequestJob;
    type State = Managing;
    type Error = Infallible;

Message is what this actor accepts. We cannot send it anything else, because sends go through an ActorRef<M>: tell takes an M, and ask takes a function which builds one from a ReplyTo.

State is what it carries between messages. Notice what we do not get: no fields on the actor value itself that change over time. Manager is a unit struct here, and everything mutable lives in Managing.

Error is how it can fail. Both init and receive return Result<_, Self::Error>, so what we get is an ordinary value rather than an escape hatch.

Each of the three has a degenerate case, and each of those is a real type rather than a convention. For a stateless actor we use (), for an infallible one Infallible, and for an actor which receives no messages at all, a pure supervisor for instance, we use Nothing, an uninhabited enum:

1
2

impl Actor for Overseer {
    type Message = Nothing;

Why bother with an uninhabited type? Because with Nothing as the message type, Incoming::Message cannot be constructed, so the only incoming such an actor can ever get is a terminated signal from an actor it watches. The compiler knows that, and so do we.

Modeling the state

init creates the initial state and receive maps the current state plus one incoming to the next state. Let’s look at a work-pulling manager which hands out jobs to workers as they ask for them:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33

struct Managing {
    jobs: Vec<&'static str>,
    workers: usize,
}

fn receive(
    &self,
    _: &ActorContext<Self::Message>,
    incoming: Incoming<Self::Message>,
    state: Self::State,
) -> Result<Control<Self::State>, Self::Error> {
    match incoming {
        Incoming::Message(RequestJob(reply_to)) => {
            let mut jobs = state.jobs;
            match jobs.pop() {
                Some(document) => reply_to.reply(Next::Job(document)),
                None => reply_to.reply(Next::Drained),
            }
            Ok(Control::Continue(Managing { jobs, ..state }))
        }

        Incoming::Terminated(_) => {
            let workers = state.workers - 1;

            if workers > 0 {
                Ok(Control::Continue(Managing { workers, ..state }))
            } else {
                println!("## All documents processed");
                Ok(Control::Stop)
            }
        }
    }
}

Notice that the state arrives by value, so we can simply move state.jobs out and mutate it locally; the struct update syntax then puts the next state together. We get no &mut self and no interior mutability, yet nothing here is expensive: the Vec is moved, not cloned.

Two more things are visible in that snippet. Control is the return: Continue carries the state which will handle the next message; Stop ends the actor, which stops its children first and terminates once all descendants are gone. And Incoming has two variants, so messages and terminated signals arrive through the same receive, in one FIFO order. The manager watches every worker it spawns and stops itself once the last one has terminated; since a worker only stops after the job queue is drained, the last terminated signal also means all the work is done. What we get on top from that shared FIFO order, a terminated signal proving that the watcher has seen every message from that actor it will ever see, is the subject of the next episode.

Failing and restarting

Because Error is an associated type, failing is just returning. Let’s look at a loader whose domain has one toxic value, so type Error = ToxicValue;:

1
2
3
4
5
6
7
8
9
10
11
12

match message {
    // The error escalates to supervision, which restarts the loader with backoff.
    Message::Load(TOXIC) => Err(ToxicValue(TOXIC)),

    Message::Load(value) => {
        let count = count + 1;
        println!("## Loaded {value}, count is {count}");
        Ok(Control::Continue(count))
    }

    Message::Finish => Ok(Control::Stop),
}

Returning Err hands the failure to the actor’s supervision strategy, and inside receive a ? does the same for anything that converts into Error. If we would rather handle a failure as part of the domain, we write an ordinary match which returns Control::Continue with whatever state we consider correct. So we choose between escalating and handling per call site, in code, with no framework-specific ceremony either way. Panics, incidentally, take the same route: they are caught and fed to supervision exactly like a returned error.

So why separate the actor value from the state at all? Here we get the payoff. Under the Restart strategy, a failure stops the actor’s children, waits out an exponentially growing backoff and re-runs init, until a streak exceeds max_restarts and the actor stops instead. What we rebuild is the state, and only the state. The actor value survives, which is why we put configuration and the refs an actor was constructed with there. The mailbox survives too, so the messages queued behind the failing one are processed by the restarted state, and messages told during the backoff simply queue up. The message which caused the failure is consumed and not redelivered, which is what keeps a toxic message from turning into a restart loop.

We need no rollback mechanism for any of this. The next state only comes into existence when receive returns it, so a receive which fails or panics halfway through has not written anything. Supervision never sees a half-updated state: the failed one is dropped, and init builds a fresh one. In a framework built on &mut self, we would need either a transaction or a convention that nobody violates under deadline pressure. Well, I would rather have the compiler enforce it.

What is next

So far we have seen the state machine from the inside. In the next episode we look at it from the outside: the mailbox, which is one FIFO queue carrying both messages and terminated signals, and the ordering guarantee that falls out of that single queue.

If you want to run these yourself, you can find the full code on GitHub: the manager is work_pulling.rs and the loader is supervision.rs.

My thoughts, written with AI assistance.