tellus, episode 3: one queue for messages and signals

In the previous episode we looked at an actor from the inside, as a state machine. In this one we look at it from the outside, at the mailbox and the guarantee I called the most valuable thing in tellus: receiving a terminated signal for a watched actor proves that the watcher has seen every message from that actor it will ever see.

Why does that not need any coordination? Because it falls out of a single design decision.

One channel for both

A mailbox is a FIFO channel of Incoming<M>, and Incoming is exactly the enum receive gets: either a message or a terminated signal. We split that channel into a sending half, which is cloned into every ActorRef pointing at the actor, and a receiving half owned by the actor’s run loop.

The decision is that terminated signals are not a side channel. They ride the very same queue as ordinary messages. So if a watched actor delivers three messages to its watcher and then terminates, those three messages and the signal enter the watcher’s queue in that order, and its run loop takes them out in that order. We have nothing else to synchronize, because a queue is already the synchronization.

With a separate signal channel we would need a rendezvous between two channels to reconstruct exactly this order. One queue gets it right by construction.

Bounded mailboxes, unbounded channel

That decision immediately collides with a second requirement. A mailbox can be bounded, and a full one drops messages. But we must never drop a terminated signal: it is what proves termination, and a lost one leaves a watcher waiting forever.

If the bound lived in the channel, those two requirements would be in direct conflict, so we do not put it there. The channel underneath is always unbounded, and we enforce the capacity with a reservation counter sitting in front of it. Let’s look at the counted side: sending a message reserves capacity first and only then enqueues, and receiving one releases the reservation:

pub(crate) fn try_send_counted(&self, item: T) -> Result<(), CountedSendError> {
    match self.quota.reserve() {
        Ok(reservation) => {
            self.item_tx.send(item).map_err(|_| Disconnected)?;
            reservation.commit();
            Ok(())
        }
        ...

A terminated signal takes the other door into the same channel:

pub(crate) fn try_send_uncounted(&self, item: T) -> Result<(), Disconnected> {
    self.item_tx.send(item).map_err(|_| Disconnected)
}

Counted and uncounted, both ride one queue. The signal keeps its place in line behind everything sent before it, and no capacity check can ever reject it. So we have stopped capacity and ordering from being the same mechanism, and they stop fighting.

What a watcher actually is

ActorContext::watch(other) reads as registering interest, but what we register is concrete: a watcher is a sending handle into the watching actor’s own mailbox, handed over to the watched actor, which keeps it in a registry.

When that actor terminates, the last thing it does is send the signal through those handles, into the same queue it has been sending messages to that watcher all along. That is the entire mechanism behind the ordering guarantee: we need no coordination and no happens-before argument spanning two channels, just a sender used twice.

Let’s look at what that buys us, in the scatter_gather example. The root actor spawns workers, watches each and sends each a request whose ReplyTo delivers the partial result back as an ordinary message:

match incoming {
    Incoming::Message(Partial(sum)) => Ok(Control::Continue(Gathering {
        total: state.total + sum,
        ..state
    })),

    // The partial result of the terminated worker has already been added.
    Incoming::Terminated(_) => {
        let remaining = state.remaining - 1;

        if remaining > 0 {
            Ok(Control::Continue(Gathering { remaining, ..state }))
        } else {
            println!("## Total is: {}", state.total);
            Ok(Control::Stop)
        }
    }
}

Notice the comment: that is the point. When the signal for a worker arrives, that worker’s partial result is already in total, because the worker replied before it stopped, and the reply went into this queue ahead of the signal. So counting terminated signals is all we need to know the sum is complete, and we need no timeout and no acknowledgement protocol.

What the guarantee does not say

So is the watcher guaranteed to have seen everything the terminated actor sent? No, and we should state the difference precisely, because the guarantee is easy to overstate.

Delivery is at most once. A message told to a full bounded mailbox, or to an actor which has already terminated, is dropped and logged as a dead letter; it never enters the queue at all, and the signal cannot say anything about a message we never queued.

So the guarantee is not “the watcher has seen every message the terminated actor sent”. It is: the watcher has seen every message from that actor it will ever see, because each one either arrived ahead of the signal or was dropped as a dead letter and was never going to arrive. Nothing is in flight anymore. That is what lets us act on the signal as a completion marker, and it is exactly as strong as at-most-once delivery permits.

Closing two smaller races

Registration is race-free in both directions. The watcher registry is a map which termination takes and closes atomically, so an actor racing to watch a terminating actor either gets in before the close and is signaled, or fails registration and learns immediately that the actor has already terminated, in which case we deliver the signal right away. Either way we lose no watcher, and registering twice registers once.

unwatch has a stronger contract than it appears to: after it returns, no terminated signal for that actor will be received, even if the actor has already terminated and its signal is already sitting in the queue. We cannot enforce that on the sending side, so we enforce it on the receiving side, in the run loop, before receive ever sees the signal:

if let Incoming::Terminated(other) = &incoming
    && !context.take_watched_for(*other)
{
    debug!(
        %actor_id,
        other_id = %*other,
        "dropping terminated signal for an unwatched actor"
    );
    continue;
}

Notice that the watching actor keeps its own record of what it watches, and we drop a signal whose sender is no longer in that record. The same bookkeeping deregisters a terminating actor from everything it still watches, so dead watchers never accumulate.

What is next

One thing above was quietly load bearing: the signal is the last thing a terminating actor does. Not the first, and not somewhere in the middle. In the next episode we look at why, and at the termination sequence that makes a terminated signal prove not just that the actor stopped receiving, but that it and its entire subtree are gone, destructors and all.

If you want to run this yourself, you can find the full code on GitHub: the example is scatter_gather.rs, and the queue machinery is in quota.rs and mailbox.rs.

Disclosure: the prose of this post was drafted with Claude Code and revised by me.