tellus 0.6: stashing, rolling upgrades and a faster cluster
tellus 0.6 is out. Earlier this month, tellus 0.5 brought the same ActorRef across machines, and a week later I wrote about how we test that cluster under permanent chaos. With this release, we get stashing, which an actor opts into, and rolling upgrades for a cluster. It also makes remote messaging several times faster, changes how hosts passivate their entities and puts a timeout on every store call. Before we look at each of them: a big shout-out to cosmir17, who wrote stashing and more than a fifth of the commits in this release.
Stashing, for the actors which ask for it
Let’s start with an actor which has to wait for something before it can handle its messages, for example a table it loads when it starts. What happens to the lookups which arrive in the meantime? With 0.6, the actor can stash them and handle them later. Here we see the directory actor from the how-to on deferring messages while it is still loading:
(State::Loading, Incoming::Message(Message::Loaded(customers))) => {
println!("Loaded {} customers", customers.len());
context.unstash_all();
State::Ready { customers }
}
(State::Loading, incoming) => {
match context.stash(incoming) {
Ok(()) => {}
Err(Incoming::Message(Message::Lookup { reply_to, .. })) => {
reply_to.reply(Err(LookupError::Busy));
}
Err(_) => unreachable!("only a lookup can be refused"),
}
State::Loading
}Notice that stash takes the Incoming as receive got it. So a terminated signal can be stashed too, and it keeps its place behind the messages of the actor which stopped. The ordering guarantee of death watch hence survives stashing. Once our table is in, unstash_all hands everything back, one receive per item and in the order it was stashed. The stash has the capacity of the mailbox, and a full stash hands the item back as an error. Then the actor decides what to do, and in our example it answers Busy.
Stashing is opt-in per actor type, like timers: our directory declares type Stash = Stashed;. An actor which declares Unstashed has no stash method at all, and its run loop contains no stash code. If you know Akka Typed, its StashBuffer is the model. The differences are that a full stash hands the item back instead of throwing, that signals can be stashed, and that unstashed items go through the run loop one receive at a time.
Rolling upgrades
A cluster which runs for months will see its message types change. During a rolling upgrade, our old and new nodes run side by side and talk to each other. So a new node has to understand the old shape of a message, and an old node must not get a shape it doesn’t know. With 0.6, every payload carries the schema version of its type, and every node announces the application version from its configuration when it connects. Every type which crosses nodes implements cluster::Schema, and for a type which keeps its shape, we can go with the defaults:
impl Schema for Greet {}When a type changes, its Schema gets a new VERSION. It decodes the old version, and it encodes the old shape for peers which still run the old application. Old code cannot learn a new shape, so it is our new code which has to do both. How that looks in code is the topic of my next post.
We test this the same way as the rest of the cluster. The cluster demo now also rolls all five nodes between two application versions, one node per pause in the chaos. The verifier checks every mixed cluster on the way.
There is one catch for this release itself. The wire protocol is now version 2, and 0.5 and 0.6 nodes refuse each other. So moving from 0.5 to 0.6 means restarting the whole cluster once. Rolling upgrades work from 0.6 on.
A faster cluster
The payload codec used to be a trait. It had to be object safe, so every payload went through erased_serde, with one dynamic call per field and per element. Now the codec is an enum whose only variant so far is Codec::Postcard, with CBOR planned as the next one, so encoding and decoding are static calls. On a 10-core arm64 machine, bulk transfer went from 97 to 713 MiB/s, and pipelined round trips from 534,000 to 1,462,000 per second. The bytes on the wire did not change. The price is that you can no longer plug in a codec of your own. If you need another format, please open an issue, and it can become another variant.
Passivation without a stop message
Let’s turn to hosts. A host spawns entities by key and, if configured, passivates them once nothing has been routed to them for a while. Hosts have existed since 0.3, but 0.6 changes how passivation reaches an entity. An entity’s message type is now Hosted<M>, where M is its domain message:
pub enum Hosted<M> {
Message(M),
Passivate,
}The host wraps every message it routes in Hosted::Message and asks the entity to stop with Hosted::Passivate. So our domain message needs no stop variant, and clients which send through the host cannot ask an entity to passivate. Also new: with the serde feature, a HostConfig can be read from YAML like the rest of the configuration.
A timeout on every store call
An event-sourced actor now bounds every call to its store, by 10 seconds unless Persistence::with_timeout sets another timeout. Before, a store which never answered kept the actor, and every ancestor waiting for it, from terminating. A load, read or append which runs into the timeout fails the actor into its supervision strategy. An append which timed out may still have landed, and the replay of the next incarnation picks it up if it has.
Upgrading
0.6 has breaking changes, and the release notes say for each of them what to do. Let’s look at the three which are the most likely to affect you. The first is the protocol change above. The second is the PostgreSQL store, whose tables are now called tellus_events and tellus_snapshots; the notes of tellus-persistence-postgres 0.3.0 have the statements which move your data. The third is the bootstrap configuration, where min_peers is now min_nodes and formation is now formation_provider. And tellus now needs Rust 1.100.
tellus is on crates.io and GitHub, and tellus.rs has the tutorials and how-tos. In the next post we will look at rolling upgrades in code. What do you think about 0.6? I am looking forward to your comments.
My thoughts, written with AI assistance.