In light of a recent Microsoft outage that, to me, looked a lot like a metastable failure, I want to revisit metastable failure recovery.  Now, I don’t really know much about the actual failure beyond the little bits of information released to the public, so I can only speculate about its metastable nature. But this snippet caught my attention and prompted me to write this post: “The servers we returned to service can become unresponsive again once they receive traffic; we’re analyzing diagnostic data to determine what’s triggering the behavior.”

So a quick refresher on metastable failures. These failures start with an initial trigger that puts the systems “out of balance,” usually by overloading a critical system resource (CPU, I/O, locks, etc.). The trigger starts an amplification loop that keeps the failure persisting even after the initial fault is addressed. A typical example of a metastable failure is retry-induced metastable failure: a trigger starts the overload; once overloaded, some requests time out, clients retry the timed-out work, and these retries create more overload, leading to even more requests timing out, and so on.

A typical recovery will have to do two things. Number one, we need to break the amplification loop, so we stop digging a deeper hole. Number two, we often need to deal with the backlog of accumulated/pending work.  Often, you can accomplish both by “luck” — rebooting the system kills the backlog that activates the feedback mechanism, so when you get out of that “reboot,” you get a clean state with no amplification and no backlog. I am grossly simplifying things here, of course, since rebooting large systems is hard, and a cold start done too fast can create a metastable failure. But generally, reducing the backlog (by dropping work) reduces the amplification.  We often combine this with trying to get more resources, so that we can avoid dropping all pending work/data that still might be useful (e.g., you may not want to lose hours of undelivered emails, for example). Prioritizing new work also helps, as it keeps new work from contributing to some forms of amplification, like retry-induced metastability.

However, sometimes we are not so lucky, and clearing the backlog queues on something that appears failed doesn’t work. This happens because the component that looks failed isn’t the one creating the amplification loop, so rebooting it or clearing that backlog doesn’t affect the active feedback loop. It is hard to describe quickly, as I am writing this without much prior thought.

So I will give an example. I modeled a simple system with an open-loop load generator creating work (a stand-in for clients), a gateway service acting as a load balancer and retry queue, and worker nodes. The load generator talk to the gateways, and the gateway load-balances work onto the servers. The gateways also keep a timeout for each work item and retry up to several times if the server does not respond. The worker nodes, well, they do simulated work on those requests. I have an image of my MESSI graph for this little model here:

This model focuses on retry-induced metastability and doesn’t include active mitigation strategies (load shedding, admission control, etc.) for simplicity. I set up the overload as a spike that starts at 20 seconds (first red dashed vertical line) of runtime and completely disappears at 80 seconds (last blue dashed vertical line); both the overload spike and return to normal is gradual over the 20-seconds interval; the base load is 70% of the combined worker node capacity, and the spike reaches 130% of the capacity. Each request can be retried 3 times — this is my amplification driver. The gateway handles this amplification and keeps track of requests that need a retry. When I run this workload, I get the following in my simulation: (1) gateways are not overloaded, but they get close to their capacity, (2) worker nodes do get overloaded and fail with no goodput after 32 seconds, (3) worker nodes accumulate the backlog:

Now let’s talk about mitigations. Only worker servers failed and overloaded here, and worker servers have the backlog, so my instinct is to reset the workers. I can do this by purging their queues (again, a destructive action that can cause data loss in real systems). In my simulation, I did the queue purge at the 100s mark. This, however, achieved diddly-squat for me. See, I may have purged the backlog, but I haven’t addressed the feedback loop: these gateways still wait on a ton of work and will retry, filling that backlog again. The goodput showed a small blip of recovery at the 100s mark right after the purge. Still, the system was quickly overrun again, because the backlog that caused the failure is separate from the retry mechanism that caused amplification, and I never reset the retry mechanism.

And so the actual recovery here requires me to not only purge the worker servers, but also stop the retry cycle at the gateway. When I do so, the system recovers:

Obviously, my example here is rather academic—I can purge queues quickly, and recovery almost seems magical and quick, while in reality things do take time. I could have simulated that as well, and when recovery actions take time, metastable loops do not stop, so you also have to contend with the speed of recovery against the speed of amplification. Nevertheless, this demonstrates what I wanted to show: the need to identify and isolate the feedback loop, and that the loop may run on components that do not feel overloaded or failed.