Asynchrony, queues and back pressure

Queues turn spikes into backlogs

Load never arrives evenly. Lunch orders bunch at 12:30, ticket buyers all press the button at 10:00:00, and a batch job dumps a million events at midnight. A synchronous service must either have capacity for the highest peak or fail during it. Putting a queue between the producer of work and the service that does it changes the arithmetic: during a spike the extra work waits in the queue, and the consumers process it at their own steady rate, catching up when the spike has passed. Microsoft's catalogue of cloud patterns calls this queue-based load levelling. Section 2.4 showed why queues form whenever arrivals bunch; this section is about using them deliberately.

The benefit is real: consumers can be sized for the average rather than the peak, a slow or restarting consumer does not lose work, and the producer is no longer temporally coupled to the consumer (Section 6.5). The price is that the work is now done later. A queue turns a spike into a backlog, and a backlog is simply a delay that every item behind it will experience. That is fine for sending confirmation emails or updating a search index, questionable for assigning a courier to an order, and unacceptable for a payment the customer is waiting to see confirmed. The first design question for any queue is therefore: how long may an item wait, and what happens to items that would wait longer?

Work queues and competing consumers

The most common use is a work queue with competing consumers: several identical workers read from one queue, each message goes to only one of them, and throughput grows by adding workers. It is the X axis of the scale cube applied to background work, and it scales naturally on the right signal: backlog per worker, or the age of the oldest message.

The mechanics matter for correctness. A worker takes a message, processes it and then acknowledges it; if the worker crashes first, the broker makes the message visible again after a timeout (SQS calls it the visibility timeout) and another worker processes it. Delivery is therefore at least once, so consumers must be idempotent (Section 6.4). A visibility timeout shorter than the real processing time causes the same message to be processed twice in parallel. Ordering is lost as soon as there is more than one consumer, unless messages are partitioned by key, in which case the number of partitions caps the number of useful consumers. And prefetching (a worker taking several messages at once to save round trips) should be kept low for slow tasks, or one worker hoards messages that idle workers could have processed.

A queue does not create capacity

A queue absorbs a temporary excess; it cannot absorb a permanent one. If work arrives faster than it is completed for long enough, the backlog grows without limit, and Little's law (L = λ × W) says the waiting time grows with it. An unbounded in-memory queue eventually exhausts memory and crashes the process, losing everything in it; an unbounded durable queue fills its disk or keeps work for hours after anyone cares about it. Worse, the queue hides the problem: producers get fast acknowledgements, error rates stay at zero, and the only symptom is that results arrive later and later. Engineers at Amazon describe such backlogs as potentially insurmountable, because a service that falls far enough behind may spend all its capacity processing work that its clients have long since given up on.

Two practices follow. First, measure the age of the oldest message as well as the queue length: age is what users experience, and it reacts immediately when consumers stall. Second, give every queue a bound derived from the waiting time you are willing to accept, using Little's law, and decide in advance what happens when the bound is reached. That decision is the subject of the next three parts of this section.

Back pressure: making producers slow down

Back pressure is any mechanism by which a slower downstream stage makes a faster upstream stage slow down, instead of letting work pile up between them. The term comes from fluid in pipes, and the idea was met in pipes and filters (Section 5.4). It is built into many layers of the stack you already use. TCP's flow control stops a sender when the receiver's buffer is full. A bounded blocking queue makes the producer's put wait for space. Pull-based consumers, as in Kafka, take work only when they are ready, so the broker's log rather than the consumer's memory holds the backlog, and a Kafka producer blocks or fails when its own send buffer is full. The Reactive Streams specification lets a subscriber request a specific number of items, so a stream never delivers more than the subscriber has asked for.

Back pressure works well inside one process or along one pipeline. Across a request-response boundary with human users it usually becomes rejection: an overloaded HTTP service cannot make a phone wait politely, so it answers 429 Too Many Requests or 503 Service Unavailable, ideally with a Retry-After header, and relies on the client to back off. When a bounded queue is full there are only three choices: block the producer, reject the new work, or drop something already accepted. Each is right somewhere; the mistake is not choosing.

Load shedding and prioritisation

A server under more load than it can handle does worse than serving its capacity and refusing the rest. It accepts everything, every request waits in its internal queues, response times rise past the clients' timeouts, and it ends up spending its whole capacity on requests whose clients have already given up. Throughput stays high while goodput, the rate of requests completed in time to be useful, collapses towards zero. Load shedding prevents this by rejecting some requests deliberately so that the rest succeed.

Good shedding has three properties. It is early and cheap: rejecting a request must cost far less than serving it, so it happens at the front door, before authentication-heavy or database work, based on a cheap signal such as in-flight requests or queue age. It is prioritised: Google's SRE book describes labelling every request with a criticality (from CRITICAL_PLUS down to SHEDDABLE) so that batch and prefetch traffic is dropped before user-facing traffic. In the food-delivery platform, checkouts and courier location updates outrank recommendations, which outrank analytics events. And it is aware of time: a request that has already waited longer than its caller's timeout should be dropped rather than processed. Facebook's engineers have described two refinements: a controlled-delay queue that allows only a short wait when a queue has not emptied recently, and switching to last-in, first-out under load, so that the freshest requests, whose users are still waiting, are served first.

A further step is to adapt the limit itself. Fixed concurrency limits are either too low on a good day or too high on a bad one; adaptive concurrency limiters raise the permitted number of in-flight requests while latency stays low and cut it when latency rises, borrowing the idea from TCP congestion control. At the very front of a system, admission control can take the form of a virtual waiting room, as in the ticket on-sale above: users are held in a queue at the edge and admitted at the rate the core can serve, which is load shedding presented as fairness.

Rate limiting algorithms

Rate limiting restricts how much work a particular client, API key, tenant or IP address may submit in a period. It differs from load shedding in purpose: shedding protects the server from total overload whoever causes it, while rate limiting shares capacity fairly and protects against any single client, buggy or malicious. Both answer the same way, usually with HTTP 429. Four algorithms dominate.

The token bucket deserves a closer look because it is the usual default. The bucket holds up to b tokens and gains r tokens per second. Each request takes a token; if none is left, it is rejected (or, in the shaping variant, delayed). A client that has been quiet can therefore send a burst of b requests at once, which suits real clients such as a mobile app opening a screen, while over any longer period it cannot exceed r on average. The leaky bucket is the mirror image: it smooths output to a constant rate, which is useful for protecting a downstream system that cannot accept bursts at all.

import math, time

class TokenBucket:
    def __init__(self, rate, capacity):
        self.rate = rate              # tokens added per second
        self.capacity = capacity      # largest burst allowed
        self.tokens = capacity        # start full
        self.updated = time.monotonic()

    def allow(self, cost=1):
        now = time.monotonic()
        elapsed = now - self.updated
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        self.updated = now
        if self.tokens >= cost:
            self.tokens -= cost
            return True, 0
        wait = (cost - self.tokens) / self.rate   # seconds until enough tokens
        return False, max(1, math.ceil(wait))      # for the Retry-After header

bucket = TokenBucket(rate=5, capacity=20)        # one bucket per API key
allowed, retry_after = bucket.allow()

HTTP 429, Retry-After and where to enforce limits

HTTP has standard signals for both situations. 429 Too Many Requests, defined in RFC 6585 (2012), tells a client that it has sent too many requests in a given time. 503 Service Unavailable says the server as a whole is temporarily overloaded or in maintenance. Either may carry a Retry-After header (RFC 9110), giving a number of seconds or a date after which retrying makes sense. Many APIs also return headers announcing the limit, the remaining allowance and the reset time, traditionally X-RateLimit-*; an IETF draft is standardising RateLimit and RateLimit-Policy fields. Well-behaved clients honour Retry-After, add jitter so that rejected clients do not all return at the same instant, and treat repeated 429s as a reason to slow down rather than to retry harder (Section 9.4).

Limits can be enforced at several layers, and mature systems use more than one: at the edge or API gateway per IP address and API key, to stop abuse and runaway clients before they cost anything; per tenant, so that one large customer cannot starve the others; and inside each service, to protect its own capacity and its dependencies. A global limit across many gateway instances needs shared state, usually atomic counters in Redis, which adds a network call per request; a common compromise is to give each of N instances a local limit of about 1 ÷ N of the total and accept some imprecision. Limits should be set from measured capacity and published, so that partners can design to them, and the number of rejections should be monitored, because a sudden rise is either an attack, a client bug or a limit that no longer matches reality.

Dead-letter queues

Some messages fail every time: a malformed payload, an order for a restaurant that no longer exists, a bug triggered by one particular input. Retried indefinitely, such a poison message blocks a partition or wastes a worker forever. A dead-letter queue (introduced in Section 6.4) receives messages that have failed a set number of times (SQS calls this the maximum receive count in a redrive policy), so that the rest of the work can continue.

A dead-letter queue is a tool for defects, not for overload: if messages land there because consumers are too slow, the fix is capacity or back pressure, not a second queue. Every message in a DLQ means something the business expected did not happen, so the DLQ needs an owner, an alert on its depth or age, tools to inspect messages with their error and attempt count, and a way to replay them once the cause is fixed. Decide in advance what replaying late means for ordering and for messages whose business moment has passed: a refund can be replayed a day late, a courier assignment cannot.