Rate Limiting & Thundering Herd Defense

Learning outcomes

• Build distributed rate limiting using Redis atomic scripts
• Protect downstream APIs from traffic spikes and abuse

Mental model

Rate limiting regulates the frequency of client API requests to protect server infrastructure, prevent Denial-of-Service (DoS) overloads, and enforce API monetization tiers.

Theory

• Token Bucket: Refills tokens at a constant rate; permits burst traffic up to bucket capacity.
• Leaky Bucket: Queues requests and processes them at a strict constant output rate; smooths out bursts.
• Fixed Window Counter: Counts requests per fixed minute/hour window; vulnerable to double-limit traffic spikes at window boundaries.
• Sliding Window Log / Counter: Measures requests over a moving time window; highly accurate.

# Redis Sliding-Window Rate Limiter in Python
import time
import redis.asyncio as aioredis

redis = aioredis.from_url("redis://localhost:6379")

async def is_rate_limited(user_id: str, limit: int = 100, window_seconds: int = 60) -> bool:
    now = time.time()
    clear_before = now - window_seconds
    key = f"rate:{user_id}"
    
    async with redis.pipeline(transaction=True) as pipe:
        pipe.zremrangebyscore(key, 0, clear_before)
        pipe.zadd(key, {str(now): now})
        pipe.zcard(key)
        pipe.expire(key, window_seconds)
        results = await pipe.execute()
        
    request_count = results[2]
    return request_count > limit

Alternatives and trade-offs

• Fixed Window: Minimal memory footprint; allows 2x limit spikes near window resets.
• Sliding Window Log (Redis ZSET): 100% boundary accuracy; higher Redis RAM usage for high request volumes.
• Token Bucket: Supports legitimate burst traffic while maintaining average rate limits.

Failure modes and misconceptions

1. Non-Atomic Rate Limit Checks: Executing GET count followed by SET count in separate non-atomic commands causes race conditions under high concurrency. Always use Redis Lua scripts or MULTI/EXEC pipelines.
2. Missing Retry-After Headers: Returning HTTP 429 Too Many Requests without an explicit Retry-After header causes aggressive client retries.

Decision scenario

Use Redis Sliding Window rate limiting for API Gateway authentication tiers to enforce accurate client rate limits without window boundary traffic spikes.

Learning outcomes

• Implement Token Bucket, Leaky Bucket, and Sliding Window rate limiting algorithms.
• Execute atomic rate limit calculations using Redis pipelines and Lua scripts.
• Attach standard HTTP rate limit headers (X-RateLimit-Limit, X-RateLimit-Remaining, Retry-After).

Trade-offs

Rate limiting guarantees API availability and protection against abuse, but requires fast in-memory stores like Redis for distributed state tracking.

Primary sources

• FastAPI Framework Architecture & Dependency Injection Specification — Tiangolo, verified 2026-07-22