Design a rate limiter as a service
45 min · Advanced
Earlier, a rate limiter lived inside one server. In a real backend, 20 API servers sit behind a load balancer, and a client's requests land on all of them. A distributed rate limiter makes those servers enforce one shared limit, such as 100 requests per minute per API key, without slowing every request down.
Topics in this lesson
- Requirements: one shared limit, low added latency
- Placement: gateway, sidecar or library
- Shared counter store: atomic increment with expiry
- Race conditions and fail-open vs fail-closed
- Rule configuration per tier and monitoring
Key points
- Many servers need one shared counter; local counts multiply the limit by the number of servers.
- Gateway placement is the usual choice; sidecar and library are options when the limiter needs app data.
- Use an atomic
INCRwith an expiry (or one Lua script); never read and then write the count. - Choose fail-open or fail-closed per endpoint, and keep rules in config with allowed/rejected metrics.
Interview questions this lesson answers
- How do you rate limit when you have 20 API servers?
- Why not let each server limit on its own?
- What race condition can happen in a distributed limiter?
- What happens when Redis is down?
- Where would you put the limiter: gateway, sidecar or inside the app?
- How do you know your limits are set right?
- Your limiter adds 5 ms per request. How would you reduce it?
Sign in to continue this lesson
Create an account to read every lesson, practise with the exercises and keep track of your progress.