⚑LearnIT
Engineering Β· Topic 13

πŸ“ˆScaling for High Concurrency

When thousands of concurrent users hit a Next.js app, bottlenecks appear roughly in this order: (1) DB connection pool exhaustion, (2) server memory, (3) API response time / N+1 queries, (4) edge bandwidth, (5) real-time/websocket limits. Adding servers doesn't help if the database is the wall.

13.1Database (usually the first wall)

  • Connection pooling. Serverless functions each open DB connections β†’ pool exhaustion fast. Use PgBouncer or Prisma's pooling / Accelerate. Keep a single global client instance (don't new PrismaClient() per request):
const globalForPrisma = globalThis;
export const prisma = globalForPrisma.prisma ?? new PrismaClient();
if (process.env.NODE_ENV !== 'production') globalForPrisma.prisma = prisma;
  • Kill N+1 queries. Use eager loading (include/select) instead of a query per relation. Fetch related data in one round trip.
  • Select only needed fields, and take a limit β€” don't over-fetch.
  • Index strategically. For multi-tenant B2B: index the tenant key (team_id), status/filter columns, and time columns for range queries. Composite indexes with the most selective column first. Full-text (GIN) indexes for search.
  • Read replicas. Route heavy reads to replicas, writes to the primary.
  • Consider sharding/partitioning only at very large scale (10M+ rows) β€” by tenant id, or time-partitioning for append-heavy tables.

13.2Caching (multi-layer, recap from Part 5)

Edge/CDN (static, long TTL) β†’ Redis (hot data, short TTL, invalidate on write) β†’ client (react-query). Each layer absorbs load so fewer requests reach the DB.

13.3API design for scale

Cursor-based pagination; partial responses (field selection); request batching; rate limiting to protect the backend; idempotency keys for safe retries on writes.

13.4Real-time at scale

  • WebSocket needs sticky sessions (load balancer routes a user to the same instance) β€” via ip_hash (Nginx) or sessionAffinity: ClientIP (K8s). Persistent connections can't be freely load-balanced per-message.
  • Server-Sent Events (SSE) are simpler than WebSockets when you only need serverβ†’client push (notifications, live updates) and don't need clientβ†’server over the same channel.
  • Batch outbound messages to avoid flooding clients.
  • Beyond ~10k concurrent connections, introduce a message broker (Redis Pub/Sub, Kafka, RabbitMQ) or a dedicated realtime service (managed Pusher/Ably, or a Socket.IO cluster with a Redis adapter) so instances can broadcast to each other.

13.5Deployment & scaling

  • Horizontal scaling β€” multiple stateless instances behind a load balancer with health checks.
  • Auto-scaling on CPU/memory (K8s HPA), with sensible min/max replicas.
  • Multi-region for global latency; route users to the nearest region.
  • Managed (Vercel) vs self-hosted (Docker + K8s) β€” Vercel handles edge/scaling for you; K8s gives control and can be cheaper at scale but you own the ops.

13.6Observability (non-negotiable at scale)

Track latency percentiles (p50/p95/p99, not just averages β€” averages hide tail pain), error rates, DB pool utilization, cache hit rates, active connections. Tools: OpenTelemetry, Sentry (errors + tracing), Datadog/Grafana. Set alert thresholds and, where possible, auto-rollback on error spikes.

🧠 Quiz: Scaling for High Concurrency

1 question

1When scaling for high concurrency, which layer is usually the first bottleneck?

Answered 0/1