In my previous article, I explained how I optimized follower comparison using O(n) complexity with JavaScript Sets. While that worked well for smaller accounts, scaling to users with 100k+ followers revealed new challenges: HTTP timeouts and GitHub’s secondary rate limits.

Fetching tens of thousands of followers requires thousands of paginated API calls, which can easily exceed typical HTTP request timeouts. To build a more resilient system, I moved away from synchronous execution and adopted a background job queue architecture powered by Redis and BullMQ.

The Architectural Shift: From Sync to Async

Instead of handling heavy computations inside a single HTTP request, I split the system into two roles.

Producer (API Server)
Receives the request, generates a unique Job ID, and pushes the task into a queue. The API immediately returns the Job ID to the client.

Consumer (Worker)
A background process that consumes jobs from the queue and performs the heavy follower analysis without blocking the main API server.

This approach ensures that even large accounts can be processed without risking server timeouts.

Infrastructure: Redis and Docker

To manage background job state and queue processing, I integrated Redis. Redis acts as the central coordination layer between the API server and the worker.

To ensure the system is reproducible and easy to deploy, the entire stack is containerized using Docker.

With a simple docker-compose.yml, the following services can be started instantly:

• Node.js API
• Worker process
• Redis instance

This setup makes the system portable and easy to scale.

Resilience: Backoff and Concurrency Control

GitHub’s API enforces strict limits to prevent abuse. To avoid triggering these limits, I implemented two key strategies using BullMQ.

Concurrency Control
The worker processes only one heavy job at a time. This prevents bursts of API requests that could trigger GitHub’s secondary rate limits.

Exponential Backoff
If the API returns a 429 Too Many Requests error, the job does not fail immediately. Instead, it retries with increasing delays (for example: 5s → 10s → 20s), allowing the system to recover gracefully.

Real-Time Progress Tracking with Socket.IO

Long-running background jobs can create a poor user experience if users are left waiting without feedback. To solve this, I implemented real-time progress updates using Socket.IO. Socket.IO is a library that enables bidirectional, event-based communication between the client and server over WebSockets (with HTTP long-polling as a fallback). Unlike traditional HTTP requests, where the client must repeatedly poll the server for updates, WebSockets allow the server to push events to the client instantly. The worker periodically reports progress while processing pagination. These updates are stored in Redis and emitted to connected clients via Socket.IO events. This enables the UI to display a live progress bar, allowing users to track the analysis in real time instead of waiting blindly.

Conclusion

Transitioning from a synchronous request model to a Worker/Queue architecture transformed the Social Graph Analyzer from a simple script into a resilient backend service.

The system can now:

• Process large accounts without timeouts
• Respect GitHub API rate limits
• Recover gracefully from temporary failures
• Provide real-time progress feedback to users

Whether a user has 100 or 100,000 followers, the system handles the workload reliably while keeping the user informed.