Hey readers,
Imagine you’re building a new feature for a photo-sharing app. When a user uploads a new profile picture, the app needs to do several things: save the original image, create three different thumbnail sizes, run it through a content moderation filter, and finally, update the user’s profile URL.
If you do all of that during the initial web request, the user is just staring at a loading spinner, waiting. If any one of those steps is slow or fails, the entire upload process fails. This is synchronous processing, and it’s fragile and creates a poor user experience.
There has to be a better way. That better way is using a message queue.
A message queue is a component that allows different parts of your system to communicate asynchronously. Instead of ordering food at a counter and having to wait there until it’s ready, a message queue is like getting a buzzer. You can go find a table, relax, and the buzzer will let you know when your food is ready to be picked up.
Let’s look at three critical problems that message queues solve.
1. Decoupling Services (The Independent Workers)
In a complex system, you want your services to be independent. The service that handles user signups shouldn’t need to know the inner workings of the service that processes images.
A message queue acts as an intermediary, or a shared to-do list, between these services. The user signup service’s only job is to add a message to the queue that says, “A new image needs to be processed for user 123.” It doesn’t know or care who does the work.
Another service, the “image processor,” listens for messages on that queue. When it sees a new message, it picks it up and does the work.
- The benefit: This creates a decoupled system. You can take the image processing service offline for maintenance, and the user signup service will continue to work perfectly, happily adding jobs to the queue. When the image processor comes back online, it will simply start catching up on the backlog.
2. Handling Spikes in Traffic (The Shock Absorber)
What happens if your app gets featured on a popular blog and 10,000 new users try to upload a profile picture in the same minute? Your small team of image processing servers would be instantly overwhelmed and crash.
A message queue is the perfect shock absorber for these situations. Instead of each request hitting a server directly, they all just add a message to the queue. The queue might grow very large, but it won’t crash.
Your pool of worker services can then pull messages from the queue and process them at their own steady, sustainable pace. It might take a few minutes to get through the spike, but the system remains stable and no data is lost.
- The benefit: This makes your application far more resilient. It allows you to handle sudden, massive spikes in traffic without having to over-provision your servers for a rare event.
3. Running Background Jobs (The Asynchronous Task)
Let’s go back to our original example. Of all the tasks involved in uploading a profile picture, which one absolutely must happen immediately? Only one: saving the original image. Everything else—creating thumbnails, content moderation, sending a confirmation email—can happen in the background.
This is the perfect use case for a message queue. The initial web request can do the one synchronous task and then put messages in a queue for all the other “background jobs.”
The flow looks like this:
- User uploads image.
- Web server saves the image and immediately sends a “Success!” response back to the user.
- Web server then adds messages to a queue:
process_thumbnails,moderate_content,send_email. - Separate worker processes pick up these jobs and complete them over the next few seconds or minutes.
- The benefit: The user gets an incredibly fast, snappy experience. They aren’t left waiting for slow, non-essential tasks to complete. This makes the application feel much more responsive.
What’s the next move?
Challenge: Think about the process of ordering a product from an online store. What parts of that process do you think happen instantly (synchronously) to give you a response? And what parts might be handled by a message queue in the background (asynchronously) after you’ve already closed the page?
Consider things like:
- Validating your credit card.
- Updating the inventory count.
- Sending you a confirmation email.
- Notifying the warehouse to start the shipping process.
Identifying this distinction is the first step to designing robust, real-world applications.
Thanks for reading!
Bou~codes and Naima from 10xdev blog.