Webhooks: Event-Driven Inter-System Communication

In any system that interacts with external services, like a payment gateway, you’ll inevitably face the challenge of inter-system communication.

Let’s start with a simple, everyday example: an e-commerce store. A customer has just completed a payment through Stripe (or any other payment gateway). They’ve entered their card details, the payment was processed, the bank approved it, and the money was deducted.

The natural question is: How does your backend system know that the transaction was actually completed on the payment gateway, which is an external system you directed the customer to?

Do you wait for the user to return to your store via a redirect? What if the payment finishes, but the client never sends anything back to your server?

This highlights a crucial question in the backend world: How do you get notified by the provider itself when a specific event occurs, especially if that event is sensitive, like a payment?

The Old Way: Polling

Before Webhooks became prevalent, the primary solution was Polling. This meant your system would periodically check with Stripe (e.g., every 10 seconds) and ask, “Has this payment finished? Are there any updates?”

This method was common in many older systems before 2010, as most APIs were designed for the client to constantly ask for updates.

However, polling had clear drawbacks:

• High Load: Your system sends requests even if nothing has changed, consuming your resources and the external service’s resources (like Stripe’s).
• Latency: You only find out about an event after your next polling interval, introducing delays.

sequenceDiagram
    participant YourSystem as Your System
    participant ExternalService as External Service

    loop Every 10 seconds
        YourSystem->>ExternalService: Any updates for Payment X?
        ExternalService-->>YourSystem: No updates / Status of Payment X
    end
    note over ExternalService: Event happens (e.g., Payment X completed)
    YourSystem->>ExternalService: Any updates for Payment X?
    ExternalService-->>YourSystem: Yes, Payment X completed!

The New Way: Webhooks

Webhooks emerged to solve the polling problem. Instead of you asking, the service sends you a notification the moment an event occurs.

What is a Webhook? A Webhook is an HTTP request sent to a specific endpoint on your server by an external service as soon as an event happens on their end.

So, Stripe sends an HTTP POST request to your designated endpoint when a payment succeeds or fails. This request includes a full payload about the event, allowing you to instantly update your order status without polling.

sequenceDiagram
    participant YourSystem as Your System
    participant ExternalService as External Service

    YourSystem->>ExternalService: Register Webhook Endpoint
    note over ExternalService: Event happens (e.g., Payment X completed)
    ExternalService->>YourSystem: HTTP POST to Webhook Endpoint (Payment X details)
    YourSystem-->>ExternalService: 200 OK
    note over YourSystem: Process event, update order status

Webhooks became the standard because they provide:

• Real-time Notifications: You know about the event almost instantly.
• Reduced Load: No constant checking, saving resources for both your system and the external service.

Naima’s Note: In the age of AI, event-driven architectures are becoming increasingly vital. AI models often need to react to real-time data streams or external triggers. Webhooks are a perfect fit for this. Imagine an AI system that needs to process a new user review the moment it’s posted, or an AI-powered fraud detection system that needs to analyze a transaction as soon as it’s authorized. Webhooks enable this immediate, reactive processing, which is crucial for timely AI insights and actions.

Challenges of Webhooks

Webhooks are powerful, but they come with their own set of challenges:

1. Duplicate Events: The same event might be sent multiple times due to network issues or retries on the sender’s side.
   • Solution: Your system must implement Idempotency. If you receive the same event twice, processing it multiple times should have the same effect as processing it once. For example, if you update an order status to “paid,” receiving the “paid” event again shouldn’t cause an error or double-charge.
2. Security: This is the biggest challenge. How do you verify that the webhook actually came from Stripe and not from an attacker trying to send fake payment success notifications?
   • Solution: External services typically provide mechanisms like digital signatures or shared secrets. Stripe, for example, signs its webhooks. Your system receives the webhook, computes its own signature using the shared secret, and compares it to the signature provided by Stripe. If they match, you can trust the webhook.

   graph TD
       A[External Service] --> B{Generate Payload + Signature};
       B --> C[Send Webhook (Payload + Signature)];
       C --> D[Your System];
       D --> E{Verify Signature (using Shared Secret)};
       E -- Valid --> F[Process Event];
       E -- Invalid --> G[Reject Event];
   

Conclusion

Webhooks emerged to solve the inefficiencies of polling, which caused latency, load, and resource consumption. Their core idea is simple: when an important event happens at an external service, your system becomes the receiver, not the constant asker. This is why they are a fundamental part of modern, event-driven architectures.

Share your experiences on how you handle external services and manage security challenges!