APIs and webhooks: why most integrations need both

Background

A webhook tells you something happened. An API lets you go and check. Use only an API and you're forever asking. Use only webhooks and you only know what you're told, which isn't always everything.

Say you're waiting on a parcel. You can keep ringing the depot to ask if it's out for delivery. That's an API: you ask, they answer, whenever you like. Or the courier can text you when it's on its way. That's a webhook: they tell you when something happens. Ringing every five minutes is exhausting, and a text that never arrives leaves you waiting in for nothing. So you take the text, and you ring the depot if nothing's turned up by lunch.

In technical terms:

• An API is a set of requests you can make to another system. "Give me record 4521." "Create this invoice." You pick the moment, and you get an answer straight back.
• A webhook is the same kind of request going the other way. You give the other system a web address, and when something happens, it calls it. "Record 4521 changed."

A webhook is quick but can be missed. An API is dependable but slow to notice anything. Each covers the other's weak spot, which is the whole argument for using both.

It's the shape of the JML build I wrote up. The HR system fires a webhook when a record changes, and a scheduled flow calls Microsoft Graph, which is an API, to do the actual work.

Setup

Why a webhook on its own isn't enough

A webhook is fast, and that's all it promises. Here's how it lets you down:

• It can get lost. Your server's down for a deploy, the network drops, or the request times out. Most providers retry for a while and then give up. How long, and how many times, depends on the provider, so read their docs and don't assume.
• It can arrive twice, or in the wrong order. Many providers promise delivery "at least once", which means a repeat is normal. A change made at 10:01 can turn up after one made at 10:02.
• It might be thin, or out of date. Some only send an id. Some send the whole record as it was when the event fired, and it may have changed again by the time you read it.
• It can be faked. Your endpoint is a public web address, so anyone can send it a request.
• It can't tell you about the past. A webhook only covers what happens after you subscribe. Last month's records come from the API.
• It can stop without telling you. Some subscriptions run out unless somebody renews them (Microsoft Graph's change notifications do), and some providers switch a webhook off after repeated failures. You don't get an error. You get silence.

Why an API on its own isn't enough

• It's wasteful. Asking every minute is 1,440 requests a day for each thing you're watching, and nearly all of them come back with "nothing's changed".
• It's slow to notice. If you check every 15 minutes, that's how late you can be.
• It runs into limits. Providers cap how often you can call, and some charge by the call. Ask too often and you get slowed down or blocked.
• Fetching everything each time doesn't last. It's fine with a few hundred records and a problem with a few hundred thousand.

The pattern: use both, in this order

1. Treat the webhook as a doorbell. When it arrives, check it's genuine, put the event on a queue and reply "got it" straight away. Senders often give up after a few seconds and send it again, so don't do the work while they wait.
2. Then ask the API for the truth. Take the record's id from the event and fetch the current version. The payload is a hint, not the data. That also deals with duplicates and out-of-order events, because you always act on the latest state.
3. Make the handler safe to run twice. Remember the event id, or the record's version, and skip anything you've already done. The same event twice should give the same result as once. In the JML build, a unique index does this job: the same HR record arriving five times makes one row, not five.
4. Add a sweep. On a timer, ask the API for everything that's changed since the last sweep finished, and run it through the same handler. It picks up whatever the webhook missed, and it's how you load the history when you first connect.
5. Move the "last swept" marker only after a sweep succeeds, and start each sweep a little before the last one ended. The overlap stops anything slipping through the gap. Duplicates don't matter, because of step 3.

It's the same trick as selecting "due today or earlier" instead of "due today": one event arriving on time shouldn't be the only thing between you and a gap.

Which one for what

When one is enough

Both isn't always the answer.

• API only is fine for a nightly report, a one-off import, or a provider that doesn't offer webhooks. Nobody minds if it's a day late.
• Webhook only is fine for a nudge where missing one doesn't matter, like a "someone filled in the form" ping.
• Both is for keeping one system's data in step with another's, or anywhere a missed event costs money, access or trust.

Receiving a webhook safely

• HTTPS only.
• Check the signature. Many providers sign each request with a shared secret. Recompute it from the raw request body, not the parsed JSON, which won't match, and compare using your language's constant-time comparison. If the provider sends a timestamp, reject old ones, so a captured request can't be replayed.
• Reply fast, work later. Send a 2xx straight away and do the work from a queue.
• Don't act on the payload alone for anything that matters. For money or access, confirm with the API first.
• Keep the signing secret in a secret store. Not in the code and not in a chat. If it's ever been pasted into either, treat it as exposed and replace it.
• Log the event id and what you did. Leave the whole payload out if it holds personal data.
• Ignore event types you don't recognise, and still reply 2xx. Providers add new ones, and if you answer with an error they'll keep retrying something you'll never handle.

Using AI to design it

An AI tool is a good first drafter and a decent second opinion here. It can't see how your provider really behaves, so check anything it says about retries, signatures or expiry against their docs. Keep real secrets and real customer payloads out of the prompt.

To review your own design:

I'm connecting [SYSTEM A] to [SYSTEM B]. Here's my design: [PASTE].
Review it as a sceptical integration engineer. List the ways it could lose
or duplicate data, most likely first: missed webhooks, duplicates,
out-of-order events, expired subscriptions, rate limits, my endpoint being
down. For each one, say what in my design would catch it, or write
"NOTHING" if nothing would. Don't redesign it yet. Say what you can't judge
without the provider's documentation.

To turn a provider's documentation into something you can compare:

Below is the documentation for [PROVIDER]'s API and webhooks. Build a
table: event name | what triggers it | what the payload contains | how it's
signed | retry behaviour | how long a subscription lasts. Use only what the
text says. If something isn't stated, write NOT STATED. Then list the
questions I'd need to ask the provider.

Documentation: [PASTE]

End either one with "list your assumptions", so you can see what it guessed.