Webhooks

Configure completion notifications

Configure webhook.url in your create-job request using a credential-free public HTTPS destination. An optional webhook.secret is a trimmed 16-200 character string. Otherwise securely store webhookSigningSecret from the accepted response; an authenticated replay of the identical request can return the same generated secret. Keep it on your server.

{
  "event": "crawl_job.completed",
  "deliveryId": "f7a35b1b-9e03-47fc-9a14-747e2968f31a",
  "data": {
    "crawlJobId": "8a1d1ac4-1bf4-4f2b-9c9c-bda9d2356e72",
    "resultUrl": "https://api.example.com/v1/crawl-jobs/8a1d1ac4-1bf4-4f2b-9c9c-bda9d2356e72/result",
    "status": "completed"
  }
}

X-Nostimates-Event: crawl_job.completed
X-Nostimates-Delivery: <stable delivery UUID>
X-Nostimates-Timestamp: <ISO timestamp>
X-Nostimates-Signature: v1=<hex HMAC-SHA256>

The callback contains completion metadata, not the collected artifact. The stable deliveryId identifies one notification across delivery attempts; crawlJobId identifies the job. completed means the result is available even if some or all collections failed. Fetch /v1/crawl-jobs/{crawlJobId}/result from your configured API origin with the owning account bearer token, then follow the storage redirect without forwarding that token to another host. Never send your token to an unchecked callback URL.

Verify the exact raw bytes

Signatures use HMAC-SHA256 with the webhook secret over the timestamp header, a literal period, then the exact raw HTTP body bytes. Read a bounded raw body before JSON middleware alters it. Do not parse and reserialize JSON before verification. The delivery and event headers must match the signed body; those headers alone are not authenticated identifiers.

import { createHmac, timingSafeEqual } from "node:crypto";

// Node.js 22+, ESM. Supply the exact raw Buffer and a Web Headers object.
export async function receiveCompletion({ rawBody, headers, secret, enqueueOnce,
  now = Date.now() }) {
  const timestamp = headers.get("X-Nostimates-Timestamp");
  const signature = headers.get("X-Nostimates-Signature");
  const time = Date.parse(timestamp ?? "");
  if (!Buffer.isBuffer(rawBody) || typeof secret !== "string" || !secret ||
      !Number.isFinite(time) || new Date(time).toISOString() !== timestamp ||
      Math.abs(now - time) > 5 * 60 * 1000 ||
      !/^v1=[0-9a-f]{64}$/.test(signature ?? "")) {
    throw new Error("Invalid webhook signature or timestamp");
  }

  const expected = createHmac("sha256", secret)
    .update(timestamp + ".").update(rawBody).digest();
  const received = Buffer.from(signature.slice(3), "hex");
  if (received.length !== expected.length || !timingSafeEqual(received, expected)) {
    throw new Error("Invalid webhook signature");
  }

  const event = JSON.parse(rawBody.toString("utf8"));
  const uuid = /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;
  if (!event || event.event !== "crawl_job.completed" ||
      !uuid.test(event.deliveryId ?? "") ||
      !uuid.test(event.data?.crawlJobId ?? "") ||
      event.data?.status !== "completed" ||
      typeof event.data?.resultUrl !== "string" ||
      headers.get("X-Nostimates-Event") !== event.event ||
      headers.get("X-Nostimates-Delivery") !== event.deliveryId) {
    throw new Error("Invalid completion metadata");
  }

  // Must atomically persist pending work under a unique deliveryId before resolving.
  // Existing IDs resolve without inserting more work; storage failures must reject.
  await enqueueOnce(event.deliveryId, event);
  return 204;
}

Pass a Node Buffer, a Web Headers object, your server-held secret, and your enqueueOnce storage adapter. Reject duplicate header values when your HTTP framework exposes them separately. This example chooses a five-minute clock-skew/replay window; that is a receiver policy, not an API deadline or retry schedule. Keep the receiver clock synchronized and adjust the window for your environment. Each delivery attempt is signed with its own timestamp.

Accept once, process durable work

CREATE TABLE webhook_inbox (
  delivery_id TEXT PRIMARY KEY,
  payload TEXT NOT NULL,
  status TEXT NOT NULL DEFAULT 'pending'
);

-- enqueueOnce binds the verified delivery ID and JSON payload as parameters.
-- Resolve only after this insert commits; duplicates leave pending work intact.
INSERT INTO webhook_inbox (delivery_id, payload)
VALUES (?, ?)
ON CONFLICT (delivery_id) DO NOTHING;

Implement enqueueOnce with your durable database: atomically insert the verified deliveryId and serialized event into a pending inbox using a unique constraint. The SQLite statement above shows the storage boundary; the JavaScript example does not provision a database or HTTP server. Return 2xx only after the insert commits or an existing durable entry is confirmed. A process-local Set is not durable deduplication. Let storage errors produce a non-2xx response so delivery can retry.

A worker must process pending inbox records independently of delivery. Mark work done only after it succeeds; recover interrupted work and retry failures. Make application side effects idempotent or commit them with the done state in one transaction. Keep delivery IDs long enough for retries and your replay policy. A duplicate callback returns 2xx without inserting another job, including when the original inbox entry is still pending.

Retries and fallback

Delivery is at least once. Eligible network and non-2xx HTTP failures receive up to six total attempts, including the first attempt. Configuration, signing-secret decryption, and network-policy failures stop immediately; an oversized receiver response can also be a permanent failure. Keep acknowledgement responses small. Attempts are tracked durably, but no exactly-once execution or delivery-time guarantee is promised.

If delivery fails permanently or attempts are exhausted, poll the existing job and download its result with account authentication. Webhook failure does not delete the result. Do not resubmit a crawl just to recover an artifact; see Quickstart and Result download for the same polling and retrieval path.