Admission and enqueue are separate commits
A crash between them creates a request no worker can see, or an event pointing to state that never committed. The transactional outbox closes that gap.
Web capture engineering
Queue, retry and idempotency for browser jobs
A browser job is slow, memory-heavy and externally visible. The queue normally provides at-least-once delivery, so a process crash can replay a capture after it rendered, uploaded or billed.
4 min readPublished Updated
What you actually see
Two workers render the same request, the loser overwrites success with failure, a webhook fires twice, or usage contains a charge with no successful request left to justify it.
Reproduce it
- 01Kill the worker after artifact upload but before queue acknowledgement and observe which effects repeat.
- 02Race a synchronous handler and a queue worker against the same request id.
- 03Send the same bulk admission twice concurrently with one idempotency key; a read-before-write check should fail this test.
The DIY version
Start with the smallest thing that works
Use the database as the ownership boundary: request plus outbox in one transaction, atomic state transition for the lease, and unique constraints for each externally visible effect.
claim-job.mts
const claimed = await db.$executeRaw`
UPDATE requests
SET status = 'running', started_at = COALESCE(started_at, now())
WHERE id = ${requestId}
AND status IN ('accepted', 'queued')
`;
if (claimed === 0) return currentTerminalResult(requestId);
try {
const artifact = await render(requestId);
await db.$transaction(async (tx) => {
await tx.request.update({ where: { id: requestId }, data: { status: "succeeded" } });
await tx.usageLedger.create({ data: { request_id: requestId, units: 1 } });
});
} catch (error) {
if (isRetryable(error)) await requeueWithBackoff(requestId);
else await markFailedWithoutCharge(requestId);
}Where it breaks
The call is small. Everything around it is the system.
The lease is only an application check
Two workers can both read `queued` before either writes `running`. Ownership must be one conditional update whose affected-row count decides the winner.
Retry policy ignores error class
A bad selector will not improve on attempt three, while a transient network failure might. Retrying terminal work consumes capacity and delays healthy jobs.
Idempotency is checked before insert
Read-then-create races. A unique constraint and conflict handling, not an earlier SELECT, makes concurrent duplicates harmless.
Keep the DIY version when
- Jobs are manual, single-process and no effect occurs outside the local filesystem.
- A duplicate artifact and duplicate notification are acceptable and billing is not attached to completion.
What you get, and what it costs
Output, limits, cost, failures
Output
One artifact and one billable ledger entry for a successful non-cached request, even if queue delivery or a caller retry occurs more than once.
Cost
One successful, non-cached capture costs one credit. Failures, including platform failures, cost zero; cache hits cost zero.
1 creditper successful non-cached capture
Limits
- Idempotency prevents duplicate effects; it does not guarantee exactly-once execution at the transport layer.
- Retryable errors are bounded by a maximum attempt count and scheduled with backoff.
- Webhook delivery has its own stable event id because capture success and callback success are separate effects.
Common errors
concurrency_limit_reachednever billed
Admission exceeds the plan's start bucket; honor `Retry-After`.
HTTP 400 · retryable
temporary_unavailablenever billed
The platform cannot safely admit work at that moment.
HTTP 503 · retryable
Try it against your own page
The runner above carries this exact configuration into the playground — no retyping, no starting over.
Tested and reviewed
Written by PageCapture Engineering. Reviewed by PageCapture data and worker maintainers. Verified 2026-08-12 by running the published code against a fixture.
Contract tests inspect atomic bulk admission and idempotency ordering; billing tests assert no error is chargeable; the executor lease and unique-ledger design are linked to repository implementation.
Sources and verification basis
Sources support the browser and API behaviors named above. PageCapture-specific limits and billing are taken from the public contract; external sources are used for the underlying browser behavior.
- INSERT — ON CONFLICTPostgreSQL
Defines the atomic uniqueness primitive used for idempotent effects.
- Async and webhooksPageCapture
Documents retry timing and stable webhook event ids.
- Bulk capturesPageCapture
Documents bulk `Idempotency-Key` replay semantics.