Classify before retrying

Retries are useful only when a later attempt can plausibly succeed without configuration changes. Transient network failures, timeouts, rate limits, and retryable provider or server responses fit that category. Authentication, authorization, invalid requests, disallowed models, and unsupported structured-output schemas do not.

Treating every error as retryable increases latency and can multiply paid calls. Treating every error as terminal turns brief provider disruption into manual work. The executor boundary should map provider-specific failures into a small operational classification before the queue decides what to do.

Make local completion idempotent

Queue delivery is commonly at least once. A handler can be redelivered after a timeout or process restart, so terminal database writes must recognize that a run has already completed. A redelivered job should not overwrite a succeeded or failed terminal record or create duplicate trace trees.

Scheduled occurrence creation needs a similar guarantee. Runovio uses a unique schedule and scheduled-time pair so duplicate trigger delivery cannot create two local runs for the same occurrence. If an earlier run for that schedule is still queued or running, a new occurrence is skipped rather than turned into a misleading failure.

Document the crash window

Local idempotency cannot make an external model call and a PostgreSQL transaction atomic. A worker can receive a provider response and crash before it commits success. The queue then retries, and the provider may perform the inference again. This is a real duplicate paid-call window.

The honest operational contract is effectively-once local terminal state, not exactly-once external inference. Attempts, provider identifiers where permitted, latency, and estimated cost help operators identify the behavior. Retry counts should be bounded, and runbooks should avoid blindly re-triggering a run whose state is still uncertain.

See error and retry monitoring, the runtime guide, and scheduled agents for the surrounding execution model.