duration
model
recoveries
files
tokens
redacted
An agent's session, shared by its author. “Use this in my agent” copies what it learned, to paste into your own agent. How it works
8 steps
Last night's payout job never finished. The logs stop at "fetching balance for sel_8KQ2" around 01:04 UTC and there's nothing after that - no error, no "payouts done" line - until we killed it at 07:00. The payments provider's status page shows a partial outage from 01:00 to 02:30 with API requests timing out. Every seller after that one didn't get paid. Can you make the job robust to this, so a stalled provider can't hang it? Add tests.Bashgit ls-files | grep -v node_modules | head -50 && cat package.json.gitignore
README.md
package.json
src/clients/payments.js
src/jobs/nightly-payouts.js
src/retry.js
test/payouts.test.js
{
"name": "payouts-worker",
"version": "2.1.0",
"private": true,
"type": "module",
"scripts": {
"payouts": "node src/jobs/nightly-payouts.js",
"test": "node --test"
},
"engines": {
"node": ">=20"
}
}Bashcat src/clients/payments.js src/jobs/nightly-payouts.js src/retry.js test/payouts.test.js README.mdimport { withRetry } from "../retry.js";
export class PaymentsApiError extends Error {
constructor(status, body) {
super(`payments API answered ${status}`);
this.name = "PaymentsApiError";
this.status = status;
this.body = body;
}
}
export function createPaymentsClient({ baseUrl = process.env.PAYMENTS_API_URL, apiKey = process.env.PAYMENTS_API_KEY } = {}) {
async function call(method, path, body) {
return withRetry(
async () => {
const response = await fetch(new URL(path, baseUrl), {
method,
headers: { authorization: `Bearer ${apiKey}`, "content-type": "application/json" },
body: body === undefined ? undefined : JSON.stringify(body),
});
const text = await response.text();
if (!response.ok) throw new PaymentsApiError(response.status, text);
return text ? JSON.parse(text) : null;
},
{ label: `${method} ${path}` }
);
}
return {
listSellers: () => call("GET", "/v1/sellers?status=active"),
getBalance: (sellerId) => call("GET", `/v1/sellers/${sellerId}/balance`),
createPayout: (sellerId, amount, currency, idempotencyKey) =>
call("POST", `/v1/sellers/${sellerId}/payouts`, { amount, currency, idempotency_key: idempotencyKey }),
};
}
import { createPaymentsClient } from "../clients/payments.js";
export async function runNightlyPayouts(client = createPaymentsClient(), today = new Date().toISOString().slice(0, 10)) {
const summary = { paid: 0, skipped: 0, failed: [] };
const { data: sellers } = await client.listSellers();
for (const seller of sellers) {
try {
console.log(`fetching balance for ${seller.id}`);
const balance = await client.getBalance(seller.id);
if (balance.available < 100) {
summary.skipped++;
continue;
}
await client.createPayout(seller.id, balance.available, balance.currency, `payout-${seller.id}-${today}`);
summary.paid++;
} catch (error) {
console.error(`payout for ${seller.id} failed: ${error.message}`);
summary.failed.push(seller.id);
}
}
console.log(`payouts done: ${summary.paid} paid, ${summary.skipped} skipped, ${summary.failed.length} failed`);
return summary;
}
if (import.meta.url === `file://${process.argv[1]}`) {
runNightlyPayouts().then((summary) => {
process.exitCode = summary.failed.length > 0 ? 1 : 0;
});
}
export async function withRetry(fn, { attempts = 3, baseDelayMs = 500, label = "request" } = {}) {
let lastError;
for (let attempt = 1; attempt <= attempts; attempt++) {
try {
return await fn();
} catch (error) {
lastError = error;
if (!isRetryable(error) || attempt === attempts) break;
const delay = baseDelayMs * 2 ** (attempt - 1);
console.warn(`${label} failed (${error.message}), retry ${attempt}/${attempts - 1} in ${delay}ms`);
await new Promise((resolve) => setTimeout(resolve, delay));
}
}
throw lastError;
}
function isRetryable(error) {
if (error.status) return error.status === 429 || error.status >= 500;
return error.name === "TypeError"; // fetch's network errors
}
import { test, after } from "node:test";
import assert from "node:assert/strict";
import http from "node:http";
import { createPaymentsClient } from "../src/clients/payments.js";
import { runNightlyPayouts } from "../src/jobs/nightly-payouts.js";
const payouts = [];
let balanceFailuresLeft = 0;
const server = http.createServer(async (req, res) => {
const chunks = [];
for await (const chunk of req) chunks.push(chunk);
const json = (status, body) => {
res.writeHead(status, { "content-type": "application/json" });
res.end(JSON.stringify(body));
};
if (req.url === "/v1/sellers?status=active") return json(200, { data: [{ id: "sel_a" }, { id: "sel_b" }, { id: "sel_c" }] });
const balance = req.url.match(/^\/v1\/sellers\/(\w+)\/balance$/);
if (balance) {
if (balanceFailuresLeft > 0) {
balanceFailuresLeft--;
…[shortened]Or pick a public run
Bashpython3 - <<'E' p='src/clients/payments.js' s=open(p).read() s=s.replace("apiKey = process.env.PAYMENTS_API_KEY } = {}"…fetching balance for sel_c
payouts done: 2 paid, 1 skipped, 0 failed
fetching balance for sel_1
✔ a 503 from the provider is retried (528.091546ms)
GET /v1/sellers/sel_1/balance failed (The operation was aborted due to timeout), retry 1/1 in 1ms
payout for sel_1 failed: The operation was aborted due to timeout
fetching balance for sel_2
GET /v1/sellers/sel_2/balance failed (The operation was aborted due to timeout), retry 1/1 in 1ms
payout for sel_2 failed: The operation was aborted due to timeout
aborting after 2 consecutive failures; not attempted: sel_3, sel_4
payouts done: 0 paid, 0 skipped, 4 failed
fetching balance for sel_a
fetching balance for sel_c
payout for sel_a failed: timed out
payouts done: 1 paid, 0 skipped, 1 failed
✔ a stalled provider times out instead of hanging the job (221.906367ms)
✔ a timeout on one seller doesn't stop later sellers once the provider recovers (1.273156ms)
ℹ tests 4
ℹ suites 0
ℹ pass 4
ℹ fail 0
ℹ cancelled 0
ℹ skipped 0
ℹ todo 0
ℹ duration_ms 1033.902342python3 - <<'E'
p='src/clients/payments.js'
s=open(p).read()
s=s.replace("apiKey = process.env.PAYMENTS_API_KEY } = {}","apiKey = process.env.PAYMENTS_API_KEY, timeoutMs = 10_000, retry = {} } = {}")
s=s.replace(""" body: body === undefined ? undefined : JSON.stringify(body),
});""",""" body: body === undefined ? undefined : JSON.stringify(body),
// covers connect, headers and body read, so a stalled provider can't hang us
signal: AbortSignal.timeout(timeoutMs),
});""")
s=s.replace("{ label: `${method} ${path}` }","{ label: `${method} ${path}`, ...retry }")
open(p,'w').write(s)
p='src/retry.js'
s=open(p).read()
s=s.replace(' return error.name === "TypeError"; // fetch\'s network errors',' // fetch\'s network errors and our per-request timeout (payout POSTs carry an idempotency key)\n return error.name === "TypeError" || error.name === "TimeoutError";')
open(p,'w').write(s)
p='src/jobs/nightly-payouts.js'
s=open(p).read()
s=s.replace("today = new Date().toISOString().slice(0, 10)) {","today = new Date().toISOString().slice(0, 10), { maxConsecutiveFailures = 5 } = {}) {")
s=s.replace(" for (const seller of sellers) {\n",""" let consecutiveFailures = 0;
for (const [index, seller] of sellers.entries()) {
if (consecutiveFailures >= maxConsecutiveFailures) {
// provider looks down: stop burning timeouts, report the rest as unpaid
const rest = sellers.slice(index).map((s) => s.id);
console.error(`aborting after ${consecutiveFailures} consecutive failures; not attempted: ${rest.join(", ")}`);
summary.failed.push(...rest);
break;
}
""")
s=s.replace(" summary.paid++;\n"," summary.paid++;\n consecutiveFailures = 0;\n")
s=s.replace(" summary.skipped++;\n"," summary.skipped++;\n consecutiveFailures = 0;\n")
s=s.replace(" summary.failed.push(seller.id);\n"," summary.failed.push(seller.id);\n consecutiveFailures++;\n")
s=s.replace(…[shortened]The job can no longer hang on a stalled provider. All 4 tests pass, including two new ones. Cause:fetchin payments.js had no timeout. When the provider accepted the connection and never answered,getBalanceforsel_8KQ2waited forever. That fits your logs: no error, no "payouts done" line, and nothing logged after that seller. Changes - Per-request timeout: every API call now has a 10s limit by default (timeoutMsoption). It covers reading the response body too. - Retries on timeout: retry.js now retries timeouts with the existing backoff. This is safe for payouts because each payout request carries an idempotency key. - Outage breaker: in nightly-payouts.js, the job stops after 5 consecutive failures (maxConsecutiveFailures). It logs which sellers were not attempted and adds them tofailed. Without this, a 90-minute outage could still take hours: each seller would burn about 3 attempts × 10s plus backoff. - Exit code: iflistSellersthrows, the job now logs the error and exits 1. Before, that would have been an unhandled rejection. New tests: one runs against a server that never responds. It checks that the job finishes quickly, trips the breaker, and reports all sellers as failed. The other checks that one seller's timeout doesn't stop later sellers. Still to do: the job doesn't re-run failed sellers by itself. Last night's unpaid sellers need a manual re-run. That should be safe because the idempotency key ispayout-<seller>-<date>, but the key's date comes from the run date, so a re-run today won't reuse last night's keys. Check that no seller was paid before the hang. A payout may have been created forsel_8KQ2or earlier sellers without the response being logged. I'd also have cron alert on a non-zero exit or a missing "payouts done" line.