From 238689ddcc233442606bb2f0d5316c86463739b0 Mon Sep 17 00:00:00 2001 From: Keysat Date: Mon, 15 Jun 2026 18:15:00 -0500 Subject: [PATCH] Persist payment-webhook dedup; declare BTCPay required; scope CORS MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Replace the in-memory dedup Sets in the BTCPay and Zaprite webhook handlers (and the BTCPay rescan path) with a persistent JSON-backed store (server/webhook-dedup.js). The in-memory sets were cleared on restart, so a duplicate webhook delivery straddling a relay restart could double-credit (BTCPay) or double-extend a subscription (Zaprite). The store atomically writes /data/processed-webhooks.json, namespaces keys per rail (storeId|invoiceId vs zaprite:orderId), and prunes entries older than 180 days (safely beyond any retry window). Also: - BTCPay is a required running dependency (operator decision). Config was already optional:false/kind:'running'; corrected the contradictory "optional" comment in the manifest to match. - Scope cors() to /relay/* only — off /admin/* and the same-origin dashboard, which don't need permissive CORS. - Add money-path unit tests (commitCredit/refundCredit/applyTierPromotion) and webhook-dedup tests (incl. the survives-a-restart guarantee). - Fix two AGENTS.md auth-doc drifts; refresh Current state. Version 0.2.125 -> 0.2.126. --- AGENTS.md | 24 +++---- server/index.js | 8 ++- server/routes/credits.js | 36 +++++----- server/routes/zaprite-webhook.js | 24 +++---- server/test/money-path.test.js | 111 ++++++++++++++++++++++++++++++ server/test/webhook-dedup.test.js | 70 +++++++++++++++++++ server/webhook-dedup.js | 93 +++++++++++++++++++++++++ startos/manifest/index.ts | 18 ++--- startos/versions/index.ts | 5 +- startos/versions/v0.2.126.ts | 12 ++++ 10 files changed, 345 insertions(+), 56 deletions(-) create mode 100644 server/test/money-path.test.js create mode 100644 server/test/webhook-dedup.test.js create mode 100644 server/webhook-dedup.js create mode 100644 startos/versions/v0.2.126.ts diff --git a/AGENTS.md b/AGENTS.md index 5b8618f..19b4539 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -8,7 +8,7 @@ Operator-side, credit-metered service that sits in front of Gemini and the opera ## Stack - **Server**: Node.js (`type: module`, ES modules). Same dev box as the app (`v25.6.1`); container runtime is whatever the `Dockerfile` pins. -- **HTTP**: `express` + `multer` (audio upload). Admin routes under `/admin/*` behind an admin-session-cookie gate; relay-to-relay routes under `/relay/*` behind the operator key. +- **HTTP**: `express` + `multer` (audio upload). Admin routes under `/admin/*` behind an admin-session-cookie gate. `/relay/*` uses per-call header auth — install-id/license, or operator-key + user-id for the cloud control plane (a few routes like `health`/`policy`/`capabilities` are public). See the Auth model under Endpoints. `cors()` is scoped to `/relay/*` only. - **Dashboard**: `public/dashboard.html` — single-file vanilla JS, render-string-into-innerHTML, same shape as the app's `index.html`. - **Packaging**: `@start9labs/start-sdk` under `startos/` — version graph at `startos/versions/index.ts`. - **Storage**: filesystem under the StartOS data dir (`/data`). Internal meetings persist as `/data/internal-meetings/.json`. No SQLite here. @@ -62,7 +62,7 @@ All routes mount in `server/index.js`. Public paths sit under `/relay/*`; operat - **`X-Recap-Operator-Key`** + **`X-Recap-User-Id`** → "cloud" path. The Recaps cloud server (`recaps.cc`) authenticates once with a shared operator key (`relay_cloud_operator_key`) and names the acting user. Credit pool keyed `user:`, tier comes from the relay's stored row, NOT a per-user license. See `server/identity.js`. - **`X-Recap-Install-Id`** (+ optional `Authorization: `) → "license" path. Self-hosted installs and the operator's single-mode app. Credits/tier come from the resolved Keysat license + install id. -- **Admin session cookie** → `/admin/*`. Cookie issued by `POST /admin/login`; `/admin/login` and `/admin/status` are exempt inside `setupAdminAuthMiddleware`. +- **Admin session cookie** → `/admin/*`. Cookie issued by `POST /admin/login`; `/admin/login`, `/admin/status`, and `/admin/btcpay/callback` are exempt inside `setupAdminAuthMiddleware`. - **Webhook signature** → `POST /relay/btcpay/webhook` validates `BTCPay-Sig` against `relay_btcpay_webhook_secret`. Zaprite's webhook re-fetches the order through the Zaprite API to verify, so no shared-secret signing. - **`X-Recap-Job-Id`** is a billing key, not auth: the first call with a given id charges one credit; later calls with the same id are free (so transcribe + analyze for one summary = one credit total). @@ -141,14 +141,14 @@ this. When unsure whether a change is contract-affecting, assume it is and check - **Never edit a `startos/versions/.ts` that's already been built/installed** — add a new version file. - **Don't push to GitHub by default** — remote is self-hosted Gitea. -## Current state — post-eval security pass landed (2026-06-13) +## Current state — Users tab + webhook-dedup/P2 batch landed (2026-06-15) -- **Box, local tree, git aligned at relay `0.2.124`** (app `0.2.155`); `current: v_0_2_124`. Gitea remote `origin` now set up (`ssh://git@immense-voyage.local:59916/grant/recap-relay.git`); `master` pushed and tracking `origin/master`. Working tree clean. **Suite green at 60 tests** (`cd server && npm test`); server boots clean. -- **Full independent eval done** (evaluator + security-auditor + exerciser + doc-auditor + start9-spec-checker) → `EVALUATION.md` (overwritten in place each run, so re-running diffs cleanly). -- **All P0/P1 fixed** this session (commits `8ad7c54`/`d2caa98`/`3a601e1`): SSRF guard on caller-supplied media URLs (new `server/safe-url.js`), the early-renewal credit-reset money-leak (`extendUserTier`/`setUserTier` `resetCycle`), and the `multer`→`^2.0.1` DoS bump. None touch the `../recap` client contract. -- **Three P2 fixed** (commits `cbd9748`/`da1bba2`/`693d724`): meeting-`:id` path-traversal guard (`meetingPath()`), constant-time operator-key compare, and a JSON error handler that closes the malformed-body stack-trace leak. -- **Next (open P2), in priority order:** - 1. Persist webhook dedup so a restart can't double-credit/double-extend — `routes/credits.js:63`, `zaprite-webhook.js:27`. - 2. **Needs operator decision:** is BTCPay a hard requirement or truly optional? It's `optional:false`/`kind:'running'` despite "optional" comments, so StartOS won't start the relay without BTCPay co-installed — `startos/manifest/index.ts:38-49` + `dependencies.ts`. Then make manifest/deps/comment agree. - 3. Money-path unit tests (`commitCredit`/`refundCredit`/`applyTierPromotion`/grant handlers); scope `cors()` off `/admin/*` (`index.js`); split the 2225-line `routes/internal-meetings.js`; fix the two AGENTS.md auth-doc drifts (Stack-line `/relay/*` auth; missing `/admin/btcpay/callback` exempt path). -- **Risks/notes:** SSRF guard leaves a DNS-rebinding TOCTOU open (acceptable for a private box; revisit if exposed). P3+ deferred tail + pre-existing speaker-tool/empty-section backlog → `ROADMAP.md` / `docs/issues-backlog.md`. +- **Box, local tree, git aligned at relay `0.2.126`** (app `0.2.155`); `current: v_0_2_126`. Gitea remote `origin` (`ssh://git@immense-voyage.local:59916/grant/recap-relay.git`); `master` tracks `origin/master`. Working tree clean. **Suite green at 79 tests** (`cd server && npm test`); server boots clean. +- **Users dashboard tab** (`0.2.125`): new cookie-gated tab — every credit-ledger row (typed cloud/license/install) with computed remaining/total balances, key filter, and a per-row "grant free credits" action. `GET /admin/credits` (enriched read) + `POST /admin/credits/grant {credit_key, amount}` (free top-up via `addPurchasedCredits`, guards: positive int ≤1M, must be an existing row). Admin-only; no `../recap` contract change. +- **Webhook dedup now persistent** (`0.2.126`): new `server/webhook-dedup.js` (JSON store at `/data/processed-webhooks.json`, atomic writes, 180-day prune) replaces the in-memory Sets in `routes/credits.js` + `zaprite-webhook.js` (and the rescan path) — a duplicate delivery straddling a restart can no longer double-credit/double-extend. Keys namespaced `|` vs `zaprite:`. +- **BTCPay is REQUIRED** (operator decision, 2026-06-15): config was already `optional:false`/`kind:'running'`; corrected the contradictory "optional" comment in `startos/manifest/index.ts`. It's the only paid rail, so the relay shouldn't run without it. +- **CORS scoped to `/relay/*`** (`index.js`) — off `/admin/*` + dashboard (same-origin). Plus money-path unit tests (`commitCredit`/`refundCredit`/`applyTierPromotion`) and the two AGENTS.md auth-doc drift fixes. +- **Next (open P2 / deferred):** + 1. Split the 2225-line `routes/internal-meetings.js` — **deferred as likely overkill** for a private service; do only if it becomes painful to work in. + 2. P3+ deferred tail (no `/relay/*` rate limiting, container-as-root, dashboard `innerHTML` XSS surface, prune 126 version files, `/relay/health` stale `0.2.11`, etc.) + speaker-tool/empty-section backlog → `ROADMAP.md` / `docs/issues-backlog.md`. +- **Risks/notes:** webhook dedup keeps the pre-existing check-then-mark race for *truly simultaneous* duplicate deliveries (vanishingly rare on a private box; would need locking). SSRF guard leaves a DNS-rebinding TOCTOU open (acceptable for a private box). Full prior eval → `EVALUATION.md`. diff --git a/server/index.js b/server/index.js index 1ae1d1d..077910a 100644 --- a/server/index.js +++ b/server/index.js @@ -17,6 +17,7 @@ import { initCredits } from "./credits.js"; import { initAuditLog } from "./audit-log.js"; import { initJobCredits } from "./job-credits.js"; import { initOutputStore } from "./output-store.js"; +import { initWebhookDedup } from "./webhook-dedup.js"; import { setupAdminAuthMiddleware, setupAdminAuthRoutes, @@ -49,9 +50,14 @@ await initCredits({ dataDir: DATA_DIR }); await initJobCredits({ dataDir: DATA_DIR }); await initAuditLog({ dataDir: DATA_DIR }); await initOutputStore({ dataDir: DATA_DIR }); +await initWebhookDedup({ dataDir: DATA_DIR }); const app = express(); -app.use(cors()); +// CORS only on /relay/* — those are the cross-origin clients (the Recaps +// app + cloud server). /admin/* and the dashboard are served same-origin +// to the operator's browser, so they don't need (and shouldn't get) the +// permissive Access-Control-Allow-Origin a global cors() would apply. +app.use("/relay", cors()); app.use(cookieParser()); // Admin auth must run BEFORE the admin routes register so the cookie diff --git a/server/routes/credits.js b/server/routes/credits.js index f82143b..94b9328 100644 --- a/server/routes/credits.js +++ b/server/routes/credits.js @@ -49,18 +49,16 @@ import { BtcPayError, } from "../btcpay-client.js"; import { recordCall } from "../audit-log.js"; +import { isWebhookProcessed, markWebhookProcessed } from "../webhook-dedup.js"; import { envelope, errorEnvelope } from "./envelope.js"; -// In-memory dedup of processed BTCPay invoice ids. BTCPay retries -// webhook deliveries on non-2xx responses or network errors, so the -// same InvoiceSettled event can land more than once. We don't want -// to grant credits twice for one paid invoice. -// -// Cleared on relay restart, which means an unlucky webhook duplicate -// straddling a restart would double-credit. Acceptable tradeoff for -// v1 (operator can manually adjust the ledger if it happens), but -// worth swapping for a persistent set once the relay sees real load. -const processedInvoices = new Set(); +// Dedup of processed BTCPay invoice ids. BTCPay retries webhook +// deliveries on non-2xx responses or network errors, so the same +// InvoiceSettled event can land more than once — we don't want to grant +// credits twice for one paid invoice. Backed by the persistent +// webhook-dedup store (../webhook-dedup.js) so a duplicate straddling a +// relay restart can't double-credit. Keys are namespaced +// `|` to share the store with the Zaprite rail. export function creditsRouter() { const router = express.Router(); @@ -452,7 +450,7 @@ export function creditsRouter() { } const dedupKey = `${cfg.relay_btcpay_store_id}|${invoiceId}`; - if (processedInvoices.has(dedupKey)) { + if (isWebhookProcessed(dedupKey)) { return res .status(200) .json({ ok: true, ignored: "already_processed", invoiceId }); @@ -487,7 +485,7 @@ export function creditsRouter() { const subTier = meta.tier; const periodDays = Number(meta.period_days) || 30; if (!subUserId || (subTier !== "pro" && subTier !== "max")) { - processedInvoices.add(dedupKey); + await markWebhookProcessed(dedupKey); return res .status(200) .json({ ok: true, ignored: "bad_tier_metadata", invoiceId }); @@ -498,7 +496,7 @@ export function creditsRouter() { tier: subTier, periodDays, }); - processedInvoices.add(dedupKey); + await markWebhookProcessed(dedupKey); console.log( `[btcpay/webhook] ${subTier} +${periodDays}d for user ${subUserId.slice(0, 8)}… (invoice ${invoiceId}) → expires ${row.subscription_expires_at}`, ); @@ -529,7 +527,7 @@ export function creditsRouter() { // Not ours — could be an invoice from another product on the // same store. Mark processed so we don't keep retrying, and // 200 so BTCPay stops calling. - processedInvoices.add(dedupKey); + await markWebhookProcessed(dedupKey); return res .status(200) .json({ ok: true, ignored: "no_recap_metadata" }); @@ -541,7 +539,7 @@ export function creditsRouter() { creditKey, amount: credits, }); - processedInvoices.add(dedupKey); + await markWebhookProcessed(dedupKey); await recordCall({ install_id: installId, license_fingerprint: buyerFp, @@ -614,8 +612,8 @@ function rewriteCheckoutUrl(url, browserBase) { // Recovery: when the BTCPay webhook URL was broken and paid // invoices never got credited, this scans BTCPay's recent settled // invoices and grants credits for ones the relay hasn't processed. -// Idempotent via the same processedInvoices dedup the webhook uses, -// so re-running is safe. +// Idempotent via the same persistent webhook-dedup store the webhook +// uses, so re-running is safe. // // Exported so the admin route in btcpay-setup.js can call it. Not // exposed via /relay/* because it's operator-initiated, not buyer. @@ -660,7 +658,7 @@ export async function rescanSettledInvoices() { const details = []; for (const invoice of invoices || []) { const dedupKey = `${cfg.relay_btcpay_store_id}|${invoice.id}`; - if (processedInvoices.has(dedupKey)) { + if (isWebhookProcessed(dedupKey)) { alreadyProcessed++; continue; } @@ -684,7 +682,7 @@ export async function rescanSettledInvoices() { creditKey, amount: credits, }); - processedInvoices.add(dedupKey); + await markWebhookProcessed(dedupKey); await recordCall({ install_id: installId, license_fingerprint: buyerFp, diff --git a/server/routes/zaprite-webhook.js b/server/routes/zaprite-webhook.js index 154926e..e3ad814 100644 --- a/server/routes/zaprite-webhook.js +++ b/server/routes/zaprite-webhook.js @@ -15,16 +15,14 @@ import express from "express"; import { extendUserTier } from "../credits.js"; import { getZapriteConfig } from "../config.js"; import { getOrder, isOrderPaid, orderIdFromWebhook } from "../zaprite-client.js"; +import { isWebhookProcessed, markWebhookProcessed } from "../webhook-dedup.js"; -// In-memory dedup of fully-processed orders (mirrors the BTCPay handler's -// processedInvoices). Zaprite retries on non-200, and may fire multiple -// events per order, so we guard the grant. Cleared on restart — a -// re-delivered webhook after restart re-fetches + re-grants, but -// extendUserTier is keyed per order via this set within a process; a -// duplicate grant across a restart is the same harmless "extend by one -// period" the operator-comp path already tolerates. (Acceptable: card -// double-fires across a restart are vanishingly rare.) -const processedZaprite = new Set(); +// Dedup of fully-processed orders (mirrors the BTCPay handler). Zaprite +// retries on non-200, and may fire multiple events per order, so we +// guard the grant. Backed by the persistent webhook-dedup store +// (../webhook-dedup.js) so a re-delivered webhook straddling a relay +// restart can't double-extend a subscription. Keys are namespaced +// `zaprite:` to share the store with the BTCPay rail. export function zapriteWebhookRouter() { const router = express.Router(); @@ -49,7 +47,7 @@ export function zapriteWebhookRouter() { return res.status(200).json({ ok: true, ignored: "no_order_id" }); } const dedupKey = `zaprite:${orderId}`; - if (processedZaprite.has(dedupKey)) { + if (isWebhookProcessed(dedupKey)) { return res .status(200) .json({ ok: true, ignored: "already_processed", orderId }); @@ -91,7 +89,7 @@ export function zapriteWebhookRouter() { const meta = order.metadata || {}; if (meta.product !== "recap_tier_subscription") { - processedZaprite.add(dedupKey); + await markWebhookProcessed(dedupKey); return res .status(200) .json({ ok: true, ignored: "not_a_tier_order", orderId }); @@ -100,7 +98,7 @@ export function zapriteWebhookRouter() { const subTier = meta.tier; const periodDays = Number(meta.period_days) || 30; if (!subUserId || (subTier !== "pro" && subTier !== "max")) { - processedZaprite.add(dedupKey); + await markWebhookProcessed(dedupKey); return res .status(200) .json({ ok: true, ignored: "bad_tier_metadata", orderId }); @@ -112,7 +110,7 @@ export function zapriteWebhookRouter() { tier: subTier, periodDays, }); - processedZaprite.add(dedupKey); + await markWebhookProcessed(dedupKey); console.log( `[zaprite/webhook] ${subTier} +${periodDays}d for user ${subUserId.slice(0, 8)}… (order ${orderId}) → expires ${row.subscription_expires_at}`, ); diff --git a/server/test/money-path.test.js b/server/test/money-path.test.js new file mode 100644 index 0000000..e651429 --- /dev/null +++ b/server/test/money-path.test.js @@ -0,0 +1,111 @@ +// Core billing primitives: commitCredit (debit), refundCredit (inverse), +// and applyTierPromotion (the Core→paid upgrade bookkeeping). Default +// quotas (no config file → getTierQuotas falls back): Core lifetime=10, +// geminiCapLifetime=5; Pro monthly=50, geminiCapMonthly=25. + +import { test, describe, before } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { + initCredits, + getOrCreateRow, + getUserCreditRow, + setUserTier, + commitCredit, + refundCredit, + applyTierPromotion, +} from "../credits.js"; + +before(async () => { + await initCredits({ dataDir: mkdtempSync(path.join(tmpdir(), "relay-money-")) }); +}); + +describe("commitCredit", () => { + test("Core debits the lifetime bucket; Gemini tracked separately", async () => { + const creditKey = "inst:core-commit"; + await commitCredit({ creditKey, tier: "core", backend: "gemini" }); + let row = await getOrCreateRow({ creditKey }); + assert.equal(row.lifetime_consumed, 1); + assert.equal(row.lifetime_gemini_consumed, 1); + + // A hardware call bumps lifetime but NOT the Gemini sub-counter. + await commitCredit({ creditKey, tier: "core", backend: "hardware" }); + row = await getOrCreateRow({ creditKey }); + assert.equal(row.lifetime_consumed, 2); + assert.equal(row.lifetime_gemini_consumed, 1); + }); + + test("paid tier debits the monthly bucket", async () => { + await setUserTier({ userId: "p-commit", tier: "pro" }); + await commitCredit({ creditKey: "user:p-commit", tier: "pro", backend: "gemini" }); + const row = await getUserCreditRow("p-commit"); + assert.equal(row.monthly_consumed, 1); + assert.equal(row.monthly_gemini_consumed, 1); + }); + + test("spend order: once the tier bucket is exhausted, debit purchased", async () => { + const creditKey = "inst:core-exhausted"; + const row = await getOrCreateRow({ creditKey }); + row.lifetime_consumed = 10; // at the Core lifetime cap + row.purchased_balance = 3; + await commitCredit({ creditKey, tier: "core", backend: "hardware" }); + const after = await getOrCreateRow({ creditKey }); + assert.equal(after.lifetime_consumed, 10, "tier counter must not exceed the cap"); + assert.equal(after.purchased_balance, 2, "overflow comes out of purchased"); + }); +}); + +describe("refundCredit", () => { + test("mirrors a Core commit (tier bucket first, Gemini sub-counter too)", async () => { + const creditKey = "inst:core-refund"; + await commitCredit({ creditKey, tier: "core", backend: "gemini" }); + await refundCredit({ creditKey, tier: "core", backend: "gemini" }); + const row = await getOrCreateRow({ creditKey }); + assert.equal(row.lifetime_consumed, 0); + assert.equal(row.lifetime_gemini_consumed, 0); + }); + + test("refund with an empty tier bucket credits the purchased bucket", async () => { + const creditKey = "inst:refund-to-purchased"; + const row = await getOrCreateRow({ creditKey }); + row.lifetime_consumed = 0; + row.purchased_balance = 0; + await refundCredit({ creditKey, tier: "core", backend: "hardware" }); + const after = await getOrCreateRow({ creditKey }); + assert.equal(after.lifetime_consumed, 0, "must floor at 0, not go negative"); + assert.equal(after.purchased_balance, 1, "refund lands in purchased when tier is already 0"); + }); +}); + +describe("applyTierPromotion", () => { + test("Core→paid transfers leftover Core credits to purchased and resets the cycle", async () => { + const row = await getOrCreateRow({ creditKey: "inst:promo" }); + row.lifetime_consumed = 4; // 6 of 10 Core credits unused + row.monthly_consumed = 2; // stale paid-counter noise that must be zeroed + const promoted = await applyTierPromotion(row, "pro"); + assert.equal(promoted, true); + assert.equal(row.tier_snapshot, "pro"); + assert.equal(row.purchased_balance, 6, "leftover Core credits carry forward as durable top-up"); + assert.equal(row.purchased_total_ever, 6); + assert.equal(row.monthly_consumed, 0, "promotion starts a fresh monthly cycle"); + }); + + test("idempotent: a second promotion on an already-paid row is a no-op", async () => { + const row = await getOrCreateRow({ creditKey: "inst:promo-again" }); + row.lifetime_consumed = 4; + await applyTierPromotion(row, "pro"); // first: fires + const purchasedAfterFirst = row.purchased_balance; + const promoted = await applyTierPromotion(row, "max"); // second: must bail + assert.equal(promoted, false); + assert.equal(row.purchased_balance, purchasedAfterFirst, "no second leftover transfer"); + assert.equal(row.tier_snapshot, "pro", "bails before flipping tier again"); + }); + + test("promoting to Core is a no-op", async () => { + const row = await getOrCreateRow({ creditKey: "inst:promo-core" }); + assert.equal(await applyTierPromotion(row, "core"), false); + }); +}); diff --git a/server/test/webhook-dedup.test.js b/server/test/webhook-dedup.test.js new file mode 100644 index 0000000..5656456 --- /dev/null +++ b/server/test/webhook-dedup.test.js @@ -0,0 +1,70 @@ +// Persistent webhook dedup — the store that stops a settled-payment +// webhook duplicate from double-crediting (BTCPay) or double-extending +// (Zaprite) when the duplicate straddles a relay restart. + +import { test, describe } from "node:test"; +import assert from "node:assert/strict"; +import { mkdtempSync, writeFileSync } from "node:fs"; +import { tmpdir } from "node:os"; +import path from "node:path"; + +import { + initWebhookDedup, + isWebhookProcessed, + markWebhookProcessed, +} from "../webhook-dedup.js"; + +// The module is a singleton; each test inits a fresh dir to reset state. +const freshDir = () => mkdtempSync(path.join(tmpdir(), "relay-dedup-")); + +describe("webhook dedup (persistent)", () => { + test("unknown key is not processed; marking makes it processed", async () => { + await initWebhookDedup({ dataDir: freshDir() }); + assert.equal(isWebhookProcessed("store1|inv1"), false); + await markWebhookProcessed("store1|inv1"); + assert.equal(isWebhookProcessed("store1|inv1"), true); + }); + + // Headline guarantee for this change. + test("processed keys survive a restart (reload from disk)", async () => { + const dir = freshDir(); + await initWebhookDedup({ dataDir: dir }); + await markWebhookProcessed("store1|inv-restart"); + // Simulate a relay restart: re-init from the SAME data dir. + await initWebhookDedup({ dataDir: dir }); + assert.equal( + isWebhookProcessed("store1|inv-restart"), + true, + "a key marked before restart must still be deduped after restart" + ); + }); + + test("BTCPay and Zaprite keys share the store without colliding", async () => { + await initWebhookDedup({ dataDir: freshDir() }); + await markWebhookProcessed("store1|inv9"); + await markWebhookProcessed("zaprite:order9"); + assert.equal(isWebhookProcessed("store1|inv9"), true); + assert.equal(isWebhookProcessed("zaprite:order9"), true); + assert.equal(isWebhookProcessed("zaprite:inv9"), false); + }); + + test("marking is idempotent", async () => { + await initWebhookDedup({ dataDir: freshDir() }); + await markWebhookProcessed("k"); + await markWebhookProcessed("k"); // must not throw or duplicate + assert.equal(isWebhookProcessed("k"), true); + }); + + test("stale entries past the retention window are pruned on load", async () => { + const dir = freshDir(); + const ancient = new Date(Date.now() - 365 * 24 * 60 * 60 * 1000).toISOString(); + const recent = new Date().toISOString(); + writeFileSync( + path.join(dir, "processed-webhooks.json"), + JSON.stringify({ keys: { "store1|old": ancient, "store1|new": recent } }) + ); + await initWebhookDedup({ dataDir: dir }); + assert.equal(isWebhookProcessed("store1|old"), false, "1y-old key should be pruned"); + assert.equal(isWebhookProcessed("store1|new"), true, "recent key should survive"); + }); +}); diff --git a/server/webhook-dedup.js b/server/webhook-dedup.js new file mode 100644 index 0000000..ec0214c --- /dev/null +++ b/server/webhook-dedup.js @@ -0,0 +1,93 @@ +// Persistent dedup of settled-payment webhook deliveries — BTCPay +// invoices AND Zaprite orders share this one store. It replaces the +// per-process in-memory Sets the two webhook handlers used to keep: +// those were cleared on restart, so a duplicate delivery that straddled +// a relay restart would double-credit (BTCPay) or double-extend a +// subscription (Zaprite). Persisting the processed-keys to disk closes +// that window. +// +// JSON-file backed (single file at /processed-webhooks.json), +// same rationale as the credit ledger in credits.js: at most one +// mutation per settled payment, so a plain JSON file with serial writes +// is plenty. +// +// Keys are namespaced by their caller so the two rails can't collide in +// the shared store: BTCPay uses `|`, Zaprite uses +// `zaprite:`. We store key → ISO timestamp (not a bare set) so +// entries far older than any processor's retry window can be pruned on +// load, keeping the file bounded. + +import fs from "fs/promises"; +import path from "path"; + +// Payment processors retry webhook delivery for hours-to-days, never +// months. 180 days is safely beyond any retry window, so pruning past +// it cannot re-open a double-grant gap. +const RETENTION_MS = 180 * 24 * 60 * 60 * 1000; + +let storePath = null; +let processed = {}; // key → ISO timestamp +let writing = null; // serializes concurrent writes + +export async function initWebhookDedup({ dataDir } = {}) { + const dir = dataDir || "/data"; + storePath = path.join(dir, "processed-webhooks.json"); + await fs.mkdir(dir, { recursive: true }).catch(() => {}); + try { + const raw = await fs.readFile(storePath, "utf8"); + const parsed = JSON.parse(raw); + processed = + parsed && typeof parsed === "object" && parsed.keys ? parsed.keys : {}; + } catch (err) { + if (err.code !== "ENOENT") { + console.warn( + `[webhook-dedup] failed to read ${storePath}: ${err.message} — starting empty` + ); + } + processed = {}; + } + // Prune anything older than the retention window so the file stays + // bounded over the relay's lifetime. + const cutoff = Date.now() - RETENTION_MS; + let pruned = 0; + for (const [k, ts] of Object.entries(processed)) { + const t = Date.parse(ts); + if (!Number.isFinite(t) || t < cutoff) { + delete processed[k]; + pruned += 1; + } + } + if (pruned > 0) await persist(); + console.log( + `[webhook-dedup] loaded ${Object.keys(processed).length} processed key(s)` + + `${pruned ? `, pruned ${pruned} stale` : ""} from ${storePath}` + ); +} + +async function persist() { + // Coalesce concurrent writes — mirrors credits.js persist(). + if (writing) await writing; + writing = (async () => { + const tmp = storePath + ".tmp"; + await fs.writeFile(tmp, JSON.stringify({ keys: processed }), { mode: 0o600 }); + await fs.rename(tmp, storePath); + })(); + try { + await writing; + } finally { + writing = null; + } +} + +export function isWebhookProcessed(key) { + return Object.prototype.hasOwnProperty.call(processed, key); +} + +// Mark a webhook key processed and persist immediately. Callers invoke +// this AFTER the grant/extend succeeds, so a failed grant leaves the key +// unmarked and a processor retry can complete it. Idempotent. +export async function markWebhookProcessed(key) { + if (isWebhookProcessed(key)) return; + processed[key] = new Date().toISOString(); + await persist(); +} diff --git a/startos/manifest/index.ts b/startos/manifest/index.ts index 483e6f0..3eff92d 100644 --- a/startos/manifest/index.ts +++ b/startos/manifest/index.ts @@ -31,15 +31,15 @@ export const manifest = setupManifest({ start: null, stop: null, }, - // Relay has no REQUIRED dependencies — Gemini is internet-fronted - // and the optional Parakeet/Gemma backends are at user-configured - // URLs (typically a separate machine on the operator's LAN). - // - // BTCPay Server is declared optional so the dashboard's "Connect - // BTCPay" flow can auto-discover its URL via - // sdk.serviceInterface.getAll() when both are installed on the - // same Start9 box. When BTCPay is not installed, the relay still - // runs fine — only the credit-purchase flow is disabled. + // BTCPay Server is a REQUIRED running dependency (optional: false + // here, kind: 'running' in dependencies.ts). It's the only payment + // rail through which the operator actually gets paid, so the relay + // shouldn't run without it. The dependency also wires up the internal + // docker hostname (btcpayserver.startos:23000) the relay-to-BTCPay API + // calls rely on, and lets the dashboard's "Connect BTCPay" flow + // auto-discover the URL via sdk.serviceInterface.getAll(). (Gemini is + // internet-fronted and the Parakeet/Gemma backends are at + // user-configured LAN URLs, so neither is a StartOS dependency.) dependencies: { btcpayserver: { description: { diff --git a/startos/versions/index.ts b/startos/versions/index.ts index 4e6d213..0f9d8f4 100644 --- a/startos/versions/index.ts +++ b/startos/versions/index.ts @@ -126,8 +126,9 @@ import { v_0_2_122 } from './v0.2.122' import { v_0_2_123 } from './v0.2.123' import { v_0_2_124 } from './v0.2.124' import { v_0_2_125 } from './v0.2.125' +import { v_0_2_126 } from './v0.2.126' export const versionGraph = VersionGraph.of({ - current: v_0_2_125, - other: [v_0_2_124, v_0_2_123, v_0_2_122, v_0_2_121, v_0_2_120, v_0_2_119, v_0_2_118, v_0_2_117, v_0_2_116, v_0_2_115, v_0_2_114, v_0_2_113, v_0_2_112, v_0_2_111, v_0_2_110, v_0_2_109, v_0_2_108, v_0_2_107, v_0_2_106, v_0_2_105, v_0_2_104, v_0_2_103, v_0_2_102, v_0_2_101, v_0_2_100, v_0_2_99, v_0_2_98, v_0_2_97, v_0_2_96, v_0_2_95, v_0_2_94, v_0_2_93, v_0_2_92, v_0_2_91, v_0_2_90, v_0_2_89, v_0_2_88, v_0_2_87, v_0_2_86, v_0_2_85, v_0_2_84, v_0_2_83, v_0_2_82, v_0_2_81, v_0_2_80, v_0_2_79, v_0_2_78, v_0_2_77, v_0_2_76, v_0_2_75, v_0_2_74, v_0_2_73, v_0_2_72, v_0_2_71, v_0_2_70, v_0_2_69, v_0_2_68, v_0_2_67, v_0_2_66, v_0_2_65, v_0_2_64, v_0_2_63, v_0_2_62, v_0_2_61, v_0_2_60, v_0_2_59, v_0_2_58, v_0_2_57, v_0_2_56, v_0_2_55, v_0_2_54, v_0_2_53, v_0_2_52, v_0_2_51, v_0_2_50, v_0_2_49, v_0_2_48, v_0_2_47, v_0_2_46, v_0_2_45, v_0_2_44, v_0_2_43, v_0_2_42, v_0_2_41, v_0_2_40, v_0_2_39, v_0_2_38, v_0_2_37, v_0_2_36, v_0_2_35, v_0_2_34, v_0_2_33, v_0_2_32, v_0_2_31, v_0_2_30, v_0_2_29, v_0_2_28, v_0_2_27, v_0_2_26, v_0_2_25, v_0_2_24, v_0_2_23, v_0_2_22, v_0_2_21, v_0_2_20, v_0_2_19, v_0_2_18, v_0_2_17, v_0_2_16, v_0_2_15, v_0_2_14, v_0_2_13, v_0_2_12, v_0_2_11, v_0_2_10, v_0_2_9, v_0_2_8, v_0_2_7, v_0_2_6, v_0_2_5, v_0_2_4, v_0_2_3, v_0_2_2, v_0_2_1, v_0_2_0, v_0_1_0], + current: v_0_2_126, + other: [v_0_2_125, v_0_2_124, v_0_2_123, v_0_2_122, v_0_2_121, v_0_2_120, v_0_2_119, v_0_2_118, v_0_2_117, v_0_2_116, v_0_2_115, v_0_2_114, v_0_2_113, v_0_2_112, v_0_2_111, v_0_2_110, v_0_2_109, v_0_2_108, v_0_2_107, v_0_2_106, v_0_2_105, v_0_2_104, v_0_2_103, v_0_2_102, v_0_2_101, v_0_2_100, v_0_2_99, v_0_2_98, v_0_2_97, v_0_2_96, v_0_2_95, v_0_2_94, v_0_2_93, v_0_2_92, v_0_2_91, v_0_2_90, v_0_2_89, v_0_2_88, v_0_2_87, v_0_2_86, v_0_2_85, v_0_2_84, v_0_2_83, v_0_2_82, v_0_2_81, v_0_2_80, v_0_2_79, v_0_2_78, v_0_2_77, v_0_2_76, v_0_2_75, v_0_2_74, v_0_2_73, v_0_2_72, v_0_2_71, v_0_2_70, v_0_2_69, v_0_2_68, v_0_2_67, v_0_2_66, v_0_2_65, v_0_2_64, v_0_2_63, v_0_2_62, v_0_2_61, v_0_2_60, v_0_2_59, v_0_2_58, v_0_2_57, v_0_2_56, v_0_2_55, v_0_2_54, v_0_2_53, v_0_2_52, v_0_2_51, v_0_2_50, v_0_2_49, v_0_2_48, v_0_2_47, v_0_2_46, v_0_2_45, v_0_2_44, v_0_2_43, v_0_2_42, v_0_2_41, v_0_2_40, v_0_2_39, v_0_2_38, v_0_2_37, v_0_2_36, v_0_2_35, v_0_2_34, v_0_2_33, v_0_2_32, v_0_2_31, v_0_2_30, v_0_2_29, v_0_2_28, v_0_2_27, v_0_2_26, v_0_2_25, v_0_2_24, v_0_2_23, v_0_2_22, v_0_2_21, v_0_2_20, v_0_2_19, v_0_2_18, v_0_2_17, v_0_2_16, v_0_2_15, v_0_2_14, v_0_2_13, v_0_2_12, v_0_2_11, v_0_2_10, v_0_2_9, v_0_2_8, v_0_2_7, v_0_2_6, v_0_2_5, v_0_2_4, v_0_2_3, v_0_2_2, v_0_2_1, v_0_2_0, v_0_1_0], }) diff --git a/startos/versions/v0.2.126.ts b/startos/versions/v0.2.126.ts new file mode 100644 index 0000000..6f6d9b0 --- /dev/null +++ b/startos/versions/v0.2.126.ts @@ -0,0 +1,12 @@ +import { VersionInfo } from '@start9labs/start-sdk' + +export const v_0_2_126 = VersionInfo.of({ + version: '0.2.126:0', + releaseNotes: { + en_US: 'Persist payment-webhook dedup across restarts (no double-credit/double-extend); declare BTCPay a required dependency; scope CORS to /relay/*; add money-path + webhook-dedup tests; doc fixes', + }, + migrations: { + up: async ({ effects }) => {}, + down: async ({ effects }) => {}, + }, +})