Persist payment-webhook dedup; declare BTCPay required; scope CORS
Replace the in-memory dedup Sets in the BTCPay and Zaprite webhook handlers (and the BTCPay rescan path) with a persistent JSON-backed store (server/webhook-dedup.js). The in-memory sets were cleared on restart, so a duplicate webhook delivery straddling a relay restart could double-credit (BTCPay) or double-extend a subscription (Zaprite). The store atomically writes /data/processed-webhooks.json, namespaces keys per rail (storeId|invoiceId vs zaprite:orderId), and prunes entries older than 180 days (safely beyond any retry window). Also: - BTCPay is a required running dependency (operator decision). Config was already optional:false/kind:'running'; corrected the contradictory "optional" comment in the manifest to match. - Scope cors() to /relay/* only — off /admin/* and the same-origin dashboard, which don't need permissive CORS. - Add money-path unit tests (commitCredit/refundCredit/applyTierPromotion) and webhook-dedup tests (incl. the survives-a-restart guarantee). - Fix two AGENTS.md auth-doc drifts; refresh Current state. Version 0.2.125 -> 0.2.126.
This commit is contained in:
Keysat
@@ -8,7 +8,7 @@ Operator-side, credit-metered service that sits in front of Gemini and the opera
|
||||
## Stack
|
||||
|
||||
- **Server**: Node.js (`type: module`, ES modules). Same dev box as the app (`v25.6.1`); container runtime is whatever the `Dockerfile` pins.
|
||||
- **HTTP**: `express` + `multer` (audio upload). Admin routes under `/admin/*` behind an admin-session-cookie gate; relay-to-relay routes under `/relay/*` behind the operator key.
|
||||
- **HTTP**: `express` + `multer` (audio upload). Admin routes under `/admin/*` behind an admin-session-cookie gate. `/relay/*` uses per-call header auth — install-id/license, or operator-key + user-id for the cloud control plane (a few routes like `health`/`policy`/`capabilities` are public). See the Auth model under Endpoints. `cors()` is scoped to `/relay/*` only.
|
||||
- **Dashboard**: `public/dashboard.html` — single-file vanilla JS, render-string-into-innerHTML, same shape as the app's `index.html`.
|
||||
- **Packaging**: `@start9labs/start-sdk` under `startos/` — version graph at `startos/versions/index.ts`.
|
||||
- **Storage**: filesystem under the StartOS data dir (`/data`). Internal meetings persist as `/data/internal-meetings/<id>.json`. No SQLite here.
|
||||
@@ -62,7 +62,7 @@ All routes mount in `server/index.js`. Public paths sit under `/relay/*`; operat
|
||||
|
||||
- **`X-Recap-Operator-Key`** + **`X-Recap-User-Id`** → "cloud" path. The Recaps cloud server (`recaps.cc`) authenticates once with a shared operator key (`relay_cloud_operator_key`) and names the acting user. Credit pool keyed `user:<id>`, tier comes from the relay's stored row, NOT a per-user license. See `server/identity.js`.
|
||||
- **`X-Recap-Install-Id`** (+ optional `Authorization: <license>`) → "license" path. Self-hosted installs and the operator's single-mode app. Credits/tier come from the resolved Keysat license + install id.
|
||||
- **Admin session cookie** → `/admin/*`. Cookie issued by `POST /admin/login`; `/admin/login` and `/admin/status` are exempt inside `setupAdminAuthMiddleware`.
|
||||
- **Admin session cookie** → `/admin/*`. Cookie issued by `POST /admin/login`; `/admin/login`, `/admin/status`, and `/admin/btcpay/callback` are exempt inside `setupAdminAuthMiddleware`.
|
||||
- **Webhook signature** → `POST /relay/btcpay/webhook` validates `BTCPay-Sig` against `relay_btcpay_webhook_secret`. Zaprite's webhook re-fetches the order through the Zaprite API to verify, so no shared-secret signing.
|
||||
- **`X-Recap-Job-Id`** is a billing key, not auth: the first call with a given id charges one credit; later calls with the same id are free (so transcribe + analyze for one summary = one credit total).
|
||||
|
||||
@@ -141,14 +141,14 @@ this. When unsure whether a change is contract-affecting, assume it is and check
|
||||
- **Never edit a `startos/versions/<v>.ts` that's already been built/installed** — add a new version file.
|
||||
- **Don't push to GitHub by default** — remote is self-hosted Gitea.
|
||||
|
||||
## Current state — post-eval security pass landed (2026-06-13)
|
||||
## Current state — Users tab + webhook-dedup/P2 batch landed (2026-06-15)
|
||||
|
||||
- **Box, local tree, git aligned at relay `0.2.124`** (app `0.2.155`); `current: v_0_2_124`. Gitea remote `origin` now set up (`ssh://git@immense-voyage.local:59916/grant/recap-relay.git`); `master` pushed and tracking `origin/master`. Working tree clean. **Suite green at 60 tests** (`cd server && npm test`); server boots clean.
|
||||
- **Full independent eval done** (evaluator + security-auditor + exerciser + doc-auditor + start9-spec-checker) → `EVALUATION.md` (overwritten in place each run, so re-running diffs cleanly).
|
||||
- **All P0/P1 fixed** this session (commits `8ad7c54`/`d2caa98`/`3a601e1`): SSRF guard on caller-supplied media URLs (new `server/safe-url.js`), the early-renewal credit-reset money-leak (`extendUserTier`/`setUserTier` `resetCycle`), and the `multer`→`^2.0.1` DoS bump. None touch the `../recap` client contract.
|
||||
- **Three P2 fixed** (commits `cbd9748`/`da1bba2`/`693d724`): meeting-`:id` path-traversal guard (`meetingPath()`), constant-time operator-key compare, and a JSON error handler that closes the malformed-body stack-trace leak.
|
||||
- **Next (open P2), in priority order:**
|
||||
1. Persist webhook dedup so a restart can't double-credit/double-extend — `routes/credits.js:63`, `zaprite-webhook.js:27`.
|
||||
2. **Needs operator decision:** is BTCPay a hard requirement or truly optional? It's `optional:false`/`kind:'running'` despite "optional" comments, so StartOS won't start the relay without BTCPay co-installed — `startos/manifest/index.ts:38-49` + `dependencies.ts`. Then make manifest/deps/comment agree.
|
||||
3. Money-path unit tests (`commitCredit`/`refundCredit`/`applyTierPromotion`/grant handlers); scope `cors()` off `/admin/*` (`index.js`); split the 2225-line `routes/internal-meetings.js`; fix the two AGENTS.md auth-doc drifts (Stack-line `/relay/*` auth; missing `/admin/btcpay/callback` exempt path).
|
||||
- **Risks/notes:** SSRF guard leaves a DNS-rebinding TOCTOU open (acceptable for a private box; revisit if exposed). P3+ deferred tail + pre-existing speaker-tool/empty-section backlog → `ROADMAP.md` / `docs/issues-backlog.md`.
|
||||
- **Box, local tree, git aligned at relay `0.2.126`** (app `0.2.155`); `current: v_0_2_126`. Gitea remote `origin` (`ssh://git@immense-voyage.local:59916/grant/recap-relay.git`); `master` tracks `origin/master`. Working tree clean. **Suite green at 79 tests** (`cd server && npm test`); server boots clean.
|
||||
- **Users dashboard tab** (`0.2.125`): new cookie-gated tab — every credit-ledger row (typed cloud/license/install) with computed remaining/total balances, key filter, and a per-row "grant free credits" action. `GET /admin/credits` (enriched read) + `POST /admin/credits/grant {credit_key, amount}` (free top-up via `addPurchasedCredits`, guards: positive int ≤1M, must be an existing row). Admin-only; no `../recap` contract change.
|
||||
- **Webhook dedup now persistent** (`0.2.126`): new `server/webhook-dedup.js` (JSON store at `/data/processed-webhooks.json`, atomic writes, 180-day prune) replaces the in-memory Sets in `routes/credits.js` + `zaprite-webhook.js` (and the rescan path) — a duplicate delivery straddling a restart can no longer double-credit/double-extend. Keys namespaced `<storeId>|<invoiceId>` vs `zaprite:<orderId>`.
|
||||
- **BTCPay is REQUIRED** (operator decision, 2026-06-15): config was already `optional:false`/`kind:'running'`; corrected the contradictory "optional" comment in `startos/manifest/index.ts`. It's the only paid rail, so the relay shouldn't run without it.
|
||||
- **CORS scoped to `/relay/*`** (`index.js`) — off `/admin/*` + dashboard (same-origin). Plus money-path unit tests (`commitCredit`/`refundCredit`/`applyTierPromotion`) and the two AGENTS.md auth-doc drift fixes.
|
||||
- **Next (open P2 / deferred):**
|
||||
1. Split the 2225-line `routes/internal-meetings.js` — **deferred as likely overkill** for a private service; do only if it becomes painful to work in.
|
||||
2. P3+ deferred tail (no `/relay/*` rate limiting, container-as-root, dashboard `innerHTML` XSS surface, prune 126 version files, `/relay/health` stale `0.2.11`, etc.) + speaker-tool/empty-section backlog → `ROADMAP.md` / `docs/issues-backlog.md`.
|
||||
- **Risks/notes:** webhook dedup keeps the pre-existing check-then-mark race for *truly simultaneous* duplicate deliveries (vanishingly rare on a private box; would need locking). SSRF guard leaves a DNS-rebinding TOCTOU open (acceptable for a private box). Full prior eval → `EVALUATION.md`.
|
||||
|
||||
+7
-1
@@ -17,6 +17,7 @@ import { initCredits } from "./credits.js";
|
||||
import { initAuditLog } from "./audit-log.js";
|
||||
import { initJobCredits } from "./job-credits.js";
|
||||
import { initOutputStore } from "./output-store.js";
|
||||
import { initWebhookDedup } from "./webhook-dedup.js";
|
||||
import {
|
||||
setupAdminAuthMiddleware,
|
||||
setupAdminAuthRoutes,
|
||||
@@ -49,9 +50,14 @@ await initCredits({ dataDir: DATA_DIR });
|
||||
await initJobCredits({ dataDir: DATA_DIR });
|
||||
await initAuditLog({ dataDir: DATA_DIR });
|
||||
await initOutputStore({ dataDir: DATA_DIR });
|
||||
await initWebhookDedup({ dataDir: DATA_DIR });
|
||||
|
||||
const app = express();
|
||||
app.use(cors());
|
||||
// CORS only on /relay/* — those are the cross-origin clients (the Recaps
|
||||
// app + cloud server). /admin/* and the dashboard are served same-origin
|
||||
// to the operator's browser, so they don't need (and shouldn't get) the
|
||||
// permissive Access-Control-Allow-Origin a global cors() would apply.
|
||||
app.use("/relay", cors());
|
||||
app.use(cookieParser());
|
||||
|
||||
// Admin auth must run BEFORE the admin routes register so the cookie
|
||||
|
||||
+17
-19
@@ -49,18 +49,16 @@ import {
|
||||
BtcPayError,
|
||||
} from "../btcpay-client.js";
|
||||
import { recordCall } from "../audit-log.js";
|
||||
import { isWebhookProcessed, markWebhookProcessed } from "../webhook-dedup.js";
|
||||
import { envelope, errorEnvelope } from "./envelope.js";
|
||||
|
||||
// In-memory dedup of processed BTCPay invoice ids. BTCPay retries
|
||||
// webhook deliveries on non-2xx responses or network errors, so the
|
||||
// same InvoiceSettled event can land more than once. We don't want
|
||||
// to grant credits twice for one paid invoice.
|
||||
//
|
||||
// Cleared on relay restart, which means an unlucky webhook duplicate
|
||||
// straddling a restart would double-credit. Acceptable tradeoff for
|
||||
// v1 (operator can manually adjust the ledger if it happens), but
|
||||
// worth swapping for a persistent set once the relay sees real load.
|
||||
const processedInvoices = new Set();
|
||||
// Dedup of processed BTCPay invoice ids. BTCPay retries webhook
|
||||
// deliveries on non-2xx responses or network errors, so the same
|
||||
// InvoiceSettled event can land more than once — we don't want to grant
|
||||
// credits twice for one paid invoice. Backed by the persistent
|
||||
// webhook-dedup store (../webhook-dedup.js) so a duplicate straddling a
|
||||
// relay restart can't double-credit. Keys are namespaced
|
||||
// `<storeId>|<invoiceId>` to share the store with the Zaprite rail.
|
||||
|
||||
export function creditsRouter() {
|
||||
const router = express.Router();
|
||||
@@ -452,7 +450,7 @@ export function creditsRouter() {
|
||||
}
|
||||
|
||||
const dedupKey = `${cfg.relay_btcpay_store_id}|${invoiceId}`;
|
||||
if (processedInvoices.has(dedupKey)) {
|
||||
if (isWebhookProcessed(dedupKey)) {
|
||||
return res
|
||||
.status(200)
|
||||
.json({ ok: true, ignored: "already_processed", invoiceId });
|
||||
@@ -487,7 +485,7 @@ export function creditsRouter() {
|
||||
const subTier = meta.tier;
|
||||
const periodDays = Number(meta.period_days) || 30;
|
||||
if (!subUserId || (subTier !== "pro" && subTier !== "max")) {
|
||||
processedInvoices.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
return res
|
||||
.status(200)
|
||||
.json({ ok: true, ignored: "bad_tier_metadata", invoiceId });
|
||||
@@ -498,7 +496,7 @@ export function creditsRouter() {
|
||||
tier: subTier,
|
||||
periodDays,
|
||||
});
|
||||
processedInvoices.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
console.log(
|
||||
`[btcpay/webhook] ${subTier} +${periodDays}d for user ${subUserId.slice(0, 8)}… (invoice ${invoiceId}) → expires ${row.subscription_expires_at}`,
|
||||
);
|
||||
@@ -529,7 +527,7 @@ export function creditsRouter() {
|
||||
// Not ours — could be an invoice from another product on the
|
||||
// same store. Mark processed so we don't keep retrying, and
|
||||
// 200 so BTCPay stops calling.
|
||||
processedInvoices.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
return res
|
||||
.status(200)
|
||||
.json({ ok: true, ignored: "no_recap_metadata" });
|
||||
@@ -541,7 +539,7 @@ export function creditsRouter() {
|
||||
creditKey,
|
||||
amount: credits,
|
||||
});
|
||||
processedInvoices.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
await recordCall({
|
||||
install_id: installId,
|
||||
license_fingerprint: buyerFp,
|
||||
@@ -614,8 +612,8 @@ function rewriteCheckoutUrl(url, browserBase) {
|
||||
// Recovery: when the BTCPay webhook URL was broken and paid
|
||||
// invoices never got credited, this scans BTCPay's recent settled
|
||||
// invoices and grants credits for ones the relay hasn't processed.
|
||||
// Idempotent via the same processedInvoices dedup the webhook uses,
|
||||
// so re-running is safe.
|
||||
// Idempotent via the same persistent webhook-dedup store the webhook
|
||||
// uses, so re-running is safe.
|
||||
//
|
||||
// Exported so the admin route in btcpay-setup.js can call it. Not
|
||||
// exposed via /relay/* because it's operator-initiated, not buyer.
|
||||
@@ -660,7 +658,7 @@ export async function rescanSettledInvoices() {
|
||||
const details = [];
|
||||
for (const invoice of invoices || []) {
|
||||
const dedupKey = `${cfg.relay_btcpay_store_id}|${invoice.id}`;
|
||||
if (processedInvoices.has(dedupKey)) {
|
||||
if (isWebhookProcessed(dedupKey)) {
|
||||
alreadyProcessed++;
|
||||
continue;
|
||||
}
|
||||
@@ -684,7 +682,7 @@ export async function rescanSettledInvoices() {
|
||||
creditKey,
|
||||
amount: credits,
|
||||
});
|
||||
processedInvoices.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
await recordCall({
|
||||
install_id: installId,
|
||||
license_fingerprint: buyerFp,
|
||||
|
||||
@@ -15,16 +15,14 @@ import express from "express";
|
||||
import { extendUserTier } from "../credits.js";
|
||||
import { getZapriteConfig } from "../config.js";
|
||||
import { getOrder, isOrderPaid, orderIdFromWebhook } from "../zaprite-client.js";
|
||||
import { isWebhookProcessed, markWebhookProcessed } from "../webhook-dedup.js";
|
||||
|
||||
// In-memory dedup of fully-processed orders (mirrors the BTCPay handler's
|
||||
// processedInvoices). Zaprite retries on non-200, and may fire multiple
|
||||
// events per order, so we guard the grant. Cleared on restart — a
|
||||
// re-delivered webhook after restart re-fetches + re-grants, but
|
||||
// extendUserTier is keyed per order via this set within a process; a
|
||||
// duplicate grant across a restart is the same harmless "extend by one
|
||||
// period" the operator-comp path already tolerates. (Acceptable: card
|
||||
// double-fires across a restart are vanishingly rare.)
|
||||
const processedZaprite = new Set();
|
||||
// Dedup of fully-processed orders (mirrors the BTCPay handler). Zaprite
|
||||
// retries on non-200, and may fire multiple events per order, so we
|
||||
// guard the grant. Backed by the persistent webhook-dedup store
|
||||
// (../webhook-dedup.js) so a re-delivered webhook straddling a relay
|
||||
// restart can't double-extend a subscription. Keys are namespaced
|
||||
// `zaprite:<orderId>` to share the store with the BTCPay rail.
|
||||
|
||||
export function zapriteWebhookRouter() {
|
||||
const router = express.Router();
|
||||
@@ -49,7 +47,7 @@ export function zapriteWebhookRouter() {
|
||||
return res.status(200).json({ ok: true, ignored: "no_order_id" });
|
||||
}
|
||||
const dedupKey = `zaprite:${orderId}`;
|
||||
if (processedZaprite.has(dedupKey)) {
|
||||
if (isWebhookProcessed(dedupKey)) {
|
||||
return res
|
||||
.status(200)
|
||||
.json({ ok: true, ignored: "already_processed", orderId });
|
||||
@@ -91,7 +89,7 @@ export function zapriteWebhookRouter() {
|
||||
|
||||
const meta = order.metadata || {};
|
||||
if (meta.product !== "recap_tier_subscription") {
|
||||
processedZaprite.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
return res
|
||||
.status(200)
|
||||
.json({ ok: true, ignored: "not_a_tier_order", orderId });
|
||||
@@ -100,7 +98,7 @@ export function zapriteWebhookRouter() {
|
||||
const subTier = meta.tier;
|
||||
const periodDays = Number(meta.period_days) || 30;
|
||||
if (!subUserId || (subTier !== "pro" && subTier !== "max")) {
|
||||
processedZaprite.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
return res
|
||||
.status(200)
|
||||
.json({ ok: true, ignored: "bad_tier_metadata", orderId });
|
||||
@@ -112,7 +110,7 @@ export function zapriteWebhookRouter() {
|
||||
tier: subTier,
|
||||
periodDays,
|
||||
});
|
||||
processedZaprite.add(dedupKey);
|
||||
await markWebhookProcessed(dedupKey);
|
||||
console.log(
|
||||
`[zaprite/webhook] ${subTier} +${periodDays}d for user ${subUserId.slice(0, 8)}… (order ${orderId}) → expires ${row.subscription_expires_at}`,
|
||||
);
|
||||
|
||||
@@ -0,0 +1,111 @@
|
||||
// Core billing primitives: commitCredit (debit), refundCredit (inverse),
|
||||
// and applyTierPromotion (the Core→paid upgrade bookkeeping). Default
|
||||
// quotas (no config file → getTierQuotas falls back): Core lifetime=10,
|
||||
// geminiCapLifetime=5; Pro monthly=50, geminiCapMonthly=25.
|
||||
|
||||
import { test, describe, before } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import {
|
||||
initCredits,
|
||||
getOrCreateRow,
|
||||
getUserCreditRow,
|
||||
setUserTier,
|
||||
commitCredit,
|
||||
refundCredit,
|
||||
applyTierPromotion,
|
||||
} from "../credits.js";
|
||||
|
||||
before(async () => {
|
||||
await initCredits({ dataDir: mkdtempSync(path.join(tmpdir(), "relay-money-")) });
|
||||
});
|
||||
|
||||
describe("commitCredit", () => {
|
||||
test("Core debits the lifetime bucket; Gemini tracked separately", async () => {
|
||||
const creditKey = "inst:core-commit";
|
||||
await commitCredit({ creditKey, tier: "core", backend: "gemini" });
|
||||
let row = await getOrCreateRow({ creditKey });
|
||||
assert.equal(row.lifetime_consumed, 1);
|
||||
assert.equal(row.lifetime_gemini_consumed, 1);
|
||||
|
||||
// A hardware call bumps lifetime but NOT the Gemini sub-counter.
|
||||
await commitCredit({ creditKey, tier: "core", backend: "hardware" });
|
||||
row = await getOrCreateRow({ creditKey });
|
||||
assert.equal(row.lifetime_consumed, 2);
|
||||
assert.equal(row.lifetime_gemini_consumed, 1);
|
||||
});
|
||||
|
||||
test("paid tier debits the monthly bucket", async () => {
|
||||
await setUserTier({ userId: "p-commit", tier: "pro" });
|
||||
await commitCredit({ creditKey: "user:p-commit", tier: "pro", backend: "gemini" });
|
||||
const row = await getUserCreditRow("p-commit");
|
||||
assert.equal(row.monthly_consumed, 1);
|
||||
assert.equal(row.monthly_gemini_consumed, 1);
|
||||
});
|
||||
|
||||
test("spend order: once the tier bucket is exhausted, debit purchased", async () => {
|
||||
const creditKey = "inst:core-exhausted";
|
||||
const row = await getOrCreateRow({ creditKey });
|
||||
row.lifetime_consumed = 10; // at the Core lifetime cap
|
||||
row.purchased_balance = 3;
|
||||
await commitCredit({ creditKey, tier: "core", backend: "hardware" });
|
||||
const after = await getOrCreateRow({ creditKey });
|
||||
assert.equal(after.lifetime_consumed, 10, "tier counter must not exceed the cap");
|
||||
assert.equal(after.purchased_balance, 2, "overflow comes out of purchased");
|
||||
});
|
||||
});
|
||||
|
||||
describe("refundCredit", () => {
|
||||
test("mirrors a Core commit (tier bucket first, Gemini sub-counter too)", async () => {
|
||||
const creditKey = "inst:core-refund";
|
||||
await commitCredit({ creditKey, tier: "core", backend: "gemini" });
|
||||
await refundCredit({ creditKey, tier: "core", backend: "gemini" });
|
||||
const row = await getOrCreateRow({ creditKey });
|
||||
assert.equal(row.lifetime_consumed, 0);
|
||||
assert.equal(row.lifetime_gemini_consumed, 0);
|
||||
});
|
||||
|
||||
test("refund with an empty tier bucket credits the purchased bucket", async () => {
|
||||
const creditKey = "inst:refund-to-purchased";
|
||||
const row = await getOrCreateRow({ creditKey });
|
||||
row.lifetime_consumed = 0;
|
||||
row.purchased_balance = 0;
|
||||
await refundCredit({ creditKey, tier: "core", backend: "hardware" });
|
||||
const after = await getOrCreateRow({ creditKey });
|
||||
assert.equal(after.lifetime_consumed, 0, "must floor at 0, not go negative");
|
||||
assert.equal(after.purchased_balance, 1, "refund lands in purchased when tier is already 0");
|
||||
});
|
||||
});
|
||||
|
||||
describe("applyTierPromotion", () => {
|
||||
test("Core→paid transfers leftover Core credits to purchased and resets the cycle", async () => {
|
||||
const row = await getOrCreateRow({ creditKey: "inst:promo" });
|
||||
row.lifetime_consumed = 4; // 6 of 10 Core credits unused
|
||||
row.monthly_consumed = 2; // stale paid-counter noise that must be zeroed
|
||||
const promoted = await applyTierPromotion(row, "pro");
|
||||
assert.equal(promoted, true);
|
||||
assert.equal(row.tier_snapshot, "pro");
|
||||
assert.equal(row.purchased_balance, 6, "leftover Core credits carry forward as durable top-up");
|
||||
assert.equal(row.purchased_total_ever, 6);
|
||||
assert.equal(row.monthly_consumed, 0, "promotion starts a fresh monthly cycle");
|
||||
});
|
||||
|
||||
test("idempotent: a second promotion on an already-paid row is a no-op", async () => {
|
||||
const row = await getOrCreateRow({ creditKey: "inst:promo-again" });
|
||||
row.lifetime_consumed = 4;
|
||||
await applyTierPromotion(row, "pro"); // first: fires
|
||||
const purchasedAfterFirst = row.purchased_balance;
|
||||
const promoted = await applyTierPromotion(row, "max"); // second: must bail
|
||||
assert.equal(promoted, false);
|
||||
assert.equal(row.purchased_balance, purchasedAfterFirst, "no second leftover transfer");
|
||||
assert.equal(row.tier_snapshot, "pro", "bails before flipping tier again");
|
||||
});
|
||||
|
||||
test("promoting to Core is a no-op", async () => {
|
||||
const row = await getOrCreateRow({ creditKey: "inst:promo-core" });
|
||||
assert.equal(await applyTierPromotion(row, "core"), false);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,70 @@
|
||||
// Persistent webhook dedup — the store that stops a settled-payment
|
||||
// webhook duplicate from double-crediting (BTCPay) or double-extending
|
||||
// (Zaprite) when the duplicate straddles a relay restart.
|
||||
|
||||
import { test, describe } from "node:test";
|
||||
import assert from "node:assert/strict";
|
||||
import { mkdtempSync, writeFileSync } from "node:fs";
|
||||
import { tmpdir } from "node:os";
|
||||
import path from "node:path";
|
||||
|
||||
import {
|
||||
initWebhookDedup,
|
||||
isWebhookProcessed,
|
||||
markWebhookProcessed,
|
||||
} from "../webhook-dedup.js";
|
||||
|
||||
// The module is a singleton; each test inits a fresh dir to reset state.
|
||||
const freshDir = () => mkdtempSync(path.join(tmpdir(), "relay-dedup-"));
|
||||
|
||||
describe("webhook dedup (persistent)", () => {
|
||||
test("unknown key is not processed; marking makes it processed", async () => {
|
||||
await initWebhookDedup({ dataDir: freshDir() });
|
||||
assert.equal(isWebhookProcessed("store1|inv1"), false);
|
||||
await markWebhookProcessed("store1|inv1");
|
||||
assert.equal(isWebhookProcessed("store1|inv1"), true);
|
||||
});
|
||||
|
||||
// Headline guarantee for this change.
|
||||
test("processed keys survive a restart (reload from disk)", async () => {
|
||||
const dir = freshDir();
|
||||
await initWebhookDedup({ dataDir: dir });
|
||||
await markWebhookProcessed("store1|inv-restart");
|
||||
// Simulate a relay restart: re-init from the SAME data dir.
|
||||
await initWebhookDedup({ dataDir: dir });
|
||||
assert.equal(
|
||||
isWebhookProcessed("store1|inv-restart"),
|
||||
true,
|
||||
"a key marked before restart must still be deduped after restart"
|
||||
);
|
||||
});
|
||||
|
||||
test("BTCPay and Zaprite keys share the store without colliding", async () => {
|
||||
await initWebhookDedup({ dataDir: freshDir() });
|
||||
await markWebhookProcessed("store1|inv9");
|
||||
await markWebhookProcessed("zaprite:order9");
|
||||
assert.equal(isWebhookProcessed("store1|inv9"), true);
|
||||
assert.equal(isWebhookProcessed("zaprite:order9"), true);
|
||||
assert.equal(isWebhookProcessed("zaprite:inv9"), false);
|
||||
});
|
||||
|
||||
test("marking is idempotent", async () => {
|
||||
await initWebhookDedup({ dataDir: freshDir() });
|
||||
await markWebhookProcessed("k");
|
||||
await markWebhookProcessed("k"); // must not throw or duplicate
|
||||
assert.equal(isWebhookProcessed("k"), true);
|
||||
});
|
||||
|
||||
test("stale entries past the retention window are pruned on load", async () => {
|
||||
const dir = freshDir();
|
||||
const ancient = new Date(Date.now() - 365 * 24 * 60 * 60 * 1000).toISOString();
|
||||
const recent = new Date().toISOString();
|
||||
writeFileSync(
|
||||
path.join(dir, "processed-webhooks.json"),
|
||||
JSON.stringify({ keys: { "store1|old": ancient, "store1|new": recent } })
|
||||
);
|
||||
await initWebhookDedup({ dataDir: dir });
|
||||
assert.equal(isWebhookProcessed("store1|old"), false, "1y-old key should be pruned");
|
||||
assert.equal(isWebhookProcessed("store1|new"), true, "recent key should survive");
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,93 @@
|
||||
// Persistent dedup of settled-payment webhook deliveries — BTCPay
|
||||
// invoices AND Zaprite orders share this one store. It replaces the
|
||||
// per-process in-memory Sets the two webhook handlers used to keep:
|
||||
// those were cleared on restart, so a duplicate delivery that straddled
|
||||
// a relay restart would double-credit (BTCPay) or double-extend a
|
||||
// subscription (Zaprite). Persisting the processed-keys to disk closes
|
||||
// that window.
|
||||
//
|
||||
// JSON-file backed (single file at <dataDir>/processed-webhooks.json),
|
||||
// same rationale as the credit ledger in credits.js: at most one
|
||||
// mutation per settled payment, so a plain JSON file with serial writes
|
||||
// is plenty.
|
||||
//
|
||||
// Keys are namespaced by their caller so the two rails can't collide in
|
||||
// the shared store: BTCPay uses `<storeId>|<invoiceId>`, Zaprite uses
|
||||
// `zaprite:<orderId>`. We store key → ISO timestamp (not a bare set) so
|
||||
// entries far older than any processor's retry window can be pruned on
|
||||
// load, keeping the file bounded.
|
||||
|
||||
import fs from "fs/promises";
|
||||
import path from "path";
|
||||
|
||||
// Payment processors retry webhook delivery for hours-to-days, never
|
||||
// months. 180 days is safely beyond any retry window, so pruning past
|
||||
// it cannot re-open a double-grant gap.
|
||||
const RETENTION_MS = 180 * 24 * 60 * 60 * 1000;
|
||||
|
||||
let storePath = null;
|
||||
let processed = {}; // key → ISO timestamp
|
||||
let writing = null; // serializes concurrent writes
|
||||
|
||||
export async function initWebhookDedup({ dataDir } = {}) {
|
||||
const dir = dataDir || "/data";
|
||||
storePath = path.join(dir, "processed-webhooks.json");
|
||||
await fs.mkdir(dir, { recursive: true }).catch(() => {});
|
||||
try {
|
||||
const raw = await fs.readFile(storePath, "utf8");
|
||||
const parsed = JSON.parse(raw);
|
||||
processed =
|
||||
parsed && typeof parsed === "object" && parsed.keys ? parsed.keys : {};
|
||||
} catch (err) {
|
||||
if (err.code !== "ENOENT") {
|
||||
console.warn(
|
||||
`[webhook-dedup] failed to read ${storePath}: ${err.message} — starting empty`
|
||||
);
|
||||
}
|
||||
processed = {};
|
||||
}
|
||||
// Prune anything older than the retention window so the file stays
|
||||
// bounded over the relay's lifetime.
|
||||
const cutoff = Date.now() - RETENTION_MS;
|
||||
let pruned = 0;
|
||||
for (const [k, ts] of Object.entries(processed)) {
|
||||
const t = Date.parse(ts);
|
||||
if (!Number.isFinite(t) || t < cutoff) {
|
||||
delete processed[k];
|
||||
pruned += 1;
|
||||
}
|
||||
}
|
||||
if (pruned > 0) await persist();
|
||||
console.log(
|
||||
`[webhook-dedup] loaded ${Object.keys(processed).length} processed key(s)` +
|
||||
`${pruned ? `, pruned ${pruned} stale` : ""} from ${storePath}`
|
||||
);
|
||||
}
|
||||
|
||||
async function persist() {
|
||||
// Coalesce concurrent writes — mirrors credits.js persist().
|
||||
if (writing) await writing;
|
||||
writing = (async () => {
|
||||
const tmp = storePath + ".tmp";
|
||||
await fs.writeFile(tmp, JSON.stringify({ keys: processed }), { mode: 0o600 });
|
||||
await fs.rename(tmp, storePath);
|
||||
})();
|
||||
try {
|
||||
await writing;
|
||||
} finally {
|
||||
writing = null;
|
||||
}
|
||||
}
|
||||
|
||||
export function isWebhookProcessed(key) {
|
||||
return Object.prototype.hasOwnProperty.call(processed, key);
|
||||
}
|
||||
|
||||
// Mark a webhook key processed and persist immediately. Callers invoke
|
||||
// this AFTER the grant/extend succeeds, so a failed grant leaves the key
|
||||
// unmarked and a processor retry can complete it. Idempotent.
|
||||
export async function markWebhookProcessed(key) {
|
||||
if (isWebhookProcessed(key)) return;
|
||||
processed[key] = new Date().toISOString();
|
||||
await persist();
|
||||
}
|
||||
@@ -31,15 +31,15 @@ export const manifest = setupManifest({
|
||||
start: null,
|
||||
stop: null,
|
||||
},
|
||||
// Relay has no REQUIRED dependencies — Gemini is internet-fronted
|
||||
// and the optional Parakeet/Gemma backends are at user-configured
|
||||
// URLs (typically a separate machine on the operator's LAN).
|
||||
//
|
||||
// BTCPay Server is declared optional so the dashboard's "Connect
|
||||
// BTCPay" flow can auto-discover its URL via
|
||||
// sdk.serviceInterface.getAll() when both are installed on the
|
||||
// same Start9 box. When BTCPay is not installed, the relay still
|
||||
// runs fine — only the credit-purchase flow is disabled.
|
||||
// BTCPay Server is a REQUIRED running dependency (optional: false
|
||||
// here, kind: 'running' in dependencies.ts). It's the only payment
|
||||
// rail through which the operator actually gets paid, so the relay
|
||||
// shouldn't run without it. The dependency also wires up the internal
|
||||
// docker hostname (btcpayserver.startos:23000) the relay-to-BTCPay API
|
||||
// calls rely on, and lets the dashboard's "Connect BTCPay" flow
|
||||
// auto-discover the URL via sdk.serviceInterface.getAll(). (Gemini is
|
||||
// internet-fronted and the Parakeet/Gemma backends are at
|
||||
// user-configured LAN URLs, so neither is a StartOS dependency.)
|
||||
dependencies: {
|
||||
btcpayserver: {
|
||||
description: {
|
||||
|
||||
@@ -126,8 +126,9 @@ import { v_0_2_122 } from './v0.2.122'
|
||||
import { v_0_2_123 } from './v0.2.123'
|
||||
import { v_0_2_124 } from './v0.2.124'
|
||||
import { v_0_2_125 } from './v0.2.125'
|
||||
import { v_0_2_126 } from './v0.2.126'
|
||||
|
||||
export const versionGraph = VersionGraph.of({
|
||||
current: v_0_2_125,
|
||||
other: [v_0_2_124, v_0_2_123, v_0_2_122, v_0_2_121, v_0_2_120, v_0_2_119, v_0_2_118, v_0_2_117, v_0_2_116, v_0_2_115, v_0_2_114, v_0_2_113, v_0_2_112, v_0_2_111, v_0_2_110, v_0_2_109, v_0_2_108, v_0_2_107, v_0_2_106, v_0_2_105, v_0_2_104, v_0_2_103, v_0_2_102, v_0_2_101, v_0_2_100, v_0_2_99, v_0_2_98, v_0_2_97, v_0_2_96, v_0_2_95, v_0_2_94, v_0_2_93, v_0_2_92, v_0_2_91, v_0_2_90, v_0_2_89, v_0_2_88, v_0_2_87, v_0_2_86, v_0_2_85, v_0_2_84, v_0_2_83, v_0_2_82, v_0_2_81, v_0_2_80, v_0_2_79, v_0_2_78, v_0_2_77, v_0_2_76, v_0_2_75, v_0_2_74, v_0_2_73, v_0_2_72, v_0_2_71, v_0_2_70, v_0_2_69, v_0_2_68, v_0_2_67, v_0_2_66, v_0_2_65, v_0_2_64, v_0_2_63, v_0_2_62, v_0_2_61, v_0_2_60, v_0_2_59, v_0_2_58, v_0_2_57, v_0_2_56, v_0_2_55, v_0_2_54, v_0_2_53, v_0_2_52, v_0_2_51, v_0_2_50, v_0_2_49, v_0_2_48, v_0_2_47, v_0_2_46, v_0_2_45, v_0_2_44, v_0_2_43, v_0_2_42, v_0_2_41, v_0_2_40, v_0_2_39, v_0_2_38, v_0_2_37, v_0_2_36, v_0_2_35, v_0_2_34, v_0_2_33, v_0_2_32, v_0_2_31, v_0_2_30, v_0_2_29, v_0_2_28, v_0_2_27, v_0_2_26, v_0_2_25, v_0_2_24, v_0_2_23, v_0_2_22, v_0_2_21, v_0_2_20, v_0_2_19, v_0_2_18, v_0_2_17, v_0_2_16, v_0_2_15, v_0_2_14, v_0_2_13, v_0_2_12, v_0_2_11, v_0_2_10, v_0_2_9, v_0_2_8, v_0_2_7, v_0_2_6, v_0_2_5, v_0_2_4, v_0_2_3, v_0_2_2, v_0_2_1, v_0_2_0, v_0_1_0],
|
||||
current: v_0_2_126,
|
||||
other: [v_0_2_125, v_0_2_124, v_0_2_123, v_0_2_122, v_0_2_121, v_0_2_120, v_0_2_119, v_0_2_118, v_0_2_117, v_0_2_116, v_0_2_115, v_0_2_114, v_0_2_113, v_0_2_112, v_0_2_111, v_0_2_110, v_0_2_109, v_0_2_108, v_0_2_107, v_0_2_106, v_0_2_105, v_0_2_104, v_0_2_103, v_0_2_102, v_0_2_101, v_0_2_100, v_0_2_99, v_0_2_98, v_0_2_97, v_0_2_96, v_0_2_95, v_0_2_94, v_0_2_93, v_0_2_92, v_0_2_91, v_0_2_90, v_0_2_89, v_0_2_88, v_0_2_87, v_0_2_86, v_0_2_85, v_0_2_84, v_0_2_83, v_0_2_82, v_0_2_81, v_0_2_80, v_0_2_79, v_0_2_78, v_0_2_77, v_0_2_76, v_0_2_75, v_0_2_74, v_0_2_73, v_0_2_72, v_0_2_71, v_0_2_70, v_0_2_69, v_0_2_68, v_0_2_67, v_0_2_66, v_0_2_65, v_0_2_64, v_0_2_63, v_0_2_62, v_0_2_61, v_0_2_60, v_0_2_59, v_0_2_58, v_0_2_57, v_0_2_56, v_0_2_55, v_0_2_54, v_0_2_53, v_0_2_52, v_0_2_51, v_0_2_50, v_0_2_49, v_0_2_48, v_0_2_47, v_0_2_46, v_0_2_45, v_0_2_44, v_0_2_43, v_0_2_42, v_0_2_41, v_0_2_40, v_0_2_39, v_0_2_38, v_0_2_37, v_0_2_36, v_0_2_35, v_0_2_34, v_0_2_33, v_0_2_32, v_0_2_31, v_0_2_30, v_0_2_29, v_0_2_28, v_0_2_27, v_0_2_26, v_0_2_25, v_0_2_24, v_0_2_23, v_0_2_22, v_0_2_21, v_0_2_20, v_0_2_19, v_0_2_18, v_0_2_17, v_0_2_16, v_0_2_15, v_0_2_14, v_0_2_13, v_0_2_12, v_0_2_11, v_0_2_10, v_0_2_9, v_0_2_8, v_0_2_7, v_0_2_6, v_0_2_5, v_0_2_4, v_0_2_3, v_0_2_2, v_0_2_1, v_0_2_0, v_0_1_0],
|
||||
})
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
import { VersionInfo } from '@start9labs/start-sdk'
|
||||
|
||||
export const v_0_2_126 = VersionInfo.of({
|
||||
version: '0.2.126:0',
|
||||
releaseNotes: {
|
||||
en_US: 'Persist payment-webhook dedup across restarts (no double-credit/double-extend); declare BTCPay a required dependency; scope CORS to /relay/*; add money-path + webhook-dedup tests; doc fixes',
|
||||
},
|
||||
migrations: {
|
||||
up: async ({ effects }) => {},
|
||||
down: async ({ effects }) => {},
|
||||
},
|
||||
})
|
||||
Reference in New Issue
Block a user