Phase 0 foundation: canonical schema, ingest pipeline, CRM MCP server

Workstream A–C substrate for the Ten31 agentic system:
- A1: docs/crm-overview.md; CLAUDE.md conventions + guardrail #9
- A2: additive/reversible core migration (canonical_entities, entity_links,
  interaction_log, relationship_edges, soft-delete) + ledgered runner
- B1/B3: chunking + deterministic entity resolution (backend/ingest)
- B2: dense (bge-m3) + BM25 sparse ingest to Qdrant crm_chunks
- C: CRM MCP server (reads, retrieval modes, logged writes) — no outbound tools
- docs: redaction/re-hydration, Gmail enablement runbook
- synthetic test data; .env.example; housekeeping (.gitignore, untrack crm.db,
  drop legacy files + start9/0.3.5)

Verified end-to-end on synthetic data + live Sparks (hybrid > dense on entity
queries). Real backfill runs on Ten31 infra; index holds synthetic data only.
Branch snapshot also captures pre-existing working-tree changes.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Keysat
2026-06-05 08:11:28 -05:00
parent 7027efd777
commit c7ce44d963
99 changed files with 10676 additions and 7817 deletions
+6
View File
@@ -0,0 +1,6 @@
node_modules/
javascript/
javascript.old/
.DS_Store
.home/
*.s9pk
+231
View File
@@ -0,0 +1,231 @@
# Ten31 Database — StartOS 0.4 deployment guide
This guide walks through building the `ten-database` 0.4 service package and
sideloading it onto an x86_64 StartOS 0.4 beta machine.
The `start9/0.4/` folder is intentionally self-contained. It does not share
any files with `start9/0.3.5/`, so the legacy package stays intact and can
be rebuilt later if needed.
---
## 0 — How data preservation works
Starting with **0.1.0:40**, this package no longer ships a seed snapshot. The
0.3.5 → 0.4 migration is complete and the live `/data` volume on the StartOS
host is the sole source of truth.
Key facts:
- StartOS preserves the `main` volume across sideloads. Reinstalling a new
`.s9pk` does **not** touch `/data/crm.db`, `/data/backups/`, or
`/data/.crm-secret`. Live edits made between releases are kept.
- Only `Uninstall` from the StartOS UI destroys the `main` volume. As long
as you only `Stop → Sideload new .s9pk → Start`, your data persists.
- Use StartOS-level **Backups → Create Backup** for full volume snapshots,
and the in-app **Settings → Admin → Run Backup** for JSON exports under
`/data/backups/`.
Container paths (unchanged from 0.3.5):
- `/data/crm.db` — primary SQLite DB (WAL journal mode)
- `/data/backups/` — JSON exports
- `/data/.crm-secret` — JWT signing key (kept across restarts so sessions stick)
> Historical note: `0.1.0:39` shipped a baked-in seed snapshot and a
> first-boot copy guarded by `! -f /data/crm.db`. That code path was removed
> in `0.1.0:40`. If you ever need to bootstrap a fresh host again, sideload
> `0.1.0:39` first, let it seed, then upgrade to the latest.
---
## 1 — Build-machine prerequisites
The 0.4 build runs on any machine with:
- Node.js ≥ 20 and npm
- Docker with buildx enabled (Docker Desktop on macOS works; Linux Docker
must have the `buildx` plugin)
- `start-cli` (Start9 SDK) — install per
https://docs.start9.com/packaging/0.4.0.x/environment-setup.html
- `jq`, `make`, `s3cmd` (s3cmd only if you also plan to `make publish`)
Recommended one-time setup:
```sh
# Initialize the Start9 developer key (run once per build machine)
start-cli init-key
# Create ~/.startos/config.yaml so `make install` can sideload:
cat > ~/.startos/config.yaml <<'YAML'
# Replace with the hostname of your 0.4 beta node
host: http://start9.local
YAML
```
---
## 2 — Build the x86_64 .s9pk
From the repo root:
```sh
cd start9/0.4
# One-time dependency install (pulls start-sdk + friends):
npm ci
# Clean build (produces ten-database_x86_64.s9pk):
make clean
make x86
```
Output:
- `ten-database_x86_64.s9pk` in `start9/0.4/`
- Build summary printed by s9pk.mk (title, version, arch, SDK version,
git hash)
> Note: `make` by default builds x86, arm, and riscv. The `Makefile`
> in this folder overrides `ARCHES := x86` so only x86_64 is produced.
> If you later need arm64 too, switch to `ARCHES := x86 arm`.
### If the build fails
Common causes and fixes:
- **`.git/HEAD` or `.git/index` missing** — s9pk.mk requires a real git
repo. It looks at `../../.git` relative to `start9/0.4/` (i.e. the repo
root). Make sure you're building inside the actual repo.
- **`start-cli not found`** — install the Start9 SDK CLI.
- **docker buildx error** — run `docker buildx create --use` once.
- **Permission denied removing `javascript/` between builds** — macOS
extended attributes can make ncc output files immutable. Run
`chmod -R u+w start9/0.4/javascript` and retry, or just `rm -rf
start9/0.4/javascript` from Finder.
---
## 3 — Sideload onto the StartOS 0.4 beta node
Two options:
### Option 1 — `make install` (uses ~/.startos/config.yaml)
```sh
cd start9/0.4
make install
```
This runs `start-cli package install -s ten-database_x86_64.s9pk` against
whatever host you set in `~/.startos/config.yaml`.
### Option 2 — StartOS web UI
1. Copy `ten-database_x86_64.s9pk` onto a machine that can reach the
StartOS 0.4 UI.
2. In the UI: **System → Sideload Service → pick the .s9pk → Install.**
3. After the install completes, open the service and click **Start**.
### First-boot verification
After upgrading the service:
1. Open the Ten31 Database UI from the Interfaces page.
2. Log in with your existing account — passwords and sessions persist
because `/data/.crm-secret` is preserved.
3. Spot-check a few rows in the fundraising grid against what you saw
before the upgrade.
4. Run one manual backup (Settings → Admin → Run Backup) to confirm the
app's write path works.
---
## 4 — Rollback plan
If a new sideload misbehaves:
1. **Stop** the service in StartOS — do not Uninstall (that deletes the
`main` volume).
2. Sideload the previous `.s9pk` (keep one around) and Start.
3. Investigate by opening the service logs from the StartOS UI.
For full disaster recovery, restore the `main` volume from a StartOS-level
Backup.
---
## 5 — File map (what lives where)
```
start9/0.4/
├── DEPLOY_040.md # this file
├── README.md # short overview
├── Dockerfile # self-contained; refs only start9/0.4/ paths
├── Makefile # thin override: ARCHES := x86
├── s9pk.mk # shared 0.4 build plumbing (do not edit)
├── package.json, -lock.json # start-sdk + build tooling
├── tsconfig.json
├── docker_entrypoint.sh # ensures /data dirs + JWT secret, starts server.py
├── healthcheck.sh # curl /api/health (diagnostics only)
├── icon.svg # service icon
├── LICENSE
├── refresh_seed.sh # (LEGACY) scp helper from 0.3.5; kept for reference
├── assets/
│ └── ABOUT.md # user-facing install description
├── seed/ # (LEGACY) historical seed snapshot, NOT shipped
│ ├── README.md
│ └── data/ # crm.db + backups from initial 0.3.5 → 0.4 cut
└── startos/ # SDK source (manifest, main, interfaces…)
├── index.ts # SDK entry (no edits normally needed)
├── sdk.ts # typed SDK instance
├── utils.ts # shared constants
├── i18n.ts # simple passthrough
├── manifest/
│ ├── index.ts # id, title, images, arches, volumes, alerts
│ └── i18n.ts # localized short/long description
├── versions/
│ ├── index.ts # versionGraph wiring
│ ├── v0.1.0.39.ts # first 0.4 release (with seed)
│ └── v0.1.0.40.ts # current release (seed removed)
├── init/index.ts # setupInit ordering
├── main.ts # daemon + health check
├── interfaces.ts # HTTP interface on port 8080
├── backups.ts # Backups.ofVolumes('main')
├── dependencies.ts # (none)
└── actions/index.ts # (none)
```
The `seed/` directory and `refresh_seed.sh` are no longer referenced by the
build and can be deleted from the repo at any time. They are kept on disk
purely as a historical snapshot of the data that was migrated off the 0.3.5
host on first cutover.
---
## 6 — Things to remember
- Package id stays `ten-database` across both 0.3.5 and 0.4 so there is
exactly one service to manage on each host.
- The service volume id is `main` on both sides and mounts at `/data`
inside the container. This is what makes data preservation trivial.
- The 0.4 release is x86_64 only. If you later deploy to aarch64, change
`ARCHES` in the Makefile and rebuild.
- The built `.s9pk` is not committed — treat it as a build artifact.
`.gitignore` already ignores `*.s9pk` and `javascript/`.
- If you change anything under `startos/`, run `npm run check` (tsc) and
`npm run build` (ncc) before re-packaging.
---
## 7 — Quick cheat sheet
```sh
cd start9/0.4
make clean
make x86
make install
```
After `make install` completes, open the service in the StartOS UI,
hit Start, and verify the app still works.
+52
View File
@@ -0,0 +1,52 @@
# ─────────────────────────────────────────────────────────────────
# Ten31 Database — StartOS 0.4 container image
# ─────────────────────────────────────────────────────────────────
# Build context (from the startos manifest dockerBuild.workdir)
# is the repository root (two levels up from start9/0.4/), so all
# COPY paths below are relative to the repo root.
#
# This image is intentionally self-contained under start9/0.4/:
# no files are pulled from start9/0.3.5/ so the two packages can
# evolve independently.
#
# As of 0.1.0:40 the image NO LONGER ships a seed snapshot. The
# initial migration from 0.3.5 has been completed; from this
# release forward the live /data volume on the StartOS host is
# the sole source of truth and is preserved across sideloads.
# ─────────────────────────────────────────────────────────────────
FROM python:3.11-slim
ENV PYTHONDONTWRITEBYTECODE=1 \
PYTHONUNBUFFERED=1 \
CRM_ENV=production \
CRM_HOST=0.0.0.0 \
CRM_PORT=8080 \
CRM_DATA_DIR=/data \
CRM_FRONTEND_DIR=/app/frontend
WORKDIR /app
RUN apt-get update \
&& apt-get install -y --no-install-recommends ca-certificates curl \
&& rm -rf /var/lib/apt/lists/*
# ── Python dependencies ─────────────────────────────────────────
# Only one hard dep for now: `cryptography` is required by the Gmail
# integration's RS256 JWT signing (DWD bearer tokens). Everything else
# server.py needs is stdlib.
RUN pip install --no-cache-dir cryptography==42.0.5
# ── Application source ──────────────────────────────────────────
COPY backend/server.py /app/backend/server.py
COPY backend/email_integration /app/backend/email_integration
COPY frontend /app/frontend
# ── StartOS wrapper scripts ─────────────────────────────────────
COPY start9/0.4/docker_entrypoint.sh /usr/local/bin/docker_entrypoint.sh
COPY start9/0.4/healthcheck.sh /usr/local/bin/healthcheck.sh
RUN chmod +x /usr/local/bin/docker_entrypoint.sh \
/usr/local/bin/healthcheck.sh
EXPOSE 8080
ENTRYPOINT ["/usr/local/bin/docker_entrypoint.sh"]
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Ten31
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+4
View File
@@ -0,0 +1,4 @@
# overrides to s9pk.mk must precede the include statement
ARCHES := x86
include s9pk.mk
+50 -7
View File
@@ -1,9 +1,52 @@
# Start9 Wrapper (0.4 placeholder)
# Ten31 Database — StartOS 0.4 wrapper (x86_64)
This directory is reserved for the StartOS 0.4 package wrapper.
This directory is the self-contained StartOS 0.4 service package for
Ten31 Database. It is the x86_64 successor to the 0.3.5 (aarch64)
wrapper in `../0.3.5/`. Both packages share the same package id
(`ten-database`) and the same `/data` volume layout so data can be
preserved across the migration.
Migration plan from 0.3.5:
1. Keep package id stable (`ten-database`) if StartOS migration path allows.
2. Keep mounted data directory contract unchanged (`/data/crm.db`, `/data/backups`).
3. Rebuild wrapper files against 0.4 packaging spec and verify with current start-sdk.
4. Test upgrade on a staging node using production backup restore before live cutover.
## Start here
**Read `DEPLOY_040.md` first.** It covers:
1. How the image-seed data-preservation mechanism works.
2. How to refresh the seed with live production data from the 0.3.5 host
(via `./refresh_seed.sh` or manual scp).
3. How to install the build prerequisites (Node, Docker, `start-cli`).
4. How to build the x86_64 `.s9pk`.
5. How to sideload onto the StartOS 0.4 beta node.
6. A rollback plan and a post-install verification checklist.
## Quick cheat sheet
```sh
# From this directory:
./refresh_seed.sh embassy@embassy.local # pull live prod data into seed/
make clean
make x86
make install # uses ~/.startos/config.yaml
```
## Data layout (unchanged from 0.3.5)
Inside the container:
- `/data/crm.db` — SQLite database
- `/data/backups/` — app-level JSON exports
- `/data/.crm-secret` — JWT signing key (created on first boot if absent)
The entrypoint seeds an empty volume from the image's baked-in snapshot on
first boot, and is a no-op for every later boot. Existing volumes are
never overwritten.
## Status
- Source scaffold: complete and `tsc --noEmit` clean against
`@start9labs/start-sdk` 0.4.0.
- Dockerfile: self-contained under `start9/0.4/` with no cross-folder
references to `start9/0.3.5/`.
- Seed snapshot: present at `seed/data/` (repo dev DB — replace with live
prod data before building).
- Not yet built into a `.s9pk` here; build on a machine with Docker +
`start-cli` per `DEPLOY_040.md`.
+11
View File
@@ -0,0 +1,11 @@
Ten31 Database is a self-hosted investor CRM and fundraising database.
This StartOS 0.4 package is the x86_64 successor to the 0.3.5 (aarch64) wrapper. It preserves the original runtime data layout inside the service volume:
- `/data/crm.db` — SQLite database (investors, contacts, fundraising grid, views, users, backups, feature requests, app settings)
- `/data/backups/` — app-level JSON snapshot exports
- `/data/.crm-secret` — JWT signing key (generated on first boot if absent)
First boot seeds the service volume from a snapshot baked into the image so the new install comes up with existing data already populated. The seed is skipped if the volume already contains a `crm.db`, so it is safe to reinstall or restore from a future StartOS 0.4 backup without losing data.
The wrapper's only differences from upstream are StartOS container wiring, the private web interface on internal port 8080, and backup integration (the whole `main` volume is included in StartOS backups).
+61
View File
@@ -0,0 +1,61 @@
#!/bin/sh
# ═══════════════════════════════════════════════════════════════
# Ten31 Database container entrypoint (StartOS 0.4 wrapper)
# ═══════════════════════════════════════════════════════════════
#
# Responsibilities:
# 1. Ensure the mounted /data volume directories exist.
# 2. Ensure a persistent CRM_SECRET_KEY exists so issued JWTs
# survive container restarts.
# 3. Launch the Python backend server.
#
# Note: This entrypoint NO LONGER seeds /data from a baked-in
# snapshot. The 0.3.5 → 0.4 migration is complete; from 0.1.0:40
# forward the live /data volume on the StartOS host is the sole
# source of truth. StartOS preserves /data across sideloads, so
# upgrades will not disturb live data.
# ═══════════════════════════════════════════════════════════════
set -eu
DATA_DIR="${CRM_DATA_DIR:-/data}"
SECRET_FILE="$DATA_DIR/.crm-secret"
SECRETS_DIR="$DATA_DIR/secrets"
EMAIL_ATTACHMENTS_DIR="$DATA_DIR/email_attachments"
GMAIL_SA_KEY="$SECRETS_DIR/gmail-service-account.json"
mkdir -p "$DATA_DIR" "$DATA_DIR/backups" "$SECRETS_DIR" "$EMAIL_ATTACHMENTS_DIR"
# /data/secrets holds the Gmail service-account key; lock it down so only
# the container user can read the directory. chmod on the file itself is
# the operator's responsibility when they drop the key in.
chmod 700 "$SECRETS_DIR" 2>/dev/null || true
# ── Persistent JWT secret ───────────────────────────────────────
if [ -z "${CRM_SECRET_KEY:-}" ]; then
if [ -f "$SECRET_FILE" ]; then
CRM_SECRET_KEY="$(cat "$SECRET_FILE")"
else
CRM_SECRET_KEY="$(head -c 48 /dev/urandom | base64 | tr -d '\n' | tr '/+' 'ab')"
printf '%s' "$CRM_SECRET_KEY" > "$SECRET_FILE"
chmod 600 "$SECRET_FILE"
fi
export CRM_SECRET_KEY
fi
# ── Gmail integration env vars ──────────────────────────────────
# The integration is enabled only if the service-account key file is
# actually present on the /data volume. This makes the package
# self-disabling on fresh installs until an operator drops the key in.
if [ -f "$GMAIL_SA_KEY" ]; then
export CRM_GMAIL_INTEGRATION_ENABLED="${CRM_GMAIL_INTEGRATION_ENABLED:-true}"
export CRM_GMAIL_AUTH_METHOD="${CRM_GMAIL_AUTH_METHOD:-dwd}"
export CRM_GMAIL_SA_KEY_PATH="${CRM_GMAIL_SA_KEY_PATH:-$GMAIL_SA_KEY}"
export CRM_GMAIL_WORKSPACE_DOMAIN="${CRM_GMAIL_WORKSPACE_DOMAIN:-ten31.xyz}"
export CRM_GMAIL_SYNC_INTERVAL_MIN="${CRM_GMAIL_SYNC_INTERVAL_MIN:-180}"
echo "[entrypoint] Gmail integration: ENABLED (key at $GMAIL_SA_KEY)"
else
echo "[entrypoint] Gmail integration: DISABLED (no key at $GMAIL_SA_KEY)"
fi
# ── Launch the app ──────────────────────────────────────────────
exec python3 /app/backend/server.py
+9
View File
@@ -0,0 +1,9 @@
#!/bin/sh
# Container-side health probe for the Ten31 Database service.
# The StartOS 0.4 daemon uses checkPortListening at the platform
# level, but this script is kept for parity with the 0.3.5 wrapper
# and so the same image can be exec'd directly for diagnostics.
set -eu
PORT="${CRM_PORT:-8080}"
curl -fsS "http://127.0.0.1:${PORT}/api/health" >/dev/null
+43
View File
@@ -0,0 +1,43 @@
<?xml version="1.0" encoding="UTF-8"?>
<svg id="Layer_1" data-name="Layer 1" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 722.69 280.85">
<defs>
<style>
.cls-1 {
font-family: LTCGoudyOldstylePro-Bold, 'LTC Goudy Oldstyle Pro';
font-size: 192px;
font-weight: 700;
}
.cls-1, .cls-2, .cls-3 {
fill: #fff;
}
.cls-2, .cls-4 {
stroke-width: 3px;
}
.cls-2, .cls-4, .cls-3 {
stroke: #fff;
stroke-miterlimit: 10;
}
.cls-4 {
fill: none;
}
.cls-5 {
letter-spacing: -.06em;
}
</style>
</defs>
<text class="cls-1" transform="translate(120.54 208.45)"><tspan class="cls-5" x="0" y="0">T</tspan><tspan x="120.96" y="0">en31</tspan></text>
<g>
<polygon class="cls-3" points="95.52 140.42 54.54 154.4 54.54 126.45 95.52 140.42"/>
<line class="cls-2" x1="0" y1="140.42" x2="60.54" y2="140.42"/>
</g>
<rect class="cls-4" x="97.1" y="1.5" width="527.95" height="277.85"/>
<g>
<polygon class="cls-3" points="721.15 140.42 680.16 154.4 680.16 126.45 721.15 140.42"/>
<line class="cls-2" x1="625.62" y1="140.42" x2="686.16" y2="140.42"/>
</g>
</svg>

After

Width:  |  Height:  |  Size: 1.2 KiB

+330
View File
@@ -0,0 +1,330 @@
{
"name": "ten-database-startos-040",
"lockfileVersion": 3,
"requires": true,
"packages": {
"": {
"name": "ten-database-startos-040",
"dependencies": {
"@start9labs/start-sdk": "^0.4.0-beta.66"
},
"devDependencies": {
"@types/node": "^22.19.0",
"@vercel/ncc": "^0.38.4",
"prettier": "^3.6.2",
"typescript": "^5.9.3"
}
},
"node_modules/@iarna/toml": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/@iarna/toml/-/toml-3.0.0.tgz",
"integrity": "sha512-td6ZUkz2oS3VeleBcN+m//Q6HlCFCPrnI0FZhrt/h4XqLEdOyYp2u21nd8MdsR+WJy5r9PTDaHTDDfhf4H4l6Q==",
"license": "ISC"
},
"node_modules/@noble/curves": {
"version": "1.9.7",
"resolved": "https://registry.npmjs.org/@noble/curves/-/curves-1.9.7.tgz",
"integrity": "sha512-gbKGcRUYIjA3/zCCNaWDciTMFI0dCkvou3TL8Zmy5Nc7sJ47a0jtOeZoTaMxkuqRo9cRhjOdZJXegxYE5FN/xw==",
"license": "MIT",
"dependencies": {
"@noble/hashes": "1.8.0"
},
"engines": {
"node": "^14.21.3 || >=16"
},
"funding": {
"url": "https://paulmillr.com/funding/"
}
},
"node_modules/@noble/hashes": {
"version": "1.8.0",
"resolved": "https://registry.npmjs.org/@noble/hashes/-/hashes-1.8.0.tgz",
"integrity": "sha512-jCs9ldd7NwzpgXDIf6P3+NrHh9/sD6CQdxHyjQI+h/6rDNo88ypBxxz45UDuZHz9r3tNz7N/VInSVoVdtXEI4A==",
"license": "MIT",
"engines": {
"node": "^14.21.3 || >=16"
},
"funding": {
"url": "https://paulmillr.com/funding/"
}
},
"node_modules/@start9labs/start-sdk": {
"version": "0.4.0",
"resolved": "https://registry.npmjs.org/@start9labs/start-sdk/-/start-sdk-0.4.0.tgz",
"integrity": "sha512-PFfO7tV9nzQFZL3KXaZyf16C5VZtM+dCDlRhLHpmwssTKtcjyCEhBrB9locuS2yFqu69rj+5kLFzCWDHeRRibg==",
"license": "MIT",
"dependencies": {
"@iarna/toml": "^3.0.0",
"@noble/curves": "^1.8.2",
"@noble/hashes": "^1.7.2",
"@types/ini": "^4.1.1",
"deep-equality-data-structures": "^2.0.0",
"fast-xml-parser": "^5.5.6",
"ini": "^5.0.0",
"isomorphic-fetch": "^3.0.0",
"mime": "^4.0.7",
"yaml": "^2.7.1",
"zod": "^4.3.6",
"zod-deep-partial": "^1.2.0"
}
},
"node_modules/@types/ini": {
"version": "4.1.1",
"resolved": "https://registry.npmjs.org/@types/ini/-/ini-4.1.1.tgz",
"integrity": "sha512-MIyNUZipBTbyUNnhvuXJTY7B6qNI78meck9Jbv3wk0OgNwRyOOVEKDutAkOs1snB/tx0FafyR6/SN4Ps0hZPeg==",
"license": "MIT"
},
"node_modules/@types/node": {
"version": "22.19.17",
"resolved": "https://registry.npmjs.org/@types/node/-/node-22.19.17.tgz",
"integrity": "sha512-wGdMcf+vPYM6jikpS/qhg6WiqSV/OhG+jeeHT/KlVqxYfD40iYJf9/AE1uQxVWFvU7MipKRkRv8NSHiCGgPr8Q==",
"dev": true,
"license": "MIT",
"dependencies": {
"undici-types": "~6.21.0"
}
},
"node_modules/@vercel/ncc": {
"version": "0.38.4",
"resolved": "https://registry.npmjs.org/@vercel/ncc/-/ncc-0.38.4.tgz",
"integrity": "sha512-8LwjnlP39s08C08J5NstzriPvW1SP8Zfpp1BvC2sI35kPeZnHfxVkCwu4/+Wodgnd60UtT1n8K8zw+Mp7J9JmQ==",
"dev": true,
"license": "MIT",
"bin": {
"ncc": "dist/ncc/cli.js"
}
},
"node_modules/deep-equality-data-structures": {
"version": "2.0.0",
"resolved": "https://registry.npmjs.org/deep-equality-data-structures/-/deep-equality-data-structures-2.0.0.tgz",
"integrity": "sha512-qgrUr7MKXq7VRN+WUpQ48QlXVGL0KdibAoTX8KRg18lgOgqbEKMAW1WZsVCtakY4+XX42pbAJzTz/DlXEFM2Fg==",
"license": "MIT",
"dependencies": {
"object-hash": "^3.0.0"
}
},
"node_modules/fast-xml-builder": {
"version": "1.1.4",
"resolved": "https://registry.npmjs.org/fast-xml-builder/-/fast-xml-builder-1.1.4.tgz",
"integrity": "sha512-f2jhpN4Eccy0/Uz9csxh3Nu6q4ErKxf0XIsasomfOihuSUa3/xw6w8dnOtCDgEItQFJG8KyXPzQXzcODDrrbOg==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
],
"license": "MIT",
"dependencies": {
"path-expression-matcher": "^1.1.3"
}
},
"node_modules/fast-xml-parser": {
"version": "5.5.12",
"resolved": "https://registry.npmjs.org/fast-xml-parser/-/fast-xml-parser-5.5.12.tgz",
"integrity": "sha512-nUR0q8PPfoA/svPM43Gup7vLOZWppaNrYgGmrVqrAVJa7cOH4hMG6FX9M4mQ8dZA1/ObGZHzES7Ed88hxEBSJg==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
],
"license": "MIT",
"dependencies": {
"fast-xml-builder": "^1.1.4",
"path-expression-matcher": "^1.5.0",
"strnum": "^2.2.3"
},
"bin": {
"fxparser": "src/cli/cli.js"
}
},
"node_modules/ini": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/ini/-/ini-5.0.0.tgz",
"integrity": "sha512-+N0ngpO3e7cRUWOJAS7qw0IZIVc6XPrW4MlFBdD066F2L4k1L6ker3hLqSq7iXxU5tgS4WGkIUElWn5vogAEnw==",
"license": "ISC",
"engines": {
"node": "^18.17.0 || >=20.5.0"
}
},
"node_modules/isomorphic-fetch": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/isomorphic-fetch/-/isomorphic-fetch-3.0.0.tgz",
"integrity": "sha512-qvUtwJ3j6qwsF3jLxkZ72qCgjMysPzDfeV240JHiGZsANBYd+EEuu35v7dfrJ9Up0Ak07D7GGSkGhCHTqg/5wA==",
"license": "MIT",
"dependencies": {
"node-fetch": "^2.6.1",
"whatwg-fetch": "^3.4.1"
}
},
"node_modules/mime": {
"version": "4.1.0",
"resolved": "https://registry.npmjs.org/mime/-/mime-4.1.0.tgz",
"integrity": "sha512-X5ju04+cAzsojXKes0B/S4tcYtFAJ6tTMuSPBEn9CPGlrWr8Fiw7qYeLT0XyH80HSoAoqWCaz+MWKh22P7G1cw==",
"funding": [
"https://github.com/sponsors/broofa"
],
"license": "MIT",
"bin": {
"mime": "bin/cli.js"
},
"engines": {
"node": ">=16"
}
},
"node_modules/node-fetch": {
"version": "2.7.0",
"resolved": "https://registry.npmjs.org/node-fetch/-/node-fetch-2.7.0.tgz",
"integrity": "sha512-c4FRfUm/dbcWZ7U+1Wq0AwCyFL+3nt2bEw05wfxSz+DWpWsitgmSgYmy2dQdWyKC1694ELPqMs/YzUSNozLt8A==",
"license": "MIT",
"dependencies": {
"whatwg-url": "^5.0.0"
},
"engines": {
"node": "4.x || >=6.0.0"
},
"peerDependencies": {
"encoding": "^0.1.0"
},
"peerDependenciesMeta": {
"encoding": {
"optional": true
}
}
},
"node_modules/object-hash": {
"version": "3.0.0",
"resolved": "https://registry.npmjs.org/object-hash/-/object-hash-3.0.0.tgz",
"integrity": "sha512-RSn9F68PjH9HqtltsSnqYC1XXoWe9Bju5+213R98cNGttag9q9yAOTzdbsqvIa7aNm5WffBZFpWYr2aWrklWAw==",
"license": "MIT",
"engines": {
"node": ">= 6"
}
},
"node_modules/path-expression-matcher": {
"version": "1.5.0",
"resolved": "https://registry.npmjs.org/path-expression-matcher/-/path-expression-matcher-1.5.0.tgz",
"integrity": "sha512-cbrerZV+6rvdQrrD+iGMcZFEiiSrbv9Tfdkvnusy6y0x0GKBXREFg/Y65GhIfm0tnLntThhzCnfKwp1WRjeCyQ==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
],
"license": "MIT",
"engines": {
"node": ">=14.0.0"
}
},
"node_modules/prettier": {
"version": "3.8.2",
"resolved": "https://registry.npmjs.org/prettier/-/prettier-3.8.2.tgz",
"integrity": "sha512-8c3mgTe0ASwWAJK+78dpviD+A8EqhndQPUBpNUIPt6+xWlIigCwfN01lWr9MAede4uqXGTEKeQWTvzb3vjia0Q==",
"dev": true,
"license": "MIT",
"bin": {
"prettier": "bin/prettier.cjs"
},
"engines": {
"node": ">=14"
},
"funding": {
"url": "https://github.com/prettier/prettier?sponsor=1"
}
},
"node_modules/strnum": {
"version": "2.2.3",
"resolved": "https://registry.npmjs.org/strnum/-/strnum-2.2.3.tgz",
"integrity": "sha512-oKx6RUCuHfT3oyVjtnrmn19H1SiCqgJSg+54XqURKp5aCMbrXrhLjRN9TjuwMjiYstZ0MzDrHqkGZ5dFTKd+zg==",
"funding": [
{
"type": "github",
"url": "https://github.com/sponsors/NaturalIntelligence"
}
],
"license": "MIT"
},
"node_modules/tr46": {
"version": "0.0.3",
"resolved": "https://registry.npmjs.org/tr46/-/tr46-0.0.3.tgz",
"integrity": "sha512-N3WMsuqV66lT30CrXNbEjx4GEwlow3v6rr4mCcv6prnfwhS01rkgyFdjPNBYd9br7LpXV1+Emh01fHnq2Gdgrw==",
"license": "MIT"
},
"node_modules/typescript": {
"version": "5.9.3",
"resolved": "https://registry.npmjs.org/typescript/-/typescript-5.9.3.tgz",
"integrity": "sha512-jl1vZzPDinLr9eUt3J/t7V6FgNEw9QjvBPdysz9KfQDD41fQrC2Y4vKQdiaUpFT4bXlb1RHhLpp8wtm6M5TgSw==",
"dev": true,
"license": "Apache-2.0",
"bin": {
"tsc": "bin/tsc",
"tsserver": "bin/tsserver"
},
"engines": {
"node": ">=14.17"
}
},
"node_modules/undici-types": {
"version": "6.21.0",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-6.21.0.tgz",
"integrity": "sha512-iwDZqg0QAGrg9Rav5H4n0M64c3mkR59cJ6wQp+7C4nI0gsmExaedaYLNO44eT4AtBBwjbTiGPMlt2Md0T9H9JQ==",
"dev": true,
"license": "MIT"
},
"node_modules/webidl-conversions": {
"version": "3.0.1",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-3.0.1.tgz",
"integrity": "sha512-2JAn3z8AR6rjK8Sm8orRC0h/bcl/DqL7tRPdGZ4I1CjdF+EaMLmYxBHyXuKL849eucPFhvBoxMsflfOb8kxaeQ==",
"license": "BSD-2-Clause"
},
"node_modules/whatwg-fetch": {
"version": "3.6.20",
"resolved": "https://registry.npmjs.org/whatwg-fetch/-/whatwg-fetch-3.6.20.tgz",
"integrity": "sha512-EqhiFU6daOA8kpjOWTL0olhVOF3i7OrFzSYiGsEMB8GcXS+RrzauAERX65xMeNWVqxA6HXH2m69Z9LaKKdisfg==",
"license": "MIT"
},
"node_modules/whatwg-url": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/whatwg-url/-/whatwg-url-5.0.0.tgz",
"integrity": "sha512-saE57nupxk6v3HY35+jzBwYa0rKSy0XR8JSxZPwgLr7ys0IBzhGviA1/TUGJLmSVqs8pb9AnvICXEuOHLprYTw==",
"license": "MIT",
"dependencies": {
"tr46": "~0.0.3",
"webidl-conversions": "^3.0.0"
}
},
"node_modules/yaml": {
"version": "2.8.3",
"resolved": "https://registry.npmjs.org/yaml/-/yaml-2.8.3.tgz",
"integrity": "sha512-AvbaCLOO2Otw/lW5bmh9d/WEdcDFdQp2Z2ZUH3pX9U2ihyUY0nvLv7J6TrWowklRGPYbB/IuIMfYgxaCPg5Bpg==",
"license": "ISC",
"bin": {
"yaml": "bin.mjs"
},
"engines": {
"node": ">= 14.6"
},
"funding": {
"url": "https://github.com/sponsors/eemeli"
}
},
"node_modules/zod": {
"version": "4.3.6",
"resolved": "https://registry.npmjs.org/zod/-/zod-4.3.6.tgz",
"integrity": "sha512-rftlrkhHZOcjDwkGlnUtZZkvaPHCsDATp4pGpuOOMDaTdDDXF91wuVDJoWoPsKX/3YPQ5fHuF3STjcYyKr+Qhg==",
"license": "MIT",
"funding": {
"url": "https://github.com/sponsors/colinhacks"
}
},
"node_modules/zod-deep-partial": {
"version": "1.4.4",
"resolved": "https://registry.npmjs.org/zod-deep-partial/-/zod-deep-partial-1.4.4.tgz",
"integrity": "sha512-aWkPl7hVStgE01WzbbSxCgX4O+sSpgt8JOjvFUtMTF75VgL6MhWQbiZi+AWGN85SfSTtI9gsOtL1vInoqfDVaA==",
"license": "MIT",
"peerDependencies": {
"zod": "^4.1.13"
}
}
}
}
+23
View File
@@ -0,0 +1,23 @@
{
"name": "ten-database-startos-040",
"scripts": {
"build": "rm -rf ./javascript && ncc build startos/index.ts -o ./javascript",
"prettier": "prettier --write startos",
"check": "tsc --noEmit"
},
"dependencies": {
"@start9labs/start-sdk": "^0.4.0-beta.66"
},
"devDependencies": {
"@types/node": "^22.19.0",
"@vercel/ncc": "^0.38.4",
"prettier": "^3.6.2",
"typescript": "^5.9.3"
},
"prettier": {
"trailingComma": "all",
"tabWidth": 2,
"semi": false,
"singleQuote": true
}
}
+116
View File
@@ -0,0 +1,116 @@
#!/bin/bash
# ═══════════════════════════════════════════════════════════════
# refresh_seed.sh
# Pull the live Ten31 Database data off a StartOS 0.3.5 host
# and stage it as the seed snapshot baked into the 0.4 image.
# ═══════════════════════════════════════════════════════════════
#
# Usage:
# ./refresh_seed.sh <ssh-user@host> [remote-data-dir]
#
# Examples:
# ./refresh_seed.sh start9@192.168.1.50
# ./refresh_seed.sh embassy@embassy.local \
# /embassy-data/package-data/volumes/ten-database/data/main
#
# What it does:
# 1. Finds the remote /data directory for the ten-database service.
# 2. Copies crm.db, backups/, and (optionally) .crm-secret into
# start9/0.4/seed/data/ on this machine.
# 3. Prints a row-count summary so you can verify content.
#
# After it finishes, run:
# make clean && make x86
# from this (start9/0.4/) directory to rebuild the .s9pk.
# ═══════════════════════════════════════════════════════════════
set -eu
if [ $# -lt 1 ]; then
echo "Usage: $0 <ssh-user@host> [remote-data-dir]"
echo ""
echo "Remote data dir defaults (tried in order):"
echo " /embassy-data/package-data/volumes/ten-database/data/main"
echo " /mnt/embassy-os/package-data/volumes/ten-database/data/main"
echo " /var/lib/embassy/services/ten-database/data"
exit 1
fi
REMOTE="$1"
REMOTE_DIR="${2:-}"
SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)"
SEED_DIR="$SCRIPT_DIR/seed/data"
echo ""
echo " Staging production seed from $REMOTE"
echo " into $SEED_DIR"
echo ""
# Auto-detect remote data dir if not supplied
if [ -z "$REMOTE_DIR" ]; then
echo " Probing for remote data directory..."
for candidate in \
"/embassy-data/package-data/volumes/ten-database/data/main" \
"/mnt/embassy-os/package-data/volumes/ten-database/data/main" \
"/var/lib/embassy/services/ten-database/data"; do
if ssh "$REMOTE" "[ -f \"$candidate/crm.db\" ]" 2>/dev/null; then
REMOTE_DIR="$candidate"
echo " found: $REMOTE_DIR"
break
fi
done
if [ -z "$REMOTE_DIR" ]; then
echo " Could not auto-detect a valid data directory with crm.db on $REMOTE."
echo " Re-run this script and pass the path explicitly as the 2nd argument."
exit 2
fi
fi
mkdir -p "$SEED_DIR/backups"
echo ""
echo " Copying crm.db ..."
scp "$REMOTE:$REMOTE_DIR/crm.db" "$SEED_DIR/crm.db"
echo " Copying backups/ (if present) ..."
if ssh "$REMOTE" "[ -d \"$REMOTE_DIR/backups\" ]" 2>/dev/null; then
scp -r "$REMOTE:$REMOTE_DIR/backups/." "$SEED_DIR/backups/" || true
else
echo " (none found, skipping)"
fi
echo " Copying .crm-secret (optional — keeps existing JWTs valid) ..."
if ssh "$REMOTE" "[ -f \"$REMOTE_DIR/.crm-secret\" ]" 2>/dev/null; then
read -r -p " Include .crm-secret in the baked image? [y/N] " ans
case "$ans" in
[yY]*) scp "$REMOTE:$REMOTE_DIR/.crm-secret" "$SEED_DIR/.crm-secret" ;;
*) echo " skipping .crm-secret; a fresh secret will be generated on first boot" ;;
esac
else
echo " (no .crm-secret on remote)"
fi
echo ""
echo " Summary of staged seed:"
ls -la "$SEED_DIR"
echo ""
if command -v python3 >/dev/null 2>&1 && [ -f "$SEED_DIR/crm.db" ]; then
python3 - <<PY
import sqlite3
db = sqlite3.connect("$SEED_DIR/crm.db")
cur = db.cursor()
cur.execute("PRAGMA integrity_check")
print(" integrity_check:", cur.fetchone()[0])
for t in ("users","fundraising_state","fundraising_funds","fundraising_views",
"contacts","organizations","audit_log","feature_requests","app_settings"):
try:
cur.execute(f"SELECT COUNT(*) FROM {t}")
print(f" {t:30s} {cur.fetchone()[0]} rows")
except Exception as e:
print(f" {t}: n/a ({e})")
PY
fi
echo ""
echo " Seed refreshed. Next: cd $(dirname "$SCRIPT_DIR")/0.4 && make clean && make x86"
+130
View File
@@ -0,0 +1,130 @@
# ** Plumbing. DO NOT EDIT **.
# This file is imported by ./Makefile. Make edits there.
PACKAGE_ID := $(shell awk -F"'" '/id:/ {print $$2}' startos/manifest/index.ts)
INGREDIENTS := $(shell start-cli s9pk list-ingredients 2>/dev/null)
REPO_GIT_DIR := ../../.git
ARCHES ?= x86 arm riscv
TARGETS ?= arches
ifdef VARIANT
BASE_NAME := $(PACKAGE_ID)_$(VARIANT)
else
BASE_NAME := $(PACKAGE_ID)
endif
.PHONY: all arches aarch64 x86_64 riscv64 arm arm64 x86 riscv arch/* clean install check-deps check-init package ingredients
.DELETE_ON_ERROR:
.SECONDARY:
define SUMMARY
@manifest=$$(start-cli s9pk inspect $(1) manifest); \
size=$$(du -h $(1) | awk '{print $$1}'); \
title=$$(printf '%s' "$$manifest" | jq -r .title); \
version=$$(printf '%s' "$$manifest" | jq -r .version); \
arches=$$(printf '%s' "$$manifest" | jq -r '[.images[].arch // []] | flatten | unique | join(", ")'); \
sdkv=$$(printf '%s' "$$manifest" | jq -r .sdkVersion); \
gitHash=$$(printf '%s' "$$manifest" | jq -r .gitHash | sed -E 's/(.*-modified)$$/\x1b[0;31m\1\x1b[0m/'); \
printf "\n"; \
printf "\033[1;32m✅ Build Complete!\033[0m\n"; \
printf "\n"; \
printf "\033[1;37m $$title\033[0m \033[36mv$$version\033[0m\n"; \
printf "───────────────────────────────\n"; \
printf " \033[1;36mFilename:\033[0m %s\n" "$(1)"; \
printf " \033[1;36mSize:\033[0m %s\n" "$$size"; \
printf " \033[1;36mArch:\033[0m %s\n" "$$arches"; \
printf " \033[1;36mSDK:\033[0m %s\n" "$$sdkv"; \
printf " \033[1;36mGit:\033[0m %s\n" "$$gitHash"; \
echo ""
endef
all: $(TARGETS)
arches: $(ARCHES)
universal: $(BASE_NAME).s9pk
$(call SUMMARY,$<)
arch/%: $(BASE_NAME)_%.s9pk
$(call SUMMARY,$<)
x86 x86_64: arch/x86_64
arm arm64 aarch64: arch/aarch64
riscv riscv64: arch/riscv64
$(BASE_NAME).s9pk: $(INGREDIENTS) $(REPO_GIT_DIR)/HEAD $(REPO_GIT_DIR)/index
@$(MAKE) --no-print-directory ingredients
@echo " Packing '$@'..."
start-cli s9pk pack -o $@
$(BASE_NAME)_%.s9pk: $(INGREDIENTS) $(REPO_GIT_DIR)/HEAD $(REPO_GIT_DIR)/index
@$(MAKE) --no-print-directory ingredients
@echo " Packing '$@'..."
start-cli s9pk pack --arch=$* -o $@
ingredients: $(INGREDIENTS)
@echo " Re-evaluating ingredients..."
install: | check-deps check-init
@HOST=$$(awk -F'/' '/^host:/ {print $$3}' ~/.startos/config.yaml); \
if [ -z "$$HOST" ]; then \
echo "Error: You must define \"host: http://server-name.local\" in ~/.startos/config.yaml"; \
exit 1; \
fi; \
S9PK=$$(ls -t *.s9pk 2>/dev/null | head -1); \
if [ -z "$$S9PK" ]; then \
echo "Error: No .s9pk file found. Run 'make' first."; \
exit 1; \
fi; \
printf "\n Installing %s to %s ...\n" "$$S9PK" "$$HOST"; \
start-cli package install -s "$$S9PK"
publish: | all
@REGISTRY=$$(awk -F'/' '/^registry:/ {print $$3}' ~/.startos/config.yaml); \
if [ -z "$$REGISTRY" ]; then \
echo "Error: You must define \"registry: https://my-registry.tld\" in ~/.startos/config.yaml"; \
exit 1; \
fi; \
S3BASE=$$(awk -F'/' '/^s9pk-s3base:/ {print $$3}' ~/.startos/config.yaml); \
if [ -z "$$S3BASE" ]; then \
echo "Error: You must define \"s3base: https://s9pks.my-s3-bucket.tld\" in ~/.startos/config.yaml"; \
exit 1; \
fi; \
command -v s3cmd >/dev/null || \
(echo "Error: s3cmd not found. It must be installed to publish using s3." && exit 1); \
printf "\n Publishing to %s; indexing on %s ...\n" "$$S3BASE" "$$REGISTRY"; \
for s9pk in *.s9pk; do \
age=$$(( $$(date +%s) - $$(stat -f %m "$$s9pk" 2>/dev/null || stat -c %Y "$$s9pk") )); \
if [ "$$age" -gt 3600 ]; then \
printf "\033[1;33m⚠️ %s is %d minutes old. Publish anyway? [y/N] \033[0m" "$$s9pk" "$$((age / 60))"; \
read -r ans; \
case "$$ans" in [yY]*) ;; *) echo "Skipping $$s9pk"; continue ;; esac; \
fi; \
start-cli s9pk publish "$$s9pk"; \
done
check-deps:
@command -v start-cli >/dev/null || \
(echo "Error: start-cli not found. Please see https://docs.start9.com/latest/developer-guide/sdk/installing-the-sdk" && exit 1)
@command -v npm >/dev/null || \
(echo "Error: npm not found. Please install Node.js and npm." && exit 1)
check-init:
@if [ ! -f ~/.startos/developer.key.pem ]; then \
echo "Initializing StartOS developer environment..."; \
start-cli init-key; \
fi
javascript/index.js: $(shell find startos -type f) tsconfig.json node_modules
npm run build
node_modules: package-lock.json
npm ci
package-lock.json: package.json
npm i
clean:
@echo "Cleaning up build artifacts..."
@rm -rf $(PACKAGE_ID).s9pk $(PACKAGE_ID)_x86_64.s9pk $(PACKAGE_ID)_aarch64.s9pk $(PACKAGE_ID)_riscv64.s9pk javascript node_modules
+58
View File
@@ -0,0 +1,58 @@
# Seed Snapshot (baked into the Docker image)
Anything under `seed/data/` is copied into the container image at build time
and placed at `/app/seed/data/`. On first boot, if `/data/crm.db` is not
present on the StartOS service volume, `docker_entrypoint.sh` copies
`/app/seed/data/.` into `/data/.` so the new 0.4 install starts with the
preserved data instead of an empty database.
## What's currently baked in
Initial snapshot was taken from the repo-level `data/` directory at build
time (the same DB that the 0.3.5 dev workflow pointed at). Files:
- `seed/data/crm.db` — SQLite database (investors, contacts, fundraising
rows, views, feature_requests, users, app_settings, etc.)
- `seed/data/backups/*.json` — app-level snapshot exports
## Refreshing the seed before a build
If you want the 0.4 deploy to come up with the absolute latest production
state from the 0.3.5 StartOS server, replace the files in `seed/data/`
BEFORE running `make`:
```sh
# 1) On the 0.3.5 StartOS server, take a fresh app backup and/or grab
# the live database file:
# /media/embassy/services/ten-database/data/crm.db (canonical)
# /media/embassy/services/ten-database/data/backups/*.json (optional)
# /media/embassy/services/ten-database/data/.crm-secret (optional)
# Exact path may differ by StartOS 0.3.5 build.
#
# 2) scp them into this folder:
scp embassy@<old-host>:/media/.../ten-database/data/crm.db \
start9/0.4/seed/data/crm.db
# (Optional) include backups + secret:
scp embassy@<old-host>:/media/.../ten-database/data/backups/* \
start9/0.4/seed/data/backups/
scp embassy@<old-host>:/media/.../ten-database/data/.crm-secret \
start9/0.4/seed/data/.crm-secret
#
# 3) Rebuild:
cd start9/0.4 && make clean && make x86
```
## Keeping `.crm-secret` out of the image
By default `seed/data/.crm-secret` is NOT included. The first boot on the
new machine generates a fresh JWT secret. Existing password hashes in
`crm.db` remain valid, so users just log in once on the new host.
If you WANT to preserve the exact secret (so already-issued JWTs remain
valid), drop the file at `seed/data/.crm-secret` and rebuild.
## Safety
The entrypoint never overwrites an existing `/data/crm.db`. If the volume
already contains data (StartOS restore, manual SSH pre-seed, prior install)
the seed is skipped and a `.seeded` marker is written.
+3
View File
@@ -0,0 +1,3 @@
import { sdk } from '../sdk'
export const actions = sdk.Actions.of()
+7
View File
@@ -0,0 +1,7 @@
import { sdk } from './sdk'
export const { createBackup, restoreInit } = sdk.setupBackups(async () =>
// Preserve the entire service volume so crm.db, backup JSON files, and the
// persisted JWT secret all remain compatible with the prior package layout.
sdk.Backups.ofVolumes('main'),
)
+5
View File
@@ -0,0 +1,5 @@
import { sdk } from './sdk'
export const setDependencies = sdk.setupDependencies(async () => {
return {}
})
+1
View File
@@ -0,0 +1 @@
export const i18n = (text: string) => text
+13
View File
@@ -0,0 +1,13 @@
/**
* Plumbing. DO NOT EDIT.
*/
export { createBackup } from './backups'
export { main } from './main'
export { init, uninit } from './init'
export { actions } from './actions'
import { buildManifest } from '@start9labs/start-sdk'
import { manifest as sdkManifest } from './manifest'
import { versionGraph } from './versions'
export const manifest = buildManifest(versionGraph, sdkManifest)
+16
View File
@@ -0,0 +1,16 @@
import { sdk } from '../sdk'
import { setDependencies } from '../dependencies'
import { setInterfaces } from '../interfaces'
import { versionGraph } from '../versions'
import { actions } from '../actions'
import { restoreInit } from '../backups'
export const init = sdk.setupInit(
restoreInit,
versionGraph,
setInterfaces,
setDependencies,
actions,
)
export const uninit = sdk.setupUninit(versionGraph)
+25
View File
@@ -0,0 +1,25 @@
import { i18n } from './i18n'
import { sdk } from './sdk'
import { WEB_PORT } from './utils'
export const setInterfaces = sdk.setupInterfaces(async ({ effects }) => {
const uiMulti = sdk.MultiHost.of(effects, 'ui-multi')
const uiMultiOrigin = await uiMulti.bindPort(WEB_PORT, {
protocol: 'http',
})
const ui = sdk.createInterface(effects, {
name: i18n('Web UI'),
id: 'ui',
description: i18n('The web interface of Ten31 Database'),
type: 'ui',
masked: false,
schemeOverride: null,
username: null,
path: '',
query: {},
})
const uiReceipt = await uiMultiOrigin.export([ui])
return [uiReceipt]
})
+33
View File
@@ -0,0 +1,33 @@
import { i18n } from './i18n'
import { sdk } from './sdk'
import { DATA_MOUNT_PATH, IMAGE_ID, WEB_PORT } from './utils'
export const main = sdk.setupMain(async ({ effects }) => {
console.info(i18n('Starting Ten31 Database'))
return sdk.Daemons.of(effects).addDaemon('primary', {
subcontainer: await sdk.SubContainer.of(
effects,
{ imageId: IMAGE_ID },
sdk.Mounts.of().mountVolume({
volumeId: 'main',
subpath: null,
mountpoint: DATA_MOUNT_PATH,
readonly: false,
}),
'ten31-database-main',
),
exec: {
command: ['/usr/local/bin/docker_entrypoint.sh'],
},
ready: {
display: i18n('Web Interface'),
fn: () =>
sdk.healthCheck.checkPortListening(effects, WEB_PORT, {
successMessage: i18n('CRM API is responding.'),
errorMessage: i18n('CRM API is not responding.'),
}),
},
requires: [],
})
})
+13
View File
@@ -0,0 +1,13 @@
export const short = {
en_US: 'Self-hosted investor and fundraising database for Ten31.',
}
export const long = {
en_US:
'Ten31 Database is an Airtable-style investor CRM with fundraising grid, communications logging, views, backups, and CSV import. This StartOS 0.4 wrapper preserves the existing /data layout for upgrade-safe persistence.',
}
export const alertUpdate = {
en_US:
'This 0.4 package is designed to keep using the existing /data/crm.db, /data/backups, and /data/.crm-secret layout from the 0.3.5.1 package.',
}
+35
View File
@@ -0,0 +1,35 @@
import { setupManifest } from '@start9labs/start-sdk'
import { alertUpdate, long, short } from './i18n'
export const manifest = setupManifest({
id: 'ten-database',
title: 'Ten31 Database',
license: 'MIT',
packageRepo: 'https://github.com/ten31/ten31-database-startos',
upstreamRepo: 'https://github.com/ten31/ten31-database',
marketingUrl: 'https://ten31.vc',
donationUrl: null,
docsUrls: ['https://docs.start9.com/packaging/0.4.0.x/'],
description: { short, long },
volumes: ['main'],
images: {
main: {
source: {
dockerBuild: {
dockerfile: './Dockerfile',
workdir: '../..',
},
},
arch: ['x86_64', 'aarch64'],
},
},
alerts: {
install: null,
update: alertUpdate,
uninstall: null,
restore: null,
start: null,
stop: null,
},
dependencies: {},
})
+9
View File
@@ -0,0 +1,9 @@
import { StartSdk } from '@start9labs/start-sdk'
import { manifest } from './manifest'
/**
* Plumbing. DO NOT EDIT.
*
* The exported `sdk` const is used throughout this package codebase.
*/
export const sdk = StartSdk.of().withManifest(manifest).build(true)
+16
View File
@@ -0,0 +1,16 @@
// Informational constants shared across the startos/ modules.
// The authoritative id, title and version for the package come
// from manifest/index.ts (id, title) and versions/ (version).
export const PACKAGE_ID = 'ten-database'
export const PACKAGE_TITLE = 'Ten31 Database'
// ExVer form of the current 0.4 wrapper release (upstream 0.1.0, wrapper rev 41).
// * 0.3.5 wrapper: 0.1.0.38 (legacy, aarch64)
// * First 0.4: 0.1.0:39 (shipped seed snapshot for migration)
// * Cleanup: 0.1.0:40 (seed removed + multi-threaded server + abuser auto-ban)
// * Current: 0.1.0:41 (frontend persists auth across refreshes)
export const PACKAGE_VERSION = '0.1.0:41'
export const DATA_MOUNT_PATH = '/data'
export const WEB_PORT = 8080
export const IMAGE_ID = 'main'
export const VOLUME_ID = 'main'
+11
View File
@@ -0,0 +1,11 @@
import { VersionGraph } from '@start9labs/start-sdk'
import { v_0_1_0_39 } from './v0.1.0.39'
import { v_0_1_0_40 } from './v0.1.0.40'
import { v_0_1_0_41 } from './v0.1.0.41'
import { v_0_1_0_42 } from './v0.1.0.42'
import { v_0_1_0_43 } from './v0.1.0.43'
export const versionGraph = VersionGraph.of({
current: v_0_1_0_43,
other: [v_0_1_0_39, v_0_1_0_40, v_0_1_0_41, v_0_1_0_42],
})
+38
View File
@@ -0,0 +1,38 @@
import { VersionInfo } from '@start9labs/start-sdk'
// First StartOS 0.4 release of Ten31 Database.
//
// Upgrade context:
// * The 0.3.5 wrapper shipped at 0.1.0.38 (legacy, aarch64 only).
// * This 0.4 wrapper is built for x86_64 and is intended for a
// parallel install on a new StartOS 0.4 host.
// * Data continuity is NOT handled by a StartOS-level in-place
// upgrade (that path does not exist across StartOS majors).
// Instead the container image is pre-seeded with a snapshot of
// /data (crm.db, backups/, optional .crm-secret). On first boot
// docker_entrypoint.sh copies that snapshot into the mounted
// `main` volume if it is empty.
//
// Because both "up" and "down" paths are inside the same wrapper
// lineage (and the first 0.4 release has no earlier 0.4 version
// to migrate from), the migration functions are intentionally
// no-ops. Future 0.4.x releases can chain off this node in the
// version graph.
export const v_0_1_0_39 = VersionInfo.of({
version: '0.1.0:39',
releaseNotes: {
en_US: [
'First StartOS 0.4 package for Ten31 Database.',
'Built for x86_64; sideload-only during beta.',
'Container image ships with a baked-in /data snapshot so the',
'service boots with the existing investor and fundraising data,',
'saved views, backups, users, and app settings already in place.',
'No StartOS-level migration is performed from the 0.3.5 package;',
'this package is installed fresh on a 0.4 host.',
].join(' '),
},
migrations: {
up: async () => {},
down: async () => {},
},
})
+57
View File
@@ -0,0 +1,57 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Post-migration cleanup + hardening release.
//
// Context:
// * 0.1.0:39 was the first 0.4 package and shipped a baked-in
// /data snapshot that docker_entrypoint.sh copied into the
// mounted `main` volume on first boot (only if the volume was
// empty). That snapshot did its job and the live host now has
// a populated /data with all real investor + fundraising data.
// * 0.1.0:40 removes the seed snapshot from the image and the
// seeding logic from the entrypoint. The live /data volume is
// the sole source of truth from here on. StartOS preserves the
// volume across sideloads, so this upgrade does not disturb
// any data — it just slims the image and removes a code path
// that should never run again.
// * 0.1.0:40 also hardens the backend HTTP server against the
// vulnerability scanners that find the StartTunnel-exposed
// interface within hours of going live:
// - HTTPServer → ThreadingHTTPServer so one slow request or
// a wave of scanner probes can't block legit users.
// - Per-IP GET rate limit (default 600/min) in addition to
// the existing login/write limits.
// - 404-burst auto-ban: any IP that produces ABUSE_404_THRESHOLD
// 404s within ABUSE_404_WINDOW_SEC (default 15 in 60s) is
// parked on a class-level blacklist for ABUSE_BAN_SEC
// (default 15 minutes). Banned IPs get an instant 429 with
// no DB or filesystem work.
// - All limits stay tunable via env vars
// (CRM_GET_RATE_LIMIT_PER_MIN, CRM_ABUSE_404_THRESHOLD,
// CRM_ABUSE_404_WINDOW_SEC, CRM_ABUSE_BAN_SEC).
//
// No data migration is required: the SQLite schema is unchanged
// and the live DB on /data is left exactly as-is.
export const v_0_1_0_40 = VersionInfo.of({
version: '0.1.0:40',
releaseNotes: {
en_US: [
'Removes the baked-in /data seed snapshot now that the',
'0.3.5 → 0.4 migration is complete. The live /data volume',
'on the StartOS host is the sole source of truth and is',
'preserved across sideloads, so no live data is touched by',
'this upgrade. Image is smaller and the first-boot seeding',
'code path has been removed. Also hardens the backend',
'against vulnerability scanners hitting the public',
'StartTunnel interface: the HTTP server is now multi-threaded',
'so one slow request can no longer block legit users, GET',
'requests are rate-limited per IP, and any IP that bursts',
'too many 404s in a short window is auto-banned for 15',
'minutes with no DB work performed.',
].join(' '),
},
migrations: {
up: async () => {},
down: async () => {},
},
})
+42
View File
@@ -0,0 +1,42 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Frontend convenience release: persist auth across page reloads.
//
// Background: through 0.1.0:40 the auth token + user object were held only
// in React state in memory. Any refresh, tab close, or browser restart
// dropped the token and forced the user back to the login screen. Since
// the JWT is signed with /data/.crm-secret (which already survives sideloads
// and container restarts), the underlying token is still valid for its full
// 24-hour lifetime — we just weren't keeping it anywhere persistent.
//
// 0.1.0:41 stores the JWT and user object in localStorage on login (and
// rehydrates from there on app mount), so refreshes and reopened tabs stay
// signed in until the token expires. The api() helper now also dispatches
// a 'crm:unauthorized' event whenever an authenticated request comes back
// with a 401, and the AuthProvider listens for that event to clear the
// stored auth — so an expired or rejected token immediately bounces the
// user back to the login screen instead of leaving the app in a broken
// "loaded but every request fails" state.
//
// Backend is unchanged: the JWT still carries the user's true role and is
// re-verified on every request, so a tampered localStorage user object
// cannot escalate privileges (the next admin call would just 401/403).
//
// No data migration is required.
export const v_0_1_0_41 = VersionInfo.of({
version: '0.1.0:41',
releaseNotes: {
en_US: [
'Logins now persist across page refreshes and tab closures for',
'the full 24-hour token lifetime. Previously every reload bounced',
'you to the login screen even though the token was still valid.',
'If the server later rejects a stored token (expired, secret key',
'changed, etc.) the app automatically clears it and shows the',
'login screen instead of leaving requests silently failing.',
].join(' '),
},
migrations: {
up: async () => {},
down: async () => {},
},
})
+58
View File
@@ -0,0 +1,58 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Gmail integration — Phase 1.
//
// Background: the CRM previously had no ingestion path for email
// activity. Contacts were logged manually; correspondence history lived
// only in our mailboxes. This release adds a one-way capture pipeline
// that ingests sent and received mail for every Workspace user at
// ten31.xyz, matches messages against existing investor records, and
// records metadata (+ bodies and attachments for matched threads) into
// the CRM database.
//
// Auth model: domain-wide delegation via a Google service account. The
// service-account JSON key is stored on the /data volume at
// /data/secrets/gmail-service-account.json (chmod 600, operator-dropped).
// The integration is self-disabling: if the key file is absent, the
// scheduler doesn't start and /api/email/* routes return 503. No key →
// no behavior change from 0.1.0:41.
//
// When the key IS present, docker_entrypoint.sh auto-enables the
// integration and sets sensible defaults (3-hour sync interval, domain
// ten31.xyz, DWD auth). All defaults can still be overridden via env.
//
// Database: migration 0001 adds eight new tables under the email_
// namespace (emails, email_accounts, email_recipients,
// email_account_messages, email_attachments, email_threads,
// email_investor_links, email_sync_runs). All CREATE TABLE IF NOT EXISTS,
// so the migration is safely idempotent — re-applying is a no-op.
//
// Backend: wholly isolated under backend/email_integration/. Three tiny,
// feature-flag-guarded hooks in server.py (migration call, scheduler
// startup, /api/email/* route dispatch). Removing or disabling the
// integration leaves server behavior identical to 0.1.0:41.
//
// New Python dep: cryptography==42.0.5 (required for RS256 JWT signing
// in DWD bearer token exchange). Now installed in the image.
//
// No data migration code needed — new tables, additive only.
export const v_0_1_0_42 = VersionInfo.of({
version: '0.1.0:42',
releaseNotes: {
en_US: [
'Adds a Gmail capture pipeline. When a Google Workspace',
"service-account key is dropped into the server's /data/secrets",
'folder, the CRM begins pulling sent and received mail for every',
'ten31.xyz user on a 3-hour cycle, matching messages against',
'existing investor records and storing metadata (plus bodies and',
'attachments for matched threads) in the database. With no key',
'present the feature is dormant and this release behaves',
'identically to 0.1.0:41. Eight new email_* tables are added',
'additively; no existing data is touched.',
].join(' '),
},
migrations: {
up: async () => {},
down: async () => {},
},
})
+44
View File
@@ -0,0 +1,44 @@
import { VersionInfo } from '@start9labs/start-sdk'
// Hotfix for 0.1.0:42.
//
// Issue 1 (critical): POST requests to /api/email/* hung indefinitely.
// server.py's do_POST called get_body() early in the dispatch to support
// /api/auth/login, which reads bytes off the request stream. My Gmail
// integration hook then ran route handlers that called get_body() a
// second time — but the stream was already drained, so the second read
// blocked waiting for bytes that never came. GET requests (which don't
// read a body) were unaffected.
//
// Fix: get_body() now caches the parsed JSON on the handler instance
// on first call. Repeat calls return the cached value. Handler
// instances are per-request in ThreadingHTTPServer, so the cache is
// naturally request-scoped and thread-safe.
//
// Issue 2 (minor): the /api/email/accounts/enroll endpoint required
// both `email_address` and `user_id` in the body, making it painful to
// call for the common single-admin-enrolling-themselves case.
//
// Fix: the endpoint now also accepts `email` as an alias, and if
// user_id isn't supplied it auto-resolves by looking up the email in
// the users table (falling back to the authenticated admin's own id
// if no match).
//
// No schema changes, no data migration.
export const v_0_1_0_43 = VersionInfo.of({
version: '0.1.0:43',
releaseNotes: {
en_US: [
'Hotfix for the Gmail integration in 0.1.0:42. POST requests to',
'/api/email/* endpoints were hanging because the request body was',
'being read twice from a single-shot stream. This release caches',
'the parsed body on the request so subsequent reads are safe, and',
'also relaxes the enroll endpoint to accept just an email and',
'auto-resolve the CRM user.',
].join(' '),
},
migrations: {
up: async () => {},
down: async () => {},
},
})
+11
View File
@@ -0,0 +1,11 @@
{
"include": ["startos/**/*.ts", "node_modules/**/startos"],
"compilerOptions": {
"target": "ES2018",
"module": "CommonJS",
"moduleResolution": "node",
"esModuleInterop": true,
"strict": true,
"skipLibCheck": true
}
}