← Migration Readiness Report Readiness Runbook Restore script

Paperclip Cutover Runbook

Phase 5 — Documentation only. This runbook is a controlled, reversible procedure. Nothing in this document is to be executed during authoring. All steps are gated behind the Go/No-Go gate (see below) and require recorded sign-offs. The legacy host 44.205.14.86 (".86") stays live throughout to guarantee instant rollback.

0. Overview & Topology

Item	Value
Target host (new)	`98.84.88.134` ("the .134")
Legacy host (rollback)	`44.205.14.86` (".86") — keep live
Cutover DNS record	`paperclip.searskairos.ai` (A-record)
Cutover window	TBD — fill in at execution time
Runbook owner	Srini
Rollback authority	Srini (may be invoked unilaterally)

Required sign-offs (all three required before Go)

Area	Sign-off owner	Notes
AskPulse	Ankam	Confirms AskPulse agent secrets + routines ready
Infrastructure	Rajesh	Confirms host, DNS, ngrok, backup target
Executive	Eddie	Via Srini ONLY. Never message Eddie directly. Srini relays and records Eddie's approval.

1. Pre-Cutover Checklist

Complete and check off all items before approaching the Go/No-Go gate. Do not flip anything here — these are read-only verifications and confirmations.

Target host 98.84.88.134 reachable over SSH and healthy (uptime, disk, memory verified).
Application deployed and running on .134 in standby (email disabled, ngrok not enabled, crons commented, routines paused).
Current DNS state for paperclip.searskairos.ai captured and recorded (see Step 1 pre-check).
Current TTL of the A-record recorded; TTL lowered (e.g. to 60s) at least one TTL-period before the window to speed rollback. (Done in advance, not at cutover.)
Fresh database dump from production source available and checksummed.
bridge-views.sql present on .134, version-pinned, and reviewed.
Backup of .134 application .env taken (/opt/paperclip/.env.prebrief.bak).
Values for SHSAI_ACTIVITY_DIGEST_TO and DTCC_NOTIFICATION_ALLOWLIST retrieved from the secrets vault and staged (NOT yet written).
ngrok-paperclip.service unit file installed on .134 but disabled.
Agent secrets list reviewed; the selective subset to bind identified and approved by Ankam.
Routines inventory reviewed; the selective subset to un-pause identified and approved by Ankam.
Heartbeat monitors identified; re-enable plan confirmed.
Backup target (new) confirmed reachable and writable; old target noted for rollback.
.86 confirmed still live and serving (rollback path intact).
Rollback owner (Srini) on call for the full window.
Comms channel open; Eddie's approval relayed and recorded by Srini.

2. Go / No-Go Gate

Do not proceed past this gate unless ALL of the following are TRUE. Any single NO = No-Go, abort, remain on .86.

#	Gate condition	Go?
G1	Pre-Cutover Checklist 100% complete	☐
G2	Ankam (AskPulse) sign-off recorded	☐
G3	Rajesh (infra) sign-off recorded	☐
G4	Eddie (exec) approval relayed by Srini and recorded	☐
G5	`.86` verified live and healthy (rollback path proven)	☐
G6	DNS TTL confirmed low (fast rollback possible)	☐
G7	Fresh data dump checksum verified	☐
G8	Rollback owner (Srini) present and acknowledged	☐

Gate decision: ☐ GO ☐ NO-GO Recorded by: __________ Time: __________

If NO-GO at any point: stop, do not flip remaining steps, execute the Full Rollback for any steps already applied, and remain on .86.

3. Step 1 — Repoint DNS A-record

Point paperclip.searskairos.ai → 98.84.88.134.

Pre-check — capture and confirm current record (still pointing at .86):

dig +short paperclip.searskairos.ai A
# EXPECT: 44.205.14.86
dig +short paperclip.searskairos.ai A | tee /tmp/paperclip_dns_before.txt

Command — set the A-record to the new host (example uses Route 53; adapt to your DNS provider):

cat > /tmp/r53-cutover.json <<'JSON'
{
  "Comment": "Paperclip cutover: point to .134",
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "paperclip.searskairos.ai.",
      "Type": "A",
      "TTL": 60,
      "ResourceRecords": [{ "Value": "98.84.88.134" }]
    }
  }]
}
JSON
aws route53 change-resource-record-sets \
  --hosted-zone-id "$PAPERCLIP_ZONE_ID" \
  --change-batch file:///tmp/r53-cutover.json

Post-verify — confirm propagation to the new host:

dig +short paperclip.searskairos.ai A @1.1.1.1
# EXPECT: 98.84.88.134
curl -sS -o /dev/null -w '%{http_code}\n' https://paperclip.searskairos.ai/healthz
# EXPECT: 200

ROLLBACK — repoint DNS back to .86:

cat > /tmp/r53-rollback.json <<'JSON'
{
  "Comment": "Paperclip ROLLBACK: point to .86",
  "Changes": [{
    "Action": "UPSERT",
    "ResourceRecordSet": {
      "Name": "paperclip.searskairos.ai.",
      "Type": "A",
      "TTL": 60,
      "ResourceRecords": [{ "Value": "44.205.14.86" }]
    }
  }]
}
JSON
aws route53 change-resource-record-sets \
  --hosted-zone-id "$PAPERCLIP_ZONE_ID" \
  --change-batch file:///tmp/r53-rollback.json
dig +short paperclip.searskairos.ai A @1.1.1.1   # EXPECT: 44.205.14.86

4. Step 2 — Fresh data re-restore, THEN re-run `bridge-views.sql`

Order is mandatory: restore first, bridge-views second. Re-running bridge-views before the restore completes will bind views to stale/partial tables.

Pre-check — verify dump integrity and snapshot current DB for rollback:

# Verify the fresh dump checksum matches the recorded value
sha256sum -c /opt/paperclip/data/fresh_dump.sql.gz.sha256
# EXPECT: fresh_dump.sql.gz: OK

# Take a rollback snapshot of the CURRENT .134 database BEFORE touching it
pg_dump -Fc -h localhost -U paperclip paperclip \
  > /opt/paperclip/data/rollback_predump_$(date +%Y%m%d_%H%M%S).dump
ls -lh /opt/paperclip/data/rollback_predump_*.dump   # EXPECT: non-zero file

Command — restore fresh data, THEN run bridge-views:

# 1) Restore fresh data
gunzip -c /opt/paperclip/data/fresh_dump.sql.gz \
  | psql -h localhost -U paperclip -d paperclip -v ON_ERROR_STOP=1

# 2) ONLY after restore succeeds, run bridge-views
psql -h localhost -U paperclip -d paperclip -v ON_ERROR_STOP=1 \
  -f /opt/paperclip/sql/bridge-views.sql

Post-verify — confirm data freshness and views resolve:

psql -h localhost -U paperclip -d paperclip -At \
  -c "SELECT max(ingested_at) FROM source_events;"
# EXPECT: timestamp within the fresh dump window

psql -h localhost -U paperclip -d paperclip -At \
  -c "SELECT count(*) FROM bridge_active_view;"
# EXPECT: non-zero, matches expected row count

ROLLBACK — restore the pre-cutover .134 snapshot (drops fresh data + bridge-views):

psql -h localhost -U paperclip -d postgres -v ON_ERROR_STOP=1 -c \
  "DROP DATABASE paperclip; CREATE DATABASE paperclip OWNER paperclip;"
pg_restore -h localhost -U paperclip -d paperclip \
  /opt/paperclip/data/rollback_predump_<TIMESTAMP>.dump
# (If staying on .86 entirely, this DB rollback is optional — .86 is authoritative.)

5. Step 3 — Enable email + restore notification recipients

Set PAPERCLIP_EMAIL_ENABLED=true and restore SHSAI_ACTIVITY_DIGEST_TO and DTCC_NOTIFICATION_ALLOWLIST.

Pre-check — confirm email currently disabled and recipients currently empty/safe:

grep -E 'PAPERCLIP_EMAIL_ENABLED|SHSAI_ACTIVITY_DIGEST_TO|DTCC_NOTIFICATION_ALLOWLIST' \
  /opt/paperclip/.env
# EXPECT: PAPERCLIP_EMAIL_ENABLED=false ; the two recipient vars empty or unset

# Confirm backup of .env exists (from checklist)
ls -l /opt/paperclip/.env.prebrief.bak   # EXPECT: present

Command — write the values (staged from vault), then restart the service:

# Re-snapshot .env immediately before edit
cp -a /opt/paperclip/.env /opt/paperclip/.env.step3.bak

# Apply (values pulled from vault into shell env beforehand; not hardcoded here)
sed -i 's/^PAPERCLIP_EMAIL_ENABLED=.*/PAPERCLIP_EMAIL_ENABLED=true/' /opt/paperclip/.env
sed -i "s|^SHSAI_ACTIVITY_DIGEST_TO=.*|SHSAI_ACTIVITY_DIGEST_TO=${SHSAI_ACTIVITY_DIGEST_TO}|" /opt/paperclip/.env
sed -i "s|^DTCC_NOTIFICATION_ALLOWLIST=.*|DTCC_NOTIFICATION_ALLOWLIST=${DTCC_NOTIFICATION_ALLOWLIST}|" /opt/paperclip/.env

systemctl restart paperclip.service

Post-verify — confirm values loaded and a test send is gated to allowlist only:

systemctl show paperclip.service -p ActiveState   # EXPECT: ActiveState=active
grep PAPERCLIP_EMAIL_ENABLED /opt/paperclip/.env   # EXPECT: true
# Send a single canary to an internal allowlisted address only:
curl -sS -X POST https://paperclip.searskairos.ai/internal/email-canary \
  -H 'X-Cutover: 1'   # EXPECT: 202 Accepted; canary received internally

ROLLBACK — disable email and clear recipients (restore the backup):

cp -a /opt/paperclip/.env.step3.bak /opt/paperclip/.env
# or, surgically:
sed -i 's/^PAPERCLIP_EMAIL_ENABLED=.*/PAPERCLIP_EMAIL_ENABLED=false/' /opt/paperclip/.env
sed -i 's/^SHSAI_ACTIVITY_DIGEST_TO=.*/SHSAI_ACTIVITY_DIGEST_TO=/' /opt/paperclip/.env
sed -i 's/^DTCC_NOTIFICATION_ALLOWLIST=.*/DTCC_NOTIFICATION_ALLOWLIST=/' /opt/paperclip/.env
systemctl restart paperclip.service
grep PAPERCLIP_EMAIL_ENABLED /opt/paperclip/.env   # EXPECT: false

6. Step 4 — Enable ngrok tunnel

Enable and start ngrok-paperclip.service.

Pre-check — confirm unit is installed and currently disabled/stopped:

systemctl is-enabled ngrok-paperclip.service   # EXPECT: disabled
systemctl is-active  ngrok-paperclip.service   # EXPECT: inactive

Command:

systemctl enable --now ngrok-paperclip.service

Post-verify — confirm tunnel is up and reachable:

systemctl is-active ngrok-paperclip.service    # EXPECT: active
curl -sS http://127.0.0.1:4040/api/tunnels | jq -r '.tunnels[].public_url'
# EXPECT: an https public_url present

ROLLBACK — stop and disable the tunnel:

systemctl disable --now ngrok-paperclip.service
systemctl is-active ngrok-paperclip.service    # EXPECT: inactive

7. Step 5 — Bind agent secrets (selective), un-pause routines (selective), re-enable heartbeats

Selective = only the approved subset from the checklist (Ankam-approved). Do not bulk-enable.

Pre-check — confirm current paused/unbound state and capture inventory for rollback:

# Record which secrets are currently bound (for rollback diff)
paperclip-cli secrets list --bound > /tmp/secrets_bound_before.txt
# Record current routine states
paperclip-cli routines list --status > /tmp/routines_status_before.txt
# Record heartbeat state
paperclip-cli heartbeats status > /tmp/heartbeats_before.txt

Command — apply the approved selective lists only:

# Bind ONLY the approved subset (example names — use the Ankam-approved list)
for s in $(cat /opt/paperclip/cutover/approved_secrets.txt); do
  paperclip-cli secrets bind "$s"
done

# Un-pause ONLY the approved routines
for r in $(cat /opt/paperclip/cutover/approved_routines.txt); do
  paperclip-cli routines resume "$r"
done

# Re-enable heartbeats
paperclip-cli heartbeats enable --all

Post-verify:

paperclip-cli secrets list --bound        # EXPECT: approved subset now bound
paperclip-cli routines list --status      # EXPECT: approved subset = running
paperclip-cli heartbeats status           # EXPECT: heartbeats reporting healthy

ROLLBACK — re-pause routines, unbind the secrets just bound, disable heartbeats:

# Re-pause the routines we resumed
for r in $(cat /opt/paperclip/cutover/approved_routines.txt); do
  paperclip-cli routines pause "$r"
done
# Unbind the secrets we bound
for s in $(cat /opt/paperclip/cutover/approved_secrets.txt); do
  paperclip-cli secrets unbind "$s"
done
# Disable heartbeats
paperclip-cli heartbeats disable --all

8. Step 6 — Uncomment crons, point backup target

Pre-check — back up crontab and current backup config; confirm crons are still commented:

crontab -l > /tmp/crontab_before.txt
grep -n '^#' /tmp/crontab_before.txt        # EXPECT: paperclip cron lines commented
cp -a /opt/paperclip/backup.conf /opt/paperclip/backup.conf.bak
grep BACKUP_TARGET /opt/paperclip/backup.conf   # EXPECT: old target (record it)

Command — uncomment paperclip crons and repoint backup target:

# Uncomment only the paperclip-managed cron lines
sed -i '/# >>> paperclip >>>/,/# <<< paperclip <<</ s/^#\( \)\?//' \
  <(crontab -l) > /tmp/crontab_new.txt
crontab /tmp/crontab_new.txt

# Point backup at the new target
sed -i "s|^BACKUP_TARGET=.*|BACKUP_TARGET=${NEW_BACKUP_TARGET}|" /opt/paperclip/backup.conf

Post-verify:

crontab -l | grep -A20 '# >>> paperclip >>>'   # EXPECT: lines now active (uncommented)
grep BACKUP_TARGET /opt/paperclip/backup.conf  # EXPECT: new target
# Dry-run a backup to confirm target is writable
paperclip-cli backup run --dry-run             # EXPECT: success, writes to new target

ROLLBACK — recomment crons and restore old backup target:

crontab /tmp/crontab_before.txt                # restores commented crons
cp -a /opt/paperclip/backup.conf.bak /opt/paperclip/backup.conf
grep BACKUP_TARGET /opt/paperclip/backup.conf  # EXPECT: old target restored

9. Step 7 — Keep `.86` live for rollback

.86 is not decommissioned at cutover. It remains live, serving, and authoritative as the rollback target until cutover is declared stable (post-soak).

Pre-check / standing verification — .86 healthy throughout the window:

ssh ops@44.205.14.86 'uptime && systemctl is-active paperclip.service'
# EXPECT: active
curl -sS -o /dev/null -w '%{http_code}\n' --resolve paperclip.searskairos.ai:443:44.205.14.86 \
  https://paperclip.searskairos.ai/healthz
# EXPECT: 200 (proves .86 still serves the app directly)

Command — none. Explicitly do nothing to .86:

# INTENTIONALLY NO-OP. Do not stop, disable, or wipe .86.
echo "Leaving .86 (44.205.14.86) live for rollback. No action taken."

Post-verify — .86 still ready to take traffic on rollback:

curl -sS -o /dev/null -w '%{http_code}\n' --resolve paperclip.searskairos.ai:443:44.205.14.86 \
  https://paperclip.searskairos.ai/healthz   # EXPECT: 200

ROLLBACK — .86 IS the rollback. Repoint DNS to it (see Full Rollback). Decommission only after a clean soak and explicit sign-off (out of scope for this runbook).

10. Full Rollback

Invoke at any point on No-Go, failed post-verify, or incident. Rollback authority: Srini (may act unilaterally). Execute in this order (reverse of cutover); skip steps not yet applied.

Repoint DNS back to .86 (highest priority — restores user-facing traffic):

aws route53 change-resource-record-sets \
  --hosted-zone-id "$PAPERCLIP_ZONE_ID" \
  --change-batch file:///tmp/r53-rollback.json
dig +short paperclip.searskairos.ai A @1.1.1.1   # EXPECT: 44.205.14.86

Disable email + clear recipients (prevents the new host from sending):

sed -i 's/^PAPERCLIP_EMAIL_ENABLED=.*/PAPERCLIP_EMAIL_ENABLED=false/' /opt/paperclip/.env
sed -i 's/^SHSAI_ACTIVITY_DIGEST_TO=.*/SHSAI_ACTIVITY_DIGEST_TO=/' /opt/paperclip/.env
sed -i 's/^DTCC_NOTIFICATION_ALLOWLIST=.*/DTCC_NOTIFICATION_ALLOWLIST=/' /opt/paperclip/.env
systemctl restart paperclip.service

Disable ngrok tunnel:

systemctl disable --now ngrok-paperclip.service

Re-pause routines + unbind secrets + disable heartbeats (selective, reverse of Step 5):

for r in $(cat /opt/paperclip/cutover/approved_routines.txt); do paperclip-cli routines pause "$r"; done
for s in $(cat /opt/paperclip/cutover/approved_secrets.txt); do paperclip-cli secrets unbind "$s"; done
paperclip-cli heartbeats disable --all

Recomment crons + restore old backup target:

crontab /tmp/crontab_before.txt
cp -a /opt/paperclip/backup.conf.bak /opt/paperclip/backup.conf

(Optional) Restore .134 DB snapshot — only if .134 must be reverted; .86 is authoritative so this is usually unnecessary:

psql -h localhost -U paperclip -d postgres -c \
  "DROP DATABASE paperclip; CREATE DATABASE paperclip OWNER paperclip;"
pg_restore -h localhost -U paperclip -d paperclip \
  /opt/paperclip/data/rollback_predump_<TIMESTAMP>.dump

Confirm .86 serving and healthy:

curl -sS -o /dev/null -w '%{http_code}\n' --resolve paperclip.searskairos.ai:443:44.205.14.86 \
  https://paperclip.searskairos.ai/healthz   # EXPECT: 200

Post-rollback: notify Ankam and Rajesh directly; relay status to Eddie via Srini only. Record rollback time, trigger, and the last successfully applied step.

11. Sign-off Record

Role	Name	Notes
AskPulse	Ankam
Infrastructure	Rajesh
Executive (via Srini)	Eddie	Relayed by Srini — Eddie not contacted directly
Runbook owner / Rollback authority	Srini

Reminder: Eddie is only reached through Srini. Do not message Eddie directly under any circumstance — Srini relays both the request for approval and any rollback notifications.