This is the complete procedure for recovering an Ntrospect platform onto a replacement server after the loss of the original host: what must be prepared in advance (the survival kit), the recovery itself in six ordered phases — from a bare machine to a fully restored platform with its data — and the troubleshooting of each step.
Follow the phases in order; each one builds on the last. Keep a printed copy with your disaster-recovery materials, and confirm the survival kit is in place today — every item in it must exist somewhere other than the platform server before it is needed.
What must survive the server (the survival kit)
- The archive storage — the backup destination: a NAS share (NFS/SMB) or S3 bucket holding the
.ntbakand.nttelarchives. Configure it in Settings → Backup & Recovery → Destination → Network share (the platform mounts it itself) — never leave backups on the platform's own disk. - The backup passphrase — written down where the server's death cannot reach it. Confirm the fingerprint shown on the Configure tab matches your note. Without it, every archive is noise. It cannot be recovered.
- The release bundle (
ntersect-observability-<ver>-<arch>.tar.gz) — your install media. Rebuild it whenever platform code changes; a stale bundle restores yesterday's bugs. - Warm/cold tier data — lives on its own external storage with its own protection. The platform's backups do not carry it; the hot-tier snapshot covers everything newer than the hot→warm boundary.
What each archive kind is
| File | Contains | Cadence | Pick at restore |
|---|---|---|---|
.ntbak |
Configuration: fleet, dashboards, alert rules, settings, secrets/keyring, TLS CA, share definitions, tier layout, table identities | Every 15 min (anchorable) | Newest — complete in itself, never combined |
.nttel |
Hot-tier data: metrics, logs, flows, traces rows | Your chosen interval; current + one previous kept | Newest — the previous generation exists in case the newest is damaged |
The recovery, in order
The order matters and is enforced by mechanics, not convention: configuration first (it brings back the passphrase file and recreates the telemetry tables as the same tables), restart, then data.
Phase 0 — a fresh machine
- Install the OS, plus the two client packages the appliance cannot bundle:
nfs-common(NFS) andcifs-utils(SMB). - Unpack and start:
tar xzf <bundle>.tar.gz && cd <dir> && ./run.sh - Grant the narrow mount privilege once:
sudo ./scripts/setup-share-mounts.sh <service-user> - Open the UI. It is a blank platform. That is correct.
Phase 1 — connect the archive storage
Under Settings → Backup & Recovery → Configure → Destination → Network share:
- + Add share → NFS (server + export) or SMB (+ username/password) → Add & mount → green
mountedchip. - Click use → amber selected — save to apply → click Save changes → solid in use.
The trap this rehearsal hit three times: use fills the form; only Save changes commits it. The Restore tab reads the saved destination.
Phase 2 — restore the configuration
Under Backup & Recovery → Restore:
- Available archives lists the share's
.ntbakfiles. Select the newest. - The ceremony: Pause the platform, and type this machine's hostname to confirm.
- Enter the archive passphrase. Stage restore. A grey button names the missing precondition beside it — note that any restart clears the pause.
- Read the staged card: the table counts (recognize your platform in these numbers) and the key note — on a replacement server it reports a key merge: the archive's key becomes primary; this machine's own young key is preserved (
secret.key.superseded) as decrypt-only. Both eras stay readable. - Click Restart platform now and watch: requested → down (ticking clock) → ✓ platform is back — the restore has been applied → reload.
Phase 3 — verify identity before touching data
- The fleet is back (check the agent count) · the network share remounted itself · Configure loads with backups enabled, on the share, schedule anchor intact · the Gateway TLS panel shows the same CA fingerprint you recorded.
- Not automatic: rendered TLS artifacts (
api.crt, collector overlay) — the restored CA can re-issue them; per-gateway client certs never left the gateways.
Phase 4 — restore the hot-tier data
Under Backup & Recovery → Restore → Hot tier (telemetry) recovery:
- Newest
.nttel→ Preview (reads the whole archive, reports rows and range, writes nothing) → Restore rows (enabled only after a matching preview). Additive inserts — the worst wrong-archive outcome is duplicate rows, never an emptied table. - Restored data is historical — widen chart time ranges before concluding anything is missing.
Phase 5 — reattach the storage tiers
Under Settings → Storage Tier:
- Stage definitions restored with the configuration; their locations are physical facts — the platform refuses to invent a missing tier path.
- Network share tab → pick the share (
/warmand/coldsubdirectories keep stages separate) → enable → Apply layout → the chip flips to in use. "Selected — save to apply" after Apply means the Apply did not happen. - Reattached tier data is recognized by table identity (carried in every
.ntbak). For storage that cannot be reattached, Recover data from a drive (rescue import) copies — never moves — from any directory of ClickHouse parts.
Phase 6 — the fleet reconnects
- Gateways buffered telemetry in their disk spools throughout the outage and replay it on reconnect — the outage window costs delay, not data.
- The gap that is lost: last hot-tier snapshot → failure (RPO = your snapshot interval). Run snapshots hourly if that number matters.
Troubleshooting the recovery
| Symptom | Meaning |
|---|---|
| "the NFS server refused this host" | The NAS export list must include this machine's IP; a Kerberos-only export is unsupported — use SMB with credentials |
| "no export at that path" | Check the export path on the NAS |
| "server rejected the username or password" | SMB credential is wrong |
| Share mounted but Stage lists nothing | Destination not saved — the amber chip is still amber |
| Stage button grey, no obvious reason | It names the missing precondition beside it; a restart clears the maintenance pause |
| "a passphrase is required" | Enter it in the Archive passphrase box; on the original server the stored file answers instead |
| Charts empty after the data restore | Widen the time range — restored data is historical |
| Tier stage "not reachable on this host" | The location does not exist here yet — reattach the storage, then Apply |