Running & Monitoring
On this page 7 sections
Most of the day-to-day picture, node health, origin health, cluster topology, request volume, HEADLESS events, and licence status, is available without ever logging into a node. As long as your cluster can reach spooksystems.org:443, every node continuously reports its state to the control plane and the dashboard renders it in real time.
In-dashboard alerts
The control panel raises alerts automatically on the events you care about most:
- Node down. If a node stops heartbeating, the cluster page flags it red and surfaces an alert badge on the account overview within seconds.
- Origin degraded. When a node enters HEADLESS mode for an origin, the panel shows the origin in the “degraded” state with the headless reason and the age of the cache it is serving.
- Master moved / role change. Promotions and demotions inside a cluster are recorded and shown on the cluster timeline.
- Licence / quota issues. Expiring licences, over-quota node counts, and failed control-plane validations all surface as account-level notices.
SpookVis
SpookVis is the origin-scoped analytics tab in the control panel. It pulls from /api/vis/stats, the same aggregate stream every node reports into, and renders six widgets over the data set you select with the filter bar.
Filters
Three filter rows sit above the widgets and every widget reacts to them in real time:
- Domain, one tab per cluster on your account.
- Origin, one tab per origin group within the selected domain.
www.variants of a hostname are merged into the same group so the numbers stay sane. The row hides itself if the domain has only one group. - Period,
1D,1W,1M, orALL.
The six widgets
- Geo Heatmap
- A world map showing where your traffic is actually coming from, with your own QSCS nodes pinned on top so you can see at a glance how well your footprint matches your audience.
- Traffic
- Request volume over time for the selected origin, with the top source countries broken out alongside the total so you can spot regional shifts and spikes.
- Active Sessions
- A live feed of who is on the site right now, where they are, how recently they hit you, and the path they have taken through the origin.
- Substrate Traffic
- A side-by-side view of substrate (in-browser WASM) traffic versus plain external traffic, so you can see how much of your audience is benefiting from substrate caching and what they are requesting most recently.
- QSCS Mesh Stats
- The headline efficiency view: what proportion of requests the mesh answered without going to origin, and roughly how much egress that saved you. Single-node origins are recognised and shown as such rather than charted as empty.
- Per-Node Traffic
- How the work was actually distributed across your nodes over the selected period, split by substrate and origin traffic so you can see where the cluster is pulling its weight.
spooksystems.org:443 outbound. There is nothing to scrape, no Prometheus exporter to expose to the public internet, and no separate admin channel to firewall. If your nodes can talk to the control plane, the dashboard and SpookVis stay live.
The rest of this page covers the lower-level on-host commands and metrics endpoint, useful when you want to confirm something on a specific node or wire QSCS into your own monitoring stack.
Service control
| Goal | Command |
|---|---|
| Start | sudo systemctl start qscs |
| Stop | sudo systemctl stop qscs |
| Restart (after config change) | sudo systemctl restart qscs |
| Enable on boot | sudo systemctl enable qscs |
| Show current status | sudo systemctl status qscs |
Logs
QSCS logs to systemd's journal. Useful incantations:
# tail the live log
sudo journalctl -u qscs -f
# last 100 lines, no pager
sudo journalctl -u qscs -n 100 --no-pager
# only lines since the last restart
sudo journalctl -u qscs -b -u qscs --no-pager
# only warnings and errors
sudo journalctl -u qscs -p warningHealth from the outside
Send a normal HTTP request through QSCS:
curl -i -H "Host: <your-origin>" http://<node-ip>:8080/Look for one of these X-QSCS-Mode values: full, incremental, nochange, headless. Any of them means the daemon is running and replying.
Metrics
QSCS exposes a Prometheus-style metrics endpoint at http://127.0.0.1:8080/qscs/metrics. The endpoint is loopback-only by design, see Metrics & Health for details and the metric list.
What to watch in production
- The journal during deploys. Confirm "DomainRegistry: N peers, role=…" prints the role you expected on each node.
qscs_origin_degraded. Should normally be 0. A sustained 1 means an upstream is failing.qscs_origin_headless_requests_total. Should normally tick over slowly. A burst means QSCS is masking an outage.- Disk usage of
/opt/qscs. The cache is bounded, but verify on first deploys.
LogLevel = info prints only on subscribe, restart, role detection, and origin state transitions. If the journal is silent for hours, the node is doing what it should.