Alert system – Description
Severities
| Severity | Meaning |
|---|---|
| Alarm | A condition that needs attention now, such as a value outside its alarm limits or a satellite that may have gone silent. |
| Caution | A condition that may need attention, such as a value outside its caution limits. |
| Event | Something that happened once, such as a subsystem restart. Events do not clear; you acknowledge them. |
Each channel can have four limits: alarm low, caution low, caution high and alarm high. A channel without limits shows No limits rather than Nominal, so an empty limit is never mistaken for a good value.
What raises an alert
| Condition | Severity | Default threshold |
|---|---|---|
| Channel outside its caution limits | Caution | Set per channel |
| Channel outside its alarm limits | Alarm | Set per channel |
| No new decoded frames | Caution, then alarm | Caution after 12 hours; alarm after 24 hours |
| Satellite may have gone silent | Alarm | 3 scheduled observations in a row deliver nothing, counting passes at least 15° high, after a 3-hour upload grace period |
| Server cannot reach SatNOGS | Alarm | No successful fetch for three fetch intervals (90 minutes at the default 30) |
| SatNOGS rejects the API key, or no key is set | Alarm | Immediately |
| Server disk getting full | Caution, then alarm | Caution below 10% or 1 GB free; alarm below 3% or 300 MB free |
| Subsystem restart | Event | A restart counter goes up |
Channels that drift away from their learned normal are flagged on the channel and in the spacecraft status, and sent to the phone, separately from the limit alerts.
Alert states
Open, not acknowledged: new. Shown at the top of the list with an Acknowledge button.
Open, acknowledged: an operator has seen it, but the condition is still active.
Shelved: hidden for a chosen time (1 hour, 6 hours, 1 day or 1 week) with a reason. Shelved alerts are listed under Shelved below the main list.
Closed: acknowledged, and the condition has cleared.
An alert closes only when it has been acknowledged and its condition has cleared. A condition that clears by itself without acknowledgement stays in the list, so nothing is missed between shifts.
Phone alerts
If the server has an ntfy topic set, the fetch job pushes these to the operator's phone: alarm-level limit breaks, restarts, a satellite that may have gone silent, a channel starting to drift, three failed fetches in a row and low disk space. Each event is sent once. A morning summary of the last 24 hours is also sent at the configured hour.